Java Projects

Tools and techniques

Mike Godfrey
Information Technology Services / The University of Texas at Austin
 Outline

Look at three java projects

• Different architectures adopted

• Problems encountered and lessons learnt
• Tools and other useful techniques
 Who are we?
• Remains of centralized data processing
• Part of the unit providing IT services across campus
• Traditionally mainframe based applications with web
front-ends scripted in a simple in-house language
• Changing mission as our administrative application
development role is done by departments
• Project list that requires a much more powerful language
pushing adoption of Java
• Changes are forcing developer to acquire new skills and
ways of working – java camp.
 The Systems

• Clips – a web-based file system for distributing

course materials
• Jwebagent – a java runtime for our in-house
scripting language
• Tech lounge – a web-based developer
documentation store
Clips – file system
 Clips Requirements

• A simple to use, intuitive interface

• Handle all file types
• Accessible from any location
• Date controlled display of materials
• Easy transfer of materials from class to class
• Work well with existing mainframe class
management system
• Work within the UTDirect portal framework
 Basic Design Choices

• File system metaphor for user interface

• Browser access for both students and faculty
• Use native XML database to model the file
system in a recursive schema
• Store binary files in same XML database
• JSP with scriptlets for display
• clips interface
 Use of XML for Meta Data
• File system metaphor seems ideally suited to the tree
structure of XML
• Meta data structure based on two nodes types; directory-
nodes and binary-nodes
• One XML document contains all meta information
needed to build file structure for a given class
• Easy to re-build hierarchy of resources from XML tree
• Database uses schema to specify stored document
structure
 Meta-Data Details
• Directory nodes store information about a
directory: name, creation time, description, etc
• Binary nodes store information about a binary
file: name, foreign key, size, content type, etc
• Directory nodes can contain either binary nodes
or other directory nodes (hierarchy)
• clips meta data
 Clips Architecture

XML database

Tamino api
(binaries and

JSP presentation layer

Meta-data)

Helper User
Classes
Authorization and class info
Broker api

mainframe Broker
Natural
Existing class
Adabas management system
 New Issues for us

• Handling and auditing file uploads from web

pages (multipart encoded forms)
• Building and manipulating XML records
• How to best make large scale use of JSPs in a
complex Java system
• How to simplify communication with mainframe
 Useful Tools
• Log4j – used for debugging and recording
production exceptions (system written before
Java 1.4)
• JDOM – very user friendly api for manipulating
the XML
• cos.jar – used to parse the multipart encoded
forms
• Entire-X broker ACI for Java
• Java stub generator for mainframe modules
• Tamino - native XML database
 Lessons Learnt

• Be careful of scriplets in JSP; they expand to fill

the available space!
• Better to use a controller servlet to organize
navigation through JSPs
• XP and pair-programming works
• A Java developer’s best friend is Goggle
 Alternative JSP architecture

View
View Maintain Update Copy
jspMaintain
jsp jsp jsp jsp Controller
jsp
servlet Update
jsp
Copy
Auth Form Resource Copy jsp
Task Parser List Task
Update
Task Auth Form Resource Copy
Task Parser List Task
Update
Task

mainframe XML store

mainframe XML store
 Jwebagent

.WBX Jwebagent
Jwebagent HTML
file file
 What is Webagent?

• A scripting language with a syntax like Natural

(mainframe language)
• Scripts transformed into C++ objects forming a
virtual machine which is serialized to a .WBX file
• Runtime reads .WBX file, rebuilds the C++
virtual machine and executes it.
• Pro: easy and productive for creating web front
ends to mainframe applications.
• Con: limited functionality
 Why Jwebagent?

• Same as Webagent but the rebuilding of the

runtime is done in Java …plus
• Enable java classes to run existing Webagent
scripts
• Enable Webagent scripts to use java classes
• Improve execution speed – no process forking
and script buffering
 Main Runtime Elements
• Servlet – handles all execution requests for a WBX file
• Executor – coordinates loading file, running the VM, and
handling errors for an individual file request
• File loader – re-builds the serialized script objects
• Context objects – handle data specific to that particular
request
• VmOpCodes – functional elements of the VM
• Data objects – variables, literals
 Runtime

vmOp
broker vmOp
broker vmOp
broker vmOp
vmOp
file request vmOp
Serial code object

Servlet Executor Builds then

Script context
env
executes http http
Data buffer req resp
Request context
WBX
file
 Useful Tools
• Junit – unit testing framework (junit.org)
• JProbe – application analyzer (Quest Software)
• JMeter – load testing suite (Apache.org)
• HttpExplorer – gui http request and response
tool (in-house)
• Jedit – text editor (Jedit.org)
• IntelliJ IDEA – Java IDE (Jet Brains)
• CVS (Tortoise, WinCVS, Putty for SSH)
• Xvi32 – hex editor
 Junit – unit testing
• Provides easy way to set and teardown test
situations
• Automated running of many tests
• Organize tests in suites and suites into meta-
suites
• Text or gui UI for test reporting
• Refactor with confidence! (very valuable)
• Trick is to know what tests to write
• Good with Jwebagent as it is not database
driven
 JProbe – code analyser

• Threadalizer – finds deadlocks and conflicts (did

not show any problems for Jwebagent)
• Memory Debugger – shows memory use by
classes and can find hanging objects (useful –
strings to byte arrays)
• Profiler – tracks objects and the call pattern,
records method time and number of calls (most
useful tool –identify prime objects to optimize)
 JProbe Code Optimizations

• Strings to byte arrays – saved memory on string

generation
• io to nio gave 25% increase in through-put
• In-lining of frequently used methods to reduce
calling overhead
• Combining of related classes when frequently
constructed
• Un-doing OOP improved performance!
 JMeter – load testing
• Build and store test plans
• Control number of threads, ramp-up times and thread
delay to simulate different testing scenarios
• Test http apps, SOAP and web services, LDAP, JDBC,
etc
• Display test results in graphs, tables, files, etc
• Build tests with logical flows to suit application
• Apply pre and post requests during tests
• Very valuable for performance tuning (with JProbe)
 JVM Tuning

• Changed default of client mode to server mode

(20% improvement)
• Other settings have given some improvements
• Use incremental GC rather than ‘stop the world’
style
• Uses the flags that control survivor ratio for the
GC generations – we have short lived objects
• Increased default memory allocation
 Did we make it?
• First tests showed consistently slower times than the C+
+ version ( up to 33* slower)
• JProbe analysis and JVM tuning reduced this to about 2*
slower on average
• Through-put is slightly lower
• JMeter shows a mean that is much faster than C++, but
and average that is much higher
• Problem appears to be in the multi-treading of Broker
• Tried JNI against C++ broker and see same results
 Lessons learned

• Always build a test suite before refactoring

• Never optimize until you know what to optimize
• Do not assume GC defaults are optimal
• Java needs lots of memory!
• Allow for distractions in your project estimates
(estimated time 9 months, actual time start to
finish 2.5 years, actual time spent on project 10
months)
Tech Lounge
 Project Goals

• Nice, easy to use documentation store

• Replace mainframe based system
• Provide better editing and display options
• Allow limited access to subsets of articles
• Control the life-time of articles to ensure
relevancy
• Provide good search capabilities and cross-
referencing options
• Store multiple document formats
 Design Choices
• Store articles in native XML database
• Article content stored against a subset of XHTML strict
DTD – easy to display and construct
• Each article has a meta-data XML file for owners, dates,
related articles, number of reads, etc
• Article display via XSLT for HTML and pdf
• Article maintenance via jsps and custom tag libraries
• tech lounge
 Article Maintenance Architecture

New Article
Controller jsp
servlet Maintain
Article
jsp

Auth Form
Article Bean
Task Parser

mainframe XML store

 Useful Tools

• Many things from Apache

• DHTML
• Graphic designer

Created Tools/utilities
• Tamino Wrappers – abstraction layer
• UID (QUID-like foreign key generator)
 Problems/Mistakes

• Not using CVS well during first half of project

• Not writing unit tests
• Not using logging for debug messages
 Evolution

• Display of XML – moving towards all XSLT

• JSP usage – moving away from interlinked
pages towards controller server style
• JSP structure – scriplets bad, custom tags
good(?)
• Building up institutional tools/utilities and
standards that are increasing productivity
• Individual experts towards wide spread Java use
through in-house training courses
 Review of our Java Experience

Pros:
• More flexible and capable
• Greater abstraction possible, increased
elegance
• Many very powerful tools/utilities freely available
Cons:
• Requires higher competency of developers
• Less productive for our typical applications
• Needs larger machines
 Other projects

• PdaParser – code generation to ease

communication with mainframe modules
• Fatcookie – authentication
• Virtual File System extension to JEdit to work
with our unix server script editing proceedure
Current
• Jar management database (entirely XML/XSLT)
• Web service wrapper for mainframe modules
• Dynamic pdf generation from mainframe data
Questions?
 Useful Project Management Tip

Remember the Bullfighter!