Troubleshooting

Your Java
Applications
Lee Chuk Munn
Staff Engineer
Sun Microsystems
1
About This Presentation
• This is not a session on debugging
> As in an IDE with watches, breakpoints, etc
• Troubleshooting here referes to locating the
possible source of problem
> During deployment
> Without the benefit of source code
• Postmoterm analysis

2
Improvements in the JavaVM
http://weblogs.java.net/blog/mandychung/archive/2006/12/java_se_6_monit_1.html
• Dynamic attach
> Attach monitoring tools to a running JVM
> Uses JMX
• Heap dump and analysis
> Request heap dump on demand and analysis
• OutOfMemoryError diagnosability
> Allocation failure, automatic heap dump generation
• DTrace on Solairs
> Tracing the whole stack
• Troubleshooting guide
> http://java.sun.com/javase/6/webnotes/trouble/ 3
Garbage Collector

4
GC Responsibilities
• Garbage detection
> Garbage are objects that are no longer reachable
• Garbage reclamation
> Make space available to the running program again
• Variety of approaches
> “Stop-the-world” effectively freezes the heap
> Compacting / non compacting
> Algorithms: copying, mark-sweep, mark-compact, etc

5
Empirical Statistics
• Most objects are very short lived
> 80-98% of all newly allocated objects
die within a few million instructions
> 80-98% of all newly allocated objects
die before another megabyte has been allocated

6
Assumptions
• The weak generation hypothesis
> Most new objects will die young
> Concentrate effort on managing the 'young generation'
> Make allocate/manage/deallocate cycle fast and efficient
• For 'older' objects, manage as little as possible
• Keep young and old objects separate
• Use different GC algorithms for each generation
> Different requirements on each of these object groups

7
HotSpot VM Heap Layout

8
How Generational GC Works?

9
Generational GC

Track These Object Allocation

(Remembered Set)
Young Generation

Object Promotion

Old Generation

10
JDK Tools

11
jps
• List the JVMs that are currently running including
embeded Vms
> Associate a 'process number' with the running application
• jps
27798 Jps
25301 Main
• jps -s
28029 sun.tools.jps.Jps
25301 org.netbeans.Main
• jps -l
25301 Main -Djdk.home=/opt/java/javase/jdk1.6
-Dnetbeans.dirs=/opt/java/tools/netbeans-
6.0.1/nb6.0:/opt/java/tools/netbeans-
6.0.1/ide8:/opt/java/tools/netbeans-
6.0.1/java1:/opt/java/tools/netbeans- 12
 jinfo
• List configuration information from a running VM or
a core file
> Information includes VM properties and command line
flags
• jinfo <pid from jps>
Java System Properties:
java.vendor = Sun Microsystems Inc.
netbeans.user = /home/shulk/.netbeans/6.0
sun.java.launcher = SUN_STANDARD
sun.management.compiler = HotSpot Client Compiler
...
VM Flags:
-Djdk.home=/opt/java/javase/jdk1.6 -Dnetbeans.dirs=/opt/java/tools/netbeans-
6.0.1/nb6.0:/opt/java/tools/netbeans-6.0.1/ide8:/opt/java/tools...
13
 jstat
• List the statistics for a given VM
> Class loading, GC on all spaces, hotspot compilation
> See Troubleshooting Guide
• Provide a sample interval and the number of
samples to take
Process id Interval No. of sample

jstat -gcutil 25301 1000 10

S0 S1 E O P YGC YGCT FGC FGCT GCT
73.79 0.00 81.32 42.10 99.56 1344 8.880 36 17.363 26.243
73.79 0.00 81.32 42.10 99.56 1344 8.880 36 17.363 26.243
73.79 0.00 81.32 42.10 99.56 1344 8.880 36 17.363 26.243

14
jstack
• Prints the stack traces of all the the threads
attached to a virtual machine
> Application thread, internal VM thread,
• Also performs deadlock detection with -l option
• Use -F to force stack if VM is hung
> Solaris and Linux only

15
 jstack – Sample Output

"Java Source Worker Thread" prio=10 tid=0x08267800 nid=0x63a5 waiting on condition

[0x4532c000..0x4532d040]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x54b8d090> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
...
at java.util.concurrent.ThreadPoolExecuto$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)

Locked ownable synchronizers:

- <0x54b8cdd0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

16
jmap
• Prints memory related statistics
> Details the overall memory configuration
> Details section on each of the space with capacity, free
and used

17
jmap Sample Output
Mark Sweep Compact GC
Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 169869312 (162.0MB)
NewSize = 1048576 (1.0MB)
MaxNewSize = 4294901760 (4095.9375MB)
OldSize = 4194304 (4.0MB)
NewRatio = 12
SurvivorRatio = 8
PermSize = 33554432 (32.0MB)
MaxPermSize = 209715200 (200.0MB)
...
Eden Space:
capacity = 6881280 (6.5625MB)
used = 4437312 (4.23175048828125MB)
free = 2443968 (2.33074951171875MB)
64.48381696428571% used
18
HPROF
• Uses the JVMTI to get information from a Java VM
• Data capture includes
> CPU usages, heap dump, thread states
• Start at commandline
> java -Xrunhprof:<options> MyJavaApp
> Includes format=b for jhat analysis
• Useful options
> heap=all – displays all heap info
> cpu=sample – sample active thread

• Dump on application exit or CTRL-\

19
jhat
• Allows you to interactively work with a memory
snapshot captured by jmap
> Use jmap -dump:format=b,file=heap_dump_file
• Use jhat to 'mount' heap file
> Hosted on a web server
> Access through a standard browser
• Shows the following (standard query)
> All classes
> Object on the heap
> Instances
> Objects reachable from root set
20
jhat Object Query Language
• Develop custom query with object query language
> SQL like, uses JavaScript for expression in from and
where clause
> A set of built-in functions like heap, referrers, reachables,
sizeof, etc.
• Use to answer questions like
> Find all String instances that are over 1K in size
select s from j.l.Sring s where s.count >= 1024
> Find all URL instances that is referenced by 2 or more
objects
select u from j.n.URL u where count(referrers(u)) > 2

Note: some queries takes a long time to run on large heaps

21
Visual Tool - jconsole
• Bundle with JDK
> Graphical console that enables you monitor and manage
Java applications
> API to create your own plugin to jconsole
• Provides information on
> Memory usage and GC activities
> Threads, thread stack trace, locks
> Objects pending finalization
> Runtime information such as uptime, CPU time
> JVM information such as classpath, properties,
command line arguments, etc.
22
Visual Tool – visualvm
• Open source project
> Based on NetBeans platform, uses the update center
> Current plugin includes VisualGC, jconsole, thread dump
analyzer, profiler
> Runs only on JDK 6 and later
> http://visualvm.dev.java.net

23
Common Problems

24
Insufficient Memory
• Heap memory error – Java heap space
> Does not have sufficient space to accommodate new
objects
• Non heap error – PermGen space
> Permanent generation and interned strings are stored here
> When full cannot load classes
• Native memory error – Request <size> bytes for <reason>.
Out of swap space?
> Memory area used by JNI and native libraries
• May also mean you have incorrectly sized the heap

25
Possible Causes of Memory Problem
• Unintentional memory retention in Java application
> Event listeners
> Values in maps
> Resources not freed eg Graphics, JFrame, socket
connection, etc
• Rate of GC is too slow
> Use of finalizers may do that

26
Object Monitors
• Deadlocks
> Threads waiting for resource that are never freed
• High lock contention
> Lots of BLOCKED or WAITING threads
> Excessively synchronizing resources in a heavily
threaded environment
> May lead to unresponsiveness

27
Data Collection
and Analysis

28
1. Determining Memory Retention
• Let application run to a steady state first
• Use jconsole to visually determine if your heap is
growing
> Memory tab → Heap Memory Usage chart
• Use -XX:+PrintGCDetails and -Xloggc:logfile.txt to
capture GC info
> Use shell script or Perl to extract information
> Plot on spreadsheet
• jmap with -histo option to examine suspiciously
large allocations
29
 Reading GC Output
java -Xloggc:some_file.txt -XX:+PrintGCDetails
hello.World
Time taken including promotion
Total usable heap size

13.208: [GC 11570K->10852K(11796K), 0.0058870 secs]

Minor collection; before and after

Time from the start of application

30
2. Get A Copy of the Heap – 1
• Perform this on running application that has reach a
steady state
• Invoke the MBean operation on jconsole
> MBeans tab → com.sun.management → operations →
HotSpotDiagnostic → dumpHeap
• jmap with -dump in binary format
• If application dies with OutOfMemoryError
> Run JVM with -XX:+HeapDumpOnOutOfMemoryError
option, creates heap dump when JVM dies
> Use jinfo to set it dynamically
– jinfo -flag +HeapDumpOnOutMemoryError <pid>
31
2. Get A Copy of the Heap – 2
• jmap with -histo options breaks down class/instance
usage
num #instances #bytes class name
----------------------------------------------
1: 1123 145697064 [I
2: 11588 1441480 <constMethodKlass>
3: 11588 927296 <methodKlass>
4: 13805 917744 [C
5: 13208 845312 java.awt.geom.AffineTransform
6: 8549 683920 java.lang.reflect.Method
...
• Start Java VM with -XX:+PrintClassHistogram
> Will dump out similar info on Ctrl-\ or Ctrl-Break

32
2. Get A Copy of the Heap – 3
• Use the JVMTI heap walker demo
> Shipped with JDK in demo/jvmti directory
> Reference for developing your custom JVMTI tools
> Dumps heap on SIGQUIT signal
> java -agentpath
:$JAVA_HOME/demo/jvmti/heapViewer/lib/libheapViewer.so
MyJavaApp

• Dump heap in binary format for analysis with jhat

> Use jmap -dump:format=b,file=heap_dump_file
> Use java -Xrunhprof:format=b MyJavaApp
• Use DTrace on Solaris
33
3. Finding Object Retention
• Question to ask
> What objects are still alive?
> What is keeping it alive?
> Where is this allocated?
• jhat session with heap dump created by HPROF
> Show instance count of all classes (excluding platform)
> Locate the suspicious objects and look at instances
> Select any one of the instance
– Object allocated from gives you the trace of how this is created
– References to this object

34
Finalizers
• Finalizers may also cause memory retention
• Use jmap -finalizerinfo <pid> for count of objects
pending finalization
• jconsole provide the same info
> VM Summary tab, look at Pending Finalization

35
Detecting Deadlocks – 1
• Use jconsole to connect to running instance
> Goto Threads tab
> Click on 'Detect Deadlock' button for hints
• Look for threads that are BLOCKED or WAITING
> Will show the monitor owner
Name: Thread-1
State: WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@19abd2b owned
by: Thread-0
Total blocked: 0 Total waited: 2

Stack trace:
sun.misc.Unsafe.park(Native Method)
...
LockIt$RunningThread.run(LockIt.java:20)
java.lang.Thread.run(Thread.java:619) 36
Detecting Deadlocks – 2
• Same information available from command line
> Start VM with -XX:+PrintConcurrentLocks
> jstack -l <pid>

• Lock contention can cause performance problems

> Statistics turned off by default
> Enable it via jconsole

37
 Locating CPU Hogs
• Use HPROF with cpu=samples
> Will indicate which CPU
CPU SAMPLES BEGIN (total = 2484) Thu Feb 28 13:11:08 2008
rank self accum count trace method
1 81.48% 81.48% 2024 300411 sun.awt.X11.XToolkit.waitForEvents
2 3.46% 84.94% 86 301559 sun.java2d.loops.MaskBlit.MaskBlit
3 1.49% 86.43% 37 300638 java.lang.Thread.yield

• Use jtop
> Available from http://jtop.sourceforge.net/

38
Troubleshooting
Your Java
Applications
Lee Chuk Munn
Staff Engineer
Sun Microsystems
39
40
3. Inserting New or Existing Slides
● To add a new slide to your presentation, select
Insert>Slide. You can also duplicate slides in the
Slide Sorter using copy and paste.
• To add a slide(s) from another presentation,
copy and paste from one file to the other in
the Slide Sorter.
● To insert an entire presentation, select Insert>File.

41
Template – Text Slide
with Two Line Title and Subtitle
This is a Subtitle with Initial Caps Each Major Word

• To insert a Subtitle on other slides, copy and paste

the Subtitle text block
• On a slide with a two line Title and Subtitle (as
shown here) move the bullet text block down to
make room for the Subtitle. You can also duplicate
this slide and replace the content.

42
GlassFish V2

43
44
45
46
47
48
49
50
51
52
53
54
55
Title Here
Lee Chuk Munn
Staff Engineer
Sun Microsystems

56