Friday, January 7, 2011

Inspecting HotSpot JVM Options

Introduction
A method for inspecting the comprehensive set of HotSpot JVM options and some examples comparing different
outputs.
Background
The Oracle JVM provides an enormous spectrum of options to control the runtime JVM. These options are given a
criminally short and non-comprehensive discussion at the OTN here. OTN divides these options into three categories:
Behavioural, Performance and Debugging. This is a useful abstraction but some examples Oracle gives in each
category
can
be
misleading
due
to
the
overlap
between
Behavioural
and
Performance.
The JVM options themselves can be controlled in a number of ways:

via the command line on JVM startup.

via JMX for certain options where this is allowed.

indirectly via the command line by a super-option which then sets other options.

automatically by the JVM. The JVM has ergonomic capability to detect features of the host and set options
accordingly.
Using the first two methods we are explicitly setting options ourselves and can easily track what value each option
has (true, false, 20, 100 etc). These last two methods for setting options can cause developers the most
confusion, as we have less insight and therefore less understanding over what options are being set. This puts us in
an unfavourable position when explaining the runtime behaviour of our applications. To dump outevery JVM option
and its value there is an option to do this hidden away in the JVM source code:
-XX:+PrintFlagsFinal
Usage
Let's see this in action! First let's run the following:
$

jdk1.6.0_23/bin/java

-XX:+UnlockDiagnosticVMOptions

-XX:+PrintFlagsFinal

-version

(I have added -XX:+UnlockDiagnosticVMOptions as this unlocks more options for viewing and control. Also I
have added -version at the end so Java doesn't complain that it has nothing to run and to supplement the results).
The sysout is quite large so I've saved it here. Firstly you may be surprised at the sheer quantity of JVM options
available: 717! That's a lot of configuration! Before we dive into the options lets look at our host specification and
the output of the -version option. The host has 8 dual core 64bit AMD Opteron processors running at 3.2GHz so 16
cores
in
total
with
64GB
of
RAM.
The
Java
-version
output
shows:
java
Java(TM)
SE
Java
HotSpot(TM)

Runtime
Server

version
Environment
VM
(build

(build
19.0-b09,

"1.6.0_23"
1.6.0_23-b05)
mixed
mode)

this tells us we are running Java 6 update 23 using the Server version of the HotSpot VM in 32bit mode. Now let's look
closer at the output. For each option we can see the following columns:
1. Datatype. bool for booleans, uintx for unsigned integers etc
2. Name
3. Assignment. = means this is the default. := means this option was modified from the default value
by command line or ergonomics.
4. Value
5. Category. There are two large categories diagnostic or product. (I find this a better grouping
than Behavioural, Performance and Debugging.) and also manageable if this option can be set via
JMX. C1relates to the JIT compiler used in the Client JVM. C2 means the same but for the Server JVM
and I suspect pd means Platform Dependent as most pd options are Solaris only.

Let's look closer at the modified options which have the := assignment:
uintx InitialHeapSize
uintx MaxHeapSize
uintx ParallelGCThreads
bool PrintFlagsFinal
bool UnlockDiagnosticVMOptions
bool UseParallelGC

:= 67108864
:= 1073741824
:= 13
:= true
:= true
:= true

{product}
{product}
{product}
{product}
{diagnostic}
{product}

apart from the two flags which I set myself on the command line we can see Java ergonomics has given the heap a
sensible range of 64MB to 1GB. The fairly old Parallel Garbage Collector has been chosen with an enormous 13
threads spawned to do the collection. Straightaway we've got a lot more insight into how our JVM is configured and
a few ideas for improvements.
Client vs Server JVM Analysis
We know from looking at the -version output Java automatically chose the server JVM instead of the client JVM
(Java automatically chooses server if it detects 64bit processors). But how do we know what options are set
because Java chose the Server JVM? What options were set because of our host specification? Lets run the following
commands and review the output:
$ jdk1.6.0_23/bin/java -client -XX:+UnlockDiagnosticVMOptions -XX:+PrintFlagsFinal version
$ jdk1.6.0_23/bin/java -server -XX:+UnlockDiagnosticVMOptions -XX:+PrintFlagsFinal version
I have saved the client output here and the diff client server here. Firstly lets look at the output from the
Client JVM; it has 661 options, 56 fewer than the Server version. Now let's look solely at the Client options which
have been modified, excluding the two options I set on the command line there are only two values which were
changed:
uintx InitialHeapSize
uintx MaxHeapSize

:= 16777216
:= 268435456

{product}
{product}

so Java has chosen some fairly sensible values for the Client JVM: 16MB initial heap size with a max size of 256MB.
There are many entries in the diff file mostly by virtue of the extra options available to the Server JVM or options
which relate to memory simply having higher values. Here are the more interesting differences:
<
>
<
>

intx
intx
intx
intx

CICompilerCount
CICompilerCount
CompileThreshold
CompileThreshold

=
=
=
=

1
2
1500
10000

{product}
{product}
{pd product}
{pd product}

The Client VM waits only 1,500 invocations before compiling a method to native code and uses only one thread for
compilation vs the Server which waits for 10,000 invocations and uses two threads.
>

bool DoEscapeAnalysis

= true

{C2 product}

In this build the Server VM has escape analysis enabled by default whereas the Client VM cannot enable escape
analysis at all. I don't think this is sensible on Oracle's part as whilst escape analysis can improve performance for
scientific computing (where many primitives are created within a method and then discarded once a final value has
been computed) for other applications escape analysis can lower performance when arguments do escape their
methods and are better left on the heap. Slipping in such a change transparently can also be problematic as
developers upgrading to a new JVM may see performance degradation in their applications when no visible
command line option has changed. Developers should benchmark accordingly when upgrading JVM builds.
<
<
>
>

bool
bool
bool
bool

RewriteBytecodes
RewriteFrequentPairs
RewriteBytecodes
RewriteFrequentPairs

=
=
=
=

false
false
true
true

An optimisation to rewrite two bytecode operations as one operation.

{pd
{pd
{pd
{pd

product}
product}
product}
product}

Super Option Activation

As mentioned earlier some options are super-options and will enable other options. Let's take a look at a common
super option: -XX:+AggressiveOpts
The two commands are:
jdk1.6.0_23/bin/java -server -XX:+UnlockDiagnosticVMOptions -XX:+PrintFlagsFinal version
jdk1.6.0_23/bin/java -server -XX:+AggressiveOpts -XX:+UnlockDiagnosticVMOptions XX:+PrintFlagsFinal -version
here are all the differences:
<
>
<
>
<
>
<
>
<
>
<
>

bool
bool
intx
intx
intx
intx
bool
bool
bool
bool
bool
bool

AggressiveOpts
AggressiveOpts
AutoBoxCacheMax
AutoBoxCacheMax
BiasedLockingStartupDelay
BiasedLockingStartupDelay
EliminateAutoBox
EliminateAutoBox
OptimizeFill
OptimizeFill
OptimizeStringConcat
OptimizeStringConcat

=
:=
=
=
=
=
=
=
=
=
=
=

false
true
128
20000
4000
500
false
true
false
true
false
true

{product}
{product}
{C2 product}
{C2 product}
{product}
{product}
{C2 diagnostic}
{C2 diagnostic}
{C2 product}
{C2 product}
{C2 product}
{C2 product}

So a number of options have been enable and some others have been given more aggressive values. We know XX:+AggressiveOpts changes with each JVM build. Let's look at an older build:
(NB. Java6u20 is the oldest build to support this form of option dumping)
jdk1.6.0_20/bin/java -server -XX:+UnlockDiagnosticVMOptions -XX:+PrintFlagsFinal version
jdk1.6.0_20/bin/java -server -XX:+AggressiveOpts -XX:+UnlockDiagnosticVMOptions XX:+PrintFlagsFinal
<
>
<
>
<
>
<
>
<
>
<
>

bool
bool
intx
intx
intx
intx
bool
bool
bool
bool
bool
bool

AggressiveOpts
AggressiveOpts
AutoBoxCacheMax
AutoBoxCacheMax
BiasedLockingStartupDelay
BiasedLockingStartupDelay
DoEscapeAnalysis
DoEscapeAnalysis
EliminateAutoBox
EliminateAutoBox
UseLoopPredicate
UseLoopPredicate

=
:=
=
=
=
=
=
=
=
=
=
=

false
true
128
20000
4000
500
false
true
false
true
false
true

{product}
{product}
{C2 product}
{C2 product}
{product}
{product}
{C2 product}
{C2 product}
{C2 diagnostic}
{C2 diagnostic}
{C2 product}
{C2 product}

We can see in 6u20 DoEscapeAnalysis was an AggressiveOpt but in the new build
6u23DoEscapeAnalysis became the default (as did UseLoopPredicate). Along the same
linesOptimizeFill and OptimizeStringConcat may become default values in future builds.
Conclusion
There is enormous scope for coarse and fine grained control of JVM behaviour. The option XX:+PrintFlagsFinal allows comprehensive reporting of the options and their values. The available options
vary by build and JVM type (server or client). Recording and auditing this output is an important step in any Java
benchmarking or continuous monitoring exercise. Oracle can surreptitiously enable options in new Java builds
which may cause inexplicable changes in performance and behaviour of existing applications.