Analyzing Java Out Of Memory Problems with IBM Heapdump Facility

Analyzing Java Out Of Memory Problems with IBM Heapdump Facility

Applies to:
SAP NetWeaver Application Server Java 6.40, 04s and all applications running on top of it on IBM JVM.

Summary
This is a step-by-step guide for analyzing and finding a possible cause of Java OutOfMemory errors while running SAP NetWeaver Java components on top of an IBM JVM. Author(s): Myriana Markova Company: SAP Created on: November 2006

Author Bio
Myriana Markova works for SAP as a developer in Java Application Server area.

SAP DEVELOPER NETWORK | sdn.sap.com 2006 SAP AG

BUSINESS PROCESS EXPERT COMMUNITY | bpx.sap.com 1

Analyzing Java Out Of Memory Problems with IBM Heapdump Facility

Table of Contents
Applies to: ........................................................................................................................................ 1 Summary.......................................................................................................................................... 1 Author Bio ........................................................................................................................................ 1 Heapdump Facility When is it Useful?.......................................................................................... 3 Tools to Use..................................................................................................................................... 3 Step-by-Step Procedure for Analyzing Heapdumps........................................................................ 3 Types View................................................................................................................................... 3 Objects View ................................................................................................................................ 4 Tree View ..................................................................................................................................... 6 IBM HeapAnalyzer Quick Checklist................................................................................................. 9 Copyright........................................................................................................................................ 10

SAP DEVELOPER NETWORK | sdn.sap.com 2006 SAP AG

BUSINESS PROCESS EXPERT COMMUNITY | bpx.sap.com 2

Analyzing Java Out Of Memory Problems with IBM Heapdump Facility

Heapdump Facility When is it Useful?

Heapdump is a JVM facility that generates a dump of all live objects that are on the Java heap at a certain point in time. The IBM Heapdump contains two lines per object. The first line displays the address of the object, its size, and type information. The second line contains a list of the memory addresses of objects that have been referenced by that object. When the java heap is exhausted the JVM will generate a heap dump by default. To enable signal-based heap dumps (on SIGQUIT) one has to set in the environment variables of the user with which the JVM is started the option IBM_HEAPDUMP=TRUE In general one will use heapdumps as an alternative to profilers when trying to find the root cause for a memory leak (steady increase of memory with possible plateaus until the heap is completely exhausted). Heapdumsp are not so useful in case of an OoutOfMemory caused by a big object allocation, memory fragmentation or native out of memory.

Tools to Use
Heap Analyzer. Newest version can be downloaded from: http://www.alphaworks.ibm.com/tech/heapanalyzer Memory Dump Diagnostic for Java can be downloaded from http://www-128.ibm.com/developerworks/websphere/downloads/memory_dump.html . Has pretty much the same features as HeapAnalyzer, but allows for comparison of two heap dumps. Requires IBM JDK.

The step-by-step procedure in the next section is based on HeapAnalyzer.

Step-by-Step Procedure for Analyzing Heapdumps

Types View First check the Types view, which provides input similar to class histogram in SUN JDK. To access the Types View choose the option Types List from the Analysis menu. The types view allows sorting the objects by Sum of sizes, thus showing objects of which type take up the most space in memory. Those will be usually char [], byte [] or HashMap and Hashtable entry objects.

SAP DEVELOPER NETWORK | sdn.sap.com 2006 SAP AG

BUSINESS PROCESS EXPERT COMMUNITY | bpx.sap.com 3

Analyzing Java Out Of Memory Problems with IBM Heapdump Facility

Here one can see that besides char [] and the arrays of HashMap entries two other classes are in the top 5. Types view allows also sorting by object count revealing which are the objects with the biggest number of instances on the heap:

Objects View To access the Objects view choose Objects List from the Analysis menu. In the objects view you can look at a list of all the objects in the memory dump. For each object the following is listed: The total size the sum of the object size and the sizes of all the objects that it can reach through its children The size of the object. The number of children the object has. The memory address of the object.

SAP DEVELOPER NETWORK | sdn.sap.com 2006 SAP AG

BUSINESS PROCESS EXPERT COMMUNITY | bpx.sap.com 4

Analyzing Java Out Of Memory Problems with IBM Heapdump Facility

The class name for the object.

Each column in the view can be sorted by clicking on the column label. In this view one should check for objects with biggest number of children, and the objects biggest in size.

SAP DEVELOPER NETWORK | sdn.sap.com 2006 SAP AG

BUSINESS PROCESS EXPERT COMMUNITY | bpx.sap.com 5

Analyzing Java Out Of Memory Problems with IBM Heapdump Facility

The screenshots above taken from different heap dumps show sorting by number of children and by size of the object. In the first case one can see there are 5 arrays of same size with the same number of children, then another five of the same size and same number of children. On the second picture on the other hand there are quite a few big byte [] which are most likely taking up the biggest part of the heap. In both cases one needs to find the top 5 10 of the objects in the heap tree and check the references. Tree View The Tree View can be accessed by selecting Tree view option from the Analysis menu.

Each node from the picture above has the following format: TotalSize of object plus children, [Size of actual object], NumberOfChildren of the Object (Number of root objects), Name, Address Here one can look for a specific object by its address (right mouse button -> Choose Find an address) and find its references.

SAP DEVELOPER NETWORK | sdn.sap.com 2006 SAP AG

BUSINESS PROCESS EXPERT COMMUNITY | bpx.sap.com 6

Analyzing Java Out Of Memory Problems with IBM Heapdump Facility

In version 1.4.1 of heap wizard one has the possibility to go directly to the object in the tree from Objects view by clicking on the object with right mouse button and then choosing Find Object in a Tree View.

SAP DEVELOPER NETWORK | sdn.sap.com 2006 SAP AG

BUSINESS PROCESS EXPERT COMMUNITY | bpx.sap.com 7

Analyzing Java Out Of Memory Problems with IBM Heapdump Facility

Lets take one of the big byte[] from the previous example and check its references in the tree view:

Here one can see that the specific byte array has no children and its parent is an object of type SQLServerStatement, so most likely this is a 93mb result set retrieved from the database. If all the other byte [] have the same parent it would seem too much data is fetched on each round trip to the database and queries might need to be optimized. Right clicking in the Tree View allows using the following options: Search for total size drop Locates a drop between the total size of a parent and the biggest total size of a child. This option can be used as a way to search for a leak suspects from a given object. Go to the largest drop in sub trees Takes you to the largest drop in the tree from the chosen point. The drop is the difference in the amount of memory the object is using and the amount of memory the contained object is using. Locate a leak suspect Takes you to a leak suspect in the tree from the chosen point. A leak suspect is an object with a large drop in memory size between itself and the parent containing object. Compile leak suspects Looks through all roots for leak suspects (as described above). The above options will usually lead you to the biggest collections in the heap. However keep in mind that caches and object pools are also usually pretty big collections and sometimes will be identified as leak suspects even though they might not be leaking. Hence to avoid such detection of caches as memory leaks one needs to take multiple dumps, and avoid if possible taking dumps at startup or right after startup, as this is the time when the caches and object pools will start to fill.

SAP DEVELOPER NETWORK | sdn.sap.com 2006 SAP AG

BUSINESS PROCESS EXPERT COMMUNITY | bpx.sap.com 8

Analyzing Java Out Of Memory Problems with IBM Heapdump Facility

IBM HeapAnalyzer Quick Checklist

Check Type View o Which are the objects taking up most memory? - sort by Sum of sizes o Which are the classes with biggest number of instances? sort by Object count Check Objects View o Which are the biggest collections/ object arrays? sort by number of children o Are there any huge objects with own size tens of megabytes? sort by Size Check the Tree View o Search for the suspicious objects identified in the previous two steps o Check for big differences between total size of parent and children

SAP DEVELOPER NETWORK | sdn.sap.com 2006 SAP AG

BUSINESS PROCESS EXPERT COMMUNITY | bpx.sap.com 9

Analyzing Java Out Of Memory Problems with IBM Heapdump Facility

Copyright
Copyright 2006 SAP AG. All rights reserved. No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. Microsoft, Windows, Outlook, and PowerPoint are registered trademarks of Microsoft Corporation. IBM, DB2, DB2 Universal Database, OS/2, Parallel Sysplex, MVS/ESA, AIX, S/390, AS/400, OS/390, OS/400, iSeries, pSeries, xSeries, zSeries, z/OS, AFP, Intelligent Miner, WebSphere, Netfinity, Tivoli, Informix, i5/OS, POWER, POWER5, OpenPower and PowerPC are trademarks or registered trademarks of IBM Corporation. Adobe, the Adobe logo, Acrobat, PostScript, and Reader are either trademarks or registered trademarks of Adobe Systems Incorporated in the United States and/or other countries. Oracle is a registered trademark of Oracle Corporation. UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group. Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of Citrix Systems, Inc. HTML, XML, XHTML and W3C are trademarks or registered trademarks of W3C, World Wide Web Consortium, Massachusetts Institute of Technology. Java is a registered trademark of Sun Microsystems, Inc. JavaScript is a registered trademark of Sun Microsystems, Inc., used under license for technology invented and implemented by Netscape. MaxDB is a trademark of MySQL AB, Sweden. SAP, R/3, mySAP, mySAP.com, xApps, xApp, SAP NetWeaver, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary. These materials are subject to change without notice. These materials are provided by SAP AG and its affiliated companies ("SAP Group") for informational purposes only, without representation or warranty of any kind, and SAP Group shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP Group products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty. These materials are provided as is without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. SAP shall not be liable for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. SAP does not warrant the accuracy or completeness of the information, text, graphics, links or other items contained within these materials. SAP has no control over the information that you may access through the use of hot links contained in these materials and does not endorse your use of third party web pages nor provide any warranty whatsoever relating to third party web pages. Any software coding and/or code lines/strings (Code) included in this documentation are only examples and are not intended to be used in a productive system environment. The Code is only intended better explain and visualize the syntax and phrasing rules of certain coding. SAP does not warrant the correctness and completeness of the Code given herein, and SAP shall not be liable for errors or damages caused by the usage of the Code, except if such damages were caused by SAP intentionally or grossly negligent.

SAP DEVELOPER NETWORK | sdn.sap.com 2006 SAP AG

BUSINESS PROCESS EXPERT COMMUNITY | bpx.sap.com 10