You are on page 1of 55

IBM Integration Bus

How to analyse your system to optimise


performance and throughput in IIBv9
Dave Gorman
gormand@uk.ibm.com

2013 IBM Corporation


Overview

 The purpose of this presentation is to demonstrate how to find the cause of poor
performance for an IBM Integration Bus node (broker) for two different types of problem.

 The examples are obtained on a Windows system but the principles of investigation and
problem determination apply equally on all platforms. The system level tools will differ
though.

2 2013 IBM Corporation IBM Integration Bus


Agenda

 Introduction

 Tools

 Techniques

 Demonstration

3 2013 IBM Corporation IBM Integration Bus


What are the main performance costs in message flows?

Parsing Tree Navigation Tree Copying


A B C X Y Z

Root.Body.Level1.Level2.
Level3.Description.Line[1]; Set OutputRoot = InputRoot;

Resource Access Processing Logic

4 2013 IBM Corporation IBM Integration Bus


Integration Bus Processes

Task Function
bipservice Administration agent availability
bipbroker Administrative agent

biphttplistener HTTP Listener (broker wide)

DataFlowEngine [n] Execution group provides message flow runtime environment

 Important to understand:
Topology of broker
SOAP and HTTP listeners

6 2013 IBM Corporation IBM Integration Bus


Which resources and how much
 In busy times expect to use what is needed (!)
Exactly what will depend on the configuration and the applications
Typical to use CPU and memory plus I/O to some level

 In quiet times Message Broker and MQ processes


should use very little CPU
Should use very little I/O capacity
Will retain memory unless memory utilisation is very high

 The amount of CPU and memory used will depend on the situation
A complex configuration many MQ channels, hundreds of message flows will use much more
memory and CPU than a single message flow

 Some memory sizes


bipservice 4.6 MB
bipbroker 162 MB
biphttplistener 48 MB
DataFlowEngine 191 MB for JavaTransform samples
can use from ~100MB to GigaBytes depending on
The number of flows, the complexity of the message flow, the size of the messages
 MQ processes
Expect it to be less than Message Broker (192 MB for simple queue manager on xLinux)
Will depend on number of open queues, channels, queue buffer sizes etc.

7 2013 IBM Corporation IBM Integration Bus


Tools That are Needed

 Monitoring Tools
At the operating system level to observe
System resource usage CPU, memory, I/O activity
Heaviest resource users

At the component level to observe


Behaviour within the particular component (MQ/Integration Bus)

Both types of tools of needed


They have different views of the world
They are complimentary

 Driving Tools
Needed to generate a continuous workload
Important to assess performance after warm-up during sustained activity

8 2013 IBM Corporation IBM Integration Bus


System Configuration: lcpu=64 mem=8192MB
UNIX Tools kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
1 0 1977672 25823 0 0 0 0 0 0 3 958 696 4 0 96 0
 Vmstat 1 0 1977838 25719 0 2 0 98 100 0 29 2941 2250 4 0 96 0
1 0 1977685 25872 0 0 0 0 0 0 2 636 483 4 0 96 0
System configuration: lcpu=64 drives=5 paths=6 vdisks=2

tty: tin tout avg-cpu: % user % sys % idle % iowait


0.0 29.5 3.6 0.1 96.2 0.0

 Iostat Disks: % tm_act Kbps tps Kb_read Kb_wrtn


hdisk3 0.0 0.0 0.0 0 0
hdisk2 0.0 0.0 0.0 0 0
hdisk0 0.0 4.0 1.0 8 0
hdisk1 0.0 0.0 0.0 0 0
cd0 0.0 0.0 0.0 0 0

 Nmon

 filemon

11 2013 IBM Corporation IBM Integration Bus


Process Explorer on Windows

 Watch system activity in detail


on Windows

 Watch
CPU Usage
Commit Charge
I/O Activity
Physical Memory History
Summary Information
Individual Processes

Download from http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx


12 2013 IBM Corporation IBM Integration Bus
Process Explorer

 DataFlowEngine.exe is the Execution Group


 amqzlaa0.exe is the MQ agent for the broker
 Can quickly see system is busy
 Customise by selecting columns of choice

13 2013 IBM Corporation IBM Integration Bus


z/OS Tools

 SDSF

 RMFMON II

15 2013 IBM Corporation IBM Integration Bus


Key Tools at the Component Level

 Integration Bus

User Trace
Trace Nodes
WebUI
Compare flow statistics at the node (broker), server (execution group),
container (application or library) or at an individual message flow level
IIB Explorer
View resource use at the execution group level
View Activity Log

 MQ Explorer

 Java Healthcenter

17 2013 IBM Corporation IBM Integration Bus


Statistics Scope
Resource
Node (broker) Statistics
Server (execution group)
Accounting &
Thread Statistics
Message Flow
Terminals
Message Model Node

18 2013 IBM Corporation IBM Integration Bus


WebUI Statistics
 Using the WebUI in Integration
Bus v9:
Control statistics at all levels
Easily view and compare
flows, helping to understand
which are processing the
most messages or have the
highest elapsed time
Easily view and compare
nodes, helping to understand
which have the highest CPU
or elapsed times.
View all statistics metrics
available for each flow
View historical flow data

23 2013 IBM Corporation IBM Integration Bus


Integration Bus Explorer & Resource Statistics

 View resource
statistics for
resource managers
in IIB such as JVM,
ODBC, JDBC etc

24 2013 IBM Corporation IBM Integration Bus


Integration Bus Explorer & Activity Log
 View activity as it happens using
explorer
 Filter by resource managers

25 2013 IBM Corporation IBM Integration Bus


MQ Explorer

MQ Explorer
Trace
Statistics
26 2013 IBM Corporation IBM Integration Bus
IBM Support Assistant and Java Health Centre

 Java Health Centre is provided as part of the IBM Support Assistant


Offers very low overhead monitoring tool
Runs along side an IBM Java application

 Get visibility, monitoring and profiling in the following application areas:


Performance
Java method profiling
Lock analysis
Garbage collection
Memory
System
Java Class
File input and
Object

 Enable the application JVM prior to use


IBM_JAVA_OPTIONS=-Xhealthcenter

27 2013 IBM Corporation IBM Integration Bus


Demonstration of Analysing Performance Issues
 Identify problems in two message flows using
Process Explorer
WebUI Statistics
MQ Explorer
Java Healthcenter

 Message Flows

Coordinated Request Reply JavaComputeTransformNoXpath

28 2013 IBM Corporation IBM Integration Bus


Demo 1

Analysing a performance problem in the Coordinated


Request Reply Sample

29 2013 IBM Corporation IBM Integration Bus


Coordinated Request Reply flows
 Consists of three message flows
Request
Converts incoming message from XML to CWF
Saves the incoming message in a queue for subsequent reply processing
Writes a message for the back end reply message flow

Backend Application
Sets the completion time in the message
Writes a reply message

Reply
Reads the message from the back end message flow
Retreives the original message saved by the request message flow
Writes an output message

30 2013 IBM Corporation IBM Integration Bus


Coordinated Request Reply queues
 The queues
Request

GET_REQREP_IN GET_BACKEND_REQ

GET_REPTO_STORE

BackendReplyApp

GET_BACKEND_REQ GET_BACKEND_REP

Reply

GET_BACKEND_REP GET_REQREP_OUT

GET_REPTO_STORE

31 2013 IBM Corporation IBM Integration Bus


Run and Investigate

 Steps
1. Ensure all components are started and the application works as expected
Message flows, databases, external applications etc
2. Start a load generator [JMSPerfharness in this case]
3. Look at activity
Is processing happening at the expected rate
Is CPU usage as expected
Is memory usage as expected
4. If things do not seem as expected
Look for a build up of messages
Poor service times
5. Enable and view statistics
6. Analyse statistics
7. Examine message flows

32 2013 IBM Corporation IBM Integration Bus


Step 1 Check flows are running using the WebUI

 Check the server is running


 Check the flows are
running
 Check the event/sys log for
any errors
 Processing messages and
no errors

33 2013 IBM Corporation IBM Integration Bus


Step 2 Start a load generator
 Run Perfharness
Use 10 threads

 All threads start successfully


Each thread PUTs a
message then GETs a
message so should be no
messages on queues for
any period of time

 Check event/sys log for any


error messages

34 2013 IBM Corporation IBM Integration Bus


Step 3 Look at CPU activity

 Messages being processed but


Rate is low, much lower than expected
Very little CPU being used
Execution group does not register any
CPU activity

35 2013 IBM Corporation IBM Integration Bus


Step 4 Look for a build up of messages
 Key queues are GET_REQREP_IN
GET_REPTO_STORE Request
GET_BACKEND_REQ
Build up of msgs on queues:
GET_REPTO_STORE
GET_BACKEND_REQ BackendReplyApp GET BACKEND_REQ
GET_BACKEND_REP

GET_BACKEND_REP What does this mean?


GET_REPTO_STORE Reply
GET_REQREP_OUT

36 2013 IBM Corporation IBM Integration Bus


Step 4 Look for a build up of messages
 Looking at the flows
Queue GET_REPTO_STORE is used by Request and Reply message flows
Queue GET_BACKEND_REQ is used by BackendReplyApp message flow

GET_REPTO_STORE is used mid-flow (so flows using this are less likely to be problem)
GET_BACKEND_REQ is the input queue for the BackendReplyApp,
Indicates flow is not running fast enough or not enough instances allocated

 Need to investigate what is happening with BackendReplyApp


For this use WebUI flow statistics

37 2013 IBM Corporation IBM Integration Bus


Step 5 Enable flow statistics

 Start and stop statistics


using the WebUI for:
All flows in a server
All flows in a container
Individual flows

38 2013 IBM Corporation IBM Integration Bus


Step 5 View statistics
Flow comparison views

 Select the statistics view


 Drill down to the problem flow
 Start by comparing flows
 Flow Analysis view for most detail
39 2013 IBM Corporation IBM Integration Bus
Step 6 Compare flows

 Compare flows to determine which one might be causing the problem


 We can see that the BackenReplyApp flow has an average elapsed time of
1010.1 millseconds. It only has 1 active thread, and has processed 20
messages in the 20 second statistical snapshot period.
 This matches the rate we see from PerfHarness!

40 2013 IBM Corporation IBM Integration Bus


Step 5 Analyse the flow

 Display historical flow details such as


message rate, CPU and elapsed time
 View all nodes within the flow to determine
and sort by average elapsed and CPU
times
 The compute node Modify_CompletionTime
seems to be a problem!

 What does high elapsed time and low CPU


time suggest the problem might be?

41 2013 IBM Corporation IBM Integration Bus


5 Review the code

 Having worked out which node is causing the problem


 We can quickly see which why the node is taking 1 sec elapsed time but little CPU

42 2013 IBM Corporation IBM Integration Bus


Problem Found!

 1 second sleep in the compute node within the message flow is causing slow processing
times and no CPU usage
Matches the observations at the start
Low CPU and low rate

 Unlikely to be so easy in future but slow service times, like slow synchronous web service
invocation would have the same effect

 If it was slow web service response then allocate more instances to improve processing rate

43 2013 IBM Corporation IBM Integration Bus


Summary of Steps for this Investigation

 Use a systematic approach


Key steps used were
1. Ensure all components are started and the application works as expected
Message flows, databases, external applications etc
2. Start a load generator
3. Look at activity
Is processing happening at the expected rate
Is CPU usage as expected
Is memory usage as expected
4. If things do not seem as expected
Look for build up of messages
5. Analyse accounting and statistics
6. Examine message flows

 It is very important to
Use tools
System level and component level
Start at a high level system level and then close-in on the problem

44 2013 IBM Corporation IBM Integration Bus


Demo 2

Analysing a performance problem in the


JavaNoXPath Sample

45 2013 IBM Corporation IBM Integration Bus


Demonstration JavaComputeTransformNoXpath
 Consists of one message flow
JavaComputeTransformNoXpath
Reads an XML message
Transforms to a different format using a Java Compute node

JAVACOMPUTE.TRANSFORMNOXPATH.IN JAVACOMPUTE.TRANSFORMNOXPATH.OUT

46 2013 IBM Corporation IBM Integration Bus


What is the problem we need to solve?

 The problem is characterised by


Low message rate
High CPU usage at both system and Execution Group level
Sufficient messages on the input queue

 Likely issue is one of high CPU usage in a message flow


But which flow and which node?
47 2013 IBM Corporation IBM Integration Bus
Compare the flows

All of the Elapsed and CPU time is in JavaComputeTransformNoXpathFlow so continue


investigation of this flow

48 2013 IBM Corporation IBM Integration Bus


Finding the Processing Node for Investigation

The majority of the elapsed and CPU time


within in the flow is spent in the
JavaComputeTransformNoXpath node

What might cause this?

As this is a Java Compute Node continue


investigation using the Java Healthcenter

49 2013 IBM Corporation IBM Integration Bus


Find the Execution Group JVM Health Port for Java Health Center
Environment variable:

IBM_JAVA_OPTIONS=-Xhealthcenter

Open ports starting 1974, the


JavaComputeTransformNoXPathFlow
DataFlowEngine is 1976

50 2013 IBM Corporation IBM Integration Bus


Alternate Method for Finding the port number
Find the Process ID of the execution group with mqsilist and netstat

51 2013 IBM Corporation IBM Integration Bus


Invoking the Java Health Center

52 2013 IBM Corporation IBM Integration Bus


Attaching to the Execution Group JVM

53 2013 IBM Corporation IBM Integration Bus


Connect to a port

54 2013 IBM Corporation IBM Integration Bus


Connection Complete and Ready to Analyse

55 2013 IBM Corporation IBM Integration Bus


Analysis and Recommendations - Classes

56 2013 IBM Corporation IBM Integration Bus


Analysis and Recommendations - Environment

57 2013 IBM Corporation IBM Integration Bus


Analysis and Recommendations Garbage Collection

58 2013 IBM Corporation IBM Integration Bus


Analysis and Recommendations Native Memory

59 2013 IBM Corporation IBM Integration Bus


Analysis and Recommendations - Profiling

60 2013 IBM Corporation IBM Integration Bus


The cause

 Having worked out which


node is causing the
problem
 We can quickly see why
the node is consuming a
lot of CPU
 A call to the method
bubble_sort() just before
propagating out of the
node is sorting the entire
output message

61 2013 IBM Corporation IBM Integration Bus


If you suspect there is a product problem

 Identify the problem as best you can

 Find the simplest test that recreates the problem

 Collect the data identified in the Must Gather list


For IIB http://www.ibm.com/support/docview.wss?rs=849&uid=swg21209857
For MQ http://www.ibm.com/support/docview.wss?uid=swg21229861#MG6

62 2013 IBM Corporation IBM Integration Bus


Summary

 Wide range of tools available covering operating system and component performance
Expect to use multiple tools
After all it is important to understand what is happening at different levels
Demonstration has shown how to use the key tools for MQ and IIB to debug a problem

 Practice before hand


Being familiar with the tools is a great help in a crisis
Learning a new tool and solving a crisis is not a good combination

 Know your applications and systems


What is normal in terms of processing rate, CPU usage etc.
This information allows to know whether there is a problem and to what extent

63 2013 IBM Corporation IBM Integration Bus


Additional Information

 WebSphere Message Broker: Designing for Performance


 http://www.ibm.com/support/docview.wss?rs=849&uid=swg24006518

 WebSphere Message Broker V7-Message display, test & performance utilities (IH03)
http://www-01.ibm.com/support/docview.wss?rs=171&uid=swg24000637

 IBM Monitoring and Diagnostic Tools for Java - Getting started with Health Center
http://www.ibm.com/developerworks/java/jdk/tools/healthcenter/getting_started.html

 IBM Monitoring and Diagnostic Tools for Java - Health Center Version
http://www.ibm.com/developerworks/java/jdk/tools/healthcenter/

 IBM Monitoring and Diagnostic Tools for Java - Health Center


http://publib.boulder.ibm.com/infocenter/hctool/v1r0/index.jsp

65 2013 IBM Corporation IBM Integration Bus