Velikanovs - Performance Tuning and Troubleshooting For Oracle OC4J (Slides - Long Version)

Performance Tuning and
Troubleshooting for Oracle OC4J
Jurijs Velikanovs
7+ years Oracle DBA, OCP 7/8/8i/9i, OCA 9iAS R2
Topics:
z Introduction
z Usage considerations
z OC4J Internals
z OC4J Troubleshooting
z OC4J Performance tuning
z BC4J
Introduction
Who am I? And what are my responsibilities?
z Who am I?
I am a DBA team leader
I’m neither a Java fan nor a Java Developer nor a Java
Guru
My team has spent a lot of time researching and
struggling to stabilize the production environments
based on OC4J
z Responsibilities
To support production and development environments
To ensure production availability
To troubleshoot production system
To provide all the necessary information to the
developers or support organizations to solve a problem
How were we involved?
z Local Oracle Marketing team has pushed

customers to demand the use of OC4J in their
tenders’ conditions
z In 2002 the company management decided to
switch the whole Development from Oracle Forms
to OC4J/BC4J/JDeveloper technology
z We had
100 forms developers without any Java experience
A few Java fans
Oracle 9iAS R2 (9.0.2), later 9.0.3.1
In 2003 the first phase of the first application was
deployed
Example of the Applications we are working with
z Healthcare financial system

Enterprise application
2+ years in the production and still under development
Oracle 9iAS 9.0.3.1 OC4J, Oracle 9.2.0.5 DB
JSP = 1448 files, CLASS = 3283 files
965 tables, 593 views, 251 types, 220 packages,
DB Size 50 GB, REDO 2-8 GB Daily
800 heavy users per day
z 10x5 hour availability

Why is it so important?
z OC4J is the present strategic direction for Oracle

JDeveloper, Forms, Reports, Discoverer, OEM, etc.
z Difficult to troubleshoot
Complex technology
Many components
Java on Java
Why is it so important?
z OC4J is a strategic direction for Oracle

JDeveloper, Forms, Reports, Discoverer, OEM
z Difficult to troubleshoot
Complex technology
Many components
Java on Java
z There is not much information about this product

z Poor vendor support
z Many Oracle customers use it and have troubles
I will not tell you how to develop in Java …
I’m not going to tell you about

Web and EJB application technology
VMC
etc.
The technical layer is of great interest

I’m going to speak about OC4J internals …
ONS
Thread
OC4JMonitor
PORT: 3001 AJP new req Thread
Connection ApplicationServerThread
Listener Thread
ApplicationServerThread
Java native
AJP 1.3 ApplicationServerThread
... ONet
Memory Structures
Application objects
Http Session List ... JDBC Connection pool
Java native
RMI
RMIServer
Server TaskManager VM Periodic Task
3101
PORT: 3101
TaskManager VM Thread VMThread
Periodic Task
VM Thread
Thread
JMSServer
3201
Signal Dispatcher Finalizer

PORT: 3201 Signal Dispatcher Finalizer
JMSServer
JMSServer Suspend Checker Reference
JMSServer Suspend
JMSServer
Thread ThreadChecker Reference
Handler
Thread Thread Handler
Usage considerations
or
how to avoid troubles
Oracle AS Architecture Considerations
z You may make the configuration

complicated, with many
components highly
dependent on
each other
Oracle AS Architecture Considerations
BUT:
z Keep it as simple as possible
z At first you should be sure that the system works
properly in a simple configuration
z Make components as independent as possible
z Don’t use the infrastructure if there is no strong
need for it (i.e. Single-Signon, Discoverer,
Clustering)
Oracle AS Typical Architecture
DMS-metrics OPMN DCM-daemon

DMS.JAR (java) PM, ONS (c) DCM (java)
DCM
statistics reflecting strat, stop, ping, configuration
repository
notification management
opmn.xml
httpd.conf server.xml
Apache OHS (c) OC4J JVM
JVM(java)
Child
Child Server
Server
Child Server JMSServer
Child Server JMSServer
JMSServer
Child Server Thread
JMSServer
Child Server Thread
Thread ApplicationServerThread
Child
Child Server
Server Thread
Child
Child
Server
Server ApplicationServerThread
Child Server
Child Server ApplicationServerThread
Child Server ApplicationServerThread DB
Child Server
Child Server
Child Server ApplicationServerThread
PORT: 80, 7780
Child
Child
Child Server
Server
Server ApplicationServerThread
Child Server AJP
HTTP AJP 1.3
PORT: 3001
Child Server
WAN Child Server
Child Server
Child Server
AJP
Connection
Connection
Listener
Child
Child Server
Server Listener OC4JMonitor
Child
Child
Server
Server TaskManager
TaskManager
Child Server OC4JMonitor
Thread TaskManager
Child Server TaskManager
Child Server Thread (dms)
Child Server
Child Server
Child
Child Server Java native
Child Server
Server Java native
VM Periodic Task
VM Thread VMThread
Periodic Task
VM Thread
Thread
RMI
RMIServer
Server Signal Dispatcher Finalizer
Suspend Checker Reference
JMSServer Suspend
PORT: 3201
ThreadChecker Reference
Handler
Thread Handler
Installation Considerations
z Use the latest version of Oracle AS depending on

your circumstances
z Use the latest versions of Oracle Forms, Reports,
Discoverer, Portal. Install them in the OH
separated from your custom development OC4J
OH
z Install the latest patches for OS, JDK, OC4J,
JDBC components
z Install the version of BC4J which was used by the
Development. Don’t use the default version of
BC4J/ADF/JHeadstart (See. JDeveloper install.html)
Operational Considerations
z Keep running only the components you need

z Don’t startup in the production environment:
OracleEnterprise Manager 10g Application Server
Control
OC4J_Demos component
“home” component
z Don’t deploy your applications to the “home”
component
z Create your own component and use it to deploy
and run your applications
z Use the opmnctl, dcmctl, dmstool utilities to
maintain your AS environment
Deployment Considerations
z Get a commitment from the Development to use industry

standards for the Application development
Use EAR files to deploy any application changes
Use Ant tool (http://ant.apache.org) to make EAR/WAR files
In emergency cases use WAR files to deploy small changes to
application
z Use undeployApplication/deployApplication combination
rather than redeployApplication dcmctl command
z Use dcmctl saveInstance before deploying to backup
current configuration
z Deploy BC4J.EAR to your OC4J component to keep
“home” down
Versions Considerations
Recommendations Application
Have a dedicated test
environment and verify APP Frame Work
all the changes there
before implementing in BC4J 9.0.3.10.7
production
If you have the
OC4J JDBC
automatic stresstesting c.c.c.x c.c.c.x
scenarios, run them after 9.0.3.1 9.2.0.5
each change in the test
environment JDK c.c.x_x, 1.3.1_15
Monitor the impact of
changes and if the
situation becomes worse OS 2.4.9-e.38enterprise
rollback them
z OS
Application
It is important to install the latest
patches from an OS vendor
(Kernel, Network, etc.) APP Frame Work
Usually the patches can be
rollbacked easily BC4J 9.0.3.10.7
The change of OS main release
requires more testing and efforts
than patching
z JDK OC4J JDBC
c.c.c.x c.c.c.x
Install the latest versions of JDK 9.0.3.1 9.2.0.5
within c.c. boundaries
Use the JDK from Sun
ww.JavaSoft.com JDK c.c.x_x, 1.3.1_15
To install it replace the OH/jdk
directory. These changes can be
rollbacked easily OS 2.4.9-e.38enterprise
The change of c.c. version
requires the Development
involvement and more testing
z JDBC
Install the latest version of JDBC Application
within c.c.c. version
To install it replace OH/jdbc APP Frame Work
directory. These changes can be
rollbacked easily
It requires the minimal BC4J 9.0.3.10.7
functionality testing
z OC4J
Can be easily upgraded within a
c.c.c. version boundaries OC4J JDBC
c.c.c.x c.c.c.x
It requires the minimal
9.0.3.1 9.2.0.5
functionality testing
Follow the patch installation
instructions (9iAS 9.0.3.1) JDK c.c.x_x, 1.3.1_15
There are 50+ one-off patches
(*.class)
The change of c.c.c. version
requires the Development OS 2.4.9-e.38enterprise
involvement and more testing
z BC4J Application
A BC4J version sets the
requirements for OC4J version APP Frame Work
It can’t be changed without
Development involvement BC4J 9.0.3.10.7
even within x.x.x.x. version
boundaries
Often there are needs for OC4J JDBC
application changes c.c.c.x c.c.c.x
9.0.3.1 9.2.0.5
It requires a lot of testing
New version can have negative
JDK c.c.x_x, 1.3.1_15
impact to your application
The change of c.c.c. version
requires more application code OS 2.4.9-e.38enterprise
changes, testing, and other
efforts
z How to Determine the Versions

OS: uname -r
$ uname -r
2.4.9-e.38enterprise
JRE: java –version

$ $ORACLE_HOME/jdk/bin/java -version
java version "1.3.1_13"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.1_13-b03)
Java HotSpot(TM) Client VM (build 1.3.1_13-b03, mixed mode)
OC4J: java -jar oc4j.jar –version

$ $ORACLE_HOME/jdk/bin/java -jar $ORACLE_HOME/j2ee/home/oc4j.jar -version
Oracle9iAS (9.0.3.1.0) Containers for J2EE (build 031214.BP1.2075)
JDBC: a program with a call to meta.getDriverVersion

(location $OH/jdbc/lib/classes12dms.jar, Notes:73629.1/244074.1)
$ $ORACLE_HOME/jdk/bin/java JDBCVersion
Oracle JDBC driver 9.2.0.5.0
BC4J: a program with a call to meta.getDriverVersion

(location $BC4J/lib/bc4jct.jar)
$ $ORACLE_HOME/jdk/bin/java BC4JVersion
9.0.3.10.7
Other Considerations
z Have at least one senior level developer, who understands used

technology well, is capable to make technical decisions and to
lead the framework development, maintenance and
troubleshooting
z Synchronize OC4J, BC4J versions with Development
z Use an Automatic testing tool for the application stresstesting
z A commitment from Oracle is desirable. If you can manage a
connection to the Oracle JDeveloper development team to
support your project, it will be an invaluable help during
troubleshooting and decision making
z Have a skilled DBA for environments maintenance, monitoring
and troubleshooting. The DBA provides all possible
information to the development for application troubleshooting
z Have a monitoring framework
Before going to life
!!! Stresstest your application !!!

(or ask for results)
OC4J Internals
OC4J Internals
ONS
Thread
OC4JMonitor
new req Thread
PORT: 3001
AJP
Listener Thread
Java native
... ONet
Memory Structures
Application objects
Http Session List ... JDBC Connection pool
Java native
RMI
RMIServer
3101
PORT: 3101
Periodic Task
VM Thread
Thread
JMSServer
3201

JMSServer
JMSServer Suspend
JMSServer
Handler
Processes (threads) within OC4J
z Java native threads

VM Thread, VM Periodic Task Thread, Signal Dispatcher,
Suspend Checker Thread, Finalizer, Reference Handler
Exist in any Java based application
z OC4J Listener - threads
AJP/HttpConnectionListener, RMIServer (2), JMSServer
Accepting clients requests, assigning them to the worker threads
The configuration of listened ports is in opmn.xml file
z Set of processes for communicating with ONS
OC4JMonitorThread, Thread(2)
These processes make connection to the OPMN immediately after
the startup of OC4J
They send event messages to the OPMN to be broadcasted to
subscribers
Processes (threads) within OC4J
z TaskManager
Responsible for a periodic tasks executing (housekeeping)
server.http.SessionTimeoutTask (drops expired session)
server.ApplicationServerTask (clear expired threads)
sql.DriverManagerXADataSource
…
You can set an executing frequency in server.xml file
taskmanager-granularity=1000 (default 1 sec)
Use the dmstool to monitor the statistics of the process
dmstool -table oc4j_task
z ApplicationServerThreads (Global Thread Pool)
Execute users requests, run an application code, communicate
with the Apache processes and the database
Initial count of the processes can be set in server.xml file
global-thread-pool min=“200" max="200" queue="20“
OC4J Troubleshooting
Classification of Problem Areas
OC4J
JVM(java)
4.
PORT: 3001
AJP
JMSServer
JMSServer
Connection
Listener ApplicationServerThread
New Request
as
Apache
sig
n ApplicationServerThread
1.
TaskManager
3. DB
2. ApplicationServerThread
JDK
OS
1. OC4J container inside (infinitive cycles, waits on other threads, deadlocks)

2. Interface to the Apache processes
3. Interface to the database
4. 500 Internal Error
Possible Approaches & Information Sources
z Java Full thread dump

z Other
Log files
OS Monitoring commands
OS Tracing commands
Apache side information
Java code decompilation
Documentation, External searches and Metalink
z Troubleshooting information gathering (example)

Java “Full thread dump”
z How to get a Full thread dump?

z The Full thread dump interpretation
Different types of the threads within OC4J container
Different states of the threads
Some threads examples
z Useful java runtime parameter

z Java DEADLOCK problem
How to get a “Full thread dump”?
z Get pid of the OC4J process

Use : ps, pstree or opmnadmin
$ opmnadmin debug comp=pm
HTTP/1.1 200 OK
Content-Length: 677
Content-Type: text/html
Response:
======== PM ========
PM requests processed: 5
PROCESS TABLE
UID PID FLAGS TYPE STATUS REF HTTP AJP RMI JMS
---------- ----- -------- ------- ------- ------ ----- ----- ----- -----
456236 13512 00000040 OC4J Alive 1 0 3003 3104 3204
390700 13511 00000040 OC4J Alive 1 0 3002 3103 3203
521772 15874 00000040 OC4J Alive 1 0 3001 3102 3202
259628 13415 00000040 OC4J Alive 1 0 3000 3101 3201
194092 13338 00000040 Apache Alive 1 7780 0 0 0
128556 13281 00000040 Generic Alive 1 0 0 0 0
z Find out the output file name (OH/opmn/logs/*)

$ ps -fw --width 4000 -p 16114
UID PID PPID C STIME TTY TIME CMD
oracle 15874 15902 0 23:40 ? 00:00:00
/u01/app/oracle/product/J2EE01/jdk/bin/i386/native_threads/java -Xms512M -
Xmx512M -Doracle.ons.oraclehome=/u01/app/oracle/product/J2EE01 -
Doracle.ons.oracleconfighome=/u01/app/oracle/product/J2EE01 -
Doracle.ons.clusterid=1 -Doracle.ons.instanceid=10.130.2.9.20dcd9.fdc4ac6949.-
8000 -Doracle.ons.indexid=PANDA.default_island.1 -DOPMN=true -jar oc4j.jar -
config /u01/app/oracle/product/J2EE01/j2ee/PANDA/config/server.xml -properties -
instance -vmId C_10.130.2.9.20dcd9.fdc4ac6949.-
8000#J2EE01.ap1.voava.lv#521772#PANDA#PANDA#default_island -ajp 3001 -rmi 3102 -
jms 3202
$ ps -wo args --width 4000 -p 15874 | grep -v COMMAND | awk -F"oracle.ons.indexid="

'{print $2}' | awk '{print $1}'
PANDA.default_island.1
z Send SIGQUIT (Quit from keyboard) signal

$ kill -3 15874
z Check the output file
OH/opmn/logs/PANDA.default_island.1
Full thread dump:
"Thread-13" daemon prio=1 tid=0x82c8690 nid=0x3f15 waiting on monitor

[0xa51ff000..0xa51ff89c]
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:427)
at java.util.TimerThread.run(Timer.java:380)
"Thread-12" prio=1 tid=0x67b0cce0 nid=0x3e02 waiting on monitor [0..0xbffeb62c]
"OC4JMonitorThread" daemon prio=1 tid=0x67691ec8 nid=0x3ef2 waiting on monitor

[0xa53ff000..0xa53ff89c]
at oracle.ons.NotificationQueue.internalDequeue(NotificationQueue.java:251)
at oracle.ons.NotificationQueue.dequeue(NotificationQueue.java:224)
at oracle.ons.Subscriber.receive(Subscriber.java:136)
at com.evermind.server.OC4JMonitorThread.run(OC4JMonitorThread.java:419)
at java.lang.Thread.run(Thread.java:479)
"Thread-10" daemon prio=1 tid=0x676930f0 nid=0x3ef1 waiting on monitor

[0xa55ff000..0xa55ff89c]
at java.lang.Object.wait(Object.java:415)
at oracle.ons.SenderThread.run(SenderThread.java:81)
The “Full thread dump” Interpretation
=====================================================
"OC4JMonitorThread" daemon prio=1 tid=0x67691ec8 nid=0x3ef2 waiting on monitor
[0xa53ff000..0xa53ff89c]
=====================================================
[OC4JMonitorThread] – name (type) of the thread

OC4JMonitorThread.java
Thread.currentThread().setName("OC4JMonitorThread");
[nid=0x3ef2] – Thread ID (in decimal = 16114)
[waiting on monitor] – State of the thread at the snap time
[oracle.ons.NotificationQueue.internalDequeue] – java stack

TYPES of the Threads
OC4J Specific
AJPConnectionListener [0.0.0.0/0.0.0.0:3001] 1
RMIServer [0.0.0.0/0.0.0.0:3102] count:1 1
RMIServer [0.0.0.0/0.0.0.0:3102] count:2 1
JMSServer 1
OC4JMonitorThread 1
TaskManager 1
Thread 5
ApplicationServerThread 195
JAVA GENERIC
VM Thread 1
VM Periodic Task Thread 1
Reference Handler 1
Signal Dispatcher 1
Suspend Checker Thread 1
Finalizer 1
1
STATES of the Threads
z Waiting on monitor
In most of cases it is an idle state of the thread. The thread is waiting
for a task to execute or voluntary sleeps
All the available ApplicationServerThreads are in this state
z Runnable
The thread is executing application code or reading from the socket
All listeners must be in this state
z Waiting for monitor entry
Waiting to lock an object (other thread is holding the lock). This
happens if two or more threads are trying to execute synchronized java
code
None of threads must be in this state for a long time
If the system thread stays in this state for a long time you are in trouble
If many threads are in this state you have the situation of serialization
Threads Examples
1. OC4JMonitorThread is waiting for a work. Usual state for most of the
threads
"OC4JMonitorThread" daemon prio=1 tid=0x679a3518 nid=0x573 waiting on monitor [0xa53ff000..0xa53ff89c]
2. "ApplicationServerThread-5" – idle worker thread, is waiting for a task

"ApplicationServerThread-5" prio=1 tid=0x6794da88 nid=0x43f waiting on monitor [0xb15ff000..0xb15ff89c]
at EDU.oswego.cs.dl.util.concurrent.BoundedBuffer.poll(BoundedBuffer.java:170)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask(PooledExecutor.java:768)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:802)
3. "Thread-9" is reading from the socket (connected to the ONS process,

port 7100). This thread most of time is waiting for a data to be read from
the socket
"Thread-9" daemon prio=1 tid=0x67a02920 nid=0x571 runnable [0xa57ff000..0xa57ff89c]
at java.net.SocketInputStream.socketRead(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:85)
at oracle.ons.InputBuffer.readMoreData(InputBuffer.java:267)
at oracle.ons.InputBuffer.getNextString(InputBuffer.java:222)
at oracle.ons.ReceiverThread.run(ReceiverThread.java:225)
Threads Examples
4. "AJPConnectionListener” is waiting for new connections from the

Apache processes
"AJPConnectionListener [0.0.0.0/0.0.0.0:3001]" prio=1 tid=0x67911828 nid=0x439 runnable
[0xb19ff000..0xb19ff89c]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:463)
at java.net.ServerSocket.implAccept(ServerSocket.java:238)
at java.net.ServerSocket.accept(ServerSocket.java:217)
at com.evermind.server.http.AJPConnectionListener.run(AJPConnectionListener.java:58)
5. "ApplicationServerThread-5" is reading the request from the Apache

process
"ApplicationServerThread-77" prio=1 tid=0x8391048 nid=0x4a75 runnable [0xb4dff000..0xb4dff89c]
at com.evermind.io.SingleReadBufferInputStream.readChunk(SingleReadBufferInputStream.java:116)
at com.evermind.io.SingleReadBufferInputStream.read(SingleReadBufferInputStream.java:35)
at com.evermind.server.http.AJPRequestHandler.initRequest(AJPRequestHandler.java:345)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:131)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:804) Apache
Threads Examples
6. "ApplicationServerThread-71" is executing database query, waiting for the results to be retrieved
"ApplicationServerThread-71" prio=1 tid=0x8379130 nid=0x4a6f runnable [0xb59fe000..0xb59ff89c]
at oracle.net.ns.Packet.receive(Unknown Source)
at oracle.net.ns.DataPacket.receive(Unknown Source)
at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
at oracle.net.ns.NetInputStream.read(Unknown Source)
at oracle.jdbc.ttc7.MAREngine.unmarshalUB1(MAREngine.java:931)
at oracle.jdbc.ttc7.MAREngine.unmarshalSB1(MAREngine.java:893)
at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:375)
at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1986)
at oracle.jdbc.ttc7.TTC7Protocol.parseExecuteFetch(TTC7Protocol.java:1144) DB
at oracle.jdbc.driver.OracleStatement.doExecuteQuery(OracleStatement.java:2484)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:2850)
at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:609)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:537)
at oracle.jbo.server.ViewObjectImpl.getQueryHitCount(ViewObjectImpl.java:1522)
at oracle.jbo.server.QueryCollection.getEstimatedRowCount(QueryCollection.java:1530)
at oracle.jbo.server.ViewRowSetImpl.getEstimatedRowCount(ViewRowSetImpl.java:1482)
at oracle.jbo.server.ViewObjectImpl.getEstimatedRowCount(ViewObjectImpl.java:4596)
at _common._comp._CMN__DataScrollerComponent._jspService(_CMN__DataScrollerComponent.java:115)
at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:56)
at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:317)
at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:476)
at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:390)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:65)
at oracle.security.jazn.oc4j.JAZNFilter.doFilter(JAZNFilter.java:283)
at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:566)
at com.evermind.server.http.ServletRequestDispatcher.include(ServletRequestDispatcher.java:119)
at com.evermind.server.http.GetParametersRequestDispatcher.include(GetParametersRequestDispatcher.java:95)
at com.evermind.server.http.EvermindPageContext.include(EvermindPageContext.java:280)
at oracle.jbo.html.jsp.datatags.ComponentTag.doStartTag(ComponentTag.java:70)
at _skd._d001._D2260F__Browse._jspService(_D2260F__Browse.java:249)
at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:65)
at oracle.security.jazn.oc4j.JAZNFilter.doFilter(JAZNFilter.java:283)
at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:566)
at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:306)
at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:767)
Useful Java Runtime Parameter
z -XX:+JavaMonitorsInStackTrace (opmn.xml)
<java-option value="-server -Xms512M -Xmx512M -verbose:gc -XX:+JavaMonitorsInStackTrace"/>
z The parameter wasn’t set

"ApplicationServerThread-18" prio=1 tid=0x692c40e0 nid=0x2f92 waiting for monitor entry [0xbc1ff000..0xbc1ff89c]
at oracle.jbo.server.ViewRowSetIteratorImpl.setRangeSize(ViewRowSetIteratorImpl.java:376)
at oracle.jbo.server.ViewRowSetImpl.setRangeSize(ViewRowSetImpl.java:1659)
at oracle.jbo.server.ViewObjectImpl.setRangeSize(ViewObjectImpl.java:4626)
at lv.vovaa.panda.common.datatags.DataSourceCmn.doStartTag(DataSourceCmn.java:189)
at _nsk._nsk001._M2801F__Edit._jspService(_M2801F__Edit.java:150)
...
OC4J
z The parameter was set
"ApplicationServerThread-96" prio=1 tid=0x69d6c8c0 nid=0xdce waiting for monitor entry [0xb25ff000..0xb25ff89c]
at oracle.jbo.server.ViewRowSetIteratorImpl.setRangeSize(ViewRowSetIteratorImpl.java:376)
- waiting to lock <460e0338> (a oracle.jbo.JboSyncLock)
at oracle.jbo.server.ViewRowSetImpl.setRangeSize(ViewRowSetImpl.java:1659)
at oracle.jbo.server.ViewObjectImpl.setRangeSize(ViewObjectImpl.java:4626)
at lv.vovaa.panda.common.datatags.DataSourceCmn.doStartTag(DataSourceCmn.java:189)
- locked <45f722d0> (a lv.vovaa.panda.common.datatags.DataSourceCmn)
at _nsk._nsk001._AP3220F__Tickets__browse._jspService(_AP3220F__Tickets__browse.java:105)
...
Useful Java Runtime Parameter
z It is possible to determine the blocking thread easily
"ApplicationServerThread-55" prio=1 tid=0x69d512b8 nid=0xda4 runnable [0xb77fe000..0xb77ff89c]
at oracle.net.ns.Packet.receive(Unknown Source)
...
at oracle.jbo.server.QueryCollection.executeQuery(QueryCollection.java:549)
at oracle.jbo.server.ViewObjectImpl.executeQueryForCollection(ViewObjectImpl.java:2600)
- locked <460e0338> (a oracle.jbo.JboSyncLock)
at lv.vovaa.panda.common.pbo.PboViewObjectImpl.pboExecuteQueryForCollection(PboViewObject.java:324)
at lv.vovaa.panda.common.pbo.PboViewObjectImpl.executeQueryForCollection(PboViewObjectImp.java:309)
at oracle.jbo.server.ViewRowSetImpl.execute(ViewRowSetImpl.java:521)
...
z Then we can locate the source code for troubleshooting purposes

ViewObjectImpl.java
protected void executeQueryForCollection(Object obj, Object aobj[], int i)

{
synchronized(getSyncLock())
{
int j = 0;
if(InstrumentedEvent.isActive)
j = InstrumentedEvent.startEvent(EventGroup.EXECUTE_QUERY, "ViewObject
executeQueryForCollection " + getName());
((QueryCollection)obj).executeQuery(aobj, i);
if(InstrumentedEvent.isActive)
InstrumentedEvent.endEvent(j);
}
}
Java DEADLOCK
z Information from the OC4J process “Full thread dump”

Full thread dump:
"ApplicationServerThread-126" prio=1 tid=0x881f668 nid=0x5d59 waiting for monitor
entry [0xa19ff000..0xa19ff89c]
...
...
FOUND A JAVA LEVEL DEADLOCK:
----------------------------
"ApplicationServerThread-58":
waiting to lock monitor 0x80b9804 (object 0x4595b408, a java.lang.Object),
which is locked by "ApplicationServerThread-51"
"ApplicationServerThread-51":
waiting to lock monitor 0x80b9564 (object 0x45955750, a
com.evermind.server.http.EvermindHttpSession),
which is locked by "ApplicationServerThread-58"
JAVA STACK INFORMATION FOR THREADS LISTED ABOVE:
------------------------------------------------
Java Stack for "ApplicationServerThread-58":
==========
at oracle.jbo.http.HttpContainer.setValue(HttpContainer.java:352)
- waiting to lock <4595b408> (a java.lang.Object)
z JDK/JRE (before 1.5.X) is unable to solve DEADLOCK situations
z OPMN doesn’t detect such kind of situations
z You need to take action from outside
Gather troubleshooting information and restart OC4J process
Classification of Problem Areas
OC4J
JVM(java)
4.
PORT: 3001
AJP
JMSServer
JMSServer
Connection
Listener ApplicationServerThread
New Request
as
Apache
sig
n ApplicationServerThread
1.
TaskManager
3. DB
2. ApplicationServerThread
JDK
OS
1. OC4J container inside (infinitive cycles, waits on other threads, deadlocks)

2. Interface to the Apache processes
3. Interface to the database
4. 500 Internal Error
Java “Full thread dump” (conclusion)
z It is an extremely useful source of information for

understanding OC4J internals and troubleshooting
z Take a few snaps to see dynamics
z You need to analyze whole OC4J process “Full thread

dump” to make conclusions
z Java on Java. The ideal person for such analysis is the

person who has a good knowledge of java, used
technologies, application framework as well as application
running on the container
z DBA/Administrator should ensure all necessary

information to be available to application troubleshooting
person
Log Files
z OC4J output file

Located at
OH/opmn/log/<comp_name>.default_island.<proc_nu
m> (example PANDA.default_island.1)
Store standard output of OC4J process (JAVA DUMP,
task manager info, debug info -Dajp.io.debug=true)
It can be useful to add time stamp information
* * * * * echo "Timestamp: `date`" >>
/u01/app/oracle/product/J2EE01/opmn/logs/PANDA.default_island.1
z OC4J default-web-access.log
Located at
OH/ j2ee/PANDA/log/PANDA_default_island_1/default-web-access.log
It is an analogue of access_log for Apache

OS Monitoring Commands
z System wide:
vmstat, iostat, uptime, pstree
z Top resources consumers:

top, ps -eo pcpu,pid,args | sort -k1n | tail -10
z Particular process information

$ while [ 1 ] ; do ps -p 13285 -o pcpu | grep -v CPU ; sleep 1 ;
done
25.3
25.3
25.3
$ while [ 1 ] ; do ps -p 19273 -o vsz,rss | grep -v VSZ ; sleep 1 ;
done
1095372 258244
1095372 258244
1095372 258244
$ ps -wo args --width 4000 -p 6018
OS Tracing Commands
z strace - trace system calls and signals for particular process
$ strace –s1000 -f -p 7495
read(19,
$ strace –s1000 –f -p 27021
--- SIGSTOP (Stopped (signal)) ---
read(104, "\0224\0\2\0\0", 4096) = 6
send(104, "A", 1, 0) = 1
send(104, "B", 1, 0) = 1
send(104, "\0", 1, 0) = 1
send(104, "\3", 1, 0) = 1
send(104, "\6", 1, 0) = 1
send(104, "\37", 1, 0) = 1
send(104, "\372", 1, 0) = 1
read(104, "\0224\0\2\0\0", 4096) = 6
send(104, "A", 1, 0) = 1
send(104, "B", 1, 0) = 1
send(104, "\0", 1, 0) = 1
send(104, "\3", 1, 0) = 1
send(104, "\6", 1, 0) = 1
send(104, "\37", 1, 0) = 1
send(104, "\372", 1, 0) = 1
read(104, "\0224\0\2\0\0", 4096) = 6
...
z pstack - print a stack trace of running processes
Apache Side Information
z Apache Server Status page
http://ap1.voava.lv:7780/server-status?refresh=15
z log files
Apache log files $OH/Apache/Apache/logs
Sometimes information can mislead
access_log
192.168.2.12 - - [01/Feb/2005:16:43:34 +0200] "GET
/PANDA/NSK/NSK003/NS0291F_Browse.jsp HTTP/1.1" 500 0
erorr_log
[Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0080: After calling
Apache's ap_bwrite(), got an return value: -1 and the client could have aborted in the
middle of receiving the chunk data.
[Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0058: Failed to handle
response chunk data passed in from oc4j via ajp13.
[Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0035: After hdndling
ajp13 response message, got an invalid value: -2.
[Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0121: Failed to service
request with network worker: PANDA_15 and it is not recoverable.
[Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0013: Failed to call
destination: PANDA's service() to service the request.
Java Code Decompilation
z Extremely useful for troubleshooting purposes
Unzip archive (jar/zip)
Decompile *.class files (http://www.kpdus.com/jad.html)
z OC4J
OH/j2ee/home/oc4j.jar
z JDBC
OH/jdbc/lib/classes12.zip
OH/jdbc/lib/classes12dms.jar
z DCM
OH/dcm/lib/dcm.jar
OH/dcm/lib/oc4j_deploy_tools.jar
OH/dcm/lib/oc4j_remote_deploy.jar
z DMS
OH/lib/dms.jar
z BC4J
OH/BC4J/redist/bc4j.ear
OH/BC4J/jlib/*
OH/BC4J/lib/*
Java Code Decompilation Usage Example
z Decompile all oc4j.jar classes

z Grep through *.java files for “System.getProperty”
9.0.4 oc4j.jar
…
opmnPingInterval
opmn.query.debug
oracle.arraylist.deepCopy
oracle.aurora.jem.scheduling.threads
oracle.dms.console.DMSConsole
oracle.dms.gate
oracle.dms.sensors
oracle.dms.transtrace.ecidenabled
oracle.home
oracle.ias.jcache
Debugging Switches
z Debug switches are documented in
Containers for J2EE User's Guide
10g Release 2 (10.1.2)
Part No. B14011-01
B Additional Information
B.8 OC4J Command-Line Options and System Properties
z Debug information example (-Dajp.io.debug=true)
PANDA.default_island.1
1107387907360:ThrP[50] ThrQ[0]
1107387907360:CxP[20] CxQ[0] - WebC[0] RmiC[0]
AJP: Input request length is: 1
AJP: AJPOutputStream::write() of 1
AJP: AJPOutputStream::writeChunk(): 1
1107387908370:ThrP[50] ThrQ[0]
1107387908370:CxP[20] CxQ[0] - WebC[0] RmiC[0]
z Some of available categories are : ajp, http, jms, rmi, jdbc
Documentation and Internet
z The information in 9.0.X documentation is very limited

z Oracle AS Documentation Library 10g Release 2 (10.1.2)
z Highly Recommended books
Containers for J2EE User's Guide (144)
Containers for J2EE Standalone User's Guide (134)
Server Administrator's Guide (542)
OPMN Administrator's Guide (158)
DCM Administrator's Guide (116)
Server Performance Guide (234)
z 10g Best Practices 10g (9.0.4) Part No. B12223-01
z BC4J JDeveloper Online Help system.
z www.google.com
z http://otn.oracle.com
z http://metalink.oracle.com
Metalink Patches
z Look through available patches.
………………………………………………………………..
Metalink Patches
z Strongly recommended in case of 9.0.3 OC4J

Metalink Patches
z Monitor for new patches, which can be relative to your
environment
Troubleshooting information gathering (example)
Performance tuning
Our Findings (Java Workers)
z Preset java workers threads for OC4J process

OH/j2ee/<comp_name>/config/server.xml
<global-thread-pool
min="100" max="300" queue="30" keepAlive="-1"
cx-min="100" cx-max="300" cx-queue="30" cx-keepAlive="-1"
debug="true"/>
Set min to high value. It allows OC4J not to create the additional
threads on the fly
You don’t have to take care on max, keepAlive and queue parameters then
OC4J doesn’t need to manage queue
debug="true" – Allows to see statistics in the OC4J process
standard output file
It is useful as troubleshooting information as well. Record appears in the output file at
the end of each Task manager cycle.
<PANDA.default_island.1>
1107390315893:ThrP[100] ThrQ[0]
1107390315893:CxP[100] CxQ[0] - WebC[0] RmiC[0]
1107390316903:ThrP[100] ThrQ[0]
1107390316903:CxP[100] CxQ[0] - WebC[0] RmiC[0]
...
Our Findings (OC4J Processes)
z Use many OC4J processes without enabled state replication

OH/opmn/conf/opmn.xml
<oc4j maxRetry="3" instanceName="PANDA" gid="PANDA" numProcs="1">
<config-file path="/u01/app/oracle/product/J2EE01/j2ee/PANDA/config/server.xml"/>
<java-option value="-server -Xms512M -Xmx512M"/>
<oc4j-option value="-properties "/>
<port ajp="3001-3100" jms="3201-3300" rmi="3101-3200"/>
<island id="default_island" numProcs="3"/>
</oc4j>
z Initially it was recommended by Oracle support while TAR resolution

z OC4J container is more stable on a lower users load
z mod_oc4j distributes requests through 3 OC4J identical containers
z 1/3 of users load is handled by each OC4J process
z Do not use state replication
It can add significant load on CPU
It complicates configuration
It can lead to opposite effect
z Be careful
You have to test this configuration for your application. It can lead to unpredictable DB connection
behavior
Memory/threads configuration is for each process of the OC4J instance
Our Findings (Memory)
OH/opmn/conf/opmn.xml
<oc4j …
<java-option value="-server -Xms512M -Xmx512M"/>
</oc4j>
z Set max=min
JVM doesn’t need to manage memory allocation dynamically
You have reserved memory on the server
z Java Garbage collector
There is a lot of information around
http://developers.sun.com/techtopics/mobility/midp/articles/garbagecollection2/
http://www.cons.org/cmucl/doc/gc-tuning.html
-verbose:gc
z Java Profilers
Add significant overhead to be used in production environments
Extremely useful in test/dev environments for troubleshooting
Our Findings (Memory)
z Monitor memory usage in production

grep log files for “out of memory” error
use OS utilities
use dmstool
$ dmstool -table JVM | grep 3001 | grep Memory

/ap1.voava.lv/OC4J:3001:7003/JVM/freeMemory.value 495796 kbytes
/ap1.voava.lv/OC4J:3001:7003/JVM/freeMemory.minValue 495796.0 kbytes
/ap1.voava.lv/OC4J:3001:7003/JVM/freeMemory.maxValue 522468.0 kbytes
/ap1.voava.lv/OC4J:3001:7003/JVM/totalMemory.value 524032 kbytes
/ap1.voava.lv/OC4J:3001:7003/JVM/totalMemory.minValue 524032.0 kbytes
/ap1.voava.lv/OC4J:3001:7003/JVM/totalMemory.maxValue 524032.0 kbytes
ap1.voava.lv*J2EE01-/u01/app/oracle/product
$
Our Findings (Stresstesting)
z The best way to tune performance is a

stresstesting
z According to our experience
After Development had performed the performance
testing, a lot of problems were solved
An automatic testing tool has been used (Rational
Robot)
Different modules were tested
Different load profiles were tested
Testing was repeated several times until acceptable
results were achieved
Response Time Monitoring by DMS
Apache OC4J
OHS [c] mod_plsql JVM [java]
handle
Send
Request B
Handle B ApplicationServerThread
Read
Resolve
Process B
JSP Processing
mod_oc4j Servlet Processing
DB
handle
Process E
Handle E
Handle B
Handle E
Request E
mod_perl
Recive mod_php
...
DMS Sensors DMS Sensors
DMS-client
DMS.JAR (java)
statistics reflecting
Response Time Breakdown Example
$ dmstool -table ohs_server | grep time
.../Apache/connection.time 48123608767 usecs
.../Apache/request.time 15916131942 usecs
.../Apache/handle.time 15913988395 usecs
rhoracle.voava.lv*J2EE01-/u01/app/oracle/product/J2EE01/Apache/Apache/conf
$
$ dmstool -table ohs_module | grep time | sort -k2n | tail -2
.../Apache/Modules/mod_dms.c/handle.time 8081655 usecs
.../Apache/Modules/mod_oc4j.c/handle.time 15905072698 usecs
$
$ dmstool -table oc4j_jspExec | grep time
.../oc4j/PANDA/WEBs/PANDA/JSPs/processRequest.time 17 563 793 msecs
$
$ dmstool -table "oc4j_jsp(threadsafe=true)" | grep time | sort -k2n | tail -5
.../oc4j/PANDA/WEBs/PANDA/JSPs/NSK/NSK003/NS0285F_searchForm.jsp/service.time 1 072 570 msecs
.../oc4j/PANDA/WEBs/PANDA/JSPs/SKD/d001/D2261F_PersonsBrowse.jsp/service.time 1 407 663 msecs
.../oc4j/PANDA/WEBs/PANDA/JSPs/NSK/NSK003/NS0291F_Browse.jsp/service.time 2 942 655 msecs
.../oc4j/PANDA/WEBs/PANDA/JSPs/SKD/d001/D2261F_Edit.jsp/service.time 3 151 184 msecs
.../oc4j/PANDA/WEBs/PANDA/JSPs/SKD/d001/D2260F_Browse.jsp/service.time 4 847 408 msecs
rhoracle.voava.lv*J2EE01-/u01/app/oracle/product/J2EE01/Apache/Apache/conf
$
DMS Sensors Classification
z Informative sensors – reflect a configuration information and some

system wide statistics
ohs_child, ohs_responses,
JVM, oc4j_ear, oc4j_opmn, oc4j_task,
JDBC_Driver , JDBC_Connection
z Average performance sensors – reflect response time totals by
modules, applications, etc.
ohs_server , ohs_module, ohs_virtualHost,
oc4j_context, oc4j_web_module, oc4j_jspExec
z Units performance sensors – reflect response time statistics for each
unit
oc4j_servlet
oc4j_jsp(threadsafe=true)
Collecting and Reflecting DMS Statistics
z DMS Sensors statistics is incremental

z Make periodic snaps of statistics for further analysis
z Develop interface to access statistics easily
Graphical interface for system wide statistics
Textual interface for units (JSP/Servlet) response time analysis

Collecting and Reflecting DMS Statistics
SYS:PROD> @sp_jsp
Completed Snapshots
Snap
Id Snap Started
------- -------------------
530 2005-02-03 12:33:26
531 2005-02-03 13:03:24
532 2005-02-03 13:33:27
549 2005-02-03 14:03:32
Specify the Begin and End Snapshot Ids

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Enter value for begin_snap: 532
Enter value for end_snap: 549
Saved in c:\temp\532_549.log
Snap Id Snap Time

------- ------------------
Begin Snap: 532 2005-02-03 13:33:27
End Snap: 549 2005-02-03 14:03:32
Elapsed: 30.08 (mins)
JSP ordered by Response time
JSP Name Response Execution Average

(sec) Time(sec) Count Response
------------------------------------------------------------------ --------- ---------- ----------
../3001/../NSK/NSK001/AP3220F_TicketImportProcessOneByOne.jsp 956.439 2 478.22
../3002/../NSK/NSK001/AP3220F_Tickets_browse.jsp 708.967 71 9.99
../3001/../NSK/NSK001/M2801F_Browse.jsp 533.389 74 7.21
../3003/../NSK/NSK001/P2600F_XMLImportProcess.jsp 471.106 1 471.11
../3003/../LRG/l001/L3000F_Browse.jsp 293.775 92 3.19
../3002/../NSK/NSK001/AP3221F_ApnAccountingDocFilter.jsp 291.382 6 48.56
../3002/../common/comp/CMN_DataFilterComponent.jsp 238.75 35 6.82
../3001/../NSK/NSK001/SP0301F.jsp 216.499 32 6.77
../3001/../SKD/d001/D2262F_Browse.jsp 208.709 7 29.82
../3001/../NSK/NSK001/SP0301F_filter.jsp 168.957 6 28.16
../3001/../NSK/NSK001/Z3601F_Save.jsp 80.127 30 2.67
../3001/../common/comp/CMN_LOVcomp.jsp 73.544 52 1.41
../3003/../NSK/NSK001/SP0301F.jsp 72.548 29 2.5
../3002/../NSK/NSK001/AP3221F_Save.jsp 68.509 91 .75
../3001/../common/comp/CMN_DataHandlerComponent.jsp 68.46 765 .09
../3002/../common/comp/CMN_DataHandlerComponent.jsp 57.144 722 .08
SYS:PROD>
Monitor All the Tiers of the System
z Do not forget to monitor

OS and HW resources utilization
Network
Database
Instrument Your Code
z AskTom.oracle.com
http://asktom.oracle.com/pls/ask/f?p=4950:8:16675873436862412624::NO::F4950_P8_DISPLAYID,F4950_P8_CRITERIA:12836314571537
…
Another example is asktom.oracle.com. Just go there and click on any article
from the home page. You'll see a URL similar to:
http://asktom.oracle.com/pls/ask/f?p=...::NO::
If you simply type over that URL and replace the word NO with YES - you'll see
the same page but with lots and lots of state/timing information dumped into it.
…
BC4J
OC4J Internals & BC4J
ONS
Thread
OC4JMonitor
new req Thread
PORT: 3001
AJP
Listener Thread
Java native
... ONet
Memory Structures
JBO Application Module Pool JBO Connection pool

JBO Application Module Pool
JBO Application Module Pool
JBO Application Module Pool Application objects
JBO View Objects
Http Session List ... ...
Java native
RMI
RMIServer
3101
PORT: 3101
Periodic Task
VM Thread
Thread
JMSServer
3201

JMSServer
JMSServer Suspend
JMSServer
Handler
Usage Considerations
z Most information is available under JDeveloper

cover
z Use development version of BC4J
z Deploy BC4J.EAR to your OC4J component
(do not use home component)
z There is the BC4J monitoring page
Useful Information Sources
0. Metalink Note:273350.1 BC4J configuration tuning tips
1. Metalink Note:283476.1 BC4J (ADF BC) Configuration Tuning Tips
2. Metalink Note:238634.1 JBO / BC4J Parameters And Defaults
3. Metalink Note:231141.1 BC4J Row Spillover and tuning
4. How to Performance Tune an ADF Business Components (ADF BC) Application
September 23, 2004
Steve Anderson and Michael Gantman
http://www.oracle.com/technology/products/jdev/tips/muench/ampooling/index.html
5. Understanding Application Module Pooling Concepts and Configuration Parameters
Author: Steve Muench, BC4J Development Team
Date: August 5, 2004
http://www.oracle.com/technology/products/jdev/tips/muench/ampooling/index.html
6. Getting Application Module Pool Statistics to Aid with Size Tuning
By Steve Muench
http://radio.weblogs.com/0118231/stories/2004/05/10/gettingApplicationModulePoolStatisticsToAidWithSizeTuning.html
7. Overview of Temporary Tables Created By BC4J
An Oracle technical white paper
April 2002
http://www.oracle.com/technology/products/jdev/htdocs/bc4j/bc4j_temp_tables.html
8. How To Support Dynamic JDBC Credentials
An Oracle BC4J technote
Version 3: April 23rd 2002
http://www.oracle.com/technology/products/jdev/howtos/bc4j/howto_dynamic_jdbc.html
9. BC4J/JClient Performance Study
Written by Steve Muench, Oracle Corporation
January, 2004
http://www.oracle.com/technology/products/jdev/tips/muench/jclientperf/index.html
10. Dive into BC4J and ADF
Tips and tricks from Steve Muench on using Oracle's J2EE frameworks, JDeveloper Java IDE, and XML technologies...
http://radio.weblogs.com/0118231/
Performance Tuning Parameters
jbo.pers.max.active.nodes=1000
jbo.pers.max.rows.per.node=1000
jbo.doconnectionpooling=true
jbo.poolminavailablesize=0
jbo.poolmaxavailablesize=600
jbo.poolmonitorsleepinterval=600000
jbo.poolmaxinactiveage=600000
jbo.ampool.monitorsleepinterval=300000
jbo.ampool.maxinactiveage=600000
JBO Parameter lookup order:

- Set programmatically or configuration in bc4j.xcfg file)
- Applet tags
- -D flags (or oc4j.properties file)
- BC4J.properties file in current working directory of Java VM
- /oracle/jbo/BC4J.properties resource in the runtime class path
- /oracle/jbo/server/jboserver.properties resource (by default found inside bc4jmt.jar)
- /oracle/jbo/common/Diagnostic.properties resource (by default found inside bc4jmt.jar)
- System defined default as a fallback
Summary
Summary
z Introduction
z Usage considerations
z OC4J Internals
z OC4J Troubleshooting
z OC4J Performance tuning
z BC4J
Q&A

Velikanovs - Performance Tuning and Troubleshooting For Oracle OC4J (Slides - Long Version)

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Velikanovs - Performance Tuning and Troubleshooting For Oracle OC4J (Slides - Long Version)

Uploaded by

Copyright:

Available Formats

Performance Tuning and

Troubleshooting for Oracle OC4J

z Local Oracle Marketing team has pushed

z Healthcare financial system

z 10x5 hour availability

z OC4J is the present strategic direction for Oracle

z OC4J is a strategic direction for Oracle

z There is not much information about this product

 I’m not going to tell you about

 The technical layer is of great interest

Http Session List ... JDBC Connection pool

Signal Dispatcher Finalizer

z You may make the configuration

DMS-metrics OPMN DCM-daemon

z Use the latest version of Oracle AS depending on

z Keep running only the components you need

z Get a commitment from the Development to use industry

z How to Determine the Versions

 JRE: java –version

 OC4J: java -jar oc4j.jar –version

 JDBC: a program with a call to meta.getDriverVersion

 BC4J: a program with a call to meta.getDriverVersion

z Have at least one senior level developer, who understands used

!!! Stresstest your application !!!

Http Session List ... JDBC Connection pool

Signal Dispatcher Finalizer

z Java native threads

1. OC4J container inside (infinitive cycles, waits on other threads, deadlocks)

z Java Full thread dump

z Troubleshooting information gathering (example)

z How to get a Full thread dump?

z Useful java runtime parameter

z Get pid of the OC4J process

z Find out the output file name (OH/opmn/logs/*)

$ ps -wo args --width 4000 -p 15874 | grep -v COMMAND | awk -F"oracle.ons.indexid="

z Send SIGQUIT (Quit from keyboard) signal

"Thread-13" daemon prio=1 tid=0x82c8690 nid=0x3f15 waiting on monitor

"Thread-12" prio=1 tid=0x67b0cce0 nid=0x3e02 waiting on monitor [0..0xbffeb62c]

"OC4JMonitorThread" daemon prio=1 tid=0x67691ec8 nid=0x3ef2 waiting on monitor

"Thread-10" daemon prio=1 tid=0x676930f0 nid=0x3ef1 waiting on monitor

[OC4JMonitorThread] – name (type) of the thread

[nid=0x3ef2] – Thread ID (in decimal = 16114)

[waiting on monitor] – State of the thread at the snap time

[oracle.ons.NotificationQueue.internalDequeue] – java stack

2. "ApplicationServerThread-5" – idle worker thread, is waiting for a task

3. "Thread-9" is reading from the socket (connected to the ONS process,

4. "AJPConnectionListener” is waiting for new connections from the

5. "ApplicationServerThread-5" is reading the request from the Apache

z The parameter wasn’t set

z Then we can locate the source code for troubleshooting purposes

protected void executeQueryForCollection(Object obj, Object aobj[], int i)

z Information from the OC4J process “Full thread dump”

1. OC4J container inside (infinitive cycles, waits on other threads, deadlocks)

z It is an extremely useful source of information for

z Take a few snaps to see dynamics

z You need to analyze whole OC4J process “Full thread

z Java on Java. The ideal person for such analysis is the

z DBA/Administrator should ensure all necessary

z OC4J output file

 It is an analogue of access_log for Apache

z Top resources consumers:

z Particular process information

z Decompile all oc4j.jar classes

I’m not going to tell you about

The technical layer is of great interest

JRE: java –version

OC4J: java -jar oc4j.jar –version

JDBC: a program with a call to meta.getDriverVersion

BC4J: a program with a call to meta.getDriverVersion

It is an analogue of access_log for Apache

Textual interface for units (JSP/Servlet) response time analysis