Professional Documents
Culture Documents
Version 1.0
Jurijs Velikanovs
http://otn.oracle.com/ocm
Introduction....................................................................................................................2
OC4J Internals ...............................................................................................................3
Physical structure .......................................................................................................3
Memory structures .....................................................................................................3
Processes ....................................................................................................................3
OC4J Troubleshooting ...................................................................................................9
Classification of Problem Areas ................................................................................9
Possible Approaches & Information Sources ..........................................................10
Java “Full thread dump” ..........................................................................................11
How to get Full thread dump? .............................................................................11
The interpretation of Full thread dump................................................................13
Useful Java runtime parameter ............................................................................16
Java DEADLOCK problem .................................................................................17
Log files ...................................................................................................................18
OC4J output file...................................................................................................18
OC4J default-web-access.log...............................................................................19
OS Monitoring commands.......................................................................................19
OS Tracing commands.............................................................................................20
Apache side information ..........................................................................................22
Apache Status page ..............................................................................................22
Apache log files ...................................................................................................22
Java code decompilation ..........................................................................................23
Debugging Switches ................................................................................................23
Documentation, External searches and Metalink.....................................................24
Documentation.....................................................................................................24
External searches .................................................................................................24
OC4J Performance .......................................................................................................26
Our Findings ............................................................................................................26
Java Workers........................................................................................................26
OC4J Processes....................................................................................................26
Memory................................................................................................................27
Stresstesting .........................................................................................................28
Response Time Monitoring by DMS.......................................................................29
Response Time Breakdown Example ..................................................................29
DMS Sensors Classification ................................................................................30
Collecting and Reflecting DMS Statistics ...........................................................30
Appendix A. How to diagnose opened files, sockets, pipes by OC4J process? ..........32
Introduction
There is almost no information about OC4J troubleshooting and performance tuning.
This paper covers available methods of OC4J performance and availability
troubleshooting. The information provided is based on several years of author’s
experience working with this technology. The main question paper is trying to cover
is how to troubleshoot OC4J production environments, where it is almost impossible
to use the traditional Java troubleshooting methods such as Java profilers because of
the significant overhead.
The first part of the paper describes the internals of OC4J. It is vital to understand
OC4J main processes (threads) and their responsibilities to be able to troubleshoot
OC4J related problems successfully. What are the main thread (process) types? What
tasks are they responsible for? What are the relationships between different types of
the processes? “OC4J Internals” part of this paper gives the answers to these
questions.
There is no any single method which would give the complete information about the
problem within OC4J environment. The “OC4J Troubleshooting” section gives you
the information about available methods. Use all available methods to gather the
sufficient troubleshooting data for further analysis, localization of the problem area
and elimination of the problem.
“OC4J Performance” describes author’s main findings in the context of OC4J
performance tuning and available performance monitoring capabilities.
Note: As English is not the author’s native language, author apologizes for possible
"syntax errors" in this document.
OC4J Internals
Warning!!! Please note that all the information provided in this section is based on author’s experience
working with OC4J processes. Java “Full thread dump” and Java classes decompilation were the main
methods to get the information. Because of that and because the fact that Oracle Corporation doesn’t
publish OC4J architecture publicly, the OC4J architecture described below possibly is not 100%
accurate. Author hopes that it is correct in main terms and will be useful for reader to understand better
how to troubleshoot and tune OC4J.
Physical structure
OC4J physical structure (source, configuration and application files structure) is
described in the Oracle documentation (See. OC4J Standalone User's Guide1).
However you can find useful information about how to diagnose opened files,
sockets, pipes by OC4J process in the appendix (see. Appendix A. How to diagnose
opened files, sockets, pipes by OC4J process?).
Memory structures
Memory structures of OC4J process are not covered in this paper in details. They are
highly dependent on the application you are using. For sure there are OC4J specific
memory structures for storing such information as HTTP session list, managing
threads and connections pools etc. as well. But they are not covered in this paper.
There will be additional paper on OC4J memory structures in the future probably.
Processes
For troubleshooting and performance purposes it is extremely important to understand
the architecture of OC4J instance (process). What are the main treads? What are their
responsibilities? OC4J process architecture is a matter of a great interest in a context
of this paper.
OC4J is written entirely in Java language and it is a Java application. Not depending
on the configuration you are using, JRE2 is used to run OC4J instance. The command
below is used to start-up any OC4J instance:
<jre location>/bin/java <1 set of parameters> -jar oc4j.jar <2 set of
parameters>
Any java process is multithreaded. It is vital to know multithreading concepts for
understanding OC4J internals. Multithreading is the way how many tasks (threads)
can be executed within a single java process. Threads are executed in the same
context and have access to all process memory structures. Different operational
systems have different implementations of multithreading within the single process.
In most of the operational systems you will see OC4J as single process. There are
different possibilities on different OS to monitor threads within the single process.
Sun Solaris for example realizes multithreading through light-weight processes.
Light-weight processes can be monitored easily. You can use prstat -Lp <pid> to see
CPU utilization by different threads within the single process. Unlike most of the OS
1
The physical structures of OC4J is the mostly the same despite OC4J version. Author recommends to
read “Oracle® Application Server Containers for J2EE Standalone User's Guide, 10g Release 2
(10.1.2), Part No. B14361-01” chapter “Advanced Configuration and Development”
2
There is JDK used actually, OC4J instance for JSP executing compile any JSP to an *.java file and
*.calss file afterwards.
Linux realizes multithreading by running each process thread as dedicated process.
You can monitor each thread using regular OS utilities such as ps, strace, pstree etc.
There is an OC4J single process on the Figure 1 “OC4J instance” below. The OC4J
instance consists of the processes or threads (quadrangles) and memory structures
(ovals).
ONS
Thread
OC4JMonitor
new req Thread
PORT: 3001
AJP
Connection ApplicationServerThread
Listener Thread
ApplicationServerThread
ApplicationServerThread
ApplicationServerThread
Java native
AJP 1.3 ApplicationServerThread
... ONet
ApplicationServerThread
ApplicationServerThread
Memory Structures
Application objects
Java native
RMI
RMIServer
Server TaskManager VM Periodic Task
3101
PORT: 3101
TaskManager VM Thread VMThread
Periodic Task
VM Thread
Thread
JMSServer
3201
Each group of processes is responsible for different tasks executing within OC4J
instance. Let’s go through OC4J process groups and describe their responsibilities:
PM requests processed: 7
PROCESS TABLE
UID PID FLAGS TYPE STATUS REF HTTP AJP RMI JMS
---------- ----- -------- ------- ------- ------ ----- ----- ----- -----
750407 20502 00000040 OC4J Alive 1 0 3002 3103 3203
619335 4535 00000040 OC4J Alive 1 0 3003 3104 3204
488263 4533 00000040 OC4J Alive 1 0 3001 3102 3202
226119 15875 00000040 OC4J Alive 1 0 3000 3101 3201
160583 15513 00000040 Apache Alive 1 7780 0 0 0
95047 15456 00000040 Generic Alive 1 0 0 0 0
$
3
Unless you run OC4J for EJB calls processing
4
See http://jakarta.apache.org/tomcat/connectors-doc-archive/jk2/common/AJPv13.html#Intro for
more information about AJP.
RMIServer (2) – There are two processes. First one accepts clients’ RMI requests
through public network interfaces (external IP addresses). Second listener process is
mainly used for administrating requests processing. Independently of configuration
you are using (standalone or Oracle AS) the OC4J process is listening on
127.0.0.1:RMIport for shutdown/reload and other administration commands. Even if
you are using Oracle AS configuration, a OMPN calls admin.jar to contact OC4J
process through RMI port. For example the command issued by the OPMN to
shutdown OC4J process listed below6:
/u01/app/oracle/product/J2EE01/jdk/bin/java
-DOPMN=true
-Doracle.home=/u01/app/oracle/product/J2EE01
-jar
admin.jar
ormi://localhost:3104
admin
null
-shutdown
Public RMI listener accepts RMI requests to call methods of the objects located inside
OC4J. In general terms this protocol enables to make distributed java calls to objects
located inside OC4J. Description of RMI protocols is out of the scope of this paper.
As AJP listener thread RMIServer thread doesn’t process calls. It just assigns request
to available ApplicationServerThread threads. Used RMI ports are negotiated7 during
the start-up of the OC4J process. You can set a range of ports in the opmn.xml file:
OH/opmn/conf/opmn.xml
<oc4j ... >
...
<port ... rmi="3101-3200".../>
...
</oc4j>
JMSServer – It is like RMI listeners from technical point of view. Java Message
Service (JMS) is one of the required protocols by J2EE specification. JMS enables to
send and receive data and events between containers asynchronously.
ApplicationServerThread
ApplicationServerThread – is a group of threads which processes users’ requests.
Basically these treads execute application code, compile and execute JSP,
communicate with database and send a response to the Apache child processes. They
are workers who are doing most the job within OC4J container.
The worker threads are linked up to the thread pool managed by TaskManager thread.
Listeners assign users’ requests to the available ApplicationServerThread threads.
You can define minimal and maximal number of the worker threads in server.xml file.
OH/j2ee/<comp_name>/config/server.xml
<application-server ...>
...
6
You can see it in the Process Manager log file OH/opmn/logs/ipm.log with 6 logging level switched
on. You can change the PM logging level in the OH/opmn/conf/opmn.xml file:
<log-file path="OH/opmn/logs/ipm.log" level="6"/>
PS Logging overhead is minimal.
7
Unless you use stand alone configuration then port is configured in rmi.xml
<global-thread-pool min="100" max="300" queue="30" keepAlive="-1" cx-
min="100" cx-max="300" cx-queue="30" cx-keepAlive="-1" debug="true"/>
...
</application-server>
min, max, queue, keepAlive parameters are not mandatory. If you do not set them
OC4J creates a single ApplicationServerThread in the thread pool during the start-up
and adds additional threads on an as-needed basis. Threads allocation is an expensive
task. OC4J spends CPU resource to allocate/deallocate worker threads. You can pre-
allocate threads in the thread pool by setting min parameter for better performance.
OC4J allocates min threads during the start-up. TaskManager never reduces the
number of threads in the pool less then min value during OC4J life time. If there is no
available thread in the pool OC4J puts new request in the queue. In case the queue
length reaches a limit set by queue parameter OC4J spawns a new
ApplicationServerThread thread. OC4J newer creates more threads then set by max
parameter. If thread is in an idle state longer then keepAlive parameter value (in milli-
seconds) TaskManager removes the thread from the pool.
If you set debug parameter to true TaskManager process prints the thread pools
information every time of the execution.
1108429632732:ThrP[100] ThrQ[0]
1108429632732:CxP[100] CxQ[0] - WebC[0] RmiC[0]
1108429633742:ThrP[100] ThrQ[0]
1108429633742:CxP[100] CxQ[0] - WebC[0] RmiC[0]
1108429634752:ThrP[100] ThrQ[0]
1108429634752:CxP[100] CxQ[0] - WebC[0] RmiC[0]
TaskManager
TaskManager process is responsible for regular tasks execution. You can change the
default execution frequency (1000 = 1 sec) by setting parameter in the server.xml file:
<application-server ... taskmanager-granularity="10000" ...>
In addition to thread pool housekeeping task TaskManager has other tasks.
TaskManager for example is responsible for removing expired http and JDBC
sessions. You can monitor different tasks statistics using dmstool utility:
$ dmstool -table oc4j_task | grep 3001 | grep completed
.../TASK[1]_com.evermind.server.ApplicationServerTask/run().completed 3811 ops
.../TASK[2]_com.evermind.server.jms.JMSTask/run().completed 3811 ops
.../TASK[3]_com.evermind.server.rmi.RMITask/run().completed 3811 ops
.../TASK[4]_com.evermind.server.rmi.RMITask/run().completed 3811 ops
.../TASK[5]_default/run().completed 3811 ops
.../TASK[6]_com.evermind.sql.DriverManagerXADataSource/run().completed 3811 ops
.../TASK[7]_com.evermind.sql.OrionPooledDataSource/run().completed 3811 ops
.../TASK[8]_com.evermind.sql.OrionCMTDataSource/run().completed 3811 ops
.../TASK[9]_BC4J/run().completed 3811 ops
.../TASK[10]_PANDA/run().completed 3811 ops
.../TASK[11]_com.evermind.server.http.SessionTimeoutTask/run().completed 3811 ops
$
Note: There is another set of parameters with prefix cx-. Documentation refers to this
set of parameters as parameters for second type of pool – conection pool. Author was
unable to find practical differences between these two types of pools.
OC4JMonitorThread+
OC4JMonitorThread and two Thread type treads are responsible for a communication
with ONS (Oracle Notification Service) process. One of Thread type treads makes
connection to the ONS process immediately after container start-up. Those threads are
responsible for sending and receiving notifications from other Oracle AS processes.
OC4J Troubleshooting
The troubleshooting in OC4J environment is connected strongly to a performance
tuning: often the root cause of the performance problems is serialization (bugs)
introduced in the container, framework or application code, often users claim that
OC4J system hangs when there are the performance problems within a database tier.
It is vital to have and to understand the available troubleshooting methods for
implementing and maintaining OC4J based systems successfully.
Using information and troubleshooting methods described in this paper you will be
able to identify the areas of the problems, to gather all the necessary information and
to provide it to the developers of the applications or to Oracle Support. In most cases
there is necessity to correct the code of the application, used framework or even
OC4J. So, it is extremely important to localize the problem area, gather all necessary
information and provide it to the appropriate organization.
OC4J
JVM(java)
4.
PORT: 3001
AJP
JMSServer
JMSServer
Connection
Listener ApplicationServerThread
New Request
ApplicationServerThread
as
Apache
sig
n ApplicationServerThread
ApplicationServerThread
ApplicationServerThread
1.
ApplicationServerThread
TaskManager
3. DB
2. ApplicationServerThread
ApplicationServerThread
ApplicationServerThread
JDK
OS
Figure 2
8
In a context of this section author uses terms “application thread” or thread to reference
ApplicationServerThreads (see. OC4J internals) which are worker threads within OC4J environment.
Apache child process send the AJP messages to each other in the infinitive
loop can happen. The threads can have problems while receiving or sending
the information to Apache processes.
3. Interface to the database. Often the database problems (such as enqueues,
long running queries or DML operations, parsing or hanging cases) are
interpreted as OC4J problems. It is important to have the possibilities to
troubleshoot and to identify problem areas.
4. 500 Internal Errors or OC4J unavailability. Unlike previous types of
problems this one can be identified easily. If there are problems with AJP
(HTTP, RMI, JMS) Listener thread the whole OC4J process is not available
for application users’ requests. Users will receive “500 Internal Error”
message on the screens of the browsers. The problems of this type as well as
described above will be useful in further investigation and troubleshooting.
======== PM ========
PM requests processed: 5
PROCESS TABLE
UID PID FLAGS TYPE STATUS REF HTTP AJP RMI JMS
---------- ----- -------- ------- ------- ------ ----- ----- ----- -----
456236 13512 00000040 OC4J Alive 1 0 3003 3104 3204
390700 13511 00000040 OC4J Alive 1 0 3002 3103 3203
521772 15874 00000040 OC4J Alive 1 0 3001 3102 3202
259628 13415 00000040 OC4J Alive 1 0 3000 3101 3201
194092 13338 00000040 Apache Alive 1 7780 0 0 0
128556 13281 00000040 Generic Alive 1 0 0 0 0
9
Java HotSpot version from Sun Microsystems used by all versions of OC4J and hashave Java “Full
thread dump” build-in functionality.
10
opmnadmin is Oracle AS utility. Starting from 9.0.4 version of Oracle AS it is renamed to opmnctl.
Output format differs in deferent Oracle AS versions.
opmnadmin command displays PID-s as well as other useful information about each
component managing bu OPMN process.
Check the log file. You will see output like provided below.
Full thread dump:
Java “Full thread dump” begins with a header Full thread dump. There is a list of all
the threads after the header.
The interpretation of Full thread dump
As mentioned above Java “Full thread dump” begins with the header, the list of all
the threads follows. Optionally there is Java-level deadlocks information if detected
by the JVM at the end of the threads dump.
This is an example of the available information about the single thread in the dump:
"OC4JMonitorThread" daemon prio=1 tid=0x67691ec8 nid=0x3ef2 waiting on monitor
[0xa53ff000..0xa53ff89c]
at Java.lang.Object.wait(Native Method)
at oracle.ons.NotificationQueue.internalDequeue(NotificationQueue.Java:251)
at oracle.ons.NotificationQueue.dequeue(NotificationQueue.Java:224)
at oracle.ons.Subscriber.receive(Subscriber.Java:136)
at com.evermind.server.OC4JMonitorThread.run(OC4JMonitorThread.Java:419)
at Java.lang.Thread.run(Thread.Java:479)
Log files
There are a dozen of log files within the standard OC4J installation. Author would
like to pay your attention to two of them.
11
TaskManager process discussed in more details in “OC4J Internals” part of this paper
regularly a timestamp to the file. You can achieve this by adding line below 12to the
UNIX user crontab:
* * * * * echo "Timestamp: `date`" >>
/u01/app/oracle/product/J2EE01/opmn/logs/PANDY.default_island.1
OC4J process output file is one of the best sources of the troubleshooting information.
OC4J default-web-access.log
This file is an analogue of Apache access_log file. The information of each request
processed by OC4J process is reflected in the file. This information can be extremely
useful to track which OC4J process have processed the request in case you are using
configuration with many OC4J processes acting as single OC4J instance13.The
example of file location is seen below:
OH/ j2ee/PANDY/log/PANDY_default_island_1/default-web-access.log
OS Monitoring commands
Please note that multithreading implementation differs and strongly
depends on used OS. Monitoring commands used in this section are
relative to Linux OS. Look for equal functionality commands for your
OS.
As it was mentioned before each source of OC4J troubleshooting information does
not provide a complete view of the problem. You have to use all the appropriate
methods to get sufficient information to troubleshoot the problem.
By author’s experience the most useful sources of OC4J process runtime information
are Java “Full thread dump” and OS commands. Use “nid=0x3ef2” – thread’s id
provided in Java dump to connect Java dump information to the information gathered
by OS commands. When the problem occurs you can go from Java dump to OS
commands and to reverse the direction.
For example if during OC4J unavailability you observe the high utilization of CPU
(using vmstat, iostat, uptime commands), you can use top or ps -eo pcpu,pid,args |
sort -k1n | tail -10 OS commands to detect top CPU consumers. Note pid-s of the
processes and convert them to the hexadecimal values. Make a couple of Java “Full
thread dump” and navigate to the threads you have identified by OS command. Note
the information about the status of the thread (in most cases the state of top CPU
consumers’ threads will be runnable), java stack (provides the information about the
executed application code) and name of the thread. Provide information you gathered
to the developers for further investigation.
In other cases the reverse investigation path is more appropriate. Making Java “Full
thread dump” you observe that many Java threads are waiting in the waiting for
monitor entry state for the object locked by the tread in the runnable state. Use OS
command to identify if thread is consuming resources or is waiting to receive the
information from TCP/IP socket. For this purpose you can use ps command as shown
below:
$ while [ 1 ] ; do ps -p 13285 -o pcpu | grep -v CPU ; sleep 1 ; done
25.3
25.3
12
Replace the name of the output file to correct one for your environment
13
OC4J Performance section for information about multiple OC4J configuration
25.3
$ while [ 1 ] ; do ps -p 19273 -o vsz,rss | grep -v VSZ ; sleep 1 ;
done
1095372 258244
1095372 258244
1095372 258244
In case the thread is using CPU resource you can use OS tracing commands
(described in the next section) to get the additional information.
Author has found the OS monitoring commands listed below as the most useful for
OC4J process troubleshooting:
• System wide:
vmstat, iostat, uptime, pstree
• Top consumers of the resources
top, ps -eo pcpu,pid,args | sort -k1n | tail -10
• Particular process information
while [ 1 ] ; do ps -p <pid> -o pcpu | grep -v CPU ; sleep 1 ; done
while [ 1 ] ; do ps -p <pid> -o vsz,rss | grep -v VSZ ; sleep 1 ; done
OS Tracing commands
Many operational systems (OS-s) have tracing utilities. They are able to attach to
running process and to reflect all calls made by process to the OS. In some situations
OS processes’ tracing capabilities add invaluable information for problems
troubleshooting. Let us illustrate OS tracing commands’ possibilities within OC4J
troubleshooting context by the example.
You have noted for example that there are two processes consuming most of CPU
resources for a longer time using top command:
$ top -d 1
3:23pm up 7 days, 5:24, 4 users, load average: 1.36, 1.27, 1.49
1080 processes: 1076 sleeping, 3 running, 0 zombie, 1 stopped
CPU2 states: 5.0% user, 21.0% system, 0.0% nice, 72.1% idle
CPU3 states: 25.0% user, 64.0% system, 0.0% nice, 9.1% idle
...
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
27021 oracle 25 0 587M 587M 25096 R 65.7 15.2 244:22 java
26018 oracle 15 0 11228 10M 8184 S 36.0 0.2 165:02 httpd
9514 oracle 15 0 2232 2228 836 R 8.1 0.0 0:00 top
16047 root 15 0 1172 1172 852 S 6.3 0.0 15:11 cmahostd
1 root 15 0 524 524 452 S 0.0 0.0 0:04 init
One of the processes is the Apache child process, the second – java thread. From Java
dump you can see that the thread with id 27021 is in the runnable state. Java stack
indicates that this thread is sending AJP packages:
"ApplicationServerThread-31" prio=1 tid=0x69cd3c50 nid=0x698d runnable [0xba7ff000..0xba7ff89c]
at java.net.SocketOutputStream.socketWrite(Native Method)
at java.net.SocketOutputStream.write(SocketOutputStream.java:60)
at com.evermind.server.http.AJPBodyInputStream.refillReadBuffer(AJPBodyInputStream.java:130)
at com.evermind.server.http.AJPBodyInputStream.read(AJPBodyInputStream.java:102)
at com.evermind.server.http.EvermindServletInputStream.read(EvermindServletInputStream.java:193)
at com.evermind.server.http.EvermindHttpServletRequest.parsePostData(EvermindHttpServletRequest.java:2103)
at com.evermind.server.http.EvermindHttpServletRequest.parseParameters(EvermindHttpServletRequest.java:2047)
at com.evermind.server.http.EvermindHttpServletRequest.getParameter(EvermindHttpServletRequest.java:2133)
at javax.servlet.ServletRequestWrapper.getParameter(ServletRequestWrapper.java:203)
at _nsk._nsk001._SP0301F__Save._jspService(_SP0301F__Save.java:68)
at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:56)
at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:317)
at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:476)
at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:390)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:65)
at oracle.security.jazn.oc4j.JAZNFilter.doFilter(JAZNFilter.java:283)
at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:566)
at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:306)
at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:767)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:166)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:72)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:804)
at java.lang.Thread.run(Thread.java:479)
14
Please note that you have to allow access to the page by editing OH/Apacha/Apache/conf/httpd.conf
file.
Java code decompilation
Unlike units written in other languages Java classes can be decompiled easily. There
are many utilities available for this purpose15. As mentioned above OC4J (oc4j.jar)
and other related components (JDBC, BC4J, etc.) are written in Java and can be
decompiled.
For troubleshooting purposes this adds information for you investigation. You can
unzip the related archive files and decompile class files. Author has found useful to
decompile java components listed below:
OC4J
OH/j2ee/home/oc4j.jar
JDBC
OH/jdbc/lib/classes12.zip
OH/jdbc/lib/classes12dms.jar
DCM
OH/dcm/lib/dcm.jar
OH/dcm/lib/oc4j_deploy_tools.jar
OH/dcm/lib/oc4j_remote_deploy.jar
DMS
OH/lib/dms.jar
BC4J
OH/BC4J/redist/bc4j.ear
OH/BC4J/jlib/*
OH/BC4J/lib/*
Debugging Switches
OC4J debugging switches documented well in the Oracle AS 10g Documentation set
Containers for J2EE User's Guide
10g Release 2 (10.1.2)
Part No. B14011-01
B Additional Information
B.8 OC4J Command-Line Options and System Properties
If your OC4J environment is on the earlier version you can still use the book
mentioned above. Author has found that this is the first release of OC4J
documentation where the debugging switches described relatively well.
You set OC4J debugging switches in the OH/opmn/conf/opmn.xml file (java-option
section). There is the example of OC4J output file with AJP debugging switch turned
on (-Dajp.io.debug=true) shown below.
15
One of these utilities is available from http://www.kpdus.com/jad.html
Output file PANDA.default_island.1
1107387907360:ThrP[50] ThrQ[0]
1107387907360:CxP[20] CxQ[0] - WebC[0] RmiC[0]
AJP: Input request length is: 1
AJP: AJPOutputStream::write() of 1
AJP: AJPOutputStream::writeChunk(): 1
1107387908370:ThrP[50] ThrQ[0]
1107387908370:CxP[20] CxQ[0] - WebC[0] RmiC[0]
Each switch is useful in different situation. Some of available useful categories are:
ajp, http, jms, rmi, jdbc.
Please do not forget to test each change of the configuration in the test environment
before to implement it in the production.
External searches
Unfortunately there is no much information about OC4J. Sometimes you can get
information you are interested in using www.google.com search capabilities.
http://otn.oracle.com has a couple of forums on OC4J related questions; unfortunately
the searching capabilities are very limited there. Try to ask question you are interested
in and probably a person from Oracle will answer you.
http://metalink.oracle.com does not have OC4J related forums and available
information is rather poor. There are no many technical notes related to OC4J
technology and OC4J troubleshooting. Author however has found very useful to
review Metalink Patches section of the Metalink to get the solution for some critical
scalability problems.
OC4J Performance
In addition to the troubleshooting methods described in “OC4J troubleshooting” part
this section describes author’s findings in the context of OC4J performance tuning
and available performance monitoring capabilities as well as other author’s
recommendations.
Our Findings
Java Workers
OC4J allows you to set a minimal number of ApplicationServerThread (Workers16),
as well as the maximum number of Workers and the size of requests’ queue. In the
ideal situation when workers do not block each other and the database responds
quickly to any request sent by Worker threads, the initial number of workers can be
set to the minimal value (20) or even to be unset17.
Author however has found that the setting of the minimal number to the highest value
improves OC4J instance availability, scalability and overall response time. If there is
a blocking situation in your environment and there is no available worker thread,
OC4J will put the requests of each user in a queue. OC4J will spawn the additional
Worker thread only when queue achieves the limit set in the configuration file. This
behavior can significantly limit OC4J process availability and performance.
Set the Worker thread’s parameters in the server.xml file (see. example below).
OH/j2ee/<comp_name>/config/server.xml
<application-server ...>
...
<global-thread-pool
min="100" max="300" queue="30" keepAlive="-1"
cx-min="100" cx-max="300" cx-queue="30" cx-keepAlive="-1"
debug="true"/>
...
</application-server>
Setting the min parameter to the high value you make OC4J free from resources’
intensive procedure of creating the new Workers, as well as from queue managing
routine. Reader can argue that the high number of unused threads adds the additional
overhead and utilization of CPU resources. Author however did not note the
significant overhead while setting the parameter in the production environments.
OC4J Processes
It is possible to startup the multiple OC4J processes, which act as one OC4J instance.
mod_oc4j (Apache module) will spread users’ requests across all the available OC4J
processes. Even if OC4J instance is located on a single host you can earn a significant
improvement in terms of system availability and performance by distributing the load
through multiple OC4J instances.
Set numProcs parameter in opmn.xml file for particular instance. There are three
processes for instance PANDY in the following example.
OH/opmn/conf/opmn.xml
16
Workers – are the threads which process users’ requests and execute the application code. For the
additional information about this type of threads see “OC4J Internals”
17
If you do not set the initial number of Workers OC4J initializes 1 Worker thread at the startup
<oc4j maxRetry="3" instanceName="PANDY" gid="PANDY" numProcs="1">
<config-file
path="/u01/app/oracle/product/J2EE01/j2ee/PANDY/config/server.xml"/>
<java-option value="-server -Xms512M -Xmx512M"/>
<oc4j-option value="-properties "/>
<port ajp="3001-3100" jms="3201-3300" rmi="3101-3200"/>
<island id="default_island" numProcs="3"/>
</oc4j>
Please note that OPMN will start three dedicated OC4J processes. Each of them will
utilize its own memory areas. In the example above all the PANDY instance OC4J
processes will utilize 1.5 GB of RAM. Ensure that you have the sufficient memory
resources for all instances. Please note that each process acts independently, each
process will have separate DMS and BC4J statistics, as well as own AJP/RMI/JMS
ports.
The mod_oc4j however will send users’ requests to one particular OC4J process.
Routing information for each user session is stored in the JSESSIONID coke18.
Memory
Author has found reasonable to set the minimal size of memory available to the single
OC4J process equal to the maximum size. However this is not a strict
recommendation. You set memory usage parameters in opmn.xml file:
OH/opmn/conf/opmn.xml
<oc4j …
<java-option value="-server -Xms512M -Xmx512M"/>
</oc4j>
Setting min=max JVM does not need memory to be allocated dynamically. Each
OC4J process reserves memory during startup.
You have to monitor regularly the usage of memory within OC4J process. For this
purpose you can set -verbose:gc java parameter, as well as dmstool functionality.
DMS utility allows you to monitor memory usage any time:
$ dmstool -table JVM | grep 3001 | grep Memory
/ap1.voava.lv/OC4J:3001:7003/JVM/freeMemory.value 495796 kbytes
/ap1.voava.lv/OC4J:3001:7003/JVM/freeMemory.minValue 495796.0 kbytes
/ap1.voava.lv/OC4J:3001:7003/JVM/freeMemory.maxValue 522468.0 kbytes
/ap1.voava.lv/OC4J:3001:7003/JVM/totalMemory.value 524032 kbytes
/ap1.voava.lv/OC4J:3001:7003/JVM/totalMemory.minValue 524032.0 kbytes
/ap1.voava.lv/OC4J:3001:7003/JVM/totalMemory.maxValue 524032.0 kbytes
ap1.voava.lv*J2EE01-/u01/app/oracle/product
$
The single snapshot of memory metrics does not add any significant value. It might be
more effective to collect metrics regularly.
18
You can observe cookie by entering javascript:alert(document.cookie); string in the Internet
Browser address bar after you logged to OC4J application.
The above example reflects the typical behavior of OC4J JVM. Garbage Collector
(GC) frees the significant memory areas periodically. There is a lot of information
regarding the tuning of GC. For example you may find the following links useful:
http://java.sun.com/docs/hotspot/gc/
http://developers.sun.com/techtopics/mobility/midp/articles/garbagecollection2/
http://www.cons.org/cmucl/doc/gc-tuning.html
Grep OC4J log files for out of memory errors. If you allocated a significant amount of
memory it is possible that there are the memory leaks in the application code. Ask the
application support or the development organizations for memory allocation
recommendations.
In case of memory leaks Java Profilers19 such as JProfiler from Quest Software or
others can be used to locate and to find root of the problem. Be aware that Java
Profilers add significant overhead and are not usable normally in the production
environments.
Stresstesting
Often a need for the stresstesting is underestimated. However, according to author’s
experience the stresstesting is one of the best methods to tune OC4J application
performance. Weak areas can be identified easily. Author considers using the
automatic stresstesting tools. This will allow you to repeat the testing for several times
and to estimate the code correction results. It is almost impossible to test all the
functionality and dynamic behavior of system users. Include the most used forms and
functionality in the stresstesting scenarios. Repeat testing – performance the problem
correction cycles for several times until you get the appropriate results.
By implementing the stresstesting process you can avoid a lot of problems in future.
19
The profiling with Oracle JDeveloper is possible on Win32 platform only.
Response Time Monitoring by DMS
Apache OC4J
OHS [c] mod_plsql JVM [java]
handle
Send
Request B
Handle B ApplicationServerThread
Read
Resolve
Process B
JSP Processing
mod_oc4j Servlet Processing
DB
handle
Process E
Handle E
Handle B
Handle E
Request E
mod_perl
Recive mod_php
...
DMS-client
DMS.JAR (java)
statistics reflecting
The figure above reflects the example of single user’s request processing. Apache as
well as OC4J and other Oracle AS components have the build-in DMS (Dynamic
Monitoring Services) sensors, which reflect different performance metrics. You can
monitor this statistics using dmstool.
Snap
Id Snap Started
------- -------------------
530 2005-02-03 12:33:26
531 2005-02-03 13:03:24
532 2005-02-03 13:33:27
549 2005-02-03 14:03:32
SYS:PROD>
Appendix A. How to diagnose opened files, sockets,
pipes by OC4J process?
It can be useful for diagnostic and troubleshooting purposes to see all files opened by
the process. For this purposes you can use lsof like utility20. The lsof utility lists all
files opened by the process. There is an example of lsof output for OC4J process
below (see Figure 3 lsof utility output example). Using mentioned utility you can see
easily all the log files, sockets, *.JAR files opened by OC4J process. It’s interesting to
see for example that OC4J process uses different *.JAR files from directories listed
below:
/u01/app/oracle/product/J2EE01/lib/dms.jar
/u01/app/oracle/product/J2EE01/opmn/lib/ons.jar
/u01/app/oracle/product/J2EE01/j2ee/home/oc4j.jar
/u01/app/oracle/product/J2EE01/j2ee/home/lib/http_client.jar
/u01/app/oracle/product/J2EE01/config/Queries.jar
...
20
lsof utility is available on Linux and Sun Solaris platforms. Look for similar utility on your platform.
Figure 3 lsof utility output example
COMM PI USER FD TYPE DEVIC SIZE NODE NAME
AND D E
java 8010 oracle 18u Ipv4 -1 79444636 TCP *:3001 (LISTEN)
java 8010 oracle 13u Ipv4 -1 79444643 TCP *:3201 (LISTEN)
java 8010 oracle rtd DIR 104,7 1024 2 /
java 8010 oracle 0r CHR 1,3 9511 /dev/null
java 8010 oracle mem REG 104,7 466557 60577 /lib/ld-2.2.4.so
java 8010 oracle mem REG 104,7 5774329 60490 /lib/libc-2.2.4.so
java 8010 oracle mem REG 104,7 65509 60494 /lib/libdl-2.2.4.so
java 8010 oracle mem REG 104,7 616450 60496 /lib/libm-2.2.4.so
java 8010 oracle mem REG 104,7 451085 60499 /lib/libnsl-2.2.4.so
java 8010 oracle mem REG 104,7 73929 60512 /lib/libnss_dns-2.2.4.so
java 8010 oracle mem REG 104,7 257535 60515 /lib/libnss_files-2.2.4.so
java 8010 oracle mem REG 104,7 351452 60523 /lib/libnss_nisplus-2.2.4.so
java 8010 oracle mem REG 104,7 561991 60526 /lib/libpthread-0.9.so
java 8010 oracle mem REG 104,7 263234 60528 /lib/libresolv-2.2.4.so
java 8010 oracle mem REG 104,8 120257 2998503 /u01/app/oracle/product/J2EE01/config/Queries.jar
java 8010 oracle cwd DIR 104,8 4096 3571783 /u01/app/oracle/product/J2EE01/j2ee/home
java 8010 oracle mem REG 104,8 1431995 3571807 /u01/app/oracle/product/J2EE01/j2ee/home/iiop.jar
java 8010 oracle mem REG 104,8 37265 3571808 /u01/app/oracle/product/J2EE01/j2ee/home/iiop_gen_bin.jar
java 8010 oracle mem REG 104,8 126451 3571810 /u01/app/oracle/product/J2EE01/j2ee/home/jaxb-rt-1.0-ea.jar
java 8010 oracle mem REG 104,8 938119 3571789 /u01/app/oracle/product/J2EE01/j2ee/home/jazn.jar
java 8010 oracle mem REG 104,8 45386 2769040 /u01/app/oracle/product/J2EE01/j2ee/home/lib/activation.jar
java 8010 oracle mem REG 104,8 726013 2769036 /u01/app/oracle/product/J2EE01/j2ee/home/lib/aqapi.jar
java 8010 oracle mem REG 104,8 515920 2769054 /u01/app/oracle/product/J2EE01/j2ee/home/lib/bcel.jar
java 8010 oracle mem REG 104,8 17978 2769032 /u01/app/oracle/product/J2EE01/j2ee/home/lib/connector.jar
java 8010 oracle mem REG 104,8 58142 2769033 /u01/app/oracle/product/J2EE01/j2ee/home/lib/cos.jar
java 8010 oracle mem REG 104,8 206289 2769041 /u01/app/oracle/product/J2EE01/j2ee/home/lib/crimson.jar
java 8010 oracle mem REG 104,8 12822 2769029 /u01/app/oracle/product/J2EE01/j2ee/home/lib/ejb.jar
java 8010 oracle mem REG 104,8 228847 2769037 /u01/app/oracle/product/J2EE01/j2ee/home/lib/http_client.jar
java 8010 oracle mem REG 104,8 104195 2769030 /u01/app/oracle/product/J2EE01/j2ee/home/lib/jaas.jar
java 8010 oracle mem REG 104,8 1070096 2769038 /u01/app/oracle/product/J2EE01/j2ee/home/lib/jasper.zip
java 8010 oracle mem REG 104,8 28404 2769031 /u01/app/oracle/product/J2EE01/j2ee/home/lib/jaxp.jar
java 8010 oracle mem REG 104,8 6727 2769034 /u01/app/oracle/product/J2EE01/j2ee/home/lib/jdbc.jar
java 8010 oracle mem REG 104,8 424080 2769039 /u01/app/oracle/product/J2EE01/j2ee/home/lib/jem.jar
java 8010 oracle mem REG 104,8 27724 2769044 /u01/app/oracle/product/J2EE01/j2ee/home/lib/jms.jar
java 8010 oracle mem REG 104,8 98496 2769047 /u01/app/oracle/product/J2EE01/j2ee/home/lib/jndi.jar
java 8010 oracle mem REG 104,8 8674 2769048 /u01/app/oracle/product/J2EE01/j2ee/home/lib/jta.jar
java 8010 oracle mem REG 104,8 280984 2769049 /u01/app/oracle/product/J2EE01/j2ee/home/lib/mail.jar
java 8010 oracle mem REG 104,8 605174 2769001 /u01/app/oracle/product/J2EE01/j2ee/home/lib/ojsp.jar
java 8010 oracle mem REG 104,8 53389 2769051 /u01/app/oracle/product/J2EE01/j2ee/home/lib/servlet.jar
java 8010 oracle mem REG 104,8 4783072 3571813 /u01/app/oracle/product/J2EE01/j2ee/home/oc4j.jar
java 8010 oracle 7w REG 104,8 1114323 5030160 /u01/app/oracle/product/J2EE01/j2ee/PANDA/application-deployments/BC4J/PANDA_default_island_1/application.log
java 8010 oracle 9w REG 104,8 6690838 6340902 /u01/app/oracle/product/J2EE01/j2ee/PANDA/application-deployments/PANDA/PANDA_default_island_1/application.log
java 8010 oracle 35r REG 104,8 1438 2184291 /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/entity/ApnNskAdDiagnosis_AS.xml
java 8010 oracle 49r REG 104,8 1422 2184306 /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/entity/ApnNskForward_AS.xml
java 8010 oracle 32r REG 104,8 1433 2184453 /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/entity/NmpNskAdClassify_AS.xml
java 8010 oracle 54r REG 104,8 1339 2184505 /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/entity/NskAdDiagnosis_AS.xml
java 8010 oracle 28r REG 104,8 1359 2184528 /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/entity/NskAdPatientGroups_AS.xm
java 8010 oracle 29r REG 104,8 1386 2184557 /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/entity/NskCadAdsFkAssoc.xml
java 8010 oracle 38r REG 104,8 1418 2184596 /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/entity/NskHpsAdsFkAssoc.xml
java 8010 oracle 37r REG 104,8 3761 2184998 /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/nsk001/P2601F_NmpAdAttributes_V
java 8010 oracle 43r REG 104,8 1496 2185078 /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/nsk001/SP0301F_NskAccountingDoc
java 8010 oracle 30r REG 104,8 1464 2185147 /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/nsk001/SP0301F_NskXmlDocErrors_
java 8010 oracle 36r REG 104,8 6031 2185426 /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/nsk002/P2631F_NskLcSubcriteria_
java 8010 oracle 34r REG 104,8 1387 2186115 /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/skd/d001/D2262F_Persons_Det5_VL.xml
java 8010 oracle 52r REG 104,8 7517 2186174 /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/skd/entity/SkdPersonDiagnoses_EO.xm
java 8010 oracle 45r REG 104,8 6657 2186205 /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/skd/entity/SkdVcurrentPvaDoctors.xm
java 8010 oracle 10w REG 104,8 55981920 5095446 /u01/app/oracle/product/J2EE01/j2ee/PANDA/log/PANDA_default_island_1/default-web-access.log
java 8010 oracle 6w REG 104,8 133330 5095445 /u01/app/oracle/product/J2EE01/j2ee/PANDA/log/PANDA_default_island_1/global-application.log
java 8010 oracle 12w REG 104,8 42600 5095448 /u01/app/oracle/product/J2EE01/j2ee/PANDA/log/PANDA_default_island_1/jms.log
java 8010 oracle 15w REG 104,8 85101 5095449 /u01/app/oracle/product/J2EE01/j2ee/PANDA/log/PANDA_default_island_1/rmi.log
java 8010 oracle 5w REG 104,8 148705 5095444 /u01/app/oracle/product/J2EE01/j2ee/PANDA/log/PANDA_default_island_1/server.log
java 8010 oracle 11w REG 104,8 10 6865677 /u01/app/oracle/product/J2EE01/j2ee/PANDA/persistence/PANDA_default_island_1/transaction.state
java 8010 oracle mem REG 104,8 400813 2293843 /u01/app/oracle/product/J2EE01/javacache/lib/cache.jar
java 8010 oracle mem REG 104,8 1227663 5783844 /u01/app/oracle/product/J2EE01/jdbc/lib/classes12dms.jar
java 8010 oracle mem REG 104,8 1846926 5783845 /u01/app/oracle/product/J2EE01/jdbc/lib/nls_charset12.jar
java 8010 oracle txt REG 104,8 27090 3998180 /u01/app/oracle/product/J2EE01/jdk/bin/i386/native_threads/java
java 8010 oracle mem REG 104,8 3228357 6357412 /u01/app/oracle/product/J2EE01/jdk/jre/lib/i18n.jar
java 8010 oracle mem REG 104,8 3968300 2441654 /u01/app/oracle/product/J2EE01/jdk/jre/lib/i386/client/libjvm.so
java 8010 oracle mem REG 104,8 4323150 6357389 /u01/app/oracle/product/J2EE01/jdk/jre/lib/i386/libawt.so
java 8010 oracle mem REG 104,8 221718 6357381 /u01/app/oracle/product/J2EE01/jdk/jre/lib/i386/libjava.so
java 8010 oracle mem REG 104,8 214414 6357388 /u01/app/oracle/product/J2EE01/jdk/jre/lib/i386/libmlib_image.so
java 8010 oracle mem REG 104,8 45805 6357385 /u01/app/oracle/product/J2EE01/jdk/jre/lib/i386/libnet.so
java 8010 oracle mem REG 104,8 81155 6357380 /u01/app/oracle/product/J2EE01/jdk/jre/lib/i386/libverify.so
java 8010 oracle mem REG 104,8 100867 6357382 /u01/app/oracle/product/J2EE01/jdk/jre/lib/i386/libzip.so
java 8010 oracle mem REG 104,8 60924 2441652 /u01/app/oracle/product/J2EE01/jdk/jre/lib/i386/native_threads/libhpi.so
java 8010 oracle mem REG 104,8 13887946 6357411 /u01/app/oracle/product/J2EE01/jdk/jre/lib/rt.jar
java 8010 oracle mem REG 104,8 86267 6357399 /u01/app/oracle/product/J2EE01/jdk/jre/lib/sunrsasign.jar
java 8010 oracle mem REG 104,8 4723739 3522887 /u01/app/oracle/product/J2EE01/jdk/lib/tools.jar
java 8010 oracle mem REG 104,8 47977 3719344 /u01/app/oracle/product/J2EE01/jlib/infratool.jar
java 8010 oracle mem REG 104,8 19055 3719459 /u01/app/oracle/product/J2EE01/jlib/javax-ssl-1_1.jar
java 8010 oracle mem REG 104,8 97158 3719348 /u01/app/oracle/product/J2EE01/jlib/jndi.jar
java 8010 oracle mem REG 104,8 50921 3719462 /u01/app/oracle/product/J2EE01/jlib/jssl-1_1.jar
java 8010 oracle mem REG 104,8 162059 3719351 /u01/app/oracle/product/J2EE01/jlib/ldapjclnt9.jar
java 8010 oracle mem REG 104,8 246743 3719457 /u01/app/oracle/product/J2EE01/jlib/netcfg.jar
java 8010 oracle mem REG 104,8 23443 3719364 /u01/app/oracle/product/J2EE01/jlib/ojmisc.jar
java 8010 oracle mem REG 104,8 50413 3719352 /u01/app/oracle/product/J2EE01/jlib/repository.jar
java 8010 oracle mem REG 104,8 260225 5783839 /u01/app/oracle/product/J2EE01/lib/dms.jar
java 8010 oracle mem REG 104,8 463863 5783772 /u01/app/oracle/product/J2EE01/lib/jsse.jar
java 8010 oracle mem REG 104,8 658404 5783805 /u01/app/oracle/product/J2EE01/lib/xmlparserv2.jar
java 8010 oracle mem REG 104,8 53464 6832352 /u01/app/oracle/product/J2EE01/opmn/lib/ons.jar
java 8010 oracle 1w REG 104,8 23185778 6144409 /u01/app/oracle/product/J2EE01/opmn/logs/PANDA.default_island.1
java 8010 oracle 2w REG 104,8 23185778 6144409 /u01/app/oracle/product/J2EE01/opmn/logs/PANDA.default_island.1
java 8010 oracle 8w REG 104,8 23185778 6144409 /u01/app/oracle/product/J2EE01/opmn/logs/PANDA.default_island.1
java 8010 oracle mem REG 104,8 297060 819453 /u01/app/oracle/product/J2EE01/soap/lib/soap.jar
java 8010 oracle mem REG 104,8 419100 6471908 /u01/app/oracle/product/J2EE01/sqlj/lib/runtime12ee.jar
java 8010 oracle mem REG 104,8 3717072 6471909 /u01/app/oracle/product/J2EE01/sqlj/lib/translator.jar
java 8010 oracle mem REG 104,8 142704 3276954 /u01/app/oracle/product/J2EE01/webservices/lib/wsdl.jar
java 8010 oracle mem REG 104,8 123656 3276958 /u01/app/oracle/product/J2EE01/webservices/lib/wsserver.jar
java 8010 oracle 22w REG 104,8 7154 2999584 /u01/app/PANDA/logs/panda-2005-01-30-08-28-14-958.log
java 8010 oracle mem REG 104,5 1144432 98829 /usr/lib/libstdc++-2-libc6.1-1-2.9.0.so
java 8010 oracle mem REG 104,5 160 427556 /usr/lib/locale/en_US/LC_ADDRESS
java 8010 oracle mem REG 104,5 21499 213158 /usr/lib/locale/en_US/LC_COLLATE
java 8010 oracle mem REG 104,5 173408 98491 /usr/lib/locale/en_US/LC_CTYPE
java 8010 oracle mem REG 104,5 370 131090 /usr/lib/locale/en_US/LC_IDENTIFICATION
java 8010 oracle mem REG 104,5 28 263082 /usr/lib/locale/en_US/LC_MEASUREMENT
java 8010 oracle mem REG 104,5 57 196715 /usr/lib/locale/en_US/LC_MESSAGES/SYS_LC_MESSAGES
java 8010 oracle mem REG 104,5 291 262181 /usr/lib/locale/en_US/LC_MONETARY
java 8010 oracle mem REG 104,5 82 427555 /usr/lib/locale/en_US/LC_NAME
java 8010 oracle mem REG 104,5 59 98484 /usr/lib/locale/en_US/LC_NUMERIC
java 8010 oracle mem REG 104,5 39 393342 /usr/lib/locale/en_US/LC_PAPER
java 8010 oracle mem REG 104,5 64 426055 /usr/lib/locale/en_US/LC_TELEPHONE
java 8010 oracle mem REG 104,5 2456 132545 /usr/lib/locale/en_US/LC_TIME
java 8010 oracle mem REG 104,5 82080 361727 /usr/X11R6/lib/libICE.so.6.3
java 8010 oracle mem REG 104,5 31544 361731 /usr/X11R6/lib/libSM.so.6.0
java 8010 oracle mem REG 104,5 1009336 361733 /usr/X11R6/lib/libX11.so.6.2
java 8010 oracle mem REG 104,5 51632 361741 /usr/X11R6/lib/libXext.so.6.4
java 8010 oracle mem REG 104,5 26788 361753 /usr/X11R6/lib/libXp.so.6.2
java 8010 oracle mem REG 104,5 306428 361759 /usr/X11R6/lib/libXt.so.6.0
java 8010 oracle mem REG 104,5 17712 361761 /usr/X11R6/lib/libXtst.so.6.1
java 8010 oracle 53u IPv4 - 988260755 TCP ap1.voava.lv:28475->prod2.vovaa.lv:1526 (ESTABLISHED)
java 8010 oracle 50u IPv4 -1 9403290 TCP ap1.voava.lv:30001->prod2.vovaa.lv:1526 (ESTABLISHED)
java 8010 oracle 16u IPv4 -1 79444641 TCP ap1.voava.lv:3101 (LISTEN)
java 8010 oracle 24u IPv4 -1 33559729 TCP ap1.voava.lv:31435->prod2.vovaa.lv:1526 (ESTABLISHED)
java 8010 oracle 26u IPv4 - 982099578 TCP ap1.voava.lv:38392->prod2.vovaa.lv:1526 (ESTABLISHED)
java 8010 oracle 41u IPv4 - 981641728 TCP ap1.voava.lv:39100->prod2.vovaa.lv:1526 (ESTABLISHED)
java 8010 oracle 33u IPv4 -1 30681729 TCP ap1.voava.lv:42991->prod2.vovaa.lv:1526 (ESTABLISHED)
java 8010 oracle 39u IPv4 -1 20319458 TCP ap1.voava.lv:45951->prod2.vovaa.lv:1526 (ESTABLISHED)
java 8010 oracle 42u IPv4 -1 20230199 TCP ap1.voava.lv:46351->prod2.vovaa.lv:1526 (ESTABLISHED)
java 8010 oracle 23u IPv4 -1 46160142 TCP ap1.voava.lv:46866->prod2.vovaa.lv:1526 (ESTABLISHED)
java 8010 oracle 44u IPv4 -1 19680489 TCP ap1.voava.lv:48228->prod2.vovaa.lv:1526 (ESTABLISHED)
java 8010 oracle 27u IPv4 - 975439727 TCP ap1.voava.lv:48625->prod2.vovaa.lv:1526 (ESTABLISHED)
java 8010 oracle 40u IPv4 -1 329717 TCP ap1.voava.lv:49006->prod2.vovaa.lv:1526 (ESTABLISHED)
java 8010 oracle 31u IPv4 -1 19232844 TCP ap1.voava.lv:50077->prod2.vovaa.lv:1526 (ESTABLISHED)
java 8010 oracle 46u IPv4 - 999783949 TCP ap1.voava.lv:50286->prod2.vovaa.lv:1526 (ESTABLISHED)
java 8010 oracle 25u IPv4 -1 42714920 TCP ap1.voava.lv:52557->prod2.vovaa.lv:1526 (ESTABLISHED)
java 8010 oracle 21u IPv4 -1 39574270 TCP ap1.voava.lv:59195->prod2.vovaa.lv:1526 (ESTABLISHED)
java 8010 oracle 48u IPv4 -1 16842834 TCP ap1.voava.lv:59222->prod2.vovaa.lv:1526 (ESTABLISHED)
java 8010 oracle 14u sock 0,0 3215522651 can't identify protocol
java 8010 oracle 17u IPv4 -1 79444637 TCP localhost:3101 (LISTEN)
java 8010 oracle 19u IPv4 -1 79444635 TCP localhost:64214->localhost:7100 (ESTABLISHED)
java 8010 oracle 3r FIFO 0,0 3215522636 pipe
java 8010 oracle 4w FIFO 0,0 3215522636 pipe