PPT

1
Copyright 2012, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 8
Triage and Diagnostics for E-Business Suite Systems

2 Copyright 2012, Oracle and/or its affiliates. All rights reserved.
Dave Suri, Project Lead, Applications Performance Gustavo Jimenez, Senior Development Manager, ATG
Agenda
Architecture Patches/Certified Browsers/JDK versions JVM Tuning and Tools Forms Tools (strace/FRD) Java Concurrent Program Performance Case studies
Architecture Overview
The applications tier connects to the database via JDBC connections The application connections are pooled, so a database session is used to service requests from multiple users throughout the lifetime of the connection Java concurrent programs run in their own JVM - each has a special configuration. Forms runs in Socket mode (Forms Listener) or Servlet mode (HTTP/s to the JVM) Apache HTTP forwards requests to OC4J through mod_oc4j. OC4J routes them to the right oacore JVM while OPMN helps keep the uptime high with death detection.
Architecture Overview - R12 OA-Framework Flow
Architecture Overview Forms Servlet Mode

The Forms Listener Servlet is a Java servlet runs inside the OC4J container The listener servlet delivers the ability to run Forms applications over HTTP The listener servlet manages the creation of Forms Runtime process (frmweb) for each client and manages any further communications between the client and its designated runtime process. The main difference is the traffic goes through HTTP/HTTPS and leverages load balancing/security capabilities of the MT. Does not require a special port to be open
Architecture Overview Forms Socket Mode

Forms Direct Socket Connection
Client requests forms application OHS receives requests & sends the request to the forms listener Listener contacts a forms runtime engine (frmsrv/f60srvm) frmsrv/f60srvm spawns a runtime process, creates connection to the database & passes the forms params (menu, sid, form name) forms runtime communicates with the database & gets the forms logic (pl/sql,etc). forms service sends applet & the response back to client forms is running, page is rendered and direct connection to runtime process
Agenda
10
Patches, Browsers & Versions

Document ID 244040.1 Recommended Performance Patches for the Oracle E-Business Suite Recommended performance patches for all the modules and technology stack components are consolidated in this note Apply latest Release Update Packs Release 12.1.3 - Patch 9239090 Document ID 1080973.1 Oracle E-Business Suite Release 12.1.3 Readme Release 12.1 - Patch 7303030 Document ID 752619.1 Oracle E-Business Suite Release 12.1.1 Maintenance Pack Installation Instructions Release 12.0 - RUP6/7 Patch 6728000 / 7577660 Document ID 743368.1 Oracle E-Business Suite Release 12.1.1 Maintenance Pack Installation Instructions
11

Upgrade to the latest certified technology stack EBS 12.1 is certified with 11gR2, Oracle Apps Server 10gR3, JRE 6.0. Document ID 561580.1 Document ID 380482.1 Oracle E-Business Suite Release 12 Technology Stack Documentation Roadmap Central document for installation, apps, database, security and other resources. OA Framework Applications Recommended Patches for Applications: note 275880.1 (Framework Roadmap) Oracle E-Business Suite Recommended Performance Patches Document id: 244040.1 Oracle E-Business Suite Recommended Performance Patches
For a list of certifications, use Document ID 561580.1 Oracle E-Business Suite Release 12.1 Release Content Documents
12

Browsers, OAF Pages in R12.x
Firefox 3.6 is supported in 11i, R12 Windows 7 / Internet Explorer 8 / Firefox 3.6 supported in 11i/R12 Oracle E-Business Suite 12.1.1 or higher R12.0 => use R12.ATG_PF.A.DELTA.6 & Forms 10.1.2.3 or higher
11.5.10 => CU2, 11i.ATG_PF.H Rollup 7 with Document ID 783600.1 Document ID 1103276.1, 389422.1 (R12), 285218.1 (11i)
R12.1.2 have ajax customizable homepage for increased performance R12.1.3 available as patch #9239090
13
Agenda
14
JVM Tuning and Tools

Response Time/JVM Usage OutOfMemoryErrors Deadlocks Some great tools Jstat : monitor JVM Eclipse MAT : analyze heap dumps IBM Thread Dump Analyzer : analyze thread dumps Visual VM : profile, monitor, dump the JVM Jconsole : similar to Visual VM except for profiling Jrockit Mission Control : flight recorder, profiler, dump Thread Dump Analyzer

Response Time/JVM Usage
First, rule out any SQL issues. Document ID 357597.1 How To Generate A SQL Trace In OA Framework For Oracle Applications
Monitor Request Response Time
Modify $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/httpd.conf
LogFormat "%h %T Logs: $LOG_HOME/ora/10.1.3/Apache/access_log*
If there are no database-related issues, take a look into the JVM

Techniques you can use Thread dumps Send to Oracle Support for analysis GC logs for sizing, Profilers - Send to Oracle Support for analysis
16

GC tuning Review the frequency of collections, especially major collections (i.e. Full GC) in the logs for healthy and unhealthy activity Use jstat tool with the gc option to review FCGT count Important columns to pay attention to are survivor spaces, eden and old gen Jstat column details (observe the grey highlighted columns) Timestamp, S0C, S1C, S0U, S1U, EC, EU, OC, OU, PC, PU, YGC, YGCT, FGC, FGCT, GCT 933.9,59904.0,57728.0,0.0,40504.5,230784.0,0.0,699072.0, 666307.1,131072.0,119088.8,31,2.054,1,0.000,2.054
17

GC tuning Bigger heaps => Full GC will take longer Longer GCs => users may experience pauses Increase the number of JVMs used may provide the ability to handle more users with faster GC times. Each JVM has a smaller Xmx,Xms memory footprint so GCs will be faster! Also try to change the JVM collectors used and track the performance with Jconsole Enable the ParallelGC collector with ParallelGCThreads = 2
18

GC tuning Unhealthy JVM with FullGC activity 898753.288: [Full GC 825983K->743065K(913024K), 15.8442715 secs] 898772.243: [Full GC 825983K->750548K(913024K), 13.3512933 secs] 898798.124: [Full GC 825983K->731902K(913024K), 13.3466676 secs] 898832.082: [Full GC 825983K->738500K(913024K), 13.1591655 secs] 898857.239: [Full GC 825973K->728250K(913024K), 16.4536985 secs] Healthy JVM with FullGC activity 902130.499: [GC 829181K->716211K(935296K), 0.2895464 secs] 902220.498: [GC 833715K->721542K(935296K), 0.3210096 secs] 902400.049: [GC 839046K->725265K(936768K), 0.3448279 secs] 902400.394: [Full GC 725265K->652064K(936768K), 8.5214844 secs] 902455.778: [GC 771040K->661918K(935296K), 0.1618026 secs]
19

GC tuning Unhealthy JVM with FullGC activity 898753.288: [Full GC 825983K->743065K(913024K), 15.8442715 secs] 898772.243: [Full GC 825983K->750548K(913024K), 13.3512933 secs] 898798.124: [Full GC 825983K->731902K(913024K), 13.3466676 secs] 898832.082: [Full GC 825983K->738500K(913024K), 13.1591655 secs] 898857.239: [Full GC 825973K->728250K(913024K), 16.4536985 secs] Healthy JVM with FullGC activity Infrequent 902130.499: [GC 829181K->716211K(935296K), 0.2895464 secs] and shorter 902220.498: [GC 833715K->721542K(935296K), 0.3210096 secs] 902400.049: [GC 839046K->725265K(936768K), 0.3448279 secs] times! 902400.394: [Full GC 725265K->652064K(936768K), 8.5214844 secs] 902455.778: [GC 771040K->661918K(935296K), 0.1618026 secs]
Full GC too frequent!!!
20

Out of Memory Errors
Common causes of OutOfMemoryError Sizing Issues Heap is undersized for expected load Memory Leak Some objects not cleaned up Strong reached references, connection objects, etc.
Memory Hemorrhage Reading large data sets consuming vast amounts of memory in any period of time Connection leaks leading to vast memory/stack allocations SQL Statements/caching may lead to excess memory used in JDBC libraries
21

Out of Memory Errors
Out of Memory Heap Dump Example
This will be present in the $LOG_HOME/ora/10.1.3/opmn/default_group~oacore~default_group~1.log
10067.492: [Full GC 118783K->118783K(124928K), 1.2127460 secs] 10068.706: [Full GC 118783K->118783K(124928K), 1.2166770 secs]
10069.924: [Full GC 118783K->117977K(124928K), 1.2416330 secs]

java.lang.OutOfMemoryError: GC overhead limit exceeded Dumping heap to java_pid8515.hprof ... Heap dump file created [149076109 bytes in 10.240 secs]
Heap Location cd $INST_TOP/../../../apps/tech_st/10.1.3/j2ee/home ls -l *hprof -rw------- 1 oraperf dba 149076109 Jan 26 11:11 java_pid8515.hprof
22
23

JVM Tools
JDK 6 Jconsole Enable heap dumps to be collected from the attached JVM Goto java.sun.management | HotSpotDiagnostic | Operations | dumpHeap P0 : absolute file name (directory + filename) P1: leave to true so that only reachable objects are dumped
Use any binary hprof heap analyzer to analyze the heap dump and see which objects take up the most space JDK 6u7 and above has VisualVM Does thread/heap dumps, profiling, graphs
24

JVM Tools
Jstat Shows the JVM heap internals Shows Timestamps and Capacity/Utilization for Survivor Spaces where live objects become old enough to copy Eden new objects are created here Old Generation long lived objects are kept here Young Generation composed of eden + 2 survivor spaces Perm Generation native heap and other non-heap objects Shows the Full GC and Young GC collection count To get the data from PID nnnnn every 60s for 2 hours jstat -gc -t nnnnn 60000 120 > jstat_gcutil_pid_31041 &
26
Agenda
27
Forms Tools (strace/FRD)

Forms Next, use strace (linux), trace (solaris) to attach to the pid: strace -o formsR12_frmweb.log -p 22874 strace -o formsR12_frmsrv.log -p 17347 The logs will show the activity of the forms server as well as the listener while the user is performing their actions. FRD, Tracing, Logging for Forms Document ID 438652.1 FRD gives clues into forms code execution, parameters, variables, item values Along with strace, FRD can provide valuable clues into researching issues.
Agenda
29
Java Concurrent Program Performance

Observe the concurrent program log file after adding verbose:gc to see if the JVM is healthy or is it suffering from low memory
Increase the Memory when you encounter OOME in the logs location: Concurrent -> Program -> Define
Remember it runs in a separate JVM from oacore

All the regular java support utilities will work (jstat, eclipse mat, tda, etc) Can specify regular JVM commands in the define field to observe the JCP behaviour during the run
30
Agenda
31
Case Studies
Oracle Advanced Product Catalog
Issue Slow Performance with Rules Enabled vs Rules Disabled
Tool Used IBM Thread Dump Analyzer Analyzes thread dumps Can compare multiple thread dumps Shows different thread states with different colors Primary area of concentration is : AJPRequestHandlerHTTPThreadGroup-* These thread groups are the Apache/AJP Requests coming to oacore Solution
Thread dumps showed JVM waiting for DB to send results. This eventually lead to the creation of a cache to improve Rules Enabled performance
Case Studies
33
Case Studies
Saving a Oracle Service Request
Issue Error saving a service request
Tools Used Forms FRD
FRD showed a NULL being passed to a variable named control.mfg_org_id Executing USER_EXIT Built-in: In Argument 0 - Type: String Value: FND FFLEX PRE-UPDATE CSXSRISR, 17, Built-In, Entry, -1424992664, USER_EXIT Analyzed the CSXSRISR_LIBRARY.Init() method to find the NULL Analyzed the SR_CONTACTS.PRE_UPDATE to find how the NULL was passed to the control.mfg_org_id
Solution
Product team notified of the issue and a test case was given to isolate the affected code
Case Studies
Java Concurrent Program Issue

Out of Memory error experienced when running a Java Concurrent Program
Tool Used Eclipse MAT Analyzed heap dump Showed which package consumes significant percentage of the heap Helps derive clues and questions as to what is happening in the tech stack layers Can see what objects are occupying space and their values Solution
The Java Concurrent Programs max JVM memory was increased to 1GB to allow the program to finish successfully
35
Case Studies
36
Case Studies
37
Case Studies
38
Useful Documents
Patches 244040.1 561580.1 Roadmap 380482.1 275880.1 Browsers 1103276.1 389422.1 Oracle E-Business Suite Recommended Performance Patches Oracle E-Business Suite Release 12.1 Release Content Documents Oracle E-Business Suite Release 12 Technology Stack Documentation Roadmap Oracle Application Framework Release 11i Documentation Road Map Is Windows 7 Certified With Oracle E-Business Suite? Recommended Browsers for Oracle E-Business Suite Release 12
285218.1
Setup Diagnosis 362851.1 357597.1 1160285.1 ATG 438652.1
Recommended Browsers for Oracle E-Business Suite 11i

Guidelines to setup the JVM in Apps E-Business Suite 11i and R12 How To Generate A SQL Trace In OA Framework For Oracle Applications R12: Forms Runtime Diagnostics (FRD), Tracing And Logging For Forms In Oracle Applications Application Technology Group (ATG) Product Information Center
39
Q&A
40
41
42

PPT

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

PPT

Uploaded by

Copyright:

Available Formats

1

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Insert Information Protection Policy Classification from Slide 8

Triage and Diagnostics for E-Business Suite Systems

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Architecture Overview - R12 OA-Framework Flow

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Architecture Overview - R12 OA-Framework Flow

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Architecture Overview - R12 OA-Framework Flow

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Architecture Overview Forms Servlet Mode

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Architecture Overview Forms Socket Mode

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Patches, Browsers & Versions

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Patches, Browsers & Versions

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Patches, Browsers & Versions

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and Tools

JVM Tuning and Tools

If there are no database-related issues, take a look into the JVM

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and Tools

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and Tools

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and Tools

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and Tools

Full GC too frequent!!!

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and Tools

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and Tools

10069.924: [Full GC 118783K->117977K(124928K), 1.2416330 secs]

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and Tools

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and Tools

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and Tools

JVM Tuning and Tools

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Forms Tools (strace/FRD)

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Java Concurrent Program Performance

Remember it runs in a separate JVM from oacore

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Java Concurrent Program Issue

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Recommended Browsers for Oracle E-Business Suite 11i