Performance Tuning : Diagnostic Tools and Traces

Rakesh Tikku ATS Performance

Presentation_ID 2006-2007 dbPerf Inc. All rights reserved. Cisco Confidential

Agenda
SQL Trace / TKPROF PL/SQL Profiler

Forms Runtime Diagnostics

Reports Trace

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

SQL TRACE

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

SQL Trace
It is a diagnostic/debugging facility that provides runtime performance statistics for individual SQL statements that your application runs. It also records the execution plan that was used to execute each SQL statement. In most cases, enabled at the session level. It is a text file, but raw.

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

SQL Trace
Level 1 - Standard Sql Trace Level 4 - Level 1 PLUS Bind values
BINDS #1: bind 0: dty=2 mxl=22(22) mal=00 scl=00 pre=00 oacflg=03 oacfl2=0 size=24 offset=0 bfp=80000001000cbc78 bln=22 avl=04 flg=05 value=144867

Level 8 - Level 1 PLUS Wait Statistics

WAIT #1: nam='file open' ela= 0 p1=0 p2=0 p3=0
WAIT #1: nam='db file sequential read' ela= 2 p1=330 p2=121062 p3=1

Level 12 - Level 1 PLUS Bind values and Wait Statistics In >80% cases, Standard Sql Trace is good enough
Presentation_ID 2006-2007 dbPerf Inc. All rights reserved. Cisco Confidential

SQL Trace
sql_trace=false
Enables sql tracing for the entire instance

timed_statistics=true
Enables collection of CPU and elapsed times

max_dump_file_size=val [K|M]|unlimited
Specifies max size of the trace file

user_dump_dest=<directory>
Specifies location where trace files should be written

tracefile_identifier=string
Creates trace files with the name sid_ora_pid_string.trc

_trace_files_public=true
Creates trace files with mode 644 instead of 600
Presentation_ID 2006-2007 dbPerf Inc. All rights reserved. Cisco Confidential

Enabling Trace - SQL Plus/PLSQL

Enabling trace for the current session
alter session set sql_trace=true; alter session set events 10046 trace name context forever, level <x>;

dbms_session.set_sql_trace(true); dbms_support.start_trace(waits=>true,binds=>true);

Enabling trace for a different session

dbms_system.set_sql_trace_in_session (SID,SERIAL#,TRUE); DBMS_SUPPORT.START_TRACE_IN_SESSION( SID , SERIAL#, waits=>TRUE, binds=>TRUE )

(10g) dbms_monitor.session_trace_enable(SID,SERIAL#, waits=>TRUE, binds=>TRUE ) (10g) dbms_monitor.session_trace_disable(SID,SERIAL#)

Presentation_ID 2006-2007 dbPerf Inc. All rights reserved. Cisco Confidential

Enabling Trace - Forms

Set profile option Utilities:Diagnostics to Yes in order to suppress Apps password prompt.

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

Enabling Trace OA FWK

Set profile FND : Diagnostics to Yes at user level Login to Self Service as the above user Click on Diagnostics icon at the top of page

Select Set Trace Level and click Go

Select the desired trace level and click Save Perform the activity that you want to trace Exit application

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

Enabling Trace OA FWK

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

10

Enabling Trace Concurrent Program

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

11

OAM Debug Workbench

Set Profile option Concurrent: Allow Debugging to Yes OAM minipack H + rollup 1 or later is required.

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

12

OAM Debug Workbench

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

13

Enabling Trace Generic Method

Set the profile Initialization SQL Statement Custom at user level to the following value
ALTER SESSION SET TRACEFILE_IDENTIFIER=CLASSXX EVENTS =' 10046 TRACE NAME CONTEXT FOREVER, LEVEL <x> '

This will trace all the sessions of that user at level <x>
The trace file name will be of the format :
sid_ora_spid_CLASSXX.trc
- It is not recommended to use this method to enable trace for an application that uses a connection pools as it can dirty the pool with connections with SQL trace on. - Examples of applications using connection pools -OA FWK -JTF Stack
Presentation_ID 2006-2007 dbPerf Inc. All rights reserved. Cisco Confidential

14

TKPROF

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

15

TKPROF
Utility that read the raw SQL trace file and produces a user readable output file Summarizes information from the raw trace file

Can generate execution plans

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

16

TKPROF
Contains the following information
The text of the SQL statement. The SQL trace statistics in tabular form. The number of library cache misses for the parsing and execution of the statement. The user initially parsing the statement. Runtime Execution Plan, listed under Row Source Operation The execution plan generated by EXPLAIN PLAN. Wait Statistics ( 9i onwards )

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

17

TKPROF
tkprof tracefile outputfile [Optional Params] Optional Parameters
Explain=username/password
Table=<username>.<tablename> Sys=yes|no Sort=sort option(s)

Sort option is a concatenation of cursor operation and statistic type

Operation PRS parse EXE execute Statistic Type CNT Count CPU CPU Time

FCH

fetch

ELA

Elapsed Time

ROW Number of Rows DSK Number of Disk Reads

Presentation_ID 2006-2007 dbPerf Inc. All rights reserved. Cisco Confidential

18

TKPROF
Sample Output
select party_name from hz_parties where party_id=:b1

call Parse Execute

count 1 1

cpu 0.00 0.00

elapsed 0.01 0.00

disk 0 0

query 0 0

current 0 0

rows ---------0 0

------- ------

-------- ---------- ---------- ---------- ----------

Fetch
total

2
4

0.00
0.00

0.06
0.07

3
3

4
4

0
0

1
---------1

------- ------

-------- ---------- ---------- ---------- ----------

Misses in library cache during parse: 1 Optimizer goal: CHOOSE Parsing user id: 33 (APPS)

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

19

TKPROF
Rows ------1 1 Row Source Operation --------------------------------------------------TABLE ACCESS BY INDEX ROWID HZ_PARTIES INDEX UNIQUE SCAN (object id 34093)

Rows ------0 1 1

Execution Plan --------------------------------------------------SELECT STATEMENT TABLE ACCESS INDEX (UNIQUE) GOAL: CHOOSE GOAL: ANALYZED (BY INDEX ROWID) OF 'HZ_PARTIES'

GOAL: ANALYZED (UNIQUE SCAN) OF 'HZ_PARTIES_U1'

Elapsed times include waiting on following events: Event waited on ---------------------------------------SQL*Net message to client db file sequential read file open Times Waited 2 3 2 Max. Wait ---------0.00 0.03 0.01 Total Waited -----------0.00 0.06 0.01

SQL*Net message from client

9.99

9.99

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

20

TKPROF
OVERALL TOTALS FOR ALL NON-RECURSIVE STATEMENTS call Parse Execute Fetch total count 538 560 363 1461 cpu 0.70 4.11 0.44 5.25 elapsed 0.95 6.20 4.34 11.49 disk 1 0 278 279 query 670 44942 3401 49013 current 0 45 52 97 rows ---------0 224 1157 ---------1381

------- ------

-------- ---------- ---------- ---------- ----------

------- ------

-------- ---------- ---------- ---------- ----------

Misses in library cache during parse: 60

OVERALL TOTALS FOR ALL RECURSIVE STATEMENTS call Parse Execute Fetch total count 4578 10944 29021 44543 cpu 4.30 2.47 6.49 13.26 elapsed 4.71 3.19 34.64 42.54 disk 2 109 2583 2694 query 38 1365 287337 288740 current 3 3048 1902 4953 rows ---------0 580 29597

------- ------ -------- ---------- ---------- ---------- ----------

------- ------

-------- ---------- ---------- ---------- ----------

---------30177

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

21

TKPROF
Interpreting tkprof Statistics
Each row corresponds to one of the three phases of SQL processing Parse Execute Fetch The columns display the statistic value for each phase Count The number of times each statement was parsed, executed or fetched CPU Total CPU time in seconds for all parse, execute, or fetch calls for the statement Elapsed Total elapsed time in seconds for all parse, execute, or fetch calls for the statement
Presentation_ID 2006-2007 dbPerf Inc. All rights reserved. Cisco Confidential

22

TKPROF
Disk Total number of data blocks physically read from the datafiles on disk for all parse, execute, or fetch calls Query

Total number of buffers retrieved in consistent mode for all parse, execute, or fetch calls
Current Total number of buffers retrieved in current mode

Rows
Total number of rows processed by the SQL statement Does not include the rows processed by the sub-queries of the SQL statement

The last row displays the totals

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

23

TKPROF
TKPROF also lists the number of library cache misses resulting from parse and execute steps for each SQL statement. This implies a HARD parse. Run Time Execution Plan Row Source Counts
Row source counts are displayed when the cursor is closed Do not call sql_trace=false, otherwise the cursors will not be closed. Instead, exit from the session.

TKPROF generated Execution Plan, if tkprof was run with the EXPLAIN option Totals of Recursive and Non-recursive SQL statements

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

24

TKPROF
Interpreting Output - what to look out for
High Physical and Logical reads High CPU Time High Elapsed Time A large gap between elapsed and CPU time indicates that the server process was waiting for something, most often the waits are due to the process having to do physical I/O Check the number of physical reads and try to account for them Wait statistics (level 8 / level 12 sql trace ) are usually provide sufficient information to pin-point the cause Multiple parse calls for one statement Multiple execute calls for one statement

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

25

TKPROF
Misses in library cache during parse -> Hard Parse Rows fetched per fetch Totals for Non-Recursive and Recursive statements Check for Throw-away. Throw-away can be caused by: Full scans Unselective range scans Late predicate filters Wrong join order Late predicate operations Throw-away Example :
Rows ------Row Source Operation ---------------------------------------------------

11
107317043 11

FILTER
TABLE ACCESS FULL CFI_RRR_AR_ACTUALS INDEX UNIQUE SCAN SYS_C00307768 (object id 38788)

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

26

TKPROF
What does raw trace have that TKPROF does not?
Bind information Most important are the bind values and their datatypes Specific execution, instead of aggregates Chronological execution order Recursive Level RPC procedure (server side) calls from Forms

Pre 9i, wait events

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

27

TKPROF
A truncated trace file will provide incomplete information. Go to the end of the trace file and check for the following line.
*** DUMP FILE SIZE IS LIMITED TO 12345 BYTES***

Make sure there are not discrepancies between run-time execution plan and tkprof-generated execution plan If the trace file has been generated on 9i, use the 9i tkprof executable.

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

28

PL/SQL PROFILER

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

29

PL/SQL Profiler
API to profile PL/SQL applications and identify performance bottlenecks Introduced in 8i

Common Causes of PL/SQL performance issues

Initializing large data structures Passing large data structures as parameters Leaving code within a loop instead of outside Using a slow or linear search/sort method Inefficient Conditional Control Statements Implicit datatype conversions

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

30

PL/SQL Profiler - Setting it up

Create the DBMS_PROFILER package
Run $ORACLE_HOME/rdbms/admin/profload.sql in the APPS schema

Create the run-time PL/SQL profiler tables

Run $ORACLE_HOME/rdbms/admin/proftab.sql in the schema running the application, in our case APPS Profiler Tables
PLSQL_PROFILER_RUNS PLSQL_PROFILER_UNITS PLSQL_PROFILER_DATA

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

31

PL/SQL Profiler - Running it

Execute dbms_profiler.start_profiler(<run name>);
Starts profiler data collection in session

Execute PL/SQL code to be profiled

Execute dbms_profiler.stop_profiler;
Flushes data into profiler tables and stops profiler data collection Exiting session does not save data automatically Profiler data is stored in memory. You can call dbms_profiler.flush_data at various points in your code save incremental data and free memory, if you expect a lot of data to be generated

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

32

PL/SQL Profiler Querying Profiler Data

Script available $ORACLE_HOME/plsql/demo
profsum.sql Gives a summary of all the plsql units called in the run

You can also refer to a script available on Metalink. Refer to Note 243755.1 for details.

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

33

PL/SQL Profiler Sample Output

OWNER APPS APPS APPS APPS APPS Line# 5 UNIT_NAME INV_SERIAL_NUMBER_PUB INV_LOG_UTIL FND_DFLEX INV_LOT_SEL_ATTR FND_PROFILE #Occ MICROSECONDS TEXT 1 3.178 g_debug_init boolean := false; -- Log initialize SECONDS .23 .12 .11 .02 .01 PRCNT 45.65 24.39 22.28 4.64 2.33 ------ -------------------------------- --------- ---------

----- -------- ------------ --------------------------------------------------

7
8 9 22 23 24 25

1
1 18 17 1 1 1

2.208 g_trace_on number := 0;

.748 g_dbg_lvl number := 0; 490.335 g_cp_flag number := 0; 33.102 118603.878 3427.758 8.615 if (g_debug_init = false) then

-- Log ON state

select fnd_profile.value('INV_DEBUG_TRACE') into select NVL(fnd_profile.value('CONC_REQUEST_ID'), if (g_trace_on = 1) then

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

34

PL/SQL Profiler Output

Use the profsum.sql to find out the expensive plsql units that were called Check which lines show high timings and number of executions If it is unnecessary code, remove it. Otherwise think of cheaper ways of doing the same thing

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

35

PLSQL Profiler - Forms

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

36

PLSQL Profiler - Forms

Help-> Diagnostics -> Trace -> PL/SQL profiler Click checkbox to Enable Perform the action

When done, click checkbox to disable.

Exit from the application You will get a message that a concurrent report has been submitted. Login back again and view the output for this concurrent request.

Output is a hyperlinked profiler report will summary and detail sections.

Presentation_ID 2006-2007 dbPerf Inc. All rights reserved. Cisco Confidential

37

PLSQL Profiler OA FWK

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

38

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

39

FORMS RUNTIME DIAGNOSTICS

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

40

Forms Runtime Diagnostics

Forms Runtime Diagnostics (FRD) is a runtime eventbased logging system intended to aid in the debugging of Forms applications FRD is enabled at runtime on a user by user basis Trace file contains chronological representation of external user events and internal Forms processing events FRD generates extensive output useful for analysis and debugging, and it is therefore recommended FRD be utilized as a logging mechanism only when specific issues are encountered.

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

41

Forms Runtime Diagnostics

Events traced by FRD
Navigation Trigger Firing: includes name and location Built-in execution (includes name and IN and OUT parameter types and values) All external user events Messages FMX/MMX file opens Unhandled exceptions

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

42

Forms Runtime Diagnostics

Logon to Applications 11i with the following text appended to the URL: play=&record=all&log=/tmp/form1.trc Reproduce the steps that you wish to trace in Applications

Exit from the Application completely

Retrieve the trace file from the application server Analyze the trace file In case you log into Apps through PHP, you need to set your user level profile ICX:Forms Launcher. The value you need to set this to is the string mentioned above appended to the value of this profile option at the Site level.

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

43

REPORTS TRACE

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

44

Reports Trace
Enabled by appending tracing parameters to command line
ar60run TRACEFILE=<file> TRACE_OPTS=(<trace options>) TRACEMODE=trace_replace|trace_append Where trace options can be one or more of the following :
TRACE_ALL : log all possible trace information in the trace file. (DEFAULT) TRACE_APP : log trace information on all the report objects in the trace file. TRACE_BRK : list breakpoints in the trace file.

TRACE_ERR : list error messages and warnings in the trace file.

TRACE_PLS : log trace information on all the PL/SQL objects in the trace file. TRACE_PRF : log performance statistics in the trace file. TRACE_SQL : log trace information on all the SQL in the trace file. TRACE_TMS : enter a timestamp for each entry in the trace file.

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

45

Reports Trace
Sign-on to Apps with System Administrator responsibility Go to Concurrent -> Program -> Define Query up your Reports concurrent program

Locate the Options field and enter the following values :

TRACEOPTS=<trace option> TRACEFILE=<path/file>

Save the concurrent program

Running the concurrent program now will generate the reports trace file as specified by TRACEFILE
Caution : Since this involves changes at the report definition level, this operation should be performed on the test instance when no other user is running this report.
Presentation_ID 2006-2007 dbPerf Inc. All rights reserved. Cisco Confidential

46

Reports Trace
LOG :
Report: /u01/CRMPERF/appl/fnd/11.5.0/reports/US/FNDSCURS.rdf Logged onto server: Username: . . . . . . . . 23:59:57 23:59:57 23:59:57 APP ... ( APP ... ) APP ... ( Generic Graphical Object B_SECURITY_GROUP_NAME1 Generic Graphical Object B_SECURITY_GROUP_NAME1 Generic Graphical Object B_USER_NAME1 +-------------------------------------+ | Report Builder Profiler statistics | +-------------------------------------+ TOTAL ELAPSED Time: Reports Time: 2243.04 seconds

. . . . . . . .

1196.62 seconds (53.34% of TOTAL) 1046.41 seconds (46.65% of TOTAL) 1046.00 seconds 0.40 seconds 904.71 seconds
Cisco Confidential

ORACLE Time: UPI: SQL: TOTAL CPU Time used by process:

Presentation_ID 2006-2007 dbPerf Inc. All rights reserved.

47

Reports Trace
From the profile output, it is possible to see :
Execution time (total elapsed time) for the report Time that was spent on formatting the data (Reports Time)

Time spent in running Reports Triggers, Formula columns, etc.

Time spent for the data to be retrieved (Oracle Time).

UPI time is the time spent in establishing the database connection and parsing and executing the SQL.
The SQL time is the time spent on fetching the data by the database server (percent of time spent executing SRW.DO_SQL() statements, EXEC_SQL statements, PL/SQL Cursors, etc.).
Presentation_ID 2006-2007 dbPerf Inc. All rights reserved. Cisco Confidential

48

Q&A
Presentation_ID 2006-2007 dbPerf Inc. All rights reserved. Cisco Confidential

49

Presentation_ID

2006-2007 dbPerf Inc. All rights reserved.

Cisco Confidential

50