Finding the Performance Bottlenecks in Your Application

Ian Jones and Roger Schrag Database Specialists, Inc. www.dbspecialists.com

IOUG-A Live! 1999 Paper #158

Finding the Bottleneck: Half the Battle in Tuning

One bad SQL statement can spoil performance
Too much code to take the lets tune every statement approach

DBA cant be familiar with every line of code

Zero in on the Bottleneck

Use the v$ views
Use SQL Trace and timed statistics Use GUI tools

Todays Presentation
Half a dozen real-life examples
scenario command-line efforts resolution

Demo of GUI tools

Enterprise Manager freeware

The Terrifyingly Slow EDI Load

A transportation company used EDI to exchange data with customers. They loaded EDI files into a temp table with SQL*Loader and then ran a 1500 line PL/SQL stored procedure to validate the data and update application tables. As more customers began sending EDI files, the PL/SQL stored procedure could no longer keep up. Data validation took as long as 24 hours for some EDI files. Where do we start tuning?

Start Load Process and Identify the Database Session

SQL> SELECT sid, serial#, status, username, osuser, 2 module, action 3 FROM v$session; SID SERIAL# STATUS USERNAME OSUSER ---- ------- -------- -------- -------1 1 ACTIVE oracle 2 1 ACTIVE oracle 3 1 ACTIVE oracle 4 1 ACTIVE oracle 5 1 ACTIVE oracle 6 1 ACTIVE oracle 7 54959 ACTIVE BJENKINS bjenkins 8 4921 INACTIVE RTHOMAS rthomas 9 2492 INACTIVE EJOHNSON ejohnson 45 3415 ACTIVE EDI_LOAD edi MODULE ACTION -------- --------

de de de SQL*Plus validate

View the Statement Being Executed

SQL> 2 3 4 SELECT FROM WHERE AND B.sql_text v$session A, v$sqlarea B A.sid = 45 B.address = A.sql_address;

SQL_TEXT ------------------------------------------------------SELECT ITEM_ID FROM ITEM_TRANSLATIONS WHERE SOURCE_ID = :b1 AND SUBSTR(SOURCE_SKU_CODE,1,6) = :b2 AND SYSDATE BETWEEN START_DATE_ACTIVE AND NVL (END_DATE_ACTIVE, SYSDATE)

We Found a Bottleneck!
PROCEDURE edi_validate_and_load (p_cust_id IN NUMBER) IS CURSOR c_get_item_id (cp_cust_id IN NUMBER, cp_sku IN VARCHAR2) IS SELECT item_id FROM item_translations WHERE source_id = cp_cust_id AND source_sku_code LIKE cp_sku || '%' AND SYSDATE BETWEEN start_date_active AND NVL (end_date_active, SYSDATE);

Disk Array Far Too Busy

One third of a financial institutions loan processing department started using a new PowerBuilder application. Response time was acceptable, but disk utilization on the server was at 100%. What could be done to reduce I/O so that the response time will still be acceptable after the rest of the department starts using the new application?

Identify the SQL Statements Causing the Most Disk Reads

SELECT sql_text, disk_reads, executions, disk_reads / DECODE (executions, 0, 1, executions) reads_per_exec FROM v$sqlarea ORDER BY reads_per_exec;

SELECT

sql_text, disk_reads, executions, disk_reads / DECODE (executions, 0, 1, executions) reads_per_exec FROM v$sqlarea ORDER BY disk_reads;
SELECT sql_text, buffer_gets, executions, buffer_gets / DECODE (executions, 0, 1, executions) gets_per_exec FROM v$sqlarea ORDER BY buffer_gets;

Part of the Query Results

SQL_TEXT -----------------------------------------------------------DISK_READS EXECUTIONS READS_PER_EXEC ---------- ---------- -------------SELECT P.PRODUCT_DESC, CP.PRODUCT_ID, UPPER (:b1) CLIENT_ID FROM CLIENT_PRODUCT CP, PRODUCT P WHERE CP.PRODUCT_ID = P.PRODUCT_ID AND (UPPER (CP.CLIENT_ID), CP.VALID_CLIENT_LEVEL_ID) IN (SELECT UPPER (:b1), CA.VALID_CLIENT_LEVEL_ID FROM CLIENTS C, CLIENT_ADDRESS CA WHERE UPPER (C.CLIENT_ID) = UPPER(:b1) AND C.CLIENT_ID = CA.CLIENT_ID) 208734602 18657 11188.0046

The Same Query Formatted for Readability

SELECT P.product_desc, CP.product_id, UPPER (:b1) client_id FROM client_product CP, product P WHERE CP.product_id = P.product_id AND (UPPER (CP.client_id), CP.valid_client_level_id) IN (SELECT UPPER (:b1), CA.valid_client_level_id FROM clients C, client_address CA WHERE UPPER (C.client_id) = UPPER(:b1) AND C.client_id = CA.client_id)

A case-insensitive query turns out to be very inefficient. Now we know what to tune.

The So-Called Locking Problem

A developer coded a PL/SQL function called compute_irr for computing internal rate of return. Response times varied widely. The developer claimed there was a locking problem on the database. What caused the slow performance in compute_irr?

Start SQL*Plus and Identify the Database Session

SQL> SELECT sid, serial#, status, username, osuser, 2 module, action 3 FROM v$session; SID SERIAL# STATUS USERNAME ---- ------- -------- -------1 1 ACTIVE 2 1 ACTIVE 3 1 ACTIVE 4 1 ACTIVE 5 1 ACTIVE 6 1 ACTIVE 9 2041 INACTIVE GL 10 4284 INACTIVE APPLSYS 62 63 7219 ACTIVE RSCHRAG 7394 INACTIVE FA OSUSER MODULE ACTION -------- -------- -------oracle oracle oracle oracle oracle oracle swatkins applmgr rschrag rschrag SQL*Plus SQL*Plus

Check Session Statistics Before Calling compute_irr

SQL> SELECT A.name, B.value

2
3 4 5 6

FROM
WHERE AND AND

v$statname A, v$sesstat B
B.statistic# IN (12, 37, 38, 39, 119, 123, 139, 140, 141) B.sid = 63 A.statistic# = B.statistic#;

NAME VALUE ------------------------------ ---------CPU used by this session 1292 db block gets 10186 consistent gets 86810 physical reads 346 table scans (long tables) 0 table scan rows gotten 1054 sorts (memory) 826 sorts (disk) 0 sorts (rows) 4693

Check Session Statistics Again While compute_irr Is Running

SQL> SELECT A.name, B.value 2 3 4 5 6 AND AND B.sid = 63 A.statistic# = B.statistic#; FROM WHERE v$statname A, v$sesstat B B.statistic# IN (12, 37, 38, 39, 119, 123, 139, 140, 141)

NAME VALUE ------------------------------ ---------CPU used by this session 7274 db block gets 10294 consistent gets 86813 physical reads 346 table scans (long tables) 0 table scan rows gotten 1054 sorts (memory) 826 sorts (disk) 0 sorts (rows) 4693

Compute the Deltas

Session Statistic CPU Used by this session db block gets consistent gets physical reads table scans (long tables) table scan rows gotten sorts (memory) sorts (disk) sorts (rows) Before 1292 10186 86810 346 0 1054 826 0 4693 After 60 Seconds 7274 10294 86813 346 0 1054 826 0 4693 After 120 Seconds 13255 10294 86813 346 0 1054 826 0 4693

Look for an infinite loop that involves no SQL statements.

Where Is the I/O Coming From?

An application that performed well in a demo database quickly became I/O-bound when a significant amount of data was loaded into the database. It seemed as if there was a huge amount of I/O even with no users logged on to the application. What could be causing so much I/O activity?

Look at Physical Reads Instance-Wide and Per Session

SQL> 2 3 4 5 6 7 SELECT name || ' (instance-wide)', value FROM v$sysstat WHERE statistic# = 39 UNION ALL SELECT 'sid = ' || TO_CHAR (sid), value FROM v$sesstat WHERE statistic# = 39;

And the Results Are...

NAME VALUE -------------------------------- ---------physical reads (instance-wide) 6048399 sid = 1 0 sid = 2 0 sid = 3 0 sid = 4 0 sid = 5 0 sid = 6 23120 sid = 7 186 sid = 8 0 sid = 9 18984 sid = 12 1375 sid = 13 5830219 sid = 17 27821

Investigate Suspicious Session 13

SQL> SELECT sid, serial#, status, username, osuser, 2 module, action 3 FROM v$session 4 WHERE sid = 13; SID SERIAL# STATUS USERNAME OSUSER MODULE ACTION ---- ------- -------- --------- -------- -------- -------13 829 ACTIVE APPSCHEMA daemon EXTRACT GET SQL> SELECT A.name, B.value 2 FROM v$statname A, v$sesstat B

3
4 5 6

WHERE
AND AND

B.statistic# IN (12, 37, 38, 39, 119, 123,

139, 140, 141) B.sid = 13 A.statistic# = B.statistic#;

Suspicious Session 13 (continued)

NAME VALUE ------------------------------ ---------CPU used by this session 0 db block gets 5928722 consistent gets 293 physical reads 5873918 table scans (long tables) 575 table scan rows gotten 72400000 sorts (memory) 3 sorts (disk) 0 sorts (rows) 4210

What SQL Is Session 13 Executing?

SQL> SELECT B.sql_text

2
3 4

FROM
WHERE AND

v$session A, v$sqlarea B
B.address = A.sql_address A.sid = 13;

SQL_TEXT -----------------------------------------------------SELECT * FROM TRANSACTIONS WHERE EXTRACTED = 'N

Let me guess! The transactions table is about 10,000 blocks in size and the extracted column is not indexed.

Fast Reports That Were Too Slow

A software vendor built an application using Oracle, Developer/2000, and a third-party reporting tool. A key report took about six seconds to complete, but consider that users will run the report in batches of several thousand. The report has thousands of lines of spaghetti code. How do you figure out what's slowing it down?

Build a Version of the Report That Enables SQL Trace

DECLARE

c INTEGER;
i INTEGER; BEGIN c := dbms_sql.open_cursor;

dbms_sql.parse (c,
'ALTER SESSION SET TIMED_STATISTICS=TRUE', dbms_sql.native); i := dbms_sql.execute (c);

dbms_sql.close_cursor (c);
dbms_session.set_sql_trace (TRUE); END;

Trace only what you need to trace!

Run the Report, Fetch the Trace File, Run TKPROF On It

OVERALL TOTALS FOR ALL NON-RECURSIVE STATEMENTS
call count ------- -----Parse 38 Execute 38 Fetch 41 ------- -----total 117 cpu elapsed disk query -------- ---------- ---------- ---------3.01 4.22 12 591 0.40 0.65 57 218 0.57 0.77 91 294 -------- ---------- ---------- ---------3.98 5.64 160 1103

Misses in library cache during parse: 30

Look into using bind variables to reduce parsing.

The Chameleon Application

An application ran well on a test database loaded with a full set of production data. But when the application was deployed in production, queries took over a minute to complete. In the test environment these same queries completed in under ten seconds. Policies forbid modifying code in production. What could be causing the application to run slower in production?

Find a Power User to Reproduce the Slow Behavior

Identify the database session:
SQL> SELECT sid, serial#, status, username, osuser, 2 module, action 3 FROM v$session 4 WHERE username = 'MARYD';
SID SERIAL# STATUS USERNAME OSUSER MODULE ACTION ---- ------- -------- --------- -------- -------- -------17 9172 INACTIVE MARYD mbd frontend query

Enable Timed Statistics Temporarily

SQL> ALTER SYSTEM SET TIMED_STATISTICS = TRUE; System altered.

SQL>

Enable SQl Trace Just Before the Query Is Launched

SQL> BEGIN 2 dbms_system.set_sql_trace_in_session (17, 9172, TRUE); 3 END; 4 / PL/SQL procedure successfully completed. SQL>

Disable SQL Trace When the Query Is Finished

SQL> BEGIN 2 dbms_system.set_sql_trace_in_session (17, 9172, FALSE); 3 END; 4 / PL/SQL procedure successfully completed.
SQL>

SQL> ALTER SYSTEM SET TIMED_STATISTICS = FALSE;

System altered. SQL>

Fetch the Trace File and Run TKPROF

call count ------- -----Parse 1 Execute 1 Fetch 17 ------- -----total 19 cpu elapsed disk query current -------- ---------- ---------- ---------- ---------1.44 1.45 0 0 0 0.00 0.00 0 0 0 68.39 68.54 0 1878 2 -------- ---------- ---------- ---------- ---------69.83 69.99 0 1878 2 rows ---------0 0 254 ---------254

Misses in library cache during parse: 1 Optimizer goal: RULE Parsing user id: 142 (BUILD4P2) Rows ------0 0 254 115 253 254 114539 253 254 115 254 129 129 Execution Plan --------------------------------------------------SELECT STATEMENT GOAL: RULE MERGE JOIN (OUTER) SORT (JOIN) NESTED LOOPS (OUTER) NESTED LOOPS (OUTER) TABLE ACCESS GOAL: ANALYZED (BY ROWID) OF 'MNME' INDEX GOAL: ANALYZED (FULL SCAN) OF 'MNME_I1' (UNIQUE) TABLE ACCESS GOAL: ANALYZED (BY ROWID) OF 'LCONTYPE' INDEX GOAL: ANALYZED (UNIQUE SCAN) OF 'LCONTYPE_PK' (UNIQUE) TABLE ACCESS GOAL: ANALYZED (BY ROWID) OF 'MTAX' INDEX GOAL: ANALYZED (UNIQUE SCAN) OF 'MTAX_I1' (UNIQUE) SORT (JOIN) TABLE ACCESS GOAL: ANALYZED (FULL) OF 'LPOST

The optimizer can behave differently from one Oracle version to the next.

GUI Tools Demo

To Find the Bottlenecks in Your Applications:

Monitor v$sqlarea Monitor v$sysstat and v$sesstat Use SQL Trace judiciously Consider using GUI tools

Resources
Oracle Server Tuning - Overview of the tuning process - How to use SQL Trace and TKPROF Oracle Server Reference - Descriptions of all v$ views High Performance SQL Tuning by Guy Harrison - Lots of tuning tips - Discussion of GUI tools available on the Internet

www.dbspecialists.com/present.html
- Download this presentation - Download a companion white paper

Contact Information
Ian Jones: ijones@dbspecialists.com Roger Schrag: rschrag@dbspecialists.com
Database Specialists, Inc. 388 Market Street, Suite 400 San Francisco, CA 94111 415/344-0500