Wait Event Enhancements in Oracle 10g

Terry Sutton and Roger Schrag Database Specialists, Inc.

www.dbspecialists.com

Todays Session
 Twelve wait event interface enhancements in Oracle 10g that we really like.
Documentation gaps in some areas make this information harder to come by.

 We will assume everyone is familiar with wait event concepts and the wait event interface in Oracle 9i or earlier.  To learn more about wait events, read: www.dbspecialists.com/presentations.html #wait_events
2

White Paper
 Contains additional detail we wont have time to cover today.  Includes a handy reference list of all Oracle 10g wait events, wait classes, and wait event parameters.  Download: www.dbspecialists.com/presentations

Twelve Oracle 10g Enhancements

More descriptive wait event names Wait event classes v$event_name new columns v$sql / v$sqlarea new columns v$session_wait_history v$session new columns v$event_histogram v$system_wait_class / v$session_wait_class Active Session History (ASH) Automatic Workload Repository (AWR) Time model statistics Improved session tracing
4

More Descriptive Names

 Prior to Oracle 10g many wait events had vague names that covered many situations:
latch free enqueue buffer busy waits

 For these waits you had to look at parameter values to learn the wait condition.  Oracle 10g gives the most common of these types of waits more descriptive names.
5

latch free
 Prior to Oracle 10g: Could indicate a wait on any of dozens of different latches.  Oracle 10g: The 26 most common latches have their own wait event.
The rest continue to use the generic latch free wait event.

latch free Prior to Oracle 10g

SQL> SELECT event, state, p1, p2, p3 FROM v$session_wait WHERE sid = 162; EVENT STATE P1 ------------- ------- ----------latch free WAITING 15113593728 P2 P3 ------ ----97 5

SQL> SELECT * FROM v$event_name WHERE name = 'latch free'; EVENT# NAME PARAMETER1 PARAMETER2 PARAMETER3 ------ ---------- --------------- --------------- --------------3 latch free address number tries SQL> SELECT name FROM v$latch WHERE latch# = 97; NAME -------------------cache buffers chains

latch free in Oracle 10g

SQL> SELECT event, state 2 FROM v$session_wait 3 WHERE sid = 162; EVENT STATE ------------------------------ ------latch: cache buffers chains WAITING

 The more descriptive wait event name saves us the effort of decoding wait event parameters.

enqueue
 Prior to Oracle 10g: Could indicate a wait on any of a few dozen different types of enqueues.  Oracle 10g: 184 distinct wait events replace the one generic enqueue wait event:
Event names differentiate the enqueue type and sometimes even the contention type. Parameter names are more descriptive than generic id1 and id2.

enqueue Prior to Oracle 10g

SQL> SELECT event, state, seconds_in_wait FROM v$session_wait WHERE sid = 96; EVENT STATE SECONDS_IN_WAIT ----------------------------------- ------------------- --------------enqueue WAITING 24 SQL> SELECT sid, CHR (BITAND (p1,-16777216) / 16777215) || 2 CHR (BITAND (p1, 16711680) / 65535) enq, 3 DECODE (CHR (BITAND (p1,-16777216) / 16777215) || 4 CHR (BITAND (p1, 16711680) / 65535), 5 'TX', 'Transaction (RBS)', ... 13 CHR (BITAND (p1, 16711680) / 65535)) enqueue_name, 14 DECODE (BITAND (p1, 65535), 1, 'Null', 2, 'Sub-Share', 15 3, 'Sub-Exclusive', 4, 'Share', 5, 'Share/Sub-Exclusive', 16 6, 'Exclusive', 'Other') lock_mode 17 FROM v$session_wait WHERE sid = 96; SID ENQ ENQUEUE_NAME LOCK_MODE ----- ---- ------------------------------ ---------96 TX Transaction (RBS) Exclusive

10

enqueue in Oracle 10g

SQL> SELECT event, state, seconds_in_wait FROM v$session_wait WHERE sid = 143; EVENT STATE SECONDS_IN_WAIT ----------------------------------- ------------------- --------------enq: TX - row lock contention WAITING 495

 Separate events for separate TX issues:

SQL> SELECT name, parameter1, parameter2, parameter3 2 FROM v$event_name 3 WHERE name LIKE 'enq: TX%'; NAME -----------------------------enq: TX - contention enq: TX - row lock contention enq: TX - allocate ITL entry enq: TX - index contention PARAMETER1 ------------name|mode name|mode name|mode name|mode PARAMETER2 --------------usn<<16 | slot usn<<16 | slot usn<<16 | slot usn<<16 | slot PARAMETER3 ------------sequence sequence sequence sequence 11

enqueue in Oracle 10g

 More descriptive parameter names, too:
SQL> SELECT name, parameter1, parameter2, parameter3 2 FROM v$event_name 3 WHERE name IN ('enq: HW - contention', 'enq: SQ - contention'); NAME -----------------------------enq: HW - contention enq: SQ - contention PARAMETER1 ------------name|mode name|mode PARAMETER2 --------------table space # object # PARAMETER3 ------------block 0

12

Wait Event Classes

 Every wait event belongs to one of 12 classes.  Classes can help point toward a root cause.  Almost 70% of all wait events belong to a class called Other.  Most of the wait events that occur frequently belong to wait event classes with helpful names.

13

Wait Event Class Names

Administrative Application Cluster Commit Concurrency Configuration Idle Network Scheduler System I/O User I/O Other

14

Wait Class Assignments

SQL> 2 3 4 SELECT FROM WHERE ORDER BY wait_class, name v$event_name name LIKE 'enq: TX%' wait_class, name; NAME ---------------------------------------enq: TX - row lock contention enq: TX - index contention enq: TX - allocate ITL entry enq: TX - contention

WAIT_CLASS --------------Application Concurrency Configuration Other

15

v$event_name
 Three new columns show which wait class a wait event belongs to:
SQL> DESCRIBE v$event_name Name Null? ----------------------------------------- -------EVENT# EVENT_ID NAME PARAMETER1 PARAMETER2 PARAMETER3 WAIT_CLASS_ID WAIT_CLASS# WAIT_CLASS Type ------------------NUMBER NUMBER VARCHAR2(64) VARCHAR2(64) VARCHAR2(64) VARCHAR2(64) NUMBER NUMBER VARCHAR2(64)

16

v$sql and v$sqlarea

 Six new columns give more information about how time was spent executing a SQL statement:
application_wait_time concurrency_wait_time cluster_wait_time user_io_wait_time plsql_exec_time java_exec_time

 Times are in microseconds.

17

v$sqlarea Example
 In session #1:
SQL> UPDATE testtab SET numcol = numcol + 1 WHERE ROWNUM < 1000;

 In session #2:
SQL> UPDATE testtab SET numcol = numcol + 1 WHERE ROWNUM < 1000;

 In session #1:
SQL> ROLLBACK;

 In session #2:
SQL> ROLLBACK;

 In session #3:
SQL> UPDATE testtab SET numcol = numcol + 1; 18

v$sqlarea Example
SQL> SELECT sql_id, application_wait_time appl, 2 concurrency_wait_time concurr, user_io_wait_time user_io 3 FROM v$sqlarea 4 WHERE sql_text LIKE 'UPDATE testtab SET numcol%'; SQL_ID APPL CONCURR USER_IO ------------- --------- --------- ----------038m56cp4am0c 178500000 0 20000 fd5mxhdbf09ny 0 10000 105040000 SQL> SELECT sql_id, sql_text 2 FROM v$sqlarea 3 WHERE sql_id IN ('fd5mxhdbf09ny','038m56cp4am0c'); SQL_ID ------------038m56cp4am0c fd5mxhdbf09ny SQL_TEXT ----------------------------------------------------------UPDATE testtab SET numcol = numcol + 1 WHERE ROWNUM < 1000 UPDATE testtab SET numcol = numcol + 1

19

v$session_wait_history
 New view in Oracle 10g.  Similar to v$session_wait, but shows the last ten wait events for each session.  The seq# column is supposed to show the order in which the waits occurred, with 1 being the most recent wait:
Different from seq# in v$session. Does not appear to work as documented (on our 10.1.0.3 system on Solaris).
20

v$session_wait_history
SQL> DESCRIBE v$session_wait_history Name Null? ----------------------------------------- -------SID SEQ# EVENT# EVENT P1TEXT P1 P2TEXT P2 P3TEXT P3 WAIT_TIME WAIT_COUNT Type -------------------------NUMBER NUMBER NUMBER VARCHAR2(64) VARCHAR2(64) NUMBER VARCHAR2(64) NUMBER VARCHAR2(64) NUMBER NUMBER NUMBER

21

v$session_wait_history Example
SQL> 2 3 4 SELECT FROM WHERE ORDER BY sid, seq#, event, wait_time, p1, p2, p3 v$session_wait_history sid = 154 seq#;

SID SEQ# EVENT WAIT_TIME P1 P2 P3 --- ---- ------------------------ ---------- ------ ------ -----154 1 db file sequential read 28 4 3547 1 154 2 log buffer space 18 0 0 0 154 3 log buffer space 36 0 0 0 154 4 db file sequential read 0 4 3559 1 154 5 db file sequential read 0 4 1272 1 154 6 db file sequential read 0 4 3555 1 154 7 log buffer space 9 0 0 0 154 8 db file sequential read 0 4 3551 1 154 9 db file sequential read 6 4 1268 1 154 10 log buffer space 8 0 0 0

22

v$session
 All columns from v$session_wait have been added to v$session:
Saves us the effort of joining the two views..

 New blocking_session, blocking_session_status columns:

blocking_session shows the SID of the session holding the resource blocking this session. blocking_session_status contains VALID when blocking_session is populated.

23

v$session Example
 Prior to Oracle 10g:
SQL> 2 3 4 5 SELECT s.sid, w.state, w.event, w.seconds_in_wait siw, s.sql_address, s.sql_hash_value hash_value, w.p1, w.p2, w.p3 FROM v$session s, v$session_wait w WHERE s.sid = w.sid AND s.sid = 154;

 Oracle 10g:
SQL> SELECT sid, state, event, seconds_in_wait siw, 2 sql_address, sql_hash_value hash_value, p1, p2, p3 3 FROM v$session 4 WHERE sid = 154;

24

Another Example
 Why is session 154 waiting? And who is blocking session 154?
SQL> SELECT sid, blocking_session, blocking_session_status block_status, 2 username, event, seconds_in_wait siw 3 FROM v$session 4 WHERE sid = 154; BLOCKING SID _SESSION BLOCK_STATUS USERNAME EVENT SIW --- -------- ------------ -------- ------------------------------ --154 157 VALID TSUTTON enq: TX - row lock contention 318

25

v$event_histogram
 New view in Oracle 10g.  Shows instance-wide summary wait event statistics like v$system_event, but provides a wait time histogram for each. Buckets:
Less than 1 mS. 1 mS to 2 mS. 2 mS to 4 mS. 4 mS to 8 mS. ... and so on

26

v$event_histogram
SQL> DESCRIBE v$event_histogram Name Null? ----------------------------------------- -------EVENT# EVENT WAIT_TIME_MILLI WAIT_COUNT Type --------------------------NUMBER VARCHAR2(64) NUMBER NUMBER

27

v$event_histogram Example
 How significant is the row lock contention on our system?
SQL> SELECT event, total_waits, time_waited, average_wait 2 FROM v$system_event 3 WHERE event = 'enq: TX - row lock contention'; EVENT TOTAL_WAITS TIME_WAITED AVERAGE_WAIT ----------------------------- ----------- ----------- -----------enq: TX - row lock contention 17218 2101966 122

 Does the average wait of 1.22 seconds indicated by v$system_event paint an accurate picture?
28

v$event_histogram Example
SQL> SELECT event, wait_time_milli, wait_count 2 FROM v$event_histogram 3 WHERE event = 'enq: TX - row lock contention'; EVENT WAIT_TIME_MILLI WAIT_COUNT ----------------------------- --------------- ---------enq: TX - row lock contention 1 833 enq: TX - row lock contention 2 635 enq: TX - row lock contention 4 372 enq: TX - row lock contention 8 395 enq: TX - row lock contention 16 781 enq: TX - row lock contention 32 3729 enq: TX - row lock contention 64 3050 enq: TX - row lock contention 128 410 enq: TX - row lock contention 256 47 enq: TX - row lock contention 512 46 enq: TX - row lock contention 1024 37 enq: TX - row lock contention 2048 3 enq: TX - row lock contention 4096 6880 29

v$system_wait_class and v$session_wait_class

 New views in Oracle 10g.  Show wait counts and total time waited since instance startup and session start.  Similar to v$system_event and v$session_event, but summarized by wait class.  Times are in centiseconds.

30

v$system_wait_class and v$session_wait_class

SQL> DESCRIBE v$system_wait_class Name Null? ----------------------------------------- -------WAIT_CLASS_ID WAIT_CLASS# WAIT_CLASS TOTAL_WAITS TIME_WAITED SQL> DESCRIBE v$session_wait_class Name Null? ----------------------------------------- -------SID SERIAL# WAIT_CLASS_ID WAIT_CLASS# WAIT_CLASS TOTAL_WAITS TIME_WAITED Type --------------------------NUMBER NUMBER VARCHAR2(64) NUMBER NUMBER

Type --------------------------NUMBER NUMBER NUMBER NUMBER VARCHAR2(64) NUMBER NUMBER 31

v$system_wait_class Example
SQL> SELECT wait_class, time_waited 2 FROM v$system_wait_class 3 ORDER BY time_waited DESC; WAIT_CLASS TIME_WAITED ------------- ----------Idle 777450022 System I/O 1261584 User I/O 116667 Configuration 116481 Application 72301 Other 12432 Commit 3496 Concurrency 319 Network 1

32

Active Session History

 New MMNL background process samples v$session once each second.  Data captured on active sessions is available in v$active_session_history, usually for 1-4+ hours.  Automatic Workload Repository captures one out of every ten samples in its hourly snapshots.

33

Active Session History

SQL> DESCRIBE v$active_session_history Name Null? ----------------------------------------- -------SAMPLE_ID SAMPLE_TIME SESSION_ID SESSION_SERIAL# USER_ID SQL_ID SQL_CHILD_NUMBER SQL_PLAN_HASH_VALUE SQL_OPCODE SERVICE_HASH SESSION_TYPE SESSION_STATE QC_SESSION_ID QC_INSTANCE_ID EVENT EVENT_ID EVENT# SEQ# P1 P2 P3 WAIT_TIME TIME_WAITED CURRENT_OBJ# CURRENT_FILE# CURRENT_BLOCK# PROGRAM MODULE ACTION CLIENT_ID Type --------------------------NUMBER TIMESTAMP(3) NUMBER NUMBER NUMBER VARCHAR2(13) NUMBER NUMBER NUMBER NUMBER VARCHAR2(10) VARCHAR2(7) NUMBER NUMBER VARCHAR2(64) NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER VARCHAR2(48) VARCHAR2(48) VARCHAR2(32) VARCHAR2(64)

34

Active Session History

 v$active_session_history shows detailed wait information, current SQL statement, object/file/block information, and more.  When a wait that was sampled by ASH completes, Oracle fills in the time_waited column in the v$active_session_history row with the actual duration of the wait.

35

ASH is Always On
 You may not have to turn on tracing or begin monitoring v$ views when users report a problem, because ASH is already sampling data.  In some cases you may be able to diagnose and resolve a problem on first detection, even if you learn of the problem after the fact.
You may not need to get users to reproduce the problem.
36

Sampling versus Collecting

 ASH samples v$session once each second. Very different from extended SQL trace, which collects data on every wait and every OCI call.
A session could encounter hundreds of waits in one second, and ASH will only capture data for one of them.

 Use ASH to view aggregate data on many sessions over a period of time.  Dont use ASH to count waits, get maximum wait times, or look at a short time period.
37

ASH Example
SQL> 2 3 4 5 6 7 SELECT DECODE (session_state, 'WAITING', event, NULL) event, session_state, COUNT(*), SUM (time_waited) time_waited FROM v$active_session_history WHERE module = 'ARXENV' AND sample_time > SYSDATE - (2/24) GROUP BY DECODE (session_state, 'WAITING', event, NULL), session_state;

EVENT SESSION_STATE COUNT(*) TIME_WAITED ------------------------------ ------------- -------- ----------ON CPU 124 0 log file sync WAITING 2 52686 db file scattered read WAITING 2 28254 db file sequential read WAITING 1 6059 control file sequential read WAITING 1 9206 SQL*Net break/reset to client WAITING 1 9140 enq: TX - row lock contention WAITING 922 930864016

38

Automatic Workload Repository

 Conceptually similar to Statspack.  Collects snapshots of system-wide performance statistics, plus ASHs sampling of session activity.  You can generate activity reports using awrrpt.sql or Enterprise Manager interface.  Out of the box, Oracle 10g databases collect snapshots hourly and retain data for seven days.

39

AWR Management
 Snapshot data is stored in SYS-owned tables in the SYSAUX tablespace.  Use dbms_workload_repository to:
Collect a snapshot on demand. Purge snapshots. Adjust snapshot interval. Adjust snapshot retention period. Disable AWR snapshot collection.

40

AWR versus Statspack

 AWR benefits:
More tightly integrated into Oracle kernel, reducing snapshot collection overhead. Snapshots include sampling of ASH data. Data collected by AWR is accessible via views with names starting DBA_HIST. DBA_HIST views are documented.

 Statspack has been updated to be 10g aware.

41

Time Model Statistics

 New concept in Oracle 10g.  Two new views provide a more detailed picture of how Oracle spends time:
Separates out background and user processes. Provides data not previously available, such as how much time was spent in PL/SQL execution.

 Times are in microseconds.

42

Time Model Statistics

SQL> DESCRIBE v$sys_time_model Name Null? ---------------------------------------- -------STAT_ID STAT_NAME VALUE SQL> DESCRIBE v$sess_time_model Name Null? ---------------------------------------- -------SID STAT_ID STAT_NAME VALUE Type --------------------------NUMBER VARCHAR2(64) NUMBER

Type --------------------------NUMBER NUMBER VARCHAR2(64) NUMBER

43

Time Model Statistics

 Some important notes about these statistics:
Statistics do not include background processes unless background appears in the statistic name. DB time shows elapsed time spent in Oracle calls. (Basically CPU time plus non-idle waits.)

44

Time Model Statistics Example

SQL> SELECT stat_name, value / 1000000 seconds FROM v$sys_time_model 2 ORDER BY seconds DESC; STAT_NAME SECONDS ------------------------------------------------ ---------DB time 80970.190 sql execute elapsed time 75057.271 DB CPU 44448.628 background elapsed time 29333.160 PL/SQL execution elapsed time 8824.538 background cpu time 5170.311 parse time elapsed 1270.147 hard parse elapsed time 838.068 PL/SQL compilation elapsed time 176.731 sequence load elapsed time 112.334 connection management call elapsed time 44.644 failed parse elapsed time 11.946 hard parse (sharing criteria) elapsed time 5.579 hard parse (bind mismatch) elapsed time 4.610 failed parse (out of shared memory) elapsed time 0.000 Java execution elapsed time 0.000 inbound PL/SQL rpc elapsed time 0.000 45

Tracing Improvements
 Enabling extended SQL trace in another session is now more convenient.  Tracing sessions in a connection pooling or shared server environment is no longer impractical.  New dbms_monitor package offers these two benefits and more.

46

Tracing Prior to Oracle 10g

 Tracing sessions in Oracle 9i and earlier had annoyances:
dbms_support usually missing or not installed. dbms_system.set_ev not officially supported. ALTER SESSION SET EVENTS to set 10046 event has a syntax hard (for some of us) to remember.

47

Easier Tracing in Oracle 10g

 Control tracing of your own session:
EXECUTE dbms_monitor.session_trace_enable (waits=>TRUE, binds=>TRUE); EXECUTE dbms_monitor.session_trace_disable ();

 Control tracing of another session:

EXECUTE dbms_monitor.session_trace_enable (session_id=>153, waits=>TRUE); EXECUTE dbms_monitor.session_trace_disable (session_id=>153);

 Tracing sessions in Oracle 10g is easier because dbms_monitor is provided and installed in every Oracle 10g database by default.
48

Tracing Prior to Oracle 10g

 Straightforward if session being traced uses dedicated server connection.  Tracing session using shared server architecture leads to multiple trace files that have to be merged manually.  Tracing an end-user session that does not map to one Oracle session (eg: connection pooling or multiplexing) is not practical.

49

New Tracing Features in 10g

 Tracing can be enabled for sessions with a specific client identifier:
EXECUTE dbms_session.set_identifier ('client identifier'); EXECUTE dbms_session.clear_identifier ();

 Tracing can be enabled for sessions with specific service / module / action combination:
EXECUTE dbms_application_info.set_module ('module name', 'action name'); EXECUTE dbms_application_info.set_action ('action name');

 Trace data split over multiple trace files can be merged into one trace file for TKPROF processing:
$ trcsess output= [clientid=] [service=] [module=] [action=] [session=] 50

Connection Pooling Example

 Application server connection pooling code:
EXECUTE dbms_session.set_identifier ('session_id from application table'); ...do the work for this end user session... EXECUTE dbms_session.clear_identifier ();

 Trace the end-user session assigned session_id 12345 in the applications sessions table:
SQL> EXECUTE dbms_monitor.client_id_trace_enable > (client_id=>'12345', waits=>TRUE, binds=>TRUE);

 Consolidate trace data into one trace file suitable for use by TKPROF:
$ cd $ORACLE_BASE/admin/$ORACLE_SID/udump $ trcsess output=mytracefile.trc clientid=12345 51

Twelve Oracle 10g Enhancements

More descriptive wait event names Wait event classes v$event_name new columns v$sql / v$sqlarea new columns v$session_wait_history v$session new columns v$event_histogram v$system_wait_class / v$session_wait_class Active Session History (ASH) Automatic Workload Repository (AWR) Time model statistics Improved session tracing
52

Wrapping Up
 Oracle 10g includes many enhancements to the wait event interface that should make performance management using wait event methodologies easier than ever.  Some are just conveniences:
More descriptive event names, wait event classes, dbms_monitor.session_trace_enable...

 Some provide data previously unavailable:

New v$sql columns, time model statistics, v$event_histogram...
53

White Paper
 Contains additional detail we didnt have time to cover today.  Includes a handy reference list of all Oracle 10g wait events, wait classes, and wait event parameters.  Download: www.dbspecialists.com/presentations

54

Contact Information
Terry Sutton and Roger Schrag
Database Specialists, Inc. 388 Market Street, Suite 400 San Francisco, CA 94111 Tel: 415/344-0500 Toll-free: 888/648-0500 www.dbspecialists.com tsutton@dbspecialists.com rschrag@dbspecialists.com
55