Performance Tuning Oracle Database

NIIT Technologies Confidential

.

Performance Tuning
Document Information Author Reviewer Approver Effective Date File

NIIT Technologies

Richard Van Der Leeden

Distribution List Sl. No Distributed to Issue Date Media Issued by Remarks

1.0

NIIT Transition Team, Eurostar Transition Team

Data Room

Change History Versio n# 1.0 Release Date 23-Jun2011 Sections Changed NA

Author NA

Change Description First Release

NIIT Technologies Confidential

Performance Tuning

Table of Contents
1 Purpose..............................................................................................4 2 Audience.............................................................................................4 3 References..........................................................................................4 4 Glossary & Conventions Used...............................................................4 4.1 Acronyms.............................................................................................................4 5 Conventions Used................................................................................5 6 Performance Tuning Approaches for Oracle(8.1.6 - 9.2.0.5) on UNIX .....5 7 Application Tuning ............................................................................7 8 Transition Phase End Sign-off ............................................................14

NIIT Technologies Confidential

Performance Tuning

1 Purpose
The Purpose of this document is to give an overview of finding performance issue in the database and do the necessary steps to overcome those.

2 Audience
The intended audiences of this document are the delivery / project management team and the SME's of Eurostar involved in the Transition process.

3 References

The List of all the referred documents under various sections is given below:

Topic

Link

4 Glossary & Conventions Used

4.1 Acronyms

Acronym KT SME
NIIT Technologies Confidential

Description Knowledge Transfer/Transition Subject Matter Expert

Performance Tuning

5 Conventions Used

All text presented in COURIER NEW font is either a script code or a terminal output.

6 Performance Tuning Approaches for Oracle(8.1.6 9.2.0.5) on UNIX

As a database administrator, you will often be confronted with users who say that the response time on the system is unacceptable. What is unacceptable performance? There are two aspects to this: Quantifiable performance Immeasurable performance like user dissatisfaction.

This note will help identify quantifiable performance problems and methods to solve them. When performance-tuning databases, it is a good to split the problem into parts to facilitate the process. Performance tuning Oracle databases can be divided into three subcategories: Application tuning RDBMS tuning UNIX system tuning

Each of the above categories has a different maximum possible effect on the performance of your ORACLE database. Below is the screenshot from Spotlight Performance Management tool being used in eurostar infrastructure. This document covers all the details to tune the various Performance bottlenecks.

NIIT Technologies Confidential

Performance Tuning

NIIT Technologies Confidential

Performance Tuning

Application Tuning

Application tuning is by far the most effective aspect for database tuning. But the most important issue is to identify what(module/package functionality, etc) is running slow. Tuning your SQL statements is the first and foremost investigation since they intervolve both your application and database. To analyze your SQL statements you can set the EVENT 10046 and then trace the SQL statements: Add the following line to your init.ora file: TIMED_STATISTICS = true Restart the database There are different ways to set the EVENT 10046 If you can run the statement via SQLPLUS then. SQL >alter session set events '10046 trace name context forever, level 8'; SQL > ---run the SQL statement--if the SQL hangs then let this run at least 10 minutes longer than it usually takes SQL > ---If it completes just EXIT OR CTRL +C to stop the query and exit--A trace file will be output in your user_dump_dest (execute a "show parameter user" in SQLPLUS on the server to find the path) . Format this file with TKPROF $ tkprof <event_10046_file> <new_file_name> sys=no explain =<username>/<password> *username password of the user who executed the query Please see NOTE:32951.1 Tkprof Interpretation Use ORADEBUG to trace a another existing session to get a EVENT 10046 trace file Please see Note: 1058210.6 HOW TO ENABLE SQL TRACE FOR ANOTHER SESSION USING ORADEBUG Look at formatted output of trace command and make sure that your SQL statement is using indexes correctly. Please see Note:163563.1 Resolving Query Tuning Issues There are links to many more Notes for Query Tuning as well in the above note RDBMS Tuning The default init.ora database configuration file is inadequate for large databases.The main tools to use to investigate database tuning issues is : STATSPACK
NIIT Technologies Confidential

Performance Tuning
HANGANALYZE and SYSTEMSTATE outputs for hanging issues Notes on Database Tuning available from Metalink STATSPACK ~~~~~~~~~ **Make sure that TIMED_STATISTICS=true in the init.ora To Install STATSPACK: SQL> connect / as sysdba SQL> @?/rdbms/admin/spcreate

You can install STATSPACK objects into its own tablespace to reduce contention. ***Do not install it in the SYSTEM tablespace, create a dedicated tablespace for the statspack schema preferably on DISKS that are not used by other datafiles To gather stats: sqlplus perfstat/perfstat SQL> execute statspack.snap(i_snap_level=>5); wait 20 mins SQL> execute statspack.snap(i_snap_level=>5);

To be able to make comparisons of performance there must be multiple snapshots taken over
a period of time.

A minimum of two snapshots are required before any performance haracteristics of the
application and database can be made.

There are different snap levels as well (see reference Notes below) A good measure is about 20-30 mins apart
To gather a report : SQL> connect perfstat/perfstat SQL> @?/rdbms/admin/spreport

This will ask you beginning and ending snapshot ids as well as a output file name For complete reference on STATSPACK: Note:149121.1 Gathering a StatsPack snapshot Note:149113.1 Installing and Configuring StatsPack Package Note:149124.1 Creating a StatsPack performance report There are also the following specific Notes that may be helpful: Note.228913.1 Systemwide Tuning using STATSPACK Reports

NIIT Technologies Confidential

Performance Tuning

HANGANALYZE and SYSTEMSTATE for 8.1.7 % svrmgrl svrmgr>connect internal for 9i and above % sqlplus "/ as sysdba" ALTER SESSION SET EVENTS 'IMMEDIATE TRACE NAME HANGANALYZE LEVEL 3'; wait 90 seconds ALTER SESSION SET EVENTS 'IMMEDIATE TRACE NAME HANGANALYZE LEVEL 3'; ***EXIT ... then reconnect*** ALTER SESSION SET MAX_DUMP_FILE_SIZE=UNLIMITED; ALTER SESSION SET EVENTS 'IMMEDIATE TRACE NAME SYSTEMSTATE LEVEL 10'; wait 90 seconds ALTER SESSION SET EVENTS 'IMMEDIATE TRACE NAME SYSTEMSTATE LEVEL 10'; wait 90 seconds ALTER SESSION SET EVENTS 'IMMEDIATE TRACE NAME SYSTEMSTATE LEVEL 10';

The output will be in the user_dump_dest directory. To interpret the hanganalyze and systemstate : Please see Note.215858.1 Interpreting HANGANALYZE trace files to diagnose hanging and performance problems Notes On Metalink to help diagnose Performance issues: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Note.223299.1 Top Oracle 9i init.ora Parameters Affecting Performance Note.262946.1 Performance Issues After Increasing Workload Note.223730.1 Automatic PGA Memory Managment in 9i UNIX System Analysis It is useful to subdivide UNIX system analysis into three subcategories: Memory CPU IO

NIIT Technologies Confidential

Performance Tuning

1. MEMORY One of the most common problems when running large numbers of concurrent users on UNIX machines is lack of memory.In this case, a quick review of UNIX memory management is useful to see what effect lack of RAM can have on performance. A UNIX machine has virtual memory: the total addressable memory range.Virtual memory is composed of RAM, DISK and SWAP space.Generally, you will want to have the available SWAP space equal to 2 to 3 times the RAM. How does UNIX use SWAP space? It uses two memory management policies: swapping and paging. Swapping occurs when UNIX transfers an entire process from RAM to a SWAP device. This frees up a large amount of RAM. Paging occurs when UNIX only transfers a "PAGE" of memory to the SWAP device. Only a tiny portion of a process might actually be "paged out" to a SWAP device. While swapping frees up memory - it is slower than paging. Paging generally is more efficient but does not allow for large amounts of memory to be freed simultaneously. Most UNIX systems today use a combination of paging and swapping to manage memory. Generally, you will see the following behaviour: System lightly used: no paging or swapping occurs. System under a medium load: paging occurs as RAM memory runs low System under a very heavy load: paging stops and swapping begins.

When analyzing your UNIX machine, make sure that the machine is not swapping at all and at worst paging lightly. This indicates a system with a healthy amount of memory available. To analyze paging and swapping, use the following commands. Commands used in Berkeley UNIX based systems will be marked as BSD. Commands used in ATT system V will be marked as ATT. - vmstat 5 5 (BSD) procs memory page disk faults cpu

r b w avm fre re at pi po fr de sr d0 s1 d2 s3 in sy cs us sy id 0 0 0 0 1088 0 2 2 0 1 0 0 0 0 0 0 26 72 24 0 1 98

Note: There are NO pageouts (po) occurring on this system. There are also 1088 * 4k pages of free RAM available (4 Meg). It is OK and normal to have page out (po) activity. You should get worried when the number of page ins (pi) starts rising. This indicates that you system is starting to page.

NIIT Technologies Confidential

Performance Tuning

- pstat -s (BSD) 12112k allocated + 3252k reserved = 15364k used, 37280k available

Note: pstat will also give you the amount of RAM and SWAP currently available on your machine.

- sar -wpg 5 5 (ATT) 09:54:29 swpin/s pswin/s swpot/s pswot/s pswch/s atch/s pgin/s ppgin/s pflt/s vflt/s slock/s pgout/s ppgout/s pgfree/s pgscan/s %s5ipf 09:54:34 0.00 0.00 0.00 0.22 0.00 0.0 0.00 0.0 12 0.87

0.22 0.00

0.65 0.00

3.90 0.00

Note: There is absolutely no swapping or paging going on. (swpin,swpot,ppgin,ppgout).

- sar -r 5 5 (ATT) 10:10:22 freemem freeswp 10:10:27 790 5862

This will give you a good indication of how much free swap and RAM you have on your machine. There are 790 pages of memory available and 5862 disk blocks of SWAP available. There are other platform dependent tools to diagnose memory: HPUX --> see Note:166490.1 Diagnosing Oracle memory on HP using GLANCE AIX --> see Note:166491.1 Diagnosing Oracle Memory on AIX using SVMON SUN --> see Note:163763.1 Diagnosing Oracle memory on Sun Solaris using PMAP

NIIT Technologies Confidential

Performance Tuning

CPU Once you have monitored your systems available memory you will want to make sure the the CPU(s)are not being overloaded. Here is some general information about how processes get allocated CPU time. UNIX is a multi-processing operating system. That means that a UNIX machine has to manage and process multiple user processes simultaneously. UNIX does this in the same way that people wait in line to buy groceries. When a process is ready to be processed by a CPU it will be placed on the waiting line or RUN-QUEUE. This is a queue of processes waiting to be run. Obviously there are limits within which one wants to keep the RUN-QUEUE size. Another factor of interest is the percentage of time the CPU spends in user mode, system mode, or idle mode. Some commands that determine whether or not there is a CPU resource problem occurring: - vmstat 5 5 (BSD) procs memory page disk faults cpu

r b w avm fre re at pi po fr de sr d0 s1 d2 s3 in sy cs us sy id 0 0 0 0 1088 0 2 2 0 1 0 0 0 0 0 0 26 72 24 0 1 98 Note: The CPU is spending most of its time in IDLE mode (id). That means that the CPU is not being heavily used at all. There are no processes that are waiting to be run (r), blocked (b), or waiting for IO (w) in the RUN QUEUE. - sar -qu 5 5 (ATT) 10:58:02 runq-sz %runocc swpq-sz %swpocc 10:58:07 2.8 100 %usr 0 %sys 2 %wio %idle 4 94

Note: The CPU is spending most (94%) of its time in idle mode. This CPU is not being heavily used at all. Generally it is alright to have a CPU that is 0% idle as long as the RUN-QUEUE is not too large. In this case the run queue is 100% occupied (there is always a process waiting to be run) and there are an average of 2.8 processes waiting to be run in the RUN-QUEUE. You want to keep the RUN-QUEUE under 5-6 for a single CPU machine. If the run queue gets any larger that those values you will see some performance degradation. Two solutions to this are: Obtain a faster processor Use more CPU's. Avoid overloading your CPU. Response time on your machine will suffer if it is overloaded. Try to keep the run queue 100% occupied and have less that 6 processes waiting to be run for one CPU. This changes as you add more CPU's or a faster CPU. You may also want to avoid the CPU spending most of
NIIT Technologies Confidential

Performance Tuning
its time (more than 50%) in system mode. This may indicate that you are spending too much time in kernel mode servicing interrupts, swapping processes etc. I/O The last step in analyzing your UNIX machine is taking a look at IO. After having looked at SQLDBA monitor to see which datafiles are being used heavily you may also want to take a look at what UNIX says about file IO. These commands are for analyzing file IO on file systems.

- iostat -d 5 5(BSD) sd1 sd3

bps tps msps bps tps msps 1 0 0 .0 4 0 0.0

iostat will display the number of kilobytes transferred per second, the number of transfers per second, and the milliseconds per average seek. In the example above, both SD1 and SD3 are not used heavily at all.BPS rates over 30 indicate heavy usage of a particular disk. If only one disk shows heavy usage, consider moving some of your datafiles off it or striping your data across several disks.

- sar -d 5 5 (ATT) 09:17:20 09:17:26 device %busy avque r+w/s blks/s avwait avserv 1.18 2.66 1.11 1.34 1.64 1.17 9 8 11 27 18 20 107 39.26 512.66 132 31.36 24.10 165 31.95 294.02 518 96.26 219.96 113 43.70 19.58 79 3.73 22.89

iop0/pdisk010 472.45 iop0/pdisk000 18.43 iop0/pdisk020 317.08 iop0/pdisk021 590.88 iop0/pdisk040 34.94 iop0/pdisk041 45.33

Note: The "-d" option reports activity for each block device.The following is an explanation on the output. %busy - percentage of time I/O was busy avque - portion of time device was busy servicing a transfer request r+w/s - average number of requests outstanding during that time blks/s - number of data transfers from or to device, avwait - number of bytes transferred in 512-byte units avserv - average time (in ms) that transfer requests wait idly on queue,and average time to be serviced.
NIIT Technologies Confidential

Performance Tuning
There is a relationship between the number of blocks transferred per second and the average wait time.You can use this to identify which disks are heavily used and which are underutilized.

- sar -b 5 5 (ATT) 15:52:57 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s 15:53:12 0 2 90 1 2 38 0 0

Note: The "-b" option indicates the overall health of the IO subsystem. The %rcache should be greater than 90% and %wcache should be greater than 60%. If this is not the case, your system may be bound by disk IO. The sum of bread, bwrit, pread, and pwrit gives a good indicator of how well your file subsystem is doing. The sum should not be greater than 40 for 2 drives and 60 for 4-8 drives. If you exceed these values, your system may be IO bound. When analyzing disk IO, make sure that you have balanced the load on your system. Here is a "wish" list of steps for designing a disk layout for Oracle: -Make sure that your logfiles and archived logfiles are NOT on the same disk as your datafiles. This is a basic safety precaution against disk failure. -Put your files on raw devices. -Allocate one disk for the User Data Tablespace. -Place Rollback, Index, and System Tablespaces on separate disks. -Consider using Raid level 5 disk striping to stripe your datafiles across separate disks. You can also use the above-mentioned UNIX commands to monitor the IO on your system and identify problem areas. Conclusion: There are many ways to approach Oracle performance issues. A structured approach like the one discussed above will allow the system administrator to systematically analyze her system and identify any problem areas. Once this has been accomplished, measures can be taken to correct problems. Performance is subjective, so find out what is expected.

8 Transition Phase End Sign-off

Transition Phase End Sign Off Prepared by: Name:

Approved by: Name:

NIIT Technologies Confidential

Performance Tuning

(To be completed and returned by Eurostar) Accepted by: Name: Date : / / ___

NIIT Technologies Confidential