You are on page 1of 87

Spotlight on Oracle 7.

5 Getting Started Guide

2010 Quest Software, Inc. ALL RIGHTS RESERVED. This guide contains proprietary information protected by copyright. The software described in this guide is furnished under a software license or nondisclosure agreement. This software may be used or copied only in accordance with the terms of the applicable agreement. No part of this guide may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying and recording for any purpose other than the purchaser's personal use without the written permission of Quest Software, Inc. The information in this document is provided in connection with Quest products. No license, express or implied, by estoppel or otherwise, to any intellectual property right is granted by this document or in connection with the sale of Quest products. EXCEPT AS SET FORTH IN QUEST'S TERMS AND CONDITIONS AS SPECIFIED IN THE LICENSE AGREEMENT FOR THIS PRODUCT, QUEST ASSUMES NO LIABILITY WHATSOEVER AND DISCLAIMS ANY EXPRESS, IMPLIED OR STATUTORY WARRANTY RELATING TO ITS PRODUCTS INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT. IN NO EVENT SHALL QUEST BE LIABLE FOR ANY DIRECT, INDIRECT, CONSEQUENTIAL, PUNITIVE, SPECIAL OR INCIDENTAL DAMAGES (INCLUDING, WITHOUT LIMITATION, DAMAGES FOR LOSS OF PROFITS, BUSINESS INTERRUPTION OR LOSS OF INFORMATION) ARISING OUT OF THE USE OR INABILITY TO USE THIS DOCUMENT, EVEN IF QUEST HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Quest makes no representations or warranties with respect to the accuracy or completeness of the contents of this document and reserves the right to make changes to specifications and product descriptions at any time without notice. Quest does not make any commitment to update the information contained in this document. If you have any questions regarding your potential use of this material, please contact: Quest Software World Headquarters LEGAL Dept 5 Polaris Way Aliso Viejo, CA 92656 www.quest.com email: legal@quest.com Refer to our Web site for regional and international office information. Trademarks: Quest, Quest Software, the Quest Software logo, AccessManager, ActiveRoles, Aelita, Akonix, Benchmark Factory, Big Brother, BridgeAccess, BridgeAutoEscalate, BridgeSearch, BridgeTrak, BusinessInsight, ChangeAuditor, CI Discovery, Defender, DeployDirector, Desktop Authority, Directory Analyzer, Directory Troubleshooter, DS Analyzer, DS Expert, Foglight, GPOADmin, Help Desk Authority, Imceda, IntelliProfile, InTrust, Invirtus, iToken, JClass, JProbe, LeccoTech, LiteSpeed, LiveReorg, LogADmin, MessageStats, Monosphere, NBSpool, NetBase, NetControl, Npulse, NetPro, PassGo, PerformaSure, Point, Click, Done!, Quest vToolkit, Quest vWorkSpace, ReportADmin, RestoreADmin, ScriptLogic, SelfServiceADmin, SharePlex, Sitraka, SmartAlarm, Spotlight, SQL Navigator, SQL Watch, SQLab, Stat, StealthCollect, Storage Horizon, Tag and Follow, Toad, T.O.A.D., Toad World, vAutomator, vConverter, vEcoShell, VESI,vFoglight, vPackager, vRanger, vSpotlight, vStream, vToad, Vintela, Virtual DBA, VizionCore, Vizioncore vAutomation Suite, Vizioncore vEssentials, Vizioncore vWorkflow, WebDefender, Webthority, Xaffire, and XRT are trademarks and registered trademarks of Quest Software, Inc in the United States of America and other countries. Other trademarks and registered trademarks are property of their respective owners.
Spotlight on Oracle Getting Started Guide Updated May 2010

Table of Contents
Introduction ......................................................................................................................................................... 5 Welcome to Spotlight on Oracle ........................................................................................................................ 7
An overview of the Oracle architecture .................................................................................................................... 7 An overview of Spotlight on Oracle .......................................................................................................................... 9 Features of Spotlight .............................................................................................................................................. 11

Getting started with Spotlight on Oracle ....................................................................................................... 15


Installing Spotlight .................................................................................................................................................. 15 Starting Spotlight .................................................................................................................................................... 18 Creating a new Spotlight user ................................................................................................................................ 19 Creating an Oracle database connection ............................................................................................................... 21 Connecting to an Oracle database......................................................................................................................... 26 Installing and using DTrace within Spotlight .......................................................................................................... 27

Using Spotlight on Oracle ................................................................................................................................ 29


The Spotlight on Oracle home page ...................................................................................................................... 29 Spotlight on Oracle alarms ..................................................................................................................................... 37 Spotlight on Oracle drilldowns ................................................................................................................................ 46 Spotlight menus...................................................................................................................................................... 49 Using Spotlight to tune your database ................................................................................................................... 50 Using Spotlight to predict database performance .................................................................................................. 51

Spotlight on Oracle RAC .................................................................................................................................. 57


What is Oracle RAC? ............................................................................................................................................. 57 Installing and authorizing Spotlight on Oracle RAC ............................................................................................... 59

Spotlight on Oracle Getting Started Guide


Starting Spotlight on Oracle RAC ........................................................................................................................... 59 Creating a new Spotlight user ................................................................................................................................ 60 Creating a connection to an Oracle RAC database ............................................................................................... 60 Connecting to an Oracle RAC database ................................................................................................................ 64 Using Spotlight on Oracle RAC .............................................................................................................................. 65

Troubleshooting ................................................................................................................................................ 75
Product authorization errors ................................................................................................................................... 75 Connection problems .............................................................................................................................................. 76 Drilldown problems ................................................................................................................................................. 77 High Spotlight load on database server.................................................................................................................. 79

Appendix: Contact Quest ................................................................................................................................. 81


Contact Quest Support ........................................................................................................................................... 81 Contact Spotlight support ....................................................................................................................................... 81 Contact Quest Software ......................................................................................................................................... 82 About Quest Software, Inc ...................................................................................................................................... 82

Index ................................................................................................................................................................... 83

1
Introduction
This guide provides the basic information you need to start using Spotlight on Oracle to diagnose the performance of your Oracle databases and clusters. You will read about: How Spotlight displays information about your Oracle databases. How to install and start Spotlight on Oracle. How to create a Spotlight user for your database. How to make a Spotlight on Oracle connection to a database. How to use that Spotlight connection. How to make and use a Spotlight connection to an Oracle RAC cluster. How to resolve some common Spotlight problems. How to contact Quest Software for further assistance.

2
Welcome to Spotlight on Oracle
Spotlight on Oracle is Quest Softwares powerful diagnostic and problemresolution tool for Oracle databases. Its unique user interface provides you with an intuitive visual representation of the activity on your database and operating system. This chapter explains the main features of Spotlight on Oracle.

An overview of the Oracle architecture


This section explains how data flows into and out of your databases. This architecture has been used to design the main Spotlight window.

Basic components of an Oracle server


The following diagram illustrates some of the basic components of an Oracle server at the memory, database and disk layers. The numbers indicate the order of the flow of information.

Spotlight on Oracle Getting Started Guide

The numbered labels in the Oracle architecture diagram correspond to the following activities: 1 The client program (for example, SQL*PLUS, Oracle Power Objects, or some other tool) sends a SELECT statement to the server process. 2 The server process looks in the shared pool for a matching SQL statement. If none is found, the server process parses the SQL and inserts the SQL statement into the shared pool. 3 The server process looks in the buffer cache for the data blocks required. If found, the data block must be moved on to the most recently used end of the Least Recently Used (LRU) list. 4 If the block cannot be found in the buffer cache the server process must fetch it from the disk file. This requires a disk I/O. 5 The server process returns the rows retrieved to the client process. This may involve some network or communications delay. 6 When the client issues the UPDATE statement the process of parsing the SQL and retrieving the rows to be updated must occur. The update statement then changes the relevant blocks in shared memory and updates entries in the rollback segment buffers. 7 The update statement also makes an entry in the redo log buffer that records the transaction details.

Chapter 2 Welcome to Spotlight on Oracle 8 The database-writer background process copies modified blocks from the buffer cache to the database files. The Oracle session performing the update does not have to wait for this to occur.

9 When the COMMIT statement is issued the log writer process must copy the contents of the redo log buffer to the redo log file. The COMMIT statement does not return control to the Oracle session issuing the commit until this write is complete. 10 If running in ARCHIVELOG mode, the archiver process copies full redo logs to the archive destination. A redo log is not eligible for re-use until it has been archived. 11 At regular intervals, or when a redo log switch occurs, Oracle performs a checkpoint. A checkpoint requires all modified blocks in the buffer cache to be written to disk. A redo log file cannot be re-used until the checkpoint completes.

Recommended reading
The best information to help you tune and configure Oracle databases can be found online. Use as your first point of reference Web sites such as: The Quest Software Oracle Solutions Web site. The Oracle home page.

An overview of Spotlight on Oracle


Home page
The Spotlight on Oracle home page provides a detailed view of the Oracle database as a whole. The home page helps you locate system bottlenecks quickly. Related server statistics are grouped together on panels that are connected by a series of graphical flows and icons. Spotlight updates these flows in real time so that you can see how quickly data is moving through the system. The icons change color as Spotlight alarms are raised, upgraded, downgraded and canceled.

10

Spotlight on Oracle Getting Started Guide The Spotlight on Oracle home page below has six main panels that reflect the Oracle architecture (see page 7). The panels are: The Sessions panel. The Host panel. The Server Processes panel. The SGA panel. The Background Processes panel. The Disk Storage panel.

For more information on the panels in the Spotlight home page, and on using them to diagnose a specified Oracle database, see Panels on page 30.

Chapter 2 Welcome to Spotlight on Oracle

11

Alarms
More detailed information on situations where the diagnosed Oracle database exceeds acceptable performance thresholds can be viewed in the alarms that Spotlight raises. For more information on Spotlight alarms, see Spotlight on Oracle alarms on page 37.

Drilldowns
More detailed information on the performance of the Oracle database is available in the charts and tables of the various drilldown pages that can be accessed from the Spotlight home page. For more information on Spotlight drilldowns, and on using them to view the detailed performance of a specified Oracle database, see Spotlight on Oracle drilldowns on page 45.

Features of Spotlight
Spotlight is designed for ease of installation and use, and offers a range of tools for problem diagnosis.

Features common to Spotlight applications


Features that are common to most Spotlight applications include: Easy installation. The display of real-time data. Comprehensive information in charts and tables. Visible, audible and configurable alarms. Context-sensitive help. Calibration. Recording and playback of historical data.

12

Spotlight on Oracle Getting Started Guide Common Spotlight technology. Spotlight on Oracle can run within the same Spotlight console as several other Quest Spotlight products.

Features specific to Spotlight on Oracle


Spotlight features that are specific to Spotlight on Oracle include: The ability to add new Spotlight on Oracle users, and to convert the accounts of existing Oracle users so they can use Spotlight on Oracle. Access to a number of other Quest applications for viewing detailed information on the specified Oracle database. For details see Spotlight Menus on page 49, or refer to the online help under Using Spotlight on Oracle to access detailed Oracle information.

Features specific to Spotlight on Oracle RAC


Spotlight features that are specific to Spotlight on Oracle RAC include: The ability to verify connection details for Oracle RAC instances. The ability to open Spotlight on Oracle from Spotlight on Oracle RAC.

Predictive Diagnostics
Predictive Diagnostics allows Spotlight to collect and analyze performance metrics for: SQL statements as they are executed on the database instance. System bottlenecks as they affect database throughput and response time. Database resources as they reach their full capacity and affect database throughput and concurrency. When enough performance data is collected, Spotlight may be able to predict the future performance for individual SQL statements on that database, for potential bottlenecks in the system, and for database CPU, memory, and disk I/O resources.
Note: When you start using Predictive Diagnostics, Spotlight will take about 10 days to collect enough data to make valid predictions on the future performance of the database.

Chapter 2 Welcome to Spotlight on Oracle For more information on Predictive Diagnostics, see Using Spotlight to predict database performance on page 51.

13

DTrace analysis (Solaris 10 and later)


DTrace is a dynamic analysis tool for Sun Microsystems operating systems. It was first introduced in the Solaris 10 operating system, and can be used ONLY on Sun platforms from Solaris 10 onwards. Spotlight on Oracle can use DTrace to capture details of the interaction between Oracle and the operating system when Oracle uses the O/S services provided by Solaris. To use DTrace within Spotlight, you need to install and configure a number of DTrace scripts on the server under investigation, and then enable the DTrace functionality within Spotlight itself. For detailed instructions on how to do this, see Installing and Using DTrace within Spotlight on page 27.

3
Getting started with Spotlight on Oracle
This chapter describes what you need to do to start using Spotlight on Oracle. It explains: How to install Spotlight. How to start Spotlight. How to set up a Spotlight user. How to create a Spotlight on Oracle connection. How to connect to an Oracle database.

Installing Spotlight
To install Spotlight on Oracle, complete the following steps. Close all other applications before loading the CD into your PC or (if you have downloaded Spotlight from the Quest Web site) executing the downloaded Quest installer. You can cancel the installation at any time by pressing the Esc key or by clicking the Cancel button.

Before installing and running Spotlight


Before you install and run Spotlight, you should ensure that the Spotlight installer and desktop user have the appropriate client permissions on the Spotlight client machine.

16

Spotlight on Oracle Getting Started Guide

Note:

The Spotlight application can be installed only on a supported Windows system where the current user has Administrator privileges.

At installation
During installation, Spotlight requires write access to the following folders: The Spotlight installation folder (by default, Program Files\Quest Software\Spotlight) Program Files\Common Files\Quest Shared Documents and Settings\All Users\Application Data\Quest Software\Spotlight Documents and Settings\<current user>\Application Data\Quest Software\Spotlight and to the following Windows registry keys and sub-keys: HKEY_LOCAL_MACHINE\SOFTWARE\Quest Software\Products HKEY_LOCAL_MACHINE\SOFTWARE\Quest Software\Spotlight HKEY_CLASSES_ROOT In many cases, the Spotlight installer will run at elevated privileges in order to perform the required actions. These privileges can specifically be allowed or disallowed by system administrators by means of standard Microsoft Installer (MSI) policy settings.

When running Spotlight


During operation, Spotlight requires write access to the following file locations under the Documents and Settings folder: <current user>\Application Data\Quest Software\Shared <current user>\Application Data\Quest Software\Spotlight You should also verify that you can run the required Oracle client on the Spotlight machine.

When authenticating a Spotlight license


When you install a new Spotlight application or update license details on an existing application, Spotlight requires: Administrator access to the Windows registry

Chapter 3 Getting Started with Spotlight on Oracle

17

Write access to the Console\Licenses sub-folder of the Spotlight installation folder.

To install Spotlight on Oracle


Quest Spotlight installers can be delivered on CD or as downloaded executable files. If installing Spotlight from a CD: 1 Insert the Quest Installer CD into your CD-ROM drive. Depending on your system's configuration, the installation program may start automatically. If this happens, follow the instructions on the screen to complete the installation. 2 If the installation program does not start, select Run from the Windows Start menu and type the name of the installation package. For Spotlight delivered as part of the Toad DBA Suite, type d:\TOAD DBA Suite for Oracle vvv.exe (where d: is the CD-ROM identifier, and vvv is the version), and click OK. For Spotlight delivered as a standalone installer type d:\SpotlightonOracle_750.exe (where d: is the CD-ROM identifier) and click OK. 3 If you have downloaded the Quest Spotlight installer from the Quest Web site, execute the downloaded installer. 4 Follow the instructions displayed on the screen. By default, Spotlight on Oracle is installed into the following directory, though you can choose a different installation path. C:\Program Files\Quest Software\Spotlight
Note: If the database to be diagnosed is running on Windows, and you want to be able to view operating system metrics in Spotlight, you must have permission to read the registry of the database machine. For more information, see Connection problems on page 76.

Multiple installations of Spotlight on Oracle


Although many Spotlight applications can run within a single Spotlight installation, the standalone Spotlight installer will allow you to install and separately upgrade multiple copies of Spotlight on your computer.

18

Spotlight on Oracle Getting Started Guide

MSI installation of Spotlight on Oracle


MSI installers are available for Spotlight on Oracle. These installers can be run silently, and produce an installation that can be captured and re-used via automatic deployment techniques. Contact Quest Software for more information on Spotlight MSI installers.

To authorize Spotlight on Oracle


Spotlight comes with a time-limited trial license for Spotlight on Oracle. Contact your Quest Software representative to obtain the necessary registered keys to ensure Spotlight remains available when the trial key expires. After the trial licenses have expired, Spotlight on Oracle reverts to a preview mode, which limits access to the Spotlight on Oracle home page, with one connection only at a time. To view or modify the Spotlight license after you have installed Spotlight on Oracle: 1 Select About Spotlight from the Help menu. (The Installed Products tab contains license details for all installed Spotlight products.) 2 Select View / change product license under any Spotlight on Oracle entry in the list of installed products. For more information, see Product Authorization Errors on page 75.

To uninstall Spotlight on Oracle


To uninstall Spotlight on Oracle, use the Add/Remove Programs option in the Windows Control Panel in the Start menu that appears when you click the Start button. (For Windows 7 and Windows Vista operating systems Programs and Features has replaced the Add/Remove Programs option.)

Starting Spotlight
In order to successfully connect to a database, the following steps must be completed: 1 Start the Spotlight application.

Chapter 3 Getting Started with Spotlight on Oracle 2 Create a new Spotlight user, if required. 3 Create a Spotlight database connection. 4 Connect to the database.

19

To start Spotlight on Oracle


To start Spotlight on Oracle, click the Spotlight desktop icon or 1 On the Windows task bar, click Start. 2 From the Programs menu, choose Quest Software | Spotlight | Spotlight.

Creating a new Spotlight user


Spotlight on Oracle users require special privileges, views, and tables so they can diagnose your Oracle databases. To grant these privileges, Spotlight invokes the Oracle User Wizard to create and update the required Spotlight users. Use the wizard to: Add new users to the database that is to be diagnosed. Convert the accounts of existing Oracle users so they can use Spotlight on Oracle. Reconfigure existing Spotlight on Oracle users to work with new versions of Spotlight on Oracle (when available).

To create a Spotlight user


1 Start Spotlight on Oracle. 2 From the File menu choose Oracle User Wizard... 3 Click Next. 4 Enter the following details: Enter the SQL*Net connection string Enter the database instance on which you wish to create the user.

20

Spotlight on Oracle Getting Started Guide

Enter the user name of a DBA account

Enter a DBA user name for that database instance (so that the Spotlight user and the relevant objects can be created). The user name requires SYSDBA access. Enter the DBA user password.

Enter the password 5 Click Next to continue.

6 Click Create a new user and enter the following details: User name Password Confirm password Select any table Alter system Enter a name for the new user. Enter a password for the new user. Confirm the password. Select this to enable the user to generate explain plans. Select this to enable the user to trace and terminate sessions.

7 Click Next to continue. 8 Select tablespaces to use for Spotlight tables Temporary segments The default tablespace is USERS, if available. The default temporary tablespace is TEMP, if available.

9 Click Next to continue. 10 You can choose whether to employ Predictive Diagnostics on this database instance. To use Predictive Diagnostics: Select Create/Upgrade Predictive Diagnostics schema, and then click Next. Choose whether to use an existing Predictive Diagnostics schema (recommended) or install a new schema, and then click Next.

Chapter 3 Getting Started with Spotlight on Oracle

21

Choose whether to install a set of sample data (which you can use to experiment with the Predictive Diagnostics feature), and then click Next. Choose the settings that Spotlight will use to schedule the Oracle jobs that collect data on SQL performance and impending bottlenecks, and then click Next. 11 Click Next when the user setup is complete. 12 Click Finish to complete the task.

Creating an Oracle database connection


Note: Before creating the connection, ensure that you have created a Spotlight user (see page 19) that has the special privileges required to diagnose your Oracle databases.

To create a database connection


1 From the File menu choose Connect... 2 Ensure Spotlight on Oracle is selected on the Connections menu. 3 Double-click the Add new connection icon. 4 A Properties window opens for the new connection. The Details page of the Properties window has several sections: Oracle connection details. ASM connection details. This section is available only for Oracle 10g systems or later. Server connection details. ESX connection details. StealthCollect connection details. This section appears only if you already have installed a Quest application (such as Performance Analysis) that supports StealthCollect.

22

Spotlight on Oracle Getting Started Guide 5 Enter the following information in the Details page of the Properties window: Connection String Select the name of the database that you wish to connect to. By default this will be the name of the connection. Enter the Spotlight user name. Enter the Spotlight user password. Select an appropriate Oracle Home a location where Oracle client software is installed. Choose a home that contains a version of the Oracle client that is compatible with the server under diagnosis. 6 (Available only in Oracle 10g and later) Select Monitor ASM if you want to monitor the performance of Oracle's Automated Storage Management (ASM) technology. ASM is built into the Oracle database kernel; it simplifies the creation and space management in Oracle databases. Enter the following details: ASM Connection String Enter the connection string required to link to the ASM database instance that is used for storage management for the Oracle database under investigation.
The TNSNAMES entry for the ASM instance must include the option (UR = A) for the Spotlight connection to work properly.

Oracle Username Oracle Password Oracle Home

Chapter 3 Getting Started with Spotlight on Oracle

23

ASM Password

Enter the user password that Spotlight uses when logging on to the ASM database instance.
There is NO ASM Username box for the ASM connection, as the ASM instance contains only one user SYS that can provide access to ASM-specific information. SYS access to an ASM instance does NOT create a security issue for the Oracle database but still you should limit access to the SYS account to trusted individuals. Because the SYS password is stored in encrypted format, non-privileged users can still access Spotlight on Oracle. To do so: Use a pre-configured or shared Spotlight on Oracle connection, OR Use a Spotlight connection without ASM access. Spotlight will display Oraclerelated data, but the ASM drilldown will contain no data.

7 Select Monitor OS if you want to monitor the operating system on the database server. Enter the following details: OS Server Enter the name of the server that is to be monitored. Use a period . if you wish to monitor the local database. Select the type of operating system to monitor. The supported types are: Unix(REXEC) Unix(SSH) Unix(SSH using Public/Private key) Windows

OS Type

If you select one of the Unix systems, make sure that the relevant SSH or REXEC daemon is running on the database server, and is configured to receive remote connections.

24

Spotlight on Oracle Getting Started Guide

SSH Port
(Enabled only when you choose to connect to the database server via SSH.)

Enter the port number that Spotlight will use for its SSH (secure shell) connection to the Oracle database.
The default value is 22.

OS Username

Type the username you use when you log onto the database server.
If the username and password if required are not completed, you will not be able to view the operating system performance statistics for the database server. (Windows) If your login details have been assigned remote access to the registry of the database server, you do not need to complete the OS Username and OS Password fields. (If your login details have NOT been set up on the database server, you must enter a username and password that has access to the registry of the server.)

SSH Key Type


(Enabled only when you connect to the database server via Unix(SSH using Public/Private key).

Choose the type of key to use when making the Spotlight connection.
Public-key encryption is supported under SSH2 only. The current options are RSA and DSA. For more information, see Public/Private keypairs in the online help.

SSH Private Key Filename


(See SSH Key Type above.)

Type the location of the file that contains the private key for the Spotlight connection, or click the "..." button to locate it. Type the passphrase used to decrypt the private key. Enter the user password (if required).

SSH Passphrase
(See SSH Key Type above.)

OS Password
(Enabled when you connect to the database server via Unix(REXEC), Unix(SSH), or Windows).

Chapter 3 Getting Started with Spotlight on Oracle

25

8 Select the option Connect to OS Spotlight when starting this connection to open the relevant operating-system (OS) Spotlight automatically when you open your Spotlight on Oracle connection. 9 Click the Monitor ESX option if you want to monitor the virtual machine that hosts the database. This option is enabled when Monitor OS is checked. Enter the following details. ESX Web Service ESX Username Type or select the URL of the ESX Web Service that manages the virtual machine. Type the username required to connect to the ESX Web Service.
Note: The username specified must have read only permissions for both the ESX virtual machine and the ESX host that the virtual machine resides on, or may reside on.

ESX Password Virtual Machine Name

Type the password required to connect the specified user to the ESX Web Service. Click ... to select the virtual machine.

10 Click the Use StealthCollect option if you wish to use StealthCollect to collect SQL statements and performance data from the target database.
Note StealthCollect is installed with Performance Analysis. The Use StealthCollect option is available while StealthCollect is installed. StealthCollect is Quest technology designed to collect SQL statements and performance indicators from Oracle databases and host operating systems without imposing overheads on the target database itself.

Enter the following details: SC Port No. Enter the port number where Spotlight can access StealthCollect data.
If required, click the button to enter username and password details for StealthCollect. (The default StealthCollect username and password for a new connection profile is quest.)

SC Oracle SID

Enter the StealthCollect Oracle SID for the target database.

26

Spotlight on Oracle Getting Started Guide

11 Click Test to check that Spotlight is able to connect successfully with the details provided.
Note For connections to virtual machines, testing the connection validates that Spotlight can connect to the Web Service using the specified username and password combination, and that the virtual machine exists. It does not check permissions, that the virtual machine is turned on, or that it is the same machine as the one specified in the Server Connection Details section.

12 Click the Save password box to save all the password details that have been entered. 13 Click OK to finish. A connection icon with the name you specified is created.

Connecting to an Oracle database


Notes: Before connecting to an Oracle database, ensure that you have: Created a Spotlight user (see page 19) that has the special privileges required to diagnose your Oracle database. Created a connection to the Oracle database that you want to diagnose (see page 21).

To connect to an Oracle database


1 From the File menu choose Connect... 2 Double-click the connection icon that represents the connection that you want to open.
Note: If you encounter problems when connecting, see Connection problems on page 76 for more information.

Once you have connected


Spotlight requires some additional information once you have connected.

Chapter 3 Getting Started with Spotlight on Oracle You may need to set the initialization parameter timed_statistics.

27

If you have a trial version of Spotlight, you may need to obtain a full license key to continue the full functionality of Spotlight after the initial trial period.

Setting the initialization parameter


In order for Spotlight to accurately monitor the database the timed_statistics parameter must be set to true in the init.ora file. If it is not set to true for that user, a message will be displayed that gives you the option to have Spotlight set the parameter for you. To set the initialization parameter to true  In the Warning message click Set timed statistics now.
Note: Spotlight can set the parameter only if you have granted the alter system privilege to the Spotlight user.

Obtaining a full license key


When installed under a trial key, Spotlight runs for a limited time with full functionality. You must register the product before the end of the trial period in order to maintain that full functionality. To do so, contact your Quest Software representative for information on obtaining a full license key. For further information see Product authorization errors on page 75.

Installing and using DTrace within Spotlight


DTrace is a dynamic analysis tool for Sun Microsystems operating systems from Solaris 10 onwards. You can enable and use Spotlight's DTrace drilldown to capture details about the interaction between Oracle and Solaris that would otherwise be unavailable. To use DTrace within Spotlight, you need to: 1 Install and configure a number of DTrace scripts on the server under investigation. 2 Enable DTrace functionality within Spotlight. 3 Use the DTrace drilldown to view the DTrace analysis.

28

Spotlight on Oracle Getting Started Guide

To install and configure DTrace scripts


1 Locate the SoO_DTrace_scripts.tar archive package in the Plug-ins\SoO sub-folder of the Spotlight installation folder. 2 Unpack the contents of the archive to a directory on the database server. Ensure that the unpacked files have execute permission. 3 In the .profile of the account that will be used to run O/S commands from Spotlight, define a QUEST_SOO_HOME variable that points to that directory on the server. 4 Provide that user with DTrace authority by adding the following line to /etc/user_attr:
<<username>>::::defaultpriv=basic,dtrace_proc,dtrace_kernel

To enable DTrace within Spotlight


1 From the View | Options menu, choose Spotlight on Oracle. 2 Click the DTrace button. 3 Select Show Solaris DTrace drilldowns. 4 Click OK. The DTrace icon will appear on the Spotlight toolbar, and a DTrace option will appear on the ViewGo To menu.

To view the pages in the DTrace drilldown


On the Spotlight toolbar, click the Solaris DTrace button, or From the View | Go To menu, choose DTrace.

4
Using Spotlight on Oracle
When you use Spotlight on Oracle to diagnose an Oracle database, you can: View the status of the connected database in the panels and dataflows of the Spotlight home page. View the alarms that Spotlight has raised for the connected database. View the detailed performance of the connected database in the various Spotlight drilldowns. Use Spotlight menus and shortcuts to access a number of other Quest applications for viewing detailed information the specified Oracle database.

The Spotlight on Oracle home page


The Spotlight home page shows an overview of the activity in an Oracle database. The display is based on the Oracle architecture diagram shown in An overview of the Oracle architecture on page 7.

30

Spotlight on Oracle Getting Started Guide

Panels
A panel is a visual feature that groups related information and components on the Spotlight home page. Components change color as alarms are raised for the metrics they contain. There are six main panels on the home page for Spotlight on Oracle. They are: The Sessions panel. This indicates the status of the Oracle communications between client and server. The Host panel. This indicates CPU usage, Virtualization Overhead and free physical RAM information. The Server Processes panel. This shows the status of Oracle server processes. These processes perform database activities on behalf of end users, and mediate database connections.

Chapter 4 Using Spotlight on Oracle

31

The SGA panel. This shows details of specific memory areas within the system global area (SGA). The SGA is an area of shared or common process memory that is used to cache frequently used data, SQL statements, procedures and other structures. The Background Processes panel. This displays the following key Oracle background processes: Database Writer (DBWR), Redo Log Writer (LGWR), Recovery Writer (RVWR) and Archiver (ARCH). It also contains a status indicator for Spotlight's Predictive Diagnostics capability (if available). The Disk Storage panel. This represents Oracle database and log files on disk. The panels, and the information they display, are described in more detail below. For a full discussion of the panels, see the Spotlight online help.
Note: To view a tool tip for a specific component in a panel, hover the mouse pointer over the component.

Sessions panel
The Sessions panel indicates the status of the Oracle communications between client and server. Information displayed here includes: The time (in milliseconds) that it takes for the query select user from dual to be submitted, executed and returned. The total number of users connected to the database. The number of users who are currently active. The sum of DB time over all sessions, divided by elapsed time.

Note: The Average Active Sessions gauge is not available for Oracle 9i.

The flows between the Sessions panel and the Server Processes panel represent the rate of data transfer in both directions across the SQL*NET interface.

32

Spotlight on Oracle Getting Started Guide

Host panel
The Host panel shows metrics for the operating system. The information displayed here includes: The total amount of CPU being used. The number of tasks waiting for CPU resources. The percentage of CPU that is unavailable to the virtual machine (if applicable). The available physical RAM. The amount of memory that has been reclaimed from the virtual machine by ESX (if applicable).

Chapter 4 Using Spotlight on Oracle

33

Server Processes panel


The Server Processes panel indicates the status of Oracle server processes that perform database activities on behalf of end users, and mediate database connections. Information displayed here includes: Statistics for the PGA memory allocated by server processes. Dedicated server processes that perform work on behalf of a single client process. Shared or multi-threaded servers (MTS) that perform work on behalf of more than one client process. MTS dispatchers that coordinate the allocation of shared servers to client tasks. Server processes that support parallel execution of queries. The current length of the Parallel Execution Queue. Job queue server processes that run PL/SQL commands submitted to the Oracle job queue via the DBMS_JOB package. The state of the EM agent connection between Oracle Enterprise Manager (if installed) and the database. The number of server processes that are currently active.

Note: The Parallel Exec Queue gauge applies only to Oracle 11gR2.

The flow to the Server Processes panel from the Disk Storage panel represents the rate of direct physical reads from disk. The flows between the Server Processes panel and the SGA panel represent the rates of: Blocks reads from the SGA by all server processes. Block changes in the SGA by all server processes. Redo buffer entries made by all server processes. SQL parse requests per second by all server processes. SQL execution requests per second by all server processes.

34

Spotlight on Oracle Getting Started Guide

SGA panel
The SGA panel shows details of specific memory areas within the system global area (SGA). The SGA is an area of shared or common process memory that is used to cache frequently used data, SQL statements, procedures and other structures. Information displayed here includes: The size of the buffer cache and its components. These hold data that is accessed frequently. The current size of the SGA. The size of the Java Pool, which caches class definitions, Java methods and Java objects. The size of the Large Pool allocation heap, which is used in MTS for session memory. It can be used by parallel execution and backup processes. The size of the redo buffer. This contains redo entries that are eventually written to the redo log. The size of the Shared Pool, which caches SQL statements, PL/SQL programs, object definitions and session memory for MTS sessions. The size and activity of the result cache (Oracle 11g or later). Whether the database is in manual or auto memory management mode (Oracle 10g or later).

The flow between the SGA panel and the Disk Storage panel represents the rate of block reads from disk. The flows between the SGA panel and the Background Processes panel represent the rates of: Block writes by the Database writer (DBWR) process. I/O writes by the Recovery writer (DBWR) process. Redo log block writes by the Redo Log Writer (LGWR) process.

Chapter 4 Using Spotlight on Oracle

35

Background Processes panel


The Background Processes panel displays the following key Oracle background processes: Database Writer (DBWRn). The Recovery Writer (RVWRn). The Log Writer (LGWRn). The Archiver (ARCHn).

Note: The n value in each process icon indicates the number of processes there are of that type. The flows between the Background Processes panel and the Disk Storage panel represent: The rate at which modified blocks are written from the SGA to disk by the DBWR processes. The rate of I/O write operations to the flash recovery area by the RVWR processes. The rate at which redo log entries are written to the redo log files by the LGWR processes. The rate at which the archiver copies from redo log files to archived logs.

The Background Processes panel also contains a status indicator for Spotlight's Predictive Diagnostics capability (if installed). This feature helps Spotlight to use collected data to predict database performance.

36

Spotlight on Oracle Getting Started Guide

Disk Storage panel


The Disk Storage panel represents Oracle database and log files on disk, including: Oracle database files. Size of the flash cache. Oracle flashback recovery files. Oracle redo logs. The archive log destinations.

Chapter 4 Using Spotlight on Oracle

37

Spotlight on Oracle alarms


Spotlight alerts you to problems with your database by issuing an alarm. When an alarm is triggered, Spotlight performs one or more of the following actions: Changes in color or intensity for relevant components. Audible warnings. E-mail notifications. With Spotlight alarms you can: View details of the metric(s) that caused the alarm in a relevant drilldown page. (For more information on drilldowns, see Spotlight on Oracle drilldowns on page 39.) View the help associated with each alarm. View details of the alarm itself in the Alarm Log drilldown. Change the actions that follow the raising of alarms. Enable, disable, or snooze alarms. Filter alarms. Change the thresholds and severities that define alarms. Change alarm notifications. Save alarm details.
Note: Detailed information on all these actions can be found in the Spotlight online help.

The following are brief descriptions of the alarms specifically associated with Spotlight on Oracle. For possible solutions for the problems indicated by these alarms, or for information on how to diagnose problems further, see Spotlight on Oracle alarms in the online help.

Archive Critical Failure alarm


An Archive Critical Failure alarm indicates that the number of working archive destinations for the database instance is about to fall below the minimum number needed to archive Oracle redo logs successfully. When you see this alarm, you should make space in the archive log destination by purging or by backing up to an offline medium any archived files that are not immediately needed.

38

Spotlight on Oracle Getting Started Guide

Archive Destination Failed alarm


Spotlight raises an Archive Destination Failed alarm when at least one of the archive destinations specified for redo log files has become full. See Archive Destination Failure alarm below.

Archive Destination Failure alarm


An Archive Destination Failure alarm when at least one of the archive destinations specified for redo log files is about to become full. When you see this alarm, you should make space in the archive log destination by purging it, or by backing it up to an offline medium.

Array Fetch Size alarm


When more than one row is retrieved from an Oracle database in a single SELECT (an array fetch), the amount of network traffic associated with SELECT operations is radically reduced. The Array Fetch Size alarm is raised when the array fetch size for the database may be too low.

ASM Balance alarm


This alarm indicates that there is unbalanced access to one or more of the disks in the ASM (Automatic Storage Management) disk group that has been allocated to the Oracle instance.

ASM Service Time alarm


This alarm indicates that the average time taken to perform I/O on the ASM instance has exceeded a specified threshold value. (Service time represents the response time of a disk plus any wait time experienced during disk operations.)

ASM Status alarm


When a Spotlight connection is lost, Spotlight automatically tries to re-establish the connection. The ASM Status alarm indicates that Spotlight is re-establishing its connection to the ASM diskgroup for the Oracle instance.

Average Time to Sync a Redo Log Entry alarm


The Average Time to Sync a Redo Log Entry alarm (formerly the Average Redo Write Time alarm) is activated when the time taken to write redo log entries exceeds a threshold. Unlike most other Oracle write I/Os, Oracle sessions must wait for redo log writes to complete before they can continue processing.

Chapter 4 Using Spotlight on Oracle

39

Buffer Busy Wait alarm


The Buffer Busy Wait alarm occurs when a session cannot access a block because it is in use by another session. The two most common causes are insufficient free lists for a table or insufficient rollback segments.

Cache Buffer Chains Latch alarm


The Cache Buffer Chains Latch alarm occurs because contention for the cache buffer chains latch exceeds a threshold.

Cache Buffer LRU Chains Latch alarm


The Cache buffer LRU chains latch alarm occurs because contention for the cache buffer chains latch exceeds a threshold.

Continued Fetch Rate alarm


Oracle sometimes needs to migrate a row in a database from its original block to a new block, or needs to chain an updated row across multiple blocks. These migrated and chained rows cause additional I/O, and the Continued Fetch alarm occurs when the number of these "continued row" fetches exceeds a threshold.

CPU Busy alarm


The CPU Busy alarm occurs when the total CPU utilization of the system exceeds a threshold.

CPU Virtualization Overhead alarm


The CPU Virtualization Overhead alarm is activated when Virtualization Overhead is greater than a threshold and CPU Usage is 95% or higher.

Datafile Read Time alarm


The Datafile Read Time alarm occurs if the average time for a random datafile read exceeds a given threshold.

Dispatchers Busy alarm


The Dispatchers Busy alarm occurs when all or most dispatcher processes are busy. In an MTS environment, multiple sessions share a smaller number of shared server processes. The servers are usually allocated by a smaller number of dispatcher processes.

40

Spotlight on Oracle Getting Started Guide

EM Agent Not Connected alarm


Oracle Enterprise Manager (where installed) manages Oracle components and administers Oracle databases. OEM communicates with the databases it administers via a set of EM agent processes. The EM Agent alarm indicates that OEM is installed and active, but not connected to the database.

Estimated Shared Pool Size alarm


This alarm occurs when the Shared Pool is set too small. A possible cause is the use of literals rather than bind variables in SQL statements. You can also increase the size of the Shared Pool.

Excessive RBS Activities alarm


The Excessive RBS Activities alarm occurs when there are high rollback extent/shrink/wrap activities.

Flashback Buffer Wait Percentage alarm


This alarm indicates that the percentage of available time spent waiting for the Recovery Writer (RVWR) to write data to the flashback logs has exceeded a specified threshold.

Flash Cache Insert Skip: DBWR Overloaded alarm


The Flash Cache Insert Skip: DBWR Overloaded alarm is raised when the ratio of insert skips from the DBWR being overloaded exceeds a threshold.

Flash Cache Read Latency


The Flash Cache Read Latency alarm is raised when the time taken to read a single bock from flash cache exceeds a threshold. This may be caused by a low performance flash device.

Flash Cache Write Complete Waits


The Flash Cache Write Complete Waits alarm is raised when the ratio of 'write complete waits: flash cache' to DB time exceeds a threshold.

Flashback Buffer Wait alarm


The Flashback Buffer Wait alarm indicates that the percentage of available time spent waiting for the Recovery Writer (RVWR) to write data to the flashback logs has exceeded a specified threshold.

Chapter 4 Using Spotlight on Oracle

41

Flashback Space Usage alarm


This alarm is raised when the amount of used space in the Flashback Recovery Area exceeds a threshold.

Free Buffer Waits alarm


The Free Buffer Waits alarm occurs when free buffer waits (as a proportion of total waits) exceeds a threshold.

Instance Failed alarm


Spotlight raises an Instance Failed alarm when a specified instance has failed because of an Archiver error. See Archive Critical Failure alarm above.

Insufficient Flashback Logs alarm


This alarm fires when the flashback logs in the flashback area do not cover the amount of time specified in the parameter DB_FLASHBACK_RETENTION_TARGET.

Job Processes Busy alarm


The Job Processes Busy alarm occurs when all or most job queue processes are busy.

Large Pool Full alarm


This alarm fires when the Large Pool allocation heap is full. The Large Pool is used in MTS for session memory, and can be used by parallel execution and backup processes.

Latch Free Waits alarm


If a session needs to acquire a latch that is held by another session, a latch free wait may occur. The presence of latch free waits may indicate a bottleneck within the SGA. (The specific action depends on the latch.)

Library Cache Latches alarm


This alarm fires when the number of library cache latches or library cache pin latches exceeds a threshold. Library cache latches protect the cached SQL statements and object definitions held in the library cache within the Shared Pool.

42

Spotlight on Oracle Getting Started Guide

Library Cache Miss Rate alarm


The Library Cache Miss Rate alarm occurs when the library cache hit ratio falls below a given threshold.

Lock Wait alarm


The Lock Wait alarm occurs when the proportion of time sessions spend waiting for locks exceeds a threshold.

Log Buffer Space Wait alarm


The Log Buffer Space Wait alarm occurs if the amount of time spent waiting for space in the redo log buffer exceeds a threshold.

Log Switch Time alarm


The Log Switch Time alarm occurs when the amount of time spent waiting for log switch events exceeds a threshold.

Low Free Physical RAM alarm


The Low Free Physical RAM alarm occurs when the servers available RAM is low.

Maximum Queued Time alarm


The Maximum Queued Time alarm becomes active when a session has been in the parallel execution queue for longer than the specified threshold.
Note: Applies only to Oracle 11gR2.

Multi-threaded Server alarm


The Multi-threaded Server alarm occurs when all or most multi-threaded server (MTS) or dispatcher processes are busy.

Non-reclaimable Flashback Space alarm


This alarm is raised when the amount of non-reclaimable used space in the Recovery Area (flashback area) exceeds a threshold. Non-reclaimable files include archived redo logs, which must be backed up before being deleted.

Chapter 4 Using Spotlight on Oracle

43

Oracle Memory on Disk alarm


The Oracle Memory on Disk alarm is activated when the amount of guest physical memory mapped to machine memory is less than the current combined size of Oracle's SGA and PGA.

Oracle Parameters for Predictive Diagnostics alarm


The Predictive Diagnostics feature cannot predict the behavior of Oracle databases without the access to Oracle data that is granted by several Oracle parameters.

Oracle Status alarm


When a Spotlight connection is lost, Spotlight automatically tries to re-establish the connection. The Oracle Status alarm indicates that Spotlight is re-establishing its connection to the Oracle instance.

Parallel Query Server alarms


The Parallel Query Server alarm occurs when all or most parallel servers are busy. Parallel server processes perform work to resolve SQL statements that are being executed in parallel. If all parallel server processes are busy, parallel SQL may be downgraded, or even executed serially, thus increasing its execution time. Spotlight can raise these Parallel Query Server alarms: Parallel Query Servers Busy, Parallel Queries Downgraded, and Parallel Queries Serialized.

Parse Ratio alarm


The Parse Ratio alarm is raised when the ratio of parse calls to execute calls exceeds a threshold.
Note This alarm is initially disabled; see Total Parse Wait Time Ratio alarm.

Predictive Diagnostics alarms


A Predictive Diagnostics alarm (SQL Collector or Bottleneck Collector alarm) is raised when the appropriate collection of predictive data has not been scheduled.

Predictive Diagnostics Custom SQL Error alarm


This alarm indicates that the customized SQL query use to represent database concurrency within Predictive Diagnostics has returned an error.

44

Spotlight on Oracle Getting Started Guide

Redo Allocation and Redo Copy Latch alarms


The Redo Allocation and Redo Copy Latch alarms are raised when there is contention for the redo allocation latch and redo copy latch.

Result Cache Find/Create Ratio alarm


When Oracle executes SQL queries, the result cache stores the result sets of specified queries in shared memory for later use. The Find/Create Ratio alarm occurs when the rate of find operations on the result cache drops below the rate of create operations, and indicates that the contents of the result cache are under-used.

Result Cache Latch Sleep Ratio alarm


Spotlight raises the Result Cache Latch Sleep Ratio alarm when the rate of latch sleeps for the Result Cache latch indicates that there is contention on the Result Cache latch, where multiple processes are competing to use the latch and are causing a significant degradation in response time.

Sequential Read Time alarm


This alarm fires when the average time that Oracle takes to read a single block of data typically an index-based lookup is significantly higher than the average time recorded during calibration.

Sessions Queued alarm


The Sessions Queued alarm becomes active when the number of sessions in the parallel execution queue exceeds a threshold.
Note Applies only to Oracle 11gR2.

Shared Pool Lock Percentage alarm


This alarm is raised when one user session holds a lock on object(s) in the Shared Pool for a prolonged period, and causes other sessions to wait on the same resource. This "blocking lock" happens rarely. When it does occur, use a SQL query to find the affected sessions.

Shared Server Converted into Dedicated Server alarm


Spotlight raises an alarm when many shared servers are converted, leaving too few shared servers to meet the demand. Converted shared servers will have a status of WAIT(RECEIVE) in v$shared_server.

Chapter 4 Using Spotlight on Oracle

45

Temporary IO Wait Ratio alarm


Spotlight raises the Temporary IO Wait Ratio alarm when the time spent in temporary IO waits is too high as a percentage of the time spent performing user calls on the Oracle database.

Total Parse Wait Time Ratio alarm


Spotlight raises this alarm when the amount of time spent parsing SQL statements as a percentage of the active database time exceeds a level that may cause latch contention on the shared pool and library cache latches.
Note: The Total Parse Wait Time Ratio alarm is available for Oracle 10.2 and later. When monitoring earlier Oracle systems, consider reactivating the (disabled) Parse Ratio alarm.

Unarchived Logs alarm


The Unarchived Logs alarm occurs when the number of unarchived redo logs exceeds a threshold. If all online logs need archiving, database update activity can be suspended while the redo log writer waits for the next log to be archived.

Unix Status alarm


When a Spotlight connection is lost, Spotlight automatically attempts to re-establish the connection. Where the Oracle database is hosted on a Unix server, this alarm indicates that Spotlight is re-establishing its connection to the Unix server.

WMI Status alarm


When a Spotlight connection is lost, Spotlight automatically attempts to re-establish the connection. Where the Oracle database is hosted on a Windows server, this alarm indicates that Spotlight is re-establishing its WMI connection to the Windows server.

Write Complete Wait alarm


Spotlight raises the a Write Complete Wait alarm when write complete waits (as a proportion of total waits) exceed a threshold. Write complete waits occur when a session tries to modify a block that is currently being written to disk by the database writer process.

46

Spotlight on Oracle Getting Started Guide

Spotlight on Oracle drilldowns


Spotlight on Oracle makes it possible for you to view your system at different levels of detail. At its highest level, which is the Spotlight home page, Spotlight displays a visual representation of the status of the major components in the current system. The detail at this level is designed to help you locate and identify bottlenecks quickly. When you have isolated a problem, you can display a drilldown page, whose charts and tables provide a detailed breakdown of the underlying statistics.

To open a Spotlight drilldown


To access a specified drilldown within Spotlight: Click the relevant drilldown button on the Spotlight toolbar OR Choose the relevant drilldown from the View | Go To menu OR Right-click on a component on the Spotlight home page, and choose an item from the shortcut menu's Show Details sub-menu OR Click on a component on the Spotlight home page OR Use a keyboard shortcut.

List of drilldowns in Spotlight on Oracle


Spotlight on Oracle contains the following drilldowns. Drilldown Sessions Description The Sessions drilldown displays information about the users connected to the Oracle database. The pages and sub-pages in the drilldown display information on: All users connected to the Oracle database. Session details for those users, including waits, SQL statements executed, locks, resource usage, and Oracle statistics. Parallel queries executed for all users connected to

Chapter 4 Using Spotlight on Oracle Drilldown Description the Oracle database. Oracle jobs currently scheduled for the database. SQL & Application Workload

47

The SQL & Application Workload drilldown allows you to select, sort and display SQL statements executed according to specified criteria. The drilldown includes: The Top SQL page, which displays the SQL statements that consume the most system resources for the current connection. The Array Fetch page, which displays information on array processing for the Oracle database under investigation. The Parse Activity page, which displays information on SQL statements associated with high parse rates (Oracle 10.2 and later). The Sort Activity page, which displays information on SQL statements associated with high sort activity (Oracle 10.2 and later). The Analyze Trace page (formerly a separate drilldown), which displays the contents of trace files that store information on how Oracle processes the execution of SQL statements, and on the resources it uses to do so (Oracle 10.2 and later). The Result Cache page (Oracle 11g and later) shows information about the result cache, which stores the result sets of specified queries so they can be re-used.

Activity

The Activity drilldown displays the activity on the database. The pages and sub-pages in the drilldown display information on waits, waiting events, blocking locks, lock activity, latches, query servers, transactions, rollbacks, and buffer busy waits. The I/O drilldown displays the input and output activity for the database. The pages and sub-pages in the drilldown display information on I/O by category,

I/O

48

Spotlight on Oracle Getting Started Guide

Drilldown

Description session, SQL, tablespaces, datafiles, segments and row chaining. The Configuration and Memory drilldown shows the memory utilization for the Shared Pool and the SGA, as well as shared pool and buffer cache hit ratios, Oracle alert log and Oracle parameters. The Operating System drilldown provides information about the performance, processes, and storage on the database machine under investigation. The Disk Storage drilldown displays information about online redo logs, tablespace utilization, archive destinations, and Flashback recovery. You can use the Tuning drilldown to view information about Oracle memory resources in the Oracle SGA and PGA (System and Program Global Areas), to tune memory resources where feasible, and to view and tune latch activity for systems with multiple CPUs. If your system uses Automatic Storage Management (ASM) to manage disk storage, you can use the ASM drilldown within Spotlight on Oracle to diagnose ASM performance. ASM is a storage management solution from Oracle Corporation that automatically reorganizes data based on statistics collected from usage and user access patterns over time. DTrace is a dynamic analysis tool that can be used on Sun platforms from Solaris 10 onwards. Spotlight on Oracle uses DTrace to capture details of the interaction between Oracle and the operating system when Oracle uses the O/S services provided by Solaris. For detailed instructions on displaying and using the DTrace drilldown, see Installing and using DTrace within Spotlight on page 27 Use the Predictive Diagnostics home page to identify: SQL statements ("degrading SQL") whose performance may not scale adequately in the future as data volumes and SQL execution rates increase.

Configuration & Memory

Operating System

Disk Storage

Tuning

ASM (If available)

DTrace (If available)

Predictive Diagnostics (If available)

Chapter 4 Using Spotlight on Oracle Drilldown Description Waiting events ("bottlenecks") that may in the future affect database throughput and response time. Database resources (CPU, memory, and disk I/O) whose limitations may in the future affect database performance.

49

When you start using Predictive Diagnostics, Spotlight will take about 10 days to collect enough data to make valid predictions on the future performance of the database. For more information Predictive Diagnostics, refer to the online help under Spotlight on Oracle (Professional) Predictive Diagnostics. Virtualization Use the Virtual Machine page to see detailed information about the CPU and memory use of a virtual machine. The pages in the Alarm Log drilldown display information on the alarms associated with the current Spotlight connection. You can filter the Alarm Log to display only the alarms that meet specified criteria.

Alarm Log

Spotlight menus
The Spotlight menus are common to all editions of Spotlight and are explained in the online help under Spotlight Basics. The exceptions to this within Spotlight on Oracle are: The File menu, whose Oracle User Wizard option provides the ability to add new Spotlight on Oracle users, and to convert the accounts of existing Oracle users so they can use Spotlight on Oracle. The Tools menu, which contains links to tools that may help you solve the performance issues that Spotlight highlights.

50

Spotlight on Oracle Getting Started Guide

Tools menu options


Explain Plan: The Explain Plan allows you to determine the execution plan Oracle applies to a particular SQL statement. Spotlight allows you to view graphical representations of the execution plan for SQL statements being executed by a user or those identified by the Top SQL drilldown. SQL Optimizer: SQL Optimizer (if installed) provides context-sensitive tuning advice for SQL statements based on the Oracle execution plan and the database structure.
Note: You must have DBA privileges for the database you are diagnosing.

Space Manager: Space Manager (if installed) provides a comprehensive solution for space management and reorganization. Space Manager offers preventive maintenance, problem detection and resolution and capacity planning across any number of databases. You can also launch Space Manager by right-clicking on the Disk Storage icon on the main page or by right-clicking the tables in the Disk Storage drilldown
Note: You must have DBA privileges for the database you are diagnosing.

Using Spotlight to tune your database


Spotlight supports an approach to Oracle performance tuning that could be described as tuning by bottleneck. Spotlight alerts you if any component of the Oracle architecture is forming a bottleneck. Additionally, Spotlight provides tools that allow you to identify and rectify any inefficiency in your databases configuration. In general, you tune your Oracle instance by reiteratively identifying bottlenecks, contention and critical resources, and by using Spotlight's online advice to remove the bottleneck or contention to improve the performance of the resource. The following is a summary of steps that you can undertake to tune Oracle using Spotlight. Use the Spotlight home page (see page 29) to alert you to any obvious bottlenecks. If Spotlight detects that some component of Oracle constitutes a performance bottleneck, the corresponding component generates a visual or auditory alarm. The exact appearance of the alarm depends on how you have configured alarm severities. When an alarm is current you can press F1 to

Chapter 4 Using Spotlight on Oracle

51

display help for the alarm. Clicking the object (in the main window) displays the relevant drilldown. Use the Activity drilldown (see page 51) to display overall efficiency and resource usage. In particular, the Event waits chart shows the amount of time database sessions are spending waiting for various resources, as well as the amount of CPU being utilized. Tuning efforts are most likely to succeed if they are concentrated on the resources being used most heavily. The Wait activity topic contains a discussion of the meaning and implications of various wait events. Other pages help you examine detailed activity. For instance, the Top Sessions drilldown shows the users who contribute most to database activity. You can display various details for these sessions, including the SQL statements, locks and resource usage. For further information on tuning your database, see the online help.

Using Spotlight to predict database performance


Use Predictive Diagnostics to predict the future performance of a database (and identify how to improve that performance) by: Identifying SQL statements whose performance may not scale adequately as data volumes and SQL execution rates increase. Identifying bottlenecks that may in the future affect database throughput and response time. Predicting when database resources such as CPU, memory and disk I/O are expected to reach their full capacity, and recommending strategies to enable the database to handle high loads at those times. Access these features through the Predictive Diagnostics home page. For information on the storage requirements of Predictive Diagnostics, see Storage requirements for Predictive Diagnostics.

52

Spotlight on Oracle Getting Started Guide

The page contains several sections What To Do Next, Find Impending Problems, Performance Modeling, and Configuration and a set of images and hyperlinks that indicate a basic workflow through the predictive process.

How to use Predictive Diagnostics


1 Open the Predictive Diagnostics home page in one of several ways. On the toolbar Keyboard shortcut From the View | Go To menu From the Spotlight on Oracle home page Click Predictive Diagnostics. Press CTRL+G. Select Predictive Diagnostics. Click Predictive Diagnostics on the Background Processes panel, and in the help window click Show me... Predictive Diagnostics.

Chapter 4 Using Spotlight on Oracle

53

2 Click a hyperlink or image in the Predictive Diagnostics home page to open the corresponding diagnostic page. Although you can use any feature in Predictive Diagnostics at any time, we recommend the following procedure for first-time users. Follow the suggestions in What To Do Next. The quality of Predictive Diagnostics predictions depends on the quality of the data on which they are based. What To Do Next tracks the current status of the data and indicates when that data is incomplete or outdated. This usually occurs about 10 days after you begin monitoring the database. Before that time, become familiar with how Predictive Diagnostics works by using the supplied sample data. Employ the Identify Degrading SQL and Identify Database Bottlenecks features to optimize data processing on the host system.
Note: Any changes you make MAY cause the data you collected earlier to become outdated. If so, you may need to collect new data for another 10 days.

Wait until enough real data is available.

Improve how the database is used now.

Use the Configuration pages to identify the resources available to the database.

Set parameters for the CPU, memory and disk I/O resources available to the database. In particular, use Peak Processing Periods to identify WHEN database resources are in greatest demand. This will improve the precision of your predictions.
The default settings we provide for these parameters may not be appropriate for your database.

Use the Performance Modeling pages.

Predict when and why database resources will run out, what will cause performance to suffer, and what you need to do to counter those problems.

54

Spotlight on Oracle Getting Started Guide

Use Predictive Diagnostics with other Spotlight features


When Predictive Diagnostics identifies a SQL statement that may cause problems in the future, you can: Open the Top SQL drilldown to view further details about the SQL statement. Open Explain Plan to view further information about the statement's execution plan. Open SQL Optimizer to view context-sensitive tuning advice for the SQL statement based on the Oracle execution plan and the structure of the database. When Predictive Diagnostics identifies a bottleneck that may cause problems in the future, you can: Open the Waiting Events page in the Activity drilldown to view further details about the current behavior of potential bottlenecks. When Predictive Diagnostics identifies a database resource whose limitations may cause problems in the future, you can: Open the I/O, Configuration & Memory, and Tuning drilldowns to view further details about the current behavior of CPU, memory, and disk I/O resources. For detailed information on Predictive Diagnostics, see the Spotlight online help under Spotlight on Oracle (Professional)Predictive Diagnostics.

Chapter 4 Using Spotlight on Oracle

55

Storage requirements for Predictive Diagnostics


The disk storage requirements for Predictive Diagnostics schemas comprise: For sample data For SQL statements The Predictive Diagnostics sample data requires approximately 70 MB of storage space. A fixed overhead of approximately 10 MB per schema. An additional requirement of approximately 1 KB per snapshot for each SQL statement. The number of individual snapshots collected depend on the application workload and size of the shared pool. The storage requirements also depend on the collection interval for predictive snapshots, and the length of time that each snapshot is retained. Example If there is a fixed overhead of 10 MB, and there are 500 eligible SQL statements per snapshot, and snapshots are collected twice a day, and each snapshot is kept for one year, the disk storage requirement for SQL statements is: 10MB + (500 x .001MB x 2 x 365) = 375 MB Disk storage requirements depend on how frequently data is collected, and the length of time that snapshots are retained. Examples If data is collected every 10 minutes and retained for six months, the disk storage requirement for impending bottlenecks is approximately 200 MB. If data is collected every 30 minutes and retained for six months, the disk storage requirement for impending bottlenecks is approximately 65 MB. If data is collected every 60 minutes and retained for six months, the disk storage requirement for impending bottlenecks is approximately 35 MB.

For impending bottlenecks

56

Spotlight on Oracle Getting Started Guide

For performance modeling

Disk storage requirements depend on how frequently data is collected, and the length of time that performance data is retained. Example If data is collected every 10 minutes and retained for one year, the disk storage requirement for performance modeling is approximately 130 MB.

Note

Parameters set in Predictive Diagnostics Options and the SQL Statement Trend Analysis wizard also affect the storage requirements for Predictive Diagnostics data. See the online help more for more information.

5
Spotlight on Oracle RAC
Spotlight on Oracle RAC is an optional add-on to Spotlight on Oracle. Spotlight on Oracle RAC is installed automatically on your system via the Spotlight on Oracle or TOAD DBA Suite installer.
Note: The detailed procedures for installing and authorizing Spotlight are covered in Chapter 3, Getting Started with Spotlight on Oracle.

Use Spotlight on Oracle RAC to diagnose Oracle databases on multiple host machines under the Real Application Clusters (RAC) architecture. When monitoring the behavior of an Oracle RAC cluster, a database administrator needs to know: 1 How is each of the instances in my cluster performing? 2 How is the database as a whole performing? 3 Is my database experiencing cluster-related contention? To further investigate the behavior of individual database instances within the cluster, the database administrator can use Spotlight on Oracle RAC to launch Spotlight on Oracle on any particular instance within a cluster. For more information, see Viewing Oracle RAC instances with Spotlight on Oracle on page 73.

What is Oracle RAC?


Real Application Clusters (RAC) is a clustering technology introduced by Oracle in release 9i. It supersedes Oracles previous clustering technology Oracle Parallel Server (OPS). Oracle RAC is a shared disk clustering solution that is significantly different from the architectures offered by Microsoft SQL Server and IBM DB2. Each node in an

58

Spotlight on Oracle Getting Started Guide Oracle RAC cluster has equal access to all of the database data through a shared disk subsystem. Data is neither partitioned to specific nodes, nor replicated across nodes. A high-speed network interconnect allows each node to keep its in-memory view of data consistent. An Oracle RAC configuration has the following architectural characteristics: 1 Many instances of Oracle run across several nodes. 2 Many instances share a single physical copy of a shared Oracle database. 3 All instances have common data and control files. 4 Every instance has individual redo logs and undo segments. 5 Every instance can simultaneously execute transactions against the same database. The following diagram illustrates the basic components of an Oracle RAC cluster.

A user should find an Oracle RAC database operationally identical to a database hosted on a single server. From top to bottom, the important features represented are:

Chapter 5 Spotlight on Oracle RAC Cluster interconnect: A high-speed, high-bandwidth communication facility that connects the nodes in an Oracle RAC cluster. Multiple nodes/instances: Oracle instances run on the nodes (host machines) in the Oracle RAC cluster. Each instance comprises an Oracle System Global Area (SGA) plus the corresponding Oracle background processes that retain and process Oracle database requests.

59

Shared disk subsystem: Database files in Oracle RAC systems are stored on multiple disks that are shared by all the nodes in the cluster, and all nodes must be able read and write to those disks.

Installing and authorizing Spotlight on Oracle RAC


Spotlight on Oracle RAC is installed automatically on your system via the Spotlight on Oracle or TOAD DBA Suite installer. The detailed procedures for installing and authorizing Spotlight are covered in Chapter 3, Getting Started with Spotlight on Oracle.
Note: Before installing Spotlight, you should review the normal installation requirements for the products, which include permissions required when installing and running Spotlight, and those required to authorize the Spotlight license.

Starting Spotlight on Oracle RAC


As with the standard version of Spotlight on Oracle, users of Spotlight on Oracle RAC must complete the following steps in order to connect successfully to a database: 1 Start the Spotlight application. 2 Create a new Spotlight user, if required. 3 Create a Spotlight database connection. 4 Connect to the database.

60

Spotlight on Oracle Getting Started Guide The procedure for starting Spotlight on Oracle RAC is the same as that for the standard version of Spotlight on Oracle. This is covered in Chapter 3, Getting Started with Spotlight on Oracle.

Creating a new Spotlight user


As with the standard version of Spotlight on Oracle, users of Spotlight on Oracle RAC require special privileges, views, and tables so they can diagnose your Oracle RAC databases. These privileges are the same as those required by Spotlight on Oracle, and so Spotlight invokes the Oracle User Wizard to create and update the required Spotlight users. The detailed procedure is covered in Chapter 3, Getting Started with Spotlight on Oracle.

Creating a connection to an Oracle RAC database


The procedure for creating a connection in Spotlight on Oracle RAC is similar (but not identical) to the equivalent procedure in the standard version of Spotlight on Oracle. It is described in detail below.
Note: Before creating the connection, ensure that you have created a Spotlight user (see page 19) that has the special privileges required to diagnose your Oracle RAC databases.

To create a database connection


1 From the File menu choose Connect... 2 Select Spotlight on Oracle RAC from the Connections menu. 3 Double-click the Add new connection icon. 4 A Properties window opens for the new connection. The Details page of the Properties window has several sections:

Chapter 5 Spotlight on Oracle RAC

61

Connection String

Enter the name (the SQL*Net alias) of the Oracle RAC database that you wish to connect to. The database name is defined within SQL*Net configuration utilities.
While this can be the name of any instance in the Oracle RAC cluster, we recommend using the name of the load-balanced instance the instance whose TNSNAMES entry includes the option (LOAD_BALANCE = yes)

Oracle Username

Enter the user name that Spotlight will use when logging on to the Oracle RAC database. Enter the user password that Spotlight will use when logging on to the Oracle RAC database. From the Oracle Home list, select an appropriate Oracle Home a location where Oracle software is installed.
Choose a home that contains a version of the Oracle client that is compatible with the server under diagnosis.

Oracle Password

Oracle Home

5 (Available only in Oracle 10g and later) Select Monitor ASM if you want to monitor the performance of Oracle's Automated Storage Management (ASM) technology. ASM is built into the Oracle database kernel; it simplifies the creation and space management in Oracle RAC databases. Enter the following details: ASM Connection String Enter the connection string required to link to the ASM database instance used for storage management for the Oracle RAC database under investigation.
The TNSNAMES entry for the ASM instance must include the option (UR = A) for the Spotlight connection to work properly.

ASM Password

Enter the user password that Spotlight uses when logging on to the ASM database instance.
There is NO ASM Username box for the ASM connection, as the ASM instance contains only

62

Spotlight on Oracle Getting Started Guide


one user SYS that can provide access to ASM-specific information. SYS access to an ASM instance does NOT create a security issue for the Oracle RAC database but still you should limit access to the SYS account to trusted individuals. Because the SYS password is stored in encrypted format, non-privileged users can still access Spotlight on Oracle RAC. To do so: Use a pre-configured or shared Spotlight on Oracle connection. Use a Spotlight connection without ASM access. Spotlight will display Oraclerelated data, but the ASM drilldown will contain no data.

6 Select Monitor OS if you want to monitor the operating system. Enter the following details: OS Type Select the type of operating system to monitor. The supported types are: Unix(REXEC) Unix(SSH) Unix(SSH using Public/Private key), or Windows

If you select one of the Unix systems, make sure that the relevant SSH or REXEC daemon is running on each node of the database cluster, and is configured to receive remote connections.

SSH Port
(Enabled only when you choose to connect to the database server via SSH.)

Enter the port number that Spotlight will use for its SSH (secure shell) connection to the Oracle database.
The default value is 22.

OS Username

Type the username you use when you log onto the database server.
Spotlight will connect successfully to all nodes in the cluster only if every node employs the SAME username and password. If the username and password if required are not completed, you will not be able to view the operating system

Chapter 5 Spotlight on Oracle RAC


performance statistics for the database server. (Windows) If your login details have been assigned remote access to the registry of the database server, you do not need to complete the OS Username and OS Password fields. (If your login details have NOT been set up on the database server, you must enter a username and password that has access to the registry of the server.)

63

SSH Key Type


(Enabled only when you connect to the database server via Unix(SSH using Public/Private key).

Choose the type of key to use when making the Spotlight connection.
Public-key encryption is supported under SSH2 only The current options are RSA and DSA. For more information, see Public/Private keypairs in the online help.

SSH Private Key Filename


(See SSH Key Type above.)

Type the location of the file that contains the private key for the Spotlight connection, or click the "..." button to locate it. Type the passphrase used to decrypt the private key Enter the user password (if required).

SSH Passphrase
(See SSH Key Type above.)

OS Password
(Enabled when you connect to the database server via Unix(REXEC), Unix(SSH), or Windows).

7 Select the Launch Spotlight on Oracle option to open a Spotlight on Oracle connection to the Oracle RAC database whenever you open the Spotlight on Oracle RAC connection. In most cases, this should not be necessary. 8 Select the Display Connections Verification Details option to display the Connection Details Verification window when you connect to a specified Oracle RAC cluster. The window allows you to specify and test connections to ALL instances in the cluster. For more information, see To verify connection details on page 64. 9 Click the Save password details box to save all the password details that have been entered. 10 Click OK to finish. A connection icon with the name you specified is created.

64

Spotlight on Oracle Getting Started Guide

Connecting to an Oracle RAC database


Use this procedure to connect to an Oracle database running on an Oracle RAC cluster.
Notes: Before connecting to an Oracle database, ensure that you have: Created a Spotlight user (see page 19) that has the special privileges required to diagnose your Oracle RAC databases. Created a connection to the Oracle RAC databases that you want to diagnose (see page 60).

To connect to an Oracle RAC database


Note: When installed under its own trial key, Spotlight runs for a limited time with full functionality. You must register the product before the end of the trial period in order to maintain functionality. For further information see Product Authorization Errors on page 75.

1 From the File menu choose Connect. 2 Double-click the connection icon that represents the connection that you want to open. 3 Verify connection details for the specified connection, if required.

To verify connection details


When you connect to a Spotlight on Oracle RAC connection, you can use the Connection Details Verification window to view the available connection data and provide or modify any data (port number, for example) that is missing or faulty.

Chapter 5 Spotlight on Oracle RAC The Connection Details Verification window is displayed under the following conditions:

65

When the Display this dialog for the current connection option is selected in the connection properties or in the Verification window itself, OR When Spotlight encounters an error during the connection process. For more information verifying connection details, see Verify connection details in the online help.

Using Spotlight on Oracle RAC


When you use Spotlight on Oracle RAC to diagnose an Oracle database running on an Oracle RAC cluster, you can: View the status of the connected database in the panels and dataflows of the Spotlight home page. View the alarms that Spotlight has raised for the connected database. View the detailed performance of the connected database in the various Spotlight drilldowns. View the performance of individual database instances in Spotlight on Oracle.

The Spotlight on Oracle RAC home page


The Spotlight home page shows an overview of the activity in an Oracle RAC database cluster. The display is based on the Oracle RAC architecture diagram shown in About Oracle RAC on page 57.

66

Spotlight on Oracle Getting Started Guide

Panels
There are three main panels on the home page for Spotlight on Oracle RAC. The panels are: The Interconnect and Global panel. This provides information on how well data is being shared and transferred between nodes in the cluster. The Instances panel. Together with separate panels for all participating instances, this panel provides information on the performance of individual instances in the cluster. The I/O Subsystem panel. This provides information on data storage and archiving in the cluster. These panels are described in more detail below. For a full discussion of the panels and the information they display, see the Spotlight online help.
Note: To view a tool tip for a specific component in a panel, hover the mouse pointer over the component.

Chapter 5 Spotlight on Oracle RAC Interconnect and Global panel

67

This panel at the top of the Spotlight home page shows a combination of Interconnect and Global Cache Service information, and information about the Oracle RAC database overall. It consists of information aggregated from queries sent out to the individual instances. Some information displayed here is easily calculated: for example, the Total Logical Reads spinner is the sum of the individual Logical Reads values. Other components, such as Cluster Latency and Cluster Overhead, have a more complex calculation. Spotlight uses the Global Database label to indicate alarms in one or more Oracle instances that may affect the performance of the Oracle RAC database as a whole. The flows between the Interconnect Panel and the Instances panel below represent the flow of data from one instance to another across the interconnect. Instances panel (and individual instances)

68

Spotlight on Oracle Getting Started Guide The middle panels show the participating instances of a cluster. Spotlight on Oracle RAC supports a theoretical maximum of 36 instances, with up to 6 instances shown on the home page. When there are more than 6 instances (not shown here), you can use a navigation control in the Instances panel to navigate between groups of instances. The color of the border and label on each instance represents the Spotlight on Oracle severity of the instance. You can move the mouse over a named instance to display a list of all the alarms currently raised against that instance.
Note: You can launch Spotlight on Oracle for any of the individual instances in the Instances Panel. (For more information on this, see Viewing Oracle RAC instances with Spotlight on Oracle.)

The overall status of the cluster does not necessarily reflect the status of individual instances, as problems with individual instances do not necessarily translate to a problem with the cluster as a whole. Two further metrics Balance and Availability are shown in the instances panel. Balance indicates the comparative load on the instances. Availability shows the percentage of instances for which a SQL*Net connection is currently open and working. I/O Subsystem panel

This panel shows information relating to the shared storage subsystem for the Oracle RAC cluster. Some of this information is aggregated from queries sent to the individual instances; other information is obtained from a single query to one of the instances. The ASM spinner on the left of this panel is populated only when ASM (Automatic Storage Management) is being used and has been defined in the Connection Properties window (see Creating a connection to an Oracle RAC database on page 60).

Chapter 5 Spotlight on Oracle RAC The flows between the I/O Subsystem panel and the individual instances represent the flow of physical reads and writes between the instances and the physical database.

69

Spotlight on Oracle RAC alarms


The following are brief descriptions of the alarms specifically associated with Spotlight on Oracle RAC. For possible solutions for the problems indicated by these alarms, or for information on how to diagnose problems further, see the Spotlight online help.

Aggregated Spotlight on Oracle alarms


Every Oracle instance in an Oracle RAC cluster can have its own performance issues, but problems in a single instance may not always affect the operation of the cluster as a whole. In such cases, Spotlight on Oracle RAC can display an alarm in the relevant instance panel without reporting a specific Spotlight on Oracle RAC alarm. However, if a problem in a single instance DOES affect the operation of the whole cluster or is repeated in multiple instances so that the combined effect is significant Spotlight on Oracle RAC reports the problem as an aggregated alarm. These alarms are displayed on the Spotlight on Oracle RAC home page usually in the Global Database heading of the Interconnect and Global Status panel, or on the individual components of the IO Subsystem panel.
Note: Aggregated alarms within Spotlight on Oracle RAC are the same as the corresponding alarms in Spotlight on Oracle (see page 37). Not all Spotlight on Oracle alarms are aggregated in Spotlight on Oracle RAC.

Archive Critical Failure alarm


An Archive Critical Failure alarm indicates that the number of working archive destinations for the database cluster is about to fall below the minimum number needed to archive Oracle redo logs successfully. When you see this alarm, you should make space in the archive log destination by purging or by backing up to an offline medium any archived files that are not immediately needed.

70

Spotlight on Oracle Getting Started Guide

Archive Destination Failed alarm


Spotlight raises an Archive Destination Failed alarm when at least one of the archive destinations specified for redo log files has become full. See Archive Destination Failure alarm below.

Archive Destination Failure alarm


An Archive Destination Failure alarm when at least one of the archive destinations specified for redo log files is about to become full. When you see this alarm, you should make space in the archive log destination by purging it, or by backing it up to an offline medium.

ASM Status alarm


When a Spotlight connection is lost, Spotlight automatically tries to re-establish the connection. The ASM Status alarm indicates that Spotlight is re-establishing its connection to the ASM diskgroup for the Oracle RAC cluster.

Availability alarm
This alarm indicates that the instances participating in the cluster are NOT all currently available for access. The database administrator should determine why the instances are unavailable, and restart them after taking the required actions.

(Global) Cache Miss Rate alarm


See Miss Rate alarm below.

Cluster Latency alarm


Spotlight raises the Cluster Latency alarm when cluster latency from all causes (interconnect, GES, LMS overhead) exceeds a specified threshold. Cluster latency is defined as the average elapsed time from when a requesting instance requests a block of data to the time when the requesting instance receives the block.

Cluster Overhead alarm


Spotlight raises the Cluster Overhead alarm when the percentage of time dedicated to cluster co-ordination activities exceeds a specified threshold. Cluster overhead is the amount of time spent waiting for cluster related activities as a percentage of the total time spent waiting for all activities.

Chapter 5 Spotlight on Oracle RAC

71

Corrupt or Lost Blocks alarms


Spotlight raises separate alarms for corrupt and lost blocks when a specified percentage of the blocks transferred across the interconnect are corrupted or lost during global cache consistent read requests. High levels of lost or corrupt blocks should not happen in normal circumstances. If these alarms fire at all, it indicates either very serious overloading of the Oracle RAC cluster, or software or hardware faults in the cluster.

Instance Failed alarm


Spotlight raises an Instance Failed alarm when a specified instance has failed because of an Archiver error. See Archive Critical Failure alarm above.

Load Imbalance alarm


Spotlight raises the Load Imbalance alarm when: The load across instances in the cluster is significantly unbalanced, and The imbalance is proven to be statistically significant, beyond a 95% probability. Cluster balance is a fundamental indication of the health and productivity of an Oracle RAC environment. If load is not reasonably evenly distributed across the instances in a cluster, then the DBA is likely to regret having chosen a clustered environment. Spotlight on Oracle RAC calculates balance by recording a user-definable metric of load over time on each instance (the default is Logical Reads, but you can switch this to CPU usage or Physical Writes via the Spotlight on Oracle RAC Options window) and calculating the relative balance of load across the cluster. In the normal course of events, there are always minor variations in load between instances within the cluster, so Spotlight uses a statistical technique to determine if a variation is significant.
Note: The activation of specific Oracle Services can unbalance the workload across the Oracle RAC cluster. In these cases, Spotlight triggers the Load Imbalance alarm only when a more significant threshold is reached.

Miss Rate alarm


This alarm, more specifically known as the Cache Miss Rate alarm or Global Cache Miss Rate alarm, is raised when the global cache local miss rate exceeds a specified

72

Spotlight on Oracle Getting Started Guide threshold. This rate is the percentage of time that a block required for a logical read is located in an inconsistent mode on ANOTHER instance in the cluster and must be transferred.

No Nodes Available alarm


This alarm indicates that Spotlight has lost its connection to the Oracle RAC cluster, and cannot re-connect to any of the nodes in the cluster. Check the status of your network connection, your host machines, and your database instances.

Spotlight on Oracle RAC drilldowns


Spotlight on Oracle RAC makes it possible for you to view your system at different levels of detail. At its highest level (the home page) Spotlight displays a visual representation of the status of the major components in the current system. When you have isolated a problem, you can display a drilldown page, whose charts and tables provide a detailed breakdown of the underlying statistics.

Spotlight on Oracle RAC contains the following drilldowns, most of which are equivalent to drilldowns in the standard edition of Spotlight on Oracle. For further information, see Spotlight on Oracle drilldowns on page 46. Drilldown Sessions SQL & Application Workload Description As per Spotlight on Oracle. As per Spotlight on Oracle. The SQL & Application Workload drilldown in Spotlight on Oracle RAC does NOT contain the following pages: SQL Analyze Trace Activity Result Cache

As per Spotlight on Oracle. The Activity drilldown in Spotlight on Oracle RAC does NOT contain the following pages: Lock Activity Server Rollback

Chapter 5 Spotlight on Oracle RAC Drilldown I/O Configuration & Memory Description As per Spotlight on Oracle. As per Spotlight on Oracle. The Configuration & Memory drilldown in Spotlight on Oracle RAC does NOT contain the following pages: Shared Pool Utilization Disk Storage Buffer Cache

73

As per Spotlight on Oracle. The Disk Storage drilldown in Spotlight on Oracle RAC does NOT contain the following pages: Online Redo Recovery Area

Cluster

The pages in the Cluster drilldown display information about the performance of the Oracle RAC cluster, including details on cluster latency, balance, overhead, miss ratios, and corrupt or lost blocks. For further information see Cluster drilldown in the online help. As per Spotlight on Oracle. The Services drilldown displays details of the services defined and deployed for the Oracle RAC cluster under investigation. As per Spotlight on Oracle.

ASM (If available) Services (Oracle 10.2 and later) Predictive Diagnostics (If available) Alarm Log

As per Spotlight on Oracle.

Viewing Oracle RAC instances with Spotlight on Oracle


The Spotlight on Oracle installation program installs both Spotlight on Oracle RAC and Spotlight on Oracle on your system. You can launch Spotlight on Oracle from any Instance panel on the Spotlight on Oracle RAC home page.

74

Spotlight on Oracle Getting Started Guide

When you click on a component on the Spotlight on Oracle RAC home page, Spotlight's default behavior is to display a context-sensitive help window. When you click the heading of an Instance panel, the help window contains the following item.

Click Show me the Spotlight on Oracle drilldown to open Spotlight on Oracle for the corresponding instance. (You can also use Help Options in the Spotlight Help menu to configure Spotlight to display the drilldown directly.)

6
Troubleshooting
This chapter suggests solutions to problems that may occur on opening Spotlight and making a connection.

Product authorization errors


The following are steps that you can take should you encounter an authorization problem with Spotlight. When first installed, Spotlight can be used with full functionality for a specified trial period. After this time it reverts to Preview mode for the standard edition of Spotlight on Oracle.

Preview mode
In preview mode, only the Spotlight on Oracle home page is accessible. If you click any of the drilldown buttons, the following message is displayed:
You do not have a license for Spotlight on Oracle that enables access to this functionality. Please contact Quest Software for information about obtaining the correct license.

If the trial date has not passed, check that the date on your system is set correctly. Otherwise, contact your Quest Software representative to obtain a registered key.

To enter a new authorization key


You can enter a new authorization key from Spotlight on Oracle itself. 1 From the Help menu, select About Spotlight. (The Installed Products page contains license details for all installed Spotlight products.) 2 Select View / Change Product License under any Spotlight on Oracle entry in the list of installed products. 3 In the License Viewer, click the Change this license link.

76

Spotlight on Oracle Getting Started Guide 4 Enter the new Authorization key and Site message and then click OK.

Invalid authorization key


If you enter the authorization key incorrectly, the following message is displayed:
Invalid authorization key

Make sure that you enter the authorization key exactly as stated on your Product Authorization sheet or as provided by your Quest Software distributor.

Connection problems
The following are a set of problems that may be encountered on connection. Follow the indicated solutions to resolve those problems.

Spotlight cannot connect to a database


There are a number of possible causes. Possible cause Oracle is not running. Solution Verify that the Oracle instance is running and can be accessed from a host or client workstation. Check your SQL* Net configuration and verify the connect string and alias you are using are correct. Run the User Wizard to establish the user ID in the database. Contact your DBA to add relevant permissions to the user. Remote commands may not work unless the user ID is added to the /etc/hosts.equiv file on that Unix machine.

The connect string and alias of the database are incorrect or have changed. The user ID you are using to log into Spotlight does not exist in the database. The Spotlight user has the wrong permissions in the database. Remote commands do not work when monitoring Unix OS with a user ID/password that exists on the machine.

Chapter 6 Troubleshooting

77

Error displayed after attempting to connect


The following error message is displayed:
Connection has been refused by the server

This means that Spotlight cannot access the operating statistics of the (Windows) database machine. Users must have access to the registry so that Spotlight can retrieve the operating system statistics.

To enable Spotlight to access the Operating System statistics


Give Spotlight the privileges it needs to access the registry. To do this, connect to the remote machine via a user account that has the appropriate privileges. An account with administrative rights to the target machine will allow this.

Spotlight cannot connect to ASM


When installed and available, Automatic Storage Management (ASM) stores Oracle data in a separate ASM database instance. Spotlight on Oracle displays performance metrics for ASM on the home page and in the ASM drilldown. If the database under investigation uses ASM but Spotlight cannot connect to it to retrieve the required information, you may need to edit the TNSNAMES entry for the ASM instance. 1 Locate the tnsnames.ora file in Oracles network/admin sub-folder of the Oracle installation folder on your Spotlight client. 2 Edit tnsnames.ora so that the TNSNAMES entry for the target ASM instance contains the option (UR = A).

Drilldown problems
The following are a set of problems that may be encountered. Follow the indicated solutions to resolve those problems.

Alarm help displayed instead of the drilldown


This occurs when component that is in alarm status; the alarm help is displayed instead of the drilldown.

78

Spotlight on Oracle Getting Started Guide

To display the drilldown


1 From the Help menu choose Help Options. 2 Select Always jump directly to the related drilldown.

The Top Sessions drilldown is displaying dead sessions


This occurs when the Top Sessions drilldown has not been reset recently.

To reset the Top Sessions drilldown


1 From the Top Sessions drilldown click on the Filter button. 2 Click Reset 3 Select the appropriate frequency.

Inaccurate information is displayed in drilldowns


Inaccurate information may be displayed in the following locations: Buffer Busy Waits (activity drilldown) Session Details (Top Sessions drilldown) Session waits (Top Sessions drilldown) Spotlight may display inaccurate information when you first connect to a database because the Fast Initialize Spotlight option is ON by default. To turn Fast Initialize off 1 From the View menu choose Options. 2 Under Data Collection click Oracle Initalization. 3 Click Turn Fast Initialize off. 4 Restart Spotlight.

The OS drilldown contains inaccurate information for Windows systems


The Operating System drilldown displays operating system performance statistics for the database you are diagnosing. If the database is running on a Windows operating system, users must have access to the registry so that Spotlight can retrieve the

Chapter 6 Troubleshooting

79

operating system statistics. See To enable Spotlight to access the Operating System statistics on page 77.

The OS drilldown contains errors for Unix systems


The Operating System drilldown may contain errors if it has lost access to the database machine. If the database is running on Unix, Spotlight can use one of the REXEC or SSH protocols to submit various Unix commands (for example, netstat, vmstat, iostat, sar) to monitor system activity. These commands must be accessible to an REXEC or SSH session for Spotlight to be able to monitor Unix activity. If these commands are not located in the standard search path available to the session, Spotlight displays an error.

Predictive Diagnostics is not available


You can use Predictive Diagnostics to collect and analyze performance metrics for SQL statements as they are executed on the database instance, system bottlenecks as they affect database throughput and response time, and database resources whose limitations affect database performance. If Predictive Diagnostics is unavailable for your Spotlight connections, re-run the Oracle user wizard to install Predictive Diagnostics. 1 Close and re-open Spotlight on Oracle. 2 From the File menu choose Oracle User Wizard... 3 Follow the procedure described in the section To create a Spotlight user (page 19). Make sure that you install Predictive Diagnostics as part of the procedure.
Note: When you start using Predictive Diagnostics, Spotlight will take about 10 days to collect enough data to make valid predictions on the future performance of the database. During that time, you can use the supplied sample data to view Predictive Diagnostics at work.

High Spotlight load on database server


The queries that Spotlight uses to collect data from a diagnosed system can place an additional load on the system under diagnosis. Most Spotlight queries consume a measurable but not significant amount of system resources. However, queries that

80

Spotlight on Oracle Getting Started Guide are complex or collect large amounts of data may put a significant load on the system. If you find that the load that Spotlight places on your server is too high, you can adjust collection rates for different categories of Spotlight data, and so limit the load that Spotlight places on the system under analysis. You can choose a standard Spotlight setting, or you can create your own Custom settings. 1 Close and re-open Spotlight on Oracle. 2 From the File menu choose Connect... 3 In the list of Spotlight on Oracle connections, right-click the connection whose load you want to change. 4 Select Properties from the shortcut menu. 5 Click the Overhead tab. 6 Use the Overhead Management slider to adjust the overall Spotlight load for the connection. (Alternatively, click Custom to adjust specific load categories.) 7 Open the Spotlight connection. The procedure for adjusting the Spotlight load is covered in more detail in the online help.

APPENDIX

Appendix: Contact Quest


Contact Quest Support
Quest Support is available to customers who have a trial version of a Quest product or who have purchased a Quest product and have a valid maintenance contract. Quest Support provides unlimited 24x7 access to SupportLink, our self-service portal. Visit SupportLink at http://support.quest.com. From SupportLink, you can do the following: y Retrieve thousands of solutions from our online Knowledgebase y Download the latest releases and service packs y Create, update and review Support cases View the Global Support Guide for a detailed explanation of support programs, online services, contact information, policies and procedures. The guide is available at: http://support.quest.com.
Note This document is available only in English.

Contact Spotlight support


Whenever you have a question about Spotlight, click Support Bundle or Contact Support on the Help menu. This creates a file called spotlightsupport.zip in one of the following folders: Windows XP Vista / Windows 7 In a sub-folder under the Documents and Settings folder. In a sub-folder under the Users folder.

82

Spotlight on Oracle Getting Started Guide This file contains a snapshot of your Spotlight installation. Send this file and your request for assistance to Quest Software (support@quest.com).
Notes: y Support bundles contain information for the active instance only. If you are using Spotlight to diagnose multiple instances ensure that the correct instance is displayed before creating the support bundle. y Support bundles cannot be created when the History Browser is open. For further information on the History Browser, see the Spotlight online help under Concepts and Features.

Contact Quest Software


Email Mail info@quest.com Quest Software, Inc. World Headquarters 5 Polaris Way Aliso Viejo, CA 92656 USA Web site www.quest.com

Refer to our Web site for regional and international office information.

About Quest Software, Inc


Now more than ever, organizations need to work smart and improve efficiency. Quest Software creates and supports smart systems management productshelping our customers solve everyday IT challenges faster and easier. Visit www.quest.com more information.

INDEX

Index
A Activity drilldown................................. 47, 72 Aggregated alarms ...................................... 69 Alarm Log drilldown ............................ 49, 73 Alarms .................................................. 11, 37 Aggregated alarms .................................. 69 Archive Critical Failure .................... 37, 69 Archive Destination Failed ............... 38, 70 Archive Destination Failure .............. 38, 70 Array Fetch Size ..................................... 38 ASM balance .......................................... 38 ASM service time ................................... 38 ASM status ....................................... 38, 70 Availability ............................................. 70 Average redo write ................................. 38 Average time to sync a redo log entry .... 38 Balance ................................................... 71 Bottleneck collector ................................ 43 Buffer busy wait ..................................... 39 Cache buffer chains latch........................ 39 Cache buffer LRU chains latch ............... 39 Cache Miss Rate ..................................... 71 Cluster Latency ....................................... 70 Cluster Overhead .................................... 70 Continued Fetch Rate ............................. 39 Corrupt Blocks........................................ 71 CPU busy ................................................ 39 Custom SQL Error .................................. 43 Datafile read time ................................... 39 Dispatchers busy ..................................... 39 EM Agent Not Connected ...................... 40 Estimated shared pool size ...................... 40 Excessive RBS activities ......................... 40 Flashback buffer wait percentage............ 40 Flashback space usage ............................ 41 Free buffer waits ..................................... 41 Global Cache Miss Rate .......................... 71 Instance Failed .................................. 41, 71 Insufficient flashback logs ...................... 41 Job processes busy .................................. 41 Large pool full......................................... 41 Latch free waits ....................................... 41 Library cache latches .............................. 41 Library cache miss rate ........................... 42 Load Imbalance ....................................... 71 Lock wait ................................................ 42 Log Buffer Space Wait ........................... 42 Log switch time....................................... 42 Lost Blocks ............................................. 71 Low free physical RAM .......................... 42 Miss Rate ................................................ 71 Multi-threaded server .............................. 42 No Nodes Available ................................ 72 Non-reclaimable flashback space ............ 42 Oracle parameters ................................... 43 Oracle status ............................................ 43 Parallel query server................................ 43 Parse ratio................................................ 43 Predictive Diagnostics............................. 43 Redo allocation and Redo copy latch ...... 44 Result Cache Find/Create Ratio .............. 44 Result Cache Latch Sleep Ratio .............. 44 Sequential Read Time ............................. 44

84

Spotlight on Oracle Getting Started Guide Cluster Overhead alarm .............................. 70 Components of an Oracle RAC cluster ....... 58 Components of an Oracle server ................... 7 Configuration and Memory drilldown ........ 73 Configuration drilldown .............................. 48 Connecting to a database ............................................ 26 troubleshooting........................................ 76 Connections........................................... 21, 26 Continued Fetch Rate alarm ........................ 39 Corrupt Blocks alarm .................................. 71 CPU Busy alarm ......................................... 39 Creating a database connection ................... 21 Creating a Spotlight user ....................... 19, 60 Custom SQL Error alarm ............................ 43 D Database cannot connect......................................... 76 tuning ...................................................... 50 Datafile Read Time alarm ........................... 39 Disk Storage drilldown ......................... 48, 73 Disk Storage panel ...................................... 36 Dispatchers Busy alarm .............................. 39 Drilldown problems .................................... 77 Drilldowns....................................... 11, 46, 72 Activity ............................................. 47, 72 Alarm Log ......................................... 49, 73 Application Workload ................. 47, 72, 73 Array fetch .................................. 47, 72, 73 ASM .................................................. 48, 73 Cluster ..................................................... 73 Configuration .......................................... 48 Configuration and Memory ..................... 73 Disk Storage ...................................... 48, 73 DTrace..................................................... 48 I/O ........................................................... 47 Memory ................................................... 48 Operating System .................................... 48 Parse activity ............................... 47, 72, 73 Predictive Diagnostics....................... 48, 73 Result Cache ........................................... 47 Services ................................................... 73

Shared pool lock percentage ................... 44 Shared server converted into dedicated server .................................................. 44 SQL collector .......................................... 43 Temporary IO wait ratio ......................... 45 Total parse wait time ratio ...................... 45 Unarchived logs ...................................... 45 Unix status .............................................. 45 WMI status ............................................. 45 Write complete wait ................................ 45 Archive Critical Failure alarm .............. 37, 69 Archive Destination Failed alarm ......... 38, 70 Archive Destination Failure alarm ........ 38, 70 Array fetch ...................................... 47, 72, 73 Array Fetch Size alarm ............................... 38 ASM ........................................................... 77 ASM Balance alarm.................................... 38 ASM drilldown ..................................... 48, 73 ASM Service Time alarm ........................... 38 ASM Status alarm ................................. 38, 70 Authorization errors .................................... 75 Availability alarm ....................................... 70 Average redo write time alarm ................... 38 Average Time to Sync a Redo Log Entry alarm ....................................................... 38 B Background Processes panel....................... 35 Balance alarm ............................................. 71 Benefits ......................................................... 7 Bottleneck collector alarm .......................... 43 Bottleneck, tuning by .................................. 50 Buffer busy wait alarm ............................... 39 C Cache Buffer Chains Latch alarm ............... 39 Cache Buffer LRU Chains Latch alarm ...... 39 Cache Miss Rate alarm ............................... 71 Charts .......................................................... 11 Client permissions ...................................... 15 Cluster components .............. See Oracle RAC Cluster drilldown ........................................ 73 Cluster Latency alarm ................................. 70

85 Index Sessions ............................................ 46, 72 Sort activity................................. 47, 72, 73 SQL & Application Workload .... 47, 72, 73 SQL Analyze Trace ................................ 47 Top SQL ..................................... 47, 72, 73 troubleshooting ....................................... 77 Tuning..................................................... 48 DTrace .................................................. 13, 27 DTrace drilldown ........................................ 48 E EM Agent Not Connected alarm ................ 40 Error connection has been refused by the server ..................................................... 77 Errors 1114 ........................................................ 79 authorization ........................................... 75 operating system drilldown..................... 79 Estimated Shared Pool Size alarm .............. 40 Excessive RBS Activities alarm ................. 40 Explain Plan................................................ 50 F Features......................................................... 7 DTrace .............................................. 13, 27 Spotlight on Oracle ..................... 11, 13, 27 Flashback Buffer Wait Percentage alarm ... 40 Flashback Space Usage alarm .................... 41 Free buffer waits alarm ............................... 41 G Global Cache Miss Rate alarm ................... 71 H History ........................................................ 11 Home page ........................................ 9, 29, 65 Predictive Diagnostics ............................ 48 Host panel ................................................... 32 I I/O drilldown .............................................. 47 Install .......................................................... 15 Installing Spotlight ............................... 15, 59 Instance Failed alarm ............................ 41, 71 Insufficient Flashback Logs alarm .............. 41 J Job Processes Busy alarm ........................... 41 L Large Pool Full alarm ................................. 41 Latch Free Waits alarm ............................... 41 Library Cache Latches alarm ...................... 41 Library Cache Miss Rate alarm................... 42 Load ............................................................ 79 Load Imbalance alarm ................................. 71 Lock wait alarm .......................................... 42 Log Buffer Space Wait alarm ..................... 42 Log Switch Time alarm............................... 42 Lost Blocks alarm ....................................... 71 Low Free Physical RAM alarm................... 42 M Main Spotlight window ..................... 9, 29, 65 Memory drilldown ...................................... 48 Menus.......................................................... 49 Miss Rate alarm .......................................... 71 MSI installation ........................................... 18 Multiple installations................................... 17 Multi-threaded server alarm ........................ 42 N No Nodes Available alarm .......................... 72 Non-reclaimable Flashback Space alarm .... 42 O Operating System drilldown ....................... 48 errors ....................................................... 79 Oracle architecture ................................................ 7 server components..................................... 7 Oracle Parameters alarm ............................. 43 Oracle RAC architecture .............................................. 57 cluster components.................................. 58 Oracle Status alarm ..................................... 43

86

Spotlight on Oracle Getting Started Guide Shared Server Converted into Dedicated Server alarm ............................................ 44 Sort activity ..................................... 47, 72, 73 Space Manager ............................................ 50 Spotlight alarms ................................................ 11, 37 connections ....................................... 21, 26 drilldowns ................................... 11, 46, 72 DTrace..................................................... 27 home page ..................................... 9, 29, 65 installing ............................................ 15, 59 menus ...................................................... 49 starting............................................... 18, 59 users .................................................. 19, 60 Spotlight on Oracle ............................... 29, 57 MSI installation ....................................... 18 Multiple installations............................... 17 Spotlight on Oracle RAC ............................ 65 SQL & Application Workload drilldown ... 47, 72, 73 SQL Analyze Trace ..................................... 47 SQL Collector alarm ................................... 43 SQL Optimizer ............................................ 50 Starting ........................................................ 15 Starting Spotlight .................................. 18, 59 Summary of tuning steps............................. 50 Support Bundle ........................................... 81 T Tables .......................................................... 11 Temporary IO Wait Ratio alarm ................. 45 Top SQL.......................................... 47, 72, 73 Total Parse Wait Time Ratio alarm ............. 45 Troubleshooting .......................................... 75 ASM ........................................................ 77 authorization ........................................... 75 connection problems ............................... 76 dead sessions ........................................... 78 drilldown problems ................................. 77 Predictive Diagnostics............................. 79 server load ............................................... 79 Tuning ......................................................... 50 Tuning drilldown......................................... 48

P Panels Background Processes ............................ 35 Disk Storage ........................................... 36 Host......................................................... 32 overview ................................................. 30 Server Processes ..................................... 33 Sessions .................................................. 31 SGA ........................................................ 34 Parallel Query Server alarms ...................... 43 Parse activity................................... 47, 72, 73 Parse Ratio alarm ........................................ 43 Permissions ................................................. 15 Predictive Diagnostics .................... 48, 73, 79 alarms ..................................................... 43 storage requirements ............................... 51 Preview mode ............................................. 75 Q Quest Software about ....................................................... 82 contacting................................................ 82 contacting Support .................................. 81 R RAC ...................................... See Oracle RAC Redo Allocation and Redo Copy Latch alarms ..................................................... 44 Result Cache ............................................... 47 Result Cache Find/Create Ratio alarm........ 44 Result Cache Latch Sleep Ratio alarm........ 44 S Sequential Read Time alarm ....................... 44 Server components ........................ See Oracle Server load .................................................. 79 Server Processes panel................................ 33 Services drilldown ...................................... 73 Sessions drilldown ................................ 46, 72 Sessions panel ............................................. 31 SGA panel .................................................. 34 Shared Pool Lock Percentage alarm ........... 44

87 Index U Unarchived Logs alarm............................... 45 Unix Status alarm ....................................... 45 Users ..................................................... 19, 60 W WMI Status alarm ....................................... 45 Write Complete Wait alarm ........................ 45

You might also like