Database Maintenance

Branchless Banking
11/30/2012 Inov8 Limited

Site Switching Mechanism

Table of Contents
Introduction .................................................................................................................................................. 3 Architecture: ................................................................................................................................................. 3 Periodic Maintenance Tasks: ........................................................................................................................ 4 Daily .......................................................................................................................................................... 4 Verify instance status ................................................................................................................................ 4 Alert Log: ................................................................................................................................................... 4 Configured metrics.................................................................................................................................... 4 RMAN backups .......................................................................................................................................... 4 Check for Various Performance indicators: .............................................................................................. 5 Weekly ...................................................................................................................................................... 6 Weekly Backup .......................................................................................................................................... 6 Re-compile invalid objects ........................................................................................................................ 6 Tuning: ...................................................................................................................................................... 7 Cleaning of alert logs................................................................................................................................. 7 Flash Recovery Area: ................................................................................................................................. 7 Monthly ..................................................................................................................................................... 8 Recovery tests ........................................................................................................................................... 8 Fragmentation .......................................................................................................................................... 8 Row chaining and migration: .................................................................................................................... 8 Oracle Streams Replication Maintenance: ................................................................................................... 9 Start or stop a capture process: .............................................................................................................. 10 Enable or disable propagation: ............................................................................................................... 10 Start or stop an apply process: ............................................................................................................... 11 Display an overview of the replication components: ............................................................................. 11

Inov8 Limited-Copyright 2012

Page 2

Site Switching Mechanism

Introduction
This document elaborates in detail the procedures/tasks for proactive maintenance and troubleshooting of the Database system for Timepey Branchless Banking (BB) System. These comprise of some periodic and other situation specific tasks. The database system design is based on oracle recommended maximum-availability architecture. The primary aim of the database maximum-availability architecture is to achieve superior data protection and availability by minimizing or eliminating planned and unplanned downtime at all technology stack layers including hardware or software components. Data protection and high availability are achieved regardless of the scope of a failure event - whether from hardware failures that cause data corruptions or from catastrophic acts of nature that impact a broad geographic area. This architecture involves 2 node RAC database primary and 1 Node stand alone database disaster recovery (DR) sites. The primary site is our current production database. The secondary site is the real time replica of the primary site. In case of disaster to the primary site the secondary site will assume the role of production DB. There are to two solutions available to synchronize secondary site with the primary production DB, i.e. standby database using Oracle datagaurd technology and replication using Oracle streams or Multimaster replication using Oracle Advanced Replication techniques. We are using oracle RAC for load balancing and failover and Oracle Streams replication for data replication between primary and secondary sites.

Architecture:
The primary site is located at the Askari Bank Information Technology Group (ITG) datacenter. The primary database system is comprised of Oracle 11g R2 database 2 nodes RAC connected to IBM D8300 SAN mounted by Oracle ASM for DB storage. The DR site is at Askari Bank G8 office. There is only one database server connected to the IBM D8700 SAN using Oracle ASM. The sites are interconnected on dark fiber. The two databases are kept synchronized by Oracle Stream Real Time data Replication.
Node1 IBM D8300 dblink inov8.akblbb dblink drdb1.akblbb
File System: ASM SID: inov8

IBM D8700

Drdb1

File System: ASM SID: inov8 Service: inov8.akblbb

Node2

Service: inov8.akblbb

File System: ASM Primary Site at ITG SID: inov8

Failover site at G-8

Page 3

Inov8 Limited-Copyright 2012

Service: inov8.akblbb

Site Switching Mechanism

Periodic Maintenance Tasks:

Daily Verify instance status
Connect to sqlplus as sysdba and issue the command:
select instance_name,status from gv$instance; INSTANCE_NAME STATUS

---------------- -----------INOV8 OPEN

Exit from sqlplus At linux shell issue the command: Lsnrctl status It will show the services currently handled by the LISTENER.

Alert Log:
Alert logs file is stored in the
/u01/app/oracle/diag/rdbms/inov8/inov8/trace/alert_inov8.log

Or V$DIAG_INFO dictionary view can be quiried to know the alertlog and other diagnostic trace files paths.
select * from V$DIAG_INFO;

Open the file and move to the end. Check the file for any alerts or errors in the last 24 hours.
If there is any alert or error investigate further to take the necessary action. Or Oracle Enterprise manager can also be used to view the latest alert log contents at
https://localhost:1158/em.

Configured metrics
Check for any alerts generated in the configured matrices page at the Enterprise Manager

RMAN backups
Check if the last night cumulative backup was successful.

Inov8 Limited-Copyright 2012

Page 4

Site Switching Mechanism Production database is configured to take a cumulative incremental backup of the whole database daily at 10:00 PM. The RMAN backup set is stored at the
/u01/app/oracle/admin/flash_recovery_area/inov8/Backupsets

The backup set is then copied to the tape drive via Tivoli BA client at 12:00 AM

The above figure elaborates the backup strategy for this database system. A full level 0 database + archive log backup is taken on Sunday and a cumulative incremental backup of the whole database is performed daily in the week days.

Check for Various Performance indicators:

Check for these performance indicators in the enterprise manager console: Storage CPU contention
select ss.username, se.SID, VALUE/100 cpu_usage_seconds from v$session ss, v$sesstat se, v$statname sn where se.STATISTIC# = sn.STATISTIC#

Inov8 Limited-Copyright 2012

Page 5

Site Switching Mechanism

and NAME like '%CPU used by this session%' and se.SID = ss.SID and ss.status='ACTIVE' and ss.username is not null order by VALUE desc;

waiting times memory usage network load I/O stat

Weekly Weekly Backup

The RMAN is configured to take a full backup of the database and archive logs on every Sunday 10:00 PM .The RMAN backup set is stored at the
/u01/app/oracle/admin/flash_recovery_area/inov8/Backupsets

The backup set is then copied to the tape drive via Tivoli BA client at 12:00 AM The manage current backups page of the EM or list backup set command of RMAN is used to check the backup status.

Re-compile invalid objects

Run the query
SELECT owner ,object_type ,object_name FROM dba_objects owner in ('I8_AKBL_PROD','INOV8_OLA_PROD','I8_VERIFLY_PROD','I8_PHOENIX_INTEGRATION','PAYMENT_ SERVICES') order by owner, object_type ; WHERE status != 'VALID' and

And look into the objects which have become invalid. Re-compile the objects if there is a need to recompile the objects. In case of invalid indexes, there is a need to rebuild the particular indexes. To recompile all the invalid objects other than indexes use:
SELECT 'ALTER ' || OBJECT_TYPE || ' ' || OWNER || '.' || OBJECT_NAME || ' COMPILE;' FROM dba_objects where status = 'INVALID' and owner in ('I8_AKBL_PROD','INOV8_OLA_PROD','I8_VERIFLY_PROD','I8_PHOENIX_INTEGRATION','PAYMENT_ SERVICES');

And execute the resultant alter statements.

Inov8 Limited-Copyright 2012

Page 6

Site Switching Mechanism In order to rebuild the invalid index, execute the following command for the particular index:
ALTER INDEX <index_name> REBUILD [online];

Tuning:
indexes and execution plans Top SQL

The top sql are the queries which are consuming a lot of resources. The top SQL can be found in TOP SQL page of the oracle enterprise manger or the following query can be used to find the top SQL in terms of elapsed time.
SELECT * FROM (SELECT sql_fulltext, sql_id, child_number, disk_reads, executions, first_load_time, last_load_time FROM v$sql

ORDER BY elapsed_time DESC) WHERE ROWNUM < 10;

Run the aforementioned query and analyze and try to tune the SQL.

Cleaning of alert logs

Archive the alert log file every Monday with the naming convertion
Alert_inov8_<from_date>to<to_date>.

Flash Recovery Area:

Check the size of files in the flash recovery area. If the flash recovery area files are approaching the max size allocated to the flash recovery area.

Inov8 Limited-Copyright 2012

Page 7

Site Switching Mechanism Move the archive logs and backup sets to some other location and free the area from rman by running crosscheck and delete commands from RMAN prompt. Also check that: The already backed up archive log are deleted from the flash recovery area. The obsolete back upsets are deleted from the flash recovery area.

Monthly Recovery tests

A complete recovery test should be performed at the last backup of the whole database.

Fragmentation
Check if there are any segments which are fragmented. If there are then try to reclaim the space from those segments by shrink space command.
Alter table <table_name> enable row movement; Alter table <table_name> shrink space;

If the indexes have become unusable than rebuild the specific index.

Row chaining and migration:

Identify the tables with migrated/chains rows by issuing the following query:
set pages 9999; column c1 heading "Owner" column c2 heading "Table" format a9; format a12;

column c3 heading "PCTFREE" format 99; column c4 heading "PCTUSED" format 99; column c5 heading "avg row" format 99,999; column c6 heading "Rows" column c7 heading "Chains" column c8 heading "Pct" set heading off; select 'Tables with migrated/chained rows and no RAW columns.' from dual; set heading on; format 999,999,999; format 999,999,999; format .99;

Inov8 Limited-Copyright 2012

Page 8

Site Switching Mechanism

select owner table_name pct_free pct_used avg_row_len num_rows chain_cnt c1, c2, c3, c4, c5, c6, c7,

chain_cnt/num_rows c8 from dba_tables where owner not in ('SYS','SYSTEM') and table_name not in (select table_name from dba_tab_columns where data_type in ('RAW','LONG RAW') ) and chain_cnt > 0 order by chain_cnt desc;

When the tables or indexes with migrated or chained rows have been identified then try reorganizing the objects. Try to move the tables to another tablespace by alter table move command and rebuild the indexes suffering the same problem.

Oracle Streams Replication Maintenance:

All the Streams related maintenance tasks can be performed through Oracle Enterprise manager Console. The addresses for the through Oracle Enterprise manager Console and stream administrator names are as follows:
Primary site:

Inov8 Limited-Copyright 2012

Page 9

Site Switching Mechanism

URL: https://172.24.26.21:1158/em Streams Administrator: strmadminprim DR site: URL: https://172.24.27.21:1158/em Streams Administrator: strmadmindr

Note: Passwords will be communicated verbally.

Start or stop a capture process:

1. 2. 3. 4. 5. In Oracle Enterprise Manager, log in to the database as the Oracle Streams administrator. Go to the Database Home page. Under High Availability, click the number link in Streams Components. The Manage Replication page appears, showing the Overview subpage. Click Streams to open the Streams subpage.

6. Select the capture process to start or stop in the list of components. If necessary, use the search tool to find the capture process, and then select it. 7. In the Action list, select Enable to start a disabled or aborted capture process, or select Disable to stop an enabled capture process. 8. Click Go. 9. If you are disabling the capture process, then click Yes on the confirmation page to finish disabling it.

Enable or disable propagation:

1. 2. 3. 4. 5. In Oracle Enterprise Manager, log in to the database as the Oracle Streams administrator. Go to the Database Home page. Under High Availability, click the number link in Streams Components. The Manage Replication page appears, showing the Overview subpage. Click Streams to open the Streams subpage.

Inov8 Limited-Copyright 2012

Page 10

Site Switching Mechanism 6. Select the propagation to enable or disable in the list of components. If necessary, use the search tool to find the propagation, and then select it. 7. In the Action list, select Enable to start a disabled or aborted propagation, or select Disable to stop an enabled propagation. 8. Click Go.

Start or stop an apply process:

1. 2. 3. 4. 5. 6. In Oracle Enterprise Manager, log in to the database as the Oracle Streams administrator. Go to the Database Home page. Under High Availability, click the number link in Streams Components. The Manage Replication page appears, showing the Overview subpage. Click Streams to open the Streams subpage. Select the apply process to start or stop in the list of components. If necessary, use the search tool to find the apply process, and then select it. 7. In the Action list, select Enable to start a disabled or aborted apply process, or select Disable to stop an enabled apply process. 8. Click Go. 9. If you are disabling the apply process, then click Yes on the confirmation page to finish disabling it.

Display an overview of the replication components:

1. In Oracle Enterprise Manager, log in to the database as the Oracle Streams administrator. 2. Go to the Database Home page. 3. Under High Availability, click the number link in Streams Components.

Inov8 Limited-Copyright 2012

Page 11

Site Switching Mechanism 4. The Manage Replication page appears, showing the Overview subpage.

Any errors in the streams environment can be viewed on this page. If any of the streams components is in aborted state, then it can be enabled from the action tab of the streams components page. If the any of streams component is disabled than the data is being replicated between the two sites. Try to view the error associated with the component. If there is no error in the view errors page in the actions tab then go to the alert log of database to find any error there. Resolve the error and enable the streams components.

Inov8 Limited-Copyright 2012

Page 12