DBA Daily Checks and Best Practice

Gordon D. Brown
Rutherford Appleton Laboratory
3D Workshop
PIC, Barcelona
20th-21st April 2009

Overview
Daily checks
Best practice
Future improvements
Your checks?

Which country is this?

Switzerland

Which country is this?

Finland

Which country is this?

Scotland

Which country is this?

Portugal

Which country is this?

China

Which country is this?

Iraq

Which country is this?

Somalia

Daily Checks
Grid Control
Green Pie!
Alerts, warnings
Performance

Other

Backups
Alert Logs
Interventions
Service exceptions
Your email

Grid Control

Grid Control: Home page

Grid Control: Targets down

Grid Control: Alerts

Grid Control: Check emails

Grid Control: Jobs

Grid Control: Notification rules

Grid Control: Notifications

Which country is this?

Egypt

Alert Logs

Alert Logs
Goals
Highlight any errors
Reduce time for DBAs to check

Each instance
Email logs to DBA team
Checks for lines to ignore
Reports status to central database

Central database
Has details of all instances
List of all lines to ignore
produces report (run through Grid Control)

Alert Logs: Emails

Alert Logs: Report

Alert Logs: Instance data

Alert Logs: Status

Alert Logs: Lines to ignore

Alert Logs: Instance crontab

# Alert and Listener Logs
0 8 * * * /home/oracle/scripts/alert.sh rcat rcat 10 rcat edb_alert <pword> >/dev/null 2>&1

Alert Logs: Instance Script 1

if [ `grep -v f alert_log_match.lst $ALERT_LOG_DIR/alert_${sdbi}.log | wc -l`
SUBJECT="CLEAR
else
SUBJECT="ERRORS
fi

-eq 0 ]; then

# Report the alert

sqlplus -s /nolog <<EOF
WHENEVER SQLERROR EXIT SQL.SQLCODE;
WHENEVER OSERROR EXIT FAILURE;
CONNECT $ldbu/$ldbp@$ldbs
INSERT INTO alert_log_status
(alert_log_status_id, database_name, instance_name, alert_log_status_date, status)
VALUES
(seq_alert_log_status.NEXTVAL, '${sdbs}', '${sdbi}', SYSDATE, '$SUBJECT');

Alert Logs: Instance Script 2

# Email the log
echo -e "ERRORS:\n" "`grep -v -f "/home/oracle/scripts/alert_log_match.lst"
$ALERT_LOG_DIR/alert_${sdbi}.log`" "\n\nLOG:\n" "`cat $ALERT_LOG_DIR/alert_${sdbi}.log`" |
mail -s "ORACLE - ${sdbi} ${SUBJECT}: alert log for `date`" db_admins

Alert Logs: Job

Alert Logs: Job

Alert Logs: Report script

####################################################################
#
# This script Produces an Alert Log Report to be sent to the DBAs
#
# Amendments :
#
GDB - 19-JAN-2009
Initial construction
#
####################################################################
# set the rcat environment
. /home/oracle/ora10rcat.sh
sqlplus edb_alert/<pword> <<EOF
@/home/oracle/scripts/alert_log_report.sql
EOF
cat /home/oracle/scripts/alert_log_report.txt | mailx -s "Alert Log Report for `date`" db_admins

Alert Logs: Report SQL

SET ECHO OFF;
SET PAGESIZE 100;
SET FEEDBACK OFF;
SET LINESIZE 120;
SET SQLPROMPT ""
COLUMN database_name FORMAT A20;
COLUMN instance_name FORMAT A20;
COLUMN status FORMAT A20;
SPOOL /home/oracle/scripts/alert_log_report.txt;
SELECT instance.database_name,
instance.instance_name,
alert_log_status.alert_log_status_date,
alert_log_status.status
FROM
instance,
alert_log_status
WHERE instance.database_name = alert_log_status.database_name(+)
AND
instance.instance_name = alert_log_status.instance_name(+)
AND
alert_log_status.alert_log_status_date(+) >= TO_DATE(TO_CHAR(SYSDATE, 'DD-MON-YYYY'), 'DDMON-YYYY')
ORDER BY database_name,
instance_name;

Alert Logs: Emails

Which country is this?

Turkey

Good Practice

Backups: Report

Wiki: Home Page

Wiki: User Notification

Instances: Crontab
User: Oracle
#Backups for the database
00 23 * * 1 /home/oracle/scripts/backup/rmanfullback.sh ngs11 rman <pword> rcat >/dev/null 2>&1
00 23 * * 2-7 /home/oracle/scripts/backup/rmanincback.sh ngs11 rman <pword> rcat >/dev/null 2>&1
#Alert Logs sent out
0 8 * * * /home/oracle/scripts/alert.sh ngs11 ngs111 11 rcat edb_alert <pword> >/dev/null 2>&1
#Housekeeping
0 9 * * * /home/oracle/scripts/ora11housekeep.sh ngs11 ngs111 2>&1

User: Root
#Backup Oracle OCR and Voting Disk
0 3 * * * /root/oscripts/backupOCR
10 3 * * * /root/oscripts/backupVD

Housekeeping
Log rotation
Removing old logs/trace files etc
See my talk at WLCG in 2007:
http://indico.cern.ch/sessionDisplay.py?sessionId=7&slotI
d=0&confId=20080#2007-11-30

Future
Service exceptions
Disaster Recovery
Reporting (long term)
SLS

Which country is this?

Slovakia

Your daily checks?

Questions and (hopefully) Answers

databaseservices@stfc.ac.uk