CASSIS Wave 1

Disaster Recovery Plan

CASSIS Wave 1 R/3 Disaster Recovery Plan

DOCUMENT NAME & LOCATION: VERSION: DATE: READERSHIP: SUMMARY: C1_BAI_DRP TBD 0.01 11/21/05 C1 Project This is the Disaster Recovery Plan (DRP) for C1 Production R/3 PRE.

Associated Documents (this document should be read in conjunction with) Title of Document File Location (refer to the most recent version of the file) Technical Infrastructure Plan Operations Manual

CASSIS Wave 1 Plan

Disaster Recovery

Document Information
Document History

Version
0.01

Author(s)
Anna Wick

Comments/Changes
First Draft

Revision Date
11/21/05

Document Approval

Approver's Name

Department/Group

Date Approved

Document Distribution

Name or Group

Location

Date Distributed

Review (e.g., SQA Review, Peer Review)

Review Date

Comments

Approval Signature

Page 2 of 12

CASSIS Wave 1 Plan

Disaster Recovery

Table of Contents
1. Purpose of this Document................................................................................................................. 5 2. Approach........................................................................................................................................... 5 2.1 Plan Summary and Objectives...................................................................................................... 5 2.2 Design Assumptions...................................................................................................................... 5 2.3 Process Assumptions..................................................................................................................... 5 2.4 Notification Contacts...................................................................................................................... 5 3. SAP R/3 PRE Disaster Recovery Procedures..................................................................................6 3.1 <Basis> Stop the PFE system and Database................................................................................6 3.1.1 Stop Backup if Running............................................................................................................. 6 3.1.2 Shutdown SAP & Oracle Applications........................................................................................6 3.1.3 Stop Oracle Services................................................................................................................. 6 3.1.4 Stop SAP Services..................................................................................................................... 6 3.2 <Basis> Remove old PFE files and rename PFE directories to PRE............................................6 3.2.1 Remove files.............................................................................................................................. 6 3.2.2 Rename directories.................................................................................................................... 6 3.3 <Basis> Central Instance & Application Server Installation...........................................................6 3.4 <Basis> Prepare to restore PRE database to PFE........................................................................7 3.4.1 Set the environment variables for PREadm..............................................................................7 3.4.2 Create the initPRE.utl file........................................................................................................... 7 3.4.3 Regenerate BKI file.................................................................................................................... 7 3.4.4 Obtain the backPRE.log file from tape.......................................................................................8 3.4.5 Obtain the most recent backups logfile from tape.....................................................................8 3.4.6 Obtain the initPRE.ora parameter file from tape:.......................................................................8 3.4.7 Obtain the initPRE.sap parameter file from tape:......................................................................8 3.4.8 Obtain the initPRE.dba parameter file from tape:......................................................................8 3.5 <Basis> Restore PRE Files from Tape.......................................................................................... 8 3.5.1 Restore datafiles........................................................................................................................ 8 3.5.2 Restore Archive logs.................................................................................................................. 8 3.6 <Basis> Recover PRE................................................................................................................... 9 3.6.1 Start Oracle Service for PRE..................................................................................................... 9 3.6.2 Roll forward database until last log............................................................................................9 3.7 <Basis> Open PRE database........................................................................................................ 9 3.8 <Basis & IBM Tulsa> Update /usr/sap/trans .................................................................................9 3.8.1 <Basis> Create /usr/sap/trans directory.....................................................................................9 3.8.2 <IBM Tulsa> Restore /usr/sap/trans...........................................................................................9 3.9 <Basis> Configure Oracle Files and Start Listener........................................................................9 3.10 <Basis & IBM Tulsa> Initial Start & SAP Configuration.............................................................10 3.10.1 <IBM Tulsa> Update DNS...................................................................................................... 10 3.10.2 <Basis> Request SAP license................................................................................................ 10 3.10.3 <Basis> Install New SAP License.......................................................................................... 10 3.10.4 <Basis> Restore SAP profiles from tape...............................................................................10 3.10.5 <Basis> Update SAP profiles as needed...............................................................................10 3.10.6 <Basis> Remove Batch and Spool Processes from the system............................................10 3.10.7 <Basis> Start PRE Central Instance......................................................................................10

Page 3 of 12

CASSIS Wave 1 Plan

Disaster Recovery

3.10.8 <Basis> Lock All users........................................................................................................... 10 3.10.9 <Basis> Import instance profiles............................................................................................ 10 3.10.10 <Basis> Operation Modes.................................................................................................... 10 3.10.11 <Basis> Execute the initial SAP consistency check.............................................................10 3.10.12 <Basis> Check Partner profile in PRE ................................................................................10 3.10.13 <Basis> RFC Server Groups............................................................................................... 11 3.11 <Basis> Final Start of PRE with Batch......................................................................................11 3.11.1 To Activate the batch and spool processes............................................................................11 3.11.2 Resume Batch Scheduling..................................................................................................... 11 3.11.3 Start PRE............................................................................................................................... 11 3.11.4 Setup Login Groups............................................................................................................... 11 3.12 <Basis> Final Steps................................................................................................................... 12 3.12.1 Resume Operations procedures............................................................................................. 12 3.12.2 Configure STMS..................................................................................................................... 12

Page 4 of 12

CASSIS Wave 1 Plan

Disaster Recovery

1.

PURPOSE OF THIS DOCUMENT

This document outlines the Disaster Recovery Plan (DRP) for the CASSIS Wave 1 (C1) R/3 Production instance, PRE. In general, PRE will come up on the PFE hardware at the BRDC in case of Disaster. This document assumes that the database / central instance server for PFE already exist and are set up from an infrastructure standpoint.

2. 2.1

APPROACH Plan Summary and Objectives

This document provides a detailed plan for the recovery of the C1 R/3 Production environment in the event a disaster is declared at the Tulsa Data Center (TDC) located in Tulsa, Oklahoma. The system covered is PRE SAP R/3 (database / central instance server & application server).

2.2

Design Assumptions
The following assumptions are made regarding the SAP disaster recovery procedures: The Primary Site, the Tulsa Data Center (TDC), is located at 509 S. Boston Ave, Tulsa, OK 741034602, and holds the production servers for SAP R/3 PRE. Disaster recovery site will be the Backup Recovery Data Center (BRDC) site, located approximately 8 miles from the TDC at CityPlex, 2488 East 81st, Suite 455, Tulsa, OK 74137. The BRDC is used for development and testing and serves as the disaster recovery site in the event of a disaster. Servers in the BRDC will already host the executables, and interface executables, that exist on the production servers In the event of a disaster, the production fix (PFE) environment will be used to house production. Original PFE will not be available until either a return of the PRE environment to original production hardware, or additional hardware is procured to replace the PFE environment. Network throughput is sufficient to handle traffic between TDC and BRDC. Network routing is in place such that, if the TDC is destroyed, external users could be routed through other pre-established locations (e.g. Omaha). All communication to the PRE SAP R/3 and other associated servers is done via DNS. It is assumed that changing the DNS entry to point to the new locations after disaster will allow all satellite systems to find the environments. This plan does not cover rolling disasters. That is, if the BRDC was hit by disaster at the same time or soon after the TDC was declared under disaster, this plan does not handle that recovery. This plan does not cover loss of the BRDC. If the BRDC were unavailable, there would be no recovery site for the TDC and development processes would cease until the BRDC is restored.

2.3

Process Assumptions
A disaster has been declared at the TDC site.

2.4

Notification Contacts
Once the disaster has been declared, the following process should be followed to notify Application personnel and begin the Disaster Recovery Procedures indicated in this document. The BP Disaster Declaration team will contact the following people to notify them of a disaster.

Overall Contacts to start notification process: TBD HW/OS: IBM Hosting Windows On-Call Networking: IBM Hosting Windows On-Call DBA/Basis: TBD Batch Scheduling: TBD Backup/Recovery: Basis On-Call, IBM Hosting Windows On-Call Business Contact (for recovery completion): TBD

Page 5 of 12

CASSIS Wave 1 Plan

Disaster Recovery

Once all parties have been contacted, the appropriate group will perform the procedures in their sections in the order noted below. When each group in succession has completed their step and verified the steps, they will notify the hand-off organization that will then begin their respective steps.

3.

SAP R/3 PRE DISASTER RECOVERY PROCEDURES

This section provides the process that would be followed in the event that a disaster is declared for the PRE Environment, including the database server and central instance server and/or associated storage.

3.1
3.1.1 3.1.2 3.1.3 3.1.4

<Basis> Stop the PFE system and Database

Stop Backup if Running Check if there is a PFE database backup running, if there is, stop the backup now. Shutdown SAP & Oracle Applications Use SAP Management console to stop SAP Use SAPDBA to shutdown Oracle Stop Oracle Services Stop the following Oracle Services: OracleOraHome92Agent OracleOraHome92TNSListener OracleServicePFE Stop SAP Services Stop the following SAP Services SAPOSCOL SAPPFE_10

3.2

<Basis> Remove old PFE files and rename PFE directories to PRE

3.2.1 Remove files Remove all datafiles under the sapdata directories on each drive where datafiles exist Do not remove the sapdata directories Remove origlog and mirrorlog files Do not remove origlog* and mirrlog*directories Do not remove the cntrl directory under the origlog* and mirrlog* directories 3.2.2 Rename directories Rename all <Drive>:\oracle\PFE to <Drive>:\oracle\PRE

3.3

<Basis> Central Instance & Application Server Installation

Follow the steps below to install SAP on the Central Instance (BP1TULAP113) for PRE. The database host is BP1TULAP192. 1. Start Sapinst on the host: Start -> Programs -> SAPInst Oracle Kernel_PRE -> select SAPinst Server+Gui 2. Welcome window Select SAP R/3 Oracle Default Click on Next 3. License agreement Click on I Agree 4. SAP System Common Parameters SAP System ID: PRE Select Central Instance Click on Next 5. SAP System Common Parameters Instance Number: 10 Instance Host: <BP1TULAP113.amoco.com> Database System ID: PRE Database Host: <BP1TULAP192.amoco.com> Click on Next 6. SAP Instance Host Information Choose Local Installation Ram:<8GB>

Page 6 of 12

CASSIS Wave 1 Plan

Disaster Recovery

Click on Next 7. SAP System Instance Directories Choose the <DRIVE>: for SAP system directory Host with Transport Directory : <BP1TULAP139.amoco.com> (this is the PSE instance) Click Next 8. Database System Common Parameters Database Charector Set: WE8DEC Database schema : SAPDAT DB Home : < E:\oracle\ora92> 9. User Information Name: preadm Password: Enter Password Confirm: Confirm Password 10. User Information Name:SAPservicePRE Password: Enter Password Confirm: Confirm Password 11. CD Browser CD Kernel : Enter the location of the Kernel CD 12. Monitor the Installation 13. Filesystem Nodes Check the filesystems/directories Click on Next 14. Extraction of SAP System Executables Select both Click on Next 15. Extraction of Oracle Software Select extract oracle client software Click on Next 16. SAP System Service Ports Enter/Check Port numbers (10) Click on Next 17. Monitor the Installation

3.4

<Basis> Prepare to restore PRE database to PFE

3.4.1 Set the environment variables for PREadm Go to Command Prompt Set Oracle_SID=PRE Set SAPDATA_HOME=F:\oracle\PRE Set DSMI_CONFIG= e:\usr\sap\PRE\SYS\exe\run\dsm_opt\.tsmsrv3.opt Note: Do not close the command prompt session 3.4.2 Create the initPRE.utl file Use the same command prompt session to execute the following command in order to create the initPRE.utl file on server BP1TULAP192 copy e:\oracle\ora92\database\initPFE.utl e:\oracle\ora92\database\initPRE.utl Edit the file e:\oracle\ora92\database\initPRE.utl and update the following: o Change all occurrences of PFE to PRE o Change MAX_SESSIONS to 8 o Change MAX_BACK_SESSIONS to 4 and uncomment o Change MAX_ARCH_SESSIONS to 4 and uncomment o Change MAX_RESTORE_SESSIONS to 4 and uncomment o Change SERVER to tsmsrv3 o Change SESSIONS to 8 o Change ADSMNODE to BP1TULAP131_PRE o Change BRBACKUPMGTCLASS to A35_PRE 3.4.3 Regenerate BKI file

Page 7 of 12

CASSIS Wave 1 Plan

Disaster Recovery

Use the same command prompt session to execute the following command in order to create the initPRE.bki file: backint p e:\oracle\ora92\database\initPRE.utl f password Note: - Get the password from basis pool members 3.4.4 Obtain the backPRE.log file from tape Login to the database server as PREadm backint p e:\oracle\ora92\database\initPRE.utl f inquire Enter # NULL e:\oracle\PRE\sapbackup\backPRE.log backint p e:\oracle\ora92\database\initPRE.utl f restore Enter <BackupID> e:\oracle\PRE\sapbackup\backPRE.log Wait while the backPRE.log file is restored into the directory e:\oracle\PRE\sapbackup 3.4.5 Obtain the most recent backups logfile from tape View the file e:\oracle\PRE\sapbackup\backPRE.log to find the name of the most recent complete PRE backup log. backint p e:\oracle\ora92\database\initPRE.utl f inquire Enter #NULL e:\oracle\PRE\sapbackup\<backup_log> The system will display a list of backups that hold this file, with the most recent one on top. Note the backupID for the most recent copy of <backup_log>. backint p e:\oracle\ora92\database\initPRE.utl f restore Enter <BackupID> e:\oracle\PRE\sapbackup\<backup_log> Wait while the backup log is restored into the directory e:\oracle\PRE\sapbackup 3.4.6 Obtain the initPRE.ora parameter file from tape: backinit p e:\oracle\ora92\database\initPRE.utl f inquire Enter # NULL \oracle\ora92\database\initPRE.ora The system will display a list of backups that hold this file, with the most recent one on top. Note the BackupID for the most recent copy of initPRE.ora. backint p e:\oracle\ora92\database\initPRE.utl f restore Enter <BackupID> e:\oracle\ora92\database\initPRE.ora Wait while the initPRE.ora file is restored into the directory e:\oracle\ora92\database 3.4.7 Obtain the initPRE.sap parameter file from tape: backinit p e:\oracle\ora92\database\initPRE.utl f inquire Enter # NULL e:\oracle\ora92\database\initPRE.sap The system will display a list of backups that hold this file, with the most recent one on top. Note the BackupID for the most recent copy of initPRE.sap. backint p e:\oracle\ora92\database\initPRE.utl f restore Enter <BackupID> e:\oracle\ora92\database\initPRE.sap Wait while the initPRE.sap file is restored into the directory e:\oracle\ora92\database 3.4.8 Obtain the initPRE.dba parameter file from tape: backinit p e:\oracle\ora92\database\initPRE.utl f inquire Enter # NULL e:\oracle\ora92\database\initPRE.dba The system will display a list of backups that hold this file, with the most recent one on top. Note the BackupID for the most recent copy of initPRE.dba. backint p e:\oracle\ora92\database\initPRE.utl f restore Enter <BackupID> e:\oracle\ora92\database\initPRE.dba Wait while the initPRE.dba file is restored into the directory e:\oracle\ora92\database

3.5

<Basis> Restore PRE Files from Tape

3.5.1 Restore datafiles Use the same command prompt session to execute the following commands: brrestore -b <backuplog.anf> -d util_file -r e:\oracle\ora92\database\initPRE.utl -m full -l E This action should restore all datafiles and controlfiles to the system when complete, verify the restore completed successfully 3.5.2 Restore Archive logs Restore remaining PRE archive logs to PFE by logging in to the database servers View the backup log file. Note the number next to Current log sequence at the top of the log file. This will be the first log needed, and the number you will use in the command below for <first log>.

Page 8 of 12

CASSIS Wave 1 Plan

Disaster Recovery

Its necessary to restore as many logs as are on tape, so work with with the <lastlog> number accordingly in order to retrieve all logs that exist from tape. brrestore -a <start_log#>-<end_log#> -c -d util_file -r e:\oracle\ora92\database\initPRE.utl The archivelogs will be restored into F:\oracle\PRE\oraarch

3.6
3.6.1 3.6.2

<Basis> Recover PRE

Start Oracle Service for PRE Start oracle service OracleServicePRE using computer management Roll forward database until last log SQL> recover database using backup controlfile until cancel; Enter auto at the prompt to have the database automatically roll through all archivelogs in \oracle\PRE\oraarch. Continue to roll forward until all available PRE archivelogs have been applied. Note: If recover database fails with the following error: ORA-00283: recovery session canceled due to errors. ORA-01244: unnamed datafile(s) added to control file by media recovery ORA-01110: datafile <####>:<full path of new datafile name> Create a subdirectory for the data file under the appropriate sapdata filesystem. Then go back into server manager and enter the following command: SQL> alter database create datafile '<full path of new datafile name> AS 'E:\oracle\PRE\920_64\database\UNNAMED0<####>; Where <full path of new datafile name> and <####> here is the same as <full path of new datafile name> and <####> in the error message above. From this point you should be able to resume rolling forward by reissuing the recover database command above.

3.7

<Basis> Open PRE database

In this step the database logs are reset; this is a key activity; if it fails the database restore and log recovery need to be done over again. Login to the database server as PREadm sqlplus /nolog SQL> connect / as sysdba SQL> alter database open resetlogs; The result should be a message saying, Database opened Verify the database opened successfully Please check the oracle alert log for more info. SQL> shutdown SQL> startup Ensure that normal shutdown and startup work correctly.

3.8

<Basis & IBM Tulsa> Update /usr/sap/trans

The /usr/sap/trans filesystem may have to be recreated and restored depending on the nature of the disaster. (Currently the PSE system, BP1TULAP139, holds the /usr/sap/trans filesystem.) 3.8.1 <Basis> Create /usr/sap/trans directory Create the directory /usr/sap/trans 3.8.2 <IBM Tulsa> Restore /usr/sap/trans It may be necessary Restore the contents of the last /usr/sap/trans backup into the new directory: Login to the new server housing /usr/sap/trans as admin dsmc restore server=tsmsrv3 virtualnode=tdcsapdn subdir=yes /usr/sap/trans/*

3.9

<Basis> Configure Oracle Files and Start Listener

Login to the database server as PREadm Open tnslistener file Change all occurrences of PFE to PRE in this file Ensure that the listener is configured to run on port 1521

Page 9 of 12

CASSIS Wave 1 Plan

Disaster Recovery

3.10

<Basis & IBM Tulsa> Initial Start & SAP Configuration

3.10.1 <IBM Tulsa> Update DNS Update any DNS entries pointing to BP1TULAP181 & BP1TULAP136 to now point to BP1TULAP192 & BPTULAP113 respectively. Currently known DNS entry is: SAPPRE pointing to BP1TULAP181 3.10.2 <Basis> Request SAP license An SAP license must be requested to run PRE on BP1TULAP113 instead of BP1TULAP181 Obtain this new license from SAP via the OSS. To request an emergency temporary license, call 1-800-677-7271. 3.10.3 <Basis> Install New SAP License Login to the central instance as PREadm C:\> saplicense show C:\> saplicense delete Input the information requested to delete the old license referencing PFE C:\> saplicense install Input the information requested for the new license 3.10.4 <Basis> Restore SAP profiles from tape backinit p e:\oracle\ora92\database\initPRE.utl f inquire Enter # NULL <full name and path of profile> The system will display a list of backups that hold this file, with the most recent one on top. Note the BackupID for the most recent copy of the profile you are looking for. backint p e:\oracle\ora92\database\initPRE.utl f restore Enter <BackupID> <full name and path of profile> Wait while the profile file is restored Repeat this process for all profiles needed (PRE_DVEBMGS10_<hostname>, DEFAULT, START_DVEBMGS10_<hostname>, etc.) 3.10.5 <Basis> Update SAP profiles as needed Compare the SAP profiles installed as part of the installation process to the profiles just restored from tape Update the profiles accordingly 3.10.6 <Basis> Remove Batch and Spool Processes from the system On the central instance, BP1TULAP113, edit the Central Instance SAP profiles commenting out the lines, which reference rdisp/wp_no_btc and rdisp/wp_no_spo. The SAP profiles can be found on the central instance server in e:\sapmnt\PRE\profile 3.10.7 <Basis> Start PRE Central Instance Logon to the central instance BP1TULAP113 as user PREadm Ensure that /usr/sap/trans is mounted to the system Start SAP using SAP MMC 3.10.8 <Basis> Lock All users Using transaction su01, perform a mass change to lock users. 3.10.9 <Basis> Import instance profiles Call transaction RZ10. Import the instance profiles by selecting Utilities > Import profiles > of active servers from the top menu. 3.10.10 <Basis> Operation Modes Go to transaction rz04 Change the existing Operation Modes to reflect the hostnames of the new servers. 3.10.11 <Basis> Execute the initial SAP consistency check Call transaction SICK. The only error reported should be one stating that there are no batch processes currently defined. This error is acceptable since there shouldnt be any batch process online yet. 3.10.12 <Basis> Check Partner profile in PRE Check RFC destinations with in SM59 Go to WE20 Select partner 'PRECLNT100' Check the RFC destination for port to 'PRECLNT100' Check all the outbound and inbound parameters

Page 10 of 12

CASSIS Wave 1 Plan

Disaster Recovery

Activate the partner profile (if required) Go to WE20 Select partner BP_TP5_100 Check the RFC destination for port to A000001 Check all the outbound and inbound parameters Activate the partner profile (if required) 3.10.13 <Basis> RFC Server Groups Call transaction RZ12 Recreate all PRE Server groups related to old hostnames with new hostnames Double click on bp1tulap181_PRE_30 Enter parallel_generators as server group Click on Save

3.11

<Basis> Final Start of PRE with Batch

IMPORTANT: BEFORE DOING THIS STEP COORDINATE WITH THE BUSINESS / CASSIS Wave 1 PROJECT SUPPORT TEAM. These teams may ask you to do any or all of the following steps in any order: 3.11.1 To Activate the batch and spool processes Shutdown the Central Instance. Modify the Central Instance SAP profile, un-commenting the entries, which reference rdisp/wp_no_btc and rdisp/wp_no_spo. 3.11.2 Resume Batch Scheduling The Escalation Process Team will be assembled to identify all tasks surrounding the outage. The following tasks will be performed after an outage to catch up processing: Perform reconfiguration tasks for any interfaces that need it. Work with the Functional Experts to analyze the queue of jobs that are waiting to be processed Check for sequence/dependency issues. Reschedule dependent jobs to run after predecessor. Determine business critical jobs that must run first. Coordinate application server start-up with the Basis Team If the outage is extensive (>12 hours), the jobs that are allowed to process may have to be limited, based on job class. Basis will modify the application server settings to allow only class A jobs to process first, then class B and C. For a shorter outage, Basis may be asked to bring up all application servers with full batch processing. Once the application servers are up, with full batch processing, and all jobs have been reviewed, the Functional Experts manually catch up the batch schedule by running jobs that were missed during the outage. 3.11.3 Start PRE Start PRE Central Instance Logon to the central instance, BP1TULAP113 as user PREadm Ensure that /usr/sap/trans is mounted to the system C:/> startsap To modify batch processes to run Class A jobs only: Go to transaction rz04 Double click on the current op mode running that you want to modify Double click on the current operation mode listed under the app instance you want to modify Change the batch jobs so that the numbers of class A batch jobs is equal to the total number of batch jobs Save your changes Go to transaction rz03 Double click on the app instance you changed so its highlighted From the menu, choose Control > Switch Op Mode > of selected servers 3.11.4 Setup Login Groups Go to transaction SMLG. Change the existing login groups to reflect the new servers.

Page 11 of 12

CASSIS Wave 1 Plan

Disaster Recovery

3.12
3.12.1

<Basis> Final Steps

Resume Operations procedures Reconfigure TSM Server for Backups: Login to BP1TULAP192 E:\> cd /oracle/PRE/920_64/database Update the file initPRE.utl and change ADSMNODE to BP1TULAP113_PRE Start Online Backup Use DB13 Schedule immediate online backup Schedule Archive log backups and Online Backups Use DB13 Schedule Archive log backups Hourly Schedule Online Backups Every Week 3.12.2 Configure STMS Login to PSE client 000 as user sap* Go to STMS Click on systems Select PFE Click on Delete Distribute and update the configuration Login to PRE client 0000 as user sap* Go to STMS You will get a popup window. Confirm that transport domain is DOMAIN_PSE and domain controller is PSE Now click on Save Login to GRM client 000 as user sap* Go to STMS Click on systems Select PRE Click on Approve Distribute and update the configuration Click on Transport routes Click on Change Select Standard configuration-> Three system in Group from the menu Enter the following, Development system: PSE Quality assurance: PFE Production system: PRE Click on Save Distribute and update the configuration

Page 12 of 12