Eux210 S2re Refresh Issues & Resolutions/Workarounds

Adopted By Lodh Biswajitt for ST WWDC

Database S2RE residing in the Esicom 2 reporting environment is an exact replica of the Esicom 2 Live Database ( on eux010, running on VCS cluster). The database S2RE is refreshed online from its Live environment every 6 hours, every day automatized through the maestro schedule SY-COPR@. The schedule details along with brief description & Team(s) responsible are as follows :-

Schedule Name :Job Name

EUX210#SY-COPR@ Desc

Owner System/prodops DBSL DBSL System/prodops DBSL Storage SL DBSL Storage SL DBSL System/prodops DBSL DBSL/ARTP/Appilcation owner DBSL/ARTP/Appilcation owner

Runs on eux010 to create a flag to (EUX010#)CRTFLAGS2RE indicate the start of the refresh flow ORKILLRE ORSTOPRE DBEXPRE (EUX010#)EXTRARE SYNCHRE (EUX010#)ORBEGS2 SPLITRE (EUX010#)ORENDS2 DBIMPRE ORSTARRE Kill all Oracle sessions on eux210 Stop Oracle Db on eux210 Unmount & export all oracle filesystems Extracts information related to the current DB structure Synchronizes the disks at the storage Runs on eux010 to put the Live DB s2 in backup mode Splits the disks at the storage Runs on eux010 to bring the Live DB s2 out of backup mode Imports & mounts all oracle filesystems Start/recover/rename the Oracle database. Scripts/sqls to change various DB characteristics to suit the reporting environment Scripts/sqls to change various DB characteristics to suit the reporting environment Runs on eux010 to remove a flag to indicate the completion of the refresh flow

PATCHRE

PATCHRE2

(EUX010#)RMFLAGS2RE

System/prodops

Generic Issues & Workarounds:1. EUX210#SY-COPR3.PATCHRE is running for more than 3 hrs Action :- Please invoke ARTP or Application Team or DBSL.

2. PATCHRE/PTACHRE2 failed with errors like :ORA-01400: cannot insert NULL into ("SICOMDBA"."PRODUCT"."PRODUCT_LINE__CODE") ORA-00955: name is already used by an existing object ORA-00955: name is already used by an existing object ORA-01418: specified index does not exist ORA-00955: name is already used by an existing object

Action :- Please invoke ARTP or Application Team

3. ORSTARRE failed with errors like :/home210/orare/sy-copxx.orstarxx.log:ORA-00308: cannot open archived log '/ora_re/arch/s2re/1_596416_415958725.dbf' /home210/orare/sy-copxx.orstarxx.log:ORA-27037: unable to obtain file status /home210/orare/sy-copxx.orstarxx.log:ORA-00308: cannot open archived log 'select' /home210/orare/sy-copxx.orstarxx.log:ORA-27037: unable to obtain file status /home210/orare/sy-copxx.orstarxx.log:ORA-00308: cannot open archived log 'spool' /home210/orare/sy-copxx.orstarxx.log:ORA-27037: unable to obtain file status /home210/orare/sy-copxx.orstarxx.log:ORA-00308: cannot open archived log '!grep' /home210/orare/sy-copxx.orstarxx.log:ORA-27037: unable to obtain file status /home210/orare/sy-copxx.orstarxx.log:ORA-00308: cannot open archived log 'spool' /home210/orare/sy-copxx.orstarxx.log:ORA-27037: unable to obtain file status /home210/orare/sy-copxx.orstarxx.log:ORA-00308: cannot open archived log 'ALTER' /home210/orare/sy-copxx.orstarxx.log:ORA-27037: unable to obtain file status /home210/orare/sy-copxx.orstarxx.log:ORA-00308: cannot open archived log 'shutdown' /home210/orare/sy-copxx.orstarxx.log:ORA-27037: unable to obtain file status /home210/orare/sy-copxx.orstarxx.log:ORA-00308: cannot open archived log 'exit' /home210/orare/sy-copxx.orstarxx.log:ORA-27037: unable to obtain file status

Action :- Restore the missing archives from backup or else resubmit the entire sy-copr@. The rationale behind this is that the time taken for restoring archives from backup will be more or less same compared to the rerun of the entire schedule.

4. ORSTARRE failed with errors like :rm: /ora_re/software/bin/sqlplusO not removed. Text file busy error 2 at line nb 301 executing : rm -f ${ORACLE_HOME}/bin/*O ${ORACLE_HOME}/bin/*0

Action :-Check for processes running from borneo application or do a ps ef|grep sqlplus to find sessions holding locks on oracle files. Kill them and rerun job.

5. ORSTARRE failed with errors like :ORA-01503: CREATE CONTROLFILE failed ORA-01200: actual file size of 304768 is smaller than correct size of 306048 blocks ORA-01110: data file 97: '/ora_re/data02/s2re/referential_sma03.dbf'

Action :- Invoke DBSL

6. ORSTOPRE in hanged state or failed with errors or running long :Action :- Invoke DBSL.

7. ORSTARRE failed with errors like :ERROR at line 1: ORA-01113: file 1 needs media recovery ORA-01110: data file 1: '/ora_re/sys/s2re/sys.dbf'

Action :- Invoke DBSL

8. ORSTARRE failed with errors like :/home210/orare/sy-copxx.orstarxx.log:ORA-01244: unnamed datafile(s) added to control file by media recovery /home210/orare/sy-copxx.orstarxx.log:ORA-01110: data file 122: '/oracle/index01/s2/logistics_big_idx12.dbf'

Action :- Invoke DBSL

9. ORSTARE failed with errors like :/home210/orare/sy-copxx.orstarxx.log:ORA-01503: CREATE CONTROLFILE failed /home210/orare/sy-copxx.orstarxx.log:ORA-01565: error in identifying file '/ora_re/user_temp/s2re/user_temp2.dbf'

Action :- Invoke DBSL

10. ORSTARE failed with errors like :ORA-00227: corrupt block detected in control file: (block 1, # blocks 1) ORA-00202: control file: '/ora_re/ctrl/s2re/ctrl1.ctl' ORA-00471: DBWR process terminated with error ORA-00469: CKPT process terminated with error

Action :- Invoke DBSL

11. ORBEGS2 or ORENDS2 failed with errors like :Tablespace not in proper state to be backed up or tablespace in read only mode

Action :- check for simultaneous runs of the schedule eux010#SY-BCKUP and rerun job once BEGIN010/END010 is in success. This generally happens due to the changed backup schedule timings on eux010 and comes only during the first run of the SY-COPR@ schedule of the day.

PS :- Above is a list of all known errors or list of errors encountered so far. SO dont hesitate to inform me of any new errors during run of the eux210#SY-COPR@ schedule.