I have had a response on the Oracle SR to request that other LOS perform the same

checks as they did for the EBUSPROD issue, I've pasted their update at the bottom.
For reference EBUSHA is single instance standby database sat on urrtw41, it
reported it could not contact multiple disks from 07:48 today. I list a few but
attach the relevant part of the alert log for exact details and timestamps.
On a daily basis at 7pm it is shutdown, its disk unmounted from thr rtw prod nodes
and mounted on urrtw01a/01b and an rman duplicate activity is performed to created
our daily clone. EBUSHA is normally back on urrtw41 at about 2am each day and
catches up with EBUSPROD by 4am.

WARNING: IO Failed. subsys:System dg:4, diskname:/dev/oracle/evadisk022

disk:0x0.0xf0d708d2
WARNING: IO Failed. subsys:System dg:5, diskname:/dev/oracle/xp12disk088
disk:0x8d.0xf0d708b4
WARNING: IO Failed. subsys:System dg:5, diskname:/dev/oracle/xp12disk100
disk:0x63.0xf0d708c2

Oracle request...
+ Consider all the checks we discussed so far for the issue on Prod case, on the
standby machines as well. Starting with identifying the disks, and checking OS logs
and storage logs, kernel parameter setting,

Let me know the feedback from the other LOS and I can update the SR with anything
you feel relevant
Sarah
urrtw41:[root::/home/sverma55]# more disk_alert
Archived Log entry 22823 added for thread 4 sequence 5470 ID 0xfeb3f00b dest 1:
Media Recovery Log
+EBUSHA_FRA/ebusha/archivelog/2013_03_27/thread_4_seq_5470.2583.811151269
Media Recovery Log
+EBUSHA_FRA/ebusha/archivelog/2013_03_27/thread_1_seq_5204.452.811150987
Wed Mar 27 07:48:02 2013
Archived Log entry 22824 added for thread 1 sequence 5205 ID 0xfeb3f00b dest 1:
Wed Mar 27 07:48:14 2013
Media Recovery Log
+EBUSHA_FRA/ebusha/archivelog/2013_03_27/thread_1_seq_5205.802.811151273
Wed Mar 27 07:48:32 2013
Media Recovery Waiting for thread 1 sequence 5206 (in transit)
Wed Mar 27 08:00:48 2013
Wed Mar 27 08:00:48 2013
Wed Mar 27 08:00:48 2013
Wed Mar 27 08:00:48 2013
Wed Mar 27 08:00:48 2013
Errors in file
/apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3049.trc:
ORA-27063: number of bytes read/written is incorrect
Errors in file
/apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3044.trc:
ORA-27063: number of bytes read/written is incorrect
Errors in file
/apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3053.trc:
ORA-27072: File I/O error
HPUX-ia64 Error: 11: Resource temporarily unavailable
Additional information: 4
Additional information: 3088
Additional information: -1
WARNING: IO Failed. subsys:System dg:4, diskname:/dev/oracle/evadisk022
disk:0x0.0xf0d708d2
au:14404
Errors in file
/apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3057.trc:
ORA-27063: number of bytes read/written is incorrect
WARNING: IO Failed. subsys:System dg:5, diskname:/dev/oracle/xp12disk088
disk:0x8d.0xf0d708b4
au:24962Errors in file /apps/EBUSHA/ora
cle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3059.trc:
ORA-27072: File I/O error
HPUX-ia64 Error: 11: Resource temporarily unavailable
Wed Mar 27 07:48:14 2013
Media Recovery Log
+EBUSHA_FRA/ebusha/archivelog/2013_03_27/thread_1_seq_5205.802.811151273
Wed Mar 27 07:48:32 2013
Media Recovery Waiting for thread 1 sequence 5206 (in transit)
Wed Mar 27 08:00:48 2013
Wed Mar 27 08:00:48 2013
Wed Mar 27 08:00:48 2013
Wed Mar 27 08:00:48 2013
Wed Mar 27 08:00:48 2013
Errors in file
/apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3049.trc:
ORA-27063: number of bytes read/written is incorrect
Errors in file
/apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3044.trc:
ORA-27063: number of bytes read/written is incorrect
Errors in file
/apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3053.trc:
ORA-27072: File I/O error
HPUX-ia64 Error: 11: Resource temporarily unavailable
Additional information: 4
Additional information: 3088
Additional information: -1
WARNING: IO Failed. subsys:System dg:4, diskname:/dev/oracle/evadisk022
disk:0x0.0xf0d708d2
au:14404
Errors in file
/apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3057.trc:
ORA-27063: number of bytes read/written is incorrect
WARNING: IO Failed. subsys:System dg:5, diskname:/dev/oracle/xp12disk088
disk:0x8d.0xf0d708b4
au:24962Errors in file
/apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3059.trc:
ORA-27072: File I/O error
HPUX-ia64 Error: 11: Resource temporarily unavailable
Additional information: 4
Additional information: 3088
Additional information: -1

Wed Mar 27 08:00:48 2013

WARNING: IO Failed. subsys:System dg:5, diskname:/dev/oracle/xp12disk100
disk:0x63.0xf0d708c2 au:3
iop:0x9fffffffbefbd320 bufp:0x9fffffffbf40fe00 offset(bytes):15103973376
iosz:1024
operation:
2(Write) synchronous:0

WARNING: IO Failed. subsys:System dg:5, diskname:/dev/oracle/xp12disk100

disk:0x63.0xf0d708c2 au:3
iop:0x9fffffffbef9c880 bufp:0x9fffffffbf5f7e00 offset(bytes):3162112
iosz:16384
operation:1(Read) synchronous:1
iop:0x9fffffffbefbcff0 bufp:0x9fffffffbf40fe00 offset(bytes):26175529984
iosz:57344
operation:2(Write) synchronous:0Wed Mar 27 08:00:48 2013

iop:0x9fffffffbef9d320 bufp:0x9fffffffbf5f7e00 offset(bytes):3162112

iosz:16384
operation:1(Read) synchronous:1
Errors in file
/apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3055.trc:
ORA-27072: File I/O error
HPUX-ia64 Error: 11: Resource temporarily unavailable
Additional information: 4
Additional information: 14352
Additional information: -1
Errors in file
/apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3046.trc:
ORA-27063: number of bytes read/written is incorrect
WARNING: IO Failed. subsys:System dg:4, diskname:/dev/oracle/evadisk022
disk:0x0.0xf0d708d2
au:10449 result: 4 osderr:0x0 osderr1:0x0 pid:3059 result: 4 osderr:0x0
osderr1:0x0
pid:3044
result: 4 osderr:0x0 osderr1:0x0 pid:3053Errors in file
/apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3044.trc:
ORA-15080: synchronous I/O operation to a disk failed

WARNING: IO Failed. subsys:System dg:4, diskname:/dev/oracle/evadisk024

disk:0x2.0xf0d708d4 au:14

WARNING: IO Failed. subsys:System dg:5, diskname:/dev/oracle/xp12disk083

disk:0x95.0xf0d708aa
au:25034 iop:0x9fffffffbef9d210 bufp:0x9fffffffbc722e00 offset(bytes):10956650496
iosz:968704
operatio
n:1(Read) synchronous:1
result: 4 osderr:0x0 osderr1:0x0 pid:3049 iop:0x9fffffffbef9d430
bufp:0x9fffffffbf5f7e00 offset(bytes):14696448 iosz:16384 operation:1(Read)
synchronous:1WARNING:
failed to read mirro
r side 1 of virtual extent 0 logical extent 0 of file 484 in group 5 from disk
EBUSHA_DATA_0099
allocation unit 3 reason error; if possible,will try another mirror side

result: 4 osderr:0x0 osderr1:0x0 pid:3057

result: 4 osderr:0x0 osderr1:0x0 pid:3055 iop:0x9fffffffbefbccc0
bufp:0x9fffffffbf40fe00 offset(bytes):26250451968 iosz:1024 operation:2(Write)
synchronous:0
WARNING: failed to write mirror side 1 of virtual extent 231 logical extent 0 of
file 892 in group
4 on disk 0 allocation unit 14404
WARNING: failed to read mirror side 1 of virtual extent 0 logical extent 0 of file
484 in group 5
from disk EBUSHA_DATA_0099 allocation unit 3 reason error; if possible,will try
another mirror
side
result: 4 osderr:0x0 osderr1:0x0 pid:3046WARNING: failed to read mirror
side 1 of virtual
extent 95 logical extent 0 of file 1795 in group 4 from disk EBUSHA_FRA_0022
allocation unit
10449
reason error; if possible,will try another mirror side

Errors in file
/apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3049.trc:
ORA-27063: number of bytes read/written is incorrect
Errors in file
/apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3046.trc:
ORA-27063: number of bytes read/written is incorrect
NOTE: successfully read mirror side 1 of virtual extent 0 logical extent 0 of file
484 in group
[5.4046851930] from disk EBUSHA_DATA_0099 allocation unit 3 WARNING: IO Failed.
subsys:System
dg:4, di
skname:/dev/oracle/evadisk022 disk:0x0.0xf0d708d2 au:10449NOTE: successfully read
mirror side 1 of
virtual extent 0 logical extent 0 of file 484 in group [5.4046851930] from disk
EBUSHA_DATA_0099
al
location unit 3

WARNING: failed to read mirror side 1 of virtual extent 0 logical extent 0 of file
909 in group 4
from disk EBUSHA_FRA_0024 allocation unit 14 reason error; if possible,will try
another mirror
side
Errors in file
/apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3044.trc:
ORA-00345: redo log write error block 236822 count 1
ORA-00312: online log 73 thread 4:
'+EBUSHA_FRA/ebusha/onlinelog/group_73.892.808572333'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
WARNING: IO Failed. subsys:System dg:4, diskname:/dev/oracle/evadisk024
disk:0x2.0xf0d708d4
au:14402Wed Mar 27 08:00:48 2013

ERROR: cannot read disk header of disk EBUSHA_DATA_0099 (99:4040624322)

iop:0x9fffffffbefbccc0 bufp:0x9fffffffbee0fe00 offset(bytes):10957609984

iosz:9216
operation:2(Write) synchronous:0
Errors in file
/apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3051.trc:
ORA-27063: number of bytes read/written is incorrect
Wed Mar 27 08:00:48 2013
Errors in file
/apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_ckpt_2837.trc:
ORA-27063: number of bytes read/written is incorrect
WARNING: IO Failed. subsys:System dg:4, diskname:/dev/oracle/evadisk024
disk:0x2.0xf0d708d4
au:10902 result: 4 osderr:0x0 osderr1:0x0 pid:3049 iop:0x9fffffffbefbc440
bufp:0x9fffffffbf00fe0
0 offset(bytes):15101993984 iosz:1024 operation:2(Write) synchronous:0
NOTE: process 3059 initiating offline of disk 99.4040624322 (EBUSHA_DATA_0099) with
mask 0x7e in
group 5

result: 4 osderr:0x0 osderr1:0x0 pid:3046 iop:0x9fffffffbefbcaa0

bufp:0x9fffffffbf40fe00 offset(bytes):11432335360 iosz:2048 operation:2(Write)
synchronous:0
Errors in file
/apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3049.trc:
ORA-15080: synchronous I/O operation to a disk failed

result: 4 osderr:0x0 osderr1:0x0 pid:3051

Errors in file
/apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3046.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 95 logical extent 0 of
file 1795 in group
4 on disk 0 allocation unit 10449
WARNING: failed to write mirror side 1 of virtual extent 79 logical extent 0 of
file 515 in group
4 on disk 2 allocation unit 14402
NOTE: successfully read mirror side 1 of virtual extent 0 logical extent 0 of file
909 in group
[4.2119569248] from disk EBUSHA_FRA_0024 allocation unit 14
Errors in file
/apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3049.trc:
ORA-15080: synchronous I/O operation to a disk failed
Errors in file
/apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3051.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 95 logical extent 0 of
file 746 in group
5 on disk 141 allocation unit 24962
WARNING: failed to write mirror side 1 of virtual extent 270 logical extent 0 of
file 1264 in
group 4 on disk 2 allocation unit 10902
ERROR: cannot read disk header of disk EBUSHA_FRA_0024 (2:4040624340)
NOTE: process 3055 initiating offline of disk 2.4040624340 (EBUSHA_FRA_0024) with
mask 0x7e in
group 4
Errors in file
/apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3049.trc:
ORA-00345: redo log write error block 98233 count 56
ORA-00312: online log 53 thread 2:
'+EBUSHA_DATA/ebusha/onlinelog/group_53.746.808572245'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
Errors in file
/apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3051.trc:
ORA-00345: redo log write error block 277222 count 2
ORA-00312: online log 11 thread 1:
'+EBUSHA_FRA/ebusha/onlinelog/group_11.1264.808572261'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
NOTE: successfully read mirror side 1 of virtual extent 95 logical extent 0 of file
1795 in group
[4.2119569248] from disk EBUSHA_FRA_0022 allocation unit 10449
WARNING: IO Failed. subsys:System dg:4, diskname:/dev/oracle/xpdisk148
disk:0x3.0xf0d708d1 au:4
Errors in file
/apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3046.trc:
ORA-15080: synchronous I/O operation to a disk failed
iop:0x9ffffffffd40c000 bufp:0x9ffffffffd5e7e00 offset(bytes):4489216
iosz:16384
operation:2(Write) synchronous:0
WARNING: failed to write mirror side 1 of virtual extent 79 logical extent 0 of
file 769 in group
5 on disk 149 allocation unit 25034
result: 4 osderr:0x0 osderr1:0x0 pid:2837
Errors in file
/apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_ckpt_2837.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 104 logical extent 0 of
file 909 in group
4 on disk 3 allocation unit 4
Errors in file
/apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3046.trc:
ORA-00345: redo log write error block 81287 count 1
ORA-00312: online log 63 thread 3:
'+EBUSHA_DATA/ebusha/onlinelog/group_63.769.808572319'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
Errors in file
/apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_ckpt_2837.trc:
ORA-00206: error in writing (block 6786, # blocks 1) of control file
ORA-00202: control file: '+EBUSHA_FRA/ebusha/control02.dbf'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
Errors in file
/apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_ckpt_2837.trc:
ORA-00221: error on write to control file
ORA-00206: error in writing (block 6786, # blocks 1) of control file
ORA-00202: control file: '+EBUSHA_FRA/ebusha/control02.dbf'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
CKPT (ospid: 2837): terminating the instance due to error 221
Wed Mar 27 08:00:48 2013
System state dump is made for local instance
System State dumped to trace file
/apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_diag_2803.trc
Instance terminated by CKPT, pid = 2837