Unable to Bring Up RAC Clusterware: ORA-15081: failed to submit an I/O operation to a disk, ORA-27072: File I/O error

advertisements

_____________________________________________________________________________________________________________________

Error Description

ASM instance is down with following error on one oracle RAC node.

Thu Sep 22 00:43:44 2016
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_18791.trc:
ORA-15081: failed to submit an I/O operation to a disk
ORA-27072: File I/O error

Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4816896
Additional information: 4294967295

When we try to bring up the cluster on second node, it fails at Start of 'ora.cssd' on 'Node2' failed

[root@Node2 trace]# crsctl start cluster
CRS-2672: Attempting to start 'ora.crf' on 'Node2'
CRS-2672: Attempting to start 'ora.cssd' on 'Node2'
CRS-2672: Attempting to start 'ora.diskmon' on 'Node2'
CRS-2676: Start of 'ora.diskmon' on 'Node2' succeeded
CRS-2676: Start of 'ora.crf' on 'Node2' succeeded
CRS-2674: Start of 'ora.cssd' on 'Node2' failed
CRS-2679: Attempting to clean 'ora.cssd' on 'Node2'
CRS-2681: Clean of 'ora.cssd' on 'Node2' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'Node2'
CRS-2677: Stop of 'ora.crf' on 'Node2' succeeded
CRS-4000: Command Start failed, or completed with errors.
Ocrcheck command fails with following error message.
[root@Node2 trace]# ocrcheck
PROT-602: Failed to retrieve data from the cluster registry
PROC-26: Error while accessing the physical storage Storage layer error [Insuffi                                      cient quorum to open OCR devices] [0]

Also, in ocssd logfile (ocssd.trc) shows following error message.

2016-10-17 09:33:33.359039 :    CSSD:3699062528: clssnmvDiskVerify: Successful discovery of 0 disks
2016-10-17 09:33:33.359041 :    CSSD:3699062528: clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
2016-10-17 09:33:33.359042 :    CSSD:3699062528: clssnmvFindInitialConfigs: No voting files found
2016-10-17 09:33:33.359231 :    CSSD:3699062528: (:CSSNM00070:)clssnmCompleteInitVFDiscovery: Voting file not found. Retrying discovery in 15 seconds

advertisements

 

Solution description

When I tried to query the ASM disks using following commands it gives Disk "OCRVOTE" defines an unmarked device in that particular node where I get the error messages. In the other working node it gives proper results. See below.

[root@Node2 ~]# oracleasm querydisk -p ocrvote
Disk "OCRVOTE" defines an unmarked device

To fix this I just performed oracleasm scandisks.

[root@Node2 ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...

[root@Node2 ~]# oracleasm querydisk -p ocrvote
Disk "OCRVOTE" is a valid ASM disk
/dev/sdj1: LABEL="OCRVOTE" TYPE="oracleasm"
/dev/sdr1: LABEL="OCRVOTE" TYPE="oracleasm"
/dev/mapper/mpathe1: LABEL="OCRVOTE" TYPE="oracleasm"
/dev/sdb1: LABEL="OCRVOTE" TYPE="oracleasm"
/dev/sdz1: LABEL="OCRVOTE" TYPE="oracleasm"

Now I am able bring up the cluster.

[root@Node2 /]# crsctl start cluster
CRS-2672: Attempting to start 'ora.crf' on 'Node2'
CRS-2672: Attempting to start 'ora.cssd' on 'Node2'
CRS-2672: Attempting to start 'ora.diskmon' on 'Node2'
CRS-2676: Start of 'ora.diskmon' on 'Node2' succeeded
CRS-2676: Start of 'ora.crf' on 'Node2' succeeded
CRS-2676: Start of 'ora.cssd' on 'Node2' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'Node2'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'Node2'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'Node2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'Node2'
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'Node2' succeeded
CRS-2676: Start of 'ora.ctssd' on 'Node2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'Node2'
CRS-2676: 

_____________________________________________________________________________________________________________________

Website Stats