CRS-4638 CRS-4535 CRS-4530 CRS-4534 CRS-2674: Start of 'ora.cssd' on 'Node2' failed

_____________________________________________________________________________________________________________________

Error Description

Cluster is up and running in one node but when I try to start the cluster in second node gives following error messages.

crsctl check crs command failed with following error messages.

[root@Node2 ~]# crsctl check crs

CRS-4638: Oracle High Availability Services is online

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4530: Communications failure contacting Cluster Synchronization Services daemon

CRS-4534: Cannot communicate with Event Manager

Also crsctl start res ora.cluster_interconnect.haip -init Command failed with CRS-2674: Start of 'ora.cssd' on 'Node2' failed

[root@Node2 ~]# crsctl start res ora.cluster_interconnect.haip -init

CRS-2672: Attempting to start 'ora.cssd' on 'Node2'

CRS-2672: Attempting to start 'ora.diskmon' on 'Node2'

CRS-2676: Start of 'ora.diskmon' on 'Node2' succeeded

CRS-2674: Start of 'ora.cssd' on 'Node2' failed

CRS-2679: Attempting to clean 'ora.cssd' on 'Node2'

CRS-2681: Clean of 'ora.cssd' on 'Node2' succeeded

CRS-4000: Command Start failed, or completed with errors.

Solution Description

When I checked the occsd.trc file I noticed that there are few repeated lines mentioning like has a disk HB, but no network HB. Now I realized that there are some network issues and when I pinged the interconnect/private IPs between the RAC nodes it is not pinging.

Log/Trace File: /u01/app/grid/diag/crs/Node2/crs/trace/ocssd.trc

2016-09-23 08:22:06.378508 :    CSSD:2643093248: clssscWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 1000 with cvtimewait status 4294967186

2016-09-23 08:22:06.636082 :    CSSD:2620225280: clssnmPollingThread: state(1) clusterState(0) exit

2016-09-23 08:22:06.636092 :    CSSD:2620225280: clssscExit: removeNode() already called

2016-09-23 08:22:06.636095 :    CSSD:2620225280: clssscExit: abort already set 0

2016-09-23 08:22:06.742376 :    CSSD:2623379200: clssnmvDHBValidateNCopy: node 1, Node1, has a disk HB, but no network HB, DHB has rcfg 370075968, wrtcnt, 966100, LATS 16055504, lastSeqNo 966099, uniqueness 1474627759, timestamp 1474633343/1310023344

2016-09-23 08:22:07.378630 :    CSSD:2643093248: clssscWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 1000 with cvtimewait status 4294967186

2016-09-23 08:22:07.743076 :    CSSD:2623379200: clssnmvDHBValidateNCopy: node 1, Node1, has a disk HB, but no network HB, DHB has rcfg 370075968, wrtcnt, 966101, LATS 16056504, lastSeqNo 966100, uniqueness 1474627759, timestamp 1474633344/1310024344

2016-09-23 08:22:08.378747 :    CSSD:2643093248: clssscWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 1000 with cvtimewait status 4294967186

2016-09-23 08:22:08.743849 :    CSSD:2623379200: clssnmvDHBValidateNCopy: node 1, Node1, has a disk HB, but no network HB, DHB has rcfg 370075968, wrtcnt, 966102, LATS 16057504, lastSeqNo 966101, uniqueness 1474627759, timestamp 1474633345/1310025344

Tried to ping the interconnect ip from node 1 to 2 it is not reachable and there was some VLAN problems associated with that. Network team fixed the issue as per my request and issue resolved.

[root@Node2 ~]# ping Node1priv.abnsayrate.net

PING Node1priv.abnsayrate.net (10.188.60.61) 56(84) bytes of data.

From Node2priv.abnsayrate.net (10.188.60.62) icmp_seq=1 Destination Host Unreachable

From Node2priv.abnsayrate.net (10.188.60.62) icmp_seq=2 Destination Host Unreachable

From Node2priv.abnsayrate.net (10.188.60.62) icmp_seq=3 Destination Host Unreachable

From Node2priv.abnsayrate.net (10.188.60.62) icmp_seq=4 Destination Host Unreachable

^C

--- Node1priv.abnsayrate.net ping statistics ---

7 packets transmitted, 0 received, +4 errors, 100% packet loss, time 6000ms

pipe 4

[root@Node2 ~]# ^C

[root@Node2 ~]# ping 10.188.60.61

PING 10.188.60.61 (10.188.60.61) 56(84) bytes of data.

From 10.188.60.61 icmp_seq=1 Destination Host Unreachable

From 10.188.60.61 icmp_seq=2 Destination Host Unreachable

From 10.188.60.61 icmp_seq=3 Destination Host Unreachable

From 10.188.60.61 icmp_seq=4 Destination Host Unreachable

^C^C

--- 10.188.60.61 ping statistics ---

6 packets transmitted, 0 received, +4 errors, 100% packet loss, time 5000ms

pipe 4

Now after fixing the issue with the network I am able to ping the IP and able to start the cluster thereafter.

[oracle@Node2 ~]$ ping 10.188.60.61

PING 10.188.60.61 (10.188.60.61) 56(84) bytes of data.

64 bytes from 10.188.60.61: icmp_seq=1 ttl=64 time=0.116 ms

64 bytes from 10.188.60.61: icmp_seq=2 ttl=64 time=0.049 ms

64 bytes from 10.188.60.61: icmp_seq=3 ttl=64 time=0.122 ms

^C

--- 10.188.60.61 ping statistics ---

Start the cluster

[root@Node2 ~]# crsctl start cluster

CRS-2672: Attempting to start 'ora.crf' on 'Node2'

CRS-2672: Attempting to start 'ora.cssd' on 'Node2'

CRS-2672: Attempting to start 'ora.diskmon' on 'Node2'

CRS-2676: Start of 'ora.diskmon' on 'Node2' succeeded

CRS-2676: Start of 'ora.crf' on 'Node2' succeeded

CRS-2676: Start of 'ora.cssd' on 'Node2' succeeded

CRS-2672: Attempting to start 'ora.ctssd' on 'Node2'

CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'Node2'

CRS-2676: Start of 'ora.ctssd' on 'Node2' succeeded

CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'Node2' succeeded

CRS-2672: Attempting to start 'ora.asm' on 'Node2'

CRS-2676: Start of 'ora.asm' on 'Node2' succeeded

CRS-2672: Attempting to start 'ora.storage' on 'Node2'

CRS-2676: Start of 'ora.storage' on 'Node2' succeeded

CRS-2672: Attempting to start 'ora.crsd' on 'Node2'

CRS-2676: Start of 'ora.crsd' on 'Node2' succeeded

_____________________________________________________________________________________________________________________