EXACC: Cluster Services Failed to Start Up after VM Machine Rebooted

After RAC Cluster VM machines rebooted, the cluster services are not up with the following messages in alert.log:

2021-07-12 18:48:20.741 [CLSECHO(27566)]ACFS-9327: Verifying ADVM/ACFS devices.
2021-07-12 18:48:20.791 [CLSECHO(27631)]ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
2021-07-12 18:48:20.855 [CLSECHO(27710)]ACFS-9156: Detecting control device '/dev/ofsctl'.
2021-07-12 18:48:21.469 [CLSECHO(28208)]ACFS-9294: updating file /etc/sysconfig/oracledrivers.conf
2021-07-12 18:48:21.530 [CLSECHO(28254)]ACFS-9322: completed
2021-07-12 18:48:23.302 [OSYSMOND(30089)]CRS-8500: Oracle Clusterware OSYSMOND process is starting with operating system process ID 30089
2021-07-12 18:48:23.274 [CSSDMONITOR(30080)]CRS-8500: Oracle Clusterware CSSDMONITOR process is starting with operating system process ID 30080
2021-07-12 18:48:23.698 [CSSDAGENT(30498)]CRS-8500: Oracle Clusterware CSSDAGENT process is starting with operating system process ID 30498
2021-07-12 18:48:25.138 [OCSSD(30929)]CRS-8500: Oracle Clusterware OCSSD process is starting with operating system process ID 30929
2021-07-12 18:48:26.213 [OCSSD(30929)]CRS-1713: CSSD daemon is started in hub mode
2021-07-12 18:48:27.400 [OCSSD(30929)]CRS-1656: The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /u01/app/grid/diag/crs/racnode1/crs/trace/ocssd.trc
2021-07-12 18:48:27.416 [OCSSD(30929)]CRS-1652: Starting clean up of CRSD resources.
2021-07-12 18:48:27.421 [OCSSD(30929)]CRS-1653: The clean up of the CRSD resources failed.
2021-07-12T18:48:29.448153+10:00
Errors in file /u01/app/grid/diag/crs/racnode1/crs/trace/ocssd.trc  (incident=41):
CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []
Incident details in: /u01/app/grid/diag/crs/racnode1/crs/incident/incdir_41/ocssd_i41.trc

2021-07-12 18:48:29.436 [OCSSD(30929)]CRS-8503: Oracle Clusterware process OCSSD with operating system process ID 30929 experienced fatal signal or exception code 6.
2021-07-12 18:48:30.755 [CSSDMONITOR(38238)]CRS-8500: Oracle Clusterware CSSDMONITOR process is starting with operating system process ID 38238
2021-07-12 18:48:31.229 [CSSDAGENT(38659)]CRS-8500: Oracle Clusterware CSSDAGENT process is starting with operating system process ID 38659

ocssd.trc:

2021-07-12 18:48:26.337 :    CSSD:1474272576: [     INFO] clssscUpdateInitState: Set state to 0x008c1e46, based on prior state of 0x008c1e06 and requested change of 0x00000040
2021-07-12 18:48:26.337 :    CSSD:1474272576: [     INFO] clssscGetParameterProfile: buffer passed for parameter ASM discovery (3) is too short, required 26, passed 20
2021-07-12 18:48:26.337 :    CSSD:1474272576: [     INFO] clssnmReadDiscoveryProfile: voting file discovery string(o/*/DATAC1_*,o/*/RECOC1_*)
2021-07-12 18:48:26.337 :    CSSD:1474272576: [     INFO] clssnkInit: NK generic layer initializing.
2021-07-12 18:48:26.337 :    CSSD:1474272576: [     INFO] clssnkipmiInit: start
2021-07-12 18:48:26.346 :    CSSD:1474272576: [     INFO] clssnkipmiInit: binary path none
2021-07-12 18:48:26.346 :    CSSD:1474272576: [     INFO] clssnkipmiInit: ipmi mech 0
2021-07-12 18:48:26.347 :    CSSD:4023375616: [     INFO] clssscthrdmain: Starting thread clssnmvDDiscThread
2021-07-12 18:48:26.390 :   SKGFD:4023375616: ERROR: -8(OS Error -1 (open,sskgxplp,Invalid protocol requested (2) or protocol not loaded.,Error 0)
)
2021-07-12 18:48:26.390 :   SKGFD:4023375616: ERROR: -10(OSS Operation oss_initialize failed with error 4 [Network initialization failed]
)
2021-07-12 18:48:26.390 :   SKGFD:4023375616: kgfkInit: Error lib:3 ret:-10

2021-07-12 18:48:26.399 :    CSSD:4023375616: [    ERROR] clsssnmvDDiscThread: Unable to create clsf context
2021-07-12 18:48:26.399 :    CSSD:4023375616: [     INFO] clssnmCheckForVfFailure: no voting file found

REASON and SOLUTION

Subscribe to get access

Read more of this content when you subscribe today.