CRS-1714: Unable to discover any voting files, retrying discovery in 15 seconds

One of the two RAC nodes failed to startup CRS with following messages in alert log:

2020-02-27 20:17:50.802 [OCSSD(14861)]CRS-1714: Unable to discover 
    any voting files, retrying discovery in 15 seconds; Details at 
(:CSSNM00070:)in /u01/app/grid/diag/crs/racnode1/crs/trace/ocssd.trc

ocssd.trc:

2020-02-27 20:17:50.801 : CSSD:3324163840: clssnmReadDiscoveryProfile: voting file discovery string(/dev/oracleasm/disks)
2020-02-27 20:17:50.801 : CSSD:3324163840: clssnmvDDiscThread: using discovery string /dev/oracleasm/disks for initial discovery
2020-02-27 20:17:50.801 : SKGFD:3324163840: Discovery with str:/dev/oracleasm/disks:
2020-02-27 20:17:50.801 : SKGFD:3324163840: UFS discovery with :/dev/oracleasm/disks:
2020-02-27 20:17:50.801 : SKGFD:3324163840: Execute glob on the string /dev/oracleasm/disks
2020-02-27 20:17:50.801 : SKGFD:3324163840: OSS discovery with :/dev/oracleasm/disks:
2020-02-27 20:17:50.802 : SKGFD:3324163840: Discovery skipping bad asmlib :ASM::
2020-02-27 20:17:50.802 : CSSD:3324163840: clssnmvDiskVerify: Successful discovery of 0 disks
2020-02-27 20:17:50.802 : CSSD:3324163840: clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
2020-02-27 20:17:50.802 : CSSD:3324163840: clssnmvFindInitialConfigs: No voting files found
2020-02-27 20:17:50.802 : CSSD:3324163840: (:CSSNM00070:)clssnmCompleteInitVFDiscovery: Voting file not found. Retrying discovery in 15 seconds
2020-02-27 20:17:51.687 : CSSD:3558553344: clsssc_CLSFAInit_CB: System not ready for CLSFA initialization
2020-02-27 20:17:52.687 : CSSD:3558553344: clsssc_CLSFAInit_CB: System not ready for CLSFA initialization
2020-02-27 20:17:53.687 : CSSD:3558553344: clsssc_CLSFAInit_CB: System not ready for CLSFA initialization
2020-02-27 20:17:54.687 : CSSD:3558553344: clsssc_CLSFAInit_CB: System not ready for CLSFA initialization
2020-02-27 20:17:55.688 : CSSD:3558553344: clsssc_CLSFAInit_CB: System not ready for CLSFA initialization
2020-02-27 20:17:56.688 : CSSD:3558553344: clsssc_CLSFAInit_CB: System not ready for CLSFA initialization
2020-02-27 20:17:57.688 : CSSD:3558553344: clsssc_CLSFAInit_CB: System not ready for CLSFA initialization

Subscribe to get access

Read more of this content when you subscribe today.

Reconfigure ASMLIB after Unbreakable Enterprise Kernel (UEK) Upgraded from UEKR3 to UEKR5 on Oracle Linux 7

After unbreakable enterprise kernel (UEK) is upgraded from release 3 to release 5 on Oracle Linux 7,  the oracle asm is not working well:

[root@racnode1 ~]# oracleasm listdisks
[root@racnode1 ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
Instantiating disk "ASM_DISK1"
Unable to instantiate disk "ASM_DISK1"
Instantiating disk "ASM_DISK2"
Unable to instantiate disk "ASM_DISK2"
Instantiating disk "ASM_DISK3"
Unable to instantiate disk "ASM_DISK3"
Instantiating disk "ASM_DISK4"
Unable to instantiate disk "ASM_DISK4"
Instantiating disk "ASM_DISK5"
Unable to instantiate disk "ASM_DISK5"
Instantiating disk "ASM_DISK6"
Unable to instantiate disk "ASM_DISK6"
Instantiating disk "ASM_DISK7"
Unable to instantiate disk "ASM_DISK7"
Instantiating disk "ASM_OCR_VOTE"
Unable to instantiate disk "ASM_OCR_VOTE"
[root@racnode1 ~]# oracleasm listdisks

Check oracleasm packages , and found there are one or two are missing:

[root@racnode1 ~]# rpm -qa|grep -i oracleasm
oracleasm-support-2.1.11-2.el7.x86_64
[root@racnode1 ~]#

The Oracle ASMLib kernel driver is now included in the Unbreakable Enterprise Kernel. No driver package needs to be installed when using this kernel.

Check Oracle ASMLib kernel driver has been installed.

[root@racnode1 ~]# lsmod|grep asm
oracleasm              61440  1

[root@racnode1 ~]# modinfo oracleasm
filename:       /lib/modules/4.14.35-1902.8.4.el7uek.x86_64/kernel/drivers/block/oracleasm/oracleasm.ko.xz
description:    Kernel driver backing the Generic Linux ASM Library.
author:         Joel Becker, Martin K. Petersen <martin.petersen@oracle.com>
version:        2.0.8
license:        GPL
srcversion:     DF8809442FB655948FEC227
depends:
retpoline:      Y
intree:         Y
name:           oracleasm
vermagic:       4.14.35-1902.8.4.el7uek.x86_64 SMP mod_unload modversions
signat:         PKCS#7
signer:
sig_key:
sig_hashalgo:   md4
parm:           use_logical_block_size:Prefer logical block size over physical (Y=logical, N=physical [default]) (bool)
[root@racnode1 ~]#

For RHEL linux, install  kmod-oracleasm package:

[root@racnode1 ~]# yum install kmod-oracleasm

Download and install oracleasmlib-2.0.12-1.el7.x86_64.rpm package from below url, if it is unavailable.

https://www.oracle.com/linux/downloads/linux-asmlib-v7-downloads.html

Check the required packages are all available now:

[root@racnode1 ~]# rpm -qa|grep -i oracleasm
oracleasmlib-2.0.12-1.el7.x86_64
oracleasm-support-2.1.11-2.el7.x86_64
kmod-oracleasm-2.0.8-26.0.1.el7.x86_64 ( for RHEL )

Configure ASM

[root@racnode1 ~]# oracleasm configure -i
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface [grid]:
Default group to own the driver interface [oinstall]:
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
[root@racnode1 ~]#

Re-initialize oracleasm:

[root@racnode1 sysconfig]# oracleasm init
Creating /dev/oracleasm mount point: /dev/oracleasm
Loading module "oracleasm": oracleasm
Configuring "oracleasm" to use device physical block size
Mounting ASMlib driver filesystem: /dev/oracleasm

Scan and list ASM disks:

[root@racnode1 sysconfig]# oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes


[root@racnode1 sysconfig]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
Instantiating disk "ASM_DISK1"
Instantiating disk "ASM_DISK2"
Instantiating disk "ASM_DISK3"
Instantiating disk "ASM_DISK4"
Instantiating disk "ASM_DISK5"
Instantiating disk "ASM_DISK6"
Instantiating disk "ASM_DISK7"
Instantiating disk "ASM_OCR_VOTE"


[root@racnode1 sysconfig]# oracleasm listdisks
ASM_DISK1
ASM_DISK2
ASM_DISK3
ASM_DISK4
ASM_DISK5
ASM_DISK6
ASM_DISK7
ASM_OCR_VOTE