CRS-1714: Unable to discover any voting files, retrying discovery in 15 seconds

One of the two RAC nodes failed to startup CRS with following messages in alert log:

2020-02-27 20:17:50.802 [OCSSD(14861)]CRS-1714: Unable to discover 
    any voting files, retrying discovery in 15 seconds; Details at 
(:CSSNM00070:)in /u01/app/grid/diag/crs/racnode1/crs/trace/ocssd.trc

ocssd.trc:

2020-02-27 20:17:50.801 : CSSD:3324163840: clssnmReadDiscoveryProfile: voting file discovery string(/dev/oracleasm/disks)
2020-02-27 20:17:50.801 : CSSD:3324163840: clssnmvDDiscThread: using discovery string /dev/oracleasm/disks for initial discovery
2020-02-27 20:17:50.801 : SKGFD:3324163840: Discovery with str:/dev/oracleasm/disks:
2020-02-27 20:17:50.801 : SKGFD:3324163840: UFS discovery with :/dev/oracleasm/disks:
2020-02-27 20:17:50.801 : SKGFD:3324163840: Execute glob on the string /dev/oracleasm/disks
2020-02-27 20:17:50.801 : SKGFD:3324163840: OSS discovery with :/dev/oracleasm/disks:
2020-02-27 20:17:50.802 : SKGFD:3324163840: Discovery skipping bad asmlib :ASM::
2020-02-27 20:17:50.802 : CSSD:3324163840: clssnmvDiskVerify: Successful discovery of 0 disks
2020-02-27 20:17:50.802 : CSSD:3324163840: clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
2020-02-27 20:17:50.802 : CSSD:3324163840: clssnmvFindInitialConfigs: No voting files found
2020-02-27 20:17:50.802 : CSSD:3324163840: (:CSSNM00070:)clssnmCompleteInitVFDiscovery: Voting file not found. Retrying discovery in 15 seconds
2020-02-27 20:17:51.687 : CSSD:3558553344: clsssc_CLSFAInit_CB: System not ready for CLSFA initialization
2020-02-27 20:17:52.687 : CSSD:3558553344: clsssc_CLSFAInit_CB: System not ready for CLSFA initialization
2020-02-27 20:17:53.687 : CSSD:3558553344: clsssc_CLSFAInit_CB: System not ready for CLSFA initialization
2020-02-27 20:17:54.687 : CSSD:3558553344: clsssc_CLSFAInit_CB: System not ready for CLSFA initialization
2020-02-27 20:17:55.688 : CSSD:3558553344: clsssc_CLSFAInit_CB: System not ready for CLSFA initialization
2020-02-27 20:17:56.688 : CSSD:3558553344: clsssc_CLSFAInit_CB: System not ready for CLSFA initialization
2020-02-27 20:17:57.688 : CSSD:3558553344: clsssc_CLSFAInit_CB: System not ready for CLSFA initialization

Subscribe to get access

Read more of this content when you subscribe today.

Advertisement

How to Increase ASM Disk Size in VirtualBox

FRA diskgroup has two ASM disks, we just want to have one disk and also need increase the FRA diskgroup size, here are the steps.

  • Drop ASM disk
  • Clone the disk, because ASM disk file is “Format variant: fixed default”. After cloning, the format changes to “Format variant: dynamic default”.
  • Resize the disk
  • Clone back the disk
  • Partition the disk
  • Create ASM disk
  • Add back into Disk Group

1)Check ASM disks in FRA diskgroup.

SQL>  select GROUP_NUMBER, STATE, TOTAL_MB,FREE_MB,NAME,PATH  
       from  v$asm_disk 
       where GROUP_NUMBER in ( select GROUP_NUMBER 
                                from v$asm_diskgroup 
                               where name='FRA');

GR STATE    TOTAL_MB FREE_MB  NAME      PATH
-- -------- ----- -------- ---------- ------------------------------
2  NORMAL   5119      4954 FRA_0001  /dev/oracleasm/disks/ASM_DISK6
2  NORMAL   5119      4958 FRA_0000  /dev/oracleasm/disks/ASM_DISK5

2) Drop asm disk  FRA_0001.

SQL> ALTER DISKGROUP FRA drop disk FRA_0001;

Diskgroup altered.
SQL> select * from v$asm_operation ;

no rows selected
SQL> select STATE, TOTAL_MB,FREE_MB,NAME,PATH 
      from v$asm_disk where GROUP_NUMBER=2;

STATE  TOTAL_MB FREE_MB NAME     PATH
----- -------- ------- -------- -------------------------------------
NORMAL 5119   4795  FRA_0000 ASM_DISK5 /dev/oracleasm/disks/ASM_DISK5

3)Release asm6.vdi disk file from menu “Host Media Manager”. It detached this file from both racnode1 &2 VMs.

4)Resize asm6.vdi from 5G to 50G.

We can just create a new vdi file. But here for practice purpose, we just choose hard way to resize the fixed size vdi file.

c:\Program Files\Oracle\VirtualBox>VBoxManage showhdinfo D:\virtuallab\asm6.vdi
UUID: 8a52a736-6f03-4766-b656-4e6496cce4eb
Parent UUID: base
State: created
Type: normal (base)
Location: D:\virtuallab\asm6.vdi
Storage format: VDI
Format variant: fixed default
Capacity: 5120 MBytes
Size on disk: 5122 MBytes
Encryption: disabled
Property: AllocationBlockSize=1048576

We must clone it into new one file, dynamically allocated.

c:\Program Files\Oracle\VirtualBox>VBoxManage.exe clonehd 
                   d:\virtuallab\asm6.vdi d:\virtuallab\asm6_new.vdi
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Clone medium created in format 'VDI'. UUID: e3f94421-f905-4dee-bc8e-91649d187d35
c:\Program Files\Oracle\VirtualBox>VBoxManage showhdinfo D:\virtuallab\asm6_new.vdi
UUID: e3f94421-f905-4dee-bc8e-91649d187d35
Parent UUID: base
State: created
Type: normal (base)
Location: D:\virtuallab\asm6_new.vdi
Storage format: VDI
Format variant: dynamic default
Capacity: 5120 MBytes
Size on disk: 168 MBytes
Encryption: disabled
Property: AllocationBlockSize=1048576

Resize asm6_new.vdi to 50GB, and change into fixed size for ASM disk purpose.

c:\Program Files\Oracle\VirtualBox>VBoxManage modifyhd 
                          d:\virtuallab\asm6_new.vdi --resize 50000
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%

Remove asm6.vdi from “Host Media Manager”, and clone back with fixed size now.

c:\Program Files\Oracle\VirtualBox>VBoxManage.exe clonehd 
  d:\virtuallab\asm6_new.vdi d:\virtuallab\asm6.vdi --variant Fixed
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Clone medium created in format 'VDI'. UUID: d0875c78-ac96-49cc-a4d0-2e55422b2560


c:\Program Files\Oracle\VirtualBox>VBoxManage showhdinfo D:\virtuallab\asm6.vdi
UUID: d0875c78-ac96-49cc-a4d0-2e55422b2560
Parent UUID: base
State: created
Type: normal (base)
Location: D:\virtuallab\asm6.vdi
Storage format: VDI
Format variant: fixed default
Capacity: 50000 MBytes
Size on disk: 50002 MBytes
Encryption: disabled
Property: AllocationBlockSize=1048576

Partition disk.

[root@racnode1 ~]# fdisk /dev/sde
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): p

Disk /dev/sde: 52.4 GB, 52428800000 bytes, 102400000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xe5f3fc7c

Device Boot Start End Blocks Id System
/dev/sde1 2048 10485759 5241856 83 Linux

Command (m for help): d
Selected partition 1
Partition 1 is deleted

Command (m for help): 1
1: unknown command
Command action
a toggle a bootable flag
b edit bsd disklabel
c toggle the dos compatibility flag
d delete a partition
g create a new empty GPT partition table
G create an IRIX (SGI) partition table
l list known partition types
m print this menu
n add a new partition
o create a new empty DOS partition table
p print the partition table
q quit without saving changes
s create a new empty Sun disklabel
t change a partition's system id
u change display/entry units
v verify the partition table
w write table to disk and exit
x extra functionality (experts only)

Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1):
First sector (2048-102399999, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-102399999, default 102399999):
Using default value 102399999
Partition 1 of type Linux and of size 48.8 GiB is set

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
[root@racnode1 ~]#
[root@racnode1 ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
[root@racnode1 ~]# oracleasm listdisks
ASM_DISK1
ASM_DISK2
ASM_DISK3
ASM_DISK5
ASM_DISK6
ASM_OCR_VOTE1
[root@racnode1 ~]# oracleasm deletedisk ASM_DISK6;
Clearing disk header: done
Dropping disk: done

[root@racnode1 ~]# oracleasm createdisk  ASM_DISK6  /dev/sde1
Writing disk header: done
Instantiating disk: done
[root@racnode1 ~]# blkid
/dev/sda1: UUID="fc21c76e-90ae-4d7f-9b1a-52412af3cf78" TYPE="xfs"
/dev/sda2: UUID="ifffmk-GSXU-NcKC-vww1-HMF6-Nc36-A6sewK" TYPE="LVM2_member"
/dev/sda3: UUID="DG6gQe-VQki-1oT2-V1wk-1etX-6qkp-7QP3cX" TYPE="LVM2_member"
/dev/sdb1: LABEL="ASM_DISK1" TYPE="oracleasm"
/dev/sdc1: LABEL="ASM_DISK2" TYPE="oracleasm"
/dev/sdd1: LABEL="ASM_DISK3" TYPE="oracleasm"
/dev/sde1: LABEL="ASM_DISK6" TYPE="oracleasm"
/dev/sdf1: LABEL="ASM_DISK5" TYPE="oracleasm"
/dev/sdg1: LABEL="ASM_OCR_VOTE1" TYPE="oracleasm"
/dev/sr0: UUID="2019-12-10-15-09-23-93" LABEL="VBox_GAs_6.1.0" TYPE="iso9660"
/dev/mapper/ol-root: UUID="b4379983-435f-4012-b054-89e6834c283a" TYPE="xfs"
/dev/mapper/ol-swap: UUID="3bfcff72-20c3-4e1e-ad38-20de7cca2050" TYPE="swap"
[root@racnode1 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdf 8:80 0 5G 0 disk
└─sdf1 8:81 0 5G 0 part
sdd 8:48 0 5G 0 disk
└─sdd1 8:49 0 5G 0 part
sdb 8:16 0 5G 0 disk
└─sdb1 8:17 0 5G 0 part
sr0 11:0 1 56.8M 0 rom
sdg 8:96 0 50G 0 disk
└─sdg1 8:97 0 50G 0 part
sde 8:64 0 48.8G 0 disk
└─sde1 8:65 0 5G 0 part
sdc 8:32 0 5G 0 disk
└─sdc1 8:33 0 5G 0 part
sda 8:0 0 96G 0 disk
├─sda2 8:2 0 29.5G 0 part
│ ├─ol-swap 252:1 0 8G 0 lvm [SWAP]
│ └─ol-root 252:0 0 61G 0 lvm /
├─sda3 8:3 0 66G 0 part
│ ├─ol-swap 252:1 0 8G 0 lvm [SWAP]
│ └─ol-root 252:0 0 61G 0 lvm /
└─sda1 8:1 0 500M 0 part /boot

Add disk back into diskgroup FRA.

SQL> ALTER DISKGROUP FRA ADD DISK '/dev/oracleasm/disks/ASM_DISK6' 
                                               NAME FRA_ASM_DISK6;

Diskgroup altered.
SQL> select STATE, TOTAL_MB,FREE_MB,NAME,PATH
from v$asm_disk where GROUP_NUMBER=2; 

STATE  TOTAL_MB FREE_MB NAME          PATH
------ -------- ------- ------------- ------------------------------
NORMAL    5119  5086    FRA_0000      /dev/oracleasm/disks/ASM_DISK5
NORMAL   49999 49706    FRA_ASM_DISK6 /dev/oracleasm/disks/ASM_DISK6

Drop the small size asm disk ASM_DISK5.

SQL>  ALTER DISKGROUP FRA drop DISK FRA_0000;

Diskgroup altered.
[root@racnode1 ~]#  oracleasm deletedisk ASM_DISK5;
Disk "ASM_DISK5" defines an unmarked device
Dropping disk: done

[root@racnode1 ~]#  oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...

[root@racnode1 ~]#  oracleasm listdisks
ASM_DISK1
ASM_DISK2
ASM_DISK3
ASM_DISK6
ASM_OCR_VOTE1
[root@racnode1 ~]#
SQL>select STATE, TOTAL_MB,FREE_MB,NAME,PATH
    from v$asm_disk where GROUP_NUMBER=2;

STATE TOTAL_MB   FREE_MB NAME         PATH
------ -------- ------- ------------- ------------------------------
NORMAL 49999    49706   FRA_ASM_DISK6 /dev/oracleasm/disks/ASM_DISK6

Reconfigure ASMLIB after Unbreakable Enterprise Kernel (UEK) Upgraded from UEKR3 to UEKR5 on Oracle Linux 7

After unbreakable enterprise kernel (UEK) is upgraded from release 3 to release 5 on Oracle Linux 7,  the oracle asm is not working well:

[root@racnode1 ~]# oracleasm listdisks
[root@racnode1 ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
Instantiating disk "ASM_DISK1"
Unable to instantiate disk "ASM_DISK1"
Instantiating disk "ASM_DISK2"
Unable to instantiate disk "ASM_DISK2"
Instantiating disk "ASM_DISK3"
Unable to instantiate disk "ASM_DISK3"
Instantiating disk "ASM_DISK4"
Unable to instantiate disk "ASM_DISK4"
Instantiating disk "ASM_DISK5"
Unable to instantiate disk "ASM_DISK5"
Instantiating disk "ASM_DISK6"
Unable to instantiate disk "ASM_DISK6"
Instantiating disk "ASM_DISK7"
Unable to instantiate disk "ASM_DISK7"
Instantiating disk "ASM_OCR_VOTE"
Unable to instantiate disk "ASM_OCR_VOTE"
[root@racnode1 ~]# oracleasm listdisks

Check oracleasm packages , and found there are one or two are missing:

[root@racnode1 ~]# rpm -qa|grep -i oracleasm
oracleasm-support-2.1.11-2.el7.x86_64
[root@racnode1 ~]#

The Oracle ASMLib kernel driver is now included in the Unbreakable Enterprise Kernel. No driver package needs to be installed when using this kernel.

Check Oracle ASMLib kernel driver has been installed.

[root@racnode1 ~]# lsmod|grep asm
oracleasm              61440  1

[root@racnode1 ~]# modinfo oracleasm
filename:       /lib/modules/4.14.35-1902.8.4.el7uek.x86_64/kernel/drivers/block/oracleasm/oracleasm.ko.xz
description:    Kernel driver backing the Generic Linux ASM Library.
author:         Joel Becker, Martin K. Petersen <martin.petersen@oracle.com>
version:        2.0.8
license:        GPL
srcversion:     DF8809442FB655948FEC227
depends:
retpoline:      Y
intree:         Y
name:           oracleasm
vermagic:       4.14.35-1902.8.4.el7uek.x86_64 SMP mod_unload modversions
signat:         PKCS#7
signer:
sig_key:
sig_hashalgo:   md4
parm:           use_logical_block_size:Prefer logical block size over physical (Y=logical, N=physical [default]) (bool)
[root@racnode1 ~]#

For RHEL linux, install  kmod-oracleasm package:

[root@racnode1 ~]# yum install kmod-oracleasm

Download and install oracleasmlib-2.0.12-1.el7.x86_64.rpm package from below url, if it is unavailable.

https://www.oracle.com/linux/downloads/linux-asmlib-v7-downloads.html

Check the required packages are all available now:

[root@racnode1 ~]# rpm -qa|grep -i oracleasm
oracleasmlib-2.0.12-1.el7.x86_64
oracleasm-support-2.1.11-2.el7.x86_64
kmod-oracleasm-2.0.8-26.0.1.el7.x86_64 ( for RHEL )

Configure ASM

[root@racnode1 ~]# oracleasm configure -i
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface [grid]:
Default group to own the driver interface [oinstall]:
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
[root@racnode1 ~]#

Re-initialize oracleasm:

[root@racnode1 sysconfig]# oracleasm init
Creating /dev/oracleasm mount point: /dev/oracleasm
Loading module "oracleasm": oracleasm
Configuring "oracleasm" to use device physical block size
Mounting ASMlib driver filesystem: /dev/oracleasm

Scan and list ASM disks:

[root@racnode1 sysconfig]# oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes


[root@racnode1 sysconfig]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
Instantiating disk "ASM_DISK1"
Instantiating disk "ASM_DISK2"
Instantiating disk "ASM_DISK3"
Instantiating disk "ASM_DISK4"
Instantiating disk "ASM_DISK5"
Instantiating disk "ASM_DISK6"
Instantiating disk "ASM_DISK7"
Instantiating disk "ASM_OCR_VOTE"


[root@racnode1 sysconfig]# oracleasm listdisks
ASM_DISK1
ASM_DISK2
ASM_DISK3
ASM_DISK4
ASM_DISK5
ASM_DISK6
ASM_DISK7
ASM_OCR_VOTE

How to Install and Configure ASMLIb on RHEL Linux 7

Make sure oracleasm packages are installed , go to Red Hat website to download the required ones:

$ rpm -qa|grep -i oracleasm
kmod-oracleasm-version.el7.x86_64
oracleasm-support-version.el7.x86_64
oracleasmlib-version.el7.x86_64

Check orcaleasm.service is activated:

# systemctl is-enabled oracleasm.service
enabled

Check  oracleasm kernel module is installed:

#modinfo oracleasm
filename: /lib/modules/3.10.0-514.el7.x86_64/weak-updates/oracleasm/oracleasm.ko
description: Kernel driver backing the Generic Linux ASM Library.
author: Joel Becker <joel.becker@oracle.com>
version: 2.0.8
license: GPL
rhelversion: 7.3
srcversion: 3AA2BF05E1F8A2D0EEAFD4E
depends:
vermagic: 3.10.0-510.el7.x86_64 SMP mod_unload modversions
signer: Red Hat Enterprise Linux Driver Update Program (key 3)
sig_key: BF:57:F3:E8:73:62:BC:72:29:D9:F4:65:32:17:73:DF:D1:F7:7A:80
sig_hashalgo: sha256
parm: use_logical_block_size:Prefer logical block size over physical (Y=logical, N=physical [default]) (bool)

Check whether oracleasm module is loaded into Linux kernel:

# lsmod | grep oracleasm
                          < -- not loaded yet

Initialize oracleasm module:

# oracleasm init
Creating /dev/oracleasm mount point: /dev/oracleasm
Loading module "oracleasm": oracleasm
Configuring "oracleasm" to use device physical block size
Mounting ASMlib driver filesystem: /dev/oracleasm

Check whether oracleasm module is loaded into Linux kernel:

# lsmod | grep oracleasm
oracleasm 63318 1

Configuring the Oracle ASM library driver:

# oracleasm configure -i
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface []: grid
Default group to own the driver interface []: dba
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done

Check oracleasm status:

# oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes

#cd /etc/sysconfig
#ls -ltr *oracleasm*
-rw-r--r--. 1 root root 685 Jul 10 2014 oracleasm-_dev_oracleasm
lrwxrwxrwx. 1 root root 24 Aug 24 22:59 oracleasm -> 
                    oracleasm-_dev_oracleasm <---- this is required.

# oracleasm configure
ORACLEASM_UID=grid
ORACLEASM_GID=dba
ORACLEASM_SCANBOOT=true
ORACLEASM_SCANORDER="dm"
ORACLEASM_SCANEXCLUDE="sd"
ORACLEASM_USE_LOGICAL_BLOCK_SIZE="false"


# cat oracleasm

#
# This is a configuration file for automatic loading of the Oracle
# Automatic Storage Management library kernel driver. It is generated
# By running /etc/init.d/oracleasm configure. Please use that method
# to modify this file
#
# ORACLEASM_UID: Default user owning the /dev/oracleasm mount point.
ORACLEASM_UID=grid

# ORACLEASM_GID: Default group owning the /dev/oracleasm mount point.
ORACLEASM_GID=dba

# ORACLEASM_SCANBOOT: 'true' means scan for ASM disks on boot.
ORACLEASM_SCANBOOT=true

# ORACLEASM_SCANORDER: Matching patterns to order disk scanning
ORACLEASM_SCANORDER="dm"

# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan
ORACLEASM_SCANEXCLUDE="sd"

# ORACLEASM_USE_LOGICAL_BLOCK_SIZE: 'true' means use the logical block size
# reported by the underlying disk instead of the physical. The default
# is 'false'
ORACLEASM_USE_LOGICAL_BLOCK_SIZE=false

#

Scan ASM disks, and list ASM disks:

# oracleasm scandisks
Scanning the system for Oracle ASMLib disks: [ OK ]

#oracleasm listdisks
OCR_VOTE
ASM_DISK1
ASM_DISK2
...
..
.

Check oracleasm log:

$ vi /var/log/oracleasm

How to Find Mappings Between ASM Disks and Physical Disks

Here are the simple steps for mapping between ASM disks and physical disks in Linux.

Subscribe to get access

Read more of this content when you subscribe today.