Blog

How to Install and Configure ASMLIb on RHEL Linux 7

Make sure oracleasm packages are installed , go to Red Hat website to download the required ones:

$ rpm -qa|grep -i oracleasm
kmod-oracleasm-version.el7.x86_64
oracleasm-support-version.el7.x86_64
oracleasmlib-version.el7.x86_64

Check orcaleasm.service is activated:

# systemctl is-enabled oracleasm.service
enabled

Check  oracleasm kernel module is installed:

#modinfo oracleasm
filename: /lib/modules/3.10.0-514.el7.x86_64/weak-updates/oracleasm/oracleasm.ko
description: Kernel driver backing the Generic Linux ASM Library.
author: Joel Becker <joel.becker@oracle.com>
version: 2.0.8
license: GPL
rhelversion: 7.3
srcversion: 3AA2BF05E1F8A2D0EEAFD4E
depends:
vermagic: 3.10.0-510.el7.x86_64 SMP mod_unload modversions
signer: Red Hat Enterprise Linux Driver Update Program (key 3)
sig_key: BF:57:F3:E8:73:62:BC:72:29:D9:F4:65:32:17:73:DF:D1:F7:7A:80
sig_hashalgo: sha256
parm: use_logical_block_size:Prefer logical block size over physical (Y=logical, N=physical [default]) (bool)

Check whether oracleasm module is loaded into Linux kernel:

# lsmod | grep oracleasm
                          < -- not loaded yet

Initialize oracleasm module:

# oracleasm init
Creating /dev/oracleasm mount point: /dev/oracleasm
Loading module "oracleasm": oracleasm
Configuring "oracleasm" to use device physical block size
Mounting ASMlib driver filesystem: /dev/oracleasm

Check whether oracleasm module is loaded into Linux kernel:

# lsmod | grep oracleasm
oracleasm 63318 1

Configuring the Oracle ASM library driver:

# oracleasm configure -i
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface []: grid
Default group to own the driver interface []: dba
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done

Check oracleasm status:

# oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes

#cd /etc/sysconfig
#ls -ltr *oracleasm*
-rw-r--r--. 1 root root 685 Jul 10 2014 oracleasm-_dev_oracleasm
lrwxrwxrwx. 1 root root 24 Aug 24 22:59 oracleasm -> 
                    oracleasm-_dev_oracleasm <---- this is required.

# oracleasm configure
ORACLEASM_UID=grid
ORACLEASM_GID=dba
ORACLEASM_SCANBOOT=true
ORACLEASM_SCANORDER="dm"
ORACLEASM_SCANEXCLUDE="sd"
ORACLEASM_USE_LOGICAL_BLOCK_SIZE="false"


# cat oracleasm

#
# This is a configuration file for automatic loading of the Oracle
# Automatic Storage Management library kernel driver. It is generated
# By running /etc/init.d/oracleasm configure. Please use that method
# to modify this file
#
# ORACLEASM_UID: Default user owning the /dev/oracleasm mount point.
ORACLEASM_UID=grid

# ORACLEASM_GID: Default group owning the /dev/oracleasm mount point.
ORACLEASM_GID=dba

# ORACLEASM_SCANBOOT: 'true' means scan for ASM disks on boot.
ORACLEASM_SCANBOOT=true

# ORACLEASM_SCANORDER: Matching patterns to order disk scanning
ORACLEASM_SCANORDER="dm"

# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan
ORACLEASM_SCANEXCLUDE="sd"

# ORACLEASM_USE_LOGICAL_BLOCK_SIZE: 'true' means use the logical block size
# reported by the underlying disk instead of the physical. The default
# is 'false'
ORACLEASM_USE_LOGICAL_BLOCK_SIZE=false

#

Scan ASM disks, and list ASM disks:

# oracleasm scandisks
Scanning the system for Oracle ASMLib disks: [ OK ]

#oracleasm listdisks
OCR_VOTE
ASM_DISK1
ASM_DISK2
...
..
.

Check oracleasm log:

$ vi /var/log/oracleasm

WARNING: FAILED to load library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so

Database instance starts up failure with below errors:

ORA-15183: ASMLIB initialization error [driver/agent not installed]
Wed Jan 31 17:48:16 2018
WARNING: FAILED to load library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so

Check “oracleasm.service” with errors:

# systemctl status oracleasm.service
 oracleasm.service - Load oracleasm Modules
 Loaded: loaded (/usr/lib/systemd/system/oracleasm.service; enabled; vendor preset: disabled)
 Active: inactive (dead) since Wed 2018-01-31 18:31:50 AEDT; 2s ago
 Process: 34627 ExecStop=/usr/sbin/service oracleasm stop_sysctl (code=exited, status=0/SUCCESS)
 Process: 429 ExecStart=/usr/sbin/service oracleasm start_sysctl (code=exited, status=0/SUCCESS)
 Main PID: 429 (code=exited, status=0/SUCCESS)

Jan 31 17:58:46 racnode1 systemd[1]: Starting Load oracleasm Modules...
Jan 31 17:58:46 racnode1 service[429]: Initializing the Oracle ASMLib driver: [ OK ]
Jan 31 17:58:48 racnode1 service[429]: Scanning the system for Oracle ASMLib disks: [ OK ]
Jan 31 17:58:48 racnode1 systemd[1]: Started Load oracleasm Modules.
Jan 31 18:31:50 racnode1 systemd[1]: Stopping Load oracleasm Modules...
Jan 31 18:31:50 racnode1 service[34627]: Dropping Oracle ASMLib disks: [ OK ]
Jan 31 18:31:50 racnode1 service[34627]: Shutting down the Oracle ASMLib driver: [ OK ]

Stop “oracleasm.service’, and re-enable it. After this, all works fine and database instance can be started up successfully.

# systemctl stop oracleasm.service

# systemctl status oracleasm.service
 oracleasm.service - Load oracleasm Modules
 Loaded: loaded (/usr/lib/systemd/system/oracleasm.service; enabled; vendor preset: disabled)
 Active: inactive (dead) since Wed 2018-01-31 18:31:50 AEDT; 2s ago
 Process: 34627 ExecStop=/usr/sbin/service oracleasm stop_sysctl (code=exited, status=0/SUCCESS)
 Process: 429 ExecStart=/usr/sbin/service oracleasm start_sysctl (code=exited, status=0/SUCCESS)
 Main PID: 429 (code=exited, status=0/SUCCESS)

Jan 31 17:58:46 racnode1 systemd[1]: Starting Load oracleasm Modules...
Jan 31 17:58:46 racnode1 service[429]: Initializing the Oracle ASMLib driver: [ OK ]
Jan 31 17:58:48 racnode1 service[429]: Scanning the system for Oracle ASMLib disks: [ OK ]
Jan 31 17:58:48 racnode1 systemd[1]: Started Load oracleasm Modules.
Jan 31 18:31:50 racnode1 systemd[1]: Stopping Load oracleasm Modules...
Jan 31 18:31:50 racnode1 service[34627]: Dropping Oracle ASMLib disks: [ OK ]
Jan 31 18:31:50 racnode1 service[34627]: Shutting down the Oracle ASMLib driver: [ OK ]
Jan 31 18:31:50 racnode1 systemd[1]: Stopped Load oracleasm Modules.

# systemctl enable oracleasm.service

# systemctl start oracleasm.service

# systemctl status oracleasm.service
 oracleasm.service - Load oracleasm Modules
 Loaded: loaded (/usr/lib/systemd/system/oracleasm.service; enabled; vendor preset: disabled)
 Active: active (exited) since Wed 2018-01-31 18:32:30 AEDT; 2s ago
 Process: 35360 ExecStart=/usr/sbin/service oracleasm start_sysctl (code=exited, status=0/SUCCESS)
 Main PID: 35360 (code=exited, status=0/SUCCESS)

Jan 31 18:32:29 racnode1 systemd[1]: Starting Load oracleasm Modules...
Jan 31 18:32:29 racnode1 service[35360]: Initializing the Oracle ASMLib driver: [ OK ]
Jan 31 18:32:30 racnode1 service[35360]: Scanning the system for Oracle ASMLib disks: [ OK ]
Jan 31 18:32:30 racnode1 systemd[1]: Started Load oracleasm Modules.

Alert.log

Wed Jan 31 18:40:09 2018
NOTE: Loaded library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so

Ocrconfig Failed With Error PROT-30 PROC-50

Try to add OCR into  new diskgroup ‘+OCR_VOTE’, and get below errors:

# ./ocrconfig -add OCR_VOTE
PROT-30: The Oracle Cluster Registry location to be added is not usable
PROC-50: The Oracle Cluster Registry location to be added is inaccessible on nodes racnode1.

Check crsd.trc log, the error is very obvious that  “OCR file/disk OCR_VOTE  No such file or directory”.

2018-01-25 14:55:18.658112 : OCRSRV:3726690624: utstoragetypecommon:failed in stat OCR file/disk OCR_VOTE, errno=2, os 
err string=No such file or directory. Returns [8]
2018-01-25 14:55:18.658123 : OCRSRV:3726690624: proprstcheckstoarge: sproprutstoragetypecommon Return [8]. OS error [2]. 
Error detail [No such file or directory]
 OCRSRV:3726690624: proas_replace_dev: Failed to verify the OCR location [OCR_VOTE] on local node. Retval:[8]. Error: 
[No such file or directory]

Check ocrconfig log :

$/u01/app/grid/diag/crs/racnode1/crs/trace$ cat ocrconfig_83818.trc
Trace file /u01/app/grid/diag/crs/racnode1/crs/trace/ocrconfig_83818.trc
Oracle Database 12c Clusterware Release 12.1.0.2.0 - Production Copyright 1996, 2014 Oracle. All rights reserved.
2018-01-25 15:56:13.188061 : OCRCONF: ocrconfig starts...
 default: prgdevid: Failed to open the OCR location to be added [OCR_VOTE]. Retval[2]
 OCRCONF: chkid: Failed to get the OCR id from the new device to be added/replaced[OCR_VOTE]. Retval[26]
2018-01-25 15:56:13.270558 : OCRCONF: Failed to compare the OCR id in the device to be added/replaced [OCR_VOTE] and OCR 
id in repository the context is pointing to. Skiping the check.
2018-01-25 15:56:13.271684 : OCRCLI: proac_replace_dev:[OCR_VOTE]: Failed. Retval [50]
2018-01-25 15:56:13.271745 : OCRAPI: procr_replace_dev: failed to replace device (50)
2018-01-25 15:56:13.271783 : OCRCONF: Failed to replace the OCR device. OCR error:[PROC-50: The Oracle Cluster Registry 
location to be added is inaccessible on nodes racnode1.]
2018-01-25 15:56:13.271815 : OCRCONF: The new OCR device [OCR_VOTE] cannot be opened
2018-01-25 15:56:13.271840 : OCRCONF: Exiting [status=failed]...

Checked physical disk and ASM disk are all good.

Finally check back the command and find ‘+’ is missing from diskgroup name.

# ./ocrconfig -add +OCR_VOTE

CRS-6706: Oracle Clusterware Release patch level (‘nnnnnn’) does not match Software patch level (‘nnnnnn’)

“opatchauto” failed in the middle, try to rerun again, get “CRS-6706” error.

# /u01/app/12.1.0.2/grid/OPatch/opatchauto apply /tmp/12.1.0.2/27010872 -oh /u01/app/12.1.0.2/grid
...
..
.
Using configuration parameter file: /u01/app/12.1.0.2/grid/OPatch/auto/dbtmp/bootstrap_racnode1/patchwork/crs/install/crsconfig_params
CRS-6706: Oracle Clusterware Release patch level ('173535486') does not match Software patch level ('2039526626'). Oracle Clusterware cannot be started.
CRS-4000: Command Start failed, or completed with errors.
2018/01/23 16:29:02 CLSRSC-117: Failed to start Oracle Clusterware stack


After fixing the cause of failure Run opatchauto resume

]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.

OPatchauto session completed at Tue Jan 23 16:29:04 2018
Time taken to complete the session 1 minute, 39 seconds

opatchauto failed with error code 42

On racnode1:

$/u01/app/12.1.0.2/grid/bin/kfod op=patches
---------------
List of Patches
===============
19941482
19941477
19694308
19245012
26925218   <---- Does not exist on node2

$/u01/app/12.1.0.2/grid/bin/kfod op=patchlvl
2039526626

On racnode2:

$/u01/app/12.1.0.2/grid/bin/kfod op=patches
---------------
List of Patches
===============
19941482
19941477
19694308
19245012

$/u01/app/12.1.0.2/grid/bin/kfod op=patchlvl
2039526626

We can see patch 26925218 has been applied onto racnode 1, the solution is :

1) Rollback this patch ( 26925218), and run “opatchauto” again to finish the patching successfully. then everything should be fine.

OR

2) Manually complete all the left patches in the GI. After this everything is fine.

In 12c, GI home must have identical patches for the clusterware to start unless during rolling patching.
After applied the same patches on all nodes, GI started fine.

———-

Another situation you might meet is all nodes have same patches but ‘opatch lsinventory’ shows the different patch level:

For example , on racnode1:

$ /u01/app/12.1.0.2/grid/bin/kfod op=patches
---------------
List of Patches
===============
11111111
22222222
33333333

$ /u01/app/12.1.0.2/grid/bin/kfod op=patchlvl
-------------------
Current Patch level
===================
8888888888

Node2

$ /u01/app/12.1.0.2/grid/bin/kfod op=patches
---------------
List of Patches
===============
11111111
22222222
33333333

$ /u01/app/12.1.0.2/grid/bin/kfod op=patchlvl
-------------------
Current Patch level
===================
9999999999

However opatch lsinventory shows the different patch level:

Patch level status of Cluster nodes :

Patching Level Nodes
-------------- -----
8888888888 node1                    ====>> different patch level
9999999999 node2

For 12.1.0.1/2:

Execute”/u01/app/12.1.0.2/grid/crs/install/rootcrs.sh -patch” as root user on the problematic node and the patch level should be corrected.

For 12.2

Execute”<GI_HOME>/crs/install/rootcrs.pl -prepatch”  “<GI_HOME>/crs/install/rootcrs.pl -postpatch”and as <root_user> on the problematic node and the patch level should be corrected

REFERENCES:

RS-6706: Oracle Clusterware Release patch level (‘nnn’) does not match Software patch level (‘mmm’) (Doc ID 1639285.1)