SYMPTOM
While adding a new disk into an existing diskgroup, below errors occurred:
ORA-15032: not all alterations performed ORA-15137: The ASM cluster is in rolling patch state.
INVESTIGATION
1) On both nodes, it shows the cluster in “In Rolling Patch”, and the patch levels are all the same.
SQL> SELECT SYS_CONTEXT('SYS_CLUSTER_PROPERTIES', 'CLUSTER_STATE') FROM DUAL; SYS_CONTEXT('SYS_CLUSTER_PROPERTIES','CLUSTER_STATE') -------------------------------------------------------------------- In Rolling Patch SQL> SELECT SYS_CONTEXT('SYS_CLUSTER_PROPERTIES', 'CURRENT_PATCHLVL') FROM DUAL; SYS_CONTEXT('SYS_CLUSTER_PROPERTIES','CURRENT_PATCHLVL') -------------------------------------------------------------------- 3628626982 $ asmcmd ASMCMD> showclusterstate In Rolling Patch ASMCMD> showpatches --------------- List of Patches =============== 26609817 26609966 26839277 27105253 27128906 27144050 27335416 27458609 27464465 27674384 ASMCMD> showversion ASM version : 12.2.0.1.0
2) “crsctl query crs softwarepatch” shows the same results on both nodes:
$ crsctl query crs softwarepatch Oracle Clusterware patch level on node racnode1 is [3628626982]. $ crsctl query crs softwarepatch Oracle Clusterware patch level on node racnode2 is [3628626982].
3) “crsctl query crs releasepatch” shows the same results on both nodes:
— racnode1:
$crsctl query crs releasepatch Oracle Clusterware release patch level is [3628626982] and the complete list of patches [26609817 26609966 26839277 27105253 27128906 27144050 27335416 27458609 27464465 27674384 ] have been applied on the local node.
— racnode2:
$crsctl query crs releasepatch Oracle Clusterware release patch level is [3628626982] and the complete list of patches [26609817 26609966 26839277 27105253 27128906 27144050 27335416 27458609 27464465 27674384 ] have been applied on the local node.
4) kfod command shows the same results on both nodes:
— racnode1:
$ $ORACLE_HOME/bin/kfod op=patchlvl ------------------- Current Patch level =================== 3628626982 $ $ORACLE_HOME/bin/kfod op=patches --------------- List of Patches =============== 26609817 26609966 26839277 27105253 27128906 27144050 27335416 27458609 27464465 27674384
— racnode2:
$ $ORACLE_HOME/bin/kfod op=patchlvl ------------------- Current Patch level =================== 3628626982 $ $ORACLE_HOME/bin/kfod op=patches --------------- List of Patches =============== 26609817 26609966 26839277 27105253 27128906 27144050 27335416 27458609 27464465 27674384
5) lsinventory shows the same results on both nodes:
$ $ORACLE_HOME/OPatch/opatch lsinventory | grep -i desc ARU platform description:: Linux x86-64 Patch description: "Database Apr 2018 Release Update : 12.2.0.1.180417 (27674384)" Patch description: "OCW APR 2018 RELEASE UPDATE 12.2.0.1.0(180129) (27464465)" Patch description: "ACFS APR 2018 RELEASE UPDATE 12.2.0.1.0(180129) (27458609)" Patch description: "Tomcat Release Update 12.2.0.1.0(ID:171023.0830) (27144050)" Patch description: "DBWLM RELEASE UPDATE 12.2.0.1.0(ID:170913) (26839277)"
SOLUTIONS
$crsctl stop rollingpatch CRS-1161: The cluster was successfully patched to patch level [3628626982].
recheck with above commands, the cluster status is changed from “In Rolling Patch” to “Normal” now.
SQL> SELECT SYS_CONTEXT('SYS_CLUSTER_PROPERTIES', 'CLUSTER_STATE') FROM DUAL; SYS_CONTEXT('SYS_CLUSTER_PROPERTIES','CLUSTER_STATE') -------------------------------------------------------------------------------- Normal
Generally speaking, For “ORA-15137: The ASM cluster is in rolling patch state” issue, the below steps can be followed one after another, until the issue is resolved:
a) stop rolling patch status one node by another:
SQL>Alter system stop rolling patch;
b) stop rolling patch for whole cluster:
$crsctl stop rollingpatch
c) In case of postpatch not complete successfully for some reason, which can also cause this issue:
— As super root user :
$GRID_HOME/crs/install/rootcrs.sh -prepatch $GRID_HOME/crs/install/rootcrs.sh -postpatch
d) For some reason, OCR is not updated with right patchlevel:
$GRID_HOME/crs/install/rootcrs.sh -prepatch $GI_HOME/bin/clscfg -patch $GRID_HOME/crs/install/rootcrs.sh -postpatch
e) For some reason, patches are available from “opatch lsinventory”, but they are missing from kfod output:
-- as super user $GRID_HOME/crs/install/rootcrs.sh -prepatch -- as grid owner, $GRID_HOME/bin/patchgen commit -pi 12345678 $GRID_HOME/bin/patchgen commit -pi 23456789 -- as super user $GRID_HOME/crs/install/rootcrs.sh -postpatch
Very helpful post/blog. Worked perfectly.
LikeLike
thanks for reading. so nice to hear it is helpful.
LikeLike
Really nice blog. My issue was resolved.. even Oracle had taken days to provide the fix. But this blog did in 30 mins.
LikeLike
thanks for reading. so nice to hear it is helpful.
LikeLike