opatchauto : Unable to create patchObject

SYMPTOMS

When applying 19c GI JULY 2020 RU ( 19.8.0.0), opatchauto failed with following errors:

#opatchauto apply /tmp/31305339/
...
..
.

Patch: /tmp/31305339/31281355
Log:
Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Unable to create patchObject
Possible causes are:
ORACLE_HOME/inventory/oneoffs/31281355 is corrupted. PatchObject constructor: Input file "/u01/app/19.0.0/grid/inventory/oneoffs/31281355/etc/config/actions" or "/u01/app/19.0.0/grid/inventory/oneoffs/31281355/etc/config/inventory" does not exist.
After fixing the cause of failure Run opatchauto resume
]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.
OPatchauto session completed at Mon Aug 3 21:06:44 2020
Time taken to complete the session 0 minute, 35 seconds
opatchauto failed with error code 42

Subscribe to get access

Read more of this content when you subscribe today.

opatchauto Prerequisite check “CheckActiveFilesAndExecutables” failed

SYMPTOMS

The following errors occur while running opatchauto to apply 19.8 GI RU:

[root@racnode1 ~]# opatchauto apply /tmp/31305339/
...
..
.
Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Prerequisite check "CheckActiveFilesAndExecutables" failed.

After fixing the cause of failure Run opatchauto resume

]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.

OPatchauto session completed at Mon Aug 3 17:50:34 2020
Time taken to complete the session 1 minute, 22 seconds

opatchauto failed with error code 42

Subscribe to get access

Read more of this content when you subscribe today.

CLSRSC-488: Patching the Grid Infrastructure Management Repository database failed

opatchauo failed when applying GI release update.

[root@racnode1 ~]# /u01/app/12.2.0.1/grid/OPatch/opatchauto 
                     apply /media/sf_Software/12201-190115/30501932
...
..
.
CRS-4123: Oracle High Availability Services has been started.
Oracle Clusterware active version on the cluster is [12.2.0.1.0]. 
                    The cluster upgrade state is [ROLLING PATCH]. 
The cluster active patch level is [1477401644].
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'racnode1'
CRS-2676: Start of 'ora.drivers.acfs' on 'racnode1' succeeded
SQL Patching tool version 12.2.0.1.0 Production on Sat Mar 7 15:46:51 2020
Copyright (c) 2012, 2019, Oracle. All rights reserved.

Connecting to database...OK
Note: Datapatch will only apply or rollback SQL fixes for PDBs
that are in an open state, no patches will be applied to closed PDBs.
Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
(Doc ID 1585822.1)
Bootstrapping registry and package to current versions...done
Determining current state...done
Adding patches to installation queue and performing prereq checks...done
Installation queue:
For the following PDBs: CDB$ROOT PDB$SEED GIMR_DSCREP_10
The following patches will be rolled back:
29708478 (DATABASE JAN 2019 RELEASE UPDATE REVISION 12.2.0.1.190716)
The following patches will be applied:
30593149 (DATABASE JAN 2020 RELEASE UPDATE 12.2.0.1.200114)

Installing patches...
Patch installation complete. Total patches installed: 6

Validating logfiles...
Patch 29708478 rollback (pdb CDB$ROOT): WITH ERRORS
logfile: /u01/app/grid/cfgtoollogs/sqlpatch/29708478/22988073/
   29708478_rollback__MGMTDB_CDBROOT_2020Mar07_15_47_52.log (errors)
Error at line 25435: Warning: Package Body created with compilation errors.
Error at line 25442: 24/12 PLS-00323: subprogram or cursor 'GET_OPTIONAL_PRIVILEGES' is
Error at line 25446: 2727/5 PL/SQL: Statement ignored
Error at line 25447: 2727/21 PLS-00306: wrong number or types of arguments in call to
Error at line 36437: Warning: Package Body created with compilation errors.
Error at line 36444: 18788/3 PL/SQL: Statement ignored
Error at line 36445: 18788/6 PLS-00201: identifier
Error at line 36448: 18792/3 PL/SQL: Statement ignored
Error at line 36449: 18792/3 PLS-00201: identifier
Patch 29708478 rollback (pdb PDB$SEED): WITH ERRORS
logfile: /u01/app/grid/cfgtoollogs/sqlpatch/29708478/22988073/
   29708478_rollback__MGMTDB_PDBSEED_2020Mar07_15_50_23.log (errors)
Error at line 36419: Warning: Package Body created with compilation errors.
Error at line 36426: 18788/3 PL/SQL: Statement ignored
Error at line 36427: 18788/6 PLS-00201: identifier
Error at line 36430: 18792/3 PL/SQL: Statement ignored
Error at line 36431: 18792/3 PLS-00201: identifier
Patch 29708478 rollback (pdb GIMR_DSCREP_10): WITH ERRORS
logfile: /u01/app/grid/cfgtoollogs/sqlpatch/29708478/22988073/
29708478_rollback__MGMTDB_GIMR_DSCREP_10_2020Mar07_15_50_23.log (errors)
Error at line 52094: Warning: Package Body created with compilation errors.
Error at line 52101: 18788/3 PL/SQL: Statement ignored
Error at line 52102: 18788/6 PLS-00201: identifier
Error at line 52105: 18792/3 PL/SQL: Statement ignored
Error at line 52106: 18792/3 PLS-00201: identifier

Please refer to MOS Note 1609718.1 and/or the invocation log
/u01/app/grid/cfgtoollogs/sqlpatch/sqlpatch_9475_2020_03_07_15_46_51/sqlpatch_invocation.log
for information on how to resolve the above errors.

SQL Patching tool complete on Sat Mar 7 15:54:51 2020
2020/03/07 15:54:51 CLSRSC-488: Patching the Grid Infrastructure 
                            Management Repository database failed.

After fixing the cause of failure Run opatchauto resume

]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.

OPatchauto session completed at Sat Mar 7 15:54:52 2020
Time taken to complete the session 22 minutes, 24 seconds

opatchauto failed with error code 42
[root@racnode1 ~]#

Subscribe to get access

Read more of this content when you subscribe today.

CRS-6706: Oracle Clusterware Release patch level (‘nnnnnn’) does not match Software patch level (‘nnnnnn’)

“opatchauto” failed in the middle, try to rerun again, get “CRS-6706” error.

# /u01/app/12.1.0.2/grid/OPatch/opatchauto apply /tmp/12.1.0.2/27010872 -oh /u01/app/12.1.0.2/grid
...
..
.
Using configuration parameter file: /u01/app/12.1.0.2/grid/OPatch/auto/dbtmp/bootstrap_racnode1/patchwork/crs/install/crsconfig_params
CRS-6706: Oracle Clusterware Release patch level ('173535486') does not match Software patch level ('2039526626'). Oracle Clusterware cannot be started.
CRS-4000: Command Start failed, or completed with errors.
2018/01/23 16:29:02 CLSRSC-117: Failed to start Oracle Clusterware stack


After fixing the cause of failure Run opatchauto resume

]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.

OPatchauto session completed at Tue Jan 23 16:29:04 2018
Time taken to complete the session 1 minute, 39 seconds

opatchauto failed with error code 42

On racnode1:

$/u01/app/12.1.0.2/grid/bin/kfod op=patches
---------------
List of Patches
===============
19941482
19941477
19694308
19245012
26925218   <---- Does not exist on node2

$/u01/app/12.1.0.2/grid/bin/kfod op=patchlvl
2039526626

On racnode2:

$/u01/app/12.1.0.2/grid/bin/kfod op=patches
---------------
List of Patches
===============
19941482
19941477
19694308
19245012

$/u01/app/12.1.0.2/grid/bin/kfod op=patchlvl
2039526626

We can see patch 26925218 has been applied onto racnode 1, the solution is :

1) Rollback this patch ( 26925218), and run “opatchauto” again to finish the patching successfully. then everything should be fine.

OR

2) Manually complete all the left patches in the GI. After this everything is fine.

In 12c, GI home must have identical patches for the clusterware to start unless during rolling patching.
After applied the same patches on all nodes, GI started fine.

———-

Another situation you might meet is all nodes have same patches but ‘opatch lsinventory’ shows the different patch level:

For example , on racnode1:

$ /u01/app/12.1.0.2/grid/bin/kfod op=patches
---------------
List of Patches
===============
11111111
22222222
33333333

$ /u01/app/12.1.0.2/grid/bin/kfod op=patchlvl
-------------------
Current Patch level
===================
8888888888

Node2

$ /u01/app/12.1.0.2/grid/bin/kfod op=patches
---------------
List of Patches
===============
11111111
22222222
33333333

$ /u01/app/12.1.0.2/grid/bin/kfod op=patchlvl
-------------------
Current Patch level
===================
9999999999

However opatch lsinventory shows the different patch level:

Patch level status of Cluster nodes :

Patching Level Nodes
-------------- -----
8888888888 node1                    ====>> different patch level
9999999999 node2

For 12.1.0.1/2:

Execute”/u01/app/12.1.0.2/grid/crs/install/rootcrs.sh -patch” as root user on the problematic node and the patch level should be corrected.

For 12.2

Execute”<GI_HOME>/crs/install/rootcrs.pl -prepatch”  “<GI_HOME>/crs/install/rootcrs.pl -postpatch”and as <root_user> on the problematic node and the patch level should be corrected

REFERENCES:

RS-6706: Oracle Clusterware Release patch level (‘nnn’) does not match Software patch level (‘mmm’) (Doc ID 1639285.1)