Apply 12.1.0.2.190716 PSU on GI RAC and OJVM

1)Download GI PSU 12.1.0.2.190716 Patch 29698592, and Oracle  JavaVM Component Database PSU 12.1.0.2.190716 Patch 29774383.

2) Download and install latest p6880880_121010_Linux-x86-64.zip for both GI_HOME and RDMS_HOME.

3) Set up PATH

# export PATH=$PATH:/u01/app/12.1.0/grid/OPatch

# opatch version
OPatch Version: 12.2.0.1.17

OPatch succeeded.

4) Apply GI and DB PSU on first node, then do the same on second node.

As root user, execute the following command on each node of the cluster:

# /u01/app/12.1.0/grid/OPatch/opatchauto apply /tmp/29698592

OPatchauto session is initiated at Sun Jul 28 22:39:56 2019

System initialization log file is /u01/app/12.1.0/grid/cfgtoollogs/opatchautodb/systemconfig2019-07-28_10-49-47PM.log.

Session log file is /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/opatchauto2019-07-28_10-51-20PM.log
The id for this session is B1YJ

Executing OPatch prereq operations to verify patch applicability on home /u01/app/12.1.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/12.1.0/dbhome_1
Patch applicability verified successfully on home /u01/app/oracle/product/12.1.0/dbhome_1

Patch applicability verified successfully on home /u01/app/12.1.0/grid
Verifying SQL patch applicability on home /u01/app/oracle/product/12.1.0/dbhome_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/12.1.0/dbhome_1
Preparing to bring down database service on home /u01/app/oracle/product/12.1.0/dbhome_1
Successfully prepared home /u01/app/oracle/product/12.1.0/dbhome_1 to bring down database service
Bringing down CRS service on home /u01/app/12.1.0/grid
Prepatch operation log file location: /u01/app/12.1.0/grid/cfgtoollogs/crsconfig/crspatch_racnode1_2019-07-28_11-31-15PM.log
CRS service brought down successfully on home /u01/app/12.1.0/grid
Performing prepatch operation on home /u01/app/oracle/product/12.1.0/dbhome_1
Perpatch operation completed successfully on home /u01/app/oracle/product/12.1.0/dbhome_1
Start applying binary patch on home /u01/app/oracle/product/12.1.0/dbhome_1
Binary patch applied successfully on home /u01/app/oracle/product/12.1.0/dbhome_1
Performing postpatch operation on home /u01/app/oracle/product/12.1.0/dbhome_1
Postpatch operation completed successfully on home /u01/app/oracle/product/12.1.0/dbhome_1
Start applying binary patch on home /u01/app/12.1.0/grid
Binary patch applied successfully on home /u01/app/12.1.0/grid
Starting CRS service on home /u01/app/12.1.0/grid
Postpatch operation log file location: /u01/app/12.1.0/grid/cfgtoollogs/crsconfig/crspatch_racnode1_2019-07-29_01-30-59AM.log
CRS service started successfully on home /u01/app/12.1.0/grid
Preparing home /u01/app/oracle/product/12.1.0/dbhome_1 after database service restarted
No step execution required.........
Trying to apply SQL patch on home /u01/app/oracle/product/12.1.0/dbhome_1
SQL patch applied successfully on home /u01/app/oracle/product/12.1.0/dbhome_1

OPatchAuto successful.
--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:racnode1
RAC Home:/u01/app/oracle/product/12.1.0/dbhome_1
Version:12.1.0.2.0
Summary:

==Following patches were SKIPPED:

Patch: /tmp/29698592/26983807
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /tmp/29698592/29423125
Reason: This patch is not applicable to this specified target type - "rac_database"
==Following patches were SUCCESSFULLY applied:

Patch: /tmp/29698592/29494060
Log: /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2019-07-28_23-34-47PM_1.log

Patch: /tmp/29698592/29509318
Log: /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2019-07-28_23-34-47PM_1.log
Host:racnode1
CRS Home:/u01/app/12.1.0/grid
Version:12.1.0.2.0
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /tmp/29698592/26983807
Log: /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2019-07-29_00-16-53AM_1.log

Patch: /tmp/29698592/29423125
Log: /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2019-07-29_00-16-53AM_1.log

Patch: /tmp/29698592/29494060
Log: /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2019-07-29_00-16-53AM_1.log

Patch: /tmp/29698592/29509318
Log: /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2019-07-29_00-16-53AM_1.log

OPatchauto session completed at Mon Jul 29 02:12:00 2019
Time taken to complete the session 212 minutes, 4 seconds
[root@racnode1 ~]#

From the log, we can see GI management database was patched as well.

...
..
.
2019-07-29 01:53:43: Mgmtdb is running on node: racnode1; local node: racnode1
2019-07-29 01:53:43: Mgmtdb is running on the local node
2019-07-29 01:53:43: Starting to patch Mgmt DB ...
2019-07-29 01:53:43: Invoking "/u01/app/12.1.0/grid/sqlpatch/sqlpatch -db -MGMTDB"
2019-07-29 01:53:43: Running as user grid: /u01/app/12.1.0/grid/sqlpatch/sqlpatch -db -MGMTDB
2019-07-29 01:53:43: Invoking "/u01/app/12.1.0/grid/sqlpatch/sqlpatch -db -MGMTDB" as user "grid"
2019-07-29 01:53:43: Executing /bin/su grid -c "/u01/app/12.1.0/grid/sqlpatch/sqlpatch -db -MGMTDB"
2019-07-29 01:53:43: Executing cmd: /bin/su grid -c "/u01/app/12.1.0/grid/sqlpatch/sqlpatch -db -MGMTDB"

5) Apply Patch 29774383 – Oracle JavaVM Component 12.1.0.2.190716 Database PSU.

[oracle@racnode1 patches]$ cd 29774383
[oracle@racnode1 29774383]$
[oracle@racnode1 29774383]$opatch prereq CheckConflictAgainstOHWithDetail -ph ./
Oracle Interim Patch Installer version 12.2.0.1.17
Copyright (c) 2019, Oracle Corporation. All rights reserved.

PREREQ session

Oracle Home : /u01/app/oracle/product/12.1.0/dbhome_1
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/12.1.0/dbhome_1/oraInst.loc
OPatch version : 12.2.0.1.17
OUI version : 12.1.0.2.0
Log file location : /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2019-07-31_20-53-44PM_1.log

Invoking prereq "checkconflictagainstohwithdetail"

Prereq "checkConflictAgainstOHWithDetail" passed.

OPatch succeeded.

[oracle@racnode1 29774383]$ opatch apply
-- on node 2

[oracle@racnode2 29774383]$ export PATH=$PATH:$ORACLE_HOME/OPatch
[oracle@racnode2 29774383]$ opatch apply

Loading Modified SQL Files Into the Database

[oracle@racnode2 29774383]$ sqlplus /nolog

SQL>  CONNECT / AS SYSDBA
Connected to an idle instance.

SQL> STARTUP
ORACLE instance started.
Database mounted.
Database opened.

alter system set cluster_database=false scope=spfile;
System altered.

[oracle@racnode2 29774383]$ srvctl stop database -d RACTEST
[oracle@racnode2 29774383]$ sqlplus /nolog

SQL> CONNECT / AS SYSDBA
Connected to an idle instance.

SQL> STARTUP UPGRADE
ORACLE instance started.
Database mounted.
Database opened.

SQL> alter pluggable database all open upgrade;

Pluggable database altered.
[oracle@racnode2 29774383]$ cd $ORACLE_HOME/OPatch
[oracle@racnode2 OPatch]$ ./datapatch -verbose
SQL Patching tool version 12.1.0.2.0 Production on Wed Jul 31 21:21:07 2019
Copyright (c) 2012, 2016, Oracle.  All rights reserved.

Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/
                 sqlpatch_25124_2019_07_31_21_21_07/sqlpatch_invocation.log

Connecting to database...OK
Note:  Datapatch will only apply or rollback SQL fixes for PDBs
       that are in an open state, no patches will be applied to closed PDBs.
    Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
       (Doc ID 1585822.1)
Bootstrapping registry and package to current versions...done
Determining current state...done

Current state of SQL patches:
Patch 29774383 (Database PSU 12.1.0.2.190716, Oracle JavaVM Component (JUL2019)):
  Installed in the binary registry only
Bundle series PSU:
  ID 190716 in the binary registry and ID 190716 in PDB CDB$ROOT, ID 190716 in PDB PDB$SEED, ID 190716 in PDB PDB_A

Adding patches to installation queue and performing prereq checks...
Installation queue:
  For the following PDBs: CDB$ROOT PDB$SEED PDB_A
    Nothing to roll back
    The following patches will be applied:
      29774383 (Database PSU 12.1.0.2.190716, Oracle JavaVM Component (JUL2019))

Installing patches...
Patch installation complete.  Total patches installed: 3

Validating logfiles...
Patch 29774383 apply (pdb CDB$ROOT): SUCCESS
  logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/29774383/22961858/29774383_apply_RACTEST_CDBROOT_2019Jul31_21_22_21.log (no errors)
Patch 29774383 apply (pdb PDB$SEED): SUCCESS
  logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/29774383/22961858/29774383_apply_RACTEST_PDBSEED_2019Jul31_21_25_13.log (no errors)
Patch 29774383 apply (pdb PDB_A): SUCCESS
  logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/29774383/22961858/29774383_apply_RACTEST_PDB_A_2019Jul31_21_25_13.log (no errors)
SQL Patching tool complete on Wed Jul 31 21:26:52 2019
[oracle@racnode2 OPatch]$ sqlplus /nolog

SQL*Plus: Release 12.1.0.2.0 Production on Wed Jul 31 21:29:55 2019

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

SQL> Connect / as sysdba
Connected.
SQL> alter system set cluster_database=true scope=spfile;

System altered.

SQL> SHUTDOWN
Database closed.
Database dismounted.
ORACLE instance shut down.

[oracle@racnode2 OPatch]$ srvctl start  database -d RACTEST

[oracle@racnode2 OPatch]$ sqlplus /as sysdba

SQL> @ $ORACLE_HOME/rdbms/admin/utlrp.sql

Finally check applied patches on both nodes and databases.

$opatch lsinventory
SQL> show con_name

CON_NAME
---------------
CDB$ROOT

SQL> select PATCH_ID, STATUS, VERSION, DESCRIPTION  
       from dba_registry_sqlpatch
  PATCH_ID STATUS     VERSION   DESCRIPTION
---------- ---------- --------- ------------------------------------------
  29494060 SUCCESS    12.1.0.2  DATABASE PATCH SET UPDATE 12.1.0.2.190716
  29774383 SUCCESS    12.1.0.2  Database PSU 12.1.0.2.190716, 
                                Oracle JavaVM Component (JUL2019)
SQL> alter session set container=PDB_A;

Session altered.

SQL> show con_name;

CON_NAME
------------------------------
PDB_A

SQL> select PATCH_ID, STATUS, VERSION, DESCRIPTION 
      from dba_registry_sqlpatch

PATCH_ID  STATUS  VERSION  DESCRIPTION
--------- ------- -------- --------------------------------------
29494060  SUCCESS 12.1.0.2 DATABASE PATCH SET UPDATE 12.1.0.2.190716
29774383  SUCCESS 12.1.0.2 Database PSU 12.1.0.2.190716, 
                           Oracle JavaVM Component (JUL2019)
SQL> show parameter instance_name

NAME           TYPE         VALUE
-------------- ----------- ------------------------
instance_name   string     -MGMTDB

SQL>  select PATCH_ID, STATUS, VERSION, DESCRIPTION  
        from dba_registry_sqlpatch;

  PATCH_ID STATUS  VERSION    DESCRIPTION
---------- ------- ---------- ------------------------------------------
  29494060 SUCCESS 12.1.0.2    DATABASE PATCH SET UPDATE 12.1.0.2.190716

How to Apply GI APR 2018 RU Patches

Here is the GI environment to be patched:

GI HOME:  12.2.0.1
DB HOMES: 12.2.0.1
          12.1.0.2
          11.2.0.4

Patches downloaded :

GI HOME: 12.2.0.1 : Combo OJVM Update 12.2.0.1.180417 and 
                    GI Update 12.2.0.1.180417 Patch 27726454, or
- 27468969/
          27674384 DATABASE APR 2018 RU 12.2.0.1.180417
          27464465 OCW APR 2018 RU 12.2.0.1.0.180417
          27458609 ACFS APR 2018 RU 12.2.0.1.0.180417 Only Grid Home
          26839277 DBWLM RELEASE UPDATE 12.2.0.1.170913 Only Grid Home
          27144050 TOMCAT RELEASE UPDATE 12.2.0.1.171023.0830 Only Grid Home
- 27475613/Oracle JavaVM Component Release Update 12.2.0.1.180417

--
-- 12.1.0.2
Combo OJVM PSU 12.1.0.2.180417 and GI PSU 12.1.0.2.180417 Patch 27726478
- 27475603/
          Oracle JavaVM Component 12.1.0.2.180417 Database PSU
- 27468957/
          27338013
          26983807 
          27338020  OCW 
          27338041  Database PSU 12.1.0.2.180417 Patch 27726471

--
-- 11.2.0.4
Combo OJVM PSU 11.2.0.4.180417 and GI PSU 11.2.0.4.180417 Patch 27726505
- 27475913/
           27338049 Database PSU 11.2.0.4.180417 Patch 27338049
           27441052 OCW PSU 11.2.0.4.180417
           22502505 ACFS PSU 11.2.0.4.160419

-27475598/Oracle JavaVM (OJVM) Component Database PSU 11.2.0.4.180417

1) Apply Apr 2018 RU 12.2.0.1.180417 for 12.2.0.1 GI HOME and DB HOME.

$GI_HOME/OPatch/opatchauto apply /PATCHES/27726454/27468969

2) Apply April 2018 RU for 12.1.0.2 DB HOME.

-- DB   
$cd $PATCH_TOP_LOCATION/27726478/27468957/27338041
$ opatch apply

-- OCW
$ $PATCH_TOP_LOCATION/27726478/27468957/27338020/custom/scripts/prepatch.sh 
 -dbhome  /u01/app/oracle/product/12.1.0/dbhome_1
$ opatch napply $PATCH_TOP_LOCATION/27726478/27468957/27338020 -local 
 -oh /u01/app/oracle/product/12.1.0/dbhome_1 -id 27338020
$ $PATCH_TOP_LOCATION/27726478/27468957/27338020/custom/scripts/postpatch.sh 
  -dbhome -dbhome /u01/app/oracle/product/12.1.0/dbhome_1

--OJAVA
$cd $PATCH_TOP_LOCATION/27726478/27475603
$ opatch apply -local

3)Apply April 2018 RU for 11.2.04 DB HOME.

-- DB PSU
$cd 27338049
$opatch prereq CheckConflictAgainstOHWithDetail -ph ./
$opatch apply or opatch apply -local ( RAC )

-- OJVM
$cd <PATCH_TOP_DIR>/27475598
$opatch prereq CheckConflictAgainstOHWithDetail -ph ./
$opatch apply or  opatch apply -local (if RAC)

--OCW
--11g version 
--chmod 750 on both files below if permission is not correct.

 $ ls -ltr 27475913/27441052/custom/scripts/p*patch.sh
-rw-r--r--. 1 oracle oinstall 4566 Feb 5 20:05 27475913/27441052/custom/scripts/prepatch.sh
-rw-r--r--. 1 oracle oinstall 8417 Feb 5 20:05 27475913/27441052/custom/scripts/postpatch.sh

$ chmod 750 27475913/27441052/custom/scripts/prepatch.sh
$ chmod 750 27475913/27441052/custom/scripts/postpatch.sh
$ $PATCH_TOP_LOCATION/27726505/27475913/27441052/custom/scripts/prepatch.sh 
  -dbhome /u01/app/oracle/product/11.2.0/dbhome_1
$ opatch napply -oh /u01/app/oracle/product/11.2.0/dbhome_1 
 -local PATCH_TOP_LOCATION/27726505/27475913/27441052/custom/server/27441052
$ $PATCH_TOP_LOCATION/27726505/27475913/27441052/custom/scripts/postpatch.sh 
-dbhome /u01/app/oracle/product/11.2.0/dbhome_1

4)Run “datapatch” or post patch SQL as per instructions.

Error in invoking target ‘agent nmhs’ of makefile

The following errors occurred while install 11.2.0.4 on Linux 7.4.  when relink the binary, the same error will be reported as well.

INFO: collect2: error: ld returned 1 exit status

INFO: make[1]:
INFO: *** [/u01/app/oracle/product/11.2.0/dbhome_1/sysman/lib/emdctl] Error 1

INFO: make[1]: Leaving directory `/u01/app/oracle/product/11.2.0/dbhome_1/sysman/lib'

INFO: make: *** [emdctl] Error 2

INFO: End output from spawned process.
INFO: ----------------------------------
SEVERE: oracle.sysman.oii.oiil.OiilActionException: Error in invoking target 'agent nmhs' of makefile '/u01/app/oracle/product/11.2.0/dbhome_1/sysman/lib/ins_emagent.mk'. See '/u01/app/oracle/product/11.2.0/dbhome_1/install/relinkActions2018-05-25_09-48-55-AM.log' for details.
at oracle.sysman.oii.oiis.OiisMakeDeps.invokeMakefile(OiisMakeDeps.java:537)
at oracle.sysman.oii.oiis.OiisMakeDeps.doRelink(OiisMakeDeps.java:614)
at oracle.sysman.oii.oiis.OiisMakeDeps.doOperation(OiisMakeDeps.java:799)
at oracle.sysman.oii.oiis.OiisMakeDeps.main(OiisMakeDeps.java:809)

WORKAROUND

In $ORACLE_HOME/sysman/lib/ins_emagent.mk file :

Change “$(MK_EMAGENT_NMECTL) “

into

$(MK_EMAGENT_NMECTL) -lnnz11

How to Update Opatch Utility of GI Home

Due to GI_HOME restricted access, you might get following errors when you extract opatch utility to GI_HOME:

checkdir error:  cannot create /u01/app/12.2.0.1/grid/OPatch

                 Permission denied

                 unable to process OPatch/config/opatch.properties.

Here is one of the appropriate way for extracting opatch utility :

Subscribe to get access

Read more of this content when you subscribe today.

CLSRSC-540: The root script failed to get a unique name of the OCR backup file on the ASM diskgroup

SYMPTOM

Rootupgrade.sh failed to upgrade GI from 12.1.0.2 to 12.2.0.1.

# /u01/app/12.2.0.1/grid/rootupgrade.sh
..
.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/12.2.0.1/grid/crs/install/crsconfig_params
The log of current session can be found at:
 /u01/app/grid/crsdata/racnode1/crsconfig/rootcrs_racnode1_2018-02-05_12-10-50AM.log
2018/02/05 12:11:10 CLSRSC-595: Executing upgrade step 1 of 19: 'UpgradeTFA'.
2018/02/05 12:11:10 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2018/02/05 12:12:44 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
2018/02/05 12:12:44 CLSRSC-595: Executing upgrade step 2 of 19: 'ValidateEnv'.
2018/02/05 12:13:00 CLSRSC-595: Executing upgrade step 3 of 19: 'GenSiteGUIDs'.
2018/02/05 12:13:03 CLSRSC-595: Executing upgrade step 4 of 19: 'GetOldConfig'.
2018/02/05 12:13:03 CLSRSC-464: Starting retrieval of the cluster configuration data
2018/02/05 12:13:16 CLSRSC-515: Starting OCR manual backup.
2018/02/05 12:13:38 CLSRSC-540: The root script failed to get a unique name of the OCR backup file on the ASM diskgroup using CLSTEST_backup12.1.0.2.0.ocr.
Died at /u01/app/12.2.0.1/grid/crs/install/oraocr.pm line 1353.
The command '/u01/app/12.2.0.1/grid/perl/bin/perl -I/u01/app/12.2.0.1/grid/perl/lib -I/u01/app/12.2.0.1/grid/crs/install /u01/app/12.2.0.1/grid/crs/install/rootcrs.pl -upgrade' execution failed

‘rootcrs_racnode1_2018-02-05_12-10-50AM’ under ‘/u01/app/grid/crsdata/racnode1/crsconfig’

2018-02-05 12:13:37: ORACLE_SID = +ASM1
2018-02-05 12:13:37: ORACLE_HOME = /u01/app/12.1.0.2/grid
2018-02-05 12:13:37: leftVersion=12.1.0.2.0; rightVersion=12.1.0.0.0
2018-02-05 12:13:37: [12.1.0.2.0] is higher than [12.1.0.0.0]
2018-02-05 12:13:37: Running as user grid: /u01/app/12.1.0.2/grid/bin/asmcmd --nocp find --type "OCRBACKUP" "*" "*CLSTEST_backup12.1.0.2.0.ocr*"
2018-02-05 12:13:37: s_run_as_user2: Running /bin/su grid -c ' echo CLSRSC_START; /u01/app/12.1.0.2/grid/bin/asmcmd --nocp find --type "OCRBACKUP" "*" "*CLSTEST_backup12.1.0.2.0.ocr*" '
2018-02-05 12:13:38: Removing file /tmp/rpd0KsVwL2
2018-02-05 12:13:38: Successfully removed file: /tmp/rpd0KsVwL2
2018-02-05 12:13:38: pipe exit code: 0
2018-02-05 12:13:38: /bin/su successfully executed

2018-02-05 12:13:38: Return code: 0 
Output: +OCR_VOTE1/CLSTEST/OCRBACKUP/clstest_backup12.1.0.2.0.ocr.276.967291997

2018-02-05 12:13:38: Reset the environment variable 'ORACLE_HOME' as: /u01/app/12.2.0.1/grid
2018-02-05 12:13:38: Failed to get the OMF name of the OCR backup file on DG.
2018-02-05 12:13:38: Executing cmd: /u01/app/12.2.0.1/grid/bin/clsecho -p has -f clsrsc -m 540 "CLSTEST_backup12.1.0.2.0.ocr"
2018-02-05 12:13:38: Command output:
> CLSRSC-540: The root script failed to get a unique name of the OCR backup file on the ASM diskgroup using CLSTEST_backup12.1.0.2.0.ocr. 
>End Command output
2018-02-05 12:13:38: CLSRSC-540: The root script failed to get a unique name of the OCR backup file on the ASM diskgroup using CLSTEST_backup12.1.0.2.0.ocr.
2018-02-05 12:13:38: ###### Begin DIE Stack Trace ######
2018-02-05 12:13:38: Package File Line Calling 
2018-02-05 12:13:38: --------------- -------------------- ---- ----------
2018-02-05 12:13:38: 1: main rootcrs.pl 287 crsutils::dietrap
2018-02-05 12:13:38: 2: oraClusterwareComp::oraocr oraocr.pm 1353 main::__ANON__
2018-02-05 12:13:38: 3: crsupgrade crsupgrade.pm 1370 oraClusterwareComp::oraocr::backupOcr4Restore
2018-02-05 12:13:38: 4: crsupgrade crsupgrade.pm 817 crsupgrade::get_oldconfig_info
2018-02-05 12:13:38: 5: crsupgrade crsupgrade.pm 485 crsupgrade::CRSUpgrade
2018-02-05 12:13:38: 6: main rootcrs.pl 296 crsupgrade::new
2018-02-05 12:13:38: ####### End DIE Stack Trace #######

CAUSE

It is a bug.  Apply patch 27440094: 12.1.0.2 TO 12.2.0.1 :UPGRADE IS FAILING WITH CLSRSC-540.

WORKAROUD

There are two workarounds for this issue.

a)Apply patch 27440094: 12.1.0.2 TO 12.2.0.1 :UPGRADE IS FAILING WITH CLSRSC-540.

or

b)Reboot all the cluster nodes, and rerun “rootupgrade.sh” on all RAC nodes again.

After either a) or b), then manually complete as per “How to Complete Grid Infrastructure Configuration Assistant(Plug-in) if OUI is not Available (Doc ID 1360798.1)”