Category: Install Upgrade and Patches

ohasd failed to start

SITUATION

1) Two 11.2.0.4 RAC nodes( racnode1/2). Deleted node racnode2 . Upgraded OS from RHEL 4  to RHEL 7 for racnode2, then tried to add node racnode2 back into cluster .

2) export IGNORE_PREADDNODE_CHECKS=Y, Ran addnode.sh ->  orainstRoot.sh->root.sh, then got “ohasd failed to start” error.

...
..
.
ohasd failed to start
Failed to start the Clusterware. Last 20 lines of the alert log follow:
2017-07-24 16:11:23.098:
[client(11401)]CRS-2101:The OLR was formatted using version 3.
2017-07-24 16:11:24.001:
[client(11424)]CRS-1001:The OCR was formatted using version 3.

ohasd failed to start at /u01/app/11.2.0.4/grid/crs/install/roothas.pl line 377, line 4.

SOLUTION

Subscribe to get access

Read more of this content when you subscribe today.

Subscribe

Copy Oracle Home Binary from One RAC Node to Another RAC Node

For some reason RAC oracle home binary corrected on racnode2, so it needs to be copied and reconfigured from racnode1.

Subscribe to get access

Read more of this content when you subscribe today.

Subscribe

Opatch Throws error : java.text.ParseException: Unparseable date: “2016.Aug.09 08:27:15 AEST

Oracle opatch utility should have used the latest available JRE to avoid those kind of TZ issues. It is a bug obviously.

SYMPTOMS

For 12.1.0.2 GI Home,  opatch  12.2.0.1.5 and later versions  throw below errors :

$ opatch lsinventory
Oracle Interim Patch Installer version 12.2.0.1.5
Copyright (c) 2016, Oracle Corporation. All rights reserved.


Oracle Home : /u01/app/12.1.0.2/grid
Central Inventory : /u01/app/oraInventory
 from : /u01/app/12.1.0.2/grid/oraInst.loc
OPatch version : 12.2.0.1.5
OUI version : 12.1.0.2.0
Log file location : /u01/app/12.1.0.2/grid/cfgtoollogs/opatch/opatch2016-09-08_17-08-01PM_1.log

sInstallTime = >2016.Aug.09 08:27:15 AEST<
java.text.ParseException: Unparseable date: "2016.Aug.09 08:27:15 AEST"
 at java.text.DateFormat.parse(DateFormat.java:337)
 at oracle.sysman.oii.oiii.OiiiCompsXMLHandler$OiiiInstallCompElementHandler.onStart(OiiiCompsXMLHandler.java:923)
 at oracle.sysman.oix.oixs.OixsElementHandler.startElement(OixsElementHandler.java:140)

CAUSE

  1. opatch is using $ORACLE_HOME/jdk/jre, which is version 1.6. But opatch has its own jre version 1.8, which is not used by opatch. it does not make sense.
  2. TZ entry AEST is added into $ORACLE_HOME/inventory/ContentsXML/comps.xml.
  3. Before opatch 12.2.0.1.5, TZ EST was used instead. Java1.6 does not understand AEST.

SOLUTION

  1. For “opatch” command failure. 
    $ORACLE_HOME/OPatch/opatch lsinventory -jre $ORACLE_HOME/OPatch/jre
  2. For  ‘opatchauto apply’ failure.
$cd $ORACLE_HOME/OPatch
[+ASM1] grid@racnode1:/u01/app/12.1.0.2/grid/OPatch$ mv opatch opatch.bin
[+ASM1] grid@racnode1:/u01/app/12.1.0.2/grid/OPatch$ echo "/u01/app/12.1.0.2/grid/OPatch/opatch.bin $@ -jre /u01/app/12.1.0.2/grid/OPatch/jre" > opatch
[+ASM1] grid@racnode1:/u01/app/12.1.0.2/grid/OPatch$ chmod 750 opatch
[+ASM1] grid@racnode1:/u01/app/12.1.0.2/grid/OPatch$ cat opatch
/u01/app/12.1.0.2/grid/OPatch/opatch.bin $@ -jre /u01/app/12.1.0.2/grid/OPatch/jre

Is MGMTDB Patched While Applying GI PSU/RU/RUR ?

MGMTDB will be patched as well at the same time when GI is upgraded or patched ( PSU, RU/RUR, etc).

Yes, while applying PSU patch onto GI, we can see GIMR database “-MGMTDB” is patched as well. It can be confirmed from logs.

a) Logs from “opatchauto apply” shows MGMTDB is patched while applying GI PSU:

System initialization log file is /u01/app/12.1.0.2/grid/cfgtoollogs/opatchautodb/systemconfig2016-09-19_12-46-41PM.log.

Session log file is /u01/app/12.1.0.2/grid/cfgtoollogs/opatchauto/opatchauto2016-09-19_12-46-55PM.log
The id for this session is 2G1F
[init:init] Executing OPatchAutoBinaryAction action on home /u01/app/12.1.0.2/grid

Executing OPatch prereq operations to verify patch applicability on CRS Home........

[init:init] OPatchAutoBinaryAction action completed on home /u01/app/12.1.0.2/grid successfully
[init:init] Executing GIRACPrereqAction action on home /u01/app/12.1.0.2/grid

Executing prereq operations before applying on CRS Home........

[init:init] GIRACPrereqAction action completed on home /u01/app/12.1.0.2/grid successfully
[shutdown:shutdown] Executing GIShutDownAction action on home /u01/app/12.1.0.2/grid

Performing prepatch operations on CRS Home........

Prepatch operation log file location: /u01/app/12.1.0.2/grid/cfgtoollogs/crsconfig/crspatch_racnode2_2016-09-19_12-47-24AM.log

[shutdown:shutdown] GIShutDownAction action completed on home /u01/app/12.1.0.2/grid successfully
[offline:binary-patching] Executing OPatchAutoBinaryAction action on home /u01/app/12.1.0.2/grid

Start applying binary patches on CRS Home........

[offline:binary-patching] OPatchAutoBinaryAction action completed on home /u01/app/12.1.0.2/grid successfully
[startup:startup] Executing GIStartupAction action on home /u01/app/12.1.0.2/grid

Performing postpatch operations on CRS Home........

Postpatch operation log file location: /u01/app/12.1.0.2/grid/cfgtoollogs/crsconfig/crspatch_racnode2_2016-09-19_12-50-36AM.log

[startup:startup] GIStartupAction action completed on home /u01/app/12.1.0.2/grid successfully
[finalize:finalize] Executing OracleHomeLSInventoryGrepAction action on home /u01/app/12.1.0.2/grid

Verifying patches applied on CRS Home.

[finalize:finalize] OracleHomeLSInventoryGrepAction action completed on home /u01/app/12.1.0.2/grid successfully
OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:racnode2
CRS Home:/u01/app/12.1.0.2/grid
Summary:

==Following patches were SKIPPED:

Patch: /u01/app/software/23615308/23177536
Reason: This patch is not applicable to this specified target type - "cluster"


==Following patches were SUCCESSFULLY applied:

Patch: /u01/app/software/23615308/21436941
Log: /u01/app/12.1.0.2/grid/cfgtoollogs/opatchauto/core/opatch/opatch2016-09-19_12-47-55PM_1.log

Patch: /u01/app/software/23615308/23054246
Log: /u01/app/12.1.0.2/grid/cfgtoollogs/opatchauto/core/opatch/opatch2016-09-19_12-47-55PM_1.log

Patch: /u01/app/software/23615308/23054327
Log: /u01/app/12.1.0.2/grid/cfgtoollogs/opatchauto/core/opatch/opatch2016-09-19_12-47-55PM_1.log

Patch: /u01/app/software/23615308/23054341
Log: /u01/app/12.1.0.2/grid/cfgtoollogs/opatchauto/core/opatch/opatch2016-09-19_12-47-55PM_1.log

b) Check Postpatch operation log file /u01/app/12.1.0.2/grid/cfgtoollogs/crsconfig/crspatch_racnode2_2016-09-19_12-50-36AM.log

$ tail -120 /u01/app/12.1.0.2/grid/cfgtoollogs/crsconfig/crspatch_racnode2_2016-09-19_12-50-36AM.log
...

....


2016-09-19 12:52:30: Executing cmd: /u01/app/12.1.0.2/grid/bin/srvctl config mgmtdb
2016-09-19 12:52:31: Command output:
> Database unique name: _mgmtdb
> Database name:
> Oracle home: <CRS home>
> Oracle user: grid
> Spfile: +OCR_VOTE/_MGMTDB/PARAMETERFILE/spfile.268.922884353
> Password file:
> Domain:
> Start options: open
> Stop options: immediate
> Database role: PRIMARY
> Management policy: AUTOMATIC
> Type: Management
> PDB name: racnode-cluster
> PDB service: racnode-cluster
> Cluster name: racnode-cluster
> Database instance: -MGMTDB
>End Command output
2016-09-19 12:52:31: isMgmtdbConfigured: 1
2016-09-19 12:52:31: setting ORAASM_UPGRADE to 1
2016-09-19 12:52:31: Executing cmd: /u01/app/12.1.0.2/grid/bin/crsctl query crs softwarepatch racnode2
2016-09-19 12:52:31: Command output:
> Oracle Clusterware patch level on node racnode2 is [3696455212].
>End Command output
2016-09-19 12:52:31: Oracle Clusterware patch level on node 'racnode2' is [3696455212]
2016-09-19 12:52:31: setting ORAASM_UPGRADE to 1
2016-09-19 12:52:31: Executing cmd: /u01/app/12.1.0.2/grid/bin/crsctl query crs softwarepatch racnode1
2016-09-19 12:52:31: Command output:
> Oracle Clusterware patch level on node racnode1 is [3696455212].
>End Command output
2016-09-19 12:52:31: Oracle Clusterware patch level on node 'racnode1' is [3696455212]
2016-09-19 12:52:31: The local node has the same software patch level [3696455212] as remote node 'racnode1'
2016-09-19 12:52:31: Executing cmd: /u01/app/12.1.0.2/grid/bin/crsctl query crs releasepatch
2016-09-19 12:52:31: Command output:
> Oracle Clusterware release patch level is [3696455212] and the complete list of patches [19769480 20299023 20831110 21359755 21436941 21948354 22291127 23054246 23054327 23054341 ] have been applied on the local node.
>End Command output
2016-09-19 12:52:31: Oracle Clusterware release patch level is [3696455212]
2016-09-19 12:52:31: setting ORAASM_UPGRADE to 1
2016-09-19 12:52:31: Executing cmd: /u01/app/12.1.0.2/grid/bin/crsctl query crs activeversion -f
2016-09-19 12:52:31: Command output:
> Oracle Clusterware active version on the cluster is [12.1.0.2.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [3696455212].
>End Command output
2016-09-19 12:52:31: Oracle Clusterware active patch level is [3696455212]
2016-09-19 12:52:31: The Clusterware active patch level [3696455212] has been updated to [3696455212].
2016-09-19 12:52:31: Postpatch: isLastNode is 1
2016-09-19 12:52:31: Last node: enable Mgmt DB globally
2016-09-19 12:52:31: Invoking "/u01/app/12.1.0.2/grid/bin/srvctl enable mgmtdb"
2016-09-19 12:52:31: trace file=/u01/app/12.1.0.2/grid/cfgtoollogs/crsconfig/srvmcfg1.log
2016-09-19 12:52:31: Running as user grid: /u01/app/12.1.0.2/grid/bin/srvctl enable mgmtdb
2016-09-19 12:52:31: Invoking "/u01/app/12.1.0.2/grid/bin/srvctl enable mgmtdb" as user "grid"
2016-09-19 12:52:31: Executing /bin/su grid -c "/u01/app/12.1.0.2/grid/bin/srvctl enable mgmtdb"
2016-09-19 12:52:31: Executing cmd: /bin/su grid -c "/u01/app/12.1.0.2/grid/bin/srvctl enable mgmtdb"
2016-09-19 12:52:32: Modifying 10.2 resources
2016-09-19 12:52:32: Invoking "/u01/app/12.1.0.2/grid/bin/srvctl upgrade model -pretb"
2016-09-19 12:52:32: trace file=/u01/app/12.1.0.2/grid/cfgtoollogs/crsconfig/srvmcfg2.log
2016-09-19 12:52:32: Executing /u01/app/12.1.0.2/grid/bin/srvctl upgrade model -pretb
2016-09-19 12:52:32: Executing cmd: /u01/app/12.1.0.2/grid/bin/srvctl upgrade model -pretb
2016-09-19 12:52:32: 'srvctl upgrade model -pretb' ... success
2016-09-19 12:52:32: Executing cmd: /u01/app/12.1.0.2/grid/bin/srvctl status mgmtdb -S 1
2016-09-19 12:52:32: Command output:
> #@=result[0]: dbunique_name={_mgmtdb} inst_name={-MGMTDB} node_name={racnode2} up={true} state_details={Open} internal_state={STABLE}
>End Command output
2016-09-19 12:52:32: Mgmtdb is running on node: racnode2; local node: racnode2
2016-09-19 12:52:32: Mgmtdb is running on the local node
2016-09-19 12:52:32: Starting to patch Mgmt DB ...
2016-09-19 12:52:32: Invoking "/u01/app/12.1.0.2/grid/sqlpatch/sqlpatch -db -MGMTDB"
2016-09-19 12:52:32: Running as user grid: /u01/app/12.1.0.2/grid/sqlpatch/sqlpatch -db -MGMTDB
2016-09-19 12:52:32: Invoking "/u01/app/12.1.0.2/grid/sqlpatch/sqlpatch -db -MGMTDB" as user "grid"
2016-09-19 12:52:32: Executing /bin/su grid -c "/u01/app/12.1.0.2/grid/sqlpatch/sqlpatch -db -MGMTDB"
2016-09-19 12:52:32: Executing cmd: /bin/su grid -c "/u01/app/12.1.0.2/grid/sqlpatch/sqlpatch -db -MGMTDB"
2016-09-19 12:53:39: Command output:
> SQL Patching tool version 12.1.0.2.0 on Mon Sep 19 12:52:32 2016
> Copyright (c) 2016, Oracle. All rights reserved.
>
> Connecting to database...OK
> Note: Datapatch will only apply or rollback SQL fixes for PDBs
> that are in an open state, no patches will be applied to closed PDBs.
> Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
> (Doc ID 1585822.1)
> Determining current state...done
> Adding patches to installation queue and performing prereq checks...done
> Installation queue:
> For the following PDBs: CDB$ROOT PDB$SEED RACNODE-CLUSTER
> Nothing to roll back
> The following patches will be applied:
> 23054246 (Database Patch Set Update : 12.1.0.2.160719 (23054246))
>
> Installing patches...
> Patch installation complete. Total patches installed: 3
>
> Validating logfiles...done
> SQL Patching tool complete on Mon Sep 19 12:53:39 2016
>End Command output
2016-09-19 12:53:39: Successfully patched Mgmt DB
2016-09-19 12:53:39: Invoking "/u01/app/12.1.0.2/grid/bin/cluutil -ckpt -oraclebase /u01/app/grid -writeckpt -name ROOTCRS_POSTPATCH -state SUCCESS"
2016-09-19 12:53:39: trace file=/u01/app/grid/crsdata/racnode2/crsconfig/cluutil7.log
2016-09-19 12:53:39: Running as user grid: /u01/app/12.1.0.2/grid/bin/cluutil -ckpt -oraclebase /u01/app/grid -writeckpt -name ROOTCRS_POSTPATCH -state SUCCESS
2016-09-19 12:53:39: s_run_as_user2: Running /bin/su grid -c ' echo CLSRSC_START; /u01/app/12.1.0.2/grid/bin/cluutil -ckpt -oraclebase /u01/app/grid -writeckpt -name ROOTCRS_POSTPATCH -state SUCCESS '
2016-09-19 12:53:39: Removing file /tmp/filei2kicY
2016-09-19 12:53:39: Successfully removed file: /tmp/filei2kicY
2016-09-19 12:53:39: pipe exit code: 0
2016-09-19 12:53:39: /bin/su successfully executed

2016-09-19 12:53:39: Succeeded in writing the checkpoint:'ROOTCRS_POSTPATCH' with status:SUCCESS
[+ASM2] grid@racnode2:~$

c) Querying PDB shows MGMTDB has been patched as well.

-- check CDB$ROOT

SQL> set pagesize 120
SQL> set linesize 180
SQL> select * from dba_registry_sqlpatch;

PATCH_ID PATCH_UID VERSION FLAGS ACTION STATUS ACTION_TIME
---------- ---------- -------------------- ---------- --------------- --------------- ---------------------------------------------------------------------------
DESCRIPTION BUNDLE_SERIES BUNDLE_ID
---------------------------------------------------------------------------------------------------- ------------------------------ ----------
BUNDLE_DATA
--------------------------------------------------------------------------------
LOGFILE
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

23054246 20213895 12.1.0.2 NB APPLY SUCCESS 19-SEP-16 12.53.38.034518 PM
Database Patch Set Update : 12.1.0.2.160719 (23054246) PSU 160719
<bundledata version="12.1.0.2.1" series="Patch Set Update">
 <bundle id="1" des
/u01/app/grid/cfgtoollogs/sqlpatch/23054246/20213895/23054246_apply__MGMTDB_CDBROOT_2016Sep19_12_53_18.log



-- check PDB RACNODE-CLUSTER

SQL> alter session set container=racnode-cluster;

SQL> select * from dba_registry_sqlpatch;

PATCH_ID PATCH_UID VERSION FLAGS ACTION STATUS ACTION_TIME
---------- ---------- -------------------- ---------- --------------- --------------- ---------------------------------------------------------------------------
DESCRIPTION BUNDLE_SERIES BUNDLE_ID
---------------------------------------------------------------------------------------------------- ------------------------------ ----------
BUNDLE_DATA
--------------------------------------------------------------------------------
LOGFILE
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
23054246 20213895 12.1.0.2 NB APPLY SUCCESS 19-SEP-16 12.53.38.457190 PM
Database Patch Set Update : 12.1.0.2.160719 (23054246) PSU 160719
<bundledata version="12.1.0.2.1" series="Patch Set Update">
 <bundle id="1" des
/u01/app/grid/cfgtoollogs/sqlpatch/23054246/20213895/23054246_apply__MGMTDB_RACNODE-CLUSTER_2016Sep19_12_53_32.log

How to Relink 12c Oracle GI / RAC Binaries after OS Upgrade

It is recommended to relink GI and RAC Home binaries after OS upgrading or patching.

This post demonstrates how to relink Oracle Grid Infrastructure ( GI ) and RAC Home binaries after OS upgrading or patching.

Subscribe to get access

Read more of this content when you subscribe today.

Subscribe