Unable to get logical block size for spfile ‘+OCR_VOTE/_mgmtdb/spfile-MGMTDB.ora’

SYMPTOM

In alert log of management DB in GI ( Grid Infrastructure ) environment, there are following errors everyday:

Fri Jul 03 00:54:36 2020
ERROR: Unable to get logical block size for spfile '+OCR_VOTE/_mgmtdb/spfile-MGMTDB.ora'.
Fri Jul 03 00:54:36 2020
ERROR: Unable to get logical block size for spfile '+OCR_VOTE/_mgmtdb/spfile-MGMTDB.ora'.
Fri Jul 03 00:54:36 2020
ERROR: Unable to get logical block size for spfile '+OCR_VOTE/_mgmtdb/spfile-MGMTDB.ora'.
Fri Jul 03 00:54:36 2020
ERROR: Unable to get logical block size for spfile '+OCR_VOTE/_mgmtdb/spfile-MGMTDB.ora'.
Fri Jul 03 00:54:36 2020
ERROR: Unable to get logical block size for spfile '+OCR_VOTE/_mgmtdb/spfile-MGMTDB.ora'.

Subscribe to get access

Read more of this content when you subscribe today.

Creating Oracle Grid Infrastructure Management Repository Failed with ORA-15221

When upgrading Oracle 18c GI to 19c GI, Creating Oracle Grid Infrastructure Management Repository Failed.

Subscribe to get access

Read more of this content when you subscribe today.

Retry and continue the upgrading process successfully.

PRCD-1334 PRCR-1065 CRS-2974 when stopping mgmtdb

Try to stop mgmtdb, and relocate mgmtdb to another node with below errors:

$ srvctl status mgmtdb
Database is enabled
Instance -MGMTDB is running on node racnode1

$ srvctl stop mgmtdb
PRCD-1334 : failed to stop database _mgmtdb
PRCR-1065 : Failed to stop resource ora.mgmtdb
CRS-2974: unable to act on resource 'ora.mgmtdb' on server 'racnode1' 
because that would require stopping or relocating resource 'ora.chad' 
but the -force option was not specified

Just relocate mgmtdb, it works.

$ srvctl relocate mgmtdb -node racnode2


$ srvctl status mgmtdb
Database is enabled
Instance -MGMTDB is running on node racnode2

Tablespace SYSMGMTDATA Is Full in GIMR Database (-MGMTDB)

Cluster Health Monitor ( CHM) tables could be fragmented in MGMTDB database.

1) Check the datafile of tablespace SYSMGMTDATA in PDB:

SQL> select PDB_NAME from dba_pdbs;

PDB_NAME
-------------
RACTEST
PDB$SEED

SQL> alter session set container=RACTEST;
Session altered. 

SQL> select TABLESPACE_NAME,FILE_NAME,AUTOEXTENSIBLE,
            BYTES/1024/1024,MAXBYTES/1024/1024 
      from dba_data_files 
     where TABLESPACE_NAME='SYSMGMTDATA';

TABLESPACE_NAME 
--------------- 
FILE_NAME
----------------------------------------------------------------------   
AUT BYTES/1024/1024 MAXBYTES/1024/1024
--- --------------- -------------------
SYSMGMTDATA      +OCR_VOTE/_MGMTDB/FD9B43BF6A646F8CE043B6A9E80A2815/DATAFILE/sysmgmtdata.269.922884445 
NO   2048              2048

2) Check CHM data in repository, and we can see the purging is working:

SQL> select trunc(BEGINTIME), count(*) 
       from chm.CHMOS_PROCESS_INT_TBL 
   group by trunc(BEGINTIME) 
   order by trunc(BEGINTIME);

TRUNC(BEG  COUNT(*)
---------  ---------
21-FEB-17  1222178
22-FEB-17  3265667
23-FEB-17  1327167

3) Check the current CHM repository retention in seconds :

$ oclumon manage -get reppath

CHM Repository Path = +OCR_VOTE/_MGMTDB/FD9B43BF6A646F8CE043B6A9E80A2815/DATAFILE/sysmgmtdata.269.922884445

$ oclumon manage -get repsize

CHM Repository Size = 136320 seconds

4) Check the big objects in repository, they are all from two partition tables CHMOS_PROCESS_INT_TBL and CHMOS_DEVICE_INT_TBL.

SQL> select owner, segment_name,segment_type , bytes/1024/1024 
       from dba_segments 
      where owner='CHM' 
        and bytes/1024/1024>10;
OWNER      SEGMENT_NAME            SEGMENT_TYPE       BYTES/1024/1024
---------- ----------------------  ------------------ ---------------
CHM        CHMOS_PROCESS_INT_TBL    TABLE PARTITION   18
CHM        CHMOS_DEVICE_INT_TBL     TABLE PARTITION   18
CHM        CHMOS_PROCESS_INT_TBL    TABLE PARTITION   18
CHM        CHMOS_DEVICE_INT_TBL     TABLE PARTITION   18

Solution 1 :  Move GIMR repository onto to a different disk group as per “How to Move 12c Grid Infrastructure Management Repository ( GIMR ) to Another Diskgroup“.

Solution 2: Manually truncate the big objects owned by CHM as per Doc ID 2177879.1. Normally the following two tables can be truncated :

SQL> truncate table CHM.CHMOS_PROCESS_INT_TBL;
SQL> truncate table CHM.CHMOS_DEVICE_INT_TBL;

Solution 3:  Resize datafile bigger. For example, increases the size from 2048M to 2560M

SQL> alter database datafile '+OCR_VOTE/_MGMTDB/FD9B43BF6A646F8CE043B6A9E80A2815/DATAFILE/sysmgmtdata.269.922884445' resize 2560m;

Database altered.

Is MGMTDB Patched While Applying GI PSU/RU/RUR ?

MGMTDB will be patched as well at the same time when GI is upgraded or patched ( PSU, RU/RUR, etc).

Yes, while applying PSU patch onto GI, we can see GIMR database “-MGMTDB” is patched as well. It can be confirmed from logs.

a) Logs from “opatchauto apply” shows MGMTDB is patched while applying GI PSU:

System initialization log file is /u01/app/12.1.0.2/grid/cfgtoollogs/opatchautodb/systemconfig2016-09-19_12-46-41PM.log.

Session log file is /u01/app/12.1.0.2/grid/cfgtoollogs/opatchauto/opatchauto2016-09-19_12-46-55PM.log
The id for this session is 2G1F
[init:init] Executing OPatchAutoBinaryAction action on home /u01/app/12.1.0.2/grid

Executing OPatch prereq operations to verify patch applicability on CRS Home........

[init:init] OPatchAutoBinaryAction action completed on home /u01/app/12.1.0.2/grid successfully
[init:init] Executing GIRACPrereqAction action on home /u01/app/12.1.0.2/grid

Executing prereq operations before applying on CRS Home........

[init:init] GIRACPrereqAction action completed on home /u01/app/12.1.0.2/grid successfully
[shutdown:shutdown] Executing GIShutDownAction action on home /u01/app/12.1.0.2/grid

Performing prepatch operations on CRS Home........

Prepatch operation log file location: /u01/app/12.1.0.2/grid/cfgtoollogs/crsconfig/crspatch_racnode2_2016-09-19_12-47-24AM.log

[shutdown:shutdown] GIShutDownAction action completed on home /u01/app/12.1.0.2/grid successfully
[offline:binary-patching] Executing OPatchAutoBinaryAction action on home /u01/app/12.1.0.2/grid

Start applying binary patches on CRS Home........

[offline:binary-patching] OPatchAutoBinaryAction action completed on home /u01/app/12.1.0.2/grid successfully
[startup:startup] Executing GIStartupAction action on home /u01/app/12.1.0.2/grid

Performing postpatch operations on CRS Home........

Postpatch operation log file location: /u01/app/12.1.0.2/grid/cfgtoollogs/crsconfig/crspatch_racnode2_2016-09-19_12-50-36AM.log

[startup:startup] GIStartupAction action completed on home /u01/app/12.1.0.2/grid successfully
[finalize:finalize] Executing OracleHomeLSInventoryGrepAction action on home /u01/app/12.1.0.2/grid

Verifying patches applied on CRS Home.

[finalize:finalize] OracleHomeLSInventoryGrepAction action completed on home /u01/app/12.1.0.2/grid successfully
OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:racnode2
CRS Home:/u01/app/12.1.0.2/grid
Summary:

==Following patches were SKIPPED:

Patch: /u01/app/software/23615308/23177536
Reason: This patch is not applicable to this specified target type - "cluster"


==Following patches were SUCCESSFULLY applied:

Patch: /u01/app/software/23615308/21436941
Log: /u01/app/12.1.0.2/grid/cfgtoollogs/opatchauto/core/opatch/opatch2016-09-19_12-47-55PM_1.log

Patch: /u01/app/software/23615308/23054246
Log: /u01/app/12.1.0.2/grid/cfgtoollogs/opatchauto/core/opatch/opatch2016-09-19_12-47-55PM_1.log

Patch: /u01/app/software/23615308/23054327
Log: /u01/app/12.1.0.2/grid/cfgtoollogs/opatchauto/core/opatch/opatch2016-09-19_12-47-55PM_1.log

Patch: /u01/app/software/23615308/23054341
Log: /u01/app/12.1.0.2/grid/cfgtoollogs/opatchauto/core/opatch/opatch2016-09-19_12-47-55PM_1.log

b) Check Postpatch operation log file /u01/app/12.1.0.2/grid/cfgtoollogs/crsconfig/crspatch_racnode2_2016-09-19_12-50-36AM.log

$ tail -120 /u01/app/12.1.0.2/grid/cfgtoollogs/crsconfig/crspatch_racnode2_2016-09-19_12-50-36AM.log
...

....


2016-09-19 12:52:30: Executing cmd: /u01/app/12.1.0.2/grid/bin/srvctl config mgmtdb
2016-09-19 12:52:31: Command output:
> Database unique name: _mgmtdb
> Database name:
> Oracle home: <CRS home>
> Oracle user: grid
> Spfile: +OCR_VOTE/_MGMTDB/PARAMETERFILE/spfile.268.922884353
> Password file:
> Domain:
> Start options: open
> Stop options: immediate
> Database role: PRIMARY
> Management policy: AUTOMATIC
> Type: Management
> PDB name: racnode-cluster
> PDB service: racnode-cluster
> Cluster name: racnode-cluster
> Database instance: -MGMTDB
>End Command output
2016-09-19 12:52:31: isMgmtdbConfigured: 1
2016-09-19 12:52:31: setting ORAASM_UPGRADE to 1
2016-09-19 12:52:31: Executing cmd: /u01/app/12.1.0.2/grid/bin/crsctl query crs softwarepatch racnode2
2016-09-19 12:52:31: Command output:
> Oracle Clusterware patch level on node racnode2 is [3696455212].
>End Command output
2016-09-19 12:52:31: Oracle Clusterware patch level on node 'racnode2' is [3696455212]
2016-09-19 12:52:31: setting ORAASM_UPGRADE to 1
2016-09-19 12:52:31: Executing cmd: /u01/app/12.1.0.2/grid/bin/crsctl query crs softwarepatch racnode1
2016-09-19 12:52:31: Command output:
> Oracle Clusterware patch level on node racnode1 is [3696455212].
>End Command output
2016-09-19 12:52:31: Oracle Clusterware patch level on node 'racnode1' is [3696455212]
2016-09-19 12:52:31: The local node has the same software patch level [3696455212] as remote node 'racnode1'
2016-09-19 12:52:31: Executing cmd: /u01/app/12.1.0.2/grid/bin/crsctl query crs releasepatch
2016-09-19 12:52:31: Command output:
> Oracle Clusterware release patch level is [3696455212] and the complete list of patches [19769480 20299023 20831110 21359755 21436941 21948354 22291127 23054246 23054327 23054341 ] have been applied on the local node.
>End Command output
2016-09-19 12:52:31: Oracle Clusterware release patch level is [3696455212]
2016-09-19 12:52:31: setting ORAASM_UPGRADE to 1
2016-09-19 12:52:31: Executing cmd: /u01/app/12.1.0.2/grid/bin/crsctl query crs activeversion -f
2016-09-19 12:52:31: Command output:
> Oracle Clusterware active version on the cluster is [12.1.0.2.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [3696455212].
>End Command output
2016-09-19 12:52:31: Oracle Clusterware active patch level is [3696455212]
2016-09-19 12:52:31: The Clusterware active patch level [3696455212] has been updated to [3696455212].
2016-09-19 12:52:31: Postpatch: isLastNode is 1
2016-09-19 12:52:31: Last node: enable Mgmt DB globally
2016-09-19 12:52:31: Invoking "/u01/app/12.1.0.2/grid/bin/srvctl enable mgmtdb"
2016-09-19 12:52:31: trace file=/u01/app/12.1.0.2/grid/cfgtoollogs/crsconfig/srvmcfg1.log
2016-09-19 12:52:31: Running as user grid: /u01/app/12.1.0.2/grid/bin/srvctl enable mgmtdb
2016-09-19 12:52:31: Invoking "/u01/app/12.1.0.2/grid/bin/srvctl enable mgmtdb" as user "grid"
2016-09-19 12:52:31: Executing /bin/su grid -c "/u01/app/12.1.0.2/grid/bin/srvctl enable mgmtdb"
2016-09-19 12:52:31: Executing cmd: /bin/su grid -c "/u01/app/12.1.0.2/grid/bin/srvctl enable mgmtdb"
2016-09-19 12:52:32: Modifying 10.2 resources
2016-09-19 12:52:32: Invoking "/u01/app/12.1.0.2/grid/bin/srvctl upgrade model -pretb"
2016-09-19 12:52:32: trace file=/u01/app/12.1.0.2/grid/cfgtoollogs/crsconfig/srvmcfg2.log
2016-09-19 12:52:32: Executing /u01/app/12.1.0.2/grid/bin/srvctl upgrade model -pretb
2016-09-19 12:52:32: Executing cmd: /u01/app/12.1.0.2/grid/bin/srvctl upgrade model -pretb
2016-09-19 12:52:32: 'srvctl upgrade model -pretb' ... success
2016-09-19 12:52:32: Executing cmd: /u01/app/12.1.0.2/grid/bin/srvctl status mgmtdb -S 1
2016-09-19 12:52:32: Command output:
> #@=result[0]: dbunique_name={_mgmtdb} inst_name={-MGMTDB} node_name={racnode2} up={true} state_details={Open} internal_state={STABLE}
>End Command output
2016-09-19 12:52:32: Mgmtdb is running on node: racnode2; local node: racnode2
2016-09-19 12:52:32: Mgmtdb is running on the local node
2016-09-19 12:52:32: Starting to patch Mgmt DB ...
2016-09-19 12:52:32: Invoking "/u01/app/12.1.0.2/grid/sqlpatch/sqlpatch -db -MGMTDB"
2016-09-19 12:52:32: Running as user grid: /u01/app/12.1.0.2/grid/sqlpatch/sqlpatch -db -MGMTDB
2016-09-19 12:52:32: Invoking "/u01/app/12.1.0.2/grid/sqlpatch/sqlpatch -db -MGMTDB" as user "grid"
2016-09-19 12:52:32: Executing /bin/su grid -c "/u01/app/12.1.0.2/grid/sqlpatch/sqlpatch -db -MGMTDB"
2016-09-19 12:52:32: Executing cmd: /bin/su grid -c "/u01/app/12.1.0.2/grid/sqlpatch/sqlpatch -db -MGMTDB"
2016-09-19 12:53:39: Command output:
> SQL Patching tool version 12.1.0.2.0 on Mon Sep 19 12:52:32 2016
> Copyright (c) 2016, Oracle. All rights reserved.
>
> Connecting to database...OK
> Note: Datapatch will only apply or rollback SQL fixes for PDBs
> that are in an open state, no patches will be applied to closed PDBs.
> Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
> (Doc ID 1585822.1)
> Determining current state...done
> Adding patches to installation queue and performing prereq checks...done
> Installation queue:
> For the following PDBs: CDB$ROOT PDB$SEED RACNODE-CLUSTER
> Nothing to roll back
> The following patches will be applied:
> 23054246 (Database Patch Set Update : 12.1.0.2.160719 (23054246))
>
> Installing patches...
> Patch installation complete. Total patches installed: 3
>
> Validating logfiles...done
> SQL Patching tool complete on Mon Sep 19 12:53:39 2016
>End Command output
2016-09-19 12:53:39: Successfully patched Mgmt DB
2016-09-19 12:53:39: Invoking "/u01/app/12.1.0.2/grid/bin/cluutil -ckpt -oraclebase /u01/app/grid -writeckpt -name ROOTCRS_POSTPATCH -state SUCCESS"
2016-09-19 12:53:39: trace file=/u01/app/grid/crsdata/racnode2/crsconfig/cluutil7.log
2016-09-19 12:53:39: Running as user grid: /u01/app/12.1.0.2/grid/bin/cluutil -ckpt -oraclebase /u01/app/grid -writeckpt -name ROOTCRS_POSTPATCH -state SUCCESS
2016-09-19 12:53:39: s_run_as_user2: Running /bin/su grid -c ' echo CLSRSC_START; /u01/app/12.1.0.2/grid/bin/cluutil -ckpt -oraclebase /u01/app/grid -writeckpt -name ROOTCRS_POSTPATCH -state SUCCESS '
2016-09-19 12:53:39: Removing file /tmp/filei2kicY
2016-09-19 12:53:39: Successfully removed file: /tmp/filei2kicY
2016-09-19 12:53:39: pipe exit code: 0
2016-09-19 12:53:39: /bin/su successfully executed

2016-09-19 12:53:39: Succeeded in writing the checkpoint:'ROOTCRS_POSTPATCH' with status:SUCCESS
[+ASM2] grid@racnode2:~$

c) Querying PDB shows MGMTDB has been patched as well.

-- check CDB$ROOT

SQL> set pagesize 120
SQL> set linesize 180
SQL> select * from dba_registry_sqlpatch;

PATCH_ID PATCH_UID VERSION FLAGS ACTION STATUS ACTION_TIME
---------- ---------- -------------------- ---------- --------------- --------------- ---------------------------------------------------------------------------
DESCRIPTION BUNDLE_SERIES BUNDLE_ID
---------------------------------------------------------------------------------------------------- ------------------------------ ----------
BUNDLE_DATA
--------------------------------------------------------------------------------
LOGFILE
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

23054246 20213895 12.1.0.2 NB APPLY SUCCESS 19-SEP-16 12.53.38.034518 PM
Database Patch Set Update : 12.1.0.2.160719 (23054246) PSU 160719
<bundledata version="12.1.0.2.1" series="Patch Set Update">
 <bundle id="1" des
/u01/app/grid/cfgtoollogs/sqlpatch/23054246/20213895/23054246_apply__MGMTDB_CDBROOT_2016Sep19_12_53_18.log



-- check PDB RACNODE-CLUSTER

SQL> alter session set container=racnode-cluster;

SQL> select * from dba_registry_sqlpatch;

PATCH_ID PATCH_UID VERSION FLAGS ACTION STATUS ACTION_TIME
---------- ---------- -------------------- ---------- --------------- --------------- ---------------------------------------------------------------------------
DESCRIPTION BUNDLE_SERIES BUNDLE_ID
---------------------------------------------------------------------------------------------------- ------------------------------ ----------
BUNDLE_DATA
--------------------------------------------------------------------------------
LOGFILE
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
23054246 20213895 12.1.0.2 NB APPLY SUCCESS 19-SEP-16 12.53.38.457190 PM
Database Patch Set Update : 12.1.0.2.160719 (23054246) PSU 160719
<bundledata version="12.1.0.2.1" series="Patch Set Update">
 <bundle id="1" des
/u01/app/grid/cfgtoollogs/sqlpatch/23054246/20213895/23054246_apply__MGMTDB_RACNODE-CLUSTER_2016Sep19_12_53_32.log