RMAN-06004: ORACLE error from recovery catalog database: ORA-01422: exact fetch returns more than requested number of rows

It is a good practice to upgrade RMAN CATLOG to the latest version in your environment

SYMPTOMS

Target Database : 12.2.0.1
RMAN Catalog  :  12.1.0.2

Trying to “upgrade catalog” from 12.2.0.1 RMAN and 12.1.0.2  RMAN catalog, then got following errors:

connected to target database: RACTEST (DBID=2073981615)
connected to recovery catalog database
PL/SQL package RMAN.DBMS_RCVCAT version 12.01.00.02. in RCVCAT database is too old

RMAN> upgrade catalog;

recovery catalog owner is RMAN
enter UPGRADE CATALOG command again to confirm catalog upgrade

RMAN> upgrade catalog;
error creating init_nrsp_pdb_key
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-06004: ORACLE error from recovery catalog database: ORA-01422: exact fetch returns more than requested number of rows

CAUSES

Bug 25852885 UPGRADE CATALOG WITH 12.2 RMAN ON 12.1 DATABASE FAILS W/ ORA-1422

SOLUTION

Backup then edit the recover.bsq file under target 12.2.0.1 ORACLE_HOME/rdbms/admin to change the following lines:

define init_nrsp_pdb_key
<<<
declare
   cursor nrsp_c is
      select distinct nrsp.rowid rid, dbinc.db_key db_key
        from nrsp, dbinc
       where nrsp.pdb_key is null
         and nrsp.dbinc_key = dbinc.dbinc_key
       order by dbinc.db_key;
   prev_db_key   number := 0;
   local_pdb_key number;
begin
   for r in nrsp_c loop
     if (r.db_key <> prev_db_key) then
        prev_db_key := r.db_key;
        select pdb_key into local_pdb_key
          from pdb
         where pdb.con_id in (1, 0);
     end if;

 TO:

define init_nrsp_pdb_key
<<<
declare
   cursor nrsp_c is
      select distinct nrsp.rowid rid, dbinc.db_key db_key
        from nrsp, dbinc
       where nrsp.pdb_key is null
         and nrsp.dbinc_key = dbinc.dbinc_key
       order by dbinc.db_key;
   prev_db_key   number := 0;
   local_pdb_key number;
begin
   for r in nrsp_c loop
     if (r.db_key <> prev_db_key) then
        prev_db_key := r.db_key;
        select pdb_key into local_pdb_key
          from pdb
         where pdb.con_id in (1, 0)
           and pdb.db_key = r.db_key;
     end if;

AND CHANGE :

define init_grsp_pdb_key
<<<
declare
   cursor grsp_c is
      select distinct grsp.rowid rid, dbinc.db_key db_key
        from grsp, dbinc
       where grsp.pdb_key is null
         and grsp.dbinc_key = dbinc.dbinc_key
       order by dbinc.db_key;
   prev_db_key   number := 0;
   local_pdb_key number;
begin
   for r in grsp_c loop
     if (r.db_key <> prev_db_key) then
        prev_db_key := r.db_key;
        select pdb_key into local_pdb_key
          from pdb
         where pdb.con_id in (1, 0);
     end if;

 TO:

define init_grsp_pdb_key
<<<
declare
   cursor grsp_c is
      select distinct grsp.rowid rid, dbinc.db_key db_key
        from grsp, dbinc
       where grsp.pdb_key is null
         and grsp.dbinc_key = dbinc.dbinc_key
       order by dbinc.db_key;
   prev_db_key   number := 0;
   local_pdb_key number;
begin
   for r in grsp_c loop
     if (r.db_key <> prev_db_key) then
        prev_db_key := r.db_key;
        select pdb_key into local_pdb_key
          from pdb
         where pdb.con_id in (1, 0)
           and pdb.db_key = r.db_key;
     end if;

Then re-run “upgrade catalog” twice:

RMAN> upgrade catalog;
recovery catalog owner is RMAN
enter UPGRADE CATALOG command again to confirm catalog upgrade

RMAN> upgrade catalog;
PL/SQL package RMAN.DBMS_BA version 12.02.00.01 in RCVCAT database is too old
recovery catalog upgraded to version 12.02.00.01
DBMS_RCVMAN package upgraded to version 12.02.00.01
DBMS_RCVCAT package upgraded to version 12.02.00.01.

RMAN> register database;
database registered in recovery catalog
starting full resync of recovery catalog
full resync complete

Please note: As of today ( 25/09/2017), Oracle support released patch 25852885.

  1. It works for 12.2.0.1.0 only, because updated “recover.bsq” is included in patch ( p25852885_122010_Linux-x86-64.zip) .
  2. It does NOT work for 12.2.0.1.170814, because updated “recover.bsq” is not included in  patch ( p25852885_12201170718DBRU_Linux-x86-64.zip).

How to Copy ASM Password File From Primary to Standby

It is a prerequisite for both primary and secondary databases have same copy of password file, and stored in ASM diskgroup preferred

From 12.1.0.2 on, since primary database password file is stored in ASM, so it needs to be manually copied onto standby database ASM for dataguard configuration. Otherwise the below errors will occur:

*** 2017-09-06 14:34:29.216
OCISessionBegin failed. Error -1
.. Detailed OCI error val is 1017 and errmsg is 'ORA-01017: invalid username/password; logon denied
'
ORA-01017: invalid username/password; logon denied
OCI_DBVER attribute value retrieval failed error=1017

*** 2017-09-06 14:34:30.323
OCISessionBegin failed. Error -1
.. Detailed OCI error val is 1017 and errmsg is 'ORA-01017: invalid username/password; logon denied
'
*** 2017-09-06 14:34:30.323737 4929 krsh.c
Error 1017 received logging on to the standby

Subscribe to get access

Read more of this content when you subscribe today.

How Long a Session Has been Idle or Inactive In Oracle Database ?

SQL to find how long a session has been ACTIVE or IDLE, very handy.

SYMPTOMS

There are up to one thousand sessions connected to Oracle Database from a Weblogic connection pool. User wanted to know how long those sessions have been idle ( INACTIVE), so the Weblogic connection pool can be reviewed and reconfigured properly.

SOLUTION

The below query can be run to meet this requirement:

SQL>select USERNAME,
           MACHINE,
           STATUS,
           LOGON_TIME,
           LAST_CALL_ET INACTIVE_SECONDS 
    from  gv$session 
    order by LAST_CALL_ET desc; 

USERNAME   MACHINE    STATUS    LOGON_TIME         INACTIVE_SECONDS
---------- ---------- -------   -----------------  ----------------
USERAPP    machine1  INACTIVE   20170917-17:04:17             3523
USERAPP    machine2  INACTIVE   20170918-02:57:03             2068
USERAPP    machine3  INACTIVE   20170917-17:52:09             1141
...
..
.
USERAPP    machine1 INACTIVE   20170917-17:17:01               176
USERAPP    machine1 I NACTIVE   20170917-17:17:01              176

REFERENCES

According to Oracle doc :

LAST_CALL_ETNUMBERIf the session STATUS is currently ACTIVE, then the value represents the elapsed time in seconds since the session has become active.If the session STATUS is currently INACTIVE, then the value represents the elapsed time in seconds since the session has become inactive.

The below query can also get the results as per Oracle Doc ID 365693.1.

SQL>select sid, seconds_in_wait 
      from v$session_wait
     where state='WAITING' 
       and event='SQL*Net message from client'
 order by seconds_in_wait desc;

Data Guard Not Applying Logs After RAC Standby Database Restarted

It looks like a bug. Property “PreferredApplyInstance” specifies the standby instance to apply logs

11.2.0.4 RAC standby database is not applying archivelogs after instances are all restarted.  It is configured in Data Guard management.

After investigation, there is nothing special, but only “PreferredApplyInstance” is configured for RAC node1 ‘STBTEST1’.

Original :

PreferredApplyInstance ='STBTEST1'

Modify “PreferredApplyInstance” to empty:

PreferredApplyInstance=''

Restarted RAC standby database instances , all archivelogs are applied to database from the node which is chosen randomly by Data Guard.

ORA-03135: connection lost contact to Standby Database

Newly built 12.1.0.2 DataGuard got errors.

Primary Database alert.log:

Mon Sep 11 11:35:59 2017
TT01: Attempting destination LOG_ARCHIVE_DEST_2 network reconnect (3135)
TT01: Destination LOG_ARCHIVE_DEST_2 network reconnect abandoned
Mon Sep 11 11:35:59 2017
Errors in file /u01/app/oracle/diag/rdbms/ractest/RACTEST1/trace/RACTEST1_tt01_30967.trc:
ORA-03135: connection lost contact
Error 3135 for archive log file 42 to 'STBTEST'
Mon Sep 11 11:35:59 2017
Errors in file /u01/app/oracle/diag/rdbms/ractest/RACTEST1/trace/RACTEST1_tt01_30967.trc:
ORA-03135: connection lost contact
LNS: Failed to archive log 42 thread 1 sequence 852 (3135)

Standby Database Alert.log

Fatal NI connect error 12170.

VERSION INFORMATION:
 TNS for Linux: Version 12.1.0.2.0 - Production
 Oracle Bequeath NT Protocol Adapter for Linux: Version 12.1.0.2.0 - Production
 TCP/IP NT Protocol Adapter for Linux: Version 12.1.0.2.0 - Production
 Time: 06-SEP-2017 16:50:33
 Tracing not turned on.
 Tns error struct:
 ns main err code: 12535

TNS-12535: TNS:operation timed out
 ns secondary err code: 12608
 nt main err code: 0
 nt secondary err code: 0
 nt OS err code: 0
 Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.xx)(PORT=63637))
Wed Sep 06 16:50:34 2017
RFS[37]: Possible network disconnect with primary database

Subscribe to get access

Read more of this content when you subscribe today.