DUPLICATE FROM ACTIVE DATABASE Fails With RMAN-03009 ORA-17628 ORA-19505

While creating a new physical standby database by using RMAN duplicate command, the following errors occur:

RMAN> run {
allocate channel prim1 type disk;
allocate channel prim2 type disk;
allocate auxiliary channel standby1 type disk;
DUPLICATE TARGET DATABASE
FOR STANDBY
FROM ACTIVE DATABASE
...
..
.
RMAN-03009: failure of backup command on prim2 channel 
ORA-17628: Oracle error 19505 returned by remote Oracle server
ORA-19505: failed to identify file ""

SOLUTION

  1. Check and remove the old standby database data files.
  2. Make sure directories for standby database data files created and accessible.

ORA-16751: failed to switchover to physical standby database and Switchover Ends with Two Physical Standby Databases

Trying to switch over database from primary database to physical standby database,  it ran into the issue where both database became physical standby.

DGMGRL> switchover to "STDBYDB"
Performing switchover NOW, please wait...
Operation requires a connection to instance "STDBYDB" on database "STDBYDB"

Connecting to instance "STDBYDB"...
Connected.

Error: ORA-16751: failed to switchover to physical standby database

Failed.

Unable to switchover, primary database is still "PRIMDB"

DGMGRL> 

--
-- Check database role
--

NAME       OPEN_MODE            DATABASE_ROLE    
--------- -------------------- ---------------- 
PRIMDB    CLOSED BY SWITCHOVER PHYSICAL STANDBY  
STDBYDB   MOUNTED              PHYSICAL STANDBY

Data Guard Broker log:

07/19/2019 08:07:55
SQL Execution error=604, 
sql=[ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WAIT WITH SESSION SHUTDOWN]. 
See error stack below.
ORA-00604: error occurred at recursive SQL level 1
ORA-00344: unable to re-create online log '/u05/oralog/PRIMDB/redo1a.log'
ORA-27040: file create error, unable to create file
Linux-x86_64 Error: 2: No such file or directory
Additional information:
switchover to primary command failed
Database Resource SetState Error (16751)
Command SWITCHOVER TO STDBYDB completed with error ORA-16751

DataGuard cannot create online redo logs on standby database side. The reason is the following two parameters are not set up correctly.

The parameters on standby database side should be like those:

DB_FILE_NAME_CONVERT  = 'PRIMDB','STDBYDB'
LOG_FILE_NAME_CONVERT = 'PRIMDB','STDBYDB'

Solution

Logon to standby database server.

SQL>select database_role from v$database;

DATABASE_ROLE
----------------
PHYSICAL STANDBY

SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.

SQL> startup mount
ORACLE instance started.
...
..
.
Database mounted.


SQL> alter database recover managed standby database cancel;

Database altered.

SQL> alter database recover managed standby database finish;

Database altered.

SQL> ALTER database commit to switchover to primary with session shutdown;

Database altered.

SQL> alter database open;

Database altered.

SQL>  select database_role from v$database;

DATABASE_ROLE
----------------
PRIMARY

Finally rebuild the Data Guard Broker Configuration.

Error 1033 received logging on to the standby

PRIMARY:

alert.log:

Error 1033 received logging on to the standby
Wed Jul 10 16:15:08 2019
Error 1033 received logging on to the standby
Wed Jul 10 16:16:11 2019
Error 1033 received logging on to the standby
Wed Jul 10 16:17:13 2019
Error 1033 received logging on to the standby

dr.log:

07/10/2019 16:19:57
Failed to connect to remote database TESTSTY. Error is ORA-01033
Failed to send message to site TESTSTY. Error code is ORA-01033.
Data Guard Broker Status Summary:
Type Name Severity Status
Configuration TESTDB Warning ORA-16607
Primary Database TESTDB Error ORA-16778
Physical Standby Database TESTSTY Error ORA-01033

Subscribe to get access

Read more of this content when you subscribe today.

ORA-16665: timeout waiting for the result from a database

$ oerr ora 16665
16665, 0000, "timeout waiting for the result from a database"
// *Cause: The Oracle Data Guard broker was forced to time out a network
// connection to a remote database because:
// - The network call to the remote database did not complete in
// a timely manner.
// - The remote database was unable to execute the command due to
// an instance failure.
// *Action: Check Data Guard broker log files for the details of the failure.
// If the network call did not complete in a timely manner, increase
// the CommunicationTimeout configuration property value and reissue
// the command.

Subscribe to get access

Read more of this content when you subscribe today.

Recover Standby Database By Using RMAN Incremental Backup

By using RMAN incremental backups to synchronize a physical standby database, it is fast and easy without installing datafiles or history archive logs required.

This post demonstrates how to recover standby database with a big gap behind primary database in 11g.

Stop the managed recovery process (MRP)

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;

On standby database, get the lowest SCN

SQL> SELECT CURRENT_SCN FROM V$DATABASE;

CURRENT_SCN
-----------
7.8880E+12

SQL> set numwidth 16
SQL> /

CURRENT_SCN
----------------
7888036454134

SQL> select min(fhscn) from x$kcvfh;

MIN(FHSCN)
----------------
7883734410408

SQL> select min(f.fhscn) from x$kcvfh f, v$datafile d
where f.hxfil =d.file#
and d.enabled != 'READ ONLY' ; 

MIN(F.FHSCN)
----------------
7883734410408

SQL> select CHECKPOINT_CHANGE# from v$datafile_header;

MIN(CHECKPOINT_CHANGE#)
-----------------------
7883734410408

On primary database, RMAN backup from SCN of previous step

Take a SCN backup, and copy backup onto standby server.

RMAN> BACKUP INCREMENTAL FROM SCN 7883734410408 DATABASE 
      FORMAT '/tmp/testdb_styby_%U' tag 'INCFORSTANDBY';

Prepare Standby ControlFile and New Datafiles

a) Create a standby control file from primary, and then copy onto standby database server. Replace the standby control file by using this newly created one.

-- From primary
--
SQL>ALTER DATABASE CREATE STANDBY CONTROLFILE AS '/tmp/TESTDBSTY.ctl';

-- From Standby
--
RMAN> SHUTDOWN IMMEDIATE ;
RMAN> STARTUP NOMOUNT; 
RMAN> RESTORE STANDBY CONTROLFILE FROM '/tmp/TESTDBSTY.ctl';

b) For new datafiles which are in Primary but not in standby database, copy them from primary to standby database server. Then catalog current all standby database datafiles, since the control file has been replaced by primary one.

RMAN> CATALOG START WITH '+DG1/tetsdbsty/datafile/'; 

c) Switch database to copy from RMAN.

RMAN> SWITCH DATABASE TO COPY;

Recover Standby Database

Catalog incremental backup, then recover standby database by using RMAN incremental backup of primary database.

RMAN> CATALOG START WITH '/tmp/testdb_styby';

RMAN> RECOVER DATABASE NOREDO;

Start the MRP process of standby database.

SQL>ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE
     DISCONNECT FROM SESSION;

Check DataGuard Health

There is no gap between primary database and standby database now.

DGMGRL> show configuration;
Configuration - dg_oemrep
Protection Mode: MaxPerformance
Databases:
oemrep -    Primary database
stboemrep - Physical standby database

Fast-Start Failover: Disabled

Configuration Status:
SUCCESS
DGMGRL> show database "stboemrep";

Database - stboemrep

Enterprise Manager Name: OEMREP

Role:                    PHYSICAL STANDBY
Intended State:          APPLY-ON
Transport Lag:           0 seconds (computed 1 second ago)
Apply Lag:               0 seconds (computed 1 second ago)
Apply Rate:              4.93 MByte/s
Real Time Query:         OFF
Instance(s):
STBOEMREP

Database Status:
SUCCESS

DGMGRL>