We changed WebLogic user password for enforcing security principal, then WebLogic domain targets show down on OEM. Hence we need update the monitoring credentials on OEM for WebLogic domain.
1) Verify the correct Admin Server hostname and password are correct:
EM Console > Middleware Targets > click the WebLogic Domain > WebLogic Domain dropdown menu > choose WebLogic Server Administration Console
Log in with this username/password
and note the Administration Server hostname as indicated in
- the url at the top of the page > https://<AdminServerFQDN>:<port>/console/...
and/or
- under Servers > AdminServer(admin) > Machine column shows Administration Server hostname
This step is necessary only if this WebLogic Domain is monitored by multiple agents: 2) Put the WebLogic Domain target under blackout: Monitoring > Blackouts > Create… > > choose “Blackout” (Create) Choose “Enable Full blackout…” Add the WebLogic Domain target (select)
3) Modify the password for the WebLogic Server Administr Server using the hostname and password verified in step 1
There are 2 possible ways to navigate to the Monitoring Credentials in EM:
Setup (top right) > Security > Monitoring credentials > choose Oracle WebLogic Domain and click “Manage Monitoring Credentials” > choose the Domain and “Set Credentials” The is Domain and Node Manager credentials are set in separate places here Test and Save configuration
or
From the WebLogic Domain’s Home Page > under WebLogic Domain dropdown men > choose Target setup > Monitoring Credentials Both the Domain and Node Manager credentials are set on the same page Test and Save configuration
This step is necessary only if step 2 was implemented: 4) If Blackout has not already expired,
Monitoring > Blackouts > View by Blackout Name > choose blackout and “Stop”
5) Refresh WebLogic domain. From WebLogic Domain home page > WebLogic Domain dropdown > choose “Refresh WebLogic Domain”
EM 13c: Updating Monitoring Credentials For WebLogic Domain Target shows “Password at Management Agent does not match error” (Doc ID 2610563.1)
Note: Static “_DGMGRL” entries are no longer needed as of Oracle Database 12.1.0.2 and later release in Oracle Data Guard Broker configurations that are managed by Oracle Restart, RAC On Node or RAC as the Broker will use the clusterware to restart an instance. ( Oracle Data Guard Broker and Static Service Registration (Doc ID 1387859.1)
DGMGRL> connect sys@testdb Password: Connected to "TESTDB" Connected as SYSDBA. DGMGRL> show configuration;
DGMGRL> switchover to stestdb; Performing switchover NOW, please wait... New primary database "stestdb" is opening... Oracle Clusterware is restarting database "testdb" ... Connected to "TESTDB" Switchover succeeded, new primary is "stestdb"
-- After switchover, run 'srvctl config database -d", the database role has been changed by Data Guard Broker automatically.
DGMGRL> validate static connect identifier for all Oracle Clusterware on database "testdb" is available for database restart.
Oracle Clusterware on database "stestdb" is available for database restart.
validate network configuration for all
DGMGRL> validate network configuration for all Oracle Clusterware on database "testdb" is available for database restart.
Oracle Clusterware on database "stestdb" is available for database restart.
DGMGRL> validate network configuration for all Connecting to instance "TESTDB2" on database "testdb" ... Connected to "TESTDB" Checking connectivity from instance "TESTDB2" on database "testdb" to instance "STESTDB2" on database "stestdb"... Succeeded. Checking connectivity from instance "TESTDB2" on database "testdb" to instance "STESTDB" on database "stestdb"... Succeeded. Connecting to instance "TESTDB1" on database "testdb" ... Connected to "TESTDB" Checking connectivity from instance "TESTDB1" on database "testdb" to instance "STESTDB2" on database "stestdb"... Succeeded. Checking connectivity from instance "TESTDB1" on database "testdb" to instance "STESTDB" on database "stestdb"... Succeeded. Connecting to instance "STESTDB2" on database "stestdb" ... Connected to "STESTDB" Checking connectivity from instance "STESTDB2" on database "stestdb" to instance "TESTDB2" on database "testdb"... Succeeded. Checking connectivity from instance "STESTDB2" on database "stestdb" to instance "TESTDB1" on database "testdb"... Succeeded. Connecting to instance "STESTDB" on database "stestdb" ... Connected to "STESTDB" Checking connectivity from instance "STESTDB" on database "stestdb" to instance "TESTDB2" on database "testdb"... Succeeded. Checking connectivity from instance "STESTDB" on database "stestdb" to instance "TESTDB1" on database "testdb"... Succeeded.
Oracle Clusterware on database "testdb" is available for database restart.
Oracle Clusterware on database "stestdb" is available for database restart.
This post demonstrates how to duplicate an Oracle database from a standby database instead of from a primary database normally.
The environment list:
Primaey DB
Standby DB
Dulicate DB
TESTDB
STBYDB
DUDB
Assume that all tnsnames are already configured. Password file, spfile are also already configured as the following:
Copy the password file from source database for the target database, or create a password file with the same password as source db.
Add “(UR = A)” into source and target tnsnames.
Add an entry into listener.ora for auxiliary instance static registration. You can create a temporary listener with a spare port without changing anything of the current listener.
CLUSTER_DATABASe=false in spfile for auxiliary instance.
Please note when you create the password file, you have to add the database into cluster OCR first, otherwise the following errors will occur:
ASMCMD> pwcreate --dbuniquename DUPDB '+DATA2/DUPDB/password/pwddupdb' Enter password: *********** OPW-00021: Failed to retrieve DB password file location from the CRS resource ASMCMD-9454: could not create new password file
$ srvctl add database -d DUPDB -oraclehome /u01/app/oracle/product/19.0.0/dbhome_1 -dbtype RAC -domain world -spfile +DATA2/DUPDB/PARAMETERFILE/spfileDUPDB.ora -role PRIMARY -startoption OPEN -stopoption IMMEDIATE
ASMCMD> pwcreate --dbuniquename DUPDB '+DATA2/DUPDB/password/pwddupdb' Enter password: ***********
-- The following error will occur if the database is not added into CRS before creating the password file:
ASMCMD> pwcreate --dbuniquename DUPDB '+DATA2/DUPDB/password/pwddupdb' Enter password: *********** OPW-00021: Failed to retrieve DB password file location from the CRS resource ASMCMD-9454: could not create new password file
Start Up Duplicate Database in Nomount
SQL> startup nomount pfile='/tmp/initDUPDB.ora' ;
SQL> create spfile='+DATA2/DUPDB/PARAMETERFILE/spfileDUPDB.ora' from pfile='/tmp/initDUPDB1.ora' ;
RMAN> duplicate target database to DUPDB from active database ; ... ... . contents of Memory Script: { Alter clone database open resetlogs; } executing Memory Script
database opened Finished Duplicate Db at 03-APR-25
RMAN>
Duplicate database when standby in Recovery( Mounted and Applying ) Mode
RMAN> duplicate target database to DUPDB from active database ; ... ... . contents of Memory Script: { Alter clone database open resetlogs; } executing Memory Script
database opened Finished Duplicate Db at 03-APR-25
RMAN>
Duplicate database when standby in Non-Recovery ( Mounted and Non-applying ) Mode
DGMGRL> show database verbose "stbydb" Database - stbydb
Role: PHYSICAL STANDBY Intended State: APPLY-ON Transport Lag: 0 seconds (computed 0 seconds ago) Apply Lag: 12 hours 39 minutes 52 seconds (computed 0 seconds ago) Average Apply Rate: 46.00 KByte/s Active Apply Rate: 1.17 MByte/s Maximum Apply Rate: 12.89 MByte/s Real Time Query: OFF Instance(s): STBYDB1 (apply instance) STBYDB2
Database Error(s): ORA-16766: Redo Apply is stopped
Database Warning(s): ORA-16853: apply lag has exceeded specified threshold
RMAN> duplicate target database to DUPDB from active database ; ... ... . contents of Memory Script: { Alter clone database open resetlogs; } executing Memory Script
database opened Finished Duplicate Db at 03-APR-25
RMAN>
From the above tests, we can see the duplicate process will
1) switch archivelogs on PRIMARY 2) copy archived logs onto new duplicated database 3) apply the archived logs onto new database finally before OPEN new database.
so, no matter standby database is either
1) OPEN READ ONLY MODE 2) MOUNTED and RECOVERY MODE 3) MOUNTED and NON RECOVERY MODE
DUPLICATE from standby database all works well.
The common errors and solutions
RMAN-04006: error from auxiliary database: ORA-12528: TNS:listener: all appropriate instances are blocking new connections
This error is generated because STATIC REGISTRATION on listener was not done. To fix this, we need to add an entry of SID_LIST_LISTENER into listener.ora.
RMAN-04006: error from auxiliary database: ORA-12537: TNS:connection closed
This error was generated by missing the (UR = A) from tnsnames.
RMAN-04006: error froRMAN-06136: Oracle error from auxiliary database: ORA-17629: Cannot connect to the remote database server ORA-17627: ORA-01017: invalid username/password; logon denied ORA-17629: Cannot connect to the remote database serverm auxiliary database: ORA-01017: invalid username/password; logon denied
RMAN-00571: =========================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: =========================================================== RMAN-03002: failure of Duplicate Db command at 04/03/2025 15:29:36 RMAN-05501: aborting duplication of target database RMAN-03015: error occurred in stored script Memory Script RMAN-06136: Oracle error from auxiliary database: ORA-17629: Cannot connect to the remote database server ORA-17627: ORA-01017: invalid username/password; logon denied ORA-17629: Cannot connect to the remote database server
RMAN-00571: =========================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: =========================================================== RMAN-03002: failure of Duplicate Db command at 04/03/2025 16:28:42 RMAN-05501: aborting duplication of target database RMAN-05614: Passwords for target and auxiliary connections must be the same when using active duplicate
This error is because password file was not copied to duplicate database, or the password is different.
ORA-12720: operation requires database is in EXCLUSIVE mode
RMAN-00571: =========================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: =========================================================== RMAN-03002: failure of Duplicate Db command at 04/03/2025 16:50:17 RMAN-05501: aborting duplication of target database RMAN-06136: Oracle error from auxiliary database: ORA-01503: CREATE CONTROLFILE failed ORA-12720: operation requires database is in EXCLUSIVE mode
Run the following commnad, we can see ‘-pwfile ‘+DATA1/TESTDB/PASSWORD/pwdTESTDB’ has been added into configuration automatically when we create password file.