Install Oracle RAC Software and Create Oracle RAC Database In Silent Mode

In this post, I am going to install Oracle 18c RAC software, and create a 18c RAC database in silent mode.

This post will also apply to how to install Oracle 19c RAC software, and create a 19c RAC database in silent mode.

Subscribe to get access

Read more of this content when you subscribe today.

“Fatal NI Connect Error 12516″ and ”TNS-12564: TNS:connection refused” In Clusterware Alert.log when ologgerd Connects to GIMR

ISSUES

There are repeating following errors in 19c CRS alert.log by ologgered trying to connect to GIMR.

Fatal NI connect error 12516, connecting to:
(DESCRIPTION=(CONNECT_TIMEOUT=60)(ADDRESS=(PROTOCOL=tcp)(HOST=racnode-cluster-scan.virtuallab)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=GIMR_DSCREP_10)(CID=(PROGRAM=ologgerd)(HOST=racnode1.virtuallab)(USER=root)))(SECURITY=(ENCRYPTION_CLIENT=requested)))
VERSION INFORMATION:
TNS for Linux: Version 19.0.0.0.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 19.0.0.0.0 - Production

Version 19.8.0.0.0
Time: 08-AUG-2020 18:15:01
Tracing not turned on.
Tns error struct:
ns main err code: 12564

TNS-12564: TNS:connection refused
ns secondary err code: 12560
nt main err code: 524

TNS-00524: Current operation is still in progress
nt secondary err code: 115
nt OS err code: 0

INVESTIGATION

Check mgmtlsnr or SCAN listeners, and found -MGMTDB PDB service ” gimr_dscrep_10″ is not registered properly.

[grid@racnode1 trace]$ lsnrctl services listener_scan2
...
..
.
Service "gimr_dscrep_10" has 1 instance(s).
Instance "-MGMTDB", status READY, has 1 handler(s) for this service…
Handler(s):
"DEDICATED" established:0 refused:0 state:blocked
REMOTE SERVER
(ADDRESS=(PROTOCOL=TCP)(HOST=172.16.100.62)(PORT=1526))

Subscribe to get access

Read more of this content when you subscribe today.

“ORA-15137: The ASM cluster is in rolling patch state” after applied 19c GI RU

PROBLEM

Just after applied the latest 19c GI July 2020 RU, any operations on diskgroups will cause the following errors :

ORA-15032: not all alterations performed
ORA-15137: The ASM cluster is in rolling patch state.

Subscribe to get access

Read more of this content when you subscribe today.

Check the cluster status back to Normal

-racnode1

[grid@racnode1 ~]$ kfod op=patches
List of Patches
31281355
31304218
31305087
31335188
[grid@racnode1 ~]$
ASMCMD> showclusterstate
Normal
ASMCMD>
[grid@racnode1 ~]$ crsctl query crs softwarepatch
Oracle Clusterware patch level on node racnode1 is [441346801].
[grid@racnode1 ~]$
[grid@racnode1 ~]$ crsctl query crs releasepatch
Oracle Clusterware release patch level is [441346801] and the complete list of patches [31281355 31304218 31305087 31335188 ] have been applied on the local node. The release patch string is [19.8.0.0.0].
[grid@racnode1 ~]$

-racnode2

[grid@racnode2 ~]$ kfod op=patches
List of Patches
31281355
31304218
31305087
31335188
[grid@racnode2 ~]$
ASMCMD> showclusterstate
Normal
ASMCMD>
[grid@racnode2 ~]$ crsctl query crs softwarepatch
Oracle Clusterware patch level on node racnode2 is [441346801].
[grid@racnode2 ~]$
[grid@racnode2 ~]$ crsctl query crs releasepatch
Oracle Clusterware release patch level is [441346801] and the complete list of patches [31281355 31304218 31305087 31335188 ] have been applied on the local node. The release patch string is [19.8.0.0.0].
[grid@racnode2 ~]$

Accidentally Deleted Oracle Database Datafile on Linux Server

Sysadmin accidentally deleted an Oracle database datafile on Linux server. There are a couple of ways to recover this database datafile, like restoring the deleted datafile from RMAN backup, etc.

In this special situation, sysadmin has just deleted the dbf file, and then realized it immediately. So we can try a different way to recover this datafile from OS file descriptor, which is still open in memory by Oracle database background process.

IDENTIFY which datafile has been wrongly deleted

...
..
.
ORA-63999: data file suffered media failure
ORA-01116: error in opening database file 13
ORA-01110: data file 13: '/u01/app/oracle/oradata/CUMDB/CUMDBPDB1/hr.dbf'
ORA-27041: unable to open file
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
...
.
.

Do not offline the deleted file in database until recovering datafile, otherwise file descriptor ( FD) of the deleted file will be lost and unavailable for the rest steps.

Check which OS process still have File Descriptor open for the deleted file

[oracle@oemnode1 trace]$ lsof |grep hr.dbf
ora_dbw0_ 9885 oracle 272uW REG 252,0 104865792 235862719 /u01/app/oracle/oradata/CUMDB/CUMDBPDB1/hr.dbf (deleted)
...
..
.

We can see the file descriptor is 272 by Oracle db writer background process 9885.

Recover datafile from file descriptor

[oracle@oemnode1 proc]$ cd /proc/9885/fd

[oracle@oemnode1 fd]$ ls -ltr 272
lrwx------ 1 oracle oinstall 64 Aug 5 22:33 272 -> /u01/app/oracle/oradata/CUMDB/CUMDBPDB1/hr.dbf (deleted)

[oracle@oemnode1 fd]$ cat 272 > /u01/app/oracle/oradata/CUMDB/CUMDBPDB1/hr.dbf

[oracle@oemnode1 fd]$ ls -ltr /u01/app/oracle/oradata/CUMDB/CUMDBPDB1/hr.dbf
-rw-r--r-- 1 oracle oinstall 104865792 Aug 5 22:38 /u01/app/oracle/oradata/CUMDB/CUMDBPDB1/hr.dbf
SQL> alter database datafile 13 offline;

Database altered.

SQL> recover datafile 13;

Media recovery complete.

SQL> alter database datafile 13 online;

Database altered.

Check and confirm datafile is restored and Recovered successfully

[oracle@oemnode1 fd]$ lsof |grep hr.dbf
oracle_46 4606 oracle 269u REG 252,0 104865792 235862728 /u01/app/oracle/oradata/CUMDB/CUMDBPDB1/hr.dbf
ora_dbw0_ 9885 oracle 272uW REG 252,0 104865792 235862728 /u01/app/oracle/oradata/CUMDB/CUMDBPDB1/hr.dbf
SQL>select FILE_NAME, TABLESPACE_NAME, STATUS  from dba_data_files where file_id=13

FILE_NAME                                      TABLESPACE_NAME  STATUS
----------------------------------------------- ---------------- -------
/u01/app/oracle/oradata/CUMDB/CUMDBPDB1/hr.dbf     HR            AVAILABLE

Please note this method works only when database is still up running after the datafile accidentally being deleted.

opatchauto : Unable to create patchObject

SYMPTOMS

When applying 19c GI JULY 2020 RU ( 19.8.0.0), opatchauto failed with following errors:

#opatchauto apply /tmp/31305339/
...
..
.

Patch: /tmp/31305339/31281355
Log:
Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Unable to create patchObject
Possible causes are:
ORACLE_HOME/inventory/oneoffs/31281355 is corrupted. PatchObject constructor: Input file "/u01/app/19.0.0/grid/inventory/oneoffs/31281355/etc/config/actions" or "/u01/app/19.0.0/grid/inventory/oneoffs/31281355/etc/config/inventory" does not exist.
After fixing the cause of failure Run opatchauto resume
]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.
OPatchauto session completed at Mon Aug 3 21:06:44 2020
Time taken to complete the session 0 minute, 35 seconds
opatchauto failed with error code 42

Subscribe to get access

Read more of this content when you subscribe today.