Category: GI and RAC

GI Postpatch Command Fails Due To rpm Database Corruption At The OS Level

The following errors occur when running “$ORACLE_HOME/crs/install/rootcrs.sh -postpatch”:

2024-11-09 13:31:20: Executing cmd: /bin/rpm -q sles-release
2024-11-09 13:31:20: Command output:
>  error: rpmdb: BDB0113 Thread/process 128527/140400711669824 failed: BDB1507 Thread died in Berkeley DB library
>  error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
>  error: cannot open Packages index using db5 -  (-30973)
>  error: cannot open Packages database in /var/lib/rpm
>  error: rpmdb: BDB0113 Thread/process 128527/140400711669824 failed: BDB1507 Thread died in Berkeley DB library
>  error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
>  error: cannot open Packages database in /var/lib/rpm
>  package sles-release is not installed
>End Command output
2024-11-09 13:31:20: Check if the startup mechanism systemd is being used
2024-11-09 13:31:20: Executing cmd: /bin/rpm -qf /sbin/init
2024-11-09 13:31:20: Command output:
>  error: rpmdb: BDB0113 Thread/process 128527/140400711669824 failed: BDB1507 Thread died in Berkeley DB library
>  error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
>  error: cannot open Packages index using db5 -  (-30973)
>  error: cannot open Packages database in /var/lib/rpm
>  error: rpmdb: BDB0113 Thread/process 128527/140400711669824 failed: BDB1507 Thread died in Berkeley DB library
>  error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
>  error: cannot open Packages database in /var/lib/rpm
>  error: rpmdb: BDB0113 Thread/process 128527/140400711669824 failed: BDB1507 Thread died in Berkeley DB library
>  error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
>  error: cannot open Packages database in /var/lib/rpm
>  file /sbin/init is not owned by any package
>End Command output
2024-11-09 13:31:20: Executing cmd: /u01/app/19.0.0/grid/bin/clsecho -p has -f clsrsc -m 180 '/bin/rpm -qf /sbin/init' '1'
2024-11-09 13:31:20: Executing cmd: /u01/app/19.0.0/grid/bin/clsecho -p has -f clsrsc -m 180 '/bin/rpm -qf /sbin/init' '1'
2024-11-09 13:31:20: Command output:
>  CLSRSC-180: An error occurred while executing the command '/bin/rpm -qf /sbin/init'
>End Command output
2024-11-09 13:31:20: CLSRSC-180: An error occurred while executing the command '/bin/rpm -qf /sbin/init'
2024-11-09 13:31:20: ###### Begin DIE Stack Trace ######
2024-11-09 13:31:20:     Package         File                 Line Calling
2024-11-09 13:31:20:     --------------- -------------------- ---- ----------
2024-11-09 13:31:20:  1: main            rootcrs.pl            358 crsutils::dietrap
2024-11-09 13:31:20:  2: s_crsutils      s_crsutils.pm        2729 main::__ANON__
2024-11-09 13:31:20:  3: s_crsutils      s_crsutils.pm        2701 s_crsutils::s_checkLinuxInitMethod
2024-11-09 13:31:20:  4: s_crsutils      s_crsutils.pm        3712 s_crsutils::s_is_Linux_Systemd
2024-11-09 13:31:20:  5: oraClusterwareComp::oraohasd oraohasd.pm          1537 s_crsutils::s_copy_afdinit_init
2024-11-09 13:31:20:  6: crspatch        crspatch.pm           730 oraClusterwareComp::oraohasd::update_OHASD
2024-11-09 13:31:20:  7: crspatch        crspatch.pm          1912 crspatch::updateSystemFiles
2024-11-09 13:31:20:  8: crspatch        crspatch.pm          2249 crspatch::performPostPatch
2024-11-09 13:31:20:  9: crspatch        crspatch.pm           526 crspatch::crsPostPatch
2024-11-09 13:31:20: 10: main            rootcrs.pl            371 crspatch::new
2024-11-09 13:31:20: ####### End DIE Stack Trace #######

2024-11-09 13:31:20: ROOTCRS_POSTPATCH_UPDATE_OHASD_SERVICE checkpoint has failed

Solution

OS level RPM database is corrupted.

Recreate RPM database at OS level to resolve this issue:

1.  As root OS user run the following:

# rm -f /var/lib/rpm/__*
# /bin/rpm –rebuilddb
# echo $?

2.  Validate the rpm database, as the root user:

# /bin/rpm -qa | more


3.  If last command listing rpms correctly, then re-run postpatch command or resume your opatchauto patching session:

As root OS user:

cd $GI_HOME/crs/install
For Cluster Grid Infrastructure Install:#./rootcrs.sh -postpatch

For Restart Grid Infrastructure Install:#./roothas.sh -postpatch

Reference

Postpatch Command Fails on Grid Infrastructure Home Due To rpm Database Corruption At The OS Level (Doc ID 2365433.1)

ORA-15099: disk is larger than maximum size of 2097152 MBs

The following errors occur when creating a new ASM diskgroup:

SQL> CREATE DISKGROUP DATA1 EXTERNAL REDUNDANCY disk 'AFD:DATA1_DISK1' NAME DATA1_DISK1;
CREATE DISKGROUP DATA1 EXTERNAL REDUNDANCY disk 'AFD:DATA1_DISK1' NAME DATA1_DISK1;
*
ERROR at line 1:
ORA-15018: diskgroup cannot be created
ORA-15099: disk 'AFD:DATA1_DISK1' is larger than maximum size of 2097152 MBs

SOLUTION

SQL> CREATE DISKGROUP DATA1 EXTERNAL REDUNDANCY disk 'AFD:DATA1_DISK1' NAME DATA1_DISK1 ATTRIBUTE 'compatible.rdbms' = '19.0', 'compatible.asm' = '19.0';

Diskgroup created.

How to Debug ‘cluvfy” or “runcluvfy.sh”

Debug “cluvfy” or “runcluvfy.sh”:

$ rm -rf /tmp/cvutrace
$ mkdir /tmp/cvutrace
$ export CV_TRACELOC=/tmp/cvutrace
$ export SRVM_TRACE=true
$ export SRVM_TRACE_LEVEL=1

$ cluvfy stage -pre dbinst -allnodes -r 12.2 -d /u01/app/oracle/product/12.2.0/dbhome_1

$ ls -ltr  /tmp/cvutrace
total 1960
-rw-r--r-- 1 grid oinstall       0 Sep  8 21:46 cvutrace.log.0.lck
-rw-r--r-- 1 grid oinstall       0 Sep  8 21:47 cvuhelper.log.0.lck
-rw-r--r-- 1 grid oinstall    1586 Sep  8 21:47 cvuhelper.log.0
-rw-r--r-- 1 grid oinstall 2000962 Sep  8 21:47 cvutrace.log.0

[INS-06006] Passwordless SSH connectivity not set up between the following node(s): [racnode2]

When attempting to configure 19c grid infrastructure on Linux 8 by running gridSetup.sh the following error occurs in SSH connectivity step:

[INS-06006] Passwordless SSH connectivity not set up between the following node(s): [racnode2]

Even after manually setup SSH equivalence successfully, the error still exists and user cannot continue the GI installation and configuration.

SOLUTION

Apply patch 30159782 before run “gridSetup.sh”, see (Doc ID 1410202.1) for how to apply OneOff patch before run “gridSetup.sh”.

OR

apply the RU or higher where the fix for 30159782.8 is included

OR

Before installation, as root user: (please change the path if the location of your "scp" is not the same with below)

# Rename the original scp.
mv /usr/bin/scp /usr/bin/scp.orig

# Create a new file </usr/bin/scp>.
vi /usr/bin/scp
# Add the below line to the new created file </usr/bin/scp>.
/usr/bin/scp.orig -T $*

# Change the file permission.
chmod 555 /usr/bin/scp

After installation:
mv /usr/bin/scp.orig /usr/bin/scp

[INS-08101] Unexpected error while executing the action at state: ‘supportedOSCheck’

Running Oracle 19c gridSetup.sh or runInstaller on Linux 8, the following errors occur:

[INS-08101] Unexpected error while executing the action at state: ‘supportedOSCheck’

SOLUTION

Set the following variable:

    $ export CV_ASSUME_DISTID=OEL7.8

    OR

    Edit the file cvu_config which is located in $ORACLE_HOME/cv/admin, change the line from

    # Fallback to this distribution id

    #CV_ASSUME_DISTID=OEL5

    to

    # Fallback to this distribution id

    CV_ASSUME_DISTID=OEL7.8