network unreachable resolving ‘./NS/IN’ or ‘./DNSKEY/IN’

While checking named status on Oracle Linux, the following warnings occur:

# systemctl status named
..
.
May 17 20:40:59 HOSTNAME named[8468]: network unreachable resolving './NS/IN': 2001:500:1::53#53
May 17 20:40:59 HOSTNAME named[8468]: network unreachable resolving './DNSKEY/IN': 2001:500:2d::d#53
..
.

Subscribe to get access

Read more of this content when you subscribe today.

How to Disable Transparent HugePages on Oracle Linux 7

It is recommended by Oracle to disable Linux native Transparent HugePages on Oracle database servers, to avoid potential Oracle database performance issues.

This post introduces how to disable Linux native Transparent HugePages, and verify Transparent Hugepages disabled or not.

Subscribe to get access

Read more of this content when you subscribe today.

PRVG-11551 : Required version of package “cvuqdisk” was not found

Run runcluvfy.sh for pre-upgrading GI from 12.1.0.2 to 12.2.0.1, and get “PRVG-11551 : Required version of package “cvuqdisk” was not found” warning.

$pwd
/u01/app/12.2.0.1/grid

$ ./runcluvfy.sh stage -pre crsinst -upgrade -rolling \ 
              -src_crshome /u01/app/12.1.0/grid     \
              -dest_crshome /u01/app/12.2.0.1/grid  \
              -dest_version 12.2.0.1 -fixup -verbose


Verifying Package: cvuqdisk-1.0.10-1 ...FAILED
racnode2: PRVG-11551 : Required version of package "cvuqdisk" was 
not found on node "racnode2" [Required = "cvuqdisk-1.0.10-1" ; Found =
"cvuqdisk-1.0.9-1"].

racnode1: PRVG-11551 : Required version of package "cvuqdisk" was not found on
node "racnode1" [Required = "cvuqdisk-1.0.10-1" ; Found =
"cvuqdisk-1.0.9-1"].

The old version package was installed.

[root@racnode2 grid]# rpm -qa | grep -i cvu
cvuqdisk-1.0.9-1.x86_64

Find the new cvuqdisk package from new GI HOME binary , and install ( -i ) or upgrade it ( -U ).

[root@racnode2 ~]# cd /u01/app/12.2.0.1
[root@racnode2 12.2.0.1]# cd grid

[root@racnode2 grid]# pwd
/u01/app/12.2.0.1/grid

[root@racnode2 grid]# find ./ -name cvuqdisk*
./cv/rpm/cvuqdisk-1.0.10-1.rpm
./cv/remenv/cvuqdisk-1.0.10-1.rpm

[root@racnode2 grid]# rpm -Uv ./cv/rpm/cvuqdisk-1.0.10-1.rpm
Preparing packages...
cvuqdisk-1.0.10-1.x86_64
cvuqdisk-1.0.9-1.x86_64
[root@racnode2 grid]# rpm -qa | grep cvuqdisk
cvuqdisk-1.0.10-1.x86_64
[root@racnode2 grid]#

How to Get and Install 32 bit Linux Packages

For Oracle 12c, both 32bit and 64bit packages are required to be installed, but it seems 32bit one is missing :

[root@racnode2 ~]# rpm -qa | grep glibc-devel
glibc-devel-2.17-292.0.1.el7.x86_64
[root@racnode2 ~]#

When try to install by yum, it says it has been installed already.

[root@racnode2 ~]# yum install glibc-devel
Package glibc-devel-2.17-292.0.1.el7.x86_64 already installed and latest version
Nothing to do
[root@racnode2 ~]#

Search the repository.

[root@racnode2 ~]# yum search glibc-devel
Loaded plugins: ulninfo
=================== N/S matched: glibc-devel ====================
glibc-devel.i686 : Object files for development using standard C libraries.
glibc-devel.x86_64 : Object files for development using standard C libraries.

Name and summary matches only, use "search all" for everything.

Install the 32 bit package.

[root@racnode2 ~]# yum install glibc-devel.i686
Loaded plugins: ulninfo
Resolving Dependencies
--> Running transaction check
---> Package glibc-devel.i686 0:2.17-292.0.1.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==================================================================
Package Arch Version Repository Size
==================================================================
Installing:
glibc-devel i686 2.17-292.0.1.el7 ol7_latest 1.1 M

Transaction Summary
=================================================================
Install 1 Package

Total download size: 1.1 M
Installed size: 1.0 M
Is this ok [y/d/N]: y
Downloading packages:
glibc-devel-2.17-292.0.1.el7.i686.rpm | 1.1 MB 00:00:02
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : glibc-devel-2.17-292.0.1.el7.i686 1/1
Verifying : glibc-devel-2.17-292.0.1.el7.i686 1/1

Installed:
glibc-devel.i686 0:2.17-292.0.1.el7

Complete!
[root@racnode2 ~]#

Oracle GI 12.1.0.2 and Chronyd Service

After installed or upgraded to Oracle GI 12.1.0.2, chrony is not supported for some reason.

[grid@racnode2 bin]$ systemctl status chronyd

● chronyd.service - NTP client/server
   Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; 
                                             vendor preset: enabled)
   Active: active (running) since Fri 2020-01-10 16:28:40 AEDT; 
                                                         1h 19min ago
     Docs: man:chronyd(8)
           man:chrony.conf(5)
  Process: 1184 ExecStartPost=/usr/libexec/chrony-helper 
                        update-daemon (code=exited, status=0/SUCCESS)
  Process: 1152 ExecStart=/usr/sbin/chronyd $OPTIONS 
                                    (code=exited, status=0/SUCCESS)
 Main PID: 1166 (chronyd)
   CGroup: /system.slice/chronyd.service
           └─1166 /usr/sbin/chronyd
[root@racnode2 bin]# ./crsctl check css
CRS-4529: Cluster Synchronization Services is online
[root@racnode2 bin]# ./crsctl check ctss
CRS-4701: The Cluster Time Synchronization Service is in Active mode.
CRS-4702: Offset (in msec): 0
[grid@racnode2 bin]$ crsctl stat res -t -init
-------------------------------------------------------------------
Name           Target  State        Server             State details
-------------------------------------------------------------------
Cluster Resources
-------------------------------------------------------------------
ora.asm
      1        ONLINE  ONLINE       racnode2          Started,STABLE
ora.cluster_interconnect.haip
      1        ONLINE  ONLINE       racnode2                 STABLE
ora.crf
      1        ONLINE  ONLINE       racnode2                 STABLE
ora.crsd
      1        ONLINE  ONLINE       racnode2                 STABLE
ora.cssd
      1        ONLINE  ONLINE       racnode2                 STABLE
ora.cssdmonitor
      1        ONLINE  ONLINE       racnode2                 STABLE
ora.ctssd
      1        ONLINE  ONLINE       racnode2          ACTIVE:0,STABLE
...
..
.

octssd.trc

2020-01-10 18:47:45.647193 : CTSS:1717679872: sclsctss_gvss2: 
                                      NTP default pid file not found
2020-01-10 18:47:45.647198 : CTSS:1717679872: ctss_check_vendor_sw: 
              Vendor time sync software is not detected. status [1].

CAUSE

It is a bug according oracle.

Workaround

Apply patch 20936562, which is included in majority GI PSUs since GI PSU 160719 (July 2016)

[grid@racnode2 tmp]$ opatch lsinventory | grep 20936562
19471836, 24445255, 20936562, 28805158, 25037011, 22144696, 18750781

[grid@racnode2 tmp]$ $ORACLE_HOME/OPatch/opatch lsinventory
...
..
.
Unique Patch ID: 22886676
Patch description: "OCW PATCH SET UPDATE 12.1.0.2.190716 (29509318)"
Created on 27 Jun 2019, 08:01:08 hrs PST8PDT
...
..
.
[grid@racnode2 trace]$ crsctl check css
CRS-4529: Cluster Synchronization Services is online

[grid@racnode2 trace]$ crsctl check ctss
CRS-4700:The Cluster Time Synchronization Service is in Observer mode
[grid@racnode2 trace]$ crsctl stat res -t -init
-------------------------------------------------------------------
Name           Target  State        Server             State details
-------------------------------------------------------------------
Cluster Resources
------------------------------------------------------------------
ora.asm
      1        ONLINE  ONLINE       racnode2           Started,STABLE
ora.cluster_interconnect.haip
      1        ONLINE  ONLINE       racnode2                 STABLE
ora.crf
      1        ONLINE  ONLINE       racnode2                 STABLE
ora.crsd
      1        ONLINE  ONLINE       racnode2                 STABLE
ora.cssd
      1        ONLINE  ONLINE       racnode2                 STABLE
ora.cssdmonitor
      1        ONLINE  ONLINE       racnode2                 STABLE
ora.ctssd
      1        ONLINE  ONLINE       racnode2         OBSERVER,STABLE
...
..
.

octssd.trc

2020-01-10 20:52:58.219654 :  CTSS:1530910464: sclsctss_gvss5: 
                              Chrony active, forcing observer mode
2020-01-10 20:52:58.219657 :  CTSS:1530910464: ctss_check_vendor_sw: 
                   Vendor time sync software is detected. status [2].

Just keep in mind cluvfy still checks NTPD “/etc/ntp.conf” configurations. But it is not a big deal.

[grid@racnode2]$ cluvfy comp clocksync -n all -verbose

Verifying Clock Synchronization across the cluster nodes

Checking if Clusterware is installed on all nodes...
Oracle Clusterware is installed on all nodes.

Checking if CTSS Resource is running on all nodes...
Check: CTSS Resource running on all nodes
  Node Name                             Status
  ------------------------------------  ------------------------
  racnode2                              passed
  racnode1                              passed
CTSS resource check passed

Querying CTSS for time offset on all nodes...
Query of CTSS for time offset passed

Check CTSS state started...
Check: CTSS state
  Node Name                             State
  ------------------------------------  ------------------------
  racnode2                              Observer
  racnode1                              Observer
CTSS is in Observer state. Switching over to clock synchronization 
                                                  checks using NTP


Starting Clock synchronization checks using Network Time Protocol(NTP)...

Checking existence of NTP configuration file "/etc/ntp.conf" across 
nodes:
  Node Name                             File exists?
  ------------------------------------  ------------------------
  racnode2                              no
  racnode1                              no
PRVG-1019 : The NTP configuration file "/etc/ntp.conf" does not exist 
            on nodes "racnode2,racnode1"
PRVF-5414 : Check of NTP Config file failed on all nodes. 
            Cannot proceed further for the NTP tests

Checking daemon liveness...

Check: Liveness for "ntpd"
  Node Name                             Running?
  ------------------------------------  ------------------------
  racnode2                              no
  racnode1                              no
PRVF-7590 : "ntpd" is not running on node "racnode2"
PRVF-7590 : "ntpd" is not running on node "racnode1"
PRVG-1024 : The NTP Daemon or Service was not running on any of the 
            cluster nodes.
PRVF-5415 : Check to see if NTP daemon or service is running failed
Result: Clock synchronization check using Network Time Protocol(NTP) 
        failed

PRVF-9652 : Cluster Time Synchronization Services check failed

Verification of Clock Synchronization across the cluster nodes was 
unsuccessful on all the specified nodes.