Category: Oracle

Oracle GI 12.1.0.2 and Chronyd Service

After installed or upgraded to Oracle GI 12.1.0.2, chrony is not supported for some reason.

[grid@racnode2 bin]$ systemctl status chronyd

● chronyd.service - NTP client/server
   Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; 
                                             vendor preset: enabled)
   Active: active (running) since Fri 2020-01-10 16:28:40 AEDT; 
                                                         1h 19min ago
     Docs: man:chronyd(8)
           man:chrony.conf(5)
  Process: 1184 ExecStartPost=/usr/libexec/chrony-helper 
                        update-daemon (code=exited, status=0/SUCCESS)
  Process: 1152 ExecStart=/usr/sbin/chronyd $OPTIONS 
                                    (code=exited, status=0/SUCCESS)
 Main PID: 1166 (chronyd)
   CGroup: /system.slice/chronyd.service
           └─1166 /usr/sbin/chronyd
[root@racnode2 bin]# ./crsctl check css
CRS-4529: Cluster Synchronization Services is online
[root@racnode2 bin]# ./crsctl check ctss
CRS-4701: The Cluster Time Synchronization Service is in Active mode.
CRS-4702: Offset (in msec): 0
[grid@racnode2 bin]$ crsctl stat res -t -init
-------------------------------------------------------------------
Name           Target  State        Server             State details
-------------------------------------------------------------------
Cluster Resources
-------------------------------------------------------------------
ora.asm
      1        ONLINE  ONLINE       racnode2          Started,STABLE
ora.cluster_interconnect.haip
      1        ONLINE  ONLINE       racnode2                 STABLE
ora.crf
      1        ONLINE  ONLINE       racnode2                 STABLE
ora.crsd
      1        ONLINE  ONLINE       racnode2                 STABLE
ora.cssd
      1        ONLINE  ONLINE       racnode2                 STABLE
ora.cssdmonitor
      1        ONLINE  ONLINE       racnode2                 STABLE
ora.ctssd
      1        ONLINE  ONLINE       racnode2          ACTIVE:0,STABLE
...
..
.

octssd.trc

2020-01-10 18:47:45.647193 : CTSS:1717679872: sclsctss_gvss2: 
                                      NTP default pid file not found
2020-01-10 18:47:45.647198 : CTSS:1717679872: ctss_check_vendor_sw: 
              Vendor time sync software is not detected. status [1].

CAUSE

It is a bug according oracle.

Workaround

Apply patch 20936562, which is included in majority GI PSUs since GI PSU 160719 (July 2016)

[grid@racnode2 tmp]$ opatch lsinventory | grep 20936562
19471836, 24445255, 20936562, 28805158, 25037011, 22144696, 18750781

[grid@racnode2 tmp]$ $ORACLE_HOME/OPatch/opatch lsinventory
...
..
.
Unique Patch ID: 22886676
Patch description: "OCW PATCH SET UPDATE 12.1.0.2.190716 (29509318)"
Created on 27 Jun 2019, 08:01:08 hrs PST8PDT
...
..
.
[grid@racnode2 trace]$ crsctl check css
CRS-4529: Cluster Synchronization Services is online

[grid@racnode2 trace]$ crsctl check ctss
CRS-4700:The Cluster Time Synchronization Service is in Observer mode
[grid@racnode2 trace]$ crsctl stat res -t -init
-------------------------------------------------------------------
Name           Target  State        Server             State details
-------------------------------------------------------------------
Cluster Resources
------------------------------------------------------------------
ora.asm
      1        ONLINE  ONLINE       racnode2           Started,STABLE
ora.cluster_interconnect.haip
      1        ONLINE  ONLINE       racnode2                 STABLE
ora.crf
      1        ONLINE  ONLINE       racnode2                 STABLE
ora.crsd
      1        ONLINE  ONLINE       racnode2                 STABLE
ora.cssd
      1        ONLINE  ONLINE       racnode2                 STABLE
ora.cssdmonitor
      1        ONLINE  ONLINE       racnode2                 STABLE
ora.ctssd
      1        ONLINE  ONLINE       racnode2         OBSERVER,STABLE
...
..
.

octssd.trc

2020-01-10 20:52:58.219654 :  CTSS:1530910464: sclsctss_gvss5: 
                              Chrony active, forcing observer mode
2020-01-10 20:52:58.219657 :  CTSS:1530910464: ctss_check_vendor_sw: 
                   Vendor time sync software is detected. status [2].

Just keep in mind cluvfy still checks NTPD “/etc/ntp.conf” configurations. But it is not a big deal.

[grid@racnode2]$ cluvfy comp clocksync -n all -verbose

Verifying Clock Synchronization across the cluster nodes

Checking if Clusterware is installed on all nodes...
Oracle Clusterware is installed on all nodes.

Checking if CTSS Resource is running on all nodes...
Check: CTSS Resource running on all nodes
  Node Name                             Status
  ------------------------------------  ------------------------
  racnode2                              passed
  racnode1                              passed
CTSS resource check passed

Querying CTSS for time offset on all nodes...
Query of CTSS for time offset passed

Check CTSS state started...
Check: CTSS state
  Node Name                             State
  ------------------------------------  ------------------------
  racnode2                              Observer
  racnode1                              Observer
CTSS is in Observer state. Switching over to clock synchronization 
                                                  checks using NTP


Starting Clock synchronization checks using Network Time Protocol(NTP)...

Checking existence of NTP configuration file "/etc/ntp.conf" across 
nodes:
  Node Name                             File exists?
  ------------------------------------  ------------------------
  racnode2                              no
  racnode1                              no
PRVG-1019 : The NTP configuration file "/etc/ntp.conf" does not exist 
            on nodes "racnode2,racnode1"
PRVF-5414 : Check of NTP Config file failed on all nodes. 
            Cannot proceed further for the NTP tests

Checking daemon liveness...

Check: Liveness for "ntpd"
  Node Name                             Running?
  ------------------------------------  ------------------------
  racnode2                              no
  racnode1                              no
PRVF-7590 : "ntpd" is not running on node "racnode2"
PRVF-7590 : "ntpd" is not running on node "racnode1"
PRVG-1024 : The NTP Daemon or Service was not running on any of the 
            cluster nodes.
PRVF-5415 : Check to see if NTP daemon or service is running failed
Result: Clock synchronization check using Network Time Protocol(NTP) 
        failed

PRVF-9652 : Cluster Time Synchronization Services check failed

Verification of Clock Synchronization across the cluster nodes was 
unsuccessful on all the specified nodes.

Reconfigure ASMLIB after Unbreakable Enterprise Kernel (UEK) Upgraded from UEKR3 to UEKR5 on Oracle Linux 7

After unbreakable enterprise kernel (UEK) is upgraded from release 3 to release 5 on Oracle Linux 7,  the oracle asm is not working well:

[root@racnode1 ~]# oracleasm listdisks
[root@racnode1 ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
Instantiating disk "ASM_DISK1"
Unable to instantiate disk "ASM_DISK1"
Instantiating disk "ASM_DISK2"
Unable to instantiate disk "ASM_DISK2"
Instantiating disk "ASM_DISK3"
Unable to instantiate disk "ASM_DISK3"
Instantiating disk "ASM_DISK4"
Unable to instantiate disk "ASM_DISK4"
Instantiating disk "ASM_DISK5"
Unable to instantiate disk "ASM_DISK5"
Instantiating disk "ASM_DISK6"
Unable to instantiate disk "ASM_DISK6"
Instantiating disk "ASM_DISK7"
Unable to instantiate disk "ASM_DISK7"
Instantiating disk "ASM_OCR_VOTE"
Unable to instantiate disk "ASM_OCR_VOTE"
[root@racnode1 ~]# oracleasm listdisks

Check oracleasm packages , and found there are one or two are missing:

[root@racnode1 ~]# rpm -qa|grep -i oracleasm
oracleasm-support-2.1.11-2.el7.x86_64
[root@racnode1 ~]#

The Oracle ASMLib kernel driver is now included in the Unbreakable Enterprise Kernel. No driver package needs to be installed when using this kernel.

Check Oracle ASMLib kernel driver has been installed.

[root@racnode1 ~]# lsmod|grep asm
oracleasm              61440  1

[root@racnode1 ~]# modinfo oracleasm
filename:       /lib/modules/4.14.35-1902.8.4.el7uek.x86_64/kernel/drivers/block/oracleasm/oracleasm.ko.xz
description:    Kernel driver backing the Generic Linux ASM Library.
author:         Joel Becker, Martin K. Petersen <martin.petersen@oracle.com>
version:        2.0.8
license:        GPL
srcversion:     DF8809442FB655948FEC227
depends:
retpoline:      Y
intree:         Y
name:           oracleasm
vermagic:       4.14.35-1902.8.4.el7uek.x86_64 SMP mod_unload modversions
signat:         PKCS#7
signer:
sig_key:
sig_hashalgo:   md4
parm:           use_logical_block_size:Prefer logical block size over physical (Y=logical, N=physical [default]) (bool)
[root@racnode1 ~]#

For RHEL linux, install  kmod-oracleasm package:

[root@racnode1 ~]# yum install kmod-oracleasm

Download and install oracleasmlib-2.0.12-1.el7.x86_64.rpm package from below url, if it is unavailable.

https://www.oracle.com/linux/downloads/linux-asmlib-v7-downloads.html

Check the required packages are all available now:

[root@racnode1 ~]# rpm -qa|grep -i oracleasm
oracleasmlib-2.0.12-1.el7.x86_64
oracleasm-support-2.1.11-2.el7.x86_64
kmod-oracleasm-2.0.8-26.0.1.el7.x86_64 ( for RHEL )

Configure ASM

[root@racnode1 ~]# oracleasm configure -i
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface [grid]:
Default group to own the driver interface [oinstall]:
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
[root@racnode1 ~]#

Re-initialize oracleasm:

[root@racnode1 sysconfig]# oracleasm init
Creating /dev/oracleasm mount point: /dev/oracleasm
Loading module "oracleasm": oracleasm
Configuring "oracleasm" to use device physical block size
Mounting ASMlib driver filesystem: /dev/oracleasm

Scan and list ASM disks:

[root@racnode1 sysconfig]# oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes


[root@racnode1 sysconfig]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
Instantiating disk "ASM_DISK1"
Instantiating disk "ASM_DISK2"
Instantiating disk "ASM_DISK3"
Instantiating disk "ASM_DISK4"
Instantiating disk "ASM_DISK5"
Instantiating disk "ASM_DISK6"
Instantiating disk "ASM_DISK7"
Instantiating disk "ASM_OCR_VOTE"


[root@racnode1 sysconfig]# oracleasm listdisks
ASM_DISK1
ASM_DISK2
ASM_DISK3
ASM_DISK4
ASM_DISK5
ASM_DISK6
ASM_DISK7
ASM_OCR_VOTE

Create Named Credentials by Using emcli

In order to create a named credential, you need to know:

  • Target name (unless you create a global credential)
  • Target type
  •  Credential type associated with the target type

Now we start to create a named credential for cluster database ( RACTEST ) user ( JAMES).

1) To get TARGET_TYPE from TARGET_NAME. Normally we know a couple of target_type from daily emcli practice.

$ emcli get_targets -targets="RACTEST:%"

Status ID Status Target Type  Target Name
1         Up     rac_database  RACTETS

2) To get the credential types (and their attributes) associated with “rac_database”.

$ emcli show_credential_type_info -target_type=rac_database
Target Type  Cred Type Name      Cred Type Column Name Key Column
rac_database DBCreds             DBPassword            No
                                 DBRole                No
                                 DBUserName            Yes
             DBHostCreds         HostPassword          No
                                 HostUserName          Yes
             DBKerberosCreds     DBKerberosPassword    No
                                 DBKerberosUserName    Yes
             DBPkiCreds          DBPkiUserWallet       Yes
                                 DBPkiUserWalletPassword No
                                 DBPkiUserWalletType   No
             HostSSHCreds        SSH_PUB_KEY           No
                                 SSH_PVT_KEY           No
                                 USERNAME              Yes

We can see target type rac_database has four credential types. we will use credential type  DBCreds in the following steps.

3) Create named credential for JAMES user of the RAC database.

$emcli create_named_credential -auth_target_type=rac_database \
 -cred_scope=Instance -target_type=rac_database \
-target_name=RACTEST -cred_type=DBCreds -cred_name=NC_RACTEST_JAMES \
-attributes="DBUserName:JAMES;DBPassword:yourpasswd" -test

Credential NC_RACTEST_JAMES created.

Here “-test” to test the credential before saving.

If you want to create a global named credential, just remove “cred_scope, target_type, target_name” parameters.

Clear Problems in OEM by Using EMCLI

There are many old problems and their incidents in OEM repository for many years.  Here is an example of  how to clear those problems by using emcli. Otherwise you might need log on to OEM console by clearing them from one target to another. Imagine your OEM repository might manage thousands of different kinds of targets.

Subscribe to get access

Read more of this content when you subscribe today.

Subscribe

Clears Stateless Alerts in OEM by Using EMCLI

The Enterprise Manager Agent does not automatically clear those stateless alerts.

This post explains how to clean up stateless alerts, also provides shell scripts to generate commands to automatically clean all stateless alerts if possible.

Subscribe to get access

Read more of this content when you subscribe today.

Subscribe