As we know, GI 12.1.0.2 still check ntpd though chronyd is running as per this blog.
After upgrading to 12.2.0.1 GI, How can GI work with Chronyd ?
Subscribe to get access
Read more of this content when you subscribe today.
Tag: crsctl check ctss
Oracle GI 12.1.0.2 and Chronyd Service
After installed or upgraded to Oracle GI 12.1.0.2, chrony is not supported for some reason.
[grid@racnode2 bin]$ systemctl status chronyd
● chronyd.service - NTP client/server
Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled;
vendor preset: enabled)
Active: active (running) since Fri 2020-01-10 16:28:40 AEDT;
1h 19min ago
Docs: man:chronyd(8)
man:chrony.conf(5)
Process: 1184 ExecStartPost=/usr/libexec/chrony-helper
update-daemon (code=exited, status=0/SUCCESS)
Process: 1152 ExecStart=/usr/sbin/chronyd $OPTIONS
(code=exited, status=0/SUCCESS)
Main PID: 1166 (chronyd)
CGroup: /system.slice/chronyd.service
└─1166 /usr/sbin/chronyd
[root@racnode2 bin]# ./crsctl check css CRS-4529: Cluster Synchronization Services is online [root@racnode2 bin]# ./crsctl check ctss CRS-4701: The Cluster Time Synchronization Service is in Active mode. CRS-4702: Offset (in msec): 0
[grid@racnode2 bin]$ crsctl stat res -t -init
-------------------------------------------------------------------
Name Target State Server State details
-------------------------------------------------------------------
Cluster Resources
-------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE racnode2 Started,STABLE
ora.cluster_interconnect.haip
1 ONLINE ONLINE racnode2 STABLE
ora.crf
1 ONLINE ONLINE racnode2 STABLE
ora.crsd
1 ONLINE ONLINE racnode2 STABLE
ora.cssd
1 ONLINE ONLINE racnode2 STABLE
ora.cssdmonitor
1 ONLINE ONLINE racnode2 STABLE
ora.ctssd
1 ONLINE ONLINE racnode2 ACTIVE:0,STABLE
...
..
.
octssd.trc
2020-01-10 18:47:45.647193 : CTSS:1717679872: sclsctss_gvss2:
NTP default pid file not found
2020-01-10 18:47:45.647198 : CTSS:1717679872: ctss_check_vendor_sw:
Vendor time sync software is not detected. status [1].
CAUSE
It is a bug according oracle.
Workaround
Apply patch 20936562, which is included in majority GI PSUs since GI PSU 160719 (July 2016)
[grid@racnode2 tmp]$ opatch lsinventory | grep 20936562
19471836, 24445255, 20936562, 28805158, 25037011, 22144696, 18750781
[grid@racnode2 tmp]$ $ORACLE_HOME/OPatch/opatch lsinventory
...
..
.
Unique Patch ID: 22886676
Patch description: "OCW PATCH SET UPDATE 12.1.0.2.190716 (29509318)"
Created on 27 Jun 2019, 08:01:08 hrs PST8PDT
...
..
.
[grid@racnode2 trace]$ crsctl check css CRS-4529: Cluster Synchronization Services is online [grid@racnode2 trace]$ crsctl check ctss CRS-4700:The Cluster Time Synchronization Service is in Observer mode
[grid@racnode2 trace]$ crsctl stat res -t -init
-------------------------------------------------------------------
Name Target State Server State details
-------------------------------------------------------------------
Cluster Resources
------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE racnode2 Started,STABLE
ora.cluster_interconnect.haip
1 ONLINE ONLINE racnode2 STABLE
ora.crf
1 ONLINE ONLINE racnode2 STABLE
ora.crsd
1 ONLINE ONLINE racnode2 STABLE
ora.cssd
1 ONLINE ONLINE racnode2 STABLE
ora.cssdmonitor
1 ONLINE ONLINE racnode2 STABLE
ora.ctssd
1 ONLINE ONLINE racnode2 OBSERVER,STABLE
...
..
.
octssd.trc
2020-01-10 20:52:58.219654 : CTSS:1530910464: sclsctss_gvss5:
Chrony active, forcing observer mode
2020-01-10 20:52:58.219657 : CTSS:1530910464: ctss_check_vendor_sw:
Vendor time sync software is detected. status [2].
Just keep in mind cluvfy still checks NTPD “/etc/ntp.conf” configurations. But it is not a big deal.
[grid@racnode2]$ cluvfy comp clocksync -n all -verbose
Verifying Clock Synchronization across the cluster nodes
Checking if Clusterware is installed on all nodes...
Oracle Clusterware is installed on all nodes.
Checking if CTSS Resource is running on all nodes...
Check: CTSS Resource running on all nodes
Node Name Status
------------------------------------ ------------------------
racnode2 passed
racnode1 passed
CTSS resource check passed
Querying CTSS for time offset on all nodes...
Query of CTSS for time offset passed
Check CTSS state started...
Check: CTSS state
Node Name State
------------------------------------ ------------------------
racnode2 Observer
racnode1 Observer
CTSS is in Observer state. Switching over to clock synchronization
checks using NTP
Starting Clock synchronization checks using Network Time Protocol(NTP)...
Checking existence of NTP configuration file "/etc/ntp.conf" across
nodes:
Node Name File exists?
------------------------------------ ------------------------
racnode2 no
racnode1 no
PRVG-1019 : The NTP configuration file "/etc/ntp.conf" does not exist
on nodes "racnode2,racnode1"
PRVF-5414 : Check of NTP Config file failed on all nodes.
Cannot proceed further for the NTP tests
Checking daemon liveness...
Check: Liveness for "ntpd"
Node Name Running?
------------------------------------ ------------------------
racnode2 no
racnode1 no
PRVF-7590 : "ntpd" is not running on node "racnode2"
PRVF-7590 : "ntpd" is not running on node "racnode1"
PRVG-1024 : The NTP Daemon or Service was not running on any of the
cluster nodes.
PRVF-5415 : Check to see if NTP daemon or service is running failed
Result: Clock synchronization check using Network Time Protocol(NTP)
failed
PRVF-9652 : Cluster Time Synchronization Services check failed
Verification of Clock Synchronization across the cluster nodes was
unsuccessful on all the specified nodes.
Disable Chronyd and Enable CTSSD into Active Mode in Linux 7
As we know, if any Network Time Protocol (NTP) demon ntpd or chronyd running, then Oracle cluster ware CTSS ( Cluster Time Synchronization Service ) will run in Observer mode.
$ crsctl check ctss CRS-4701:The Cluster Time Synchronization Service is in Observer mode.
$crsctl stat res -t -init
...
..
.
ora.ctssd
1 ONLINE ONLINE racnode1 BSERVER,STABLE
Now we disable Chronyd, and remove all Chrony configurations.
# systemctl stop chronyd
# systemctl disable chronyd Removed symlink /etc/systemd/system/multi-user.target.wants/chronyd.service.
# yum remove chrony Loaded plugins: ulninfo Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast Resolving Dependencies --> Running transaction check ---> Package chrony.x86_64 0:2.1.1-1.el7 will be erased --> Finished Dependency Resolution Dependencies Resolved
Check configurations files are gone, otherwise ctssd still thinks NTP servers are running.
$ ls -ltr /etc/chro* -rw-r-----. 1 root chrony 62 Nov 24 2015 /etc/chrony.keys.rpmsave
Check again, we see CTSSD running in ACTIVE mode now.
$ crsctl check ctss CRS-4701: The Cluster Time Synchronization Service is in Active mode. CRS-4702: Offset (in msec): 0
$ crsctl stat res -t -init
ora.ctssd
1 ONLINE ONLINE racnode1 ACTIVE:0,STABLE
$ cluvfy comp clocksync -n all -verbose Verifying Clock Synchronization across the cluster nodes Checking if Clusterware is installed on all nodes... Oracle Clusterware is installed on all nodes. Checking if CTSS Resource is running on all nodes... Check: CTSS Resource running on all nodes Node Name Status ----------- ------------------------ racnode2 passed racnode1 passed CTSS resource check passed Querying CTSS for time offset on all nodes... Query of CTSS for time offset passed Check CTSS state started... Check: CTSS state Node Name State ----------- ------------------------ racnode2 Active racnode1 Active CTSS is in Active state. Proceeding with check of clock time offsets on all nodes... Reference Time Offset Limit: 1000.0 msecs Check: Reference Time Offset Node Name Time Offset Status ------------ ----------- ------------------------ racnode2 0.0 passed racnode1 0.0 passed Time offset is within the specified limits on the following set of nodes: "[racnode2, racnode1]" Result: Check of clock time offsets passed Oracle Cluster Time Synchronization Services check passed Verification of Clock Synchronization across the cluster nodes was successful.
crs alert.log
2019-09-08 18:46:55.004 [OCTSSD(22044)]CRS-2410: The Cluster Time
Synchronization Service on host racnode2 is in active mode.
octssd.trc on master node ( racnode2 ):
.... .. . 2019-09-08 19:31:56.380369 : CTSS:1714730752: sclsctss_ivsr2: default pid file not found 2019-09-08 19:31:56.380386 : CTSS:1714730752: sclsctss_ivsr2: default pid file not found 2019-09-08 19:31:56.380393 : CTSS:1714730752: ctss_check_vendor_sw: Vendor time sync software is not detected. status [1]. ... .. .
octssd.trc on non-master node ( racnode1 ):
2019-09-08 19:39:07.441725 : CTSS:2003805952: ctsselect_msm: CTSS mode is [0xc4] 2019-09-08 19:39:07.441736 : CTSS:2003805952: ctssslave_swm1_2: Ready to initiate new time sync process. 2019-09-08 19:39:07.442805 : CTSS:2003805952: ctssslave_swm2_1: Waiting for time sync message from master. sync_state[2]. 2019-09-08 19:39:07.447917 : CTSS:2008008448: ctssslave_msg_handler4_1: Waiting for slave_sync_with_master to finish sync process. sync_state[3]. 2019-09-08 19:39:07.447926 : CTSS:2003805952: ctssslave_swm2_3: Received time sync message from master. 2019-09-08 19:39:07.447935 : CTSS:2003805952: ctssslave_swm15: The CTSS master is ahead this node. The local time offset [11975 usec] is being adjusted. Sync method [2] 2019-09-08 19:39:07.447938 : CTSS:2003805952: ctssslave_swm17: LT [1567935547sec 447908usec], MT [1567935547sec 139990164505707usec], Delta [6167usec] 2019-09-08 19:39:07.447940 : CTSS:2003805952: ctssslave_swm19: The offset is [-11975 usec] and sync interval set to [1] 2019-09-08 19:39:07.447943 : CTSS:2003805952: ctsselect_msm: Sync interval returned in [1] 2019-09-08 19:39:07.447950 : CTSS:2008008448: ctssslave_msg_handler4_3: slave_sync_with_master finished sync process. Exiting clsctssslave_msg_handler
CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time
In crs alert.log, below message is seen every 30 minutes in 12.2.0.1 GI.
[OCTSSD(9932)]CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time
ctss is running in observer mode, because NTP is running.
$ crsctl check ctss CRS-4700: The Cluster Time Synchronization Service is in Observer mode.
Subscribe to get access
Read more of this content when you subscribe today.
Make DBA Life Easy 