PMON (ospid: 123456): terminating the instance due to error 500

Oracle instance crashes with the following errors, sometimes the instance fails to start up with the same messages:

...
..
.
Errors in file /u02/app/oracle/diag/rdbms/ractest/RACTEST1/trace/RACTEST1_lreg_104781.trc  (incident=287152) (PDBNAME=CDB$ROOT):
ORA-07445: exception encountered: core dump [nscall()+911] [SIGSEGV] [ADDR:0x186] [PC:0x5C6F35F] [Address not mapped to object] []
Incident details in: /u02/app/oracle/diag/rdbms/ractest/RACTEST1/incident/incdir_287152/RACTEST1_lreg_104781_i287152.trc
...
..
.
Tue Aug 24 14:58:01 2021
Instance Critical Process (pid: 29, ospid: 104781, LREG) died unexpectedly
PMON (ospid: 123456): terminating the instance due to error 500
Tue Aug 24 14:58:01 2021
System state dump requested by (instance=1, osid=123456(PMON)), summary=[abnormal instance termination].
System State dumped to trace file /u02/app/oracle/diag/rdbms/ractest/RACTEST1/trace/RACTEST1_diag_104564_20210824145801.trc
Tue Aug 24 14:58:01 2021
Dumping diagnostic data in directory=[cdmp_20210824145801], requested by (instance=1, osid=123456(PMON)), summary=[abnormal instance termination].
Tue Aug 24 14:58:01 2021
Instance terminated by PMON, pid = 123456

The Call Stack Trace in the associated incident trace file ( RACTEST1_lreg_104781_i287152.trc ) shows :

nscall <- nsgrcOpen <- nsgrDo <- nsgrrg_Register <- kmmlrl <- ksucln <- ksbrdp <- opirip
  <- opidrv <- sou2o <- opimai_real <- ssthrdmain <- main
nscall()+911         signal   __sighandler()       2500000153 0F2408F7A
                                                   7F2275499F90 0F2408F7A ?
                                                   000000000 ? 000000000 ?
nsgrcOpen()+512      call     nscall()             2500000153 ? 7F2275499E30 ?
                                                   7F2275499F78 7F227549A0B0
                                                   7F227549A0C8 7F2275499F90
nsgrDo()+81          call     nsgrcOpen()          7F22753F2F30 7F2275499D78
                                                   7F2275499F78 ? 7F227549A0B0 ?
                                                   7F227549A0C8 ? 7F2275499F90 ?
nsgrrg_Register()+2  call     nsgrDo()             7F2275499D78 7F2275499D78 ?
70                                                 7F2275499F78 ? 7F227549A0B0 ?
                                                   7F227549A0C8 ? 7F2275499F90 ?
kmmlrl()+12395       call     nsgrrg_Register()    7F22753F2F30 000000008
                                                   7F2275499F78 ? 7F227549A0B0 ?
                                                   7F227549A0C8 ? 7F2275499F90 ?
kmlmain()+43         call     kmmlrl()             7F22753F2F30 ? 000000008 ?
                                                   7F2275499F78 ? 7F227549A0B0 ?
                                                   7F227549A0C8 ? 7F2275499F90 ?
ksbrdp()+1072        call     kmlmain()            06003EDA0 000000008 ?
                                                   7F2275499F78 ? 7F227549A0B0 ?
                                                   7F227549A0C8 ? 7F2275499F90 ?
opirip()+1488        call     ksbrdp()             06003EDA0 ? 000000008 ?
                                                   7F2275499F78 ? 7F227549A0B0 ?
                                                   7F227549A0C8 ? 7F2275499F90 ?
opidrv()+616         call     opirip()             000000032 000000004
                                                   7FFDA41BB698 7F227549A0B0 ?
                                                   7F227549A0C8 ? 7F2275499F90 ?
sou2o()+145          call     opidrv()             000000032 000000004
                                                   7FFDA41BB698 7F227549A0B0 ?
                                                   7F227549A0C8 ? 7F2275499F90 ?
opimai_real()+270    call     sou2o()              7FFDA41BB670 000000032
                                                   000000004 7FFDA41BB698
                                                   7F227549A0C8 ? 7F2275499F90 ?
ssthrdmain()+412     call     opimai_real()        000000000 7FFDA41BB980
                                                   000000004 ? 7FFDA41BB698 ?
                                                   7F227549A0C8 ? 7F2275499F90 ?
main()+236           call     ssthrdmain()         000000000 000000003

CAUSE and SOLUTION

Subscribe to get access

Read more of this content when you subscribe today.

“Fatal NI Connect Error 12516″ and ”TNS-12564: TNS:connection refused” In Clusterware Alert.log when ologgerd Connects to GIMR

ISSUES

There are repeating following errors in 19c CRS alert.log by ologgered trying to connect to GIMR.

Fatal NI connect error 12516, connecting to:
(DESCRIPTION=(CONNECT_TIMEOUT=60)(ADDRESS=(PROTOCOL=tcp)(HOST=racnode-cluster-scan.virtuallab)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=GIMR_DSCREP_10)(CID=(PROGRAM=ologgerd)(HOST=racnode1.virtuallab)(USER=root)))(SECURITY=(ENCRYPTION_CLIENT=requested)))
VERSION INFORMATION:
TNS for Linux: Version 19.0.0.0.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 19.0.0.0.0 - Production

Version 19.8.0.0.0
Time: 08-AUG-2020 18:15:01
Tracing not turned on.
Tns error struct:
ns main err code: 12564

TNS-12564: TNS:connection refused
ns secondary err code: 12560
nt main err code: 524

TNS-00524: Current operation is still in progress
nt secondary err code: 115
nt OS err code: 0

INVESTIGATION

Check mgmtlsnr or SCAN listeners, and found -MGMTDB PDB service ” gimr_dscrep_10″ is not registered properly.

[grid@racnode1 trace]$ lsnrctl services listener_scan2
...
..
.
Service "gimr_dscrep_10" has 1 instance(s).
Instance "-MGMTDB", status READY, has 1 handler(s) for this serviceā€¦
Handler(s):
"DEDICATED" established:0 refused:0 state:blocked
REMOTE SERVER
(ADDRESS=(PROTOCOL=TCP)(HOST=172.16.100.62)(PORT=1526))

Subscribe to get access

Read more of this content when you subscribe today.

How to Rotate and Purge Listener Logs in 19c

A 19c new feature really makes DBA Life Easy.

Before 19c, there are very limited options to purge listener logs, which could be growing day by day up to a couple of GB size. DBAs have to write their own customised scripts to do this job.

This post demonstrates how to take advantage of 19c a new feature to rotate and purge listerer logs automatically, without a line of script necessary.

Subscribe to get access

Read more of this content when you subscribe today.

TNS-01189: The listener could not authenticate the user

In 19c GI environment, TNS-01189 is still being received hourly from OEM 13cR3 for both local listeners and SCAN listeners, though Oracle said this issue has been fixed since 12.1.0.2.

The OEM alert:

Host=racnode2.virtuallab
Target type=Listener
Target name=LISTENER_SCAN1_racnode-cluster
Message=TNS-1189. Please check log for details.
Severity=Critical
Event reported time=May 1, 2020 9:26:35 PM 
Operating System=Linux
Platform=x86_64
Associated Incident Id=1513
Associated Incident Status=New
Associated Incident Owner=
Associated Incident Acknowledged By Owner=No
Associated Incident Priority=None
Associated Incident Escalation Level=0
Event Type=Metric Alert
Event name=TNS_ERRORS:tnserrmsg
Total Occurrences=115
Occurrences in this event=45
First Occurred in this event=Apr 28, 2020 10:11:23 AM 
Last Occurred in this event=May 1, 2020 8:10:06 PM 
Metric Group=TNS Errors
Metric=TNSMsg
Metric value=TNS-1189(version)
Key Value=TNS-1189
Key Column 1=TNSError
Rule Name=Incident management rule set for all targets,Create 
          incident for critical metric alerts
Rule Owner=System Generated
Update Details:
TNS-1189. Please check log for details.

The local listener and scan listeners error log:

1-MAY-2020 19:10:35 * (CONNECT_DATA=(COMMAND=version)) * version * 1189
TNS-01189: The listener could not authenticate the user

Subscribe to get access

Read more of this content when you subscribe today.

ORA-12547: TNS:lost contact

[oracle@racnode1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Mon Jan 13 16:31:25 2020

Copyright (c) 1982, 2014, Oracle. All rights reserved.

ERROR:
ORA-12547: TNS:lost contact


Enter user-name:
ERROR:
ORA-12547: TNS:lost contact

Subscribe to get access

Read more of this content when you subscribe today.