PMON (ospid: 123456): terminating the instance due to error 500

Oracle instance crashes with the following errors, sometimes the instance fails to start up with the same messages:

...
..
.
Errors in file /u02/app/oracle/diag/rdbms/ractest/RACTEST1/trace/RACTEST1_lreg_104781.trc  (incident=287152) (PDBNAME=CDB$ROOT):
ORA-07445: exception encountered: core dump [nscall()+911] [SIGSEGV] [ADDR:0x186] [PC:0x5C6F35F] [Address not mapped to object] []
Incident details in: /u02/app/oracle/diag/rdbms/ractest/RACTEST1/incident/incdir_287152/RACTEST1_lreg_104781_i287152.trc
...
..
.
Tue Aug 24 14:58:01 2021
Instance Critical Process (pid: 29, ospid: 104781, LREG) died unexpectedly
PMON (ospid: 123456): terminating the instance due to error 500
Tue Aug 24 14:58:01 2021
System state dump requested by (instance=1, osid=123456(PMON)), summary=[abnormal instance termination].
System State dumped to trace file /u02/app/oracle/diag/rdbms/ractest/RACTEST1/trace/RACTEST1_diag_104564_20210824145801.trc
Tue Aug 24 14:58:01 2021
Dumping diagnostic data in directory=[cdmp_20210824145801], requested by (instance=1, osid=123456(PMON)), summary=[abnormal instance termination].
Tue Aug 24 14:58:01 2021
Instance terminated by PMON, pid = 123456

The Call Stack Trace in the associated incident trace file ( RACTEST1_lreg_104781_i287152.trc ) shows :

nscall <- nsgrcOpen <- nsgrDo <- nsgrrg_Register <- kmmlrl <- ksucln <- ksbrdp <- opirip
  <- opidrv <- sou2o <- opimai_real <- ssthrdmain <- main
nscall()+911         signal   __sighandler()       2500000153 0F2408F7A
                                                   7F2275499F90 0F2408F7A ?
                                                   000000000 ? 000000000 ?
nsgrcOpen()+512      call     nscall()             2500000153 ? 7F2275499E30 ?
                                                   7F2275499F78 7F227549A0B0
                                                   7F227549A0C8 7F2275499F90
nsgrDo()+81          call     nsgrcOpen()          7F22753F2F30 7F2275499D78
                                                   7F2275499F78 ? 7F227549A0B0 ?
                                                   7F227549A0C8 ? 7F2275499F90 ?
nsgrrg_Register()+2  call     nsgrDo()             7F2275499D78 7F2275499D78 ?
70                                                 7F2275499F78 ? 7F227549A0B0 ?
                                                   7F227549A0C8 ? 7F2275499F90 ?
kmmlrl()+12395       call     nsgrrg_Register()    7F22753F2F30 000000008
                                                   7F2275499F78 ? 7F227549A0B0 ?
                                                   7F227549A0C8 ? 7F2275499F90 ?
kmlmain()+43         call     kmmlrl()             7F22753F2F30 ? 000000008 ?
                                                   7F2275499F78 ? 7F227549A0B0 ?
                                                   7F227549A0C8 ? 7F2275499F90 ?
ksbrdp()+1072        call     kmlmain()            06003EDA0 000000008 ?
                                                   7F2275499F78 ? 7F227549A0B0 ?
                                                   7F227549A0C8 ? 7F2275499F90 ?
opirip()+1488        call     ksbrdp()             06003EDA0 ? 000000008 ?
                                                   7F2275499F78 ? 7F227549A0B0 ?
                                                   7F227549A0C8 ? 7F2275499F90 ?
opidrv()+616         call     opirip()             000000032 000000004
                                                   7FFDA41BB698 7F227549A0B0 ?
                                                   7F227549A0C8 ? 7F2275499F90 ?
sou2o()+145          call     opidrv()             000000032 000000004
                                                   7FFDA41BB698 7F227549A0B0 ?
                                                   7F227549A0C8 ? 7F2275499F90 ?
opimai_real()+270    call     sou2o()              7FFDA41BB670 000000032
                                                   000000004 7FFDA41BB698
                                                   7F227549A0C8 ? 7F2275499F90 ?
ssthrdmain()+412     call     opimai_real()        000000000 7FFDA41BB980
                                                   000000004 ? 7FFDA41BB698 ?
                                                   7F227549A0C8 ? 7F2275499F90 ?
main()+236           call     ssthrdmain()         000000000 000000003

CAUSE and SOLUTION

Subscribe to get access

Read more of this content when you subscribe today.

Failed to Start Listener with TNS-01192 Error

In 19c GI environment, local listeners failed to start up with following errors:

[grid@racnode1 $ srvctl start listener -node racnode1
PRCR-1013 : Failed to start resource ora.LISTENER.lsnr
PRCR-1064 : Failed to start resource ora.LISTENER.lsnr on node racnode1
CRS-5016: Process "/u01/app/19.0.0.0/grid/bin/lsnrctl" spawned by agent "ORAAGENT" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/grid/diag/crs/racnode1/crs/trace/crsd_oraagent_grid.trc"
CRS-5016: Process "/u01/app/19.0.0.0/grid/bin/lsnrctl" spawned by agent "ORAAGENT" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/grid/diag/crs/racnode1/crs/trace/crsd_oraagent_grid.trc"
CRS-2674: Start of 'ora.LISTENER.lsnr' on 'racnode1' failed

Further check “crsd_oraagent_grid.trc” with following errors:

Starting /u01/app/19.0.0.0/grid/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 19.0.0.0.0 - Production
System parameter file is /u01/app/19.0.0.0/grid/network/admin/listener.ora
Log messages written to /u01/app/grid/diag/tnslsnr/racnode1/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
TNS-01192: Missing SID_LIST_ value left of equation for SID description in LISTENER.ORA

Listener failed to start. See the error message(s) above...

Subscribe to get access

Read more of this content when you subscribe today.