Tag: ORA-00603: ORACLE server session terminated by fatal error

ORA-00600: internal error code, arguments: [17287]

ORA-00600 error occurs while startup 19c RAC database in ExaCC:

ERROR at line 1:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [17287], [0x7FA1EDB80130],
[0x22F3A9C18], [], [GSMADMIN_INTERNAL], [DBMS_GSM_CLOUDADMIN], [11], [1], [],
[], [], []
Process ID: 335484
Session ID: 2850 Serial number: 17388

INCIDENT TRACE FILE with CALL STACK

...
..
.
----- Incident Context Dump -----
Address: 0x7fa1f83fa060
Incident ID: 197141
Problem Key: ORA 600 [ORA-00600: internal error code, arguments: [17287], [0x7FA1EDB80130], [0x22F3A9C18], [], [GSMADMIN_INTERNAL], [DBMS_GS]
Error: ORA-600 [ORA-00600: internal error code, arguments: [17287], [0x7FA1EDB80130], [0x22F3A9C18], [], [GSMADMIN_INTERNAL], [DBMS_GSM_CLOUDADMIN], [11], [1], [], [], [], []
] [] [] [] [] [] [] [] [] [] [] []
[00]: dbgexProcessError [diag_dde]
[01]: dbgePostErrorKGE [diag_dde]
[02]: dbkePostKGE_kgsf [rdbms_dde]
[03]: kgeade []
[04]: kgeseml []
[05]: kgesem []
[06]: OCIKSEC []<-- Signaling
[07]: gwm_refresh_params []
[08]: gwm_init []
[09]: gwm_notifier []
[10]: gwm_pdb_notifier []
[11]: kscdnfy [VOS]
[12]: kpdbaSwitchOpenClose [PROGINT]
[13]: kpdbSwitchRunAsSysCbk [PROGINT]
[14]: rpiswu2 [RPI]
[15]: kpdbSwitch [PROGINT]
[16]: kpdbaOpenPdb [PROGINT]
[17]: kpdbaOpenClose [PROGINT]
[18]: kpdbaSwitchStateFromRoot [PROGINT]
[19]: kpdbRestorePdbStates_int [PROGINT]
[20]: kpdbRestorePdbStates [PROGINT]
[21]: kpdbCDBOpen [PROGINT]
[22]: adbdrv_options []
[23]: opiexe [Time_Limit]
[24]: opiosq0 [OPI]
[25]: kpooprx [PROGINT_MISC]
[26]: kpoal8 [PROGINT_MISC]
[27]: opiodr [OPI]
[28]: ttcpip []
[29]: opitsk [OPI]
[30]: opiino [OPI]
[31]: opiodr [OPI]
[32]: opidrv [OPI]
[33]: sou2o []
[34]: opimai_real [OPI]
[35]: ssthrdmain []
[36]: main []
...
..
.

CAUSE

User has run a couple of SQL scripts to have revoked some privileges from PUBLIC for security compliance purpose. Hence there are many invalid objects for SYS, PUBLIC, GSMADMIN_INTERNAL, etc.

SOLUTION

Subscribe to get access

Read more of this content when you subscribe today.

Subscribe

ORA-00603 ORA-27504 ORA-27300 ORA-27301 ORA-27302 ORA-603 in ASM alert log file

There are following error messages in ASM alert log file:

Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_m000_39281.trc (incident=198012):
ORA-00603: ORACLE server session terminated by fatal error
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:sendmsg failed with status: 105
ORA-27301: OS failure message: No buffer space available
ORA-27302: failure occurred at: sskgxpsnd2
opidrv aborting process M000 ospid (39281) as a result of ORA-603
2019-03-12T01:00:18.919871+11:00
Process m000 died, see its trace file

+ASM1_m000_39281.trc:

...
..
.
SKGXP:[7f620a621930.0]{0}: SKGXPVFYNET: Socket self-test could not verify successful transmission of 32768 bytes (mtype 61).
SKGXP:[7f620a621930.1]{0}: The network is required to support UDP protocol sends of this size. Socket is bound to 169.254.174.228.
SKGXP:[7f620a621930.2]{0}: phase 'send', 0 tries, 100 loops, 13590 ms (last)
struct ksxpp * ksxppg_ [0x7f620a68a770, 0x7f6204ff1350) = 0x7f6204ff1348
...
..
.

Workaround

1) Shrink database instance SGA size to give more memory back to OS.
After OS gets more available memory, the issue is gone.

Physical memory utilisation( %):

Capture

OR

2) Oracle has reported this issue in Doc ID 2041723.1. by changing the MTU of the loopback interface,  and changing the value of kernel parameter min_free_kbytes.

a)  Lower MTU to 16436 by adding following to /etc/sysconfig/network-scripts/ifcfg-lo.

MTU=16436

Then restart the network service.

# systemctl restart network.service

b) Increase the value of vm.min_free_kbytes to 0.4%of the total physical memory of the server. This can be done by adding following to /etc/sysctl.conf.

  vm.min_free_kbytes = 2097152

and then run below command to make it effective:

#sysctl -p