ORA-00603 ORA-27504 ORA-27300 ORA-27301 ORA-27302 ORA-603 in ASM alert log file

There are following error messages in ASM alert log file:

Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_m000_39281.trc (incident=198012):
ORA-00603: ORACLE server session terminated by fatal error
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:sendmsg failed with status: 105
ORA-27301: OS failure message: No buffer space available
ORA-27302: failure occurred at: sskgxpsnd2
opidrv aborting process M000 ospid (39281) as a result of ORA-603
2019-03-12T01:00:18.919871+11:00
Process m000 died, see its trace file

+ASM1_m000_39281.trc:

...
..
.
SKGXP:[7f620a621930.0]{0}: SKGXPVFYNET: Socket self-test could not verify successful transmission of 32768 bytes (mtype 61).
SKGXP:[7f620a621930.1]{0}: The network is required to support UDP protocol sends of this size. Socket is bound to 169.254.174.228.
SKGXP:[7f620a621930.2]{0}: phase 'send', 0 tries, 100 loops, 13590 ms (last)
struct ksxpp * ksxppg_ [0x7f620a68a770, 0x7f6204ff1350) = 0x7f6204ff1348
...
..
.

Workaround

1) Shrink database instance SGA size to give more memory back to OS.
After OS gets more available memory, the issue is gone.

Physical memory utilisation( %):

Capture

OR

2) Oracle has reported this issue in Doc ID 2041723.1. by changing the MTU of the loopback interface,  and changing the value of kernel parameter min_free_kbytes.

a)  Lower MTU to 16436 by adding following to /etc/sysconfig/network-scripts/ifcfg-lo.

MTU=16436

Then restart the network service.

# systemctl restart network.service

b) Increase the value of vm.min_free_kbytes to 0.4%of the total physical memory of the server. This can be done by adding following to /etc/sysctl.conf.

  vm.min_free_kbytes = 2097152

and then run below command to make it effective:

#sysctl -p

ORA-27125: unable to create shared memory segment Linux-x86_64 Error: 28: No space left on device

When starting up an Oracle database, the below errors occur:

ORA-27125: unable to create shared memory segment
Linux-x86_64 Error: 28: No space left on device
Additional information: 3773
Additional information: 3221225472

Subscribe to get access

Read more of this content when you subscribe today.

ORA-27154 ORA-27300 ORA-27301 ORA-27302 from Instance Startup

Always make sure “kernel.sem = SEMMSL SEMMNS SEMOPM SEMMNI” set-up correctly to avoid issues.

The following errors occurred while starting up a 12.2.0.1 Oracle database instance.

SQL> startup

ORA-27154: post/wait create failed
ORA-27300: OS system dependent operation:semget failed with status: 28
ORA-27301: OS failure message: No space left on device
ORA-27302: failure occurred at: sskgpcreates

The error “operation:semget failed with status: 28” points out that it could be semaphore resource issue.

Oracle 12.2.0.1 doc recommended minimum value for semaphore are :

kernel.sem = SEMMSL SEMMNS SEMOPM SEMMNI
SEMMSL - max semaphores per array
SEMMNS - max semaphores system wide
SEMOPM - max ops per semop call
SEMMNI - max number of arrays

semmsl  =   250
semmns  = 32000
semopm  =   100
semmni  =   128

Check current system configuration for semaphores:

$ cat /proc/sys/kernel/sem
250 32000 100 200

$ ipcs -ls

------ Semaphore Limits --------
max semaphores per array   = 250
max semaphores system wide = 32000
max ops per semop call     = 100
max number of arrays       = 200
semaphore max value        = 32767

We can see SEMMNS is not right. it is supposed to be equal to SEMMSL*SEMMNI =250*200=50000.

If  SEMMNS  value too small, then we have to increase SEMMNI first, then adjust SEMMNS to SEMMSL*SEMMNI.

Set semaphore kernel parameter dynamically without the need for server reboot:

# sysctl -w kernel.sem="250 50000 100 200"
kernel.sem = 250 50000 100 200

-- Put into file
cat /etc/sysctl.conf | grep kernel.sem
kernel.sem = 250 50000 100 200

Or put into /etc/sysctl.conf file first, then

# sysctl -p

Then verify the current semaphore configurations:

# ipcs -ls

------ Semaphore Limits --------
max number of arrays       = 200
max semaphores per array   = 250
max semaphores system wide = 50000
max ops per semop call     = 100
semaphore max value        = 32767

Finally start up instance successfully.

SQL> startup
ORACLE instance started.

Total System Global Area 4294967296 bytes
Fixed Size 8801008 bytes
Variable Size 1073743120 bytes
Database Buffers 3187671040 bytes
Redo Buffers 24752128 bytes
Database mounted.
Database opened.