Category: ASM

ORA-00603 ORA-27504 ORA-27300 ORA-27301 ORA-27302 ORA-603 in ASM alert log file

There are following error messages in ASM alert log file:

Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_m000_39281.trc (incident=198012):
ORA-00603: ORACLE server session terminated by fatal error
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:sendmsg failed with status: 105
ORA-27301: OS failure message: No buffer space available
ORA-27302: failure occurred at: sskgxpsnd2
opidrv aborting process M000 ospid (39281) as a result of ORA-603
2019-03-12T01:00:18.919871+11:00
Process m000 died, see its trace file

+ASM1_m000_39281.trc:

...
..
.
SKGXP:[7f620a621930.0]{0}: SKGXPVFYNET: Socket self-test could not verify successful transmission of 32768 bytes (mtype 61).
SKGXP:[7f620a621930.1]{0}: The network is required to support UDP protocol sends of this size. Socket is bound to 169.254.174.228.
SKGXP:[7f620a621930.2]{0}: phase 'send', 0 tries, 100 loops, 13590 ms (last)
struct ksxpp * ksxppg_ [0x7f620a68a770, 0x7f6204ff1350) = 0x7f6204ff1348
...
..
.

Workaround

1) Shrink database instance SGA size to give more memory back to OS.
After OS gets more available memory, the issue is gone.

Physical memory utilisation( %):

Capture

OR

2) Oracle has reported this issue in Doc ID 2041723.1. by changing the MTU of the loopback interface,  and changing the value of kernel parameter min_free_kbytes.

a)  Lower MTU to 16436 by adding following to /etc/sysconfig/network-scripts/ifcfg-lo.

MTU=16436

Then restart the network service.

# systemctl restart network.service

b) Increase the value of vm.min_free_kbytes to 0.4%of the total physical memory of the server. This can be done by adding following to /etc/sysctl.conf.

  vm.min_free_kbytes = 2097152

and then run below command to make it effective:

#sysctl -p

PMON (ospid: nnnn): terminating the instance due to error 481

SYMPTOM

ASM failed to start up with below messages in ASM alert log:

Tue Oct 02 10:54:12 2018
ERROR: Network OS Ping failed to inst 1 on IP (169.254.255.151),
Tue Oct 02 10:54:55 2018
Instance Critical Process (pid: 11, ospid: 8470, LMON) died unexpectedly
PMON (ospid: 8443): terminating the instance due to error 481

Subscribe to get access

Read more of this content when you subscribe today.

Subscribe

Asmcmd Pwcopy With PRCD-1163 PRCR-1071 CRS-0245 ASMCMD-9453

After standby database rebuilt, then trying to copy database password file from primary to standby ASM disk, the below errors occurred:

ASMCMD> pwcopy --dbuniquename RACTEST '/tmp/orapwractest' '+DATA/RACTEST/orapwractest'
copying /tmp/orapwractest -> +DATA/RACTEST/orapwractest
PRCD-1163 : Failed to modify database RACTEST
PRCR-1071 : Failed to register or update resource ora.ractest.db
CRS-0245: User doesn't have enough privilege to perform the operation
ASMCMD-9453: failed to register password file as a CRS resource

The password file was copied to ASM diskgroup ‘+DATA’ successfully, but the configuring  of database failed .

WORKAROUND

Rerun the same command without “–dbuniquename RACTEST” option.

ASMCMD> pwcopy /tmp/orapwractest +DATA/RACTEST/orapwractest
copying /tmp/orapwractest -> +DATA/RACTEST/orapwractest

Then use ‘srvctl’ to re-configure database as user ‘oracle’ if needed:

$srvctl modify database -d RACTEST -pwfile '+DATA/RACTEST/orapwractest';

$srvctl config database -d RACTEST

How to Prevent ASM Diskgroup Automatic Mount

1) To disable certain ASM Diskgroup automatic mount on all RAC nodes, run below command as grid user:

$ srvctl disable diskgroup -diskgroup testdg

2) To disable certain ASM Diskgroup automatic mount on specific RAC node, run below command as grid user:

$ srvctl disable diskgroup -diskgroup testdg -node racnode1

3) To check the status of ASM diskgroup:

$ srvctl status diskgroup -diskgroup testdg -detail -verbose
Disk Group testdg is running on racnode1, racnode2
Disk Group testdg is disabled

Note: You cannot run the “srvctl start” command on a disabled object until you first re-enable the object