PMON (ospid: nnnn): terminating the instance due to error 481

SYMPTOM

ASM failed to start up with below messages in ASM alert log:

Tue Oct 02 10:54:12 2018
ERROR: Network OS Ping failed to inst 1 on IP (169.254.255.151),
Tue Oct 02 10:54:55 2018
Instance Critical Process (pid: 11, ospid: 8470, LMON) died unexpectedly
PMON (ospid: 8443): terminating the instance due to error 481

ANALYSE

To get interconnect private IPs for both nodes:

SQL> select * from gv$cluster_interconnects;

INST_ID NAME     IP_ADDRESS IS_  SOURCE  CON_ID
------- -------- --------------- ------- ----------
2       bond1:1  169.254.190.167 NO      0
1       bond1:1  169.254.255.151 NO      0

From node 2 to ping private IP of node 1 unsuccessfully:

[+ASM2] grid@racnode2:/home/grid$ ping 169.254.255.151
PING 169.254.255.151 (169.254.255.151) 56(84) bytes of data.
From 169.254.190.167 icmp_seq=1 Destination Host Unreachable
From 169.254.190.167 icmp_seq=5 Destination Host Unreachable

CAUSES

Routing table for “169.254.0.0” has been removed.

SOLUTION

Reboot the CRS

or

Add HAIP entry in routing table on node where “169.254.0.0” routing is missing:

# route add -net 169.254.0.0 netmask 255.255.0.0 dev bond1

grid@racnode1:~$ route -n |  netstat -nr
Kernel IP routing table
Destination   Gateway   Genmask       Flags Metric Ref    Use Iface
..
.
169.254.0.0   0.0.0.0   255.255.0.0     U     0      0    0  bond1
..
.

grid@racnode2:~$ route -n |  netstat -nr
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
..
.
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 bond1
..
.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.