Day: February 2, 2015

CRS-2674 Clusterware Network Resource ora.net1.network Could not Start

Network configurations should be consistent between OS and Clusterware.

ISSUES

After network changes for public interface bond0 netmask from 255.255.255.0 to 255.255.254.0, clusterware network resource ora.net1.network can’t be started. Hence VIP,SCANS,local listeners, SCAN listeners, Services are down.

[+ASM1] grid@racnode1:/u01/app/11.2.0.4/grid/bin$ ./srvctl start nodeapps -n racnode1

PRCR-1013 : Failed to start resource ora.net1.network
PRCR-1064 : Failed to start resource ora.net1.network on node racnode1
CRS-2674: Start of 'ora.net1.network' on 'racnode1' failed

<GRID_HOME>/log/racnode1/crsd/crsd.log shows:

2015-01-29 17:25:02.552: [ CRSPE][1170397536]{1:2737:1255} CRS-2674: Start of 'ora.net1.network' on 'racnode1' failed
2015-01-29 17:25:02.552: [ CRSRPT][1172498784]{1:2737:1255} Published to EVM CRS_RESOURCE_STATE_CHANGE for ora.net1.network
2015-01-29 17:25:02.552: [UiServer][1172498784]{1:2737:1255} Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-2674: Start of 'ora.net1.network' on 'racnode1' failed]
MSGTYPE:
TextMessage[1]
OBJID:
TextMessage[ora.LISTENER_SCAN1.lsnr]
WAIT:
TextMessage[0]
]

From srvctl query :

[+ASM1] grid@racnode1:/u01/app/11.2.0.4/grid/log/racnode1$ srvctl config nodeapps
Network exists: 1/10.10.12.0/255.255.255.0/bond0, type static
VIP exists: /racnode1-vip/10.10.12.130/10.10.12.0/255.255.255.0/bond0, hosting node racnode1

GSD exists
ONS exists: Local port 6100, remote port 6200, EM port 2016
[+ASM1] grid@racnode1:/u01/app/11.2.0.4/grid/bin$ srvctl config nodeapps -a

Network exists: 1/10.10.12.0/255.255.255.0/bond0, type static
VIP exists: /racnode1-vip/10.10.12.130/10.10.12.0/255.255.255.0/bond0, hosting node racnode1
[+ASM1] grid@racnode1:~$ srvctl config network
 Network exists: 1/10.10.12.0/255.255.255.0/bond0, type static

From ocrdump, the netmask is “255.255.255.0” 

[+ASM1] grid@racnode1:/u01/app/11.2.0.4/grid/srvm$ ocrdump ~/myfile
 ... 
... 
... 
[DATABASE.NODEAPPS.racnode1.VIP.NETMASK] ORATEXT : 255.255.255.0
[+ASM1] grid@racnode1:~$ crsctl stat res ora.net1.network -f
NAME=ora.net1.network
TYPE=ora.network.type
....
....
USR_ORA_IF=bond0
USR_ORA_NETMASK=255.255.255.0
USR_ORA_SUBNET=10.10.12.0
VERSION=11.2.0.2.0

From ifconfig, the netmask has been changed to “255.255.254.0”.

[+ASM1] grid@racnode1:/u01/app/11.2.0.4/grid/log/racnode1$ /sbin/ifconfig -a
bond0 Link encap:Ethernet HWaddr 00:21:5E:97:22:D8
inet addr:10.10.12.120 Bcast:10.10.13.255 Mask:255.255.254.0

[+ASM1] grid@racnode1:~$ cat /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
ONBOOT=yes
BOOTPROTO=none
IPADDR=10.10.12.120
GATEWAY=10.10.12.1
NETMASK=255.255.254.0
USERCTL=no
TYPE=Ethernet

SOLUTION

In this situation, we need change OCR entry NETMASK to match the OS setting. This command takes several minutes, so be patient.
Once network resource is modified, node VIP netmask will be changed automatically. After this, clusterware network resource ora.net1.network ,VIP,SCANS,local listeners, SCAN listeners and Services are all up.

[root@racnode1 bin]# ./srvctl modify network -k 1 -S 10.10.12.0/255.255.254.0/bond0

Above command can also be used to modify subnet IP if it is set wrongly, for example:

# srvctl modify network -k 1 -S 10.10.12.0/255.255.254.0/bond0

if CRS is before 11.2.0.2, Run this command as root user as well:

# srvctl modify nodeapps -n racnode1 -A racnode1-vip/255.255.254.0/bond0

Use crsctl stat res -t to check all resources are up accordingly.

[+ASM1] grid@racnode1:/u01/app/11.2.0.4/grid/bin$ srvctl config nodeapps -a
Network exists: 1/10.10.12.0/255.255.254.0/bond0, type static
VIP exists: /racnode1-vip/10.10.12.130/10.10.12.0/255.255.254.0/bond0, hosting node racnode1
[+ASM1] grid@racnode1:/u01/app/11.2.0.4/grid/bin$ srvctl config network
Network exists: 1/10.10.12.0/255.255.254.0/bond0, type static

Please note:

If OCR entry is right, then network/system administrator only need modify NETMASK for bond0 physical interface in “/etc/sysconfig/network-scripts/ifcfg-bond0” to match OCR entries.

Reference :

How to Modify Public Network Information including VIP in Oracle Clusterware (Doc ID 276434.1)
CRS-2674 Clusterware Network Resource ora.net1.network Could not Start (Doc ID 1270186.1)

Advertisement