CRS-2674 CRS-2632 ORA-01013 CRS-5017 PRCR-1079 PRCD-1084 Srvctl Start Service

One or two services failed to start up, while the rest services are all good to be started up or shutdown.

$ srvctl start service -d TESTDB -s reports
PRCD-1084 : Failed to start service REPORTS
PRCR-1079 : Failed to start resource ora.testdb.reports.svc
CRS-5017: The resource action "ora.testdb.reports.svc start" 
          encountered the following error:
ORA-01013: user requested cancel of current operation
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/
                     racnode1/crs/trace/crsd_oraagent_oracle.trc".
CRS-2674: Start of 'ora.testdb.reports.svc' on 'racnode1' failed
CRS-2632: There are no more servers to try to place resource 
 'ora.testdb.reports.svc' on that would satisfy its placement policy

It seems the information about this database or service is inconsistent in OCR.  We can try the following steps one after another until the it is successful.

1)Remove database from OCR, and add it back again.

$srvctl remove database....

$srvctl add database ....

$ srvctl add instance ....

$ srvctl start service -d TESTDB -s reports

2) if not working in previous step, then restart all clusterware.

#crsctl stop crs

#ctsct start crs

$srvctl start service -d TESTDB -s reports

3) For some reason database outage is unavailable, service can be started up manually.

Start up service on instance 1. Users or applications can connect to database through this service, but srvctl still shows service not running

SQL> exec DBMS_SERVICE.START_SERVICE('REPORTS','TESTDB1');

$ srvctl status service -d testdb -s reports 

Service REPORTS is not running.

srvctl failed to start up local listener

SYMPTOMS

After server rebooted, all Oracle Grid Infrastructure ( GI ) resources are up running as expected except local listener. Try to manually start up local listener, then get the following error messages.

$ srvctl start listener -node racnode1
PRCR-1013 : Failed to start resource ora.LISTENER.lsnr
PRCR-1064 : Failed to start resource ora.LISTENER.lsnr on node racnode1
...
..
.
CRS-2674: Start of 'ora.LISTENER.lsnr' on 'racnode1' failed

Subscribe to get access

Read more of this content when you subscribe today.

CRS-2674 Clusterware Network Resource ora.net1.network Could not Start

Network configurations should be consistent between OS and Clusterware.

ISSUES

After network changes for public interface bond0 netmask from 255.255.255.0 to 255.255.254.0, clusterware network resource ora.net1.network can’t be started. Hence VIP,SCANS,local listeners, SCAN listeners, Services are down.

[+ASM1] grid@racnode1:/u01/app/11.2.0.4/grid/bin$ ./srvctl start nodeapps -n racnode1

PRCR-1013 : Failed to start resource ora.net1.network
PRCR-1064 : Failed to start resource ora.net1.network on node racnode1
CRS-2674: Start of 'ora.net1.network' on 'racnode1' failed

<GRID_HOME>/log/racnode1/crsd/crsd.log shows:

2015-01-29 17:25:02.552: [ CRSPE][1170397536]{1:2737:1255} CRS-2674: Start of 'ora.net1.network' on 'racnode1' failed
2015-01-29 17:25:02.552: [ CRSRPT][1172498784]{1:2737:1255} Published to EVM CRS_RESOURCE_STATE_CHANGE for ora.net1.network
2015-01-29 17:25:02.552: [UiServer][1172498784]{1:2737:1255} Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-2674: Start of 'ora.net1.network' on 'racnode1' failed]
MSGTYPE:
TextMessage[1]
OBJID:
TextMessage[ora.LISTENER_SCAN1.lsnr]
WAIT:
TextMessage[0]
]

From srvctl query :

[+ASM1] grid@racnode1:/u01/app/11.2.0.4/grid/log/racnode1$ srvctl config nodeapps
Network exists: 1/10.10.12.0/255.255.255.0/bond0, type static
VIP exists: /racnode1-vip/10.10.12.130/10.10.12.0/255.255.255.0/bond0, hosting node racnode1

GSD exists
ONS exists: Local port 6100, remote port 6200, EM port 2016
[+ASM1] grid@racnode1:/u01/app/11.2.0.4/grid/bin$ srvctl config nodeapps -a

Network exists: 1/10.10.12.0/255.255.255.0/bond0, type static
VIP exists: /racnode1-vip/10.10.12.130/10.10.12.0/255.255.255.0/bond0, hosting node racnode1
[+ASM1] grid@racnode1:~$ srvctl config network
 Network exists: 1/10.10.12.0/255.255.255.0/bond0, type static 

From ocrdump, the netmask is “255.255.255.0”

[+ASM1] grid@racnode1:/u01/app/11.2.0.4/grid/srvm$ ocrdump ~/myfile
 ... 
... 
... 
[DATABASE.NODEAPPS.racnode1.VIP.NETMASK] ORATEXT : 255.255.255.0
[+ASM1] grid@racnode1:~$ crsctl stat res ora.net1.network -f
NAME=ora.net1.network
TYPE=ora.network.type
....
....
USR_ORA_IF=bond0
USR_ORA_NETMASK=255.255.255.0
USR_ORA_SUBNET=10.10.12.0
VERSION=11.2.0.2.0

From ifconfig, the netmask has been changed to “255.255.254.0”.

[+ASM1] grid@racnode1:/u01/app/11.2.0.4/grid/log/racnode1$ /sbin/ifconfig -a
bond0 Link encap:Ethernet HWaddr 00:21:5E:97:22:D8
inet addr:10.10.12.120 Bcast:10.10.13.255 Mask:255.255.254.0

[+ASM1] grid@racnode1:~$ cat /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
ONBOOT=yes
BOOTPROTO=none
IPADDR=10.10.12.120
GATEWAY=10.10.12.1
NETMASK=255.255.254.0
USERCTL=no
TYPE=Ethernet

SOLUTION

In this situation, we need change OCR entry NETMASK to match the OS setting. This command takes several minutes, so be patient.
Once network resource is modified, node VIP netmask will be changed automatically. After this, clusterware network resource ora.net1.network ,VIP,SCANS,local listeners, SCAN listeners and Services are all up.

[root@racnode1 bin]# ./srvctl modify network -k 1 -S 10.10.12.0/255.255.254.0/bond0

Above command can also be used to modify subnet IP if it is set wrongly, for example:

# srvctl modify network -k 1 -S 10.10.12.0/255.255.254.0/bond0

if CRS is before 11.2.0.2, Run this command as root user as well:

# srvctl modify nodeapps -n racnode1 -A racnode1-vip/255.255.254.0/bond0

Use crsctl stat res -t to check all resources are up accordingly.

[+ASM1] grid@racnode1:/u01/app/11.2.0.4/grid/bin$ srvctl config nodeapps -a
Network exists: 1/10.10.12.0/255.255.254.0/bond0, type static
VIP exists: /racnode1-vip/10.10.12.130/10.10.12.0/255.255.254.0/bond0, hosting node racnode1
[+ASM1] grid@racnode1:/u01/app/11.2.0.4/grid/bin$ srvctl config network
Network exists: 1/10.10.12.0/255.255.254.0/bond0, type static

Please note:

If OCR entry is right, then network/system administrator only need modify NETMASK for bond0 physical interface in “/etc/sysconfig/network-scripts/ifcfg-bond0” to match OCR entries.

Reference :

How to Modify Public Network Information including VIP in Oracle Clusterware (Doc ID 276434.1)
CRS-2674 Clusterware Network Resource ora.net1.network Could not Start (Doc ID 1270186.1)