DOYENSYS Knowledge Portal




We Welcome you to visit our DOYENSYS KNOWLEDGE PORTAL : Doyensys Knowledge Portal




Wednesday, May 31, 2017

Clusterware Configuration issue Resolution


In 3 node rac setup. In the process of root.sh script execution in first 2 nodes were completed successfully.  But when we run the script in the 3'rd  we faced the issue "Failed to start Nodeapps" .


#1 Error Detail:

Using configuration parameter file: /CRS/app/11.2.0/grid_new/crs/install/crsconfig_params
User ignored Prerequisites during installation
PRKO-2190 : VIP exists for node testdb6, VIP name testdb6-vip
/CRS/app/11.2.0/grid_new/bin/srvctl start nodeapps -n testdb6 ... failed
Failed to start Nodeapps at /CRS/app/11.2.0/grid_new/crs/install/crsconfig_lib.pm line 9217.
[NEW root@testdb6 ~]#


Cause analysis:

#1 Check the Logfile:

Logfile : /CRS/app/11.2.0/grid_new/cfgtoollogs/crsconfig/rootcrs_testdb6.log

2017-05-31 00:12:51: starting nodeapps...
2017-05-31 00:12:51: DHCP_flag=0
2017-05-31 00:12:51: nodes_to_start=testdb6
2017-05-31 00:15:05: exit value of start nodeapps/vip is 1
2017-05-31 00:15:05: output for start nodeapps is  PRKO-2421 : Network resource is already started on node(s): testdb6 PRKO-2420 : VIP is already started on node(s): testdb6 PRCR-1013 : Failed to start resource ora.ons PRCR-1064 : Failed to start resource ora.ons on node testdb6 CRS-5014: Agent "/CRS/app/11.2.0/grid_new/bin/oraagent.bin" timed out


#2 Check Cluster Resource Status:

[NEW oracle@testdb6 crsconfig]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------

ora.ons
               ONLINE  ONLINE       testdb4
               ONLINE  ONLINE       testdb5
               OFFLINE  OFFLINE       testdb6        CHECK TIMED OUT



#3 Check the nodeapps status:

srvctl config nodeapps
Network exists: 1/10.80.1.0/255.255.255.0/eth0, type static
VIP exists: /testdb4-vip/10.80.1.72/10.80.1.0/255.255.255.0/eth0, hosting node testdb4
VIP exists: /testdb5-vip/10.80.1.65/10.80.1.0/255.255.255.0/eth0, hosting node testdb5
VIP exists: /testdb6-vip/10.80.1.165/10.80.1.0/255.255.255.0/eth0, hosting node testdb6
GSD exists
ONS exists: Local port 6100, remote port 6200, EM port 2016

#4 Check the connection status:

[NEW oracle@testdb6 ~]$ /CRS/app/11.2.0/grid/bin/onsctli ping

Communication error with the OPMN server local port.
Check the OPMN log files

^Cons is not running ...


Resolve this issue:

#1 Check the ons process:

[NEW oracle@testdb6 ~]$ ps -ef | grep ons
root      2770     1  0 May22 ?        00:00:27 sendmail: accepting connections
root      2909     1  0 May22 hvc0     00:00:00 /sbin/agetty serial_console 9600 vt100-nav
oracle   18960     1  0 May26 ?        00:00:00 /CRS/app/11.2.0/grid/opmn/bin/ons -d
oracle   18961 18960  0 May26 ?        00:05:55 /CRS/app/11.2.0/grid/opmn/bin/ons -d
oracle   20786 20701  0 01:23 pts/0    00:00:00 grep ons


#2 Kill the process manually:

[NEW oracle@testdb6 bin]$ kill -9 18960
[NEW oracle@testdb6 bin]$ kill -9 18961

#3 Check the ons process:

[NEW oracle@testdb6 bin]$ ps -ef | grep ons
root      2770     1  0 May22 ?        00:00:27 sendmail: accepting connections
root      2909     1  0 May22 hvc0     00:00:00 /sbin/agetty serial_console 9600 vt100-nav
oracle   20823 20701  0 01:26 pts/0    00:00:00 grep ons

Note : no ons process is running.

#4 Start the Ons Process:

[NEW oracle@testdb6 bin]$ ./onsctl start
CRS-2672: Attempting to start 'ora.net1.network' on 'testdb6'
CRS-2676: Start of 'ora.net1.network' on 'testdb6' succeeded
CRS-2679: Attempting to clean 'ora.ons' on 'testdb6'
CRS-2681: Clean of 'ora.ons' on 'testdb6' succeeded
CRS-2672: Attempting to start 'ora.ons' on 'testdb6'
CRS-2676: Start of 'ora.ons' on 'testdb6' succeeded

#5 Check the ons status:

[NEW oracle@testdb6 bin]$ ./onsctl ping
ons is running ...
[NEW oracle@testdb6 bin]$ exit


#6 Re-run the root script:

[NEW root@testdb6 crsconfig]# /CRS/app/11.2.0/grid_new/perl/bin/perl -I/CRS/app/11.2.0/grid_new/perl/lib -I/CRS/app/11.2.0/grid_new/crs/install /CRS/app/11.2.0/grid_new/crs/install/rootcrs.pl
Using configuration parameter file: /CRS/app/11.2.0/grid_new/crs/install/crsconfig_params
User ignored Prerequisites during installation
PRKO-2190 : VIP exists for node testdb6, VIP name testdb6-vip
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[NEW root@testdb6 crsconfig]#

#7 Check the resource status:

[NEW oracle@testdb6 crsconfig]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------

ora.ons
               ONLINE  ONLINE       testdb4
               ONLINE  ONLINE       testdb5
               ONLINE  ONLINE       testdb6


Reference:

In 11.2 and 12c, stopping ons fails. The rootupgrade.sh, "crsctl stop crs -f", and "srvctl stop nodeapps" fail because ons fails to stop. (Doc ID 2038830.1)

No comments: