Mani's Blog

March 23, 2011

>ORA-29701: unable to connect to Cluster Manager After converting RAC into single instance.

Filed under: Oracle — mani @ 2:07 pm
Tags: , ,

>ORA-29701: unable to connect to Cluster Manager  After converting RAC into single instance.

I have converted RAC into single instance, then all looked fine.  I have moved the database to new diskgroup in order to decommission the older SAN storage.  All went well, now all DB files are located on new diskgroup created on SSD storage.   When we unpublish the LUNS the server got rebooted and couldn’t start ASM instance successfully.

Tried following solutions, finally it started working.

[oracle@m120319 dbs]$ dba

SQL*Plus: Release 11.1.0.7.0 – Production on Mon Mar 14 18:17:50 2011

Copyright (c) 1982, 2008, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup pfile=’/local/opt/oracle/product/11.1.0/db_1/dbs/init+ASM1.ora’;
ORA-29701: unable to connect to Cluster Manager
SQL>

1) Per recommendation from OTN changed the crs config as below, but didn’t help.

[root@m120319 ~]# /local/opt/oracle/product/11.1.0/db_1/bin/localconfig delete
Stopping Cluster Synchronization Services.
Unable to communicate with the Cluster Synchronization Services daemon.
Shutdown has begun. The daemons should exit soon.
Cleaning up Network socket directories
[root@m120319 ~]# /local/opt/oracle/product/11.1.0/db_1/bin/localconfig add
Successfully accumulated necessary OCR keys.
Creating OCR keys for user ‘root’, privgrp ‘root’..
Operation successful.
Configuration for local CSS has been initialized

Cleaning up Network socket directories
Setting up Network socket directories
Adding to inittab
Startup will be queued to init within 30 seconds.
Checking the status of new Oracle init process…
Expecting the CRS daemons to be up within 600 seconds.
Giving up: Oracle CSS stack appears NOT to be running.
Oracle CSS service would not start as installed
Automatic Storage Management(ASM) cannot be used until Oracle CSS service is started
[root@m120319 ~]#

2) Did cleanup CRS as below and started ./init.cssd run &manually

a. Run the rootdelete.sh script then the rootdeinstall.sh script from the
$ORA_CRS_HOME/install directory on any nodes you are removing CRS from. Running
these scripts should be sufficent to clean up your CRS install. Rootdelete.sh
accepts options like nosharedvar/sharedvar, and nosharedhome/sharedhome. Make
yourself familiar with these options by reading the Oracle Clusterware and
Oracle Real Application Clusters Administration and Deployment Guide.
If you have any problems with these scripts please open a service request.
If for some reason you have to manually remove the install due to problems
with the scripts, continue to step 2:
b. Stop the Nodeapps on all nodes:
srvctl stop nodeapps -n
c. Prevent CRS from starting when the node boots. To do this issue the following as root:
rm /etc/oracle/*
rm -f /etc/init.d/init.cssd
rm -f /etc/init.d/init.crs
rm -f /etc/init.d/init.crsd
rm -f /etc/init.d/init.evmd
rm -f /etc/rc2.d/K96init.crs
rm -f /etc/rc2.d/S96init.crs
rm -f /etc/rc3.d/K96init.crs
rm -f /etc/rc3.d/S96init.crs
rm -f /etc/rc5.d/K96init.crs
rm -f /etc/rc5.d/S96init.crs
rm -Rf /etc/oracle/scls_scr
rm -f /etc/inittab.crs
cp /etc/inittab.orig /etc/inittab
d. If they are not already down, kill off EVM, CRS, and CSS processes or reboot
the node:
ps -ef | grep crs
kill
ps -ef | grep evm
kill
ps -ef | grep css
kill
Do not kill any OS processes, for example icssvr_daemon process !
e. If there is no other Oracle software running (like listeners, DB’s, etc…),
you can remove the files in /var/tmp/.oracle or /tmp/.oracle. Example:
rm -f /var/tmp/.oracle/*
or
rm -f /tmp/.oracle/*

3) I had to start manually.

Finally restarted  as root:
# cd /etc/init.d

# ./init.cssd run &

4) Now ran the config again:
[root@m120319 oracle]# /local/opt/oracle/product/11.1.0/crs/bin/localconfig add
Successfully accumulated necessary OCR keys.
Creating OCR keys for user ‘root’, privgrp ‘root’..
Operation successful.
Configuration for local CSS has been initialized

Cleaning up Network socket directories
Setting up Network socket directories
Adding to inittab
Startup will be queued to init within 30 seconds.
Checking the status of new Oracle init process…
Expecting the CRS daemons to be up within 600 seconds.
Cluster Synchronization Services is active on these nodes.
m120319
Cluster Synchronization Services is active on all the nodes.
Oracle CSS service is installed and running under init(1M)
[root@m120319 oracle]#

Now I am able to start the ASM successfully.

[oracle@m120319 dbs]$ env | grep ORA
ORA_CRS_HOME=/local/opt/oracle/product/11.1.0/crs
ORACLE_SID=+ASM1
ORACLE_BASE=/local/opt/oracle
ORACLE_HOME=/local/opt/oracle/product/11.1.0/db_1
[oracle@m120319 dbs]$ dba

SQL*Plus: Release 11.1.0.7.0 – Production on Mon Mar 14 19:40:45 2011

Copyright (c) 1982, 2008, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup pfile=’/local/opt/oracle/product/11.1.0/db_1/dbs/init+ASM1.ora’
ASM instance started

Total System Global Area  283930624 bytes
Fixed Size            2158992 bytes
Variable Size          256605808 bytes
ASM Cache           25165824 bytes
ASM diskgroups mounted
SQL> select name ,state , free_mb, total_mb from v$asm_diskgroup;

NAME                   STATE          FREE_MB    TOTAL_MB
—————————— ———– ———- ———-
XN_DATA2               MOUNTED        17196      305250

SQL>

Advertisements

Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: