Mani's Blog

December 14, 2012

Error: ORA-16751: failed to switchover to physical standby database. Both Oracle DB says Physical Standby

I ran the switchover to dataguard box.  I ran into issue where both database became physical standby.

Problem:

DGMGRL for Linux: Version 11.1.0.7.0 – 64bit Production

Copyright (c) 2000, 2005, Oracle. All rights reserved.

Welcome to DGMGRL, type “help” for information.
DGMGRL> connect sys/oracle
Connected.
DGMGRL> swtichover to xnodb01c
Unrecognized command “swtichover”, try “help”
DGMGRL> switchover to xnodb01c
Performing switchover NOW, please wait…
Error: ORA-16751: failed to switchover to physical standby database

Failed.
Unable to switchover, primary database is still “xnodb01cdg”

Solution:

Now I have to choose one database and promote as primary.   Following steps worked well for me.

SQL> select database_role from v$database;

DATABASE_ROLE
—————-
PHYSICAL STANDBY

SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup mount
ORACLE instance started.

Total System Global Area 2.6724E+10 bytes
Fixed Size            2160272 bytes
Variable Size         8321501552 bytes
Database Buffers     1.8254E+10 bytes
Redo Buffers          146423808 bytes
Database mounted.
SQL> alter database recover managed standby database cancel;
alter database recover managed standby database cancel
*
ERROR at line 1:
ORA-16136: Managed Standby Recovery not active

SQL> alter database recover managed standby database finish;

Database altered.

SQL> ALTER database commit to switchover to primary with session shutdown;

Database altered.

SQL> alter database open;

Database altered.

SQL>  select database_role from v$database;

DATABASE_ROLE
—————-
PRIMARY

SQL> alter system switch logfile;

System altered.

SQL>

Now other DB functioning as Physical Standby with no issues.

December 12, 2012

The installer has detected that you may have an Automatic Storage Management (ASM) instance improperly configured or one that was not properly cleaned up from a previous install.

Filed under: Database,Oracle — mani @ 4:58 pm

I ran in to this issue when I re-install Oracle Binaries during prerequisites checking.

Problem:

The installer has detected that you may have an Automatic Storage Management (ASM) instance improperly configured or one that was not properly cleaned up from a previous install.

Solution:

You may see the recommendation by Oracle:  You must completely remove the ASM instance by deleting the entry for it from the oratab file, or you must configure it properly by ensuring that the oratab file is updated to point to a valid Oracle Home where ASM is configured.

But sometimes you need to make sure environment variables are set and old home directory has been removed.

 

ORA-16055: FAL request rejected ARCH: FAL archive failed. Archiver continuing

Filed under: Database,Dataguard,Oracle — mani @ 5:35 am
Tags: , , ,

Problem:

We have environment with Primary and Active standby Dataguard.   For some reason storage on Dataguard is crashed and I had rebuild the Dataguard DB from backup.  Noticed Archive log is not being shipped and when I look at alert log  found following errors.

FAL[server, ARC3]: FAL archive failed, see trace file.
Errors in file /local/opt/oracle/diag/rdbms/xnodbcor/XNODBCOR1/trace/XNODBCOR1_arc3_2216.trc:
ORA-16055: FAL request rejected
ARCH: FAL archive failed. Archiver continuing
ORACLE Instance XNODBCOR1 – Archival Error. Archiver continuing.

Checked parameter values on archive_dest on primary DB and noticed “RESET” value assigned to ” log_archive_dest_state_1″

log_archive_dest_state_1         string     RESET

Solution:

Enabled the dest_1 as below, that fixed the problem.   Error disappeared and logs are being shipped and applied on Standby site.

SQL> alter system set log_archive_dest_state_1=enable;

System altered.

SQL> alter system switch logfile;

System altered.

SQL>

November 30, 2012

Oracle Grid control connects to incorrect/wrong Database / Oracle Grid pulls wrong metrics

Issue:

Grid control connects to Primary DB, but thinks it connected Physical standby and pull Standby metrics instead of Primary DB. This has started happening once I ran the switchover from DG to Primary DB.

Story behind the issue:

Here is the story behind the issue.  We had Database environment with Master(Primary) RAC and Physical Standby (RAC).   We had to convert our RAC Database into Single Instance DB and use local storage.  First I have converted Dataguard into Single Instance DB.   Switched the traffic to DG and looked good.   Converted Primary from RAC to single instance and migrated to local disk while it was Dataguard(standby).   Everything looks fine until this moment.    We decided to move the traffic back to original Primary and we did.   Everything was working well.

Effort to fix the issue.

However from this point grid control started showing DG performance metrics on both Primary and standby.  It was so weird.  I did stopped the agent,  ran clearstate, started the agent,  Did agent syncronization.  Nothing worked.    Infact I uninstalled agent on both Primary and standby then Install the agent only on Primary DB box.   I am shocked to see DG metrics and the grid control thinks it is physical standby.

At this point I was looking at primary Database’s agent log.   I noticed listener services listens to remote listener service.  which is pointing to DG box.

Solution:

Ran “lsnrctl services” to find out remote registry on Dataguard box.  It is was missed out during the conversion of RAC to single instance process.

Removed/unregister remote listener as below.

alter system set remote_listener=” scope=both;

Ran “lsnrctl services” to confirm the remote registry is gone.

Now the grid talking and pulling metrics against correct DB.

October 10, 2012

Oracle ASM disk creation failed. Marking disk “%s” as an ASM disk: [FAILED]

Filed under: ASM,Database,Oracle — mani @ 10:21 pm

We have a RAC database, we are converting to Non-RAC (Stand-alone) DB, and also need to migrate data off SAN storage.  I ran into this error while I am creating ASM disk ( after adding 2 new local disks).

Here is the error I ran into.

# /etc/init.d/oracleasm createdisk asm_ssd1 /dev/sdz1
Marking disk “asm_ssd1” as an ASM disk:                    [FAILED]

As soon the disk has been added to system, is configured via multipath and created device under /dev/mapper.  As I don’t want to use via multipath device, so what I did was.

1. Grabbed device id by ls -lrt /dev/disk/by-id

2. Added this device id in backlist block of /etc/multipath.conf file

devnode_blacklist {
wwid 3600508e000000000d02538430ed9c60b
wwid 35000c5003192cd97
wwid 35000c500333728e3
devnode “^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*”
devnode “^hd[a-z]”
devnode “^cciss!c[0-9]d[0-9]*”
}

3. run /etc/init.d/multipathd restart

Now I am able to create ASM disk properly.

# /etc/init.d/oracleasm createdisk asm_ssd1 /dev/sdz1
Marking disk “asm_ssd1” as an ASM disk:                    [  OK  ]
# /etc/init.d/oracleasm createdisk asm_ssd2 /dev/sdaa1
Marking disk “asm_ssd2” as an ASM disk:                    [  OK  ]

Now I can see newly added disks.
# /etc/init.d/oracleasm listdisks
ASM_DATA01
ASM_DATA02
ASM_REDO01
ASM_SSD1
ASM_SSD2

October 8, 2012

ORA-16789: standby redo logs not configured

Filed under: Data Guard Broker,Database,Dataguard,Oracle — mani @ 9:42 pm
Tags: ,

In our environment, Dataguard Manager log was showing following errors.

DG 2012-10-08-21:23:11        0 2 0 RSM0: HEALTH CHECK WARNING: ORA-16789: standby redo logs not configured
DG 2012-10-08-21:23:11        0 2 760559130 Operation CTL_GET_STATUS continuing with warning, status = ORA-16789
DG 2012-10-08-21:23:11        0 2 760559130 Operation CTL_GET_STATUS continuing with warning, status = ORA-16809
DG 2012-10-08-21:24:11        0 2 0 RSM Error: LogXptMode value ‘ASYNC’ of requires this database to have status redo logs, but they are not configured.

Show Configuration reports warning “ORA-16608: one or more databases have warnings”

DGMGRL> show configuration

Configuration
Name:                aclu1c
Enabled:             YES
Protection Mode:     MaxPerformance
Databases:
xnodb01c   – Primary database
xnodb01cdg – Physical standby database

Fast-Start Failover: DISABLED

Current status for “aclu1c”:
Warning: ORA-16608: one or more databases have warnings

show database verbose xnodb01cdg  returned success.

show database verbose xnodb01c returned errors, so I am sure the primary DB has issue.
Solution: is to add standby log files to primary DB.

SQL> ALTER DATABASE ADD STANDBY LOGFILE GROUP 31 ( ‘+XN_DATA’) SIZE 209715200 reuse;

Database altered.

SQL> ALTER DATABASE ADD STANDBY LOGFILE GROUP 32 ( ‘+XN_DATA’) SIZE 209715200 reuse;

Database altered.

SQL> ALTER DATABASE ADD STANDBY LOGFILE GROUP 33 ( ‘+XN_DATA’) SIZE 209715200 reuse;

Database altered.

SQL> ALTER DATABASE ADD STANDBY LOGFILE GROUP 34 ( ‘+XN_DATA’) SIZE 209715200 reuse;

Database altered.

SQL> ALTER DATABASE ADD STANDBY LOGFILE GROUP 35 ( ‘+XN_DATA’) SIZE 209715200 reuse;

Database altered.

Problem solved:

After adding all above standby log files,   Dataguard log stopped reporting the error.  Also dgmgrl stopped reporting the warnings.

DGMGRL> show configuration

Configuration
Name:                aclu1c
Enabled:             YES
Protection Mode:     MaxPerformance
Databases:
xnodb01c   – Primary database
xnodb01cdg – Physical standby database

Fast-Start Failover: DISABLED

Current status for “aclu1c”:
SUCCESS

DGMGRL>

 

September 20, 2012

ORA-16792: configurable property value is inconsistent with database setting

I have DG broker configured on stand-alone database servers (primary and standby).  After I migrated standby database to new ASM diskgroup I ran into issue “Warning: ORA-16608: one or more databases have warnings” and “ORA-16792: configurable property value is inconsistent with database setting”

Restarting the Data guard broker fixed the issue.

DGMGRL> show configuration

Configuration
Name:                contact
Enabled:             YES
Protection Mode:     MaxPerformance
Databases:
xnodbcntdg – Primary database
xnodbcnt   – Physical standby database

Fast-Start Failover: DISABLED

Current status for “contact”:
Warning: ORA-16608: one or more databases have warnings

DGMGRL> show database xnodbcnt

Database
Name:            xnodbcnt
Role:            PHYSICAL STANDBY
Enabled:         YES
Intended State:  APPLY-ON
Instance(s):
XNODBCNT

Current status for “xnodbcnt”:
Warning: ORA-16792: configurable property value is inconsistent with database setting

I just restarted as below.

 

SQL> alter system set dg_broker_start=false scope=both;

System altered.

SQL> alter system set dg_broker_start=true scope=both;

System altered.

SQL>

Then the issue is gone.

 

DGMGRL> show database xnodbcnt

Database
Name:            xnodbcnt
Role:            PHYSICAL STANDBY
Enabled:         YES
Intended State:  APPLY-ON
Instance(s):
XNODBCNT

Current status for “xnodbcnt”:
SUCCESS

DGMGRL>

September 17, 2012

Warning: ORA-16607: one or more databases have failed Error: ORA-16816: incorrect database role

Filed under: Data Guard Broker,Database,Dataguard,Oracle — mani @ 10:01 pm
Tags:

I have Data guard broker configured on my Primary and Standby database servers.  For some reason we had to failover using SQL commnad (didn’t use DG manager).  Later on dgmgrl shows errors as below.

DGMGRL> show configuration

Configuration
Name:                CONTACT
Enabled:             YES
Protection Mode:     MaxPerformance
Databases:
XNODBCNT   – Primary database
XNODBCNTDG – Physical standby database

Fast-Start Failover: DISABLED

Current status for “CONTACT”:
Warning: ORA-16607: one or more databases have failed

DGMGRL> show database ‘XNODBCNT’

Database
Name:            XNODBCNT
Role:            PRIMARY
Enabled:         YES
Intended State:  TRANSPORT-ON
Instance(s):
XNODBCNT

Current status for “XNODBCNT”:
Error: ORA-16816: incorrect database role

Solution: I tried many thing like enabling/disabling configuration, database and other nothing helped,  I had reconfigure it again as below to make DG broker to work properly.

DGMGRL> create configuration CONTACT as primary database is XNODBCNTDG connect identifier is xnodbcntdg;
Configuration “contact” created with primary database “xnodbcntdg”
DGMGRL> add database XNODBCNT as connect identifier is xnodbcnt maintained as physical;
Database “xnodbcnt” added
DGMGRL> show configuration;

Configuration
Name:                contact
Enabled:             NO
Protection Mode:     MaxPerformance
Databases:
xnodbcntdg – Primary database
xnodbcnt   – Physical standby database

Fast-Start Failover: DISABLED

Current status for “contact”:
DISABLED

DGMGRL> enable configuration
Enabled.
DGMGRL> show configuration verbose

Configuration
Name:                contact
Enabled:             YES
Protection Mode:     MaxPerformance
Databases:
xnodbcntdg – Primary database
xnodbcnt   – Physical standby database

Fast-Start Failover: DISABLED

Current status for “contact”:
Warning: ORA-16610: command “EDIT DATABASE xnodbcnt SET PROPERTY” in progress

DGMGRL> show configuration verbose

Configuration
Name:                contact
Enabled:             YES
Protection Mode:     MaxPerformance
Databases:
xnodbcntdg – Primary database
xnodbcnt   – Physical standby database

Fast-Start Failover: DISABLED

Current status for “contact”:
SUCCESS

DGMGRL>

August 16, 2012

Oracle Startup fails with ORA-27102: out of memory Solaris-AMD64 Error: 22: Invalid argument

Filed under: Database,Oracle — mani @ 9:51 pm

Issue: Oracle startup on Solaris 10G failed with following errors.

oracle@xnodb916:~$ dba

SQL*Plus: Release 10.2.0.3.0 – Production on Thu Aug 16 21:33:24 2012

Copyright (c) 1982, 2006, Oracle.  All Rights Reserved.

Connected to an idle instance.

SQL> startup nomount pfile=/local/opt/oracle/product/10.2.0.3/db5/dbs/initDSP0001.ora
ORA-27102: out of memory
Solaris-AMD64 Error: 22: Invalid argument

Solution:

*Solution:*   This error represents that max-shm-memory is not sized correctly.  Either reduce Oracle memory usage on appropriate parameters.  (OR) Increase max-shm-memory
In increase max-shm-memory, run following command as root:  Change the memory value based on your memory availability.
root@xnodb916:~# prctl -n project.max-shm-memory -v 25gb -r -i project user.oracle
root@xnodb916:~# prctl -n project.max-shm-memory -v 35gb -r -i project user.oracle

Reducing memory allocation on init***.ora fixed the issue.

Reduced shared_pool_size, pga_aggregate_target and db_cache_size.  Then started the DB successfully.

SQL> startup nomount pfile=/local/opt/oracle/product/10.2.0.3/db5/dbs/initDSP0001.ora
ORACLE instance started.

Total System Global Area 1.2046E+10 bytes
Fixed Size                  2191376 bytes
Variable Size             731680752 bytes
Database Buffers         1.1308E+10 bytes
Redo Buffers                4325376 bytes
SQL>

If AMM is enabled then SGA_TARGET and SGA_MAX_SIZE need to be reduced.

“RMAN-06053: unable to perform media recovery because of missing log” “ORA-01194: file 2 needs more recovery to be consistent”

Filed under: Backup,Database,Oracle,RMAN — mani @ 6:23 pm

Issue:  While restoring 10g database, ran into following issue
“ORA-01194: file 2 needs more recovery to be consistent”
“RMAN-06053: unable to perform media recovery because of missing log”

RMAN> run {
allocate channel d1 device type disk;
allocate channel d2 device type disk;
allocate channel d3 device type disk;
allocate channel d4 device type disk;
restore database;
switch datafile all;
switch tempfile all;
recover database;
}2> 3> 4> 5> 6> 7> 8> 9> 10>

allocated channel: d1
channel d1: sid=156 devtype=DISK

allocated channel: d2
channel d2: sid=155 devtype=DISK

allocated channel: d3
channel d3: sid=154 devtype=DISK

allocated channel: d4
channel d4: sid=153 devtype=DISK

Starting restore at 16-AUG-12

channel d1: starting datafile backupset restore
channel d1: specifying datafile(s) to restore from backup set
restoring datafile 00001 to /local/opt/oracle/oradata/XNODBGRD/system01.dbf
restoring datafile 00002 to /local/opt/oracle/oradata/XNODBGRD/undotbs01.dbf
restoring datafile 00003 to /local/opt/oracle/oradata/XNODBGRD/sysaux01.dbf
restoring datafile 00004 to /local/opt/oracle/oradata/XNODBGRD/users01.dbf
restoring datafile 00005 to /local/opt/oracle/oradata/XNODBGRD/mgmt.dbf
restoring datafile 00006 to /local/opt/oracle/oradata/XNODBGRD/mgmt_ecm_depot1.dbf
restoring datafile 00007 to /local/opt/oracle/oradata/XNODBGRD/tools01.dbf
channel d1: reading from backup piece /local/data/oracle/nfs/XNODBGRD/backupset/2012_08_15/o1_mf_nnndf_BKP_XNODBGRD_FULL_08_82pc74fq_.bkp

channel d1: restored backup piece 1
piece handle=/local/data/oracle/nfs/XNODBGRD/backupset/2012_08_15/o1_mf_nnndf_BKP_XNODBGRD_FULL_08_82pc74fq_.bkp tag=BKP_XNODBGRD_FULL_081512050003
channel d1: restore complete, elapsed time: 00:25:45
Finished restore at 16-AUG-12

Starting recover at 16-AUG-12

starting media recovery

Oracle Error:
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 2 needs more recovery to be consistent
ORA-01110: data file 2: ‘/local/opt/oracle/oradata/XNODBGRD/undotbs01.dbf’

released channel: d1
released channel: d2
released channel: d3
released channel: d4
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 08/16/2012 18:03:00
RMAN-06053: unable to perform media recovery because of missing log
RMAN-06025: no backup of log thread 1 seq 81113 lowscn 1817591460 found to restore

RMAN>

Solution:

Set the sequence until necessary and recover the database.

RMAN> run {
2> set until sequence 81112;
3> recover database;
4> }

Next Page »

Create a free website or blog at WordPress.com.