Monday, March 5, 2012

ORA-16572 DMON: cannot open configuration file

Hi,

This can be a interesting read.
Recently we created a Data Guard Configuration on a Non-RAC Database using cooked file system. We pro-actively started monitoring the DMON logs and the alert logs so that we can trace all the steps during this setup.

Issue:
We were getting the error in the DMON log when we set the below parameters.

alter system set dg_broker_config_file1='/oracle/oradata/CDEV/drCDEV_PRIMARY_01.dat' scope=both;
alter system set dg_broker_config_file2='/oracle/oradata/CDEV/drCDEV_PRIMARY_02.dat' scope=both;
alter system dg_broker_start=true;

Error in the DMON log was as below

DMON: >> Starting Data Guard Broker bootstrap <<
DMON: Attach state object
DMON: cannot open configuration file "/oracle/oradata/CDEV/drCDEV_PRIMARY_01.dat", retrying
DMON: cannot open configuration file "/oracle/oradata/CDEV/drCDEV_PRIMARY_01.dat"
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
DMON: Error opening /oracle/oradata/CDEV/drCDEV_PRIMARY_01.dat, error = ORA-16572
DMON: Establishing /oracle/oradata/CDEV/drCDEV_PRIMARY_02.dat as the most current file
DMON: cannot open configuration file "/oracle/oradata/CDEV/drCDEV_PRIMARY_02.dat", retrying
DMON: cannot open configuration file "/oracle/oradata/CDEV/drCDEV_PRIMARY_02.dat.dat"
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3

We got this error even on the standby when we set the above parameters

Troubleshooting

From the error it was obvious that DMON was looking for the dg_broker_config file and was unable to find at the physical level.
We were sure that this file was to be created Oracle as this was not a database using the raw volumes or block devices.
Hence we checked for permission on the location and also created a file, etc., but found that everything was fine.
We are tried changing the location of the files to various mount points just to check if it was issue at the mount point level but we got the same error.


Observation

We then decided to go ahead and create the configuration using DGMGRL

On Primary database
DGMGRL> CREATE CONFIGURATION 'DR_CDEV' AS
PRIMARY DATABASE IS 'CDEV_PRIMARY'
CONNECT IDENTIFIER IS CDEV_PRIMARY_DGMGRL;

Once this command was executed we found that the dg_broker_files got created on the Primary database.

But still the files were missing on the Standby database. Encouraged by the previous finding we went ahead and added the Standby database to the configuration and when we enabled the configuration we saw that the db_broker_files got created on the Standby as well.

On Primary Database
DGMGRL> enable configuration

Regards,
Tanveer

Lesson Learnt: PRO-ACTIVENESS sometimes can be DISASTROUS :)

Sunday, March 4, 2012

Find delay between Primary and Standby Database

Hi,

This is just a tip and a ready reckoner for me

To find Delay if both Standby and Primary in same timezone

select to_char(sysdate,'YYYY-MM-DD HH24:MI:SS') WHEN,': standby is ' || trim(to_char(1440 * (sysdate - max (next_time)),99999.99) ||' minutes behind') LAG from v$archived_log where applied ='YES';

To find Delay if Standby and Primary in different timezone

1. On Primary get time at OS level
$ date
Tue Feb 14 15:25:08 CET 2012 ------------------------- A

2. On Standby
sqlplus / as sysdba
SQL> alter session set nls_date_format='DD-MON-YY HH24:MI';
SQL> select dbtimezone from dual; - gives timezone for database
SQL> select max(next_time) from v$archived_log where applied='YES';
MAX(NEXT_TIME)
---------------
14-FEB-12 14:22 --------------------------------------- B

The delay is A-B
In this case around 1 hour.


Regards,
Tanveer