Tuesday, November 26, 2013

Time for GoodBye!

As most of you have seen, we have not been updating the blog frequently, with the last post being somewhere in the month of March 2013. So, it is decision time to either keep it alive or close it for good. I think it is time to say GoodBye.

At this moment, I would like to thank Tanveer for his contributions to this blog after I stopped writing regularly. I would also like to thank all the followers and readers of this blog for their patronage and encouraging words in the form of comments to various blog posts!

Once again, thanks everyone, GoodBye! Keep troubleshooting....

-Aravind Kamath Posral

Sunday, March 3, 2013

ORA-16057: server not in Data Guard configuration

Hi,

Recently when building a DG for a Production Database we faced an issue of the RFS process not starting on the DG side.

Whenever be build DG's, we add the DG parameters to the production during the regular cold backup window when the DB gets bounced and changes take effect. Later we copy the coldbackup to the DG server and build the DG.

In this case after the coldbackup was copied to the DG side and standby controlfile was used to bring up the DG, we were seeing that the RFS process was not starting.

The production for which we were building the DG was already having 2 more DG's.

When we looked at the alert logs on both Production and DG sides we found no specific errors. We did a couple of log switches on the production side but there we no errors recorded in the alert log. The files were getting shipped to the already built DG's but not to our new DG. We also checked whether the listeners were up and tns alias were reachable, but found no issues with it

As we had seen in the previous iterations, when RFS does not start at the DG, we used defer and enable the log_archive_dest_state_n parameter on the Production side and the RFS used to start.
As is the usual practice we tried to defer and enable the log_archive_dest_state_n parameter on the production side and then did a couple of log switches. And atlast we got the below error recorded in the alert log


Tue Feb 26 05:33:25 2013Errors in file /oracle/admin/DPRD2/diag/rdbms/dprd_primary/DPRD2/trace/DBPRD2_arc8_2890.trc:ORA-16057: server not in Data Guard configurationFAL[server, ARC8]: Error 16057 creating remote archivelog file 'DBPRD_STBY_2'FAL[server, ARC8]: FAL archive failed, see trace file.Errors in file /oracle/admin/DPRD2/diag/rdbms/dprd_primary/DPRD2/trace/DPRD2_arc8_2890.trc:ORA-16055: FAL request rejectedARCH: FAL archive failed. Archiver continuingORACLE Instance DPRD2 - Archival Error. Archiver continuing.

Once we got this error we came to know that there is a issue with the log_archive_config parameter that we have setup.

On looking closely we found that the parameter value in production and our new DG was same.
log_archive_config = 'DG_CONFIG=(DPRD,DPRD_STBY_1,DPRD_STBY_2)'

This prompted us to check the already setup DG's and check why they were not getting the error. On the already setup DG's we found the parameter was set as below
log_archive_config = 'DG_CONFIG=(DPRD_PRIMARY,DPRD_STBY_1)'

The DB_UNIQUE_NAME of our production was DPRD_PRIMARY.
Once we changed the parameter on our new DG site to
log_archive_config = 'DG_CONFIG=(DPRD_PRIMARY,DPRD_STBY_2)'
and bounced the database, the RFS process started and the log files began to ship to the DG site.

Learnings

The usual notion is that we should keep the log_archive_config parameter same in both production and DG site.

But as seen from the occurrence above, we find that the log_archive_config parameter is checked only on the standby side.

Unless your configuration is used for SWITCHOVER/FAILOVER this parameter set correctly or incorrectly on the production site does not matter

-- Tanveer Madan





Sunday, December 16, 2012

ORA-01113 - When Opening the Database

Hi,

I am updating post after posts today, just to make sure I update all the posts which I had to before we get into the new year

Few months back we were cloning a database, during which we had to re-create the database controlfile inorder to rename the database. After creating the controlfile when we tried to open the database with RESETLOGS it failed with below error

SQL> alter database open resetlogs;

alter database open resetlogs
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced

On checking the alert log we found that the UNDO tablespace that was used in our pfile was not of the same name as in the source database and hence alter database open resetlogs had failed

ORA-30012: undo tablespace 'APPS_UNDOTS1' does not exist or of wrong type
Tue Apr 03 10:01:24 PDT 2012
Error 30012 happened during db open, shutting down database
USER: terminating instance due to error 30012
Instance terminated by USER, pid = 16192
ORA-1092 signalled during: alter database open resetlogs

We then shutdown the database and made the changes to the pfile and mounted the database and tried to open the database using RESETLOGS


SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01139: RESETLOGS option only valid after an incomplete database recovery

We got the above error because before the command failed the last time it had done the below (as per the alert log)

1. Reset the log sequence number
2. Recreated the new redo log files as per the new controlfile created

and probably had also completed 

1. Updating the datafiles and online redologs with the new SCN

Hence we issued the alter database open command without RESETLOGS option and it also errored with below error

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01113: file 1 needs media recovery if it was restored from backup, or END
BACKUP if it was not
ORA-01110: data file 1: '/oracle/oradata/T2C/fs01/system_01.dbf'

First we thought that database copy was wrong but the initial behaviour of the "alter database open restlogs" command and also the sanity checks we had done before issuing the command made us sure that there was no issue with the copy

As the database had abruptly shutdown during the "alter database open resetlogs" we thought that the datafile system_01.dbf was inconsistent but we werre unsure how many other files were inconsistent hence we decided to issue the alter database recover command. Below was the result of it

SQL> startup mount
ORACLE instance started.

Total System Global Area 8589934592 bytes
Fixed Size 2110032 bytes
Variable Size 5771365808 bytes
Database Buffers 2734686208 bytes
Redo Buffers 81772544 bytes
Database mounted.
SQL> alter database recover ;

Database altered.

######## alert log excerpt for the alter database recover command ##########


alter database recover
Tue Apr 03 10:36:37 PDT 2012
Media Recovery Start
parallel recovery started with 3 processes
Tue Apr 03 10:36:57 PDT 2012
Recovery of Online Redo Log: Thread 1 Group 1 Seq 1 Reading mem 0
Mem# 0: /oracle/oraredo/TS2COE/fs01/redo_1_01.log
Tue Apr 03 10:36:57 PDT 2012
Media Recovery Complete (TS2COE1)
Completed: alter database recover

##############################################################################



Once the recovery was completed we tried to open the database and were successful.

SQL> select name,open_mode from v$database;

NAME OPEN_MODE
--------- ----------
T2C MOUNTED

SQL> alter database open;

Database altered.

-- Tanveer










ORA-00001: unique constraint (APPLSYS.FND_DATABASE_INSTANCES_U1) violated when running Autoconfig


Hi,

Issue:
Sometime back we had a activity to migrate the Front-end/Concurrent Managers hosts only to VMs and the database nodes remained the same.
As part of this activity we ran autoconfig on the database nodes, when running on the second database node we found that the apps.fnd_nodes table was not getting populated with the entries for the second node.
On checking the adconfig.log at location $ORACLE_HOME/appsutil/log/<CONTEXT_NAME>/<MMDDHHMI>, we found that the autoconfig failed to run two scripts afdbprf.sh and adcrobj.sh due to error "ORA-12154: TNS:could not resolve the connect identifier specified"

This prompted us to check if adgentns.pl has run properly and we found that adgentns.pl also has errored out with below error

java.sql.SQLException: ORA-00001: unique constraint (APPLSYS.FND_DATABASE_INSTANCES_U1) violated
ORA-06512: at "APPS.FND_APP_SYSTEM", line 507
ORA-06512: at "APPS.FND_NET_SERVICES", line 951
ORA-06512: at line 1

Troubleshooting:

We checked the index APPLSYS.FND_DATABASE_INSTANCES_U1 and found that it was for table APPLSYS.FND_DATABASES_INSTANCES.
We also checked the columns on which the index APPLSYS.FND_DATABASE_INSTANCES_U1 was built

SQL> select index_owner,index_name,table_owner,table_name,column_name from dba_ind_columns where index_name='FND_DATABASE_INSTANCES_U1';

INDEX_OWNER     INDEX_NAME   COLUMN_NAME
-------------   ----------------   --------------
APPLSYS         FND_DATABASE_INSTANCES_U1  DB_GUID
APPLSYS         FND_DATABASE_INSTANCES_U1  INSTANCE_NUMBER

This made us check the xml file on the second database node to to see if the instance number parameter (s_instNumber) was mentioned correctly, but we found that this parameter s_instNumber was having the same number as first database node.

Fix:

Once we made the corrections to the xml files on the database nodes to reflect the right value for s_instNumber, autoconfig ran successfully and also populated the apps.fnd_nodes table correctly

Afterthoughts:

All the xml files in the database nodes had s_instNumber set as first node because when earlier someone has performed the upgrade of database from 11.2.0.1 to 11.2.0.3 had copied the xml file from the first database node to other database nodes and just changed the host names only.

On closely examining the autoconfig logs run during the upgrade activity we found the same error that we had got when adgentns.pl had run.
But autoconfig had completed sucessfully with no errors as they had not truncated the apps.fnd_nodes table because the FE and database nodes has remained the same during upgrade.

-- Tanveer



ERROR: Failed to bind Launcher to port


Hi,

Recently we did a R12 cloning and faced an error when starting the forms.
The fix seemed trivial at the end but fixing it let me understand few things, which you may already be aware of.
But for those who do not know, this may be a good read

Issue: Forms fails to start with error "Failed to bind Launcher to port 9020"

Troubleshooting:
As our forms was congigured to run in SOCKET mode, we used the adformsrvctl.sh to start the forms.
On checking the socket.log we found the below error recorded

FORMS CONNECTION ACTIVITY LOG FILE
Developer:Forms/LogRecord
[Sun Dec  2 05:30:11 2012 EST]::Server Start-up Data:
        Server Log Filename: /apps/local/D3P/inst/apps/D3P_c-2/logs/ora/10.1.2/forms/socket.log
        Server Hostname: c-2
        Server Port: 9020
        Server Pool: 1
        Server Process Id: 27370
[Sun Dec  2 05:30:11 2012 EST]::ERROR: Failed to bind Launcher to port 9020

We did the initial checks to see if the port 9020 was in use by any other process on the host but found none.
As our environment was multi-tiered we tried starting the forms on another host but found the same issue.
This made us clear that the issue was with our configuration and not at hosting level

In order to try another option we wanted to start the forms on another port say 9040. Hence to accomplish this we edited the appsweb.cfg pointed by the variable $FORMS_WEB_CONFIG_FILE to reflect the new port.
But again when we started the forms service we got the same error and ironically in the log files we found that the forms was starting with the same old port 9020.

This prompted us to think that forms when starting in socket mode was not reading $FORMS_WEB_CONFIG_FILE to take the port number

Fix:
The issue got fixed when we checked the /etc/hosts and found that there was no entry added in it for the load balancer to point to the same host itself.
After making the required changes to the file /etc/hosts and reverting the port to 9020 in $FORMS_WEB_CONFIG_FILE, the forms server started on port 9020

For Eg:
127.0.0.1 localhost localhost.localdomain
173.38.5.72 c-2 c-2.abc.com
173.38.5.72 www-d3p www-d3p.abc.com

This entry is required in /etc/hosts as the load balancer will not detect the port to be up and running until the process is really started on that port.
Hence when starting the forms without the entry in /etc/hosts the request was hitting the load balancer but as the load balancer had not detected anything running on port 9020 it was unable to route the connection.

After adding this entry in the /etc/hosts when starting the forms, the start process was able to resolve www-d3p by bypassing the load balancer and reaching the host itself.

Afterthoughts:
After we were able to resolve the issue and also know the root cause for it. It was still striking us why did the forms server not start with the port 9040 even after editing $FORMS_WEB_CONFIG_FILE
Further reading of scripts revealed the below things

Forms in Socket Mode

1. The script adformsrvctl.sh has the port number and servername hardcoded in it and these values are used to start the forms server and it does not read the $FORMS_WEB_CONFIG_FILE. Incase you want to start the forms on another port, manually change port number in adformsrvctl.sh

2. The script adformsrvctl.sh sources the below env files while starting the forms server
   $APPL_TOP/APPS<CONTEXT_NAME>.env
   $ORA_CONFIG_HOME/10.1.2/forms/server/socket.env

3. $FORMS_WEB_CONFIG_FILE is used to set parameters for the client forms session during runtime. Incase you make any changes to the $FORMS_WEB_CONFIG_FILE then you need not bounce the forms service but just relaunch the forms session for changes to take effect

Forms in Servlet Mode

1. The script adformsctl.sh is used to start forms in SERVLET mode

2. The script adformsctl.sh sources the below env files while starting the forms server
   $APPL_TOP/APPS<CONTEXT_NAME>.env
   $ORA_CONFIG_HOME/10.1.3/<CONTEXT_NAME>.env
   $ORA_CONFIG_HOME/10.1.2/forms/server/default.env

3. The forms service starts using the parameters in the $FORMS_WEB_CONFIG_FILE to start and also at  runtime by the client forms. Incase you want to start the forms on another port, make a change to the $FORMS_WEB_CONFIG_FILE and bounce the services

4. If a change is made to $FORMS_WEB_CONFIG_FILE then forms need to be bounced for it to take effect at forms server and client forms level as well


-- Tanveer


Friday, June 22, 2012

ORA-27048: skgfifi: file header information is invalid

Thought of penning this post as I thought this issue was quite challenging. We were in the process of upgrading from 11.2.0.1 to 11.2.0.3 PSU 1. However, when we started the db with 11.2.0.3 code tree for upgrade we observed the following error:
startup upgrade;
In the alert log:
Errors in file /oracle/admin/TSECX/diag/rdbms/tsecx/TSECX1/trace/TSECX1_dbw0_9942.trc:
ORA-01157: cannot identify/lock data file 380 - see DBWR trace file
ORA-01110: data file 380: '/oracle/oradata/TSECX/fs01/apps_undots4_16.dbf'
ORA-27048: skgfifi: file header information is invalid

Fix:
I want to cut short the troubleshooting steps. We did too much of troubleshooting. During the troubleshooting we observed that db would come up cleanly with 11.2.0.1 code tree but with 11.2.0.3 it will throw the aforesaid error and about 7 files would show up in v$recover_File view.
We opened the db with 11.2.0.1 code tree and then resized the 7 data files to add about 250MB space to each of the 7 files. Shutdown immediate. Then started the db upgrade with 11.2.0.3 code tree and it worked!
- Aravind Kamath Posral

Monday, March 5, 2012

ORA-16572 DMON: cannot open configuration file

Hi,

This can be a interesting read.
Recently we created a Data Guard Configuration on a Non-RAC Database using cooked file system. We pro-actively started monitoring the DMON logs and the alert logs so that we can trace all the steps during this setup.

Issue:
We were getting the error in the DMON log when we set the below parameters.

alter system set dg_broker_config_file1='/oracle/oradata/CDEV/drCDEV_PRIMARY_01.dat' scope=both;
alter system set dg_broker_config_file2='/oracle/oradata/CDEV/drCDEV_PRIMARY_02.dat' scope=both;
alter system dg_broker_start=true;

Error in the DMON log was as below

DMON: >> Starting Data Guard Broker bootstrap <<
DMON: Attach state object
DMON: cannot open configuration file "/oracle/oradata/CDEV/drCDEV_PRIMARY_01.dat", retrying
DMON: cannot open configuration file "/oracle/oradata/CDEV/drCDEV_PRIMARY_01.dat"
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
DMON: Error opening /oracle/oradata/CDEV/drCDEV_PRIMARY_01.dat, error = ORA-16572
DMON: Establishing /oracle/oradata/CDEV/drCDEV_PRIMARY_02.dat as the most current file
DMON: cannot open configuration file "/oracle/oradata/CDEV/drCDEV_PRIMARY_02.dat", retrying
DMON: cannot open configuration file "/oracle/oradata/CDEV/drCDEV_PRIMARY_02.dat.dat"
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3

We got this error even on the standby when we set the above parameters

Troubleshooting

From the error it was obvious that DMON was looking for the dg_broker_config file and was unable to find at the physical level.
We were sure that this file was to be created Oracle as this was not a database using the raw volumes or block devices.
Hence we checked for permission on the location and also created a file, etc., but found that everything was fine.
We are tried changing the location of the files to various mount points just to check if it was issue at the mount point level but we got the same error.


Observation

We then decided to go ahead and create the configuration using DGMGRL

On Primary database
DGMGRL> CREATE CONFIGURATION 'DR_CDEV' AS
PRIMARY DATABASE IS 'CDEV_PRIMARY'
CONNECT IDENTIFIER IS CDEV_PRIMARY_DGMGRL;

Once this command was executed we found that the dg_broker_files got created on the Primary database.

But still the files were missing on the Standby database. Encouraged by the previous finding we went ahead and added the Standby database to the configuration and when we enabled the configuration we saw that the db_broker_files got created on the Standby as well.

On Primary Database
DGMGRL> enable configuration

Regards,
Tanveer

Lesson Learnt: PRO-ACTIVENESS sometimes can be DISASTROUS :)

Sunday, March 4, 2012

Find delay between Primary and Standby Database

Hi,

This is just a tip and a ready reckoner for me

To find Delay if both Standby and Primary in same timezone

select to_char(sysdate,'YYYY-MM-DD HH24:MI:SS') WHEN,': standby is ' || trim(to_char(1440 * (sysdate - max (next_time)),99999.99) ||' minutes behind') LAG from v$archived_log where applied ='YES';

To find Delay if Standby and Primary in different timezone

1. On Primary get time at OS level
$ date
Tue Feb 14 15:25:08 CET 2012 ------------------------- A

2. On Standby
sqlplus / as sysdba
SQL> alter session set nls_date_format='DD-MON-YY HH24:MI';
SQL> select dbtimezone from dual; - gives timezone for database
SQL> select max(next_time) from v$archived_log where applied='YES';
MAX(NEXT_TIME)
---------------
14-FEB-12 14:22 --------------------------------------- B

The delay is A-B
In this case around 1 hour.


Regards,
Tanveer

Wednesday, February 29, 2012

ORA-12514 When doing Switchover to STANDBY Database

Hi,

Today we faced issue when switching over the PRIMARY database to STANDBY database we faced the below issue.

Issue:

Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys/manager
Connected.
DGMGRL> switchover to 'CV_S';
Performing switchover NOW, please wait...
Operation requires shutdown of instance "CV" on database "CV_P"
Shutting down instance "CV"...
ORA-01109: database not open

Database dismounted.
ORACLE instance shut down.
Operation requires shutdown of instance "CV" on database "CV_S"
Shutting down instance "CV"...
ORA-01109: database not open

Database dismounted.
ORACLE instance shut down.
Operation requires startup of instance "CV" on database "CV_P"
Starting instance "CV"...
Unable to connect to database
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

Failed.
You are no longer connected to ORACLE
Please connect again.
Unable to start instance "CV"
You must start instance "CV" manually
Operation requires startup of instance "CV" on database "CV_S"
You must start instance "CV" manually
Switchover succeeded, new primary is "CV_S"
DGMGRL>

Troubleshooting:

Metalink and other notes pointed out that the issue will occur if we miss adding the parameter GLOBAL_DBNAME = db_unique_name_DGMGRL to the listener on the primary and standby database as the DMON uses service db_unique_name_DGMGRL when starting the databases during switchover.

But when we checked the listener configuration we found that desired parameter was added and also configured in the listener

Primary Database
------------------
LISTENER_DG_CV =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(Host = db-d6.abc.com)(Port = 1530)(ip = first))
)


SID_LIST_LISTENER_DG_CV_db-d6 =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = CV)
(SDU = 32768)
(TDU = 32768)
(ORACLE_HOME = /oracle/10.2.0.4/CV)
(SID_NAME = CV)
)
(SID_DESC =
(GLOBAL_DBNAME = CV_P_DGMGRL)
(SDU = 32768)
(TDU = 32768)
(ORACLE_HOME = /oracle/10.2.0.4/CV)
(SID_NAME = CV)
)
)

Standby Database
------------------
LISTENER_DG_CV =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(Host = db2-d6.abc.com)(Port = 1530)(ip = first))
)


SID_LIST_LISTENER_DG_CV_db-d6 =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = CV)
(SDU = 32768)
(TDU = 32768)
(ORACLE_HOME = /oracle/10.2.0.4/CV)
(SID_NAME = CV)
)
(SID_DESC =
(GLOBAL_DBNAME = CV_S_DGMGRL)
(SDU = 32768)
(TDU = 32768)
(ORACLE_HOME = /oracle/10.2.0.4/CV)
(SID_NAME = CV)
)
)

Also the listener had registered the service name

Primary Side
-------------
$ lsnrctl status LISTENER_DG_CV_db-d6

LSNRCTL for Linux: Version 10.2.0.4.0 - Production on 29-FEB-2012 09:53:13

Copyright (c) 1991, 2007, Oracle. All rights reserved.

Connecting to (ADDRESS=(PROTOCOL=TCP)(Host=db-d6.abc.com)(Port=1530)(ip=first))
STATUS of the LISTENER
------------------------
Alias LISTENER_DG_CV_db-d6
Version TNSLSNR for Linux: Version 10.2.0.4.0 - Production
Start Date 28-FEB-2012 12:08:48
Uptime 0 days 21 hr. 44 min. 24 sec
Trace Level off
Security ON: Local OS Authentication
SNMP ON
Listener Parameter File /etc/listener.ora
Listener Log File /oracle/10.2.0.4/CV/network/log/listener_dg_cv_db-d6.log
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=173.38.0.22)(PORT=1530)))
Services Summary...
Service "CV" has 1 instance(s).
Instance "CV", status UNKNOWN, has 1 handler(s) for this service...
Service "CV_P_DGMGRL" has 1 instance(s).
Instance "CV", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully

Standby Side
-------------
$ lsnrctl status LISTENER_DG_CV_db2-d6

LSNRCTL for Linux: Version 10.2.0.4.0 - Production on 29-FEB-2012 09:53:13

Copyright (c) 1991, 2007, Oracle. All rights reserved.

Connecting to (ADDRESS=(PROTOCOL=TCP)(Host=db2-d6.abc.com)(Port=1530)(ip=first))
STATUS of the LISTENER
------------------------
Alias LISTENER_DG_CV_db2-d6
Version TNSLSNR for Linux: Version 10.2.0.4.0 - Production
Start Date 28-FEB-2012 12:08:48
Uptime 0 days 21 hr. 44 min. 24 sec
Trace Level off
Security ON: Local OS Authentication
SNMP ON
Listener Parameter File /etc/listener.ora
Listener Log File /oracle/10.2.0.4/CV/network/log/listener_dg_cv_db-d6.log
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=173.38.0.22)(PORT=1530)))
Services Summary...
Service "CV" has 1 instance(s).
Instance "CV", status UNKNOWN, has 1 handler(s) for this service...
Service "CV_S_DGMGRL" has 1 instance(s).
Instance "CV", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully

Further when ever we tried making the sqlplus session (sqlplus system/abc@CV_P_DGMGRL or sqlplus system/abc@CV_S_DGMGRL) from standby to primary or vice-versa it went perfectly fine and had no issues.

Also we had other databases in our system which had the same configuration and the switchover was working fine. When we started to compare those databases with this database having the issue we found that the parameter db_domain was set here to abc.com

It seemed like the DMON was trying to connect to service db_unique_name_DGMGRL.abc.com (db_unique_name_DGMGRL.domain.com) in the connect descriptor of the tns names it was using to start the database. It made me think that DMON constructed its own tnentry to start the database and did not use the tns entry we had made in the tnsnames.ora


Solution:

In our case
db_unique_name = CV_P on primary database
db_unique_name = CV_S on standby database

To make the failover successful we re-registered the listener with service name db_unique_name_DGMGRL.abc.com by editing the listener file as below and reloading it.

Primary Database
------------------
LISTENER_DG_CV =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(Host = db-d6.abc.com)(Port = 1530)(ip = first))
)


SID_LIST_LISTENER_DG_CV_db-d6 =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = CV.abc.com)
(SDU = 32768)
(TDU = 32768)
(ORACLE_HOME = /oracle/10.2.0.4/CV)
(SID_NAME = CV)
)
(SID_DESC =
(GLOBAL_DBNAME = CV_P_DGMGRL.abc.com)
(SDU = 32768)
(TDU = 32768)
(ORACLE_HOME = /oracle/10.2.0.4/CV)
(SID_NAME = CV)
)
)

$ lsnrctl reload LISTENER_DG_CV

$ lsnrctl status LISTENER_DG_CV_db-d6

LSNRCTL for Linux: Version 10.2.0.4.0 - Production on 29-FEB-2012 09:53:13

Copyright (c) 1991, 2007, Oracle. All rights reserved.

Connecting to (ADDRESS=(PROTOCOL=TCP)(Host=db-d6.abc.com)(Port=1530)(ip=first))
STATUS of the LISTENER
------------------------
Alias LISTENER_DG_CV_db-d6
Version TNSLSNR for Linux: Version 10.2.0.4.0 - Production
Start Date 28-FEB-2012 12:08:48
Uptime 0 days 21 hr. 44 min. 24 sec
Trace Level off
Security ON: Local OS Authentication
SNMP ON
Listener Parameter File /etc/listener.ora
Listener Log File /oracle/10.2.0.4/CV/network/log/listener_dg_cv_db-d6.log
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=173.38.0.22)(PORT=1530)))
Services Summary...
Service "CV.abc.com" has 1 instance(s).
Instance "CV", status UNKNOWN, has 1 handler(s) for this service...
Service "CV_P_DGMGRL.abc.com" has 1 instance(s).
Instance "CV", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully

Repeated same steps on the Standby database also so that same changes and take place.

Then we brought up the database manually and now due to the initial switchover Standby database role is primary and Primary database role is Physical_Standby.

Hence to check the fix we switched over from Standby to Primary and it went perfectly fine

DGMGRL> switchover to 'CV_P';
Performing switchover NOW, please wait...
Operation requires shutdown of instance "CV" on database "CV_S"
Shutting down instance "CV"...
ORA-01109: database not open

Database dismounted.
ORACLE instance shut down.
Operation requires shutdown of instance "CV" on database "CV_P"
Shutting down instance "CV"...
ORA-01109: database not open

Database dismounted.
ORACLE instance shut down.
Operation requires startup of instance "CV" on database "CV_S"
Starting instance "CV"...
ORACLE instance started.
Database mounted.
Operation requires startup of instance "CV" on database "CV_P"
Starting instance "CV"...
ORACLE instance started.
Database mounted.
Switchover succeeded, new primary is "CV_P"
DGMGRL>

Alternate Solution: Instead of reloading the listeners we can nullify the db_domain parameter on the Primary and Standby Databases. I have not tried incase you get hit by this issue you can try it.

Regards,
Tanveer




Saturday, February 18, 2012

ORA-01111 and ORA-01110 on the STANDBY DATABASE

Hi,
Recently we built a STANDBY DATABASE for one of our production databases.
The STANDBY DATABASE was 10g with raw volumes

Issue Description

The recovery was stuck with the below error on the STANDBY DATABASE.

Alert Log
Errors in file /oracle/admin/CPD/bdump/CPD_mrp0_19746.trc:
ORA-01111: name for data file 178 is unknown - rename to correct file
ORA-01110: data file 178: '/oracle/product/10.2.0.4/CPD/dbs/UNNAMED00178'
ORA-01157: cannot identify/lock data file 178 - see DBWR trace file
ORA-01111: name for data file 178 is unknown - rename to correct file
ORA-01110: data file 178: '/oracle/product/10.2.0.4/CPD/dbs/UNNAMED00178'

Trace File
MRP0: Background Media Recovery terminated with error 1111
ORA-01111: name for data file 178 is unknown - rename to correct file
ORA-01110: data file 178: '/oracle/product/10.2.0.4/CPD/dbs/UNNAMED00178'
ORA-01157: cannot identify/lock data file 178 - see DBWR trace file
ORA-01111: name for data file 178 is unknown - rename to correct file
ORA-01110: data file 178: '/oracle/product/10.2.0.4/CPD/dbs/UNNAMED00178'

Troubleshooting

The error codes pointed out that it is a issue with missing file which is present in the control file but not present physically on the STANDBY host.
As this issue occurred later after we had successfully configured the STANDBY DATABASE, we were able to make out that this could have occured due to new raw volume addition on primary database which has not been provisioned on the STANDBY DATABASE side.

Hence at first we needed to find the number of files added on primary after the STANDBY DATABASE build. The below query listed the time the raw volumes were added to the database and hence we could figure out the raw volumes added to the database

set lines 150
set pages 0
col name for a75
select name,to_char(CREATION_TIME,'HH24:MI:SS MON-DD-YYYY') from v$datafile order
by creation_time;

We found that there were 2 files added to the database after we had created the STANDBY DATABASE.

Nextly we needed to find the exact raw volume which was needed by the STANDBY DATABASE to continue recovery.
From the error we were able to know that the STANDBY DATABASE needed the datafile with file number 178
We ran the below query on the primary database, which yielded us the exact name of the missing raw volume and the same query on the STANDBY side gave the psuedo name the STANDBY DATABASE had given to the missing file on the STANDBY DATABASE side. Hence confirming the exact raw volume name.

Primary Side
------------
SQL> select name from v$datafile where file#=178;
NAME
----------------------------------------
/dev/mapper/oraCPD_data_017_4600Mp9

STANDBY Side
-------
SQL> select name from v$datafile where file#=178;
NAME
----------------------------------------
/oracle/product/10.2.0.4/CPD/dbs/UNNAMED00178


Note: Though there were 2 files added to the production that was missing on STANDBY DATABASE side, the recovery was stuck at the very first missing file.
Hence we needed to correct only that file but for the issue not to occur again due to the other missing raw volume you need to make sure even that
raw volume is made available on the STANDBY DATABASE side.
(We got the all the missing raw volumes present on the primary side added to the STANDBY to avoid such issues in the future when there is new raw volume addition)


Solution

1. Make sure that all the missing raw volumes are added to the STANDBY side at host level.

2. Stop Recovery on the Standby Server
recover managed standby database cancel;

3. Change the standby_file_management parameter to MANUAL on STANDBY DATABASE
alter system set standby_file_management=manual;

4. Rename the datafile on Standby side
alter database create datafile '/oracle/product/10.2.0.4/CPD/dbs/UNNAMED00178'
as '/dev/mapper/oraCPD_data_017_4600Mp9';

5. Change the standby_file_management parameter to AUTO on STANDBY DATABASE
alter system set standby_file_management=auto;

6. Restart the Recovery
alter database recover managed standby database disconnect;

Regards,
Tanveer

Wednesday, November 30, 2011

Not able to create new database connection: FND^@SECURITY_APPL_LOGIN_FAILED^@ when starting Concurrent Managers

Hi,
Recently we faced a issue after cloning R12 environment
Issue Description:
-----------------
The concurrent managers were not coming up after cloning

Workflow Agent Listener Service
Workflow Mailer Service
Workflow Document Web Service
Email Center Download Processor
Output Post Processor

When checking the manager logs for each of the managers I found the below error
Not able to create new database connection: FND^@SECURITY_APPL_LOGIN_FAILED^@

Troubleshooting:
---------------
Quick list of managers which were not started showed us that the mangers using the
framework were the ones affected.
Error pointed issue with the GUEST user.
Hence we checked the GUEST password in the dbc file $FND_SECURE/CDEV.dbc on CM node
and found the below entry
GUEST_USER_PWD=GUEST/CPRD

Also when we checked the context file $CONTEXT_FILE on CM node we found the parameter
s_guest_pass set to CPRD

Now we wanted to validate GUEST password at database base level
1. Conn to database as apps

2. Find the password at database level
SQL> select fnd_profile.value('GUEST_USER_PWD') from dual;
FND_PROFILE.VALUE('GUEST_USER_PWD')
——————————————————————————
GUEST/CDEV
Here we found that in the database the password for GUEST was different

3. Then we checked if the guest user was working fine with the password at database level
SQL> select FND_WEB_SEC.VALIDATE_LOGIN('GUEST','CDEV') from dual;
FND_WEB_SEC.VALIDATE_LOGIN('GUEST','CDEV')
——————————————————————————–-----
Y

4. We also found that mismatch in database and the CM node was because, as this is a multi
node architecture the autoconfig was not run last on the CM node but some other node which
had the value for s_guest_pass as CDEV. Hence we inferred that the database was updated
with the password for GUEST user with the value of s_guest_pass in the context file of the
node on which the autoconfig was run last

Solution:
--------
We can resolve this issue in two ways
Method 1
-----------
The traditional way of changing the GUEST password is to edit the parameter s_guest_pass in
the context file and running autoconfig

In our case as only the context file on the CM node had the wrong entry, we just need to make
the change in the context file on CM node and run autoconfig to fix the issue.
If you have more nodes where the context file is having the wrong password, then you need to correct the entry and run autoconfig on all those nodes

Method 2
-----------
In this method we can avoid running the autoconfig. We followed this method to fix the issue
i. Open the adgendbc.sh at $INST_TOP/admin/install and change the GUEST password to the
right password
ii. Take backup of the dbc file at location $FND_SECURE
iii. Run adgendbc.sh from location $INST_TOP/admin/install. It will prompt for Apps
username and password
iv. Check the new dbc file generated at location $FND_SECURE and see if the correct GUEST
password is generated
v. Do steps i to iv on all nodes where the dbc file has the wrong GUEST password
vi. Restart the concurrent managers
vii. Make the change in the conext file on the all nodes were the above steps i to iv were done,
so that the next autoconfig will generate the right dbc file

Regards,
Tanveer

Saturday, November 5, 2011

ORA-01620 When mounting a standby database

Hi,

Recently we had task to setup a DG for a RAC database.
The primary and the DG were 11g.

After copying the primary database to the DG server, we tried to mount the database on the
DG server before starting the recovery. But we got the below error

SQL> alter database mount standby database;
alter database mount standby database
*
ERROR at line 1:
ORA-01620: no public threads are available for mounting

The alert log had only the below entry and also no trace file was generated for the error.
ORA-1620 signalled during: alter database mount

From the error it was clear that it was issue with the redo log threads. But for building a DG we need not add redo threads at the standby database and also as we were not going to do a DG
Broker setup hence standby redo log file addition was also not necessary. Hence it became sure to us that the issue was not on the standby database side but on the primary database

TROUBLESHOOTING:

On the Primary Database when we ran the below query we found that the threads were
enabled for the RAC instances in PRIVATE mode and not in PUBLIC mode which is needed for the DG to work.

SQL> select thread#,status,enabled from v$thread;

THREAD# STATUS ENABLED
---------- ------ --------
1 OPEN PRIVATE
2 OPEN PRIVATE
3 OPEN PRIVATE
4 OPEN PRIVATE

In the RAC system, each instance has to have its own redo log groups. The redo log file groups of an instance are collectively called a thread, or more appropriately, a redo log thread. The redo threads can be private or public. If the redo thread is public then any instance
can acquire it and if it is private then only a specific instance can acquire it as mentioned in the
initialization file.

SOLUTION:

Inorder to get the DG running, we must convert all the redo threads to PUBLIC in primary
database, create the standby control file, and mount the standby database with this standby
control file.
Below are the steps followed by us, but there are can be deviations.

Note: To enable or disable threads the database has to be in open mode

1. Shut down all instances other than the instance 1

2. Disable the threads as you cannot change the mode from PRIVATE to PUBLIC directly.

SQL> alter database enable public thread 1;
alter database enable public thread 1
*
ERROR at line 1:
ORA-01612: instance TSEP (thread 1) is already enabled

Hence we have disable it and then enable in PUBLIC mode

alter database disable thread 2;
alter database disable thread 3;
alter database disable thread 4;

Also you cannot disable thread 1 at this point, as the instance 1 will be using it
SQL> alter database disable thread 1;
alter database disable thread 1
*
ERROR at line 1:
ORA-01615: instance TSEP (thread 1) is mounted - cannot disable

3. Enable the threads in PUBLIC mode

alter database enable public thread 2;
alter database enable public thread 3;
alter database enable public thread 4;

4. Shut down instance 1 and open the database in instance 2.
Repeat the disable thread and enable thread in PUBLIC mode for instance 1

alter database disable thread 2;
alter database enable public thread 2;

5. Start all the other instances of the RAC including instance 1

6. Create the standby control file again on the primary database

7. Copy the standby control file to the standby database and mount the database
SQL> alter database mount standby database;
Database mounted

8. Start the recovery process

-- Tanveer Madan



Tuesday, August 30, 2011

Issue creating FND_CTX_LOBS with missing DEFAULT_DATASORE

Your autoconfig/adadmin may fail saying that the index APPS.FND_CTX_LOBS is not found and
While trying to create the index APPS.FND_CTX_LOBS you may get the below error:

SQL> @aflobbld.sql applsys apps
declare
*
ERROR at line 1:
ORA-29855: error occurred in the execution of ODCIINDEXCREATE routine
ORA-20000: Oracle Text error:
DRG-10700: preference does not exist: CTXSYS.DEFAULT_DATASTORE
ORA-06512: at line 277

Solution:
Run the below commands in sequence

sqlplus ctxsys/ctxsys
spool ctxdef.log
@?/ctx/admin/ctxdef.sql

sqlplus ctxsys/ctxsys
@?/ctx/admin/defaults/dr0defin.sql "AMERICAN";


While running dr0defin.sql if you get the below error, please ignore and proceed further
Creating default policy...
begin
*
ERROR at line 1:
ORA-20000: Oracle Text error:
DRG-10507: duplicate index name: DEFAULT_POLICY_ORACONTAINS
ORA-06512: at "CTXSYS.DRUE", line 160
ORA-06512: at "CTXSYS.CTX_DDL", line 694
ORA-06512: at line 2


Then run aflobbld.sql it will error out with following error 1st time:

SQL> @aflobbld.sql applsys apps
declare
*
ERROR at line 1:
ORA-29855: error occurred in the execution of ODCIINDEXCREATE routine
ORA-20000: Oracle Text error:
DRG-50857: oracle error in drvxtab.create_index_tables
ORA-00955: name is already used by an existing object
ORA-06512: at line 277

Ignore the error and rerun the same script to create the index apps.fnd_ctx_lobs



- Saptha

Monday, August 29, 2011

Changes made in context file not reflecting in the configuration files

You made few changes to context file and ran autoconfig. But the changes are not propagated to the configuration files. Also, the changes you made to the xml file itself got wiped out.

Here is the deal:
In the autoconfig log, you will see the following:

-------------------ADX Database Utility Finished---------------
OAM Context editing support feature: Enabled
OAM Customization support feature : Enabled
File system template : /apps/OECPRD/apps/apps_st/appl/ad/12.0.0/admin/template/adxmlctx.tmp
Checking for customizations to Context template
Customizations found : None
----------------------------------------------------------------
File system Context file :/apps/local/OECPRD/inst/apps/OECPRD_c7-a1-06/appl/admin/OECPRD_c7-a1-06.xml

Checking the Context file for possible updates from the Database
Comparing the Context file stored in database with the Context file in the file system

Result : File system Context is below par with respect to the data base Context
Action to be taken : Copy the Data Base Context onto the file system

Result : Context file successfully copied

As seen, the autoconfig actually copied the context file from the database to the file system. This explains why the changes were not propagated and also explains why the context file changes are wiped out.

So why did this happen?
This happened because autoconfig determined that the version of the xml in the database is greater than the one on the filesystem.

So, what can cause this situation?
One thing that I can think of is that the context file was copied from another node of the same system and only host names were changed and autoconfig was run.

So regardless of what caused this issue, how do you fix it?
FIX:
====
1. select name,version,last_synchronized from apps.fnd_oam_context_files where node_name='c7-a1-06' order by serial_number desc
2. Copy back the xml with which you intended to run autoconfig to the $CONTEX_FILE location and edit the xml tag (basically it should be any value bigger than what was returned in step 1). If the value returned in step 1 was 347, update the s_contextserial in the xml to 348. Following is the code block which needs to be updated:


oa_context version="$Revision: 120.217.12000000.48 $"
oa_context_name oa_var="s_contextname" OECPRD_c7-a1-06 oa_context_name
oa_context_serial oa_var="s_contextserial" 348 oa_context_serial

3. Run autoconfig.
4. Validate that the xml is fine and you don’t see entries in the log that xml was copied from the database to filesystem.

- Aravind Kamath Posral

Of Analyzing Situations Objectively and Emotionally

Alright, this is my third post which is not technical, after "my introduction" and "Musings of the mind". This is again more of pondering over situations that life presents - mental grind of the sorts - and nothing technical. So you have been warned sufficiently that this is not a tech post and proceed to read the rest of the post at your own risk! I am also expecting quite a few flames!

Recently I spent some time with a old friend of mine and he is working through some issues. Talking to him made me think on this subject and hence this post.

I have often seen people telling others to think objectively in a structured manner and not emotionally/unorganized way to solve issues. The best part about this thought process is that most of us think objectively and in a structured logical fashion when we are dealing with problems at work or extraneous situations. Be it an Oracle apps issue, talking about promotion to your boss, changing jobs etc.. However, whenever presented with a problem associated to anything dear to ones heart, suddenly the objectivity, logic and structure in thinking seems to vanish. Most of us start thinking with our hearts - emotionally. The person starts addressing the problems in ways which defy "logic" to others.

In my life's experience, I have learnt that it is easy to ask someone to think objectively to address an issue close to heart - but the actual person who goes through it - will seldom be able to do it.

I think this is so because nothing is perfect in this world - not the thing/person close to your heart included. We have all the while focused on the positives/amiable aspects and held something/someone close to our heart and ignored everything else. Mind you, life is dynamic and when negatives start becoming predominant, we are no longer able to digest this. On many occasions, we hate to address such issues and procastinate acting on them because of a multitude of reasons - insecurity, societal issues, turbulence, grief... Mind just wants to believe that everything is ok. Howeverm when it reaches a breaking point, the issue can no longer be ignored, the person is stuck emotionally and loses all reasoning.

So it is easy to ask someone to think objectively, but when matters of heart hits us, we ourself lose our ability to think objectively/logically.

I dont have any solutions/suggestions here, but I think it certainly helps to "step into ones shoes" before being judgemental about others resolve, grit, ability to think rationally, deal with issues and so on..

Finally, everyone has something very very dear to heart - just that it differs from person to person. There may be "gifted" people with ability to think rationally and objectively in matters of heart, but from what I have seen in my life so far, they are a minority!

- Aravind Kamath Posral

Sunday, August 28, 2011

Resizing the TEMP Tablespace

The TEMP tablespace can be resized with or without outage on the database. But resizing tablespace without outage may require few extra steps.
Example considering with outage
---------------------------------------
Considering we have a single temporary tablespace in the database, and the requirement is to resize it and add it into
temporary tablespace groups

1. Check the default temporary tablespace, this is just for validation
SQL>SELECT * FROM database_properties WHERE property_name='DEFAULT_TEMP_TABLESPACE';
PROPERTY_NAME PROPERTY_VALUE DESCRIPTION
------------------------------ --------------- -------------------------------------
DEFAULT_TEMP_TABLESPACE TEMP Name of default temporary tablespace

Also if you check the TEMPORARY_TABLESPACE column in dba_users table, all the users will have TEMP as temporary tablespace

2. Check the size of the temporary tablespace
SQL> select tablespace_name,sum(bytes)/1024/1024/1000 Size_in_GB from dba_temp_files group by tablespace_name;
TABLESPACE_NAME SIZE_in_GB
--------------- -------------
TEMP 40

3. To make effective use of the volumes available, we can create another temporary tablespace with smaller size and make
note of the volumes already being used by the temporary tablespace and use the same volumes to create our new resized temporary
tablespace
SQL> create temporary tablespace TEMP3
tempfile
'/dv/pper/ora_SCOTT_p7/' size 8600M',
'/dv/pper/ora_SCOTT_p8/' size 8600M
/
You cannot drop the temporary tablespace TEMP as it is still the default temporary tablespace. If you try then you will get
the below error
Hence let us change the default temporary tablespace as TEMP3

SQL> alter database default temporary tablespace TEMP3;
Database altered

SQL>SELECT * FROM database_properties WHERE property_name='DEFAULT_TEMP_TABLESPACE';
PROPERTY_NAME PROPERTY_VALUE DESCRIPTION
------------------------------ --------------- -------------------------------------
DEFAULT_TEMP_TABLESPACE TEMP3 Name of default temporary tablespace

4. Now we can drop and recreate the TEMP tablespace as per new sizes
SQL> drop tablespace TEMP;
Tablespace dropped

As we are going to also add the temporary tablespaces to group we can create as below

SQL> create temporary tablespace TEMP1
tempfile
'/dv/pper/ora_SCOTT_p7/' size 8600M',
'/dv/pper/ora_SCOTT_p8/' size 8600M
/

SQL> create temporary tablespace TEMP2
tempfile
'/dv/pper/ora_SCOTT_p7/' size 8600M',
'/dv/pper/ora_SCOTT_p8/' size 8600M
/

SQL> select tablespace_name,sum(bytes)/1024/1024/1000 Size_in_GB from dba_temp_files group by tablespace_name;
TABLESPACE_NAME SIZE_in_GB
--------------- -------------
TEMP1 16
TEMP2 16

5. Now we can add these into temporary tablespace group

SQL> alter tablespace TEMP1 tablespace group TEMP;
Tablespace altered

SQL> alter tablespace TEMP2 tablespace group TEMP;
Tablespace altered

SQL> select * from dba_tablespace_groups;
GROUP_NAME TABLESPACE_NAME
------------- -------------------
TEMP TEMP1
TEMP TEMP2

6. Now change the default temporary tablespace to TEMP and drop TEMP3

SQL> alter database default temporary tablespace TEMP;
Database altered

SQL>SELECT * FROM database_properties WHERE property_name='DEFAULT_TEMP_TABLESPACE';
PROPERTY_NAME PROPERTY_VALUE DESCRIPTION
------------------------------ --------------- -------------------------------------
DEFAULT_TEMP_TABLESPACE TEMP Name of default temporary tablespace

SQL> drop tablespace temp3;

7. As we have retained the name of the tablespace group as TEMP which was same previously, we need not alter the users in the dba_users table.
If you change the name of the tablespace group as TEMP_A, then you need to change the temporary tablespace for all the already created users.

SQL> alter user XABC default temporary tablespace TEMP_A;

New users who will be created after this resize exercise will by default get TEMP_A as temporary tablespace.

Example considering without outage
-------------------------------------------

The steps remain the same as above but with few changes

A. Follow steps from 1 to 3

B. Now as the database is in use, the users will use the temporary tablespace TEMP though we have changed the default temporary tablespace to TEMP3.
This is because all the already created users have TEMPORARY_TABLESPACE column in dba_users table updated as TEMP.
So before dropping tablespace TEMP. we need to run alter user commands and change the temporary tablespace to TEMP3 to all the users in dba_users

Once this change is made, any future sessions from this users will us TEMP3 for their operartions.
Now you can drop the TEMP tablespace. Before dropping check that no sessions are using TEMP tablespace.

C. Follow steps 4 and 5

D. As in step B, we now need to change the temporary tablespace of the users back to TEMP if you retained the same previous name, if not then change it
to the new temporary tablespace name.

E. Follow step 6