RC-50004: Fatal: Error occurred in ApplyAppsTier

Recently while cloning an RAC enabled R12 instance, we faced the aforesaid mentioned error while running adcfgclone.pl on apps tier towards 79% completion. There was a generic java error stack accompanying the error RC-500004, which I did not capture and hence I cannot share that here. However, while investigating the issue, I noticed the following error in $INST_TOP/admin/log/ApplyAppsTier_[timestamp].log.

java.sql.SQLException: Listener refused the connection with the following error: ORA-12514, TNS:listener does not currently know of service requested in connect descriptor The Connection descriptor used by the client was: (DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=isg-db-01-vip.domain.com)(PORT=1612)))(CONNECT_DATA=(SERVICE_NAME=estg)))

Database connection to jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=isg-db-01-vip.domain.com)(PORT=1612)))(CONNECT_DATA=(SERVICE_NAME=estg))) failed UPDATE call failed with exit code 1 Updating parameters: fnd_jdbc_buffer_min=1 fnd_jdbc_buffer_max=5 fnd_jdbc_buffer_decay_interval=300 java.sql.SQLException: Listener refused the connection with the following error: ORA-12514, TNS:listener does not currently know of service requested in connect descriptor The Connection descriptor used by the client was: (DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=isg-db-01-vip.domain.com)(PORT=1612)))(CONNECT_DATA=(SERVICE_NAME=estg)))

Database connection to jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=isg-db-01-vip.domain.com)(PORT=1612)))(CONNECT_DATA=(SERVICE_NAME=estg))) failed Updating parameters: fnd_jdbc_buffer_decay_size=5 fnd_jdbc_usable_check=false fnd_jdbc_context_check=true java.sql.SQLException: Listener refused the connection with the following error: ORA-12514, TNS:listener does not currently know of service requested in connect descriptor The Connection descriptor used by the client was: (DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=isg-db-01-vip.domain.com)(PORT=1612)))(CONNECT_DATA=(SERVICE_NAME=estg)))

Database connection to jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=isg-db-01-vip.domain.com)(PORT=1612)))(CONNECT_DATA=(SERVICE_NAME=estg))) failed Updating parameters: fnd_jdbc_plsql_reset=false java.sql.SQLException: Listener refused the connection with the following error: ORA-12514, TNS:listener does not currently know of service requested in connect descriptor The Connection descriptor used by the client was: (DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=isg-db-01-vip.domain.com)(PORT=1612)))(CONNECT_DATA=(SERVICE_NAME=estg)))

Database connection to jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=isg-db-01-vip.domain.com)(PORT=1612)))(CONNECT_DATA=(SERVICE_NAME=estg))) failed

DBC generation failed with exit code 1

As it is evident from the error, the dbc file generation was failing due to which adcfgclone.pl was failing.
Digging deeper into this issue, I verified that the database instance was correctly registered with the listener. Bounce of the database listener did not help either. That is when I noticed that the connect string being used was having load balance entries, but did not have both the active RAC node information. This led me to check the Net services topology information verification where in I observed that the node information was not updated correctly. ACtually adgentns.pl is responsible for creating these entries, and I observed that adgentns.pl had not failed with any error though it had reported the ignorable unique key constraint error.
Fix:
1.Run the following command to clean up the net topology information:
perl [AD_TOP]/bin/adgentns.pl contextfile=[full path to context file] -removesystem
This will remove clean up the information in fnd_app_servers and fnd_nodes table.
2. Run autoconfig on both the db nodes and apps nodes: this should fix the issue.
3. However, in our case, there were a few corrections required in the database xml file:
i. s_dbService : dbname.domain
ii. s_dbClusterInst: 1 (on node 1) and 2 (on node 2)
iii. s_instNumber: 1 (on node 1) and 2 (on node 2)
iv. "s_instThread: 1 (on node 1) and 2 (on node 2)

After making the above corrections, I ran autoconfig and this fixed the issue.
- Aravind Kamath Posral

How to change the forms mode in R12?

Hi,
This post is more of a procedural thing rather than a problem-troubleshoot-fix kind of a post. By default, when you configure an R12 instance, rapid install will configure forms in servlet mode. The following steps will come in handy if you are entrusted with the task of changing the forms mode to socket or vice-versa.
a. Currently your forms is running in servlet mode and you want to configure socket mode.
i. Stop all the running services.
ii. Clear orphans, if any.
iii. Take a backup of the existing xml file.
iv. Run the following command:

$FND_TOP/bin/txkrun.pl -script=ChangeFormsMode -contextfile=[context file name with full path] -mode=socket -port=[forms port value] -runautoconfig=No

v. Verify that your xml has been updated with "socket" value for a few variables as shown in the log of the above script.
vi. Run autoconfig.
vii. Verify that you can connect to forms using front end and that you should see a frmsrv process running on the forms host. Run adopmnctl.sh status to verify that forms is no longer running in servlet mode.

b. Currently your forms is running in socket mode and you want to configure servlet mode.
i. stop all the running services.
ii. Clear orphans, if any.
iii. Take a backup of the existing xml file.
iv. Run the following command:

$FND_TOP/bin/txkrun.pl -script=ChangeFormsMode -contextfile=[context file name with full path] -mode=servlet -port=[forms port value] -runautoconfig=No

v. Verify that your xml has been updated with "servlet" value for a few variables as shown in the log of the above script.
vi. Run autoconfig.
vii. Verify that you can connect to forms using front end and that you do not see a frmsrv process running on the forms host. Run adopmnctl.sh status to verify that forms is indeed running in servlet mode.

- Aravind Kamath Posral

adapcctl.sh: exiting with status 204

Hi,
Happy New Year to all readers!
Recently I was cloning an R12 instance. After the cloning part was done using rapid clone, the apache failed to come up and was reporting the following error:
adapcctl.sh: exiting with status 204
However, there were no errors in error_log or in the adapcctl.txt log file that is generated as part of the startup script. Further digging into the issue revealed the following error in the HTTP_Server~1 log file under $LOG_HOME/ora/10.1.3/opmn:
Syntax error on line 17 of /apps/local/FIEST1/inst/apps/FIEST1_iftg-01/ora/10.1.3/Apache/Apache/conf/mod_osso.conf:Unable to open /apps/local/FIEST1/inst/apps/FIEST1_iftg-01/ora/10.1.3/Apache/Apache/conf/osso/FIEST1_iftg-01_osso.conf.

It turns out that this issue is because the source is SSO enabled and the target was not yet SSO enabled. Running $FND_TOP/bin/txkrun.pl -script=SetSSOReg -removereferences=Yes in target instance did not help either.

Fix:
Run detailed steps to deregister OID/SSO registration. This will run autoconfig as part of the deregistration process, which will generate the correct mod osso files. Basically, the s_mod_osso_conf_comment should have # (comment) in xml file before running autoconfig (which will be the result of running deregistration steps).

The apache came up fine subsequently.

- Aravind Kamath Posral