RC-50004: Fatal: Error occurred in ApplyAppsTier

Recently while cloning an RAC enabled R12 instance, we faced the aforesaid mentioned error while running adcfgclone.pl on apps tier towards 79% completion. There was a generic java error stack accompanying the error RC-500004, which I did not capture and hence I cannot share that here. However, while investigating the issue, I noticed the following error in $INST_TOP/admin/log/ApplyAppsTier_[timestamp].log.

java.sql.SQLException: Listener refused the connection with the following error: ORA-12514, TNS:listener does not currently know of service requested in connect descriptor The Connection descriptor used by the client was: (DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=isg-db-01-vip.domain.com)(PORT=1612)))(CONNECT_DATA=(SERVICE_NAME=estg)))

Database connection to jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=isg-db-01-vip.domain.com)(PORT=1612)))(CONNECT_DATA=(SERVICE_NAME=estg))) failed UPDATE call failed with exit code 1 Updating parameters: fnd_jdbc_buffer_min=1 fnd_jdbc_buffer_max=5 fnd_jdbc_buffer_decay_interval=300 java.sql.SQLException: Listener refused the connection with the following error: ORA-12514, TNS:listener does not currently know of service requested in connect descriptor The Connection descriptor used by the client was: (DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=isg-db-01-vip.domain.com)(PORT=1612)))(CONNECT_DATA=(SERVICE_NAME=estg)))

Database connection to jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=isg-db-01-vip.domain.com)(PORT=1612)))(CONNECT_DATA=(SERVICE_NAME=estg))) failed Updating parameters: fnd_jdbc_buffer_decay_size=5 fnd_jdbc_usable_check=false fnd_jdbc_context_check=true java.sql.SQLException: Listener refused the connection with the following error: ORA-12514, TNS:listener does not currently know of service requested in connect descriptor The Connection descriptor used by the client was: (DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=isg-db-01-vip.domain.com)(PORT=1612)))(CONNECT_DATA=(SERVICE_NAME=estg)))

Database connection to jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=isg-db-01-vip.domain.com)(PORT=1612)))(CONNECT_DATA=(SERVICE_NAME=estg))) failed Updating parameters: fnd_jdbc_plsql_reset=false java.sql.SQLException: Listener refused the connection with the following error: ORA-12514, TNS:listener does not currently know of service requested in connect descriptor The Connection descriptor used by the client was: (DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=isg-db-01-vip.domain.com)(PORT=1612)))(CONNECT_DATA=(SERVICE_NAME=estg)))

Database connection to jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=isg-db-01-vip.domain.com)(PORT=1612)))(CONNECT_DATA=(SERVICE_NAME=estg))) failed

DBC generation failed with exit code 1

As it is evident from the error, the dbc file generation was failing due to which adcfgclone.pl was failing.
Digging deeper into this issue, I verified that the database instance was correctly registered with the listener. Bounce of the database listener did not help either. That is when I noticed that the connect string being used was having load balance entries, but did not have both the active RAC node information. This led me to check the Net services topology information verification where in I observed that the node information was not updated correctly. ACtually adgentns.pl is responsible for creating these entries, and I observed that adgentns.pl had not failed with any error though it had reported the ignorable unique key constraint error.
Fix:
1.Run the following command to clean up the net topology information:
perl [AD_TOP]/bin/adgentns.pl contextfile=[full path to context file] -removesystem
This will remove clean up the information in fnd_app_servers and fnd_nodes table.
2. Run autoconfig on both the db nodes and apps nodes: this should fix the issue.
3. However, in our case, there were a few corrections required in the database xml file:
i. s_dbService : dbname.domain
ii. s_dbClusterInst: 1 (on node 1) and 2 (on node 2)
iii. s_instNumber: 1 (on node 1) and 2 (on node 2)
iv. "s_instThread: 1 (on node 1) and 2 (on node 2)

After making the above corrections, I ran autoconfig and this fixed the issue.
- Aravind Kamath Posral