Thursday, June 23, 2011

adoacorectl.sh: exiting with status 152

Today we faced an issue where the oacore jvms were coming up to init state but not alive and then would show down. No errors logged anywhere. Enabling debugging did not give out any information.

Here is how i did the troubleshooting:
1. Validate the connect string (apps_jdbc_url) provided in dbc file by connecting using sqlplus apps/@" apps jdbc url connect string". This was working.

2. When the oacore jservs are in init stage, connect to database and query v$session:

select distinct machine from v$session;
This showed that oacore was indeed making connection to database. This also means that dbc file is NOT the suspect.

3. After clearing persistence folder etc which are usual suspects, all the efforts went in vain.

4. The problem had to be with the configuration because the database connections were going through fine.

5. I analyzed the applications deployed to this jvm and each of the applications were using a custom schema name to connect to the database.

6. This node is an external node in DMZ. This implies this is protected by firewall.

7. For the external nodes, we use OCM to route the connections to the database.

8. Someone had run autoconfig which had overwritten the customization in $IAS_ORACLE_HOME/j2ee/config/oacore/data-sources.xml and it was trying to connect "directly" to database.

9. Changed the connect string in data-sources.xml to route through OCM and bingo! the oacore came "alive"

- Aravind Kamath Posral

No comments: