adoacorectl.sh: exiting with status 204

With 30 mins to golive, we encountered following error in production environment while trying to startup the Apache/Web services:

adoacorectl.sh: exiting with status 204
adoafmctl.sh: exiting with status 204

We have forms running in socket mode, so there were no issues with that. Apache came up fine. There were no issues with OPMN. No other errors were logged.

Subsequently, I figured out that the issue was due to /apps filesystem getting filled up. The processes were unable to write to the disk due to which the startup was not happening. During the course of troubleshooting, I figured out that oacorestd.out and oacorestd.err (predominantly stdout) logs were consuming all the space. We have 5 OACORE process and one oafm process. This means we have 6 logs which had grown to consume all the space.

To overcome this issue, we shutdown all the services and zipped the logs to create space.

Once this was done, I tried to start the services but still encountered the same error. However, this time around, I saw the following error in oacorestd.err:

11/06/12 09:14:05 Error initializing server: /apps/local/CEPROD/inst/apps/CEPROD_c7-iprd-54/ora/10.1.3/j2ee/oacore/config/server.xml, Fatal error at line 1 offset 1 in file:/apps/local/CEPROD/inst/apps/CEPROD_c7-iprd-54/ora/10.1.3/j2ee/oacore/config/server.xml: .:
XML-20108: (Fatal Error) Start of root element expected.

When i checked the server.xml, I found that it had been reduced to zero bytes. At this moment, one of the metalink notes suggested to run autoconfig to fix this issue.

However, we were already out of the outage window and autoconfig would be the last option.

As a workaround to this issue, I went to the logs of the last autoconfig run (we were sure that no patches that bring in new templates were applied
post that autoconfig run) and located the server.xml backup for both oacore and oafm and placed them back and started the services and everything came up as expected.

-Aravind Kamath Posral