Configuration assistant "BPEL Configuration Assistant" failed

Recently we were in the process of upgrading BPEL from 10.1.3.3 to 10.1.3.5. During the process we fcaed the following error when the configuration assistants were running:

apps/st1bfi/product/AS10gR3_BPEL/j2ee/BPEL_OC4J_FIB/applications/orabpel.ear

[java] 10/06/28 08:38:22 Notification ==>Initialize /apps/st1fib/product/AS10gR3_BPEL/j2ee/BPEL_OC4J_FIB/applications/orabpel.ear begins...

[java] 10/06/28 08:38:22 Notification ==>Operation failed with error:
[java] Unable to find/read file META-INF/application.xml in /apps/st1fib/product/AS10gR3_BPEL/j2ee/BPEL_OC4J_FIB/applications/orabpel (META-INF/application.xml)


[java] Redeploy error: Redeploy failed: Operation failed with error:
[java] Unable to find/read file META-INF/application.xml in /apps/st1fib/product/AS10gR3_BPEL/j2ee/BPEL_OC4J_FIB/applications/orabpel (META-INF/application.xml)


BUILD FAILED
/apps/st1fib/product/AS10gR3_BPEL/bpel/system/services/install/ant-tasks/redeploy.xml:51: The following error occurred while executing this line:
/apps/st1fib/product/AS10gR3_BPEL/bpel/system/services/install/ant-tasks/redeploy.xml:64: The following error occurred while executing this line:
/apps/st1fib/product/AS10gR3_BPEL/bpel/system/services/install/ant-tasks/redeploy.xml:92: The following error occurred while executing this line:
/apps/st1fib/product/AS10gR3_BPEL/bpel/system/services/install/ant-tasks/redeploy-common.xml:80: The following error occurred while executing this line:
/apps/st1fib/product/AS10gR3_BPEL/bpel/system/services/install/ant-tasks/redeploy-common.xml:152: Java returned: 1

Total time: 6 seconds
Configuration assistant "BPEL Configuration Assistant" failed


While I have strong reasons to believe that this was caused due to the oracle home being on the NFS mounted file system, we had an urgent requirement to deliver the instance to the business. We had to deliver the instance under two hours and we had to crack this issue.


There are three main parts to this apps filesystem upgrade part of BPEL upgrade:

1. Applying the patchset 10.1.3.5
2. Configuration assistants run subsequently
3. Run root.sh and exit the installer.


The aforesaid issue you get when the configuration assistant is running and there were no issues reported by the installer in the earlier stage. To fix this issue, we wanted to figure out what were the sequence of events performed by the configuration assistant (roughly, highlighting main events which i thought were important):


- Patch brings in new jar files
- Patch also updates the file $ORACLE_HOME/bpel/system/services/lib/orabpel.ear
- The configuration assistant checks server.xml to see if this application (orabpel.ear is present).
- If it is present, then it deletes the corresponding line in server.xml ($ORACLE_HOME/j2ee/BPEL_OC4J_FIB/config/server.xml) and the proceeds with the actual deployment.
- The configuration assistant then copies the above orabpel.ear file to $ORACLE_HOME/j2ee/BPEL_OC4J_FIB/applications
- The configuration assistant then extracts orabpel.ear file
- If application-deployment subfolder for this application is created if it is not present.
- The server.xml ($ORACLE_HOME/j2ee/BPEL_OC4J_FIB/config/server.xml) file is again updated with this entry.


Here is how we fixed the issue:

- When the configuration assistant failed, we clicked on the retry button.
- This time the assistant failed with the error that ORABPEL application not found.
- The reason is, from the first iteration, the entry for orabpel was deleted in server.xml.
- Make the entry for orabpel in the server.xml like ""
- Copy the orabpel.ear from $ORACLE_HOME/bpel/system/services/lib/orabpel.ear to /apps/st1fib/product/AS10gR3_BPEL/j2ee/BPEL_OC4J_FIB/applications
- Click the retry button, it was successful.


- Aravind Kamath Posral

ORA-12514: Message 12514 not found; product=RDBMS; facility=ORA

Here is a quick one..

Concurrent managers (workers) not coming up but FNDSM is up. FNDSM log has following errors:

Routine AFPCMT encountered an ORACLE error. ORA-01012: not logged on
.

Review your error messages for the cause of the error. (=)

_ 2 _
Routine AFPSMG encountered an ORACLE error. ORA-03114: not connected
to ORACLE
.

Review your error messages for the cause of the error. (=)

_ 3 _
Routine AFPCSQ encountered an ORACLE error. ORA-12514: Message 12514
not found; product=RDBMS; facility=ORA

.

Review your error messages for the cause of the error. (=)


============

Fix: Check if your database is running. In our case the database instance had crashed.

- Aravind Kamath Posral

Autoconfig failing after reporting error with afcpctx.sh

We were running autoconfig in our R12.1.1/11gR2 environment as part of SSO integration step and we ran into this issue:
[AutoConfig Error Report]The following report lists errors AutoConfig encountered during each phase of its execution. Errors are grouped by directory and phase.The report format is:
[PROFILE PHASE] AutoConfig could not successfully execute the following scripts: Directory: /apps/local/TESIF/inst/apps/TESIF_oa1-ipls-35/admin/install afcpctx.sh INSTE8_PRF 1

Autoconfig failed with the above error.

As I started troubleshooting this issue, I noticed the following error in the autoconfig log:
Uploading Context file /apps/local/TESIF/inst/apps/TESIF_oa1-ipls-35/appl/admin/TESIF_oa1-ipls-35.xml
Context File upload for /apps/local/TESIF/inst/apps/TESIF_oa1-ipls-35/appl/admin/TESIF_oa1-ipls-35.xml failed.

Check log file created by FNDCPUCF program.

SQL*Plus: Release 10.1.0.5.0 - Production on Fri Jul 9 09:37:00 2010
Copyright (c) 1982, 2005, Oracle. All rights reserved.
Enter value for 1: Enter value for 2: Enter value for 3: Connected.
COUNT(*)---------- 14
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit ProductionWith the Partitioning, Real Application Clusters, OLAP, Data Miningand Real Application Testing options
Uploading Metadata file /apps/TESIF/apps/apps_st/appl/ad/12.0.0/admin/template/adctxinf.tmpERRORCODE = 1 ERRORCODE_ENDLog filename : L470980.log

When I checked the log L470980.log, it had the following error:
Oracle error -6502: ORA-06502: PL/SQL: numeric or value error has been detected in FND_GSM_UTIL.APPEND_CTX_FRAGMENT.

Now, if you look up metalink for the above error, it talks about two possible situations:
- Hostname + Domain Name > 30 characters
- Some bug in 11.5.10 and to apply a patch.

However, in our case, I did not think any of these would apply, as in the past autoconfig was running successfully in the past. So the focus was on recent patches which were applied to the system but one key factor helped me eliminate that possibility as well i.e., the last_ddl_time of FND_GSM_UTIL did not reflect a equal or later timestamp as that of the patches. Also, the $FND_TOP/patch/115/sql/AFCPGUTB.pls did not have the timestamp of the patch application time, which confirmed that no patch changed anything as far as this package body is concerned.

After some intense troubleshooting moments, I zeroed in on the real culprit:
< xxim_top oa_var="s_xximtop" oa_type="PROD_TOP" oa_enabled="FALSE">/apps/TESIF/apps/apps_st/appl/xxim/12.0.0< / xxim_top>
< xxcau_top oa_var="s_xxcautop" oa_type="PROD_TOP" oa_enabled="FALSE">/apps/TESIF/apps/apps_st/appl/xxcau/12.0.0< / xxcau_top>
The above entries were found in our xml which turned out to be the culprits. Actually these custom top entries should be present under $APPL_TOP/XCUSTOM.env (custom environment file) and not in the context file. Got rid of the two lines and the autoconfig works the way we knew it in the past!
- Aravind Kamath Posral