Thursday, December 16, 2010

Routine AFPGRQ encountered an ORACLE error. ORA-01455: converting column overflows integer datatype

Issue:

Standard Managers not coming up. All other managers are coming up and are running fine.

Details:

Our standard managers were not coming up where as other managers were up and running. The standard manager log file had the following error logged:

Routine AFPGRQ encountered an ORACLE error. ORA-01455: converting column overflows integer datatype


Troubleshooting:

In our instance, we had 10 standard managers defined as part of the default workshift. Also, the packages used by AFPGRQ routine were the correct ones and in sync with the appl_top. So no reason to suspect any changes there.

However, there was one request pending on Standard Managers while viewing the "Administer Concurrent Managers" form. From the Administer Concurrent Managers form, queried Standard Managers and in that screen, clicked on the "Requests" button. The form did not show the details of the concurrent request.

So changed focus to fnd_concurrent_requests table and here is what I found:

sql> select request_id,phase_code,status_code from fnd_Concurrent_requests where phase_code='P';

This query returned 108 rows of which one was having the status_code "I".

So, narrowing down, here is what further investigation revealed:

SQL> select request_id,phase_code,status_code,requested_by,CONCURRENT_PROGRAM_ID from fnd_Concurrent_requests where request_id=1012680;

REQUEST_ID P S REQUESTED_BY CONCURRENT_PROGRAM_ID
---------- - - ------------ ---------------------
   1012680 P I   3.4535E+14                101330


Here, REQUESTED_BY column should be a valid id (number) which represents the user who submitted the request.

As you can see, some junk value (large integer value) was entered  which was causing the concurrent managers to crash. Now, this phenonmenon may not be unique to Standard Manager and can happen with any manager which is assigned to pick up such erroneous requests.

Querying this requests from the "Run Requests" form also would not yield any result as the form was unable to handle this condition.

Now, to overcome the issue, I ran the following update:

SQL> update fnd_concurrent_Requests set Phase_Code='C', status_Code='E', requested_by=1145 where request_id=1012680;
SQL> commit;

The requested_by id 1145 is a valid user from fnd_user table as identified by user_id column.


Then, from the administer managers screen, clicked on restart for standard managers and the managers came up clean.


Further you can identify the program in the following way:


select concurrent_program_id from fnd_concurrent_requests where request_id=1012680 ;


select user_concurrent_program_name from fnd_concurrent_programs_tl where concurrent_program_id=101330;


From this you can identify a set of users who ran this program in the past and inform them to be careful while submitting the requests programatically.

- Aravind Kamath Posral

Thursday, October 21, 2010

IBY_0001 Communication error

Our application team was trying to setup ipayment application using cybersource payment gateway solution and they reported that they encountered the following error:

IBY_0001 Communication error


Upon investigation, we found the following:

In zone.properties, following is the configuration defined for cybersource servlet. At the time of initialization, the oacore jvm uses these initialization arguments to init the servlet.

servlet.oramipp_cys.initArgs=cybsconfigfile=/usrs/cybadm/tsqc/config.xml,cybsmapfile=/usrs/cybadm/tsqc/mapping.xml

In the oacore jvm logs ($IAS_CONFIG_HOME/Apache/Jserv/logs/jvm/OACoreGroup.0.stderr / stdout), I found the following error recorded:


: XML-0108: (Fatal Error) Start of root element expected.
com.cybersource.client.integration.foundation.exception.ConfigurationException: The file {0} is missing or unreadable.at com.cybersource.client.integration.foundation.ServiceRegistry.checkFileReadability ServiceRegistry.java:315)
at com.cybersource.client.integration.foundation.ServiceRegistry.registerXMLConfigManager(ServiceRegistry.java:65)
at com.cybersource.client.integration.iPayment.oramipp_cys.init(oramipp_cys.java:50)
at org.apache.jserv.JServServletManager.load_init(JServServletManager.java:755)
at org.apache.jserv.JServServletManager.loadServlet(JServServletManager.java:659)
at org.apache.jserv.JServConnection.processRequest(JServConnection.java:394)
at org.apache.jserv.JServConnection.run(JServConnection.java:294)
at java.lang.Thread.run(Thread.java:479)


The reason why it failed was because oacore process was not able to access the cybersource config files.


/usrs> ls -ltr /usrs/cybadm/tsqc/config.xml

ls: /usrs/cybadm/tsqc/config.xml: Permission denied


For security purposes, the files related to payment gateway are owned by user called cybadm and read only permissions are given to the members of the group cyb.

Now, the oacore process is owned by apps file system owner (oatsqc) and this user was not added to cyb group.

We added oatsqc user to cyb group and then bounced oacore to overcome this issue.

- Aravind Kamath Posral

Tuesday, October 5, 2010

Getting Blank Page when accessing AppsLocalLogin.jsp

Recently, we were reported that the user was unable to login to an R12 instance.
The environment was an R12 bulit on database 11gR2 and it was integrated with SSO as well.


Issue Description:
Getting a blank page after entering the login credentials at the SSO page when accessing the application URL.


Symptoms:
We tried first to check if the application was working fine without having to go through the SSO and hence we tried the AppsLocalLogin.jsp page (http://deif.isc.com:8043/OA_HTML/AppsLocalLogin.jsp). The link was leading to a blank page.
Nextly, When we tried the logging to the application through SSO (http://deif.isc.com:8043/OA_HTML/AppsLogin), the SSO page was loading and we were able to enter the login credentials and after that it was again giving a blank page. When we looked at the browser URL we again found that the it was pointing to the AppsLocalLogin.jsp after we authenticate against the SSO


Troubleshooting Steps:
Firstly, we tried to check the services and found it all to be up and running.
Nextly we turned to the access and error logs for the HTTP_Server and found no hints in there.
We also tried to recompile the jsp using the ojspcompile.pl (perl $FND_TOP/bin/ojspcompile.jsp --compile --flush --p 2)
But this also did not help.


We check to find if any invalids objects were present in the database and found that no new invalids were present.


Then further looking into the logs, we found the below error in the oacore log (application.log) at location
$LOG_HOME/ora/10.1.3/j2ee/oacore/oacore_default_group_
10/10/04 01:57:06.314 html: chain failed javax.servlet.ServletException: java.lang.RuntimeException: Unable to create user session. Please contact your System Administrator.
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.EvermindPageContext.handlePageThrowable EvermindPageContext.java:899) at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.EvermindPageContext.handlePageException EvermindPageContext.java:816)
at _AppsLocalLogin._jspService(_AppsLocalLogin.java:303)
at com.orionserver[Oracle Containers for J2EE 10g (10.1.3.5.0) ].http.OrionHttpJspPage.service(OrionHttpJspPage.java:59)
at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:473)
at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:594)
at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:518)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)



This prompted us to check alert log which had these errors recorded
ORA-1653: unable to extend table ICX.ICX_SESSIONS by 16 in tablespace APPS_DATA
Mon Oct 04 02:21:33 2010


It turned out to be a tablespace issue. Whenever a login process happens it is recorded in the ICX_SESSIONS table as GUEST user is making a connection when the AppsLocalLogin.jsp pages gets loaded even before it asks for the login credentials. Once the login credentials is made a record is made in the table against the user id used to login.


Once we added the space to the tablespace in problem here, the page AppsLocalLogin.jsp loaded without issues and were able to login.


- Tanveer Madan

Friday, September 24, 2010

How to obtain Oracle Inventory for your Oracle Home

Hello,

Here is a "how to" post.

Requirement:

You have accidentally deleted the oracle inventory or the oracle inventory is corrupt. You need the oracle inventory to be able to apply some patch using Opatch.

How to obtain your inventory back:

a. If your database is RAC:

$ORACLE_HOME/oui/bin/runInstaller -attachHome -local -invPtrLoc /var/opt/oracle/oraInst.loc ORACLE_HOME="/oracle/product/10.2.0.4/ORADB"
ORACLE_HOME_NAME="oradb_home_10_2_0_4" "CLUSTER_NODES={db-host-01,db-host-02}" LOCAL_NODE="db-host-01"

The above command will only update the inventory on the local node (where the command is being run) without affecting the other nodes in the cluster.

b. If your database is non-RAC:

$ORACLE_HOME/oui/bin/runInstaller -attachHome -invPtrLoc /var/opt/oracle/oraInst.loc ORACLE_HOME="/oracle/product/10.2.0.4/ORADB" ORACLE_HOME_NAME="oradb_home_10_2_0_4"


I believe this should work well with 11g db also.

Disclaimer: Please make sure to test these commands thoroughly if you plan to use these in your environment. These commands have worked well for me in the past, but you should use your judgement if you plan to use them in your environment. Ultimately, you are responsible for the outcome! :)


- Aravind Kamath Posral

The system property CV_NODE_ALL has not been set to the static nodelist

Hello Readers,

I think last couple months have been quiet ones for appsdbatechstuff with not much posts from my side. Here is an issue we faced while setting up CRS. The overall scope of the activity was 11g GRID Infrastructure Installation and a 11gR2 code tree setup for a two-node RAC system.

While running the cluster verify after the CRS installation, we faced the following error:

$cluvfy stage -post crsinst -n all -verbose

Performing post-checks for cluster services setup
The system property CV_NODE_ALL has not been set to the static nodelist
Verification cannot proceed

$


Fix:

Make sure your /var/opt/oracle/oraInst.loc is pointing to the GRID_HOME inventory and not the database code tree inventory. Re run the cluster verify utility.



- Aravind Kamath Posral

Saturday, July 31, 2010

Configuration assistant "BPEL Configuration Assistant" failed

Recently we were in the process of upgrading BPEL from 10.1.3.3 to 10.1.3.5. During the process we fcaed the following error when the configuration assistants were running:

apps/st1bfi/product/AS10gR3_BPEL/j2ee/BPEL_OC4J_FIB/applications/orabpel.ear

[java] 10/06/28 08:38:22 Notification ==>Initialize /apps/st1fib/product/AS10gR3_BPEL/j2ee/BPEL_OC4J_FIB/applications/orabpel.ear begins...

[java] 10/06/28 08:38:22 Notification ==>Operation failed with error:
[java] Unable to find/read file META-INF/application.xml in /apps/st1fib/product/AS10gR3_BPEL/j2ee/BPEL_OC4J_FIB/applications/orabpel (META-INF/application.xml)


[java] Redeploy error: Redeploy failed: Operation failed with error:
[java] Unable to find/read file META-INF/application.xml in /apps/st1fib/product/AS10gR3_BPEL/j2ee/BPEL_OC4J_FIB/applications/orabpel (META-INF/application.xml)


BUILD FAILED
/apps/st1fib/product/AS10gR3_BPEL/bpel/system/services/install/ant-tasks/redeploy.xml:51: The following error occurred while executing this line:
/apps/st1fib/product/AS10gR3_BPEL/bpel/system/services/install/ant-tasks/redeploy.xml:64: The following error occurred while executing this line:
/apps/st1fib/product/AS10gR3_BPEL/bpel/system/services/install/ant-tasks/redeploy.xml:92: The following error occurred while executing this line:
/apps/st1fib/product/AS10gR3_BPEL/bpel/system/services/install/ant-tasks/redeploy-common.xml:80: The following error occurred while executing this line:
/apps/st1fib/product/AS10gR3_BPEL/bpel/system/services/install/ant-tasks/redeploy-common.xml:152: Java returned: 1

Total time: 6 seconds
Configuration assistant "BPEL Configuration Assistant" failed


While I have strong reasons to believe that this was caused due to the oracle home being on the NFS mounted file system, we had an urgent requirement to deliver the instance to the business. We had to deliver the instance under two hours and we had to crack this issue.


There are three main parts to this apps filesystem upgrade part of BPEL upgrade:

1. Applying the patchset 10.1.3.5
2. Configuration assistants run subsequently
3. Run root.sh and exit the installer.


The aforesaid issue you get when the configuration assistant is running and there were no issues reported by the installer in the earlier stage. To fix this issue, we wanted to figure out what were the sequence of events performed by the configuration assistant (roughly, highlighting main events which i thought were important):


- Patch brings in new jar files
- Patch also updates the file $ORACLE_HOME/bpel/system/services/lib/orabpel.ear
- The configuration assistant checks server.xml to see if this application (orabpel.ear is present).
- If it is present, then it deletes the corresponding line in server.xml ($ORACLE_HOME/j2ee/BPEL_OC4J_FIB/config/server.xml) and the proceeds with the actual deployment.
- The configuration assistant then copies the above orabpel.ear file to $ORACLE_HOME/j2ee/BPEL_OC4J_FIB/applications
- The configuration assistant then extracts orabpel.ear file
- If application-deployment subfolder for this application is created if it is not present.
- The server.xml ($ORACLE_HOME/j2ee/BPEL_OC4J_FIB/config/server.xml) file is again updated with this entry.


Here is how we fixed the issue:

- When the configuration assistant failed, we clicked on the retry button.
- This time the assistant failed with the error that ORABPEL application not found.
- The reason is, from the first iteration, the entry for orabpel was deleted in server.xml.
- Make the entry for orabpel in the server.xml like ""
- Copy the orabpel.ear from $ORACLE_HOME/bpel/system/services/lib/orabpel.ear to /apps/st1fib/product/AS10gR3_BPEL/j2ee/BPEL_OC4J_FIB/applications
- Click the retry button, it was successful.


- Aravind Kamath Posral

Monday, July 12, 2010

ORA-12514: Message 12514 not found; product=RDBMS; facility=ORA

Here is a quick one..

Concurrent managers (workers) not coming up but FNDSM is up. FNDSM log has following errors:

Routine AFPCMT encountered an ORACLE error. ORA-01012: not logged on
.

Review your error messages for the cause of the error. (=)

_ 2 _
Routine AFPSMG encountered an ORACLE error. ORA-03114: not connected
to ORACLE
.

Review your error messages for the cause of the error. (=)

_ 3 _
Routine AFPCSQ encountered an ORACLE error. ORA-12514: Message 12514
not found; product=RDBMS; facility=ORA

.

Review your error messages for the cause of the error. (=)


============

Fix: Check if your database is running. In our case the database instance had crashed.

- Aravind Kamath Posral

Autoconfig failing after reporting error with afcpctx.sh

We were running autoconfig in our R12.1.1/11gR2 environment as part of SSO integration step and we ran into this issue:
[AutoConfig Error Report]The following report lists errors AutoConfig encountered during each phase of its execution. Errors are grouped by directory and phase.The report format is:
[PROFILE PHASE] AutoConfig could not successfully execute the following scripts: Directory: /apps/local/TESIF/inst/apps/TESIF_oa1-ipls-35/admin/install afcpctx.sh INSTE8_PRF 1

Autoconfig failed with the above error.

As I started troubleshooting this issue, I noticed the following error in the autoconfig log:
Uploading Context file /apps/local/TESIF/inst/apps/TESIF_oa1-ipls-35/appl/admin/TESIF_oa1-ipls-35.xml
Context File upload for /apps/local/TESIF/inst/apps/TESIF_oa1-ipls-35/appl/admin/TESIF_oa1-ipls-35.xml failed.

Check log file created by FNDCPUCF program.

SQL*Plus: Release 10.1.0.5.0 - Production on Fri Jul 9 09:37:00 2010
Copyright (c) 1982, 2005, Oracle. All rights reserved.
Enter value for 1: Enter value for 2: Enter value for 3: Connected.
COUNT(*)---------- 14
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit ProductionWith the Partitioning, Real Application Clusters, OLAP, Data Miningand Real Application Testing options
Uploading Metadata file /apps/TESIF/apps/apps_st/appl/ad/12.0.0/admin/template/adctxinf.tmpERRORCODE = 1 ERRORCODE_ENDLog filename : L470980.log

When I checked the log L470980.log, it had the following error:
Oracle error -6502: ORA-06502: PL/SQL: numeric or value error has been detected in FND_GSM_UTIL.APPEND_CTX_FRAGMENT.

Now, if you look up metalink for the above error, it talks about two possible situations:
- Hostname + Domain Name > 30 characters
- Some bug in 11.5.10 and to apply a patch.

However, in our case, I did not think any of these would apply, as in the past autoconfig was running successfully in the past. So the focus was on recent patches which were applied to the system but one key factor helped me eliminate that possibility as well i.e., the last_ddl_time of FND_GSM_UTIL did not reflect a equal or later timestamp as that of the patches. Also, the $FND_TOP/patch/115/sql/AFCPGUTB.pls did not have the timestamp of the patch application time, which confirmed that no patch changed anything as far as this package body is concerned.

After some intense troubleshooting moments, I zeroed in on the real culprit:
< xxim_top oa_var="s_xximtop" oa_type="PROD_TOP" oa_enabled="FALSE">/apps/TESIF/apps/apps_st/appl/xxim/12.0.0< / xxim_top>
< xxcau_top oa_var="s_xxcautop" oa_type="PROD_TOP" oa_enabled="FALSE">/apps/TESIF/apps/apps_st/appl/xxcau/12.0.0< / xxcau_top>
The above entries were found in our xml which turned out to be the culprits. Actually these custom top entries should be present under $APPL_TOP/XCUSTOM.env (custom environment file) and not in the context file. Got rid of the two lines and the autoconfig works the way we knew it in the past!
- Aravind Kamath Posral

Wednesday, June 16, 2010

java.sql.SQLException while registering Ebiz with SSO

Hi
Recently while trying to integrate an EBiz R12 instance with OID for Single Sign On functionality we hit an while running the following command:
$FND_TOP/bin/txkrun.pl -script=SetSSOReg -registersso=yes -appspass=apps
Fri Jun 11 03:47:39 PDT 2010 =========================================
Fri Jun 11 03:47:39 PDT 2010 OracleAS Single Sign-On Registration Tool
Fri Jun 11 03:47:39 PDT 2010 =========================================
Fri Jun 11 03:47:39 PDT 2010 Parameters passed to SSO registration tool :
param0:-oracle_home_path
param1:/cdv1001/apps/tech_st/10.1.3
param2:-site_name param
3:http://sord142.corp.com:8010
param4:-config_mod_osso param
5:true param6:-logout_url param7:http://sord142.corp.com:8010/OA_HTML/AppsLogoutSSO
param8:-home_url
param9:http://sord142.corp.com:8010
param10:-config_file param11:/cdv1001/inst/apps/cdv_sord142/ora/10.1.3/Apache/Apache/conf/osso/cdv_sord142_osso.conf param12:-update_mode
param13:CREATE
param14:-success_url
param15:http://sord142.corp.com:8010/osso_login_success
param16:-cancel_url
param17:http://sord142.corp.com:8010
Fri Jun 11 03:47:39 PDT 2010 -DinstallType=
Fri Jun 11 03:47:39 PDT 2010 -DoldOracleHome=Fri Jun 11 03:47:39 PDT 2010 -DoldOHSUser=root
Fri Jun 11 03:47:39 PDT 2010 SSO database is jdbc:oracle:thin:@ldap://ldap-imdvrdc.domain.com:3051/idmdvrdc,cn=oraclecontext
Fri Jun 11 03:47:40 PDT 2010 Exception while creating database connection :java.sql.SQLException: Io exception: The Network Adapter could not establish the connectionjava.sql.SQLException: Io exception: The Network Adapter could not establish the connection at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:125) at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:162) at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:274) at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:328) at oracle.jdbc.driver.PhysicalConnection.(PhysicalConnection.java:361) at oracle.jdbc.driver.T4CConnection.(T4CConnection.java:151) at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32) at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:595) at java.sql.DriverManager.getConnection(DriverManager.java:525) at java.sql.DriverManager.getConnection(DriverManager.java:171) at oracle.security.sso.SSORegistrar.main(SSORegistrar.java:327)Fri Jun 11 03:47:40 PDT 2010 Check if your database is up and running.

Fix:
The database was up and running and there were no issues. However, in this case, the ebiz server was in a different domain than that of our OID environment and we had opened specific ports between the environments to allow access to the OID servers. When we performed telnet test on database listener port, everything was fine. However, when trying to register the error was showing up.

Further investigation revealed that oraclecontext was was pointing to database name instead of databasename.domain.com. In our telnet tests we were using fully qualified hostname to test the access. To over come this issue, we put an entry in /etc/hosts of the web server of ebiz environment as follows:

123.45.67.89 dvdb.domain.com dvdb

In the above entry, first entry is the IP address of the database host of the OID and subsequent entries are the databasehost.domain.com and databasehost values.

This resolved the issue.

- Aravind Kamath Posral

Monday, June 14, 2010

FNDOPP24553.txt:usdsop: exec failed during spawn/apps/local/TS10EF/inst/apps/TS10EF_'uname -n'/admin/scripts/java.sh: No such file or directory

Hi,
we had just cloned an R12.1.1 apps instance. In the target instance, TS10EF, the Output Post Processor and Workflow Background Listener managers were not coming up. The log of the OPP manager had the following error:
FNDOPP24553.txt:usdsop: exec failed during spawn/apps/local/TS10EF/inst/apps/TS10EF_'uname -n'/admin/scripts/java.sh: No such file or directory

Fix:
The issue is caused by incorrect AFJVAPRG environment variable setting. It should point to $ADMIN_SCRIPTS_HOME/java.sh. However, in our case, there was 'uname -n' in the environment variable which was causing this issue. To fix this, make the change in xml file for AFJVAPRG environment variable to reflect the context name (TS10EF_istg03) and run autoconfig. If you dont wish to run autoconfig, you can make the change in xml and also change AFJVAPRG the follow env file:
/apps/ts10ef/apps/apps_st/appl/istg03.env
Once the fix is done, log out , log in, source the env and start the concurrent managers. This should resolve the issue. However, in our case, it did not. In
the administer concurrent managers screen, we were seeing the following error:
"System Hold, Fix Manager before resetting counters "

To over come this issue, shutdown concurrent managers, run cmclean.sql (or basically set the process codes right) and start the managers. The problem would be fixed.

- Aravind Kamath Posral

Tuesday, May 18, 2010

Configuration assistant "Oracle Web Services Inspection Language Configuration Assistant" failed

This post is on an issue which we faced during the BPEL installation (10.1.3.1.0). While installing the 10gAS J2ee Component we got the following error at the end of the installation where the Universal Installer runs the various configuration assistants:
=======================================================
Launched configuration assistant 'Oracle Web Services Inspection Language Configuration Assistant'------------------------------------------------
Tool type is: Recommended.The command being spawned is: '/apps/dvbef/product/AS10gR3_BPL/ant/bin/ant -buildfile
/apps/dvbef/product/AS10gR3_BPL/webservices/lib/wsil-install.xml -logfile
/apps/dvbef/product/AS10gR3_BPL/cfgtoollogs/wsil.txt -DHOST=0:0:0:0:0:0:0:1 -DOPMNPORT="6029" - DADMIN_USER=oc4jadmin -DOPMNINSTANCE=ADMIN_OC4J -Denv.JAVA_HOME=/apps/dvbef/product/AS10gR3_BPL/jdk -
Denv.ANT_HOME=/apps/dvbef/product/AS10gR3_BPL/ant -Denv.ORACLE_HOME=/apps/dvbef/product/AS10gR3_BPL *Protected
value, not to be logged*'
Buildfile: /apps/dvbef/product/AS10gR3_BPL/webservices/lib/wsil-install.xml
Configuration assistant "Oracle Web Services Inspection Language Configuration Assistant" failed
------------------------------------------------Launched configuration assistant 'consoleOnExit'------------------------------------------------
Tool type is: Optional.The command being spawned is: ' '
Configuration assistant "consoleOnExit" failed
===============================================================================
The wsil.txt under $ORACLE_HOME/cfgtoollogs reported the following issue:
init: [echo] Initializing Variables
install-wsil:
install-wsil_nonjsso: [echo] Deploying WSIL App with JSSO disabled
[java] Failed at "Could not get DeploymentManager".
[java] This is typically the result of an invalid deployer URI format being supplied, the target server not being in a started state or incorrect authentication details being supplied.
[java] More information is available by enabling logging -- please see the Oracle Containers for J2EE Configuration and Administration Guide for details.

BUILD FAILED/apps/dvbef/product/AS10gR3_BPL/webservices/lib/wsil-install.xml:29: The following error occurred while executing
this line:/apps/dvbef/product/AS10gR3_BPL/webservices/lib/wsil-install.xml:35: Java returned: 1
===================================================================================
Fix: The culprit turned out to be the /etc/hosts file where the entry looked like this:
127.0.0.1 localhost localhost.domain ifdv-11 ifdv-11.acme.com
The entry should look like this:
127.0.0.1 localhost localhost.domain
172.168.10.4 ifdv-11 ifdv11.acme.com
Once you get the above entry, kill any running processes from the oracle home and delete the BPEL oracle home and restart the installation using OUI.
- Aravind Kamath Posral

Wednesday, April 28, 2010

ORA-10643: Database should be mounted in restricted mode and Exclusive mode

Hi,
Recently, we ran into the following issue while converting a system tablespace from dictionary managed to locally managed:
SQL> EXEC dbms_space_admin.tablespace_migrate_to_local('SYSTEM');
BEGIN dbms_space_admin.tablespace_migrate_to_local('SYSTEM'); END;
*ERROR at line 1:
ORA-10643: Database should be mounted in restricted mode and Exclusive mode
ORA-06512: at "SYS.DBMS_SPACE_ADMIN", line 227
ORA-06512: at line 1

Our database was a RAC database and we were trying to convert the system tablespace to locally managed. The prerequisite for such an activity is that the database should be started in restricted mode and all tablespaces other than system, temp, undo and sysaux be in read only mode. SYSAUX should be offline. We started the first instance and executed the command to initiate the conversion. However, the conversion failed with the above error.

We set the cluster_database value to false and restarted the conversion process. Thanks to my colleague, M Rajesh Kumar for his timely help and suggestion which saved us lot of time. This time around, we did not see the original error, instead a new error cropped up:

ERROR at line 1:
ORA-10647: Tablespace other than SYSTEM, UNDO, TEMP not found in read only mode
ORA-06512: at "SYS.DBMS_SPACE_ADMIN", line 227
ORA-06512: at line 1

All the tablespaces other than system,temp,undo were in read only mode and sysaux was in offline mode. That is when we realized, we have two UNDO tablespaces, one for each instance. UNDO_TS2, the undo tablespace used by other instance was also online. We cannot make that tablespace read only as that is an UNDO tbs. We tried to complete the conversion by making UNDO_TS2 offline, but it did not succeed - we continued to see the same error.

Fix:

Make note of the details of the UNDO_TS2 like data files/volumes used, auto extend on/off etc and drop UNDO_TS2 from the first instance. Perform the system tablespace conversion from Dictionary managed to LMTS. Recreate the UNDO_TS2 using the details which you have taken before dropping the tbs.

PS: I have not tried to give the exact steps involved in the LMTS conversion. The steps include things like dropping multiple temporary tablespaces, working with default temporary tablespace settings, tablespace groups etc, which we followed.

-Aravind Kamath Posral

Friday, April 16, 2010

/usr/lib/hpux64/dld.so: Unable to find library 'libsqlplus.so'

Hi,
This is a real short post - cant type too much with a fractured right hand....:)
Issue:
Following error while executing shutdown command using sqlplus:
/usr/lib/hpux64/dld.so: Unable to find library 'libsqlplus.so'
This error might pop up during other operations/commands with sqlplus also.
Fix:
Check LD_LIBRARY_PATH and SHLIB_PATH
It should point to $ORACLE_HOME/lib:$ORACLE_HOME/lib32
set it accordingly and the issue is fixed.
- Aravind Kamath Posral

Tuesday, March 23, 2010

TNS 12521: No message file for product=RDBMS, facility=ORA

Hi,
We came across an interesting issue over the weekend. After the cold backup of the database, we were not able to connect to the db from any of the apps front end servers using sqlplus. Due to this all the FE services including concurrent managers were down. Most recently we had performed some autoconfig related maintenance activity. We were seeing the following error message while trying to connect from sqlplus:
ORA-12521 No message file for product=RDBMS, facility=ORA

Following is the error logged in sqlnet.ora file:
Fatal NI connect error 12514, connecting to: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=trm.domain.com)(PORT=1682))(CONNECT_DATA=SERVICE_NAME=kvprod.domain.com)(INSTANCE_NAME=kvprod)(CID=(PROGRAM=)(HOST=11-am-11)(USER=oakvprod))))
VERSION INFORMATION:
TNS for Linux: Version 8.0.6.0.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 8.0.6.3.0 - Production Time: 21-MAR-10 09:56:41 Tracing not turned on.
Tns error struct:
nr err code: 12204
TNS-12204: TNS:received data refused from an application
ns main err code: 12564
TNS-12564: TNS:connection refused ns secondary err code: 0 nt main err code: 0 nt secondary err code: 0 nt OS err code: 0

Following is the error in listener log file on db host:
21-MAR-2010 08:15:42 * (CONNECT_DATA=(SERVICE_NAME=kvprod.domain.com)(INSTANCE_NAME=kvprod)(CID=(PROGRAM=)(HOST=11-am-11)(USER=oakvprod))) * (ADDRESS=(PROTOCOL=tcp)(HOST=145.254.224.149)(PORT=54806)) * establish * KVPROD.domain.com * 12521

The lsnrctl status command showed the following output:
LSNRCTL> Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=trm.domain.com)(PORT=1682))(CONNECT_DATA=(SID=KVPROD)))
STATUS of the LISTENER------------------------Alias KVPROD
Version TNSLSNR for HPUX: Version 9.2.0.7.0 - Production
Start Date 21-MAR-2010 10:01:27Uptime 0 days 0 hr. 30 min. 35 secTrace Level offSecurity OFFSNMP OFFListener Parameter File /oracle/product/9.2.0.7/KVPROD/network/admin/KVPROD_trm/listener.ora
Listener Log File /oracle/product/9.2.0.7/KVPROD/network/admin/KVPROD.logListening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROCKVPROD))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=trm.domain.com)(PORT=1682)))
Services Summary...
Service "KVPROD" has 1 instance(s). Instance "KVPROD", status UNKNOWN, has 1 handler(s) for this service...
Service "KVPROD.domain.com" has 1 instance(s). Instance "/usr/tools/oracle/scripts/KVPROD", status READY, has 1 handler(s) for this service...
Service "PLSExtProc" has 1 instance(s).
Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully

That is where we observed that the Instance registered with the listener was reflecting as /usr/tools/oracle/scripts/KVPROD.

Further troubleshooting led me to figure out that instance name parameter was reflecting as /usr/tools/oracle/scripts/KVPROD.

SQL> show parameter instance_name
/usr/tools/oracle/scripts/KVPROD

Further analysis revealed that this anomaly is due to the environment variable called ORACLE_PATH. If the ORACLE_PATH is set and if there is a file or folder by the same name as ORACLE_SID, then oracle apparently concludes on instance_name as $ORACLE_PATH$ORACLE_SID. That, in the absence of instance_name init parameter set explicitly in init file. If instance_name init parameter is set, this parameter takes precedence over the calculated value.

In our case, we did not have the instance_name parameter set in init file and we had a folder by the same name as that of ORACLE_SID in the oracle_path which led to this issue.
Fix:
1. Set instance_name parameter explicitly
OR
2. unset ORACLE_PATH
OR
3. Do not have a file/folder by the same name as that of oracle_sid in oracle_path

Why this setting took effect after the last bounce, why not in the past? - always the last maintenance activity is a suspect. It may have brought in the oracle_path variable. Just that file/folder with name same as that of sid not being there in any dev/test envs but being present on PROD is good enough to nullify hours and hours of trial runs which happen before we make the change in production!!
- Aravind Kamath Posral

Thursday, March 11, 2010

IBY_0001 Communication error. The payment system, the processor, or iPayment electronic commerce servlet is not available/accessible.

IBY_0001 Communication error. The payment system, the processor, or iPayment electronic commerce servlet is not available/accessible. Please make sure you have the correct system set up and resubmit the request at a later time.

Problem:
In our Oracle applications instance R11i (11.5.8), we applied some patches as part of some maintenance activity. Subsequently we ran autoconfig and then, the users complained that their iPayment credit card authorizations were failing with the aforesaid error.

Troubleshooting and Fix:
I am aware that the error message is generic in nature and can happen due to a variety of reasons including misconfiguration of payment gateway related class files in jserv.properties / zone.properties / System setup using iPayment Administrator responsibility and so on. However, in our case, we did not have a reason to suspect those factors as the change that happened was we having applied some patches and DBA related maintenance activities.

The first thing to do in this kind of a situation is to enable FND Debug Logs. Specify a location for the log file and provide the module which needs to be traced, in our case IBY.
Second thing is enable AF logs by having debug related entries in jserv.properties:
wrapper.bin.parameters=-DAFLOG_ENABLED=TRUE
wrapper.bin.parameters=-DAFLOG_MODULE=iby%
wrapper.bin.parameters=-DAFLOG_LEVEL=STATEMENT
wrapper.bin.parameters=-DAFLOG_FILENAME=/tmp/fndLog10Mar.log

However, I believe it is a good option to enable debug logging in jserv by editing jserv.properties and editing log related parameters :
log=truelog.file=[full path to jserv.log]
log.timestamp = true
log.channel.debug=true
Dont forget to enable debug logging in httpd.conf. This should also be able to trap the error.

The above debug settings resulted in the following error stack (Partial Error Stack):
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:350)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:137)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:124)
at java.net.Socket.(Socket.java:268)
at java.net.Socket.(Socket.java:95)
at oracle.apps.iby.payment.PmtAdapter._oraMippSocket(PmtAdapter.java:982)
at oracle.apps.iby.payment.VendorAdapter.oraMipp(VendorAdapter.java:146)
at oracle.apps.iby.payment.VendorAdapter.oraDispatch(VendorAdapter.java:396)
at oracle.apps.iby.payment.VendorAdapter.oraDispatch(VendorAdapter.java:328)

This resulted in analyzing why the connection refused error was showing up.
It turns out that the parameter "Listen" in httpd.conf got updated due to new templates brought in by the patch.

Before patch it was :
Listen 8001
After patch it was:
Listen [Hostname.domain.com]:8001

This was the cause of this issue.

When you have the Listen 8001 directive, the apache listens on 127.0.0.1. However, when Listen [Hostname.domain.com]:8001 is configured, then it listens on the hostname. So if the iPayment uses a load balancer URL, it will not be able to connect because in the /etc/hosts file on the physical host, the entry will look like:
127.0.0.1 http://www.qcprod.acme.com/
So, when the loop back connection from the host tries to connect on 127.0.0.1, because apache is no longer listening on 127.0.0.1 and hence it sees connection refused error, which results in the original error message thrown by iPayment. The fix is to change the Listen parameter back to original value.
- Aravind Kamath Posral

Thursday, February 25, 2010

ORA-20001: Unabled to call fnd_ldap_wrapper.create_user

Hi,
Recently we integrated our R12 instance with SSO/OID and we used provisiontype 4, which means no bidirectional flow of users. Subsequently when we tried to create users in R12 (not OID) we encountered the following error:
ORA-20001: APP-FND-02600: Unable to create user ALISBT due to the following reason(s):
ORA-20001: Unabled to call fnd_ldap_wrapper.create_user due to the followingreason:An unexpected error occurred. Please contact your System Administrator...

Fix:

When you have provision type 4 setup in your environment, you should set the following profile option correctly:

Application SSO LDAP Synchronization ==> DISABLED

In our environment it was set to enabled by default, due to which we faced the issue. Set the aforesaid provide option to disabled at site level and you will be able to create users successfully.

PS: We were able to overcome this issue quickly with help of my colleague and friend Srinivasan K, who is an expert in R12 SSO integration and I must acknowledge his excellent contribution here. Thanks Srini!

- Aravind Kamath Posral

Thursday, February 4, 2010

How to use non-default inventory location pointer with opatch

Hi,
This post again, is not a "fix-a-problem" kind of post. This is more to do with how to work around a tight spot and hence dont be surprised to see the disclaimer at the end!

If you are in a situation where you need to use a inventory location pointer from a non-default location, you can use the following command:

a. to query:
opatch lsinventory -invPtrLoc [full path to oraInst.loc]
eg. opatch lsinventory -invPtrLoc /apps/sdev/apps/tech_st/10.1.2/oraInst.loc

b. to apply a patch:

opatch apply -invPtrLoc [full path to oraInst.loc]
eg. opatch apply -invPtrLoc /apps/sdev/apps/tech_st/10.1.2/oraInst.loc

Just to add, the default location of the oraInst.loc file will be /etc/oraInst.loc and/or /var/opt/oracle/oraInst.loc. You might encounter a situation wherein there are multiple oracle homes on the same server and multiple maintenance activities are happening in parallel, in which case, if your oraInst.loc at non-default location has the correct location of the oracle inventory of the desired oracle home, you have the flexibility of using the same.

Further, if you dont have an inventory, you can create one using the following steps:

1. Define an inventory location for the new inventory information to be created.
2. Create the empty oraInventory directory with the appropriate shell username (who owns the oracle home file system) in the path specified in oraInst.loc

3. $./runInstaller -silent -attachHome -invPtrLoc /var/opt/oracle/oraInst.loc ORACLE_HOME="[full_path_to_ORACLE_HOME]" ORACLE_HOME_NAME="[ORACLE_HOME_NAME]"

DISCLAIMER: The above steps have worked well for me in various situations in the past, but I cannot guarantee that this will work well in all situations. Hence, please use your discretion if you plan to use these commands. You will be doing the same at your own risk. Have adequate back-out measures in place before you try the aforesaid steps in particular and any step from any of my other posts in general :)

- Aravind Kamath Posral

Monday, January 25, 2010

RC-50004: Fatal: Error occurred in ApplyAppsTier

Recently while cloning an RAC enabled R12 instance, we faced the aforesaid mentioned error while running adcfgclone.pl on apps tier towards 79% completion. There was a generic java error stack accompanying the error RC-500004, which I did not capture and hence I cannot share that here. However, while investigating the issue, I noticed the following error in $INST_TOP/admin/log/ApplyAppsTier_[timestamp].log.

java.sql.SQLException: Listener refused the connection with the following error: ORA-12514, TNS:listener does not currently know of service requested in connect descriptor The Connection descriptor used by the client was: (DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=isg-db-01-vip.domain.com)(PORT=1612)))(CONNECT_DATA=(SERVICE_NAME=estg)))

Database connection to jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=isg-db-01-vip.domain.com)(PORT=1612)))(CONNECT_DATA=(SERVICE_NAME=estg))) failed UPDATE call failed with exit code 1 Updating parameters: fnd_jdbc_buffer_min=1 fnd_jdbc_buffer_max=5 fnd_jdbc_buffer_decay_interval=300 java.sql.SQLException: Listener refused the connection with the following error: ORA-12514, TNS:listener does not currently know of service requested in connect descriptor The Connection descriptor used by the client was: (DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=isg-db-01-vip.domain.com)(PORT=1612)))(CONNECT_DATA=(SERVICE_NAME=estg)))

Database connection to jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=isg-db-01-vip.domain.com)(PORT=1612)))(CONNECT_DATA=(SERVICE_NAME=estg))) failed Updating parameters: fnd_jdbc_buffer_decay_size=5 fnd_jdbc_usable_check=false fnd_jdbc_context_check=true java.sql.SQLException: Listener refused the connection with the following error: ORA-12514, TNS:listener does not currently know of service requested in connect descriptor The Connection descriptor used by the client was: (DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=isg-db-01-vip.domain.com)(PORT=1612)))(CONNECT_DATA=(SERVICE_NAME=estg)))

Database connection to jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=isg-db-01-vip.domain.com)(PORT=1612)))(CONNECT_DATA=(SERVICE_NAME=estg))) failed Updating parameters: fnd_jdbc_plsql_reset=false java.sql.SQLException: Listener refused the connection with the following error: ORA-12514, TNS:listener does not currently know of service requested in connect descriptor The Connection descriptor used by the client was: (DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=isg-db-01-vip.domain.com)(PORT=1612)))(CONNECT_DATA=(SERVICE_NAME=estg)))

Database connection to jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=isg-db-01-vip.domain.com)(PORT=1612)))(CONNECT_DATA=(SERVICE_NAME=estg))) failed

DBC generation failed with exit code 1

As it is evident from the error, the dbc file generation was failing due to which adcfgclone.pl was failing.
Digging deeper into this issue, I verified that the database instance was correctly registered with the listener. Bounce of the database listener did not help either. That is when I noticed that the connect string being used was having load balance entries, but did not have both the active RAC node information. This led me to check the Net services topology information verification where in I observed that the node information was not updated correctly. ACtually adgentns.pl is responsible for creating these entries, and I observed that adgentns.pl had not failed with any error though it had reported the ignorable unique key constraint error.
Fix:
1.Run the following command to clean up the net topology information:
perl [AD_TOP]/bin/adgentns.pl contextfile=[full path to context file] -removesystem
This will remove clean up the information in fnd_app_servers and fnd_nodes table.
2. Run autoconfig on both the db nodes and apps nodes: this should fix the issue.
3. However, in our case, there were a few corrections required in the database xml file:
i. s_dbService : dbname.domain
ii. s_dbClusterInst: 1 (on node 1) and 2 (on node 2)
iii. s_instNumber: 1 (on node 1) and 2 (on node 2)
iv. "s_instThread: 1 (on node 1) and 2 (on node 2)

After making the above corrections, I ran autoconfig and this fixed the issue.
- Aravind Kamath Posral

Friday, January 22, 2010

How to change the forms mode in R12?

Hi,
This post is more of a procedural thing rather than a problem-troubleshoot-fix kind of a post. By default, when you configure an R12 instance, rapid install will configure forms in servlet mode. The following steps will come in handy if you are entrusted with the task of changing the forms mode to socket or vice-versa.
a. Currently your forms is running in servlet mode and you want to configure socket mode.
i. Stop all the running services.
ii. Clear orphans, if any.
iii. Take a backup of the existing xml file.
iv. Run the following command:

$FND_TOP/bin/txkrun.pl -script=ChangeFormsMode -contextfile=[context file name with full path] -mode=socket -port=[forms port value] -runautoconfig=No

v. Verify that your xml has been updated with "socket" value for a few variables as shown in the log of the above script.
vi. Run autoconfig.
vii. Verify that you can connect to forms using front end and that you should see a frmsrv process running on the forms host. Run adopmnctl.sh status to verify that forms is no longer running in servlet mode.

b. Currently your forms is running in socket mode and you want to configure servlet mode.
i. stop all the running services.
ii. Clear orphans, if any.
iii. Take a backup of the existing xml file.
iv. Run the following command:

$FND_TOP/bin/txkrun.pl -script=ChangeFormsMode -contextfile=[context file name with full path] -mode=servlet -port=[forms port value] -runautoconfig=No

v. Verify that your xml has been updated with "servlet" value for a few variables as shown in the log of the above script.
vi. Run autoconfig.
vii. Verify that you can connect to forms using front end and that you do not see a frmsrv process running on the forms host. Run adopmnctl.sh status to verify that forms is indeed running in servlet mode.

- Aravind Kamath Posral

Tuesday, January 19, 2010

adapcctl.sh: exiting with status 204

Hi,
Happy New Year to all readers!
Recently I was cloning an R12 instance. After the cloning part was done using rapid clone, the apache failed to come up and was reporting the following error:
adapcctl.sh: exiting with status 204
However, there were no errors in error_log or in the adapcctl.txt log file that is generated as part of the startup script. Further digging into the issue revealed the following error in the HTTP_Server~1 log file under $LOG_HOME/ora/10.1.3/opmn:
Syntax error on line 17 of /apps/local/FIEST1/inst/apps/FIEST1_iftg-01/ora/10.1.3/Apache/Apache/conf/mod_osso.conf:Unable to open /apps/local/FIEST1/inst/apps/FIEST1_iftg-01/ora/10.1.3/Apache/Apache/conf/osso/FIEST1_iftg-01_osso.conf.

It turns out that this issue is because the source is SSO enabled and the target was not yet SSO enabled. Running $FND_TOP/bin/txkrun.pl -script=SetSSOReg -removereferences=Yes in target instance did not help either.

Fix:
Run detailed steps to deregister OID/SSO registration. This will run autoconfig as part of the deregistration process, which will generate the correct mod osso files. Basically, the s_mod_osso_conf_comment should have # (comment) in xml file before running autoconfig (which will be the result of running deregistration steps).

The apache came up fine subsequently.

- Aravind Kamath Posral