Tuesday, December 15, 2009

An exception occurred in the method CacheAccess.get

We faced an issue in one of our cloned instance. OACORE j2ee container was throwing the following error in application.log. The users were seeing Internal Server Error in their browsers.


html: LeakSetting: global:true,session:false,aggressive:false,stderr:false,appslog:false,corelog:false:$ at oracle.apps.fnd.common.AppsContext.initializeContext(AppsContext.java:926) at oracle.apps.fnd.common.AppsContext.initializeContext(AppsContext.java:891) at oracle.apps.fnd.common.AppsContext.(AppsContext.java:751) at oracle.apps.fnd.common.WebAppsContext.(WebAppsContext.java:1014) at oracle.apps.fnd.sso.Utils.getAppsContext(Utils.java:525) at oracle.apps.fnd.sso.AppsLoginRedirect.AppsSetting(AppsLoginRedirect.java:120) at oracle.apps.fnd.sso.AppsLoginRedirect.init(AppsLoginRedirect.java:170) at com.evermind[Oracle Containers for J2EE 10g (10.1.3.0.0) ].server.http.HttpApplication.loadServlet(HttpApplication.java:2231) at com.evermind[Oracle Containers for J2EE 10g (10.1.3.0.0) ].server.http.HttpApplication.findServlet(HttpApplication.java:4617) at com.evermind[Oracle Containers for J2EE 10g (10.1.3.0.0) ].server.http.HttpApplication.findServlet(HttpApplication.java:4541) at com.evermind[Oracle Containers for J2EE 10g (10.1.3.0.0) ].server.http.HttpApplication.getRequestDispatcher(HttpApplication.java:2821) at com.evermind[Oracle Containers for J2EE 10g (10.1.3.0.0) ].server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:740) at com.evermind[Oracle Containers for J2EE 10g (10.1.3.0.0) ].server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:451) at com.evermind[Oracle Containers for J2EE 10g (10.1.3.0.0) ].server.http.AJPRequestHandler.run(AJPRequestHandler.java:299) at com.evermind[Oracle Containers for J2EE 10g (10.1.3.0.0) ].server.http.AJPRequestHandler.run(AJPRequestHandler.java:187) at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260) at com.evermind[Oracle Containers for J2EE 10g (10.1.3.0.0) ].util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303) at java.lang.Thread.run(Thread.java:595)Caused by: oracle.apps.jtf.base.resources.FrameworkException: Error in IAS Cache: java.lang.NullPointerException: null Connection at oracle.apps.jtf.cache.IASCacheProvider.get(IASCacheProvider.java:712) at oracle.apps.jtf.cache.CacheManager.getInternal(CacheManager.java:4802) at oracle.apps.jtf.cache.CacheManager.get(CacheManager.java:4624) at oracle.apps.fnd.cache.AppsCache.get(AppsCache.java:216) ... 30 moreCaused by: oracle.apps.jtf.base.resources.FrameworkException:An exception occurred in the method CacheAccess.getnullThe base exception is:null Connection at oracle.apps.jtf.base.resources.FrameworkException.convertException(FrameworkException.java:607) at oracle.apps.jtf.base.resources.FrameworkException.addException(FrameworkException.java:585) at oracle.apps.jtf.base.resources.FrameworkException.(FrameworkException.java:66) at oracle.apps.jtf.base.resources.FrameworkException.(FrameworkException.java:88) at oracle.apps.jtf.base.resources.FrameworkException.(FrameworkException.java:202) at oracle.apps.jtf.base.resources.FrameworkException.(FrameworkException.java:218) at oracle.apps.jtf.base.resources.FrameworkException.(FrameworkException.java:249) ... 34 more09/12/14 22:27:10.447 html: 10.1.3.0.0 Stopped

Fix:
Services work fine on one web node but not in other web node. The reason this was happening was that the node name (where error is encountered) was missing from fnd_nodes. Run autoconfig on the node where you have the issue and it should be fine.

- Aravind Kamath Posral

Wednesday, December 2, 2009

RC-40201: Unable to connect to Database LT1QSC while running adclonectx.pl

We were trying to generate the xml file in our 11.5.9 environment as part of autoconfig implementation in our environment and we were using adclonectx.pl for this purpose.

The database is a RAC instance and we starting seeing the following issue:
RC-40201: Unable to connect to Database LT1QSC.

The log revealed that database listener was rejecting jdbc connections from adclonectx.pl:
getConnection() -->
sDbHost : psf-db1
sDbDomain : domain.com
sDbPort : 1527
sDbSid : LT1QSC
sDbUser : APPS
Trying to connect using SID...getConnectionUsingSID() --> JDBC URL: jdbc:oracle:thin:@psf-db1.domain.com:1527:LT1QSC
Exception occurred: java.sql.SQLException: Io exception: Connection refused(DESCRIPTION=(TMP=)(VSNNUM=153093888)(ERR=12505)(ERROR_STACK=(ERROR=(CODE=12505)(EMFI=4))))
Trying to connect using SID as ServiceNamegetConnectionUsingServiceName() --> JDBC URL: jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=psf-db1.domain.com)(PORT=1527))(CONNECT_DATA=(SERVICE_NAME=LT1QSC))) Exception occurred: java.sql.SQLException: Io exception: Connection refused(DESCRIPTION=(TMP=)(VSNNUM=153093888)(ERR=12514)(ERROR_STACK=(ERROR=(CODE=12514)(EMFI=4))))
Trying to connect using SID as ServiceName.DomainNamegetConnectionUsingServiceName() --> JDBC URL: jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=psf-db1.domain.com)(PORT=1527))(CONNECT_DATA=SERVICE_NAME=LT1QSC.domain.com)))
Exception occurred: java.sql.SQLException: Io exception: Connection refused(DESCRIPTION=(ERR=1153)(VSNNUM=153093888)(ERROR_STACK=(ERROR=(CODE=1153)(EMFI=4)(ARGS='(ADDRESS=(PROTOCOL=tcp)(HOST=psf-db2.domain.com)(PORT=1527))'))(ERROR=(CO
DE=305)(EMFI=1)))) Connection could not be obtained; returning null

Quick Fix:
shutdown db node 2 and rerun the command, it will go through. This way we are forcing the connection to node 1 itself, as I believe this issue is specific to load balanced environment.

- Aravind Kamath Posral

Friday, November 20, 2009

RC-10203: The template provided does not match with the context file

We recently faced the following issue in one of our instances while running adclonectx.pl to clone the xml file:


AC-10006: Exception - java.io.FileNotFoundException: /appl_top/ad/11.5.0/admin/template/adxmlctx.tmp thrown while creating OAVars object for file:
../admin/template/adxmlctx.tmpRC-50004: Error occurred in CloneContext: RC-10203: The template provided does not match with the context file.Check logfile /apps/sdev/appl_top/admin/sdev/out/CloneContext_11190814.log for details.ERROR: context creation not completed successfully.Please check /tmp/adclonectx.err file for errors.

The adclonectx.pl script was not picking up the correct $AD_TOP path which is why it was throwing this error. To fix this issue, I used the following command:
perl adclonectx.pl sharedappltop contextfile=/apps/sdev/sdev.xml template=/apps/sdev/appl_top/ad/11.5.0/admin/template/adxmlctx.tmp
pairsfile=/apps/sdev/pre_Autoconfig/pairsfile_sdev.txt

Pass the template parameter as part of the adclonectx.pl script to provide the full path of the adxmlctx.tmp, which is the template referred here.


Subsequently, the command ran without any issues:
/apps/sdev/appl_top/ad/11.5.0/bin > perl adclonectx.pl sharedappltop contextfile=/apps/sdev/sdev.xml template=/apps/sdev/appl_top/ad/11.5.0/admin/template/adxmlctx.tmp
pairsfile=/apps/sdev/pre_Autoconfig/pairsfile_sdev.txt/apps/sdev/common_top/util/jre/1.1.8/bin/jre -classpath
/apps/sdev/product/iAS/jdbc/lib/classes111.zip:/apps/sdev/common_top/java/xmlparserv2.zip:/apps/sdev/common_top/util/jre/1.1.8/lib/rt.jar:/apps/sdev/common_top/java:/a pps/sdev/common_top/util/jre/1.1.8/lib/rt.jar:/apps/sdev/common_top/util/jre/1.1.8/lib/i18n.jar:/apps/sdev/common_top/util/jre/1.1.8/lib/tools.jar:/apps/sdev/common_to p/java/appsborg.zip:/apps/sdev/common_top/java/apps.zip:/apps/sdev/product/8.0.6/forms60/java:/apps/sdev/common_top/java oracle.apps.ad.context.CloneContext -e
/apps/sdev/sdev.xml -tmpl /apps/sdev/appl_top/ad/11.5.0/admin/template/adxmlctx.tmp -pairsfile /apps/sdev/pre_Autoconfig/pairsfile_sdev.txt -sharedappltop -perl
"/apps/sdev/product/iAS/Apache/perl/bin/perl" -perl5lib
"/apps/sdev/product/iAS/Apache/perl/lib/5.00503:/apps/sdev/product/iAS/Apache/perl/lib/site_perl/5.005:/apps/sdev/appl_top/au/11.5.0/perl"
Provide the values required for creation of the new APPL_TOP Context file.
Do you want to use a virtual hostname for the target node (y/n) [n] ?:
Do you want to add Admin Server node (y/n) [y] ?:
Do you want to add Concurrent Processing Server node (y/n) [y] ?:
Do you want to add Forms Server node (y/n) [y] ?:
Do you want to add Web Server node (y/n) [y] ?:
Target system 8.0.6 ORACLE_HOME directory [/apps/sdev/product/8.0.6]:
Target system iAS ORACLE_HOME directory [/apps/sdev/product/iAS]:
Location of the JDK on the target system [/opt/java1.3]:Database port is 1521
Do you want the middle-tier services on the target system to have the same port values as the source system (y/n) [y] ?:Web Listener port is 8903Complete port information available at /apps/sdev/appl_top/admin/out/sdev_thus/portpool.lst
APPS password is required to validate database connection using database port 1521Enter APPS Password [APPS]:apps
Target system proxy hostname [name.domain.com]:
Target system proxy port [80]:
UTL_FILE_DIR on database tier consists of the following directories.
1. /apps/orarpt/sdev/utlChoose a value which will be set as APPLPTMP value on the target node [1]:
Creating the new APPL_TOP Context file from : /apps/sdev/appl_top/ad/11.5.0/admin/template/adxmlctx.tmp
The new APPL_TOP context file has been created : /apps/sdev/appl_top/admin/sdev_thus.xml
Log file located at /apps/sdev/appl_top/admin/sdev/out/CloneContext_11190840.log

Tuesday, November 10, 2009

FRM-40735: POST-UPDATE trigger raised unhandled exception ORA-04062.

We encountered a strange nagging issue when users were accessing a form through one of the forms server node. They were facing the following error:
ORA-04062: signature of package "APPS.XYCCA_HDRB_EXTN_PCK" has been changed

FRM-40735: POST-UPDATE trigger raised unhandled exception ORA-04062.


We did not suspect the database package because users connected to other three nodes were perfectly fine and were able to open the same form without issues. Bounce of forms server on the erring node did not help either. Environment, db connectivity and all other form server related settings were perfectly fine.


It turns out that the erring node was seeing an "older" image of the pll file referenced by the form. We have a shared file system implemented in this system and the other three nodes were showing the correct latest version of the file. We suspect a NFS cache issue at this time but the fix was to touch a dummy file from the erring node on the file system where the plls were residing.
Unmount and mount on erring node, i am sure, will fix the issue but will require a bounce. In our case, once we touched a dummy file, the erring node started to see the current version of the plls in question and we were able to get past the issue.
- Aravind Kamath Posral

Sunday, November 8, 2009

SQL*Net break/reset to dblink and SQL*Net break/reset to client

We faced a strange situation where one of our concurrent programs was sitting "idle" doing nothing, although the status of the session showed active. The program connects to a remote database over db link, runs a select query and the output of select query is inserted into a table. However, in this situation, the session on the remote database was inactive and the wait event was "SQL*Net break/reset to client". In the concurrent program session, the wait event was "SQL*Net break/reset to dblink".

After some troubleshooting effort that revealed nothing, we decided to create a dummy table with the same structure and run the select/insert statement manually. This gave out the cause of the issue:

ORA-12899: value too large for column string (actual: string, maximum: string)

Actually, the data fetched from the remote database was not being populated in the table because of the above error.

I strongly believe that this is the result of poor design as the concurrent program should have taken care of the exception handling.

Nevertheless, since this is a data issue, there is nothing much DBA can do and hence we transferred the issue to our dev team to take care of this.

PS: Thought of posting this as there are not many articles out there about the aforesaid wait events, which are not related to bugs! Almost every hit on the above wait event will lead you to bug description or a tar to which one will not have access. In our case, the workaround was helpful to trap the error.
- Aravind Kamath Posral

Tuesday, November 3, 2009

ld: Can't find library for -llibrt.so

Today when we were trying to setup a 10.2.0.4 code tree on one of our HPUX server, we encountered the following error while relinking the oracle executable.
ORACLE_SID=ASTCTC
Host Name: dvd-db109

Error:

/oracle/product/10.2.0.4/ASTCTC/rdbms/lib > make -f ins_rdbms.mk ioracle
chmod 755 /oracle/product/10.2.0.4/ASTCTC/bin
- Linking Oracle rm -f /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/oracle cc +DD64 -Wl,+s -Wl,+n +DD64 -o /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/oracle -L/oracle/product/10.2.0.4/ASTCTC/rdbms/lib/ -L/oracle/product/10.2.0.4/ASTCTC/lib/ /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/ssoident.o `if /usr/bin/getconf KERNEL_BITS grep "64" > /dev/null ; \ then echo "-Wl,+pi,64M -Wl,+pd,L -Wl,+padtext,16M -Wl,+paddata,1M" ; \ else echo "" ; fi` -Wl,-E -Wl,-PF,linkorderfile +FPd /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/opimai.o /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/ssoraed.o /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/ttcsoi.o /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/defopt.o -Wl,+forceload -lperfsrv10 -Wl,+noforceload /oracle/product/10.2.0.4/ASTCTC/lib/nautab.o /oracle/product/10.2.0.4/ASTCTC/lib/naeet.o /oracle/product/10.2.0.4/ASTCTC/lib/naect.o /oracle/product/10.2.0.4/ASTCTC/lib/naedhs.o /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/config.o -lserver10 -lodm10 -lnnet10 -lskgxp10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 -lhasgen10 -lcore10 -lskgxn2 -locr10 -locrb10 -locrutl10 -lhasgen10 -lcore10 -lskgxn2 -lclient10 -lvsn10 -lcommon10 -lgeneric10 -lknlopt `if /usr/ccs/bin/ar tv /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/libknlopt.a grep xsyeolap.o > /dev/null 2>&1 ; then echo "-loraolap10" ; fi` -lslax10 -lpls10 -lplp10 -lserver10 -lclient10 -lvsn10 -lcommon10 -lgeneric10 `if [ -f /oracle/product/10.2.0.4/ASTCTC/lib/libavserver10.a ] ; then echo "-lavserver10" ; else echo "-lavstub10"; fi` `if [ -f /oracle/product/10.2.0.4/ASTCTC/lib/libavclient10.a ] ; then echo "-lavclient10" ; fi` -lknlopt -lslax10 -lpls10 -lplp10 -ljox10 -lserver10 /oracle/product/10.2.0.4/ASTCTC/has/lib/libclsra10.a -ldbcfg10 -locijdbcst10 -lwwg `cat /oracle/product/10.2.0.4/ASTCTC/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lnro10 `cat /oracle/product/10.2.0.4/ASTCTC/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lmm -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `cat /oracle/product/10.2.0.4/ASTCTC/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lnro10 `cat /oracle/product/10.2.0.4/ASTCTC/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `if /usr/ccs/bin/ar tv /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/libknlopt.a grep "kxmnsd.o" > /dev/null 2>&1 ; then echo " " ; else echo "-lordsdo10"; fi` -lctxc10 -lctx10 -lzx10 -lgx10 -lctx10 -lzx10 -lgx10 -lordimt10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 -lsnls10 -lunls10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `cat /oracle/product/10.2.0.4/ASTCTC/lib/sysliblist` -lm `cat /oracle/product/10.2.0.4/ASTCTC/lib/sysliblist` -lm -lpthread -lpthreadld: Can't find library for -llibrt.soFatal error.
*** Error exit code 1
Stop.

Fix:

After the routine check for LD_LIBRARY_PATH, SHLIB_PATH etc, I zeroed in on the cc compiler. It turns out that cc was being picked up from a wrong path.


/oracle/product/10.2.0.4/ASTCTC/rdbms/lib > which cc
/usr/bin/cc
/usr/bin > ls -l cc
lrwxr-xr-t 1 root sys 17 Jul 9 01:00 cc@ -> /opt/ansic/bin/cc
/usr/ccs/bin > ls -l cc
lrwxrwxr-x 1 sfmdb 191 12 Aug 18 2008 cc@ -> ./cc_bundled
/usr/ccs/bin > export PATH=/usr/ccs/bin:$PATH
/usr/ccs/bin > which cc
/usr/ccs/bin/cc
/usr/ccs/bin > cc -V
(Bundled) cc: HP aC++/ANSI C B3910B A.05.50 [May 15 2003]

In our case, the above version is the desired one. Refer to your installation manual to see which version of cc is compatible/required for your version of OS and Oracle binaries.

/usr/ccs/bin > echo $ORACLE_HOME
/oracle/product/10.2.0.4/ASTCTC
/usr/ccs/bin > echo $SHLIB_PATH
/oracle/product/10.2.0.4/ASTCTC/lib32:/oracle/product/10.2.0.4/ASTCTC/lib
/oracle/product/10.2.0.4/ASTCTC/rdbms/lib > make -f ins_rdbms.mk ioracle
chmod 755 /oracle/product/10.2.0.4/ASTCTC/bin
- Linking Oracle
rm -f /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/oracle cc +DD64 -Wl,+s -Wl,+n +DD64 -o /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/oracle -L/oracle/product/10.2.0.4/ASTCTC/rdbms/lib/ -L/oracle/product/10.2.0.4/ASTCTC/lib/ /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/ssoident.o `if /usr/bin/getconf KERNEL_BITS grep "64" > /dev/null ; \ then echo "-Wl,+pi,64M -Wl,+pd,L -Wl,+padtext,16M -Wl,+paddata,1M" ; \ else echo "" ; fi` -Wl,-E -Wl,-PF,linkorderfile +FPd /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/opimai.o /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/ssoraed.o /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/ttcsoi.o /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/defopt.o -Wl,+forceload -lperfsrv10 -Wl,+noforceload /oracle/product/10.2.0.4/ASTCTC/lib/nautab.o /oracle/product/10.2.0.4/ASTCTC/lib/naeet.o /oracle/product/10.2.0.4/ASTCTC/lib/naect.o /oracle/product/10.2.0.4/ASTCTC/lib/naedhs.o /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/config.o -lserver10 -lodm10 -lnnet10 -lskgxp10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 -lhasgen10 -lcore10 -lskgxn2 -locr10 -locrb10 -locrutl10 -lhasgen10 -lcore10 -lskgxn2 -lclient10 -lvsn10 -lcommon10 -lgeneric10 -lknlopt `if /usr/ccs/bin/ar tv /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/libknlopt.a grep xsyeolap.o > /dev/null 2>&1 ; then echo "-loraolap10" ; fi` -lslax10 -lpls10 -lplp10 -lserver10 -lclient10 -lvsn10 -lcommon10 -lgeneric10 `if [ -f /oracle/product/10.2.0.4/ASTCTC/lib/libavserver10.a ] ; then echo "-lavserver10" ; else echo "-lavstub10"; fi` `if [ -f /oracle/product/10.2.0.4/ASTCTC/lib/libavclient10.a ] ; then echo "-lavclient10" ; fi` -lknlopt -lslax10 -lpls10 -lplp10 -ljox10 -lserver10 /oracle/product/10.2.0.4/ASTCTC/has/lib/libclsra10.a -ldbcfg10 -locijdbcst10 -lwwg `cat /oracle/product/10.2.0.4/ASTCTC/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lnro10 `cat /oracle/product/10.2.0.4/ASTCTC/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lmm -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `cat /oracle/product/10.2.0.4/ASTCTC/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lnro10 `cat /oracle/product/10.2.0.4/ASTCTC/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `if /usr/ccs/bin/ar tv /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/libknlopt.a grep "kxmnsd.o" > /dev/null 2>&1 ; then echo " " ; else echo "-lordsdo10"; fi` -lctxc10 -lctx10 -lzx10 -lgx10 -lctx10 -lzx10 -lgx10 -lordimt10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 -lsnls10 -lunls10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `cat /oracle/product/10.2.0.4/ASTCTC/lib/sysliblist` -lm `cat /oracle/product/10.2.0.4/ASTCTC/lib/sysliblist` -lm -lpthread -lpthread mv -f /oracle/product/10.2.0.4/ASTCTC/bin/oracle /oracle/product/10.2.0.4/ASTCTC/bin/oracleO mv /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/oracle /oracle/product/10.2.0.4/ASTCTC/bin/oracle chmod 6751 /oracle/product/10.2.0.4/ASTCTC/bin/oracle

As you can see above, the relink was successful.
- Aravind Kamath Posral

Wednesday, October 28, 2009

creating a new RAC environment on Linux using Rapid Clone

Hi,
Here i try to explain the steps at a high level that are involved in creating a new R12 environment with RAC database on Linux platform at backend. We used a combination of manual and rapid clone steps to setup the environment. The source was production and the target was a stage environment. We have shared appl top architecture implemented in this environment. The oracle apps version is R12 (12.0.4) and runs a 10.2.0.3 RAC database on block devices. The production was not taken down for cloning purposes, instead, we used hot backups from production to bring up the target server database.
At the top level, following are the steps:
1. Set up the RDBMS code tree on target server
2. Make Hot Backup Consistent
3. Copy the source block devices from backup server to target server
4. Bring up the target database
5. Apps Precloning Steps on source steps
6. Apps post clone steps on target
Source DB Name: Source
Source DB Instance 1: Source1
Source DB Instance 2 : Source2
Target DB Name: Target
Target DB Instance 1: Target1
Target DB Instance 2: Target2
Source db Hostnames: isprd-db-01 / isprd-db-02
Source Apps Hostnames: isprd-01/isprd-02/isprd-03/isprd-04
Target DB Hostnames: isstg-db-01 / isstg-db-02
Target Apps Hostnames: isstg-01/isstg-02/isstg-03/isstg-04
Load Balancer: target.domain.com
1. Setup the RDBMS code tree on Target Server:
- This requires that we setup the RAC code tree. However, the database needs to be brought up as a single instance and then converted to RAC.- Run adpreclone.pl dbTier on source database host. Since the code tree copy would take a lot of time over the network, we copied the code tree from another environment running on the target servers. Setup the /etc/oraInst.loc to point to correct inventory locations.- Copy only the appsutil folder from production where you have run the adpreclone.pl- Ensure that $ORACLE_HOME, $ORACLE_SID, $LD_LIBRARY_PATH are all set.- Run the following command one first node to complete the cloning of code tree on target server:
$ cd $ORACLE_HOME/clone/bin/ perl clone.pl ORACLE_HOME=/oracle/product/10.2.0.3/TARGET ORACLE_HOME_NAME=ORACLE_TARGET -O"LOCAL_NODE=isstg-db-01"' '-O"CLUSTER_NODES={isstg-db-01,isstg-db-02}"'- run root.sh as root user which is present under $ORACLE_HOME
- Run the following command on the second db node to complete the cloning of code tree on target server:
$ cd $ORACLE_HOME/clone/bin/
perl clone.pl ORACLE_HOME=/oracle/product/10.2.0.3/TARGET ORACLE_HOME_NAME=ORACLE_TARGET'-O"CLUSTER_NODES={isstg-db-01,isstg-db-02}"' '-O"LOCAL_NODE=isstg-db-02"' '-O-noConfig'
$ cd $ORACLE_HOME/oui/bin/ ./runInstaller -silent -updateNodeList ORACLE_HOME=/oracle/product/10.2.0.3/TARGET "CLUSTER_NODES={isstg-db-01,isstg-db-02}"
- run root.sh from $ORACLE_HOME
2. Make Hot Backup Consistent:
This step is required because we are not taking the copy from cold backup. In our environment, we have a backup server where the hot backup (datafiles) are stored. We also have the code tree, and using this code tree, we can make the hot backup consistent.
on the backup server:
sqlplus>startup nomountsql>alter database mount;sql>recover database until cancel using backup controlfile;
apply a couple of archive logs as requested by the recovery process and the cancel.sql> cancelsql> alter database open readonly;sql> shutdown immediate;
The above steps make hot backup consistent.
- Take a backup of the control file on source:
sql> alter database backup controlfile to trace;
- Create dd scripts to copy the block devices from source to target. Sample command for one file would look like:
dd if=/dev/mapper/oraSOURCE_0213_03Z7_4400p1 bs=256kgzip --fast ssh isstg-db-01 "gunzipdd of=/dev/mapper/oraTARGET_4669_1DXH_4400p1 bs=256k "
- You can run the above dd commands in background and check for nohup.log- When the copy is going on, you can create the control file for target. The first line in our case would look like follows:
STARTUP NOMOUNTCREATE CONTROLFILE SET DATABASE "TARGET" RESETLOGS NOARCHIVELOG
- Make sure all references are to target- If you are planning to have 16 groups of redo logs, provide only 8 at this time. You can add the second thread later when we convert single instance to RAC.- Prepare the init.ora for target instance. Update the correct values for control files which will be created in the next section. Have the list of redo log volumes handy.- Set _no_recovery_through_resetlogs=TRUE in target as the source for this target is RAC.- Take care to ensure that you rename the tnsnames.ora, ldap.ora and sqlnet.ora on target database code tree ($TNS_ADMIN) location so that no damage is done to any remote production dbs due to db links that got carried over.- Set aq and job queue process to 0 so that no inadvertant damages are done.- Once the copy is over, bring up the target database using the control file script.- Take a backup of sys.link$ table and truncate the same.- run exec fnd_conc_clone.setup_clean as apps user.- Once the db comes up cleanly, convert to RAC.- Create the - Ensure that you have set the undo_tablespace parameter correctly.- Ensure that you have the RAC init parameters setup. For eg. CLUSTER_DATABASE/CLUSTER_INTERCONNECT, REMOTE_LISTENER,LOCAL_LISTENER and so on.- Once you have finished above steps, bring up the single node instance with SID pointing to target.- Convert the single node to RAC by adding additional log groups:
ALTER DATABASE ADD LOGFILE THREAD 2GROUP 9 ('/dev/mapper/oraTarget_4669_2B5A_log2_1000p1','/dev/mapper/oraTarget_4669_2B7A_log2_mirror_1000p1') SIZE 512M,GROUP 10 ('/dev/mapper/oraTarget_4669_2B5A_log2_1000p2','/dev/mapper/oraTarget_4669_2B7A_log2_mirror_1000p2') SIZE 512M,GROUP 11 ('/dev/mapper/oraTarget_4669_2B5A_log2_1000p3','/dev/mapper/oraTarget_4669_2B7A_log2_mirror_1000p3') SIZE 512M,GROUP 12 ('/dev/mapper/oraTarget_4669_2B5A_log2_1000p5','/dev/mapper/oraTarget_4669_2B7A_log2_mirror_1000p5') SIZE 512M,GROUP 13 ('/dev/mapper/oraTarget_4669_2B5A_log2_1000p6','/dev/mapper/oraTarget_4669_2B7A_log2_mirror_1000p6') SIZE 512M,GROUP 14 ('/dev/mapper/oraTarget_4669_2B5A_log2_1000p7','/dev/mapper/oraTarget_4669_2B7A_log2_mirror_1000p7') SIZE 512M;

alter database enable public thread 2;
- Startup second node after ensuring the RAC parameters have been defined in node 2 init file.- Verify alert log for any errors.
Create the xml file for node 1 and node 2 of the RAC db respectively. For node 1, use the following syntax:
[oracle@isstg-db-01 bin]$ cd $ORACLE_HOME/appsutil/clone/bin[oracle@isstg-db-01 bin]$ perl adclonectx.pl contextfile=/oracle/product/10.2.0.3/TARGET/appsutil/clone/context/db/SORUCE_isprd-db-01.xml template=$ORACLE_HOME/appsutil/template/adxdbctx.tmp initialnode
Respond to the prompts and the above command will create the xml file for you. Copy the context file from source server to target server to use as per the contextfile variable in the above command.
For the second node, use the following syntax:
cd $ORACLE_HOME/appsutil/clone/binperl adclonectx.pl contextfile=<> template=$ORACLE_HOME/appsutil/template/adxdbctx.tmp pairsffile=$ORACLE_HOME/appsutil/clone/pairsfile.txt addnode
In the contextfile variable above, provide the location of the newly created xml (the one which we created in preceding step) file copied to the secondary node from the primary node.
Run autoconfig on both the db nodes.
This completes the db part. Next move on to apps part.
5. Apps Precloning Steps on source steps
This portion is pretty much standard and what we do here is run adpreclone.pl on source node. In case of shared appl top, running on any one node will do.
perl adpreclone.pl appsTier
Then copy the apps file system to target.
6. Apps post clone steps on target
Once the copy is complete, login to target system and run the following command:
perl adcfgclone.pl appsTier
In other nodes, run the following command to create and configure the shared appl top:
perl adclonectx.pl addnode contextfile=/apps/target/TARGET_isstg-04.xml
In the above syntax, contextfile is the new created xml file in node 1 which is copied over to additional nodes.
This pretty much sums up the steps at macro level. I have not gone to the micro level and finishing tasks as that would result in a humungous post. I have putforth those steps which i thought were critical and provide direction to a similar effort.
- Aravind Kamath Posral

Tuesday, October 27, 2009

ORA-12154: TNS:could not resolve the connect identifier specified

Background:
We were in the middle of a project and we wanted to have restricted access to our database environment. We have ldap name resolution method implemented in the organisation and this means that anyone who has his or her ldap.ora setup correctly will be able to connect to any database across the organization, provide they have the login credentials. We wanted to restrict individuals from accessing our database, where as development team had a requirement to ensure that db links in remote databases were able to connect to the database undergoing project patching at all times.
We disabled the ldap entries by deleting them and provided the tnsnames entries for the remote databases which would connect to our database over db link. However, once the ldap entries were removed, the remote database failed to connect to our database with the following error:
ORA-12154: TNS:could not resolve the connect identifier specified

Correct tnsentries were made in $ORACLE_HOME/network/admin/tnsnames.ora.

Fix:
Detailed troubleshooting revealed that the environment was not sourced properly while starting the remote database. The $TNS_ADMIN variable was pointing to /var/opt/oracle location where as it should have been pointing to $ORACLE_HOME/network/admin. Since we could not bounce the remote database in between, we temporarily put the entry in /var/opt/oracle/tnsnames.ora in remote database and then the db link started working.
This situation would not arise if environment was sourced properly before starting up the database, which would then point to the correct $TNS_ADMIN.
-Aravind Kamath Posral

PCP: Managers defined to run on Secondary Node are running on Primary Node.

Background:

We have installed R12 in our environment and we have two mid-tier nodes and a database node. The database node does not contain any apps services or apps filesystem. Initially, we had planned one node to be the web apps node and one node to be the concurrent manager node.

Issue:

We decided to use PCP in our environment and move some of the concurrent queues to second node. We followed the pcp steps and brought up the concurrent managers, but all the managers would run on node one.

Fix:

The second node was configured to run web apps only. Hence, corrected the following sections of the context file on node 2:


< type="batch" title="Batch Processing Services" services="tns_apps, concmgr, icsm_srv, jtff_srv">
< oa_var="s_batch_status">disabled< /oa_service_group_status >

TO

< type="batch" title="Batch Processing Services" services="tns_apps, concmgr, icsm_srv, jtff_srv">
< oa_var="s_batch_status">enabled< /oa_service_group_status >

Subsequently autoconfig was run. This will create Service Managers for node two and PCP was found to be working satisfactorily.

-Aravind Kamath Posral

adpreclone.pl: ERROR while running Stage...

Back to blogging after a couple of months of inactivity!! Here is an issue I faced while trying to run adpreclone on dbtier:
/oracle/product/9.2.0.7_autoconfig/STPJS/appsutil/scripts/STPJS_server > perl adpreclone.pl dbTier
Running Rapid Clone with command...
perl /oracle/product/9.2.0.7_autoconfig/STPJS/appsutil/bin/adclone.pl java=/oracle/product/9.2.0.7_autoconfig/STPJS/jre/1.4.2 mode=stage
stage=/oracle/product/9.2.0.7_autoconfig/STPJS/appsutil/clone component=dbTier method=CUSTOM dbctx=/oracle/product/9.2.0.7_autoconfig/STPJS/appsutil/STPJS1_server.xml showProgress

Beginning database tier Stage - Tue Oct 27 01:46:48 2009
APPS Password : ******
Log file located at /oracle/product/9.2.0.7_autoconfig/STPJS/appsutil/log/STPJS1_server/StageDBTier_10270046.log
\ 0% completed
ERROR while running Stage...

ERROR while running perl /oracle/product/9.2.0.7_autoconfig/STPJS/appsutil/bin/adclone.pl java=/oracle/product/9.2.0.7_autoconfig/STPJS/jre/1.4.2 mode=stage stage=/oracle/product/9.2.0.7_autoconfig/STPJS/appsutil/clone component=dbTier method=CUSTOM dbctx=/oracle/product/9.2.0.7_autoconfig/STPJS/appsutil/STPJS1_server.xml showProgress ... Please check the log for more details..

The log does not have any details.

Fix:

1. comment out the following lines in STPJS1_server.env



#PATH=${PERLBIN}:${PATH}

#export PATH

#PATH=".:/usr/bin:/usr/ccs/bin:/opt/nettladm/bin:/opt/fcms/bin:/opt/resmon/bin:/
opt/perf/bin:/opt/OV/bin/OpC:/opt/ignite/bin:/usr/sbin/stm/uut/bin/progs:/usr/lo
cal/bin:/usr/local/tools:/usr/contrib/bin:/usr/sbin/diag/contrib:/opt/hparray/bi
n:/opt/graphics/common/bin:/opt/prm/bin:/opt/OV/bin:/opt/langtools/bin:/opt/NetA
pp/santools/bin:/opt/wbem/bin:/opt/wbem/sbin:/opt/sfm/bin:/users/oracle/bin:/usr
/bin:/usr/local/bin:/bin:/etc:/usr/etc:/usr/bin:/usr/bin/X11:/usr/contrib/bin:/u
sr/local/X11R5/bin:/oracle/product/9.2.0.7_autoconfig/STPJS/bin:/usr/tools/orac
le/Standard/script:/users/oracle/bin"


2. copy the correct version of zip to $ORACLE_HOME/bin. The zip version should be greater than 2.2.

3. set path as follows:
export PATH=$ORACLE_HOME/bin:$PATH

4. Which zip should show $ORACLE_HOME/bin/zip

5. Which perl should show /usr/local/bin/perl

6. Run adpreclone.pl dbTier, it should work correctly.


Troubleshooting this issue requires some proof reading of the adpreclone.pl script to understand what it does and where exactly it errors out. Here is the result of that:

since /usr/local/bin has both perl and zip executables, the lower version of zip was being picked up from /usr/local/bin. Even if you copy the newer version of zip to $ORACLE_HOME/bin and set the path manually before you start the adpreclone to PATH=$ORACLE_HOME/bin:$PATH, you will encounter the error. This is because adpreclone calls adclone.pl and before calling adclone.pl it sources the STPJS1_server.env file which will reset the PATH to original value which will wipe out $ORACLE_HOME/bin from the path. This is why we have commented out path in env file in above steps.

Revert the changes back after adpreclone runs successfully.

Permanent Fix: Copy the correct version of zip to /usr/local/bin after making sure that there are no other applications running on the server that have dependency on older version of zip.


-Aravind Kamath Posral

Sunday, June 21, 2009

RC-50004: Fatal: Error occurred in ApplyAppsTier

Error: While running perl adcfgclone.pl appsTier, encountered the following error somewhere near 74% progress..

RC-50004: Fatal: Error occurred in ApplyAppsTier


StackTrace -
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccesso
rImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructo
rAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
at
oracle.apps.ad.clone.util.CloneProcessor.run(CloneProcessor.java:67)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.lang.NoSuchMethodError:
oracle.apps.ad.autoconfig.InstantiateFile.writeReport(Ljava/util/Vector;Loracl
e/apps/ad/util/LogFile;I)V
at
oracle.apps.ad.clone.ApplyApplTop.checkAutoConfigErr(ApplyApplTop.java:590)
at
oracle.apps.ad.clone.ApplyApplTop.runAutoConfig(ApplyApplTop.java:558)
at oracle.apps.ad.clone.ApplyApplTop.doConf(ApplyApplTop.java:339)
at oracle.apps.ad.clone.ApplyApplTop.doApply(ApplyApplTop.java:382)
at oracle.apps.ad.clone.ApplyApplTop.(ApplyApplTop.java:267)
at oracle.apps.ad.clone.ApplyAppsTier.(ApplyAppsTier.java:105)
... 6 more


ERROR while running Apply...


Troubleshooting tips:

1. Metalink suggested that this is due to mid-tier and db-tier servers not being in the same domain / not entering fully qualified domain name while entering the db host name.
2. The above suggestion did not help in our case.
3. The following was recorded in "$INST_TOP/admin/logApplyAppsTier_06170114.log"

DECLARE
*
ERROR at line 1:
ORA-02019: connection description for remote database not found
ORA-06512: at "XXXX_DBLINKADM.CHK_DBLNK_CREATION", line 156
ORA-06512: at line 2
ORA-06512: at "SYS.DBMS_SYS_SQL", line 909
ORA-06512: at "SYS.DBMS_SQL", line 39
ORA-06512: at line 33


Error encountered while running $INST_TOP/admin/install/bisdblrp.sql
ERRORCODE = 256 ERRORCODE_END
.end std out.


Solution:

Actually, we have a custom solution comprising of database triggers that are meant to prevent creation of db links from production to non-production and vice-versa.One of the checks this custom solution does is to validate if the source and target databases are indeed non-prod. In this case, it turned out that since I was creating a new environment, the new environment information was not available in the repository. By the way, bisdblrp.sql creates EDW_APPS_TO_WH db link in which case,
the source and target db name will be the same (the target instance for which rapid clone is being run). This custom trigger was preventing the creation the db link
due to which autoconfig was reporting an error which in turn was reported by Rapid Clone as mentioned above.

So, the solution is to ensure that bisdblpr.sql successfully creates the db link.

-Aravind Kamath Posral

Wednesday, April 8, 2009

Determine release dates for Oracle Critical Patch Updates

Hi,

If you are trying to find out about the Critical Patch Updates (CPU) patches for various oracle products, here are some useful links:

http://www.oracle.com/technology/deploy/security/securityfixlifecycle.html

The above link gives you overview view of the program. The link below gives you the dates / schedule.

http://www.oracle.com/technology/deploy/security/alerts.htm

- Aravind Kamath Posral

Friday, April 3, 2009

Clicking on a Specific Self-Service Responsibility throws unexpected error

Hi,

Few days back, one of the developers reported an issue they encountered while clicking on a self-service responsibility (customized territory management responsibility):

You have encountered an unexpected error. Please contact the System Administrator for assistance.

The error stack was as follows:

Error Page
Exception Details.
oracle.apps.fnd.framework.OAException: Could not load application module 'oracle.apps.jtf.territory.manageTerritory.server.NonNATerritoryAM'.
at oracle.apps.fnd.framework.webui.OAJSPApplicationRegistry.registerApplicationModule(OAJSPApplicationRegistry.java:279)
at oracle.apps.fnd.framework.webui.OAJSPApplicationRegistry.registerApplicationModule(OAJSPApplicationRegistry.java:78)
at oracle.apps.fnd.framework.webui.OAPageBean.preparePage(OAPageBean.java:1280)
at oracle.apps.fnd.framework.webui.OAPageBean.preparePage(OAPageBean.java:533)
at oracle.apps.fnd.framework.webui.OAPageBean.preparePage(OAPageBean.java:421)
at _OA._jspService(_OA.java:212)
at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:59)
at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:416)
at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:478)
at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:401)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:719)
at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:376)
at com.evermind.server.http.ServletRequestDispatcher.unprivileged_forward(ServletRequestDispatcher.java:270)
at com.evermind.server.http.ServletRequestDispatcher.access$100(ServletRequestDispatcher.java:42)
at com.evermind.server.http.ServletRequestDispatcher$2.oc4jRun(ServletRequestDispatcher.java:204)
at oracle.oc4j.security.OC4JSecurity.doPrivileged(OC4JSecurity.java:283)
at com.evermind.server.http.ServletRequestDispatcher.forward(ServletRequestDispatcher.java:209)
at com.evermind.server.http.EvermindPageContext.forward(EvermindPageContext.java:322)
at _RF._jspService(_RF.java:225)
at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:59)
at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:416)
at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:478)
at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:401)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:64)
at oracle.apps.jtf.base.session.ReleaseResFilter.doFilter(ReleaseResFilter.java:26)
at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:15)
at oracle.apps.fnd.security.AppsServletFilter.doFilter(AppsServletFilter.java:318)
at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:627)
at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:376)
at com.evermind.server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:870)
at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:451)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:299)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:187)
at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260)
at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303)
at java.lang.Thread.run(Thread.java:595)
## Detail 0 ##
oracle.apps.fnd.framework.OAException: Application: FND, Message Name: FND_GENERIC_MESSAGE. Tokens: MESSAGE = java.sql.SQLException: ORA-20001: SQL_PLSQL_ERROR: N, ROUTINE, MO_GLOBAL.INIT, N, ERRNO, -20001, N, REASON, ORA-20001: SQL_PLSQL_ERROR: N, ROUTINE, MO_GLOBAL.SET_ORG_ACCESS, N, ERRNO, -20001, N, REASON, ORA-20001: APP-FND-02938: Multi-organization routine failed to initialize a session for the product: &PRODUCT. Please inform your support representative.
ORA-06512: at "APPS.FND_MESSAGE", line 509
ORA-06512: at "APPS.MO_GLOBAL", line 36
ORA-06512: at "APPS.MO_GLOBAL", line 757
ORA-06512: at "APPS.MO_GLOBAL", line 700
ORA-06512: at line 1
;

FIX:


FND_MO_PRODUCT_INIT_PKG.register_application('XX_CUSTOM','SEED','Y');
COMMIT;

select application_short_name,status from FND_MO_PRODUCT_INIT where application_short_name like 'XX%';

The result should return XX_CUSTOM and the flag should be Y.

- Aravind Kamath

Thursday, April 2, 2009

Concurrent Managers Crashing and FNDLIBR processes going defunct

Hi,

We observed an issue in one of our R12 instances where the concurrent managers were crashing and going defunct. There were no notable error messages in Internal Concurrent Manager log. The log indicated that all the concurrent queues were started up correctly as per the workshift definition.

However, the logs of the custom concurrent managers which were crashing had the following error:

APP-FND-00362: Routine afpbep cannot execute request &REQUEST for program &PROGRAM, because the environment variable &BASEPATH is not set for the application to which the concurrent program executable &EXECUTABLE belongs.

I had no reason to suspect any relinking issues even if we had patches applied to this system prior to this issue because the relink log did not show any failures. Also, there were other managers which use the FNDLIBR executable that were working fine without issues.

Further investigation revealed that in the log of the concurrent managers which were crashing, ahead of the aforesaid error, there was one concurrent request that was picked up by the manager for execution. Each of the crashed managers had tried to execute the request before they crashed.

I cancelled the identified custom concurrent request and looked up for similar requests and cancelled one more of such a request which was scheduled.

Then bring up the concurrent managers - they work like charm!

Further investigation revealed that the concurrent program definition was incorrect. Rather, the custom top under which this program was registered was not defined correctly. At the shell level, the custom top was defined as $XXCUST_TOP where as in
the custom application registration within apps, it was defined as CUST_TOP (Base Path).

I think the straight forward way of fixing this issue with the custom top is to update the base path in application registration to XXCUST_TOP.

- Aravind Kamath Posral

Wednesday, April 1, 2009

adopmnctl.sh: exiting with status 206

Hi,

Of late, I faced an issue while starting up the oacore component in our R12 environment. There were no errors logged in any of the log files except the error status code of 206 returned by opmnctl.sh .

ias-component/process-type/process-set:
OC4J/oacore/default_group/

Error
--> Process (index=1,uid=32979870,pid=13697)
failed to start a managed process after the maximum retry limit
Log:
/apps/local/INSTANCE_NAME/inst/apps/INSTANCE_NAME_SERVER_NAME/logs/ora/10.1.3/opmn/OC4J~oacore~default_group~1

ias-component/process-type/process-set:
OC4J/oafm/default_group/

Error
--> Process (index=1,uid=32979871,pid=13716)
failed to start a managed process after the maximum retry limit
Log:
/apps/local/INSTANCE_NAME/inst/apps/instance_name_server_name/logs/ora/10.1.3/opmn/OC4J~oafm~default_group~1


03/31/09-07:12:49 :: adopmnctl.sh: exiting with status 206



FIX:

- cd $ORA_CONFIG_HOME/10.1.3/j2ee/oacore/persistence/
- rm -rf *
- repeat the above step for oafm. The path will be cd $ORA_CONFIG_HOME/10.1.3/j2ee/oafm/persistence/

Troubleshooting Tips:

- use any command like lsof to see what files are being accessed to get an idea.

On the side lines, if you navigate to $IAS_ORACLE_HOME/opmn/bin and try to use opmnctl to check status or to set log levels, you will see an issue that says unable to read the opmn.xml file. For this, you need to set the $ORACLE_CONFIG_HOME to point to 10.1.3 instead of 10.1.2.

- Aravind Kamath

Sunday, February 1, 2009

AutoConfig could not successfully instantiate the following files: igccmprf.sql

Issue: While running autoconfig after applying R12 RUP 4 patch, we encountered the following error:

AutoConfig could not successfully instantiate the following files:

Directory: /apps/local/INSTANCE_NAME/inst/apps/CONTEXT_NAME/admin/install

igccmprf.sql INSTE8

It was crucial to get past the above error as autoconfig was being run as part of the post patching steps to propagate the changes brought into the system via this patch. However, it was relatively simple to fix this issue. The autoconfig log had the following entries:

SP2-0310: unable to open file "/apps/INSTANCE_NAME/apps/apps_st/comn/admin/install/CONTEXT_NAME/igccmprf.sql"

Some investigation revealed that this file is generated from the template file with the same name i.e. igccmprf.sql and was missing from the appl top.

Fix: Copy the file igccmprf.sql from $PATCH_TOP/igc/admin/template to $IGC_TOP/admin/template and rerun autoconfig.

Although the bigger question that needs to be answered is "why did the file not get copied by the patch?", we are at the moment content with getting past this issue and later revisit the above question to find an answer.

- Aravind Kamath Posral

Friday, January 23, 2009

Forms Server (frmsrv) crashing

Hi,

We were in the process of setting up a new instance by way of cloning. The apps version was 12.0.4 and database was 10.2.0.3 on linux. After the cloning process was complete, the services were started. We have configured the forms server in socket mode. The adformsrvctl.sh would start the forms server executable (frmsrv) but any connections would fail with FRM 92050. The forms executable would crash without leaving any messages in socket.log. No errors logged elsewhere too. The issue was with the library file libXm.so.2. This is apparently a motif file and I am not sure which rpm brings this in.

Troubleshooting tips:

1. set the environment
2. manually start the forms server with the following command
frmsrv host hostname port 9091 pool 1 exe frmweb log /apps/local/Instance_Name/inst/apps/INSTANCENAME_ServerNAME/logs/ora/10.1.2/forms/socket.log
3. The above command will throw the library could not be found error.
4. Ascertain the issue: ldd frmweb | grep 'not found'
5. The library libxm.so.2 should be made available under /usr/X11R6/lib/
6. The above path should be part of the $LD_LIBRARY_PATH
7. once the library file has been installed, relink the forms executable using the following command: make -f $ORACLE_HOME/forms/lib/ins_forms.mk install
8. Ascertain the executable is able to find the library file: ldd frmweb |grep 'not found'
9. Telnet from your desktop to the forms server (in this case on port 9091) and the frmsrv should not crash.

-Aravind Kamath Posral

Thursday, January 15, 2009

Verification of patch failed: Patch is not found in the Inventory.

Hi,

Happy New Year!!!

Problem:

SEVERE:OUI-67073:UtilSession failed: ApplySession failed in system modification phase... 'Verification of patch failed: Patch is not found in the Inventory.

'
INFO:--------------------------------------------------------------------------------
INFO:The following warnings have occurred during OPatch execution:
INFO:1) OUI-67124:Inventory check failed: Patch ID is NOT registered in Oracle Home inventory.
INFO:2) OUI-67124:ApplySession failed in system modification phase... 'Verification of patch failed: Patch is not found in the Inventory. '
INFO:3) OUI-67124:
NApply restored the home. Please check your ORACLE_HOME to make sure:
- files are restored properly.
- binaries are re-linked correctly.

We faced the above error while applying one of the patches using opatch.

Troubleshooting:

I checked upon the following aspects:

- The oracle home was properly defined in inventory.xml
- oraInst.loc was pointing to the correct inventory location
- OUI versions are adequate to update the inventory
- oratab contained the correct oracle home entry

None of above turned to be the culprits. This patch was being applied as part of the patches that were being applied to upgrade the database to 10.2.0.4 from

10.2.0.2. This prompted me to take a look at the oui log for this patchset and I found the following lines:

WARNING: Home:/oracle/product/10.2.0.4/OCDEV has conflicting information present in the inventory at /oracle/product/10.2.0.4/OCDEV/oraInventory/Contents
WARNING: Please delete the /oracle/product/10.2.0.4/OCDEV/oraInventory/Contents directory to avoid inventory update failures

Fix: As the above warning made it evident, I renamed the existing Contents folder and created an empty folder by the name Contents under oraInventory.

Applied the patch again using opatch and the issue was fixed.

- Aravind Kamath Posral