Friday, November 20, 2009

RC-10203: The template provided does not match with the context file

We recently faced the following issue in one of our instances while running adclonectx.pl to clone the xml file:


AC-10006: Exception - java.io.FileNotFoundException: /appl_top/ad/11.5.0/admin/template/adxmlctx.tmp thrown while creating OAVars object for file:
../admin/template/adxmlctx.tmpRC-50004: Error occurred in CloneContext: RC-10203: The template provided does not match with the context file.Check logfile /apps/sdev/appl_top/admin/sdev/out/CloneContext_11190814.log for details.ERROR: context creation not completed successfully.Please check /tmp/adclonectx.err file for errors.

The adclonectx.pl script was not picking up the correct $AD_TOP path which is why it was throwing this error. To fix this issue, I used the following command:
perl adclonectx.pl sharedappltop contextfile=/apps/sdev/sdev.xml template=/apps/sdev/appl_top/ad/11.5.0/admin/template/adxmlctx.tmp
pairsfile=/apps/sdev/pre_Autoconfig/pairsfile_sdev.txt

Pass the template parameter as part of the adclonectx.pl script to provide the full path of the adxmlctx.tmp, which is the template referred here.


Subsequently, the command ran without any issues:
/apps/sdev/appl_top/ad/11.5.0/bin > perl adclonectx.pl sharedappltop contextfile=/apps/sdev/sdev.xml template=/apps/sdev/appl_top/ad/11.5.0/admin/template/adxmlctx.tmp
pairsfile=/apps/sdev/pre_Autoconfig/pairsfile_sdev.txt/apps/sdev/common_top/util/jre/1.1.8/bin/jre -classpath
/apps/sdev/product/iAS/jdbc/lib/classes111.zip:/apps/sdev/common_top/java/xmlparserv2.zip:/apps/sdev/common_top/util/jre/1.1.8/lib/rt.jar:/apps/sdev/common_top/java:/a pps/sdev/common_top/util/jre/1.1.8/lib/rt.jar:/apps/sdev/common_top/util/jre/1.1.8/lib/i18n.jar:/apps/sdev/common_top/util/jre/1.1.8/lib/tools.jar:/apps/sdev/common_to p/java/appsborg.zip:/apps/sdev/common_top/java/apps.zip:/apps/sdev/product/8.0.6/forms60/java:/apps/sdev/common_top/java oracle.apps.ad.context.CloneContext -e
/apps/sdev/sdev.xml -tmpl /apps/sdev/appl_top/ad/11.5.0/admin/template/adxmlctx.tmp -pairsfile /apps/sdev/pre_Autoconfig/pairsfile_sdev.txt -sharedappltop -perl
"/apps/sdev/product/iAS/Apache/perl/bin/perl" -perl5lib
"/apps/sdev/product/iAS/Apache/perl/lib/5.00503:/apps/sdev/product/iAS/Apache/perl/lib/site_perl/5.005:/apps/sdev/appl_top/au/11.5.0/perl"
Provide the values required for creation of the new APPL_TOP Context file.
Do you want to use a virtual hostname for the target node (y/n) [n] ?:
Do you want to add Admin Server node (y/n) [y] ?:
Do you want to add Concurrent Processing Server node (y/n) [y] ?:
Do you want to add Forms Server node (y/n) [y] ?:
Do you want to add Web Server node (y/n) [y] ?:
Target system 8.0.6 ORACLE_HOME directory [/apps/sdev/product/8.0.6]:
Target system iAS ORACLE_HOME directory [/apps/sdev/product/iAS]:
Location of the JDK on the target system [/opt/java1.3]:Database port is 1521
Do you want the middle-tier services on the target system to have the same port values as the source system (y/n) [y] ?:Web Listener port is 8903Complete port information available at /apps/sdev/appl_top/admin/out/sdev_thus/portpool.lst
APPS password is required to validate database connection using database port 1521Enter APPS Password [APPS]:apps
Target system proxy hostname [name.domain.com]:
Target system proxy port [80]:
UTL_FILE_DIR on database tier consists of the following directories.
1. /apps/orarpt/sdev/utlChoose a value which will be set as APPLPTMP value on the target node [1]:
Creating the new APPL_TOP Context file from : /apps/sdev/appl_top/ad/11.5.0/admin/template/adxmlctx.tmp
The new APPL_TOP context file has been created : /apps/sdev/appl_top/admin/sdev_thus.xml
Log file located at /apps/sdev/appl_top/admin/sdev/out/CloneContext_11190840.log

Tuesday, November 10, 2009

FRM-40735: POST-UPDATE trigger raised unhandled exception ORA-04062.

We encountered a strange nagging issue when users were accessing a form through one of the forms server node. They were facing the following error:
ORA-04062: signature of package "APPS.XYCCA_HDRB_EXTN_PCK" has been changed

FRM-40735: POST-UPDATE trigger raised unhandled exception ORA-04062.


We did not suspect the database package because users connected to other three nodes were perfectly fine and were able to open the same form without issues. Bounce of forms server on the erring node did not help either. Environment, db connectivity and all other form server related settings were perfectly fine.


It turns out that the erring node was seeing an "older" image of the pll file referenced by the form. We have a shared file system implemented in this system and the other three nodes were showing the correct latest version of the file. We suspect a NFS cache issue at this time but the fix was to touch a dummy file from the erring node on the file system where the plls were residing.
Unmount and mount on erring node, i am sure, will fix the issue but will require a bounce. In our case, once we touched a dummy file, the erring node started to see the current version of the plls in question and we were able to get past the issue.
- Aravind Kamath Posral

Sunday, November 8, 2009

SQL*Net break/reset to dblink and SQL*Net break/reset to client

We faced a strange situation where one of our concurrent programs was sitting "idle" doing nothing, although the status of the session showed active. The program connects to a remote database over db link, runs a select query and the output of select query is inserted into a table. However, in this situation, the session on the remote database was inactive and the wait event was "SQL*Net break/reset to client". In the concurrent program session, the wait event was "SQL*Net break/reset to dblink".

After some troubleshooting effort that revealed nothing, we decided to create a dummy table with the same structure and run the select/insert statement manually. This gave out the cause of the issue:

ORA-12899: value too large for column string (actual: string, maximum: string)

Actually, the data fetched from the remote database was not being populated in the table because of the above error.

I strongly believe that this is the result of poor design as the concurrent program should have taken care of the exception handling.

Nevertheless, since this is a data issue, there is nothing much DBA can do and hence we transferred the issue to our dev team to take care of this.

PS: Thought of posting this as there are not many articles out there about the aforesaid wait events, which are not related to bugs! Almost every hit on the above wait event will lead you to bug description or a tar to which one will not have access. In our case, the workaround was helpful to trap the error.
- Aravind Kamath Posral

Tuesday, November 3, 2009

ld: Can't find library for -llibrt.so

Today when we were trying to setup a 10.2.0.4 code tree on one of our HPUX server, we encountered the following error while relinking the oracle executable.
ORACLE_SID=ASTCTC
Host Name: dvd-db109

Error:

/oracle/product/10.2.0.4/ASTCTC/rdbms/lib > make -f ins_rdbms.mk ioracle
chmod 755 /oracle/product/10.2.0.4/ASTCTC/bin
- Linking Oracle rm -f /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/oracle cc +DD64 -Wl,+s -Wl,+n +DD64 -o /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/oracle -L/oracle/product/10.2.0.4/ASTCTC/rdbms/lib/ -L/oracle/product/10.2.0.4/ASTCTC/lib/ /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/ssoident.o `if /usr/bin/getconf KERNEL_BITS grep "64" > /dev/null ; \ then echo "-Wl,+pi,64M -Wl,+pd,L -Wl,+padtext,16M -Wl,+paddata,1M" ; \ else echo "" ; fi` -Wl,-E -Wl,-PF,linkorderfile +FPd /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/opimai.o /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/ssoraed.o /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/ttcsoi.o /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/defopt.o -Wl,+forceload -lperfsrv10 -Wl,+noforceload /oracle/product/10.2.0.4/ASTCTC/lib/nautab.o /oracle/product/10.2.0.4/ASTCTC/lib/naeet.o /oracle/product/10.2.0.4/ASTCTC/lib/naect.o /oracle/product/10.2.0.4/ASTCTC/lib/naedhs.o /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/config.o -lserver10 -lodm10 -lnnet10 -lskgxp10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 -lhasgen10 -lcore10 -lskgxn2 -locr10 -locrb10 -locrutl10 -lhasgen10 -lcore10 -lskgxn2 -lclient10 -lvsn10 -lcommon10 -lgeneric10 -lknlopt `if /usr/ccs/bin/ar tv /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/libknlopt.a grep xsyeolap.o > /dev/null 2>&1 ; then echo "-loraolap10" ; fi` -lslax10 -lpls10 -lplp10 -lserver10 -lclient10 -lvsn10 -lcommon10 -lgeneric10 `if [ -f /oracle/product/10.2.0.4/ASTCTC/lib/libavserver10.a ] ; then echo "-lavserver10" ; else echo "-lavstub10"; fi` `if [ -f /oracle/product/10.2.0.4/ASTCTC/lib/libavclient10.a ] ; then echo "-lavclient10" ; fi` -lknlopt -lslax10 -lpls10 -lplp10 -ljox10 -lserver10 /oracle/product/10.2.0.4/ASTCTC/has/lib/libclsra10.a -ldbcfg10 -locijdbcst10 -lwwg `cat /oracle/product/10.2.0.4/ASTCTC/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lnro10 `cat /oracle/product/10.2.0.4/ASTCTC/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lmm -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `cat /oracle/product/10.2.0.4/ASTCTC/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lnro10 `cat /oracle/product/10.2.0.4/ASTCTC/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `if /usr/ccs/bin/ar tv /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/libknlopt.a grep "kxmnsd.o" > /dev/null 2>&1 ; then echo " " ; else echo "-lordsdo10"; fi` -lctxc10 -lctx10 -lzx10 -lgx10 -lctx10 -lzx10 -lgx10 -lordimt10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 -lsnls10 -lunls10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `cat /oracle/product/10.2.0.4/ASTCTC/lib/sysliblist` -lm `cat /oracle/product/10.2.0.4/ASTCTC/lib/sysliblist` -lm -lpthread -lpthreadld: Can't find library for -llibrt.soFatal error.
*** Error exit code 1
Stop.

Fix:

After the routine check for LD_LIBRARY_PATH, SHLIB_PATH etc, I zeroed in on the cc compiler. It turns out that cc was being picked up from a wrong path.


/oracle/product/10.2.0.4/ASTCTC/rdbms/lib > which cc
/usr/bin/cc
/usr/bin > ls -l cc
lrwxr-xr-t 1 root sys 17 Jul 9 01:00 cc@ -> /opt/ansic/bin/cc
/usr/ccs/bin > ls -l cc
lrwxrwxr-x 1 sfmdb 191 12 Aug 18 2008 cc@ -> ./cc_bundled
/usr/ccs/bin > export PATH=/usr/ccs/bin:$PATH
/usr/ccs/bin > which cc
/usr/ccs/bin/cc
/usr/ccs/bin > cc -V
(Bundled) cc: HP aC++/ANSI C B3910B A.05.50 [May 15 2003]

In our case, the above version is the desired one. Refer to your installation manual to see which version of cc is compatible/required for your version of OS and Oracle binaries.

/usr/ccs/bin > echo $ORACLE_HOME
/oracle/product/10.2.0.4/ASTCTC
/usr/ccs/bin > echo $SHLIB_PATH
/oracle/product/10.2.0.4/ASTCTC/lib32:/oracle/product/10.2.0.4/ASTCTC/lib
/oracle/product/10.2.0.4/ASTCTC/rdbms/lib > make -f ins_rdbms.mk ioracle
chmod 755 /oracle/product/10.2.0.4/ASTCTC/bin
- Linking Oracle
rm -f /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/oracle cc +DD64 -Wl,+s -Wl,+n +DD64 -o /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/oracle -L/oracle/product/10.2.0.4/ASTCTC/rdbms/lib/ -L/oracle/product/10.2.0.4/ASTCTC/lib/ /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/ssoident.o `if /usr/bin/getconf KERNEL_BITS grep "64" > /dev/null ; \ then echo "-Wl,+pi,64M -Wl,+pd,L -Wl,+padtext,16M -Wl,+paddata,1M" ; \ else echo "" ; fi` -Wl,-E -Wl,-PF,linkorderfile +FPd /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/opimai.o /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/ssoraed.o /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/ttcsoi.o /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/defopt.o -Wl,+forceload -lperfsrv10 -Wl,+noforceload /oracle/product/10.2.0.4/ASTCTC/lib/nautab.o /oracle/product/10.2.0.4/ASTCTC/lib/naeet.o /oracle/product/10.2.0.4/ASTCTC/lib/naect.o /oracle/product/10.2.0.4/ASTCTC/lib/naedhs.o /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/config.o -lserver10 -lodm10 -lnnet10 -lskgxp10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 -lhasgen10 -lcore10 -lskgxn2 -locr10 -locrb10 -locrutl10 -lhasgen10 -lcore10 -lskgxn2 -lclient10 -lvsn10 -lcommon10 -lgeneric10 -lknlopt `if /usr/ccs/bin/ar tv /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/libknlopt.a grep xsyeolap.o > /dev/null 2>&1 ; then echo "-loraolap10" ; fi` -lslax10 -lpls10 -lplp10 -lserver10 -lclient10 -lvsn10 -lcommon10 -lgeneric10 `if [ -f /oracle/product/10.2.0.4/ASTCTC/lib/libavserver10.a ] ; then echo "-lavserver10" ; else echo "-lavstub10"; fi` `if [ -f /oracle/product/10.2.0.4/ASTCTC/lib/libavclient10.a ] ; then echo "-lavclient10" ; fi` -lknlopt -lslax10 -lpls10 -lplp10 -ljox10 -lserver10 /oracle/product/10.2.0.4/ASTCTC/has/lib/libclsra10.a -ldbcfg10 -locijdbcst10 -lwwg `cat /oracle/product/10.2.0.4/ASTCTC/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lnro10 `cat /oracle/product/10.2.0.4/ASTCTC/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lmm -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `cat /oracle/product/10.2.0.4/ASTCTC/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lnro10 `cat /oracle/product/10.2.0.4/ASTCTC/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `if /usr/ccs/bin/ar tv /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/libknlopt.a grep "kxmnsd.o" > /dev/null 2>&1 ; then echo " " ; else echo "-lordsdo10"; fi` -lctxc10 -lctx10 -lzx10 -lgx10 -lctx10 -lzx10 -lgx10 -lordimt10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 -lsnls10 -lunls10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `cat /oracle/product/10.2.0.4/ASTCTC/lib/sysliblist` -lm `cat /oracle/product/10.2.0.4/ASTCTC/lib/sysliblist` -lm -lpthread -lpthread mv -f /oracle/product/10.2.0.4/ASTCTC/bin/oracle /oracle/product/10.2.0.4/ASTCTC/bin/oracleO mv /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/oracle /oracle/product/10.2.0.4/ASTCTC/bin/oracle chmod 6751 /oracle/product/10.2.0.4/ASTCTC/bin/oracle

As you can see above, the relink was successful.
- Aravind Kamath Posral