ld: Can't find library for -llibrt.so

Today when we were trying to setup a 10.2.0.4 code tree on one of our HPUX server, we encountered the following error while relinking the oracle executable.
ORACLE_SID=ASTCTC
Host Name: dvd-db109

Error:

/oracle/product/10.2.0.4/ASTCTC/rdbms/lib > make -f ins_rdbms.mk ioracle
chmod 755 /oracle/product/10.2.0.4/ASTCTC/bin
- Linking Oracle rm -f /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/oracle cc +DD64 -Wl,+s -Wl,+n +DD64 -o /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/oracle -L/oracle/product/10.2.0.4/ASTCTC/rdbms/lib/ -L/oracle/product/10.2.0.4/ASTCTC/lib/ /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/ssoident.o `if /usr/bin/getconf KERNEL_BITS grep "64" > /dev/null ; \ then echo "-Wl,+pi,64M -Wl,+pd,L -Wl,+padtext,16M -Wl,+paddata,1M" ; \ else echo "" ; fi` -Wl,-E -Wl,-PF,linkorderfile +FPd /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/opimai.o /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/ssoraed.o /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/ttcsoi.o /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/defopt.o -Wl,+forceload -lperfsrv10 -Wl,+noforceload /oracle/product/10.2.0.4/ASTCTC/lib/nautab.o /oracle/product/10.2.0.4/ASTCTC/lib/naeet.o /oracle/product/10.2.0.4/ASTCTC/lib/naect.o /oracle/product/10.2.0.4/ASTCTC/lib/naedhs.o /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/config.o -lserver10 -lodm10 -lnnet10 -lskgxp10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 -lhasgen10 -lcore10 -lskgxn2 -locr10 -locrb10 -locrutl10 -lhasgen10 -lcore10 -lskgxn2 -lclient10 -lvsn10 -lcommon10 -lgeneric10 -lknlopt `if /usr/ccs/bin/ar tv /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/libknlopt.a grep xsyeolap.o > /dev/null 2>&1 ; then echo "-loraolap10" ; fi` -lslax10 -lpls10 -lplp10 -lserver10 -lclient10 -lvsn10 -lcommon10 -lgeneric10 `if [ -f /oracle/product/10.2.0.4/ASTCTC/lib/libavserver10.a ] ; then echo "-lavserver10" ; else echo "-lavstub10"; fi` `if [ -f /oracle/product/10.2.0.4/ASTCTC/lib/libavclient10.a ] ; then echo "-lavclient10" ; fi` -lknlopt -lslax10 -lpls10 -lplp10 -ljox10 -lserver10 /oracle/product/10.2.0.4/ASTCTC/has/lib/libclsra10.a -ldbcfg10 -locijdbcst10 -lwwg `cat /oracle/product/10.2.0.4/ASTCTC/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lnro10 `cat /oracle/product/10.2.0.4/ASTCTC/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lmm -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `cat /oracle/product/10.2.0.4/ASTCTC/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lnro10 `cat /oracle/product/10.2.0.4/ASTCTC/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `if /usr/ccs/bin/ar tv /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/libknlopt.a grep "kxmnsd.o" > /dev/null 2>&1 ; then echo " " ; else echo "-lordsdo10"; fi` -lctxc10 -lctx10 -lzx10 -lgx10 -lctx10 -lzx10 -lgx10 -lordimt10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 -lsnls10 -lunls10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `cat /oracle/product/10.2.0.4/ASTCTC/lib/sysliblist` -lm `cat /oracle/product/10.2.0.4/ASTCTC/lib/sysliblist` -lm -lpthread -lpthreadld: Can't find library for -llibrt.soFatal error.
*** Error exit code 1
Stop.

Fix:

After the routine check for LD_LIBRARY_PATH, SHLIB_PATH etc, I zeroed in on the cc compiler. It turns out that cc was being picked up from a wrong path.

/oracle/product/10.2.0.4/ASTCTC/rdbms/lib > which cc
/usr/bin/cc
/usr/bin > ls -l cc
lrwxr-xr-t 1 root sys 17 Jul 9 01:00 cc@ -> /opt/ansic/bin/cc
/usr/ccs/bin > ls -l cc
lrwxrwxr-x 1 sfmdb 191 12 Aug 18 2008 cc@ -> ./cc_bundled
/usr/ccs/bin > export PATH=/usr/ccs/bin:$PATH
/usr/ccs/bin > which cc
/usr/ccs/bin/cc
/usr/ccs/bin > cc -V
(Bundled) cc: HP aC++/ANSI C B3910B A.05.50 [May 15 2003]

In our case, the above version is the desired one. Refer to your installation manual to see which version of cc is compatible/required for your version of OS and Oracle binaries.

/usr/ccs/bin > echo $ORACLE_HOME
/oracle/product/10.2.0.4/ASTCTC
/usr/ccs/bin > echo $SHLIB_PATH
/oracle/product/10.2.0.4/ASTCTC/lib32:/oracle/product/10.2.0.4/ASTCTC/lib
/oracle/product/10.2.0.4/ASTCTC/rdbms/lib > make -f ins_rdbms.mk ioracle
chmod 755 /oracle/product/10.2.0.4/ASTCTC/bin
- Linking Oracle
rm -f /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/oracle cc +DD64 -Wl,+s -Wl,+n +DD64 -o /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/oracle -L/oracle/product/10.2.0.4/ASTCTC/rdbms/lib/ -L/oracle/product/10.2.0.4/ASTCTC/lib/ /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/ssoident.o `if /usr/bin/getconf KERNEL_BITS grep "64" > /dev/null ; \ then echo "-Wl,+pi,64M -Wl,+pd,L -Wl,+padtext,16M -Wl,+paddata,1M" ; \ else echo "" ; fi` -Wl,-E -Wl,-PF,linkorderfile +FPd /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/opimai.o /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/ssoraed.o /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/ttcsoi.o /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/defopt.o -Wl,+forceload -lperfsrv10 -Wl,+noforceload /oracle/product/10.2.0.4/ASTCTC/lib/nautab.o /oracle/product/10.2.0.4/ASTCTC/lib/naeet.o /oracle/product/10.2.0.4/ASTCTC/lib/naect.o /oracle/product/10.2.0.4/ASTCTC/lib/naedhs.o /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/config.o -lserver10 -lodm10 -lnnet10 -lskgxp10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 -lhasgen10 -lcore10 -lskgxn2 -locr10 -locrb10 -locrutl10 -lhasgen10 -lcore10 -lskgxn2 -lclient10 -lvsn10 -lcommon10 -lgeneric10 -lknlopt `if /usr/ccs/bin/ar tv /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/libknlopt.a grep xsyeolap.o > /dev/null 2>&1 ; then echo "-loraolap10" ; fi` -lslax10 -lpls10 -lplp10 -lserver10 -lclient10 -lvsn10 -lcommon10 -lgeneric10 `if [ -f /oracle/product/10.2.0.4/ASTCTC/lib/libavserver10.a ] ; then echo "-lavserver10" ; else echo "-lavstub10"; fi` `if [ -f /oracle/product/10.2.0.4/ASTCTC/lib/libavclient10.a ] ; then echo "-lavclient10" ; fi` -lknlopt -lslax10 -lpls10 -lplp10 -ljox10 -lserver10 /oracle/product/10.2.0.4/ASTCTC/has/lib/libclsra10.a -ldbcfg10 -locijdbcst10 -lwwg `cat /oracle/product/10.2.0.4/ASTCTC/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lnro10 `cat /oracle/product/10.2.0.4/ASTCTC/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lmm -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `cat /oracle/product/10.2.0.4/ASTCTC/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lnro10 `cat /oracle/product/10.2.0.4/ASTCTC/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `if /usr/ccs/bin/ar tv /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/libknlopt.a grep "kxmnsd.o" > /dev/null 2>&1 ; then echo " " ; else echo "-lordsdo10"; fi` -lctxc10 -lctx10 -lzx10 -lgx10 -lctx10 -lzx10 -lgx10 -lordimt10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 -lsnls10 -lunls10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `cat /oracle/product/10.2.0.4/ASTCTC/lib/sysliblist` -lm `cat /oracle/product/10.2.0.4/ASTCTC/lib/sysliblist` -lm -lpthread -lpthread mv -f /oracle/product/10.2.0.4/ASTCTC/bin/oracle /oracle/product/10.2.0.4/ASTCTC/bin/oracleO mv /oracle/product/10.2.0.4/ASTCTC/rdbms/lib/oracle /oracle/product/10.2.0.4/ASTCTC/bin/oracle chmod 6751 /oracle/product/10.2.0.4/ASTCTC/bin/oracle

As you can see above, the relink was successful.
- Aravind Kamath Posral