Generate Jar Files Hangs While Applying A Patch

Recently while i was applying a patch in our environment, the generation of jar files was hung. It would not progress. Now, on a windows 2000 server, it is known to us that when the number of characters in the path statement exceeds 1000 characters, the patch gets stuck at adjkey (signing the jar files process) which we refer to as "genjars.cmd". However, this time around it did not seem to be the case.

The problem was the missing "identitydb.obj" file under the %HOME% directory. The identitydb.obj file aka identity database, holds the certificate authority details alongside the certificate and is required to sign a jar file that is delivered by the patch.

Actually, it looks like that the profile of the owner of the applications file system, in other words, the domain user account profile that was used to install the apps, was deleted from this particular server.

- Aravind Kamath

Segmentation Violation - adrelink.sh, adlibout.sh

I was in the process of 'adding' an additional mid-tier node to our existing multi-node environment. After following the standard steps (cloning etc), this node went out of sync with other nodes in terms of patch levels, as this activity took a back seat making way to other priority issues. Hence, i started patching the new node that i was setting up, which was not patched alongside the other mid-tiers because it was not ready.

This activity brought to the forefront an issue with extraction of object modules / relinking of executables during patching. I encountered "Segmentation violation" error and the patch aborted. The patch log did not contain the segmentation violation error message. It would come up only in the command prompt where i was doing the patching.

I started troubleshooting this issue and after a while, i found that the issue was with MKS toolkit version. MKS Toolkit 8.0 patch 1 was required which actually fixed the issue.

Troubleshooting hints:

- A simple test.sh script which displays the hostname was executed successfully
- adrelink.sh would throw segmentation violation error.
- adlibout.sh would throw segmentation violation error
- At times, sourcing the environment file to relink would throw the above error
- Finally, invoking KSH from start>programs>MKS Toolkit>ksh would throw the segmentation fault.

This helped me eliminate other factors like environment files, environment variables, path etc and narrow down on MKS as just invoking ksh had nothing to do with oracle apps.

- Aravind Kamath

Oracle Home Inventory Issue

I was in the process of applying iAS Rollup 5 patch to our oracle applications environment. Despite following the steps to apply the patch, where in oracle_home, classpath, jdk home etc are set, the installer refused to continue with the following error:

"Not able to find ORACLE_HOME to be patched in the inventory please set the same properly and try again.."

There was a suggestion to create "Global Inventory" by applying the OUI patch and stuff, but i thought there should be a way to get past this without having to recreate the global inventory.

Here is how i fixed the issue:

- set the environment variable %ORAINST% to point to the directory that contained the inventory.xml file under %program files%\oui\..... This would ensure that the installer of the rollup patch would look for the inventory.xml in this location. (We have alternative location created by different oracle software installation)
- Most Importantly, edited the Oracle_Home name information in inventory.xml to read consistent with the information in the registry, ie HKLM\Software\Oracle\Home1(which is the $IAS_ORACLE_HOME). For some reason, the home names were inconsistent and which is why we saw the error stated above.
- Then I ran the installer and yes, the installation completed successfully.
- How do I know? ==>
- Log file of the installation.
- cd $IAS_ORACLE_HOME/Apache/Apache
- apache -v
- Output confirms that rup5 was successfully installed.

- Aravind Kamath

Mystery Connections from Concurrent Manager Server to Online Server

Hi,

It has been quite sometime since i posted my last "AppsDBATechStuff" blog. I was held up with certain "time-bound" activities, that occupied most of my time.

Now, i would like to share with you an interesting problem that i got to work on. We were working on certain memory management issues with Jserv in our environment and it was discovered that there were connections between the Online Servers and the Concurrent Manager server. Though it was not the likely culprit in the "memory" issue that we were chasing, it caught our attention and i started working on this.

Upon my investigation, i was able to determine that the connections were originating from the concurrent manager box and terminating on the Online Servers. I tied the ports and the processess together and determined that "Workflow Java Deferred Agent Listener" process was where the connections were originating and the "OA Core Group" Jserv was where the connection was terminating.

Looking at the logs (javacache.log), i was able to determine that Workflow Java Deferred Agent Listener was making these connections to all the Jservs on the online servers. Further analysis revealed that these connections were being made as part of the "Distributed mode Java Caching" framework. The "Database Invalidation" mechanism in which the WF Java Deferred Agent Listener processes the events relating to the data objects caching invalidation, is also responsible for notifying the Jserv that the data objects that were cached are now invalid.

This is the reason we saw the connections. Though we do not rely/leverage on the component cache mechanism, the above connections were seen as part of the framework being available and enabled. We continue to rely on "time based invalidation".


- Aravind Kamath