Oracle E-Business Suite R12 Upgrade

August 2009 Vasu Balla Pythian

Oracle E-Business Suite R12 Upgrade p.2

A typical Oracle E-Business Suite R12 upgrade (from 11i) is normally not meant to be that complicated as the procedure is pretty much detailed in the Oracle Applications Upgrade Guide, which as we all know, can be downloaded from OTN (Oracle Technology Network). However, not everything in life is typical. During the past few months, we (Pythian) were one of the prime technical Apps database experts on a rather interesting and challenging Apps R12 upgrade for one of our clients, which entailed quite a few particularities. As a note: during the upgrade, we encountered several challenges, which we were able to solve with very limited Oracle Tech Support assistance, resulting in significant time savings. As a requirement, the final Apps R12 upgraded system (12.0.6 in this case) needed the following configuration specifications: SSL enabled load balanced external users to access iSupplier external users to access iReceivable external users to be authenticated through NAM3 (Novell Authentication Manager 3.0) have a reverse proxy server between NAM3 and the Load Balancer

Not only was the R12 upgrade successful it was achieved in 48 hrs as opposed to the estimated 72 hrs initially projected. As anyone would expect, merging the patches played an important role in speeding up the overall upgrade time. We ended up merging all Post 12.0.4 upgrade patches (including the 12.0.6 R12 RUP6 Patch) in which, we applied the rule of thumb of merging both "non-AD" and "non-FND" patches together and thereafter, to individually apply AD, FND and online-Help patches. Merging the patches did save a lot of time, both in limiting the number of adpatch runs, and also in limiting the number of times the actual adpatch maintenance steps were executed (i.e. compiling Apps schema, compiling jtf files, etc.). Following are the main issues we encountered and how we fixed them. Obviously, every Apps R12 upgrade is unique as it is based on the existing setup, version upgrading from, and R12 version upgrading to. The intent of this paper is to share our experience in the hopes that you can glean some insights to help with your next upgrade.

Oracle E-Business Suite R12 Upgrade p.3

ISSUES
Purging Non-Required Data From the "GL_INTERFACE" Table Dramatically Reduced Overall Patch Runtime In reviewing the patch runtime worker jobs, we've clearly noticed that script "jaiglintfsrcmig.sql" was taking roughly 4 hrs 55 mins to complete. After carefully reviewing the script's SQL path, we noticed that the script was actually doing several updates on the "GL_INTERFACE" table. And this was occurring on an implementation where table "GL_INTERFACE" is (was) not being purged after data loads. We then proceeded to ask the client's Development team to identify the non required data from the "GL_INTERFACE" table, which was then purged prior to engaging the Apps upgrade. Purging the non-required data from the "GL_INTERFACE" table, turned out to save approximately 4 1/2 hours of upgrade time. Apache Lock File Issues

In setting a second and third Mid-Tier nodes, we've successfully completed the configuration of a shared APPL_TOP and shared Tech Stack. Redhat GFS was used for the shared filesystem. The only issue faced with the shared filesystem, was that Apache would fail to startup due to the "adapcctl.sh" startup script timing out. After carefully diagnosing the issue (using the "strace" command to see where the "httpd" processes were hanging), we identified that the issue resulted in Apache trying to lock files on the GFS mountpoints. This appeared to be a generic issue with most shared filesystem when locking files take place (which is in fact an Apache requirement). The issue was fixed by configuring Apache parameters "s_lock_pid_dir", "s_pids_dir" and "s_web_pid_file" (in the context xml file) to point to local mount point "/var/tmp" (instead of pointing to shared mountpoints). These changes were made to all context xml files for all nodes in the cluster. R12 Login Page not Working After Applying the 12.0.4 Patch

After applying the 12.0.4 patch, for some reason, the R12 login page stopped working properly, more specifically, the R12 login page would loop between "/OA_HTML/AppsLogin" and "/OA_HTML/RF.jsp?" and consequently, the login page would not come up at all. After thorough troubleshooting, we determined that the FND tables and WF tables were not completely synchronized, thus making the "RF.jsp" go back to the AppsLogin page as user "GUEST" which in fact, does not have necessary responsibilities against WF tables. The problem was then fixed by running the "WF_LOCAL_SYNCH.VALIDATEUSERROLES" procedure. Scrolling with the Mouse Causes the Forms Session to Crash

Yes, scrolling with the mouse in the Forms screens was indeed causing the users' Forms session to crash. When researching Metalink, the only solution found was to actually patch up

Oracle E-Business Suite R12 Upgrade p.4

the Mid-Tier 10.1.2 Oracle Home to 10.1.2.3 and also to patch the 10.1.3 Oracle Home to 10.1.3.4. Applying the two Oracle Home Patchsets did fix the problem. Oracle does not yet ship the latest fixed Techstack version components with the 12.0.4 R12 Media, however the fixed versions might very well be shipped with the 12.1 Apps release. Intermittent Issues

At one point, we were faced with two types of intermittent issues. The first was with respect to a report customization where the "Payables Open Interface Import" program used to fail intermittently with error "REP-0069: Internal error, REP-50002: Server is shutting down". After researching this issue, we've identified patch 6116405 from Metalink Note 549910.1 "Error REP-0069 and REP-50002 while Generating Reports through Concurrent Managers", which in the end, fixed the issue. The other intermittent issue dealt with Apache producing errors of the form "404 not found in OA Framework pages". After carefully researching and diagnosing the issue, we narrowed it down to OC4J not behaving correctly and further more, identified that the error would occur only when OC4J oacore jvms were increased from 1 to 3. We then tracked down Metalink Note 731115.1, specifying to apply patch 7311892 to the 10.1.3 Oracle Home and also to add set context file parameter "s_oacore_jvm_start_options" to "-Doracle.ons.numprocs=<n>", and of course to also run Autoconfig to propgagate all changes. This fixed the Apache errors of the form "404 not found in OA Framework pages". Fixing OA Framework Custom Application Issues

Pythian also contributed in helping the client's development team with fixing custom application issues based on the OA framework. The application was failing through R12 Apache's mod_security configuration file "security.conf", which in fact was checking for some characters that could be used for SQL injection attacks. In this case, the custom application was using square brackets "[ ]" in variable names, thus causing Apache to filter URLs resulting in access blocked errors. Issue was then resolved by customizing the "security.conf" Autoconfig template to exclude square brackets from being filtered.

Oracle E-Business Suite R12 Upgrade p.5

Discover Upgrade 4i to 10g

As part of this upgrade project, Discoverer also had to undergo an upgrade from 4i to 10g. The issue encountered with the Discoverer upgrade was that the import process for the Discoverer Report "eex" files, caused them to hang at different times (during the import process). Applying the latest rollup patch on the 10g Discoverer setup, turned out to fix this Discoverer Report "eex" file import "hanging" issue.

SET UP Challenges
Implementing SSL Using Internal Certificate Authorities Just like several Oracle E-Business Suite implementations, one of the client's requirements was to implement SSL for their R12 configuration. What made it interesting was the fact that the client does not obtain their SSL Certificates from third party Certificate Authorities (ie Verisign), but rather uses their own internal CA for generating certificates for all their required hosts. Another interesting aspect was the fact that all certificates were "chained", implying that not all certificates were generated by the Internal CA, but rather generated by sub CAs in order to then be targeted for different systems. We successfully managed to configure SSL for the Apps R12 configuration using these internally generated chained certificates. It is also important to note here that we needed to make sure that all CAs in the certificate chain were properly imported in the wallets before importing the actual user certificate. Following this simple procedure made it possible to have internally generated certificates to work perfectly without any issues. Load Balancer Setup

Another of the client's requirements was to implement a Load Balancer (Cisco Content Switch). For optimal configuration, Pythian recommended to configure the L/B to use "sticky session" (using the cookie insert method) and also to set the L/B session timeout to 8 hrs. The purpose of this setup was to avoid users getting "unknown session" related errors throughout their working day. Sticky session using the cookie insert option provides great advantage for loop back connections (connections originating and coming back on the same middle tier). If connections don't come back on the same mid-tier server as they originated, then certain Apps applications like Configurator and iReceivables will inevitably end up having issues. With the cookie insert method, the loop back connections ensure that the user connection actually carries out the same cookie sent by the user, thereby coming back on the same mid-tier server as the user's request initially originated from.

Oracle E-Business Suite R12 Upgrade p.6

Apps Integration With NAM3 (Novell Account Manager 3.0)

Another very specific requirement for this client's E-Business Suite architecture was the underlying 11i integration with Novell's NAM3 Identity management software. Instead of using Oracle's OID to integrate with NAM3, the client is using a NAM3 feature to integrate 11i with it. Under the client's Apps 11i functionality, external users are channeled through NAM3's login page and once authenticated, NAM3 internally does a form fill of the Apps 11i login page which in turn, opens up the Apps 11i home page for that user. In addition to authenticating the users, NAM3 also works as a reverse proxy and passes all traffic through it. It modifies all the content of the traffic to use NAM3 URLs only. Direct Access to the 11i environment is not open to external users. All access is via NAM3 server only. This behavior is specific while operating under Apps 11i. A new problem arose with the R12 upgrade. Unlike the 11i login page "AppsLocalLogin.jsp", the R12 login page is not static. The R12 login page uses many hidden fields which are dynamically generated. Because of this, NAM3 was not able to perform a successful form fill of the R12 login page. The only supported Oracle method for third party Identity Management integration with R12 is actually only with Oracle's OID. Facing this challenge, Pythian's proposed solution was to integrate Apps R12 with Oracle's OID, and then use the OID login page (which in essence is static) to undergo the form fill process through NAM3. However, the fact that NAM3 internally changes URLs which are internally passed through it, caused a new problem to this whole login page process. To this, OID was giving errors when R12 URLs were modified to internal NAM3 URLs. To resolve this problem, Pythian proposed to implement virtual hosts on the Apps mid-tier servers and have Apache to run on these defined virtual hosts. These virtual hosts were to match the URLs used by NAM3, therefore OID would not have any problem authenticating the users. With this, Pythian successfully implemented the virtual hosts on the same mid-tier servers and enabled both Apache and oacore services to run using different URLs for logging in to the same R12 Instance. This setup is somewhat a bit like a DMZ node running on an internal node itself. Furthermore, all virtual hosts are simply seen as other nodes listed in the FND_NODES table, and they are not sharing the dbc file with the existing middle tiers. Server level profile options are also enabled to allow multiple login URLs. In a summarized way, Pythian configured two sets of virtual hosts on the same middle tiers (one for iSupplier and one for the iReceivable applications). Both these applications are used and accessed by external users via NAM3. This setup was successfully tested by end users and was later confirmed as properly working, which then turned out to be the final solution for the Apps R12 Production architecture. Reverse Proxy Setup Required

Another security architectural requirement came from the client's Network team, which was to configure Apache to be working in reverse proxy between the Load Balancer and the NAM3. Opening ports between NAM3 and the R12 Load balancer were seen as a security concern. To

Oracle E-Business Suite R12 Upgrade p.7

counter this, Pythian worked with the client's DBA and Unix teams to configure Apache to run on a separate set of servers and configured it to pass on the requests between NAM3 and the Load Balancer. SSL was also implemented in reverse proxy mode for achieving the highest security level possible. As there were two external application URLs for NAM3, Pythian also implemented virtual hosts in Reverse Proxy mode so that a single Apache could handle proxy requests for both R12 External URLs. No reverse proxy server had to be configured for internal URLs as the reverse proxy configuration was setup only as an extra security layer, working between NAM3 and the R12 load balancer. Upgrade/Go-Live Weekend

To have an efficient production upgrade process, Pythian thoroughly reviewed the Production upgrade project plan and worked with the client's DBA Team and included the virtual nodes and reverse proxy setups in the Production plan. It was also of utmost importance to group similar activities together in order to reduce as much as possible the number of Autoconfig runs. Pythian worked with the client's DBA team to make it a 24/7 upgrade activity and was able to complete the full production upgrade in 48 hours as opposed to the original estimated 72 hours. The 24 hour accelerated difference was accredited to both having established a very good production upgrade plan and also, to the execution speed of the powerful production servers. The production upgrade was truly a success.