When one talks about Oracle E-Business suite implementation following questions about data arise:

How existing legacy system data will be available in new system?

Will there be a complete move from OLD system to NEW system or both systems will co-exists for
sometime before complete cut over?
Will there be any communication needed between OLD and NEW system?
Most important is HOW to achieve it?
While discussing about these questions few terms get used interchangeable viz. Data Migration, Data
conversion, Interfaces etc. All can be brought under one umbrella/category called Data Movement.
This white paper talks about various strategies/technologies which can be used for data communication
between legacy system and Oracle E-Business Suite. This gives the overview about various technical
components involved in taking care of most important word called "DATA".
This white paper tries to give direction to answer above questions. To make legacy data available to new
system, one has to perform data entry manually or use tools like data loader, win-runner which simulate
keyboard/mouse action and make data entry faster. If data is huge the programmer/developers help is
needed to build the program to automate data loading. If two systems are going to coexists for some
time and communicate with each other then programs which run at regular intervals to move the data are
needed.


Defining Various Terminologies
Data Migration is a process of moving required (and most often very large) volumes of data from our
clients' existing systems to new systems. Existing systems can be anything from custom-built IT
infrastructures to spreadsheets and standalone databases.
Data conversion can be defined as a process of converting data from one structural form to another to
suit the requirements of the system to which it is migrated.
Interface is set of programs for connection between two systems in order to synchronize the Data. They
can be Manual, Batch or Real-Time. They are used repeatedly and should therefore be designed and
constructed in the most efficient manner possible.
Though Data Migration, conversion, interfaces are treated differently but technical folks should not
differentiate between them. The design of the all program should same like interfaces, because in most of
the implementation data need to be moved multiple times and for some time both legacy system and
Oracle Apps 11i happened to be active. Bottom line if data needs to be moved programmatically i.e.
without using tools like data loader or manual data entry then it is better to design the program which
can be execute multiple time as a concurrent program.


Data Movement Entities
Generally data conversion/interfaces include following entities
1. Legacy System (Oracle/Other database e.g. DB2/File System)
2. Oracle Apps custom staging tables
3. Shell scripts to read data files and load them in custom staging tables using SQL*Loader utility
4. Oracle Apps custom data validation programs
5. Oracle seeded open interface tables (e.g. MTL_TRANSACTIONS_INTERFACE)
6. Oracle seeded APIs (e.g. Process Order API)
7. Oracle Apps Seeded Import Program (e.g. Item Import, BOM Import)
Legacy System
There can be various types of legacy systems which can be based on databases like Oracle, DB2 etc or
based on file systems. Generally it is better to use files extracted from legacy system while development of
data conversion program as data volume is very high. DB Links can be used for connecting to other
databases while developing interfaces.
1. ASCII files :- As a practice most of implementations have ASCII files as an input to oracle. These files
can be comma/pipe/tab delimited files with certain file format. If excel files are received them they
need to be converted to CSV file manually.
2. XML files :-Now days XML (Extended Markup Language) is becoming the standard world wide most
of the system accept and understand XML format. Source system can generate XML files based on
format defined by DTD (Document Type Definition) or XSD (Xml Schema Definition). DTD/XSD
defines the structure/tags present in XML document.
3. Legacy Outbound staging tables :- Legacy system/home grown system can be based on databases
like Oracle, DB2 etc. Now days, there are drivers available in oracle to connect to other non-oracle
databases using DB Links. If the data volume is less then it is better to use DB Links that ASCII file so
that we can get read of FTP/SFTP failure issues. Also it is easier to update database tables directly to
update the current status of record (Success/Error etc) instead of exchanging files
Oracle Apps 11i E-Business Suite Components
Oracle Apps Custom staging tables :- As a thumb rule the structure of custom staging table should be
same as the source data file or source legacy database table. There should be presence of columns like
Interface_id, Batch_ID, Process_Flag, Error_Message, WHO columns and Extended WHO columns in the
table structure.
Shell Script with SQL*Loader:- Shell script (host) type of concurrent program should be used to load
data files in custom staging tables instead of SQL*Loader executable type. Shell script internally uses
SQL*Loader command line utility to load data and archives the data files. With usage of shell script the
data file path is not hard coded, archiving monitoring bad file is easily achievable which is not the case
with SQL*Loader executable type. As a rule there should not be hard coding in shell scripts e.g. directory
paths, username, password etc.Inbound data files can be placed in directories like
$XXINV_TOP/datafiles/in.
Shell script should read these files and load in custom tables and archive data files in
$XXINV_TOP/datafiles/in/archive with date timestamp appended to file name. SQL*Loader log files and
bad files can reside in $XXINV_TOP/datafiles/in/log and $XXINV_TOP/datafiles/in/bad folders.
This shell script should also read the bad files generated by SQL*Loader and complete the concurrent
program in Success/Warning/Error Status. Sample shell script is given in annexure.
This shell script has following sections
Extract Apps username and password
Initialize file names (data, log, bad, archive files)
Check existence of file
Run SQL*Loader
Archive data files
Check bad files
Shell Script with Java based XML Parser:- Shell script (host) type of concurrent program should be used
to call java based/PLSQL XML parser. The directory structure, archiving, logging related rules remain the
same mentioned above. In this case we have to develop a custom Java or PLSQL based XML parser which
can parse the input xml files and load in custom staging table. PLSQL based parsers user UTL_FILE
package to perform file read operations. Building a java based XML parser is a topic in itself, which can be
considered as out of scope for this white paper.
PLSQL Package for reading Legacy outbound tables:- This can be a plsql package which can read
legacy database tables using database links and writes the inbound custom staging tables in Oracle Apps
11i instance.
Oracle Apps Custom Lookup Table:- This table should be used for storing the lookup values i.e.
mapping legacy system values with oracle values. This generally store information like Unit of Measure
etc. which can be different on Oracle and Legacy system. The table structure should have columns like
Lookup_Type, Language, Legacy System, Legacy System Value, Oracle_Value, effectivity_date, end_date
and WHO columns.
PLSQL package for data validation and mapping:- This PLSQL package should perform following
activities
Validate data in custom tables
Perform data mapping activities using lookup table
Update the custom staging table Process_Flag column with following values
1 - PENDING
3 - ERROR
7 - PROCESSED/SUCCESS
5 --VALIDATION SUCCESS
2 - PICKED FOR VALIDATION
PLSQL package to write data in Oracle Open Interface tables:- This PLSQL package should read all
valid records (Process_Flag=5) and insert data in Oracle Seeded interface table and mark Process_Flag=7
Oracle Open Interface tables:- These are the inbound interface tables provided by oracle for various
base tables in oracle applications.
Oracle Open Interface Concurrent Programs:- These are oracle supplied import programs which read
oracle open interface tables, validate the data and load oracle apps base tables.
Oracle Supplied Public APIs- Generally for online processing (when batch processing is not acceptable)
public APIs are used to load oracle base tables. In this form of interfaces, open interface tables are not
used. Public PLSQL APIs accept plsql table type of input parameters and APIs perform validation and
updates of oracle base tables.
Other important components:-
Lookups and Profiles :- Custom Profile Options can be used for following purposes
Enable/Disable Debug Mode
Store Email addresses to be sent by Monitoring Routines
Store hard coded values needed by interfaces
Store file paths in case of outbound interfaces
FND_LOOKUPS (Common Lookups) can be used for following purposes
Storing mappings if custom lookup table is not utilized
Store hard coded values
Custom Lookup Table -In case of some mapping between Oracle and Legacy systems a custom lookup
table can be designed along with a custom form which will help data entry for mappings
Custom Form to update custom staging tables :-This form is used for performing UPDATE/DELETE
operations on custom staging tables. This is used to correct any data errors in custom staging tables and
re-submit data for processing. This helps avoiding manual correction of data files. This form can be a
simple data entry screen which selects custom table from list of value and displays the data in tabular
format on a stacked canvas and allows data updates.
Monitoring:- Monitoring/Error Reporting are an important aspect of data conversion or interface
development. There should a notification system to notify users about the issue or problem. Monitoring
can be achieved in multiple ways like Alerts, Workflow, and Error Reporting UNIX Emails etc.
Alert based monitoring:- This can be developed using a custom table with columns which are generally
required for an email e.g. SUBJECT, TO_EMAIL, EMAIL_MESSAGE, PROCESS_STATUS, INSTANCE and WHO
Columns. This custom table can have an event alert (INSERT) which read new records and send out an
email and update PROCESS_STATUS column value for current record to 'SENT'

Error Reporting using UNIX Emails :- To build this functionality additional efforts are needed to populate
an error table giving more details of the problem, build simple reports on it and submit these report
concurrent programs at the end of data conversion routine/interfaces and email the output file to
respective users.
Summary
To summarize the data conversion and interface development strategies, one should always have about
components like custom staging tables, shell scripts, SQL*Loader control files, validation programs and
monitoring in the design. There can be short cuts in data conversion like instead of shell scripts using
directly SQL*Loader executable types in concurrent programs, which should definitely avoided because
these short cuts involve lot of hard coding also maintenance and reusability of components cannot be
achieved. Monitoring and error report is generally ignored considering it as overhead but it helps in long
run for debugging and maintenance.
Monitoring and error report is generally ignored considering it as overhead but it helps in long run for
debugging and maintenance
Annexure I - Sample Shell Script
# XXINV2402_ITEM_OnHand
#author :
# Date Written. 09/20/2007
# Purpose. Loading data in Staging table for Master Item

#######################################################################
######## get user id #######
#######################################################################
for i in $*
do
case $i in FCP_LOGIN=*)
#echo $i
UID_PWD=`echo $i | sed 's/FCP_LOGIN=//g'`
UID_PWD=`echo $UID_PWD | sed 's/"//g'`;;
esac
done

##########################################################################
####### Initialize the variables #######
##########################################################################
TODAY_DAT=`date +%Y%m%d%H%M`
CTL_FILE=$XXINV_TOP/bin/XXINV2402_01.ctl
DATA_FILE=$XXINV_TOP/datafiles/in/Item_OnHand.csv
BAK_FILE=$XXINV_TOP/datafiles/in/archive/Item_OnHand$TODAY_DAT.bak
LOG_FILE=$XXINV_TOP/log/Item_OnHand$TODAY_DAT.log
BAD_FILE=$XXINV_TOP/datafiles/out/archive/Item_OnHand$TODAY_DAT.bad
##########################################################################
####### Check File Existence #######
##########################################################################
fileChk()
{
# if file is not exists then exit the program
if [ -f $1 ]
then
echo $1' File Exists'
else
echo $1' File Not Exists'
exit 1
fi
}
# Check Data File is exists or not
fileChk $DATA_FILE

fileChk $CTL_FILE
tr -cd '[a-z],[A-Z],[0-9],\n,`,!,@,#,$,%,^,&,*,(,),_,-,+,=,[,{,],},|,\,/,?,<,>, ,:,;,",.'< $DATA_FILE > $DATA_FILE.01
mv $DATA_FILE.01 $DATA_FILE

sqlldr $UID_PWD control=$CTL_FILE data=$DATA_FILE log=$LOG_FILE bad=$BAD_FILE
retcode=`echo $?`
echo $retcode

if [ $retcode = 0 ]
then
# take backup of datafile to out folder if successful loaded
mv $DATA_FILE $BAK_FILE
fi
# Check Bad file exists and prompt it
if [ -f $BAD_FILE ]
then
echo 'Please Check Bad file '$BAD_FILE
fi

Annexure II - Monitoring Alert
Table Name: XXINV_2149_TRX_MONITOR
TRANSACTION_ID NUMBER,
INSTANCE VARCHAR2(100),
SUBJECT VARCHAR2(1000),
ALERT_MESSAGE VARCHAR2(4000),
EMAIL_ADDRESSES VARCHAR2(4000),
EMAIL_STATUS VARCHAR2(30),
CREATION_DATE DATE DEFAULT SYSDATE,
CREATED_BY NUMBER,
LAST_UPDATED_BY NUMBER,
LAST_UPDATED_DATE DATE DEFAULT SYSDATE,
REQUEST_ID NUMBER,
PROGRAM_APPLICATION_ID NUMBER,
PROGRAM_ID NUMBER,
PROGRAM_UPDATE_DATE DATE DEFAULT SYSDATE,
ALERT_TYPE VARCHAR2(30),
PROGRAM_NAME VARCHAR2(1000)
Event Alert on this table to send out email and update EMAIL_STATUS from PENDING to
SENT