D78191GC10 SG

Oracle Data Integrator 11g:
Advanced Integration and

Development

Student Guide
D78191GC10
Edition 1.0
February 2013
D80589
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Copyright 2013, Oracle and/or its affiliates. All rights reserved.
Disclaimer

This document contains proprietary information and is protected by copyright and
other intellectual property laws. You may copy and print this document solely for your
own use in an Oracle training course. The document may not be modified or altered in
any way. Except where your use constitutes "fair use" under copyright law, you may
not use, share, download, upload, copy, print, display, perform, reproduce, publish,
license, post, transmit, or distribute this document in whole or in part without the
express authorization of Oracle.

The information contained in this document is subject to change without notice. If you
find any problems in the document, please report them in writing to: Oracle University,
500 Oracle Parkway, Redwood Shores, California 94065 USA. This document is not
warranted to be error-free.

Restricted Rights Notice

If this documentation is delivered to the United States Government or anyone using
the documentation on behalf of the United States Government, the following notice is
applicable:

U.S. GOVERNMENT RIGHTS
The U.S. Governments rights to use, modify, reproduce, release, perform, display, or
disclose these training materials are restricted by the terms of the applicable Oracle
license agreement and/or the applicable U.S. Government contract.

Trademark Notice

Oracle and J ava are registered trademarks of Oracle and/or its affiliates. Other names
may be trademarks of their respective owners.

Author
Viktor Tchemodanov
Technical Contributors
and Reviewers
Denis Gray
Alex Kotopoulis
Julien Testut
Christophe Dupupet
Rebecca Sly
Gerry Jurrens
Sophia Chen
Vishal Parashar
Richard Green
Editors
Raj Kumar
Malavika Jinka
Aju Kumar
Graphic Designers
Seema Bopaiah
Maheshwari Krishnamurthy
Publishers
Michael Sebastian
Srividya Rameshkumar
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.

iii
Contents

1 Introduction
Lesson Objectives 1-2
Course Objectives 1-3
Target Audience 1-4
Class Introductions 1-5
Agenda of Lessons 1-6
Course Environment 1-7
Course Materials 1-8
Course Practices 1-9
How can I Learn More? 1-10
Related Training 1-11
Overview of Oracle Data Integrator 1-12
Features of Oracle Data Integrator 1-13
Conventional Integration Process: ETL 1-15
EL-T 1-16
ODI Architecture 1-18
Summary 1-19
Practice 1-1: Exploring Your Environment 1-20

2 Overview of ODI Knowledge Modules
Objectives 2-2
ODI Knowledge Modules: Overview 2-3
Knowledge Modules: Overview 2-4
Global Knowledge Modules 2-5
Knowledge Modules in Action 2-6
Code Generation 2-7
Types of Knowledge Modules: Overview 2-8
Working with Knowledge Modules 2-9
How to Select Knowledge Modules? 2-10
Creating New Knowledge Modules 2-11
Developing Knowledge Modules 2-12
Editing KMs 2-13
Description of KM Steps 2-15
Details of the Steps are Generic 2-16
Setting KM Options 2-17
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.

iv
Replacing Existing KMs 2-18
Developing Knowledge Modules 2-20
Developing Your Own KM: General Guidelines 2-21
Understanding the Substitution API 2-22
Using Substitution Methods 2-23
Understanding Various ODI Substitution Methods 2-24
Substitution Methods: Basic Examples 2-25
Using Substitution Methods in Actions 2-26
Working with Object Names 2-28
Working with Lists of Tables, Columns, and Expressions 2-30
Example: Using getTargetColList to Create a Table 2-33
Example: Using getColList in an Insert Values Statement 2-34
Example: Using getSrcTableList 2-35
Generating the Source Select Statement 2-36
Generating the Source Select Statement: Example 2-37
Working with Data Sets 2-38
Obtaining Other Information with the API 2-39
Quiz 2-40
Summary 2-41
Practice 2-1 Overview: Developing and Enhancing an Integration Knowledge
Module 2-42

3 Developing Knowledge Modules
Objectives 3-2
Developing Knowledge Modules: Best Practices 3-3
Developing KMs: Target a Particular Stage of your Integration Process 3-4
Developing KMs: What to Avoid? 3-6
Developing KMs: Other Recommendations 3-7
Using Java in Knowledge Modules 3-8
Using Code Generation Tags: 3-10
Using Code Generation Tags: Examples 3-11
Understanding Code Generation Techniques 3-12
Using Substitution Methods: Code Examples 3-14
Using ODI Substitution Methods: getJoin() 3-15
Using ODI Substitution Methods: getFilter() 3-16
Using ODI Substitution Methods: getPK() 3-17
Using ODI Substitution Methods in Journalizing Knowledge Modules:
Examples 3-18
Using ODI Substitution Methods in Reverse-Engineering KM: getModel() 3-20
Troubleshooting Knowledge Modules 3-22
Troubleshooting KMs 3-23
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.

v
Quiz 3-27
Summary 3-28
Practice 3-1 Overview: Creating a New Knowledge Module for
Data Generation 3-29

4 Designing ODI Integration Interfaces
Objectives 4-2
Overview of Integration Process 4-3
Conventional Integration Process: ETL 4-4
E-LT 4-5
Integration Process: Overview 4-7
Typical Integration Process 4-9
Overview of Integration Interfaces 4-10
Integration Interfaces: Overview 4-11
Basic Integration Process: The Sequence of Operations 4-13
The Staging Area 4-14
Placing the Staging Area 4-15
Designing Integration Interfaces: Best Practices 4-16
Designing Integration Interfaces: E-LT- and ETL-Style Interfaces 4-17
Designing an ETL-Style Interface 4-18
Designing an ETL-Style Interface: Using Multiconnection IKM 4-19
Designing an ETL-Style Interface: Using an LKM and a mono-connection IKM 4-22
Designing an ETL-Style Interface: Limitations of Using an LKM and a
mono-connection IKM 4-23
Using an LKM and a Mono-Connection IKM: Steps 4-24
Maintaining Integrity of Data 4-25
Enforcing ODI Data Quality 4-26
Ways to Handle Erroneous data 4-27
Error Recycling 4-28
Building a Data Quality Framework 4-29
Quiz 4-30
Summary 4-31
Practice 4-1: Creating an ODI Interface for XML to Database Transformation with
ODI Constraint and Error Recycling 4-32

5 Designing Advanced Integration Interfaces
Objectives 5-2
ODI Interfaces: Advanced Functionality 5-3
Creating Lookups 5-4
Lookup Wizard 5-6
Using Set-Based Operators with Integration Interfaces 5-7
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.

vi
Example: Flow with Multiple Data Sets 5-9
Defining a Data Set 5-10
Using Set-Based Operators: Guidelines 5-11
Using Partitioning with ODI Interfaces 5-12
Partitioning: Definition in Data Store After Reverse Engineering 5-13
Using Partitioning in an Interface 5-14
Using Temporary Interfaces 5-15
Using Temporary Interfaces: Example 5-16
Derived Select for Temporary Interfaces 5-17
Derived Select for Temporary Interfaces: Limitations 5-18
Quiz 5-19
Summary 5-20
Practice 5-1: Implementing ODI Integration with Temporary Interfaces 5-21

6 Using Variables in ODI
Objectives 6-2
Using Variables: Overview 6-3
Using Variables in ODI: Overview 6-4
Variable Scope 6-5
Referring to a Variable 6-6
Referring to a Variable: Using : Instead of # 6-8
Using Variables in ODI Objects 6-9
Using Variables in Packages 6-10
Using Variables in Packages: Example 6-11
Using Variables in Interfaces 6-12
Using Variables in Interfaces: Examples 6-14
Using Variables in Object Properties 6-15
Using Variables in Procedures 6-16
Using Variables Within Variables 6-18
Using Variables in the Resource Name of a Data Store 6-19
Using a Variable as a Startup Parameter: Example 6-21
Using ODI Variables in Topology 6-24
Using ODI Variables in Topology: A Server URL 6-25
Using Variables in a Server URL: Example 6-26
Tracking Variables 6-28
Tracking Variables: Notes 6-32
Quiz 6-34
Summary 6-35
Practice 6-1 Overview: Using Variables in ODI Package 6-36
Practice 6-2 Overview: Using an ODI Variable as a Startup Parameter 6-37
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.

vii
7 Accelerating Development in ODI with Groovy
Objectives 7-2
Interacting Programmatically with ODI 7-3
Overview of ODI SDK 7-4
SDK-Supported ODI Operations 7-5
ODI Operations Not Supported by SDK 7-6
Combining Different APIs 7-7
Using ODI Groovy Editor 7-8
Introduction to Groovy 7-9
Introduction to the Groovy Editor 7-10
Executing Script with Groovy Editor 7-11
Performing SDK Tasks Using Java with Groovy Editor 7-12
Example of Performing an SDK Task Using Java with Groovy Editor 7-13
Performing Advanced Actions with Groovy 7-14
Using Custom Libraries 7-15
Defining Additional Groovy Execution Classpath 7-16
Read Input with the odiInputStream Variable 7-17
Automating Development Tasks: Example 1 7-18
Quiz 7-23
Summary 7-24
Practice 7-1 Overview: Automating ODI Tasks with Groovy 7-25

8 Working with Complex Files in ODI
Objectives 8-2
Complex Files Concept: Overview 8-3
Types of Files in ODI 8-4
Complex Files Concept 8-5
Knowledge Modules for Complex Files 8-7
Requirements for Working with Complex Files 8-8
Complex Files: Configuring the Topology 8-9
Setting Up the Topology 8-10
JDBC URL Properties 8-11
JDBC URL: Example 8-12
Complex File Data Server Definition: Example 8-13
Definition of Physical and Logical Schema: Example 8-15
Complex Files in Integration Project 8-16
Setting Up an Integration Project and Creating a Complex File Model 8-17
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.

viii
Setting Up an Integration project and Creating a Complex File Model:
Example 8-18
Designing an Interface with Complex File Model 8-19
Creating an nXSD File 8-20
Example of nXSD Schema 8-21
Using Native Format Builder Wizard 8-24
Quiz 8-27
Summary 8-28
Practice 8-1 Overview: Configuring ODI Topology and ODI Model with
Complex Files 8-29

9 Integration of ODI in Enterprise Environment and SOA
Objectives 9-2
ODI Integration with Java EE 9-3
Understanding ODI Java EE Agent and Standalone Agent 9-4
Java EE Agent Deployment Features 9-5
Integration of ODI with Fusion Middleware Control 9-6
Using ODI Console 9-7
Using ODI Console: Example 9-8
Overview of Using Web Services with ODI 9-9
Types of Web Services 9-10
Overview of Data Services 9-11
Generation of Data Services 9-12
Overview of Public Web Services 9-13
Using Public Web Service OdiInvoke 9-14
Installing Public Web Services 9-15
Invoking Web Services 9-16
OdiInvokeWebService Tool 9-17
Integration of ODI with SOA 9-20
ODI with SOA Integration Scenarios 9-21
Integration of ODI Within SOA in Action 9-22
Example 1: Using Data Services 9-23
Example 2: Exposing ODI Process as a Web Service 9-24
Creating the BPEL process 9-25
Creating ODI Interface and ODI Package 9-26
Creating ODI Scenario and Editing the BPEL Process 9-28
Deploying the BPEL Process to the Application Server and Invoking from Enterprise
Manager 9-29
Example 3: Calling a Web Service from Within ODI for Processing ODI Errors with
BPEL Human Workflow 9-30
Processing ODI Errors with BPEL Human Workflow 9-31
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.

ix
Quiz 9-38
Summary 9-39
Practice 9-1 Overview: Integrating ODI in the Enterprise Environment 9-40
Practice 9-2 Overview: Exposing an ODI Scenario as a Web Service 9-41
Practice 9-3 Overview: Integrating ODI with a BPEL Process Within SOA 9-42

10 Enhancing ODI Security
Objectives 10-2
Oracle Data Integrator (ODI) Security: Overview 10-3
Security Concepts: Overview 10-4
Authorizing by Object Instance 10-6
ODI Security Navigator: Overview 10-7
Using Generic and Nongeneric Profiles 10-9
Built-in Profiles 10-10
Implementing a Strongly Secured Approach 10-12
Security Policy Approach 10-13
Defining Security Policies: Using Generic Profiles 10-14
Defining Security Policies: Using Nongeneric Profiles 10-15
Using Nongeneric Profiles: Granting an Authorization by Object Instance 10-16
Unused Authorizations 10-17
Cleaning Up Unused Authorizations 10-18
ODI Security Integration: Overview 10-19
Implementing External Authentication (OPSS) 10-21
Configuring External Authentication 10-23
Using External LDAP Server 10-24
Oracle Internet Directory (OID): Architecture 10-25
Configuring ODI External Authentication 10-26
Configuring External Authentication with OID 10-27
1. Creating a New OID User with Directory Services Manager 10-28
2. Editing the jps-config.xml File to Point to OID LDAP Server 10-29
3. Running the Script (odi_credtool.cmd) to Set Up the Credentials for the Identity
Store 10-30
4. Creating New ODI Master Repository Referencing a User in the External LDAP
Server 10-31
5. Creating a New ODI Connection Referencing a User in the External OID LDAP
Server 10-33
Switching the Master Repository Authentication Mode 10-34
Using Switch Authentication Mode Wizard 10-35
Reactivating Users After Switching to Internal Authentication 10-38
Re-enabling Users After Switching to External Authentication 10-39
External Password Storage 10-40
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.

x
Setting Up External Password Storage 10-41
Implementing External Password Storage 10-42
Quiz 10-44
Summary 10-45
Practice 10-1: Implementing ODI External User Authentication 10-46

11 Choosing Integration Strategies: Best Practices
Objectives 11-2
Defining an ODI Integration Strategy 11-3
Integration Strategies 11-4
Strategies with Staging Area on the Target 11-5
Strategies with Staging Area on the Target: Append 11-7
Strategies with Staging Area on the Target: Control Append 11-8
Strategies with Staging Area on the Target: Incremental Update 11-10
Strategies with Staging Area on the Target: Incremental Update:
Optimization 11-12
Working with Slowly Changing Dimensions 11-13
Three Types of Slowly Changing Dimensions 11-14
Working with Type 2 Slowly Changing Dimensions 11-15
Type 2 SCDs: Example 11-16
Implementing Type 2 SCDs 11-17
Implementing Type 2 SCDs: Steps 11-18
Strategies with Staging Area Different from the Target 11-20
Using ODI for Bulk Processing 11-24
Using ODI for Bulk Processing: Design Patterns 11-26
Using Cross-Reference (XREF) Table: Overview 11-28
Using the XREF Knowledge Module 11-29
Real-Time Data Integration: Using ODI with Oracle GoldenGate 11-31
Loading Data Patterns 11-32
Oracle GoldenGate: Overview 11-33
Oracle GoldenGate Solutions: Overview 11-34
Oracle GoldenGate Concepts 11-35
Using ODI with Oracle GoldenGate 11-36
Using ODI and GoldenGate Together 11-37
Initialize CDC Process and Perform Change Operations with
Oracle GoldenGate 11-38
Working on ODI Projects: Best Practices 11-39
1. Use Context-Independent Design 11-40
2. Use Procedures Only when Needed 11-41
3. Always Enforce Data Quality 11-42
4. Handle Error Cases in Packages 11-43
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.

xi
5. Choose Right Knowledge Module 11-44
6. Other Best Practices 11-45
Quiz 11-46
Summary 11-47
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.

O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Introduction
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
This lesson provides a general overview of the course objectives and agenda of lessons.
Oracle Data Integrator 11g: Advanced Integration and Development 1 - 2
Lesson Objectives
After completing this lesson, you should be able to:
Describe the course objectives
Describe agenda of lessons
Identify the target audience for the course and the
suggested prerequisites
Describe the course environment
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Course Objectives
After completing this course, you should be able to:
Develop and enhance Knowledge Modules
Design advanced integration interfaces
Use complex files
Use ODI Groovy editor
Use ODI within SOA
Enhance ODI security
Use best practices for implementing integration strategies
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Target Audience
This course is directed at:
Business Analysts
Process Architects
IT Developers
Data Administrators
Professional Services Consultants
Oracle Solution Architects
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Class Introductions
Please introduce yourself:
Name, Title/Position
Company
Experience using Oracle Data Integrator, Fusion
Middleware Products
Reasons for attending
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Agenda of Lessons
Day 1:
Lesson 1: Introduction
Lesson 2: Overview of ODI Knowledge Modules
Lesson 3: Customizing Knowledge Modules
Lesson 4: Designing ODI Integration Interfaces
Day 2:
Lesson 5: Designing Advanced Integration Interfaces
Lesson 6: Using Variables in ODI
Lesson 7: Accelerating Development in ODI with Groovy
Lesson 8: Working with Complex Files in ODI
Day 3:
Lesson 9: Integration of ODI in Enterprise Environment and
SOA
Lesson 10: Enhancing ODI Security
Lesson 11: Choosing Integration Strategies: Best Practices
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
In this course, the following products are used for lessons and practices:
Oracle Database 11g, WebLogic Server 11g, Oracle Data Integrator 11g, Oracle SOA Suite,
Oracle Internet Directory, and supporting products. These layered products together comprise the
Oracle Data Integrator runtime environment.
Course Environment
Browser
Oracle Data Integrator
11g
FUSION MIDDLEWARE
WEBLOGIC SERVER
11
g
Oracle SOA Suite
Oracle Internet
Directory
JDeveloper
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The Student Guide contains the lecture slides and notes. The Activity Guide contains the practices
for the course.
Course Materials
Student Guide
Activity Guide
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Course Practices
Core topics will be reinforced with hands-on practices.
Optional topics and practices will be covered at the
instructors discretion and if time permits.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
How can I Learn More?
Topic Website
Education and Training
http://education.oracle.com
Product Documentation
http://www.oracle.com/technology/documentation
Product Downloads
http://www.oracle.com/technology/software
Product Articles
http://www.oracle.com/technology/pub/articles
Product Support
http://www.oracle.com/support
Product Forums
http://forums.oracle.com
Product Tutorials
http://www.oracle.com/oll
Oracle Data Integrator
on OTN
http://www.oracle.com/technetwork/middleware/d
ata-integrator/overview/index.html
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Your instructor can provide additional information regarding the availability and contents of the
courses listed in the slide.
Related Training
Course
Oracle Data Integrator 11g: Integration and Administration
Oracle WebLogic Server: Administration Essentials
Oracle SOA Suite 11g: Essential Concepts
Oracle Directory Services 11g: Essential Concepts
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Overview of Oracle Data Integrator
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
ODIs ELT architecture leverages disparate RDBMS engines to process and transform data.
This approach optimizes performance and scalability, and lowers overall solution costs.
ODI turns the promise of active integration into reality by providing all the key components
that are required to enable real-time data warehousing and operational data hubs. ODI
combines three styles of data integration: data-based, event-based, and service-based. ODI
unifies silos of integration by transforming large volumes of data in batch mode, by
processing events in real time through its advanced Changed Data Capture, and by
providing data services to the Oracle SOA Suite.
Oracle Data Integrator shortens implementation times with its declarative design approach.
Designers specify what they want to accomplish with their data, and then the tool generates
the details of how to perform the task. With ODI, the business user or the developer specifies
the rules to apply to the integration processes. The tool automatically generates data flows
and administers correct instructions for the various source and target systems. With
declarative design, the number and complexity of steps is greatly reduced, which in turn
shortens implementation times. Automatic code generation reduces the learning curve for
integration developers and streamlines non-IT professionals to the definition of integration
processes and data formats.
Features of Oracle Data Integrator
ELT architecture provides high performance.
ELT faster than ETL
Active integration enables real-time data warehousing and
operational data hubs.
Changed data capture technology for real-time data
warehousing
Data services provided to the Oracle SOA Suite
Declarative design improves developer productivity.
Business users specify what they want; ODI generates the
flows and code.
Knowledge modules provide flexibility and extensibility.
Predefined, reusable code templates with built-in
connectivity to all major databases
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Knowledge modules are at the core of the ODI architecture. They make all ODI processes
modular, flexible, and extensible. Knowledge modules implement the actual data flows and
define the templates for generating code across the multiple systems involved in each process.
ODI provides a comprehensive library of knowledge modules, which can be tailored to
implement existing best practices (for example, for highest performance, for adhering to
corporate standards, or for specific vertical knowhow). By helping companies capture and
reuse technical expertise and best practices, ODIs knowledge module framework reduces the
cost of ownership. It also enables metadata-driven extensibility of product functionality to meet
the most demanding data integration challenges.
ODI streamlines the high-performance movement and transformation of data between
heterogeneous systems in batch, real-time, synchronous, and asynchronous modes. It dramatically
enhances user productivity with an innovative, modularized design approach and built-in connectivity
to all major databases, data warehouse appliances, analytic applications, and SOA suites.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
This integration process is also known as an extract, transform, and load (ETL) process.
The first part of an ETL process involves extracting data from the source systems. Most data
warehousing projects consolidate data from different source systems.
The transform stage applies a series of rules or functions to the data extracted from the source to
derive the data for loading into the target. Some data sources will require very little or even no
manipulation of data. In other cases, transformations (such as filtering, joining, sorting, and so on)
may be required to meet the business and technical needs of the target database.
The load phase loads the data into the target, usually the data warehouse.
Note: You can add to this process the checks that ensure the quality of data flow, as shown in the
slide.
Conventional Integration Process: ETL
Integration process
Extract - Transform (check) - Load
Source
ORDERS
LINES
Target
SALES
Errors
Transform
A machine
A machine
A machine
Extract Transform Load
CORRECTIONS
File
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Data is one of the most important assets of any company, and data integration constitutes the
backbone of any enterprises IT systems. Choosing the technology for data integration is critical
for productivity and responsiveness of business divisions within an enterprise.
ELT stands for extract, load, and transform. It includes the processes that enable companies to
move data from multiple sources, reformat and cleanse the data, and load it into another
database, or a data warehouse for analysis, to support a business process.
ODI provides a strong and reliable integration platform for IT infrastructure. Built on the next-
generation architecture of extract, load, and transform (ELT), ODI delivers superior performance
and scalability connecting heterogeneous systems at a lower cost than traditional, proprietary ETL
products. Unlike the conventional extract, transform, and load (ETL) design, with ODI, ELT
architecture extracts data from sources, loads it into a target, and transforms it by using the
database power according to business rules. The tool automatically generates data flows,
manages their complexity, and administers correct instructions for the various source and target
systems.
EL-T
1. Extract: Extracting data from various sources
2. Load: Loading the data into the destination target
3. Transform: Transforming the data according to a set of
business rules
Extract
Transform
Conventional ETL architecture
Load Load Extract
Transform
Transform
Next-generation ELT architecture
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Extract
The first step in the ELT process is extracting data from various sources. Each of the source
systems may store its data in a completely different format. The sources are usually flat files or
RDBMS, but almost any data storage can be used as a source for an ELT process.
Load
This step involves loading the data into the destination target, which might be a database or data
warehouse.
Transform
After the data has been extracted and loaded, the next step is to transform the data according to a
set of business rules. The data transformation may involve various operations including, but not
limited to filtering data, sorting data, aggregating data, joining data, cleaning data, generating
calculated data based on existing values, and validating data.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The repository forms the central component of the ODI architecture. This stores configuration
information about the IT infrastructure; the metadata for all applications, projects, scenarios, and
execution logs. Repositories can be installed in an online transaction processing (OLTP) relational
database. The repository also contains information about the ODI infrastructure, defined by the
administrators. The two types of ODI repositories are Master and Work Repositories.
At design time, developers work in a repository to define metadata and business rules. The
resulting processing jobs are executed by the agent, which orchestrates the execution by
leveraging existing systems. The agent connects to available servers and asks them to execute
the code. It then stores all return codes and messages in the repository. The agent also stores
statistics, such as the number of records processed, and the elapsed time. Several repositories
can coexist in an IT infrastructure. The graphic in this slide shows two repositories: one for the
development environment and the other for the production environment. Developers release their
projects in the form of scenarios that are sent to production.
In production, these scenarios are scheduled and executed on a Scheduler Agent that also stores
all its information in the repository. Operators have access to this information and can monitor the
integration processes in real time.
Business users, as well as developers, administrators, and operators, can gain web-based read
access to the repository by using the ODI Console.
ODI Architecture
Desktop
Repositories
ODI Studio
Operator
Designer
Topology
Security
ODI Master
Repository
ODI Work
Repository
Sources and Targets
Legacy Applications
ERP/CRM/PLM/SCM
Files / XML DBMS DW / BI / EPM
JVM
Java EE
Application
ODI SDK
WebLogic 11g / Application Server
Data Sources Connection Pool
Web Service Container
Public
WS
Data
Services
FMW Console
ODI Plug-in
Servlet Container
ODI
Console
Java EE
Application
ODI SDK
Runtime
WS
Java EE
Agent
JVM
Run-time
WS
Standalone
Agent
Open Web Services for
decoupled invocation
of any ODI jobs
Heterogeneous
access
SDK APIs for Java
generation of any ODI
system metadata
Metadata is an
open relational
model with built-
in Flex Fields
Open Tools for adding
OS functionality to
any ELT/ETL job
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Summary
In this lesson, you should have learned to:
Describe the course objectives
Describe agenda of lessons
Identify the target audience for the course and the
suggested prerequisites
Describe the course environment
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Practice 1-1: Exploring Your Environment
In this practice, you will explore the technical environment for
your practices.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Overview of ODI Knowledge Modules
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Objectives
Describe the guidelines for working with Knowledge
Modules (KMs)
Use ODI substitution API for developing KMs
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
ODI Knowledge Modules: Overview
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Technically, this term describes a template containing the code necessary to implement a
particular data integration task. These tasks include loading data, checking it for errors, or setting
up triggers necessary to implement journalization. However, all Knowledge Modules basically
work the same way: ODI uses them to generate code, which is then executed by a technology at
run time.
Note: Knowledge Modules are independent of the structure of the source and target data stores.
The same KM can be used, no matter which source table you have, or how many source tables
you have. Likewise, all target tables can use the same Knowledge Module.
Knowledge Modules: Overview
A Knowledge Module is a code template containing the
sequence of commands necessary to implement a data
integration task.
There are different predefined Knowledge Modules for
loading, integration, checking, reverse engineering,
journalizing, and deploying data services.
All Knowledge Modules work by generating code to be
executed at run time.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
ODI 11.1.1.6 introduces Global Knowledge Modules (KMs) allowing specific KMs to be shared
across multiple projects. In previous versions of ODI, Knowledge Modules were always specific to
a project and could only be used within the project into which they were imported. Global KMs are
listed in the Designer Navigator in the Global Objects accordion.
Global Knowledge Modules
KMs can be specified as Global, allowing them to be shared
across multiple projects.
Project KMs
Global KMs
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Knowledge Modules define a way of implementing business rules on a given technology. They
connect the abstract logic of your business rules to the concrete reality of your data servers.
If your data servers change, you just select a different Knowledge Module to match. Your business
rules remain unchanged. If your business rules change, you do not have to modify any codeyou
just update the SQL expressions that define them in ODI.
In your example, you have Loading Knowledge Modules that describe how to move data from the
Sybase source server to the DB2/UDB staging area. Integration Knowledge Modules perform the
work of creating temporary tables and moving data onto the Oracle target server. Lastly, Check
Knowledge Modules implement constraint checking and isolating errors in a separate table.
Note: Knowledge Modules are generic because they enable data flows to be generated
regardless of the transformation rules. And they are highly specific because the code they
generate and the integration strategy they implement are finely tuned for a given technology.
Knowledge Modules in Action
Knowledge Modules implement the required operations.
DB2/UDB
Staging Area
Target (Oracle)
SALES
ERRORS
Source (Sybase)
ORDERS
LINES
CORRECTIONS
File
TEMP_1
TEMP_2
Join/Transform
TEMP_
SALES
Extract/Transform
Transform and
integrate
Check constraints/
Isolate errors
Extract/Join/
Transform
CKM
IKM
LKM
LKM
IKM
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Business rules and their implementation on a server are being carefully kept separate. Now you
will see how ODI combines the two at run time.
For example, you have a mapping that the net income of an employee is the sum of the income
components multiplied by their coefficients. This is an expression in SQL, but there is no code to
perform this integration.
You have a Knowledge Module that can wipe a destination table and fill it with source data.
However, it knows nothing about your business rules.
Then you have all the other metadata that you defined in ODI: the Topology of your servers, the
models that exist on them, the technologies used by each server, and so on.
When you put these three things together, ODI generates a code to carry out the integration. This
code is specific to a technology, a layout, and a set of business rules. However, if any of these
three things changes, all you need to do is regenerate the code.
At run time, the ODI Agent orchestrates the running of the generated code. Based on execution
locations, various parts of the generated code will be executed either on the Source, Staging or
Target.
Code Generation
Generated code
Truncate Table SCOTT.EMP
Insert into SCOTT.EMP
Select Round(Sum(Sal*Coeff))
Begin If .. Then Else
ftp ...
Implemented business
rules
Inc_net = Round
(Sum(Inc * coeff))
Knowledge module
Truncate Table <%=odiRef.getTable( )%>
Insert into <%=odiRef.getColList()%>
Select <%=odiRef.getFrom()%>
Begin If .. Then Else
ftp ...
Design
time
ODI Agent orchestrates the
running of the generated code
Run
time
Truncate Table SCOTT.EMP
Insert into SCOTT.EMP
Other metadata
- Topology of your
servers
- Models in servers
- Technology, and
so on
Targets
Sources
plus
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The first group of Knowledge Modules is critical for doing any work with interfaces.
Loading Knowledge Modules (LKMs) extract data from the source of interfaces. So, if your
data is stored in flat files, you will need the File to SQL LKM.
Integration Knowledge Modules (IKMs) implement a particular strategy for loading the target
of an interface. Thus, to load an Oracle table while taking into account the slowly changing
dimensions properties, a particular IKM can be used.
Check Knowledge Modules (CKMs) are selected in Interfaces and constraints can be
individually enforced by ODI at the interface level.
The second group is used for setting up, checking, and configuring models.
Check Knowledge Modules (CKMs) enforce constraints defined on the target data store.
CKMs are used by models to perform static checks outside of interfaces.
Reverse Engineering Knowledge Modules (RKMs) are needed only to perform customized
reverse engineering. They are used to recover the structure of a data model and are used
when standard reverse engineering cannot be performed.
Journalizing Knowledge Modules (JKMs) are used to set up Changed Data Capture. This
makes interfaces react only to changes in data and can vastly reduce the amount of data
that needs to be transferred.
Service Knowledge Module (SKM) is the code template for generating data services.
Types of Knowledge Modules: Overview
There are six types of Knowledge Modules you may import:
I
n
t
e
r
f
a
c
e
s
M
o
d
e
l
s
KM Type Description
LKM Loading
Assembles data from source data stores to the Staging
Area
IKM Integration
Uses a given strategy to populate the target data store from
the Staging Area
CKM Check Checks data in a data store for errors statically or during an
integration process
RKM Reverse
engineering
Retrieves the structure of a data model from a database. It
is needed only for customized reverse engineering.
JKM Journalizing Sets up a system for Changed Data Capture to reduce the
amount of data that needs to be processed
SKM Data
Services
Deploys data services that provide access to data in data
stores
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Working with Knowledge Modules
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
You now know what the various types of Knowledge Modules are. But, how do you know which
ones need to be imported into your specific project?
The most important thing to remember is to import all Knowledge Modules that may be used in a
project. Each time you create a project, you must import the Knowledge Modules that will be used
by that project.
The basic strategy for importing Knowledge Modules is as follows:
First, import the most basic SQL Knowledge Modules. These work on almost every database
management system (DBMS) with acceptable performance.
Then, consider importing more specific Knowledge Modules for the particular technologies
involved in your project. If you are transferring data to an Oracle server, consider adding the
Oracle-based IKMs.
Technology-specific Knowledge Modules can take advantage of certain characteristics or the
special tools provided by the technologies. However, it is often best to begin with the most
generic Knowledge Modules to start your interface, then choose specific Knowledge
Modules later to increase performance. Refer to:
Oracle Fusion Middleware Knowledge Module Developer's Guide for Oracle Data
Integrator 11g Release 1 (11.1.1)
Oracle Fusion Middleware Connectivity and Knowledge Modules Guide for Oracle Data
Integrator 11g Release 1 (11.1.1)
How to Select Knowledge Modules?
Which Knowledge Modules would you need?
Start with basic SQL Knowledge Modules.
Add technology-specific Knowledge Modules as needed.
For your projects, you will need LKMs, IKMs, and CKMs
only.
You need to become familiar with the library of available
Knowledge Modules, and what they can do for you.
Examine the following ODI documentation:
Knowledge Modules Developers Guide
Connectivity and Knowledge Modules Guide
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
In real practice, there are few cases when you create a Knowledge Module from the beginning.
To create a new KM:
1. Add new Knowledge Module of appropriate type.
2. Specify the name and add the details to create the functionality of your KM based on your
project needs
Creating New Knowledge Modules
In real practice, there are few cases when you create a
Knowledge Module from the beginning. To create a new KM:
1. Add new Knowledge Module of appropriate type.
2. Specify the name and add the details to create the
functionality of your KM based on your project needs.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
In most cases, you dont create a Knowledge Module from the beginning but rather perform
developing a KM starting from some existing Knowledge Module. To develop a Knowledge
Module, import a KM of appropriate type, and then modify it by adding a new functionality
according to your project requirements.
Developing Knowledge Modules
In most cases, you do not create a Knowledge Module from the
beginning but rather develop a KM starting from some existing
KM. To develop a Knowledge Module:
1. Import a Knowledge Module of appropriate type.
2. Modify imported KM to add new functionality based on
your project requirements.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Use the Knowledge Module Editor to create your customized Knowledge Modules. In this
example, the third line is highlighted, Create work table. If you double-click this line, a detailed
editor opens, as shown in the next slide.
Editing KMs
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The KM Editor has a section to edit general information about an item within the KM, such as this
Create work table item in the LKM SQL to Oracle Knowledge Module.
The KM Editor also has a section to edit the KMs options.
Knowledge Modules are used later in this course. For additional detailed information about KMs,
see the ODI Knowledge Modules Reference Guide.
Editing KMs
Options
editing
General
editing
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
After the execution, the step names will then be viewable in ODI Operator.
Description of KM Steps
A knowledge
module is made
of steps.
Each step has
a name and a
template for the
code to be
generated.
Steps are listed on the
Details tab.
After the execution,
ODI lists the same
step names in
Operator.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Each step has a command on source and command on target. In the Command window, you can
view and modify each command manually.
Details of the steps are generic:
The source and target tables are not known, only the technologies are known.
Substitution methods are the placeholders for the table names and column names
Parameters of the substitution methods: Tables or columns
Details of the Steps are Generic
Source and Target
are not known. Only
Technologies are
known.
Substitution methods are
the placeholders for
the table names
and column names.
Parameters of the
substitution
methods: Tables
or columns
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Each option can be On or Off and is defined on the Options tab of each step of the KM.
Setting KM Options
KMs have options
that will let users:
Turn options on
or off
Specify or modify
values used by
the KM
Options are defined
in the projects tree,
under the KM.
Options are used in
the KM code with the
substitution method
<%=odiRef.getOption
(OptionName)%>.
KM options are defined on the
Options tab of each KM step.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Normally, if you import a Knowledge Module with the same name as an existing Knowledge
Module, you end up with two copies in your project. This can be useful if you want to make
changes to the Knowledge Module without breaking your existing interfaces.
However, another mode of importing is available: the import replace mode. In this mode, the
existing interface is replaced with the version imported from the disk. All interfaces that used the
old Knowledge Module are automatically updated to use the new version. Also, values that are set
for the Knowledge Module options in these interfaces are transferred to the new version. However,
any existing scenarios are not regenerated. If you want to incorporate changes to the Knowledge
Module into your scenarios, you must regenerate them.
There are various reasons why you would want to replace a Knowledge Module. The most
common reason is that a newer version of the Knowledge Module is released, perhaps by
someone on your team. You import the Knowledge Module again and all your existing interfaces
will still work.
Similarly, you may have made some undesirable changes to your Knowledge Module. You can
quickly undo these changes by reimporting from a saved version.
Note: Any interface that uses the replaced KM also will be impacted.
Replacing Existing KMs
Import replace mode
Updates all interfaces that use the KM
Preserves existing options
Does not update generated scenarios
Why replace an existing KM?
Because a newer version is released
To undo any undesirable changes made
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
To import a Knowledge Module in replacement mode, perform the following:
1. Right-click the name of the Knowledge Module that is to be replaced.
2. Select Import Replace from the context menu.
3. Enter the name of the XML file that contains the new version of the Knowledge Module. You
can use the browse button to select a file. Note that you can replace only one Knowledge
Module at a time.
4. Consider regenerating any scenarios that use the Knowledge Module to take the changes
into account. This includes both the interfaces and packages. Regenerating scenarios is
discussed later in the course.
Note: You should regenerate any scenarios that use the KM.
Replacing Existing KMs
1. Right-click the KM.
2. Select Import Replace.
3. Select the replacement file.
Browse for the file
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
One of the main guidelines when developing your own KM is to avoid starting from the beginning.
ODI provides more than 100 KMs. Take a look at these existing KMs, even if they are not written
for your technology. The more examples you have, the faster you develop your own code. You
can, for example, duplicate an existing KM and start enhancing it by changing its technology, or
copying lines of code from another KM.
To speed up development, duplicate existing steps and modify them. This prevents typos in the
syntax of the odiRef methods.
When developing your own KM, remember that it is targeted to a particular stage of the integration
process. As a reminder:
LKMs are designed to load remote source data sets to the staging area (into C$ tables).
IKMs apply the source flow from the staging area to the target. They start from the C$
tables, may transform and join them into a single I$ table, may call a CKM to perform data
quality checks on this I$ table, and finally write the flow data to the target.
CKMs check data quality in a data store or a flow table (I$) against data quality rules
expressed as constraints. The rejected records are stored in the error table (E$).
Developing Your Own KM: General Guidelines
Very few KMs are ever created from scratch. They usually
are extensions or modifications of existing KMs.
Duplicate existing steps and modify them. This prevents
typos in the syntax of the odiRef methods.
All interfaces using the KM inherit the new behavior.
Remember to make a copy of the KM if you do not want to
alter existing interfaces. Modify the copy, not the original.
Modifying a KM that is already used is a very efficient way
to implement modifications in the data flow and affect all
the existing developments.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
KMs are written as templates by using the Oracle Data Integrator substitution API. The API
methods are Java methods that return a string value. They all belong to a single object instance
named "odiRef". The same method may return different values depending on the type of KM that
invokes it. The following example illustrates how you would write a CREATE TABLE statement in
a KM and what it would generate. The following code is entered in a KM:
CREATE TABLE <%=odiRef.getTable("L", "INT_NAME",
"A")%>(<%=odiRef.getColList("", "\t[COL_NAME] [DEST_CRE_DT]", ",\n", "",
"")%>)
The generated code for the PRODUCT table is:
CREATE TABLE db_staging.I$_PRODUCT( P RODUCT_ID numeric(10),
PRODUCT_NAME varchar(250), FAMILY_ID numeric(4), SKU varchar(13),
LAST_DATE timestamp)
The generated code for the CUSTOMER table is:
CREATE TABLE db_staging.I$_CUSTOMER( CUST_ID numeric(10), CUST_NAME
varchar(250), ADDRESS varchar(250), CITY varchar(50), ZIP_CODE
varchar(12), COUNTRY_ID varchar(3))
Once executed with appropriate metadata, the KM has generated a different code for the
PRODUCT and CUSTOMER tables.
Understanding the Substitution API
The code entered in a KM:
CREATE TABLE<%=odiRef.getTable("L", "INT_NAME",
"A")%>(<%=odiRef.getColList("", "\t[COL_NAME]
[DEST_CRE_DT]",
",\n", "", "")%>)<%=odiRef.method_name(parameters)%>
The generated code for the table is:
CREATE TABLE db_staging.I$_PRODUCT( P RODUCT_ID
numeric(10),
PRODUCT_NAME varchar(250), FAMILY_ID numeric(4), SKU
varchar(13),
LAST_DATE timestamp)
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Substitution methods are direct calls to ODI methods implemented in Java. These methods are
used to generate text that corresponds to the metadata stored in the ODI repository or session
information.
If you write an expression to express a business rule, you do not generally know in which context
this expression is run. For instance, if you refer to a specific schema by its name in your test
setup, it may not work in your production environment. But, you want to be able to specify your
table in a generic way. Substitution methods enable you to do this. You can specify the table
name with a substitution method. At run time, ODI automatically adds the name of the appropriate
physical schema for the context. You can also use substitution methods to retrieve information
about the current session. For example, you can include the time when the session was launched,
its name, or the code for the current context. Similarly, substitution methods give you access to the
metadata about the source and target of your interface. The general syntax to use a substitution
method is as follows:
<%=odiRef.method_name(parameters)%>
Angle brackets with percentage signs enclose all substitution method calls. The equal sign tells
ODI to replace the tags and everything inside with the result of the method call.
odiRef is a special ODI Java object, which contains all the methods for performing substitution.
You can obtain more information about these methods in Oracle Fusion Middleware Knowledge
Module Developer's Guide for Oracle Data Integrator 11g Release 1 (11.1.1).
Using Substitution Methods
These methods are used to write generic code, such as:
Table names in a context-dependent format
Information about the session
Source and target metadata information
Use the method with the following syntax:
<%=odiRef.method_name(parameters)%>
Refer to the Oracle Fusion Middleware Knowledge
Module Developer's Guide for Oracle Data Integrator
11g Release 1 (11.1.1).
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The substitution methods can be categorized depending on the type of Knowledge Module into
which they can be used. Global methods are the methods that can be used in any situation (in all
Knowledge Modules and in actions). In addition to the methods from the Global Methods list,
some methods can be used specifically in Journalizing Knowledge Modules (JKMs), Loading
Knowledge Modules (LKMs), Integration Knowledge Modules (IKMs), Check Knowledge Modules
(CKMs), Reverse-Engineering Knowledge Modules (RKMs), or in Service Knowledge Modules
(SKMs).
For details of using each substitution method, refer to Appendix A Substitution API Reference in
Oracle Fusion Middleware Knowledge Module Developer's Guide for Oracle Data Integrator 11g
Release 1 (11.1.1).
Understanding Various ODI Substitution Methods
Global methods: These methods can be used in any
situation.
They can be used in any KM or in actions.
Methods that can be used specifically in KMs of certain
type:
Journalizing KM methods
Loading KM methods
Integration KM methods
Check KM methods
Reverse-Engineering KM methods
Service KM methods
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
These examples show how to use substitution methods. First, you look at how to access the
default value of a column. In ODI, you can specify a default value for each column. To access this
default value, use the getColDefaultValue substitution method. If your column is a string, you
must enclose the entire method call within single quotation marks. If the default value for this
column is unknown, the result of this expression is the unknown string with single quotation
marks.
You might to want to record the system date into a last updated column. Normally, you use a
function available in your database engine. What if your target is not a database, but a flat file? In
this case, you can use the getSysDate substitution method. The system date on the machine
where the agent is running is used. To call this function, pass one argument, which is the date
format to be used. By passing yyyy, a four-digit date is returned. Thus, the final string is
something like The year is: 2006.
The next example shows how to use a SELECT statement in a filter. For example, you want to
import orders from the ORDERS table. You are retrieving only those orders processed in the last
week. To do this, you create a subquery that retrieves the maximum order date minus seven days
from the ORDERS table. However, the schema containing the ORDERS table may change on
different servers. Therefore, use the getObjectName substitution method to refer to the table
name. The generated code will contain the qualified name of the table. Thus, the final filter
expression refers to something like MSSQL_ORDERS_PROD.SRC_ORDERS.
Substitution Methods: Basic Examples
Mapping a target column to its default value defined in
ODI:
'<%=odiRef.getColDefaultValue()%>'
Mapping a column by using the system date:
'The year is:
<%=odiRef.getSysDate("yyyy")%>
Writing a generic SELECT subquery in a filter:
ORDER_DATE >=(SELECT MAX(ORDER_DATE)-7 FROM
<%=odiRef.getObjectName("SRC_ORDERS")%>)
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
An action corresponds to a DDL operation (CREATE TABLE, DROP REFERENCE, and so on) used
to generate a procedure to implement in a database the changes performed in a data integrator
model (GENERATE DDL operation). Each action contains several Action Lines, corresponding to
the commands required to perform the DDL operation (for example, dropping a table requires
dropping all its constraints first).
Action lines contain statements valid for the technology of the action group. Unlike procedures or
Knowledge Module commands, these statements use a single connection (SELECT ... INSERT
statements are not possible).
In the style of the Knowledge Modules, actions make use of the substitution methods to make their
DDL code generic. Action Calls methods are usable in the action lines only. Unlike other
substitution methods, they are not used to generate text, but to generate actions appropriate for
the context. For example, to perform a DROP TABLE DDL operation, you must first drop all foreign
keys referring to the table. In the DROP TABLE action, the first action line will use the
dropReferringFKs() action call method to automatically generate a Drop Foreign Key action
for each foreign key of the current table. This call is performed by creating an action line with the
following code: <% odiRef.dropReferringFKs(); %> The syntax for calling the action call
methods is: <% odiRef.method_name(); %>
Note: The action call methods must be alone in an action line, should be called without a
preceding "=" sign, and require a trailing semi-colon.
Using Substitution Methods in Actions
Action corresponds to DDL operation to generate a procedure
for implementing database changes performed in ODI model
Action Lines Code: Statements valid for the technology
For example: Dropping and checking a constraint on a table:
TABLE <%=odiRef.getTable("L", "TARG_NAME", "A") %>
DROP CONSTRAINT
<%=odiRef.getCK("COND_NAME") %>
Actions Calls Method:
<% odiRef.method_name(); %>
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The following Action Call Methods are available for Actions:
addAKs(): Call the Add Alternate Key action for all alternate keys of the current table.
dropAKs(): Call the Drop Alternate Key action for all alternate keys of the current table.
addPK(): Call the Add Primary Key for the primary key of the current table.
dropPK(): Call the Drop Primary Key for the primary key of the current table.
createTable(): Call the Create Table action for the current table.
dropTable(): Call the Drop Table action for the current table.
addFKs(): Call the Add Foreign Key action for all the foreign keys of the current table.
dropFKs(): Call the Drop Foreign Key action for all the foreign keys of the current table.
enableFKs(): Call the Enable Foreign Key action for all the foreign keys of the current table.
disableFKs(): Call the Disable Foreign Key action for all the foreign keys of the current table.
addReferringFKs(): Call the Add Foreign Key action for all the foreign keys pointing to the
current table.
dropReferringFKs(): Call the Drop Foreign Key action for all the foreign keys pointing to the
current table.
enableReferringFKs(): Call the Enable Foreign Key action for all the foreign keys pointing to
the current table.
disableReferringFKs(): Call the Disable Foreign Key action for all the foreign keys pointing
to the current table.
addChecks(): Call the Add Check Constraint action for all check constraints of the current table.
dropChecks(): Call the Drop Check Constraint action for all check constraints of the current
table.
addIndexes(): Call the Add Index action for all the indexes of the current table.
dropIndexes(): Call the Drop Index action for all the indexes of the current table.
modifyTableComment(): Call the Modify Table Comment for the current table.
AddColumnsComment(): Call the Modify Column Comment for all the columns of the current
table.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
When working in Designer, you should avoid specifying physical information such as the database
name or schema name because they may change depending on the execution context. The
correct physical information will be provided by Oracle Data Integrator at execution time by using
substitution methods.
Working with Object Names
When working in Designer, you do not specify physical
information such as the database name or schema name as
they may change depending on the execution context. The
physical information will be provided by ODI at execution time.
The correct physical information will be provided by Oracle
at execution time.
The substitution API has methods that calculate the fully
qualified name of an object or data store taking into
account the context at run time.
For example, for object name:
getObjectName("L", "OBJ_NAME", "D")
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The substitution API has methods that calculate the fully qualified name of an object or data store
taking into account the context at run time. These methods are listed in the table below in this
slide.
Working with Object Names
Qualified name
required
Method Usable In
Any object named
OBJ_NAME
getObjectName("L",
"OBJ_NAME", "D")
Anywhere
The target data
store of the
current interface
getTable("L", "TARG_NAME",
"A")
LKM, CKM, IKM, JKM
The integration
(I$) table of the
current interface.
getTable("L", "INT_NAME",
"A")
LKM, IKM
The loading table
(C$) for the
current loading
phase.
getTable("L", "COLL_NAME",
"A")
LKM
The error table
(E$) for the data
store being
checked.
getTable("L", "ERR_NAME",
"A")
LKM, CKM, IKM
The data store
being checked
getTable("L", "CT_NAME",
"A")
CKM
The data store
referenced by a
foreign key
getTable("L",
"FK_PK_TABLE_NAME", "A")
CKM
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Generating code from a list of items often requires a while or for loop. Oracle Data Integrator
addresses this issue by providing powerful methods that help you generate code based on lists.
These methods act as iterators to which you provide a substitution mask or pattern and a
separator and they return a single string with all patterns resolved separated by the separator.
All of them return a string and accept at least these four parameters:
Start: A string used to start the resulting string
Pattern: A substitution mask with attributes that will be bound to the values of each item of
the list
Separator: A string used to separate each substituted pattern from the following one
End: A string appended to the end of the resulting string
Some of them accept an additional parameter (the Selector) that acts as a filter to retrieve only
part of the items of the list. For example, list only the mapped column of the target data store of an
interface.
Some of this methods are summarized in the table on the next pages.
Working with Lists of Tables, Columns, and
Expressions
Generating code from a list of items often requires a while or
for loop. ODI provides methods that help you generate code
based on lists. These methods return a string and accept at
least four parameters:
Start: A string used to start the resulting string
Pattern: A substitution mask with attributes that will be
bound to the values of each item of the list
Separator: A string used to separate each substituted
pattern from the following one
End: A string appended to the end of the resulting string
Additional parameter (Selector): A filter to retrieve only part
of the items of the list.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Note
Instead of the getPKColList() method, you can alternatively use getColList with the
selector parameter set to "PK.
Instead of getSrcTablesList(), whenever possible use the getFrom method. The
getFrom method is discussed later in this lesson.
Instead of getFilterList() , use the getFilter() method, which is more appropriate in
most cases.
Expressions
Method Description Usable In
getColList()
Returns a list of columns and
expressions that need to be executed in
the context; accepts a "selector" as a 5th
parameter for filtering
LKM, CKM, IKM, JKM, SKM
getTargetColList()
Returns the list of columns in the target
data store
LKM, CKM, IKM, JKM, SKM
getAKColList()
Returns the list of columns defined for an
alternate key
CKM, SKM
getPKColList()
Returns the list of columns in a primary
key
CKM, SKM
getFKColList()
Returns the list of referencing columns
and referenced columns of the current
foreign key
CKM, SKM
getSrcTablesList()
Returns the list of source tables of an
interface
LKM, IKM
getFilterList()
Returns the list of filter expressions in an
interface
LKM, IKM
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Note
Instead of the getJoinList() method, you can alternatively use the getJoin method,
which is usually more appropriate.
Instead of getGrpByList(), whenever possible use the getGrpBy method.
Instead of getHavingList(), use the getHaving method, which is more appropriate in
most cases.
Expressions
getJoinList()
Returns the list of join expressions in
an interface
LKM, IKM
getGrpByList()
Returns the list of expressions that
should appear in the group by clause
when aggregate functions are detected
in the mappings of an interface
LKM, IKM
getHavingList()
Returns the list of expressions that
should appear in the having clause
when aggregate functions are detected
in the filters of an interface
LKM, IKM
getSubscriberList() Returns a list of subscribers JKM
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
In this example:
Start is set to "(\n": The generated code will start with a parenthesis followed by a carriage
return (\n).
Pattern is set to "\t[COL_NAME] [DEST_WRI_DT]": The generated code will loop over
every target column and generate a tab character (\t) followed by the column name
([COL_NAME]), a white space, and the destination writable data type ([DEST_WRI_DT]).
The Separator is set to ",\n": Each generated pattern will be separated from the next one
with a comma (,) and a carriage return (\n).
End is set to "\n)": The generated code will end with a carriage return (\n) followed by a
parenthesis.
Example: Using getTargetColList to Create a
Table
The following example shows how to use a column list to
create a table:
KM code:
Create table MYTABLE
<%=odiRef.getTargetColList("(\n", "\t[COL_NAME]
[DEST_WRI_DT]", ",\n", "\n)")%>
Generates the following statement:
Create table MYTABLE ( CUST_ID numeric(10),
CUST_NAME varchar(250), ADDRESS varchar(250),
CITY varchar(50), ZIP_CODE varchar(12),
COUNTRY_ID varchar(3) )
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
In this example, the values that need to be inserted into MYTABLE are either bind variables with
the same name as the target columns or constant expressions if they are executed on the target.
To obtain these two distinct set of items, the list is split using the Selector parameter:
"INS AND NOT TARG": First, generate a comma-separated list of columns ([COL_NAME])
mapped to bind variables in the value part of the statement (:[COL_NAME]). Filter them to
get only the ones that are flagged to be part of the INSERT statement and that are not
executed on the target.
"INS AND TARG": Then generate a comma-separated list of columns ([COL_NAME])
corresponding to expression ([EXPRESSION]) that are flagged to be part of the INSERT
statement and that are executed on the target. The list should start with a comma if any
items are found.
Example: Using getColList in an Insert Values
Statement
The following example shows how to use column listing to
insert values into a table:
KM code:
insert into MYTABLE ( <%=odiRef.getColList("",
"[COL_NAME]", ", ", "\n", "INS AND NOT TARG")%>
<%=odiRef.getColList(",", "[COL_NAME]", ", ", "", "INS AND
TARG")%> ) Values ( <%=odiRef.getColList("", ":[COL_NAME]",
", ", "\n", "INS AND NOT TARG")%> <%=odiRef.getColList(",
", "[EXPRESSION]", ", ", "", "INS AND TARG")%> )
insert into MYTABLE ( CUST_ID, CUST_NAME, ADDRESS, CITY,
COUNTRY_ID , ZIP_CODE, LAST_UPDATE ) Values ( :CUST_ID,
:CUST_NAME, :ADDRESS, :CITY, :COUNTRY_ID , 'ZZ2345',
current_timestamp )
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
In this example, getSrcTableList generates a message containing the list of resource names
used as sources in the interface to append to MYLOGTABLE. The separator used is composed of a
concatenation operator (||) followed by a comma enclosed by quotation marks (',') followed by the
same operator again. When the table list is empty, the SOURCE_TABLES column of MYLOGTABLE
will be mapped to an empty string ('').
Example: Using getSrcTableList
The following example concatenates the list of the source
tables of an interface for logging purposes:
KM code:
insert into MYLOGTABLE ( INTERFACE_NAME, DATE_LOADED,
SOURCE_TABLES ) values ( '<%=odiRef.getPop("POP_NAME")%>',
current_date, '' <%=odiRef.getSrcTablesList("|| ",
"'[RES_NAME]'", " || ',' || ", "")%> )
insert into MYLOGTABLE ( INTERFACE_NAME, DATE_LOADED,
SOURCE_TABLES ) values ( 'Int. CUSTOMER', current_date,
'' || 'SRC_CUST' || ',' || 'AGE_RANGE_FILE' || ',' ||
'C$0_CUSTOMER' )
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
getFrom() method
The FROM clause is built accordingly with the appropriate keywords (INNER, LEFT, and so
on) and parentheses when supported by the technology.
When used in an LKM, this method returns the FROM clause because it should be executed
by the source server.
When used in an IKM, it returns the FROM clause because it should be executed by the
staging area server.
getFilter method
When used in an LKM, it returns the filter clause because it should be executed by the
source server.
When used in an IKM, it returns the filter clause because it should be executed by the
staging area server.
getGrpBy() method
The GROUP BY clause includes all mapping expressions referencing columns that do not
contain aggregation functions.
Generating the Source Select Statement
getFrom()
Returns the FROM clause of a SELECT
statement with the appropriate source
tables, left, right and full outer joins
LKM, IKM
getFilter()
Returns filter expressions separated by
an AND operator
LKM, IKM
getJrnFilter()
Returns the special journal filter
expressions for the journalized source
data store. This method should be used
with the Changed Data Capture (CDC)
framework.
LKM, IKM
getGrpBy()
Returns the GROUP BY clause when
aggregation functions are detected in the
mappings
LKM, IKM
getHaving()
Returns the HAVING clause when
aggregation functions are detected in
filters. The having clause includes all
filters expressions containing
aggregation functions.
LKM, IKM
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
This slide shows you the examples of code you use to obtain the result set from any SQL RDBMS
source server and any SQL RDBMS staging area server to build your final flow data.
Note that the getColList is filtered to retrieve only expressions that are not executed on the
target and that are mapped to writable columns. Because all filters and joins start with an AND,
the WHERE clause of the SELECT statement starts with a condition that is always true (1=1).
Generating the Source Select Statement: Example
To obtain the result set from any SQL RDBMS source
server, you would use the following SELECT statement in
your LKM:
select <%=odiRef.getPop("DISTINCT_ROWS")%> <%=odiRef.getColList("",
"[EXPRESSION]\t[ALIAS_SEP] [CX_COL_NAME]", ",\n\t", "", "")%> from
<%=odiRef.getFrom()%> where (1=1) <%=odiRef.getFilter()%>
<%=odiRef.getJrnFilter()%> <%=odiRef.getJoin()%>
<%=odiRef.getGrpBy()%> <%=odiRef.getHaving()%
To obtain the result set from any SQL RDBMS staging
area server to build your final flow data, you would use the
following SELECT statement in your IKM:
select <%=odiRef.getPop("DISTINCT_ROWS")%> <%=odiRef.getColList("",
"[EXPRESSION]", ",\n\t", "", "(not TRG) and REW")%> from
<%=odiRef.getFrom()%> where (1=1) <%=odiRef.getJoin()%>
<%=odiRef.getFilter()%> <%=odiRef.getJrnFilter()%>
<%=odiRef.getGrpBy()%> <%=odiRef.getHaving()%>
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Oracle Data Integrator supports data sets. Each data set represents a group of joined and filtered
sources tables, with their mappings. Data sets are merged into the target data store using set-
based operators (UNION, INTERSECT, and so on) at the integration phase.
During the loading phase, the LKM always works on one data set. During the integration phase,
when all data sets need to merged, certain odiRef APIs that support working on a specific data set
are called using an index that identifies the data set. The example in this slide explains how this
data set merging is done. A Java For loop iterates over the data sets. The number of data sets is
retrieved using the getDataSetCount method. For each data set, a SELECT statement is
issued, each statement separated from the previous one by the data sets set-based operator
retrieved using the getDataSet method. The SELECT statement is built as in Generating the
Source Select Statement (see previous slide), except that each method call is parameterized with
i, the index of the data set being processed. For example, getFrom(i) generates the FROM
statement for the data set identified by the value of i. All the methods that support a parameter for
the data set index also support a syntax without this index value. Outside an IKM, these methods
should be used without the data set index. Within an IKM, if used without the data set index, these
methods address the first data set.
All the methods that support a parameter for the data set index also support Oracle Data
Integrator interfaces and knowledge modules.
Working with Data Sets
Each data set represents a group of joined and filtered
sources tables, with their mappings.
During the loading phase, the LKM always works on one
data set.
Data sets are merged into the target data store using set-
based operators (UNION, INTERSECT, and so on) at the
integration phase, when certain odiRef APIs are called.
Example of data set merging code:
<%for (int i=0; i < odiRef.getDataSetCount(); i++){%>
<%=odiRef.getDataSet(i, "Operator")%> select
<%=odiRef.getUserExit("OPTIMIZER_HINT")%>
<%=odiRef.getPop("DISTINCT_ROWS")%> <%=odiRef.getColList(i,"",
"[EXPRESSION]", ",\n\t", "", "(((INS or UPD) and !TRG) and REW)")%>,
<% if (odiRef.getDataSet(i, "HAS_JRN").equals("1")) { %> JRN_FLAG
IND_UPDATE <%} else {%> 'I' IND_UPDATE <%}%> from
<%=odiRef.getFrom(i)%> where (1=1) <%=odiRef.getJoin(i)%>
<%=odiRef.getFilter(i)%> <%=odiRef.getJrnFilter(i)%>
<%=odiRef.getGrpBy(i)%> <%=odiRef.getHaving(i)%> <%}%>
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
This slide shoes methods that provide additional information, which may be useful. Note that for
getFlexFieldValue() with the List methods, flexfield values can be specified as part of the
pattern parameter.
Obtaining Other Information with the API
The following methods provide additional information:
getPop()
Returns information about the
current interface
LKM, CKM, IKM, JKM
getInfo()
current running session
Any procedure or KM
getOption()
Returns the value of a particular
option
Any procedure or KM
getFlexFieldValue()
Returns information about a
flexfield value
Any procedure or KM
getJrnInfo()
Returns information about the CDC
framework
LKM, IKM, CKM
getTargetTable()
target table of an interface
LKM, IKM, CKM
getModel()
current model during a reverse-
engineering process
RKM
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Answer: b
Explanation: Duplicating an existing KM and enhancing it by changing its technology, or copying
lines of code from another KM is a regular practice for developing a KM.
Quiz
You should never duplicate an existing KM for developing a
new one.
a. True
b. False
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Summary
Describe the guidelines for working with Knowledge
Modules (KMs)
Use ODI substitution API for developing KMs
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Practice 2-1 Overview: Developing and Enhancing
an Integration Knowledge Module
In this practice, you develop an integration knowledge module
with substitution methods and options to enhance the KM with
new functionality.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Objectives
Describe best practices and techniques of developing and
enhancing Knowledge Modules
Use ODI Substitution methods with Java in KMs
Use various ODI tags in KMs
Troubleshoot KMs
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Developing Knowledge Modules: Best Practices
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Oracle Data Integrator provides a large number of Knowledge Modules out-of-the-box. It is
recommended that you start by reviewing the existing KMs and start from an existing KM that is
close to your use case. Duplicate this KM and customize it by editing the code.
When developing your own KM, remember that it is targeted at a particular stage of the integration
process. As a reminder:
LKMs are designed to load remote source data sets to the staging area into Loading (C$)
tables.
IKMs apply the source flow from the staging area to the target. They start from the Loading
tables, may transform and join them into a single integration table (I$) table, may call a
CKM to perform data quality checks on this integration table, and finally write the flow data to
the target.
Developing KMs: Target a Particular Stage of your
Integration Process
1. When developing your own KM, remember that it is
targeted at a particular stage of the integration process:
LKMs are designed to load remote source data sets to the
staging area into Loading (C$) tables
IKMs apply the source flow from the staging area to the
target:
Start from the Loading tables
May transform and join them into a single integration table ("I$")
table
May call a CKM to perform data quality checks on this
integration table
Finally, write the flow data to the target
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
CKMs check data quality in a data store or an integration table (I$) against data quality
rules expressed as constraints. The rejected records are stored in the error table (E$).
RKMs are in charge of extracting metadata from a metadata provider to the Oracle Data
Integrator repository by using the SNP_REV_xx temporary tables.
JKMs are in charge of creating and managing the Change Data Capture infrastructure.
Developing KMs: Target a Particular Stage of your
Integration Process
CKMs check data quality in a data store or an integration
table (I$) against data quality rules expressed as
constraints
The rejected records are stored in the error table (E$).
RKMs extract metadata from a metadata provider to the
Oracle Data Integrator repository.
They use the SNP_REV_xx temporary tables.
JKMs are in charge of creating and managing the Change
Data Capture infrastructure
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Avoid these common pitfalls:
Creating too many KMs: A typical project requires less than 5 KMs. Do not confuse KMs and
procedures, and do not create one KM for each specific use case. Similar KMs can be
merged into a single one and parameterized using options.
Hard-coding values, including catalog or schema names in KMs: You should instead use the
substitution methods getTable(), getTargetTable(), getObjectName(), Knowledge
Module options, or others as appropriate.
Using variables in KMs: You should instead use options or flexfields to gather information
from the designer.
Writing the KM entirely in Jython, Groovy, or Java: You should do this only if it is the
appropriate solution (for example, when sourcing from a technology that only has a Java
API). SQL is easier to read, maintain, and debug than Java, Groovy, or Jython code.
Using <%if%> statements instead of check box options for conditional code generation
Developing KMs: What to Avoid?
2. Try to avoid:
Creating too many KMs
Using hard-coded values, including catalog or schema
names in KMs
You should instead use the substitution methods getTable(),
getTargetTable(), getObjectName(), or KM options.
Using too many variables in KMs
You should instead use options or flexfields to gather
information from the designer.
Writing the KM entirely in Jython, Groovy, or Java
SQL is easier to read, maintain , and debug.
Using <%if%> statements instead of check box options for
conditional code generation
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Other common code-writing recommendations that apply to KMs:
The code should be correctly indented. The generated code should also be indented in order
to be readable. SQL keywords such as select, insert, and so on should be in lowercase
for better readability.
Developing KMs: Other Recommendations
3. Other code-writing recommendations:
The code should be correctly indented.
SQL keywords such as select, insert, and so on should be
in lowercase for better readability.
<%
String myOptionValue=odiRef.getOption("OPT001");
if (myOption.equals("TRUE"))
{
out.print("/* Option OPT001 is set to TRUE */");
}
else
{
%>
/* The OPT001 option is not properly set */
<%
}
%>
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Using Java in Knowledge Modules
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The code generation in Oracle Data Integrator is able to interpret any Java code enclosed
between <% and %> tags.
Using Java in Knowledge Modules
The code of the Knowledge Modules is partially written in
Java.
The tags <%=%> are used to enclose the fragments of
Java code.
More Java can be called using these tags.
Example:
<%
myVar = 'VAR';
%>
select ...
from ...
where COL1 = <%=myVar%>
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
This slide shows the examples of using code generation tags.
The % tag is used with most substitution methods, for example to generate a list of columns.
The ? tag is used with if statement , for example to check if the Mandatory attribute of the
columns is flagged or not. In this example, if the column is mandatory then ODI generates NOT
NULL otherwise it is NULL.
Using Code Generation Tags:
create table <%=odiRef.getTable("L", "TARG_NAME", "A")%>
(
<%=odiRef.getColList("", "\t[COL_NAME] [DEST_CRE_DT] <?if ([COL_MANDATORY]
== 1){out.print(\u0022NOT\u0022);}?> NULL", ",\n", "", "")%>
)
)
create table <?=odiRef.getObjectName("L", "CUSTOMER", "A")?>
(
CUST_ID number(10) <?if (1 == 1){out.print("NOT");}?> NULL,
CUSNAME varchar(50) <?if (1 == 1){out.print("NOT");}?> NULL,
ADDRESS varchar(80) <?if (0 == 1){out.print("NOT");}?> NULL,
COUNTRY varchar(50) <?if (0 == 1){out.print("NOT");}?> NULL,
...
)
Compiled by ODI Designer
create table DEMO_PRODUCTION.CUSTOMER
(
CUST_ID number(10) NOT NULL,
CUSNAME varchar(50) NOT NULL,
ADDRESS varchar(80) NULL,
COUNTRY varchar(50) NULL,
...
)
Compiled by the Agent
The % tag is used to
generate a list of
columns.
The ? tag is used to
check if the
Mandatory attribute
of the columns is
flagged or not.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
This slide shows examples of code generation tags used in compilation steps:
1. Procedure / KM as it appears in ODI Designer
2. Generated Code
- Independent from the context
- Scenario or session ready to be executed
3. Generated code
- Resolution of the context
- Sensitive information
- Visible in the Sunopsis logs
4. Executed code
Using Code Generation Tags: Examples
The substitution methods used in compilation steps:
connect my_user/<%=odiRef.getInfo("DEST_ENCODED_PASS")%>
insert into <%=odiRef.getTable("L","TARG_NAME", "D")%>
select <%=odiRef.getColList("", "[COL_NAME]", ", ", "", "")%>
...
connect my_user/<?=odiRef.getInfo("DEST_ENCODED_PASS")?>
insert into <?=odiRef.getObjectName("L","TARG_NAME", "MY_LSCHEMA", "D")?>
select EMP_ID, EMP_NAME, EMP_CODE
...
Compiled by ODI Designer Generation of a session or a scenario
connect my_user/<@=odiRef.getInfo("DEST_ENCODED_PASS")@>
insert into SCOTT.EMP
...
Compiled by the Agent Written in the Logs
connect my_user/M4678GHT
insert into SCOTT.EMP
...
Compiled by the Agent Execution
1
2
3
4
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
This slide shows an example of how you can use these advanced techniques. The following KM
code creates a string variable and uses it in a substitution method:
<%
String myTableName;
myTableName = "ABCDEF";
%>
drop table <%=odiRef.getObjectName(myTableName.toLowerCase())%>
This code generates the following:
drop table SCOTT.abcdef
Understanding Code Generation Techniques
Using Java Variables and String Functions:
The following KM code creates a string variable and uses it
in a substitution method:
<%
String myTableName;
myTableName = "ABCDEF";
%>
drop table
<%=odiRef.getObjectName(myTableName.toLowerCase())%>
The following is generated:
drop table SCOTT.abcdef
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
You can use conditional branching and advanced programming techniques to generate code. The
following example illustrates how you can use conditional branching:
The following KM code generates code depending on the OPT001 option value.
<%
{out.print("/* Option OPT001 is set to TRUE */");}
else
{%> /* The OPT001 option is not properly set */ <%}
%>
If OPT001 is set to TRUE, then the following is generated:
/* Option OPT001 is set to TRUE */
Otherwise the following is generated:
/* The OPT001 option is not set to TRUE */
Understanding Code Generation Techniques
Using a KM Option to generate code conditionally:
The following KM code generates code depending on the
OPT001 option value:
<%
{ out.print("/* Option OPT001 is set to TRUE */"); }
else
{ %> /* The OPT001 option is not properly set */ <% }
%>
If OPT001 is set to TRUE, then the following is generated:
/* Option OPT001 is set to TRUE */
Otherwise, the following is generated:
/* The OPT001 option is not set to TRUE */
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Using Substitution Methods: Code Examples
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The getJoin() method retrieves the SQL join string (on the source during the loading, on the
staging area during the integration) for a given data set of an interface.
Parameter
pDSIndex (Int): Index identifying which of the data sets is taken into account by this command
Note: The pDSIndex parameter can be omitted when this method is used in an LKM. It can be
also omitted for IKMs. In this case, the data set taken into account is the first one.
Using ODI Substitution Methods: getJoin()
getJoin(): Returns the entire WHERE clause section
generated for the joins of an interface
Usage: public java.lang.String
getJoin(java.lang.Int pDSIndex)
Parameter (Int):
pDSIndex : Index identifying which of the data sets is taken
into account by this command
Example:
insert into <%=odiRef.getTable("L", "COLL_NAME",
"W")%> select <%=odiRef.getColList("",
"[EXPRESSION]", ", ", "", "INS=1")%> from
<%=odiRef.getFrom()%> where (1=1)
<%=odiRef.getJoin()%> <%=odiRef.getFilter()%>
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The getFilter() method returns filter expressions separated by an AND operator. When used
in an LKM, it returns the filter clause because it should be executed by the source server. When
used in an IKM, it returns the filter clause because it should be executed by the staging area
server.
Using ODI Substitution Methods: getFilter()
getFilter(): Returns the entire WHERE clause section
generated for the filters of an interface
getFilter(java.lang.Int pDSIndex)
Parameter (Int):
pDSIndex : Index identifying which of the data sets is taken into
account by this command
Example:
insert into <%=odiRef.getTable("L", "COLL_NAME",
"W")%> select <%=odiRef.getColList("",
"[EXPRESSION]", ", ", "", "INS=1")%> from
<%=odiRef.getFrom()%> where (1=1)
<%=odiRef.getJoin()%> <%=odiRef.getFilter()%>
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
getPK() method returns information relative to the primary key of a data store during a check
procedure. In an action, this method returns information related to the primary key currently
handled by the DDL command.
Parameters
ID: Internal number of the PK constraint
KEY_NAME: Name of the primary key
MESS: Error message relative to the primary key constraint
FULL_NAME: Full name of the PK generated with the local object mask
<flexfield code>: Flexfield value for the primary key
Using ODI Substitution Methods: getPK()
getPK(): Returns information about primary key
getPK(java.lang.String pPropertyName)
Parameter (String):
pPropertyName: String that contains the name of the
requested property
Values:
ID: Internal number of the PK constraint
KEY_NAME: Name of the primary key
MESS: Error message relative to the primary key constraint
FULL_NAME: Full name of the PK generated with the local object
mask.
<flexfield code>: Flexfield value for the primary key.
Example: <%=odiRef.getPK("KEY_NAME")%>
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
This slide shows you examples of substitution methods that can be used specifically in
Journalizing Knowledge Modules (JKMs).
The getJrnFilter() method returns the SQL Journalizing filter for a given data set in the
current interface. If the journalized table is in the source, this method can be used during the
loading phase. If the journalized table is in the staging area, this method can be used while
integrating.
Parameter (Int): pDSIndex Index identifying which of the data sets is taken into account by this
command.
The getJrnInfo() method returns journalizing information about a data store.
Parameter (String): pPropertyName The name of the requested property, for example:
FULL_TABLE_NAME: Full name of the journalized data store
JRN_NAME: Name of the journal data store
JRN_SUBSCRIBER: Name of the subscriber
JRN_METHOD: Journalizing Mode (consistent or simple)
Note: This method returns information about a data stores journalizing for a JKM while
journalizing a model/data store, or for a LKM/IKM in an interface.
Using ODI Substitution Methods in Journalizing
Knowledge Modules: Examples
getJrnFilter(): Returns the SQL Journalizing filter for
a given data set in the current interface, for example:
<%=odiRef.getJrnFilter()%>
getJrnFilter(java.lang.Int pDSIndex)
Parameter: pDSIndex Index identifying which of the data
sets is taken into account by this command
getJrnInfo(): Returns journalizing information about a
data store, for example:
<%=odiRef.getJrnInfo("FULL_TABLE_NAME")%>
getJrnInfo(java.lang.String pPropertyName
Parameter: pPropertyName String containing the name
of the requested property
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The getSubscriberList() method returns a list of subscribers for a journalized table. The
pPattern parameter is interpreted and then repeated for each element of the list, and separated
from its predecessor with the parameter pSeparator. The generated string begins with pStart
and ends with pEnd. If no element is selected, pStart and pEnd are omitted and an empty string
is returned.
Parameters (String):
pStart: This sequence marks the beginning of the string to generate.
pPattern: The pattern is repeated for each occurrence in the list. The list of the attributes
usable in a pattern is detailed in the Pattern Attributes List below. Each occurrence of the
attributes in the pattern string is replaced by its value. Attributes must be between brackets ([
and ]). For example: My name is [SUBSCRIBER].
pSeparator: This parameter separates each pattern from its predecessor.
pEnd: This sequence marks the end of the string to generate.
Example:
Returning list of Subscribers: <%=odiRef.getSubscriberList("\nBegin List\n", "-
[SUBSCRIBER]", "\n", "\nEnd of List\n")%>
Using ODI Substitution Methods in Journalizing
Knowledge Modules: Examples
getSubscriberList(): Returns a list of subscribers for
a journalized table
getSubscriberList( java.lang.String pStart,
java.lang.String pPattern, java.lang.String
pSeparator, java.lang.String pEnd)
Parameters (String):
pStart: This marks the beginning of the string to generate.
pPattern: The pattern is repeated for each occurrence in the
list.
pSeparator: This parameter separates each pattern.
pEnd: This sequence marks the end of the string to generate.
Example: <%=odiRef.getSubscriberList("\nBegin
List\n", "- [SUBSCRIBER]", "\n", "\nEnd of
List\n")%>
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
This slide shows you examples of substitution methods that can be used specifically in RKM.
The getModel() method returns information on the current data model during the processing of
a personalized reverse engineering. The list of available data is described in the pPropertyName
values table.
Note: This method may be used on the source connection (data server being reverse-engineered)
as well as on the target connection (repository). On the target connection, only the properties
independent from the context can be specified (for example, the schema and catalog names
cannot be used).
Parameters:
ID: Internal identifier of the current model
MOD_NAME: Name of the current model
LSCHEMA_NAME: Name of the logical schema of the current model
MOD_TEXT: Description of the current model
REV_TYPE: Reverse engineering typeS for standard reverse, C for customize
REV_UPDATE: Update flag of the model
REV_INSERT: Insert flag for the model
Using ODI Substitution Methods in Reverse-
Engineering KM: getModel()
getModel(): Returns information on the current data
model during the processing of a personalized reverse
engineering
getModel(java.lang.String pPropertyName))
Parameter: pPropertyName String that contains the
name of the requested property
Values:
ID: Internal identifier of the current model
LSCHEMA_NAME: Name of the logical schema of the current model
MOD_TEXT: Description of the current model
REV_TYPE: Reverse engineering typeS for standard reverse, C
for customize
REV_UPDATE: Update flag of the model
REV_INSERT: Insert flag for the model
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
REV_OBJ_PATT: Mask for the objects to reverse
REV_OBJ_TYPE: List of object types to reverse-engineer for this model. This is a semicolon-
separated list of object types codes. Valid codes are: T: Table; V: View; Q: Queue; SY:
System table; AT: Table alias; SY: Synonym
TECH_INT_NAME: Internal name of the technology of the current model
LAGENT_NAME: Name of the logical execution agent for the reverse engineering
REV_CONTEXT: Execution context of the reverse
REV_ALIAS_LTRIM: Characters to be suppressed for the alias generation
CKM: Check Knowledge Module
RKM: Reverse-engineering Knowledge Module
SCHEMA_NAME: Physical Name of the data schema in the current reverse context
WSCHEMA_NAME: Physical Name of the work schema in the current reverse context
CATALOG_NAME: Physical Name of the data catalog in the current reverse context
WCATALOG_NAME: Physical Name of the work catalog in the current reverse context
<flexfield code>: Value of the flexfield for the current model
Example:
Retrieve the list of tables that are part of the mask of objects to reverse:
select TABLE_NAME, RES_NAME, replace(TABLE_NAME,
'<%=odiRef.getModel("REV_ALIAS_LTRIM")%>' , '') ALIAS, TABLE_DESC from
MY_TABLES where TABLE_NAME like '<%=odiRef.getModel("REV_OBJ_PATT")%>'
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Troubleshooting Knowledge Modules
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
This topic provides information on how to troubleshoot problems that you might encounter when
using Oracle Knowledge Modules.
Errors appear often in Oracle Data Integrator in the following way:
java.sql.SQLException: ORA-01017: invalid username/password; logon
denied at ...
at ...
at ...
java.sql.SQLExceptioncode simply indicates that a query was made to the database
through the JDBC driver, which has returned an error. This error is frequently a database or driver
error, and must be interpreted in this direction.
Only the part of text in bold must first be taken in account. It must be searched in the Oracle
documentation. If it contains an error code specific to Oracle, the error can be immediately
identified.
If such an error is identified in the execution log, it is necessary to analyze the SQL code sent to
the database to find the source of the error. The code is displayed in the description tab of the
erroneous task.
Troubleshooting KMs
Identifying database errors:
Errors appear often in the following way in ODI:
java.sql.SQLException: ORA-01017: invalid
username/password; logon denied at ...
at ...
at ...
java.sql.SQLExceptioncode indicates that a query
was made to the database through the JDBC driver, which
has returned an error (a database or driver error).
If an error message contains an error code specific to
Oracle, the error can be immediately identified.
If an error is identified in the execution log, it is necessary
to analyze the SQL code sent to the database.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
This slide describes common problems and solutions.
ORA-12154 TNS: could not resolve service name
TNS alias resolution. This problem may occur when using the OCI driver, or a KM using database
links. Check the configuration of the TNS aliases on the machines.
ORA-02019 connection description for remote database not found
You use a KM using nonexisting database links. Check the KM options for creating the database
links.
ORA-00900 invalid SQL statement
ORA-00923 FROM Keyword not found where expected
The code generated by the interface, or typed in a procedure is invalid for Oracle. This is usually
related to an input error in the mapping, filter, or join. The typical case is a missing quotation mark
or an unclosed bracket.
Troubleshooting KMs
Common problems and solutions:
ORA-12154 TNS: Could not resolve service name
This problem may occur when using the OCI driver, or a KM
using database links. Check the configuration of the TNS
aliases on the machines.
ORA-02019 connection description for remote database
not found
You use a KM using nonexisting database links. Check the
KM options for creating the database links.
ORA-00900 invalid SQL statement (or ORA-00923 FROM
Keyword not found where expected)
The code generated by the interface, or typed in a procedure
is invalid for Oracle (input error in the mapping, filter, or join).
The typical case is a missing quote or an unclosed bracket.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
A frequent cause is also the call made to a non-SQL syntax such as the call to an Oracle-stored
procedure using the syntax:
EXECUTE SCHEMA.PACKAGE.PROC(PARAM1, PARAM2)
The valid SQL call for a stored procedure is:
BEGIN
SCHEMA.PACKAGE.PROC(PARAM1, PARAM2);
END;
Note: The syntax EXECUTE SCHEMA.PACKAGE.PROC(PARAM1, PARAM2) is specific to
SQL*PLUS, and does not work with JDBC.
Troubleshooting KMs
The syntax error example: ORA 00900 Invalid SQL
Statement
Wrong statement:
EXECUTE SCHEMA.PACKAGE.PROC(PARAM1, PARAM2
Correct statement:
BEGIN
SCHEMA.PACKAGE.PROC(PARAM1, PARAM2);
END;
Note: The syntax EXECUTE
SCHEMA.PACKAGE.PROC(PARAM1, PARAM2) is specific
to SQL*PLUS, and does not work with JDBC.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Other errors
ORA-00904 invalid column name: Keying error in a mapping/join/filter. A string which is not
a column name is interpreted as a column name, or a column name is misspelled. This error
may also appear when accessing an error table linked to a data store with a recently
modified structure. Modify or drop the error tables and let Oracle Data Integrator re-create it
in the next execution.
ORA-00903 invalid table name: The table used (source or target) does not exist in the
Oracle schema. Check the mapping logical/physical schema for the context, and check that
the table physically exists on the schema accessed for this context.
ORA-00972 Identifier is too Long: There is a limit in the object identifier in Oracle (usually
30 characters). When this limit is exceeded, this error appears.
ORA-01790 expression: Must have the same data type as the corresponding expression
You are trying to connect two different values that cannot be implicitly converted (in a mapping or
a join). Use the explicit conversion functions on these values.
Troubleshooting KMs
Other errors:
ORA-00904 invalid column name
Keying error in a mapping/join/filter. A string which is not a
column name is interpreted as a column name.
ORA-00903 invalid table name
The table used (source or target) does not exist in the Oracle
schema.
ORA-00972 Identifier is too Long
There is a limit in the object identifier in Oracle (usually 30
characters). When this limit is exceeded, this error appears.
ORA-01790 expression must have the same data type as
the corresponding expression.
You are trying to connect two different values that cannot be
implicitly converted (in a mapping or join).
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Answer: b
Explanation: You should avoid using <%if%> statements instead of check box options for
conditional code generation.
Quiz
It is recommended to use <%if%> statements instead of check
box options for conditional code generation.
a. True
b. False
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Summary
Describe best practices and techniques of developing and
enhancing Knowledge Modules
Use ODI Substitution methods with Java in KMs
Use various ODI tags in KMs
Troubleshoot KMs
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Practice 3-1 Overview: Creating a New Knowledge
Module for Data Generation
In this practice, you create a new knowledge module that can
be used with ODI integration interfaces for generating
thousands of records with random data in any table in the
Oracle Database.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Designing ODI Integration Interfaces
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Objectives
Describe different styles (E-LT, ETL) of integration
interfaces
Describe a typical integration process
Use best practices for creating integration interfaces of
different styles
Maintain quality of data
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Overview of Integration Process
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
This integration process is also known as an extract, transform, and load (ETL) process.
The first part of an ETL process involves extracting data from the source systems. Most data
warehousing projects consolidate data from different source systems.
The transform stage applies a series of rules or functions to the data extracted from the source to
derive the data for loading into the target. Some data sources will require very little or even no
manipulation of data. In other cases, transformations (such as filtering, joining, sorting, and so on)
may be required to meet the business and technical needs of the target database.
The load phase loads the data into the target, usually the data warehouse.
Note: You can add to this process the checks that ensure the quality of data flow, as shown in the
slide.
Conventional Integration Process: ETL
Integration process
Extract - Transform (check) - Load
Source
ORDERS
LINES
CORRECTIONS
File
Target
SALES
Errors
Transform
A machine
A machine
A machine
Extract Transform Load
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Data is one of the most important assets of any company, and data integration constitutes the
backbone of any enterprises IT systems. Choosing the technology for data integration is critical
for productivity and responsiveness of business divisions within an enterprise.
E-LT stands for extract, load, and transform. It includes the processes that enable companies to
move data from multiple sources, reformat and cleanse the data, and load it into another
database, or a data warehouse for analysis, to support a business process.
ODI provides a strong and reliable integration platform for IT infrastructure. Built on the next-
generation architecture of extract, load, and transform (E-LT), ODI delivers superior performance
and scalability connecting heterogeneous systems at a lower cost than traditional, proprietary ETL
products. Unlike the conventional extract, transform, and load (ETL) design, with ODI, E-LT
architecture extracts data from sources, loads it into a target, and transforms it by using the
database power according to business rules. The tool automatically generates data flows,
manages their complexity, and administers correct instructions for the various source and target
systems.
E-LT
1. Extract: Extracting data from various sources
2. Load: Loading the data into the destination target
3. Transform: Transforming the data according to a set of
business rules
Extract
Transform
Conventional ETL architecture
Load Load Extract
Transform
Transform
Next-generation E-LT architecture
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Extract
The first step in the ELT process is extracting data from various sources. Each of the source
systems may store its data in a completely different format. The sources are usually flat files or
RDBMS, but almost any data storage can be used as a source for an ELT process.
Load
This step involves loading the data into the destination target, which might be a database or data
warehouse.
Transform
After the data has been extracted and loaded, the next step is to transform the data according to a
set of business rules. The data transformation may involve various operations including, but not
limited to filtering data, sorting data, aggregating data, joining data, cleaning data, generating
calculated data based on existing values, and validating data.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
An integration process is always needed in an interface. This process integrates data from the
source or loading tables into the target datastore using a temporary integration table.
An integration process uses an integration strategy, which defines the steps required in the
integration process .The following elements are used in the integration process:
An integration table (also known as the flow table) is sometimes needed to stage data after all
staging area transformation are made. This table is named after the target table, prefixed with
I$. This integration table is the image of the target table with extra fields required for the
strategy to be implemented. The data in this table is flagged, transformed or checked before
being integrated into the target table.
The source and/or loading tables (created by the LKM). The integration process loads data
from these tables into the integration table or directly into the target tables.
Integration Process: Overview
The integration process integrates data from the source or
loading tables into the target datastore, using a temporary
integration table.
The following elements are used in the integration process:
An integration table (also known as the flow table) is
sometimes needed to stage data after all staging area
transformation are made.
This loading table is named after the target table, prefixed
with I$.
The source and/or loading tables (created by the LKM)
The integration process loads data from these tables into the
integration table or directly into the target tables.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Integration Process: Overview
Check Knowledge Module. The IKM may initiate a flow
check phase to check the data in the integration table
against some of the constraints of the target table.
Invalid data is removed from the integration table (removed
from the flow).
Interface metadata, such as Insert, Update, UD1, and so
on, or model metadata such as the Slowly Changing
Dimension (SCD) behavior
They are used at the integration phase to parameterize
column-level behavior in the integration strategies.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
A typical integration process works in the following way:
1. Create a temporary integration table if needed. For example, an update flag taking value I or
U to identify which of the rows are to be inserted or updated.
2. Load data from the source and loading tables into this integration table, executing those of
the transformations (joins, filters, mapping) specified on the staging area.
3. Perform some transformation on the integration table to implement the integration strategy.
For example, compare the content of the integration table with the target table to set the
update flag.
4. Modify the content. Load data from the integration table into the target table.
Typical Integration Process
Perform
Transformations to
Implement the
Integration Strategy
Target
3
Create a Temporary
Integration Table (if
needed)
Loading Data from
the Source and
Loading Table into
Integration Table 1
2
Source
Modify the Content.
Load Data from the
Integration Table to
the Target Table
4
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Now look at the first of the three factors that structure the flow. This is the most important:
choosing where to put the staging area.
Overview of Integration Interfaces
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
An interface consists of a set of rules that define the loading of a data store or a temporary target
structure from one or more source data stores.
An integration interface is made up of and defined by the following components:
Target data store: The target data store is the element that will be loaded by the interface.
This data store may be permanent (defined in a model) or temporary (created by the
interface).
Data sets: One target is loaded with data coming from several data sets. Set-based
operators (Union, Intersect, and so on) are used to merge the different data sets into the
target data store. Each data set corresponds to one diagram of source data stores and the
mappings used to load the target data store from these source data stores.
Diagram of source data stores: A diagram of sources is made of source data stores
possibly filteredrelated using joins. The source diagram also includes lookups to fetch
additional information for loading the target. Two types of objects can be used as a source of
an interface: data stores from the models and interfaces. If an interface is used, its target
data storetemporary or notwill be taken as a source. The source data stores of an
interface can be filtered during the loading process, and must be put in relation through joins.
Joins and filters are either copied from the models or defined for the interface. Join and filters
are implemented in the form of SQL expressions.
Integration Interfaces: Overview
An integration interface consists of a set of rules that
define the loading of a data store or a temporary target
structure from one or more source data stores.
An interface is made up of and defined by the following
components:
Target data store: This is the element that will be loaded by
the interface (permanent or temporary).
Data sets: The target is loaded with data coming from
several data sets. Set-based operators (Union, Intersect, and
so on) are used to merge different data sets into the target
data store.
A diagram of source data stores: This is the diagram that
represents source data stores which have been possible
filtered using joins. These data stores may include lookups to
fetch additional information for loading the target.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Mapping: A mapping defines the transformations performed on one or several source
columns to load one target column. These transformations are implemented in the form of
SQL expressions. Each target column has one mapping per data set. If a mapping is
executed on the target, the same mapping applies for all data sets.
Staging area: The staging area is a logical schema into which some of the transformations
(joins, filters, and mappings) take place. It is by default the same schema as the targets
logical schema. It is possible to locate the staging area on a different location (including one
of the sources). It is the case if the targets logical schema is not suitable for this role. For
example, if the target is a file data store, because the file technology has no transformation
capability.
Note: Mappings can be executed either on the source or target/staging area. Filters and
joins can be executed either on the source or staging area.
Flow: The flow describes how the data flows between the sources, the staging area if it is
different from the target, and the target as well as where joins and filters take place. The flow
also includes the loading and integration methods used by this interface. These are selected
by choosing Loading and Integration Knowledge Modules (LKMs, IKMs).
Control: An interface implements two points of control. Flow control checks the flow of data
before it is integrated into the target. Post-Integration control performs a static check on the
target table at the end of the interface. The check strategy for Flow and Post-Integration
Control is defined by a Check Knowledge Module (CKM).
Integration Interfaces: Overview
Other components of an interface:
Mapping: This defines the transformations performed on one
or several source columns to load one target column.
Each target column has one mapping per data set.
If a mapping is executed on the target, the same mapping
applies for all data sets.
Staging area: This is a logical schema into which some of the
transformations (joins, filters, and mappings) take place. It is
by default the targets logical schema.
Flow: The flow describes how the data flows between the
sources, the staging area, and the target as well as where
joins and filters take place.
Control: Flow control checks the flow of data before it is
integrated into the target. Post-Integration control performs a
static check on the target table at the end of the interface.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The following sequence of operations is more or less unavoidable to implement this integration.
However, you have some latitude in controlling how and where they are performed.
First, the two orders tables must be extracted and joined. The constraint that orders must be
closed is applied here.
Second, data from the corrections file must be extracted and transformed into the correct
format. Now, data from these two sources must be joined and transformed into a temporary
table, I$_SALES. This temporary table looks identical to the target SALES table.
Lastly, the data from this table must be copied to the Oracle server. You can also do a data
check, or flow control, here.
Basic Integration Process: The Sequence of
Operations
Target (Oracle)
SALES
Transform and
integrate
I$_SALES
Source (Sybase)
ORDERS
LINES
CORRECTIONS
File
C$_0
Extract/Join/
Transform
C$_1
Extract/Transform
Join/Transform
1
2
3
4
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The staging area is where most of the transformation and error checking is usually performed.
After those tasks are performed, you just copy, or load, the data into the target server.
The staging area is a special area that you create when you set up a database in ODI. ODI
creates temporary tables in the staging area, and uses them to perform data transformation.
You can place the staging area on your source server, your target server, or another server
altogether. However, the best place for the staging area is usually on the target server. This gives
you the greatest scope for data consistency checking, and minimizes network traffic. You now look
at some of the consequences of different choices.
The Staging Area
Target (Oracle)
Staging AREA
The language is SQL
SALES
ERRORS
Source (Sybase)
ORDERS
LINES
CORRECTIONS
File
TEMP_1
TEMP_2
Join/Transform
TEMP_
SALES
1
2
3
4
5
Extract/Transform
Transform and
integrate
Check constraints/
Isolate errors
Extract/Join/
Transform
A staging area is a separate, dedicated area in an RDBMS
where ODI creates its temporary objects and executes
some of your transformation rules.
You usually place the staging area on the target data
server.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Placing the Staging Area
The staging area may be located on:
The Oracle target database (default)
A third RDBMS database server
The Sybase source database
The staging area cannot be placed on the file source data
server.
This data server is not an RDBMS.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Designing Integration Interfaces: Best Practices
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
In an E-LT-style integration interface, ODI processes the data in a staging area, which is located
on the target. Staging area and target are located on the same RDBMS. The data is loaded from
the sources to the target. To create an E-LT-style integration interface, follow the standard
procedure of creating an ODI interface. See Working with Integration Interface in the Oracle
Fusion Middleware Developer's Guide for Oracle Data Integrator 11g Release 1 (11.1.1). for
generic information about how to design integration interfaces.
In an ETL-style interface, ODI processes the data in a staging area, which is different from the
target. The data is first extracted from the sources and then loaded to the staging area. The data
transformations take place in the staging area and the intermediate results are stored in temporary
tables in the staging area. The data loading and transformation tasks are performed with the
standard ELT KMs.
Designing Integration Interfaces: E-LT- and ETL-
Style Interfaces
E-LT-style integration interface:
ODI processes the data in a staging area, which is located
on the target.
Staging area and target are located on the same RDBMS.
The data is loaded from the sources to the target.
ETL-style interface:
ODI processes the data in a staging area, which is different
from the target.
The data is first extracted from the sources and then loaded
to the staging area.
The data transformations take place in the staging area and
the intermediate results are stored in temporary tables in the
staging area.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
In this topic, you learn how to design an ETL-style interface where the staging area is Oracle
database or any ANSI-92compliant database and the target on Oracle database.
In an ETL-style interface, ODI processes the data in a staging area, which is different from the
target. Oracle Data Integrator provides two ways for loading the data from an Oracle staging area
to an Oracle target:
Using a multiconnection IKM
Using an LKM and a mono-connection IKM
Note: Depending on the KM strategy that is used, flow and static control are supported.
Designing an ETL-Style Interface
In an ETL-style interface, ODI processes the data in a staging
area, which is different from the target. Oracle Data Integrator
provides two ways for loading the data from an Oracle staging
area to an Oracle target:
Using a multiconnection IKM
A multiconnection IKM allows updating a target where the
staging area and sources are on different data servers.
If there is no dedicated multiconnection IKM, use a standard
exporting LKM in combination with a standard mono-
connection IKM.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Designing an ETL-Style Interface: Using
Multiconnection IKM
A multiconnection IKM allows updating a target where the
staging area and sources are on different data servers.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Oracle Data Integrator provides the following multiconnection IKM for handling Oracle data: IKM
Oracle to Oracle Control Append (DBLINK). You can also use the generic SQL multiconnection
IKMs for ANSI SQL-92compliant technologies: IKM SQL to SQL Incremental Update and IKM
SQL to SQL Control Append.
IKM SQL to SQL Incremental Update KM integrates data from any ANSI-SQL92compliant
database into any ANSI-SQL92compliant database target table in incremental update mode. This
IKM is typically used for ETL configurations: source and target tables are on different databases
and the interfaces staging area is set to the logical schema of the source tables or a third schema.
It allows an incremental update strategy with no temporary target-side objects. Use this KM if it is
not possible to create temporary objects in the target server. Because the application updates are
made without temporary objects on the target, the updates are made directly from source to target.
The configuration where the flow table is created on the staging area and not in the target should
be used only for small volumes of data. This KM supports flow and static control.
IKM SQL to SQL Control Append is typically used for ETL configurations: source and target tables
are on different databases and the interfaces staging area is set to the logical schema of the
source tables or a third schema. Use this KM strategy to perform control append.
Note: This KM supports flow and static control.
Multiconnection IKM
Oracle Data Integrator provides the following
multiconnection IKM for handling Oracle data: IKM
Oracle to Oracle Control Append (DBLINK)
You can also use the generic SQL multi-connection IKMs:
IKM SQL to SQL Incremental Update: Integrates data from
any ANSI-SQL92compliant database into any AINSI-
SQL92compliant database target table in incremental
update mode.
Note: Use this KM if it is not possible to create temporary
objects in the target server.
IKM SQL to SQL Control Append: Source and target tables
are on different databases and the interfaces staging area is
set to the logical schema of the source tables or a third
schema.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
To use a multi-connection IKM in an ETL-style interface, perform the following steps:
1. Create an integration interface using the standard procedure as described in the section
Working with Integration Interface in the Oracle Fusion Middleware Developer's Guide for
Oracle Data Integrator 11g Release 1 (11.1.1).
2. On the Definition tab of the Interface Editor, select Staging Area different from Target and
select the logical schema of the source tables or another logical schema that is not a source
or the target. This schema will be used as the staging area.
3. On the Flow tab, select one of the Source Sets. In the Property Inspector, select an LKM
from the LKM Selector list to load from the sources to the staging area. See the chapter in
the Oracle Fusion Middleware Connectivity and Knowledge Modules Guide for Oracle Data
Integrator 11g Release 1 (11.1.1) that corresponds to the technology of your staging area to
determine the LKM you can use. Optionally, modify the KM options.
4. Select the Target by clicking its title. In the Property Inspector, select an ETL multiconnection
IKM from the IKM Selector list to load the data from the staging area to the target.
Optionally, modify the KM options.
Note: See the chapter in Oracle Fusion Middleware Connectivity and Knowledge Modules
Guide for Oracle Data Integrator 11g Release 1 (11.1.1) that corresponds to the technology
of your staging area to determine the IKM you can use.
Multiconnection IKM
To use a multiconnection IKM in an ETL-style interface,
performing the following steps:
1. Create an integration interface using the standard
procedure.
2. In the Definition tab, select Staging Area different from
Target and select the logical schema of the source tables
or another logical schema (not source or target).
3. In the Flow tab, select one of the Source Sets. In the
Property Inspector, select an LKM to load from the sources
4. Select Target. In the Property Inspector, select ETL
multiconnection IKM standard mono-connection IKM to
load the data from the staging area to the target.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
If there is no dedicated multiconnection IKM, use a standard exporting LKM in combination with a
standard mono-connection IKM. This slide shows the configuration of an integration interface
using an exporting LKM and a mono-connection IKM to update the target data. The exporting LKM
is used to load the flow table from the staging area to the target. The mono-connection IKM is
used to integrate the data flow into the target table.
Designing an ETL-Style Interface: Using an LKM
and a mono-connection IKM
If there is no dedicated multiconnection IKM, use a standard
exporting LKM in combination with a standard mono-connection
IKM:
The exporting LKM is used to load the flow table from the
staging area to the target. The mono-connection IKM is
used to integrate the data flow into the target table.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Note that this configuration (LKM + exporting LKM + mono-connection IKM) has the following
limitations:
Neither simple CDC nor consistent CDC are supported when the source is on the same data
server as the staging area (explicitly chosen in the Interface Editor).
Temporary Indexes are not supported.
Designing an ETL-Style Interface: Limitations of
The configuration (LKM + exporting LKM + mono-connection
IKM) has the following limitations:
Neither simple Changed Data Capture (CDC) nor
consistent CDC are supported when the source is on the
same data server as the staging area.
Temporary Indexes are not supported.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
To use an LKM and a mono-connection IKM in an ETL-style interface, perform the following steps:
1. Create an integration interface using the standard procedure as described in the section
Working with Integration Interface" in the Oracle Fusion Middleware Developer's Guide for
2. On the Definition tab of the Interface Editor, select Staging Area different from Target and
select the logical schema of the source tables or a third schema.
3. On the Flow tab, select one of the Source Sets. In the Property Inspector, select an LKM
from the LKM Selector list to load from the sources to the staging area. See the chapter in
the Oracle Fusion Middleware Connectivity and Knowledge Modules Guide for Oracle Data
Integrator 11g Release 1 (11.1.1) that corresponds to the technology of your staging area to
determine the LKM you can use. Optionally, modify the KM options.
4. Select the Staging Area. In the Property Inspector, select an LKM from the LKM Selector list
to load from the staging area to the target. Optionally, modify the options.
5. Select the Target by clicking its title. In the Property Inspector, select a standard mono-
connection IKM from the IKM Selector list to update the target. Optionally, modify the KM
options.
Using an LKM and a Mono-Connection IKM: Steps
To use an LKM and a mono-connection IKM in an ETL-style
interface, perform the following steps:
1. Create an integration interface using the standard
procedure.
2. In the Definition tab, select Staging Area different from
Target and select the logical schema of the source tables
or a third schema.
3. In the Flow tab, select one of the Source Sets. In the
Property Inspector, select an LKM to load from the sources
4. Select the Staging Area. In the Property Inspector, select
an LKM from the LKM Selector list to load from the staging
area to the target.
5. Select Target. In the Property Inspector, select a standard
mono-connection IKM to update the target.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Maintaining Integrity of Data
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Audits provide statistics on the integrity of application data. They also isolate data that is detected
as erroneous by applying the business rules. After erroneous records have been identified and
isolated in error tables, they can be accessed from Oracle Data Integrator Studio, or from any
other front-end application.
Enforcing ODI Data Quality
Audits provide statistics on the integrity of application data.
They also isolate data that is detected as erroneous by
applying the business rules:
Uniqueness rules
Referential integrity rules
Validation rules that enforce consistency at the record level
After erroneous records have been identified and isolated
in error tables, they can be accessed from:
ODI Studio
Any other front-end application
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The following are four ways erroneous data might be handled:
Automatically correct data Oracle Data Integrator offers a set of tools to simplify the
creation of data cleansing interfaces that can be scheduled to run at predetermined intervals.
Accept erroneous data (for the current project) In this case, interface developers need
precise rules for filtering out erroneous data later, using Oracle Data Integrator filters.
Correct the invalid records In this situation, the invalid data is sent to application end users
via various text formats or distribution modes, such as human workflow, e-mail, HTML, XML,
flat text files, and so on, using Oracle Data Integrator packages.
Recycle data Erroneous data from an audit can be recycled into the integration process.
Using the Check Knowledge Modules, you simply have to define the control business rules, and
inconsistent data is automatically isolated for you in error tables. However, isolating erroneous
data is not the only issue in Data Quality. Even if ODI automatically detects duplicated keys,
mandatory fields, missing references, and more complex constraints, you will need to involve
business users in the process of qualifying data discrepancies and make the appropriate
decisions.
Ways to Handle Erroneous data
Automatically correct data
Create Data Cleansing interfaces
Accept erroneous data (for the current project):
In this case, interface developers need precise rules for
filtering out erroneous data later using ODI filters.
Correct the invalid records:
In this case, invalid data is sent to application end users via
various text formats or distribution models.
Examples: Human Workflow, e-mail,
HTML, XML, flat text files, and so on
Recycle data:
Erroneous data from the audit can be recycled into the
integration process.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
In some cases, it is useful to recycle errors from previous runs so that they are added to the flow
and applied again to the target. This method can be useful, for example, when receiving daily
sales transactions that reference product IDs that may not exist. Suppose that a sales record is
rejected in the error table because the referenced product ID does not exist in the product table.
This happens during the first run of the interface. In the meantime the missing product ID is
created by the data administrator. Therefore, the rejected record becomes valid and should be re-
applied to the target during the next execution of the interface.
This mechanism is implemented by IKMs with an extra task that inserts all the rejected records of
the previous executions of this interface from the error table into integration table. This operation is
made prior to calling the CKM to check the data quality, and is conditioned by a KM option usually
called RECYCLE_ERRORS.
Error Recycling
In some cases, it is useful to recycle errors from previous
runs so that they are added to the flow and applied again
to the target, for example:
When receiving daily sales transactions that reference
product IDs that may not exist
When the missing product ID is created, the rejected record
becomes valid and should be reapplied to the target during
the next execution of the interface
IKMs inserts all the rejected records of the previous
executions of this interface from the error table into the
integration table.
This operation is made before calling the CKM to check
the data quality, and is conditioned by a KM option, usually
called RECYCLE_ERRORS.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
You should have an answer to each of the questions in the slide before defining your data quality
strategy.
Building a Data Quality Framework
You should answer the following questions:
What level of data quality is required for the Data
Warehouse?
Who are the business owners of source data
What should we do with rejected records?
Do we need to define an error recycling strategy?
Do we need to involve business owners of the data and
report rejected records?
How would business users modify erroneous source data?
Do we provide a GUI to have them modify erroneous
records in the error tables?
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Answer: a
Explanation: In an ETL-style interface, ODI processes the data in a staging area, which is
different from the target. The data is first extracted from the sources and then loaded to the staging
area. The data transformations take place in the staging area and the intermediate results are
stored in temporary tables in the staging area. The data loading and transformation tasks are
performed with the standard ELT KMs.
Quiz
Which of the following is not true about ETL-style interfaces?
a. ODI processes the data in a staging area, which is on the
target.
b. The data is first extracted from the sources and then
loaded to the staging area.
c. The intermediate results are stored in temporary tables in
the staging area.
d. The data loading and transformation tasks are performed
with the standard ELT KMs.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Summary
Describe different styles (E-LT, ETL) of integration
interfaces
Describe typical integration process
Use best practices for creation integration interfaces of
different styles
Maintain quality of data
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Practice 4-1: Creating an ODI Interface for XML to
Database Transformation with ODI Constraint and
Error Recycling
In this practice, you perform the steps to build an ODI interface
that will load an XML file with a constraint to the database table.
To maintain data quality, you enable error recycling. Any rows
that do not pass the constraint will be loaded to the error table
on the target database E_CLIENT.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Designing Advanced Integration Interfaces
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Objectives
Implement lookups in ODI interfaces
Create ODI interfaces with multiple datasets
Use set-based operators in ODI interfaces
Use partitioning with ODI interfaces
Create ODI temporary interfaces
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Now look at the first of the three factors that structure the flow. This is the most important:
choosing where to put the Staging area.
ODI Interfaces: Advanced Functionality
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
A wizard is available in the interface editor to create lookups by using a source as the driving table
and a lookup datastore or interface. These lookups now appear as compact graphical objects in
the interface sources diagram.
The user can choose how each lookup is generated, either as a Left Outer Join in the FROM clause
or as an expression in the SELECT clause (in-memory lookup with nested loop). The second
syntax is sometimes more efficient in small lookup tables.
This feature simplifies the design and readability of interfaces that use lookups, and enables
optimized code for execution.
Creating Lookups
You can generate lookups by using either of these methods:
Left outer join (execution plan can be sort/merge, hash
join)
select
O.ORDER_ID,
C.COUNTRY_NAME COUNTY_NAME_LOOKUP
From ORDERS O left outer join COUNTRY C
on (O.COUNTRY_ID = C. COUNTRY_ID)
As an expression in the SELECT clause: In-memory lookup
with nested loops (lookup table accessed in-memory)
select
O.ORDER_ID,
(select C.COUNTRY_NAME from COUNTRY C
where O.COUNTRY_ID = C.
COUNTRY_ID)COUNTY_NAME_LOOKUP
From ORDERS O
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Ensure that the technology you have chosen supports lookups.
Creating Lookups
Make sure that your chosen technology supports lookups.
Three possible levels of Lookup Support: None | Join | Select
Two ways to create lookups from an interface:
File > New to open the Gallery and the Table Lookup Wizard
Click the Lookup Wizard button.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The lookup wizard has two steps. In the first step, you simply select your driving and lookup
tables. In the second step, you define the lookup condition and you can specify several options.
Lookup Wizard
1
2
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
A data set represents the data flow that comes from a group of joined and filtered source data
stores. One target can be loaded with data coming from several data sets. Set-based operators
(Union, Intersect, and son) are used to merge the different data sets into the target data store.
Each data set corresponds to one diagram of source data stores and the mappings used to load
the target data store from these source data stores.
Benefits of using data sets:
- Accelerates the interface design
- Reduces the number of interfaces needed to merge several data flows into the same
target data store
Using Set-Based Operators with Integration
Interfaces
A data set:
Represents the data flow that comes from a group of joined
and filtered source data stores
Includes the target mappings for this group of sources
Note: Several data sets can be merged into the interface target
data store by using set-based operators such as Union and
Intersect.
List of operators: INTERSECT, UNION, UNION ALL, MINUS
Merging several data flows in one target
Mappings and diagrams are per data flow
Benefits:
Faster design
Better performance
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
In this slide, several data sets are shown merged into the interface target data store using set-
based operators such as Union and Intersect.
Note, that the support for data sets as well as the set-based operators supported depend on the
capabilities of the staging areas technology.
The set-based operators are always executed on the staging area.
When designing the integration interface, the mappings for each data set must be consistent. This
means that each data set must have the same number of target columns mapped.
Using Set-Based Operators with Integration
Interfaces
ORDERS1
LINES
MODIF_File1
SALES
UNION
INTERSECT
MINUS
ORDERS2
LINES
ORD_05
CORRECT
MODIF_File1
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Example: Flow with Multiple Data Sets
DataSet1 Lines
DataSet0 - Orders
SS_0 (1): MSSQL_CRM
COUNTRY
CUSTOMER
REGION
ORDERS
Staging: MEMORY_ENGINE
-
-
SS_1 (2): DB2_HSCM
-
CITY
ORDER_LINES
CURRENCY
PRODUCTS
PERSON
C$ (SS0)
C$ (SS1)
EMPLOYEE
- LKM SQL to
Oracle
(LOAD)
LKM DB2 to Oracle
(LOAD)
IKM SQL to SQL
Append
SS_2 (3): DB2_HSCM
-
BOOKINGS
PRODUCTS
DataSet2 - Bookings
C$ (SS0)
EMPLOYEE
LKM DB2 to
Oracle
(LOAD)
UNIONALL
-
-
MINUS
Target: ORACLE_SERVER1
SALES
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
To create a new data set:
1. In the Source Diagram toolbar, click Add /Remove DataSet to display the data sets
Configuration dialog box.
2. Click Add new DataSet. A new line is added for the new data set at the bottom of the list.
3. In the DataSet Name field, give the name of the new data set. This name will be displayed in
the data set tab. In the Operator field, select the set-based operator for your data set. Repeat
steps 2 and 3 if you want to add more data sets and then click Close.
Defining a Data Set
1
2
3
Specify the UNION
operator to unite the new
source of data with the
default data set.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
This slide offers guidelines for using set-based operators successfully.
Make sure that your technology supports set-based operators.
- Check Support Set Operators.
In your interface, add several data sets:
- Save before adding data sets.
- Share the same target structure.
- Order counts for precedence.
Select an IKM that supports data sets.
- Data sets are merged from the staging area.
Using Set-Based Operators: Guidelines
Make sure that your technology supports set-based
operators.
Check Support Set Operators.
In your interface, add several data sets:
Save before adding data sets.
Share the same target structure.
Order counts for precedence.
Select an IKM that supports data sets.
Data sets are merged from the staging area.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Oracle Data Integrator is able to use database-defined partitions when processing data in
partitioned tables used as source or targets of integration interfaces. These partitions are created
in the data store corresponding to the table, either through the reverse-engineering process or
manually. For example with the Oracle technology, partitions are reverse-engineered using the
RKM Oracle.
The partitioning methods supported depend on the technology of the data store. For example, for
the Oracle technology the following partitioning methods are supported: Range, Hash, List.
After they are defined on a data store, partitions can be selected when this data store is used as a
source or a target of an interface.
Using Partitioning with ODI Interfaces
Partitioning features:
Support (sub)partitioning on data stores
Reverse-engineer partition information
Enable partition selection in interfaces
Auto-generate SQL Statements:
SELECT * FROM MY_TABLE PARTITION
(MY_PARTITION)
Defined by technology
Partitioning Support: No | Partitions | Sub-Partition
Partition Mask: %OBJECT_NAME
PARTITION(%PARTITION)
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Partition information can be reverse-engineered by using Reverse Engineering Knowledge
Modules (RKMs) or can be manually created into models.
Partitioning: Definition in Data Store After
Reverse Engineering
CREATE TABLE trg_customer
( customer_id NUMBER(6)
, cust_first_name VARCHAR2(20)
, cust_last_name VARCHAR2(20)
, cust_address VARCHAR2(20)
, nls_territory VARCHAR2(30)
, cust_email VARCHAR2(30))
PARTITION BY LIST (nls_territory) (
PARTITION asia VALUES ('CHINA', 'THAILAND'),
PARTITION europe VALUES ('GERMANY', 'ITALY', 'SWITZERLAND'),
PARTITION west VALUES ('AMERICA'),
PARTITION east VALUES ('INDIA));
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
In an interfaces mapping, you can specify the use of partitions on a target data store.
Using Partitioning in an Interface
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
You can create an interface that would not have a predefined, permanent target data store. In this
case, all rows in the target are derived from the rows of one or more source data stores. These
interfaces are referred to as temporary interfaces. Such interfaces are designed to be used by
other interfaces as sources for further transformation.
Note: Unlike permanent interfaces, a temporary interfaces are symbolized with yellow interface
symbol.
Using Temporary Interfaces
Two types of ODI Interfaces:
Target data store is permanent (predefined in the model)
Target data store is temporary (defined in the interface)
temporary interface
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
In this example, the temporary interface INT_TEMP_AGG_ORDERS, with its target data store
TEMP_AGG_ORDERS, is used in place of a source data store for the interface
INT_TRG_SALES.
In the Mapping Properties, you can see that the rows of the temporary interface are mapped to the
rows of the target data store.
Temporary interface is
used as a source data
store in the permanent
interface.
Using Temporary Interfaces: Example
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
When using a temporary data store that is the target in temporary interface as a source or as a
lookup table for another interface, you can choose:
To use a persistent temporary data store: You will run a first interface creating and
loading the temporary data store, and then a second interface sourcing from it. In this case,
you would typically sequence the two interfaces in a package.
Not to use a persistent data store: The second interface generates a sub-select
corresponding to the loading of the temporary data store. This option is not always available
because it requires all data stores of the source interface to belong to the same data server
(for example, the source interface must not have any source sets). You activate this option
by selecting Use Temporary Interface as Derived Table on the source.
Note the following when using a temporary interface as derived table:
The generated sub-select syntax can be either a standard sub-select syntax (default
behavior) or the customized syntax from the IKM used in the first interface.
All IKM commands except the one that defines the derived-table statement option Use
current command for Derived Table sub-select statement are ignored. This limitation
causes, for example, the disabling of support for temporary index management.
Derived Select for Temporary Interfaces
In Technology > Definition tab, select Support
Derived Table.
In an interface, if a temporary source is used,
a new check box appears:
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
There are some limitations of using derived select for temporary interfaces. A temporary interface
INT_1 becomes eligible to be a source derived table in another parent interface INT_2, if, and only
if:
All source data stores and subinterfaces of INT_1 are on the same physical server as the
target table of INT_2
The technology of this physical server supports derived tables in the FROM clause
No data store within INT_1 is used as journalized
Derived Select for Temporary Interfaces:
Limitations
A temporary interface INT_1 becomes eligible to be a source
derived table in another interface INT_2, if, and only if:
All source data stores and subinterfaces of INT_1 are on
the same physical server as the target table of INT_1
The technology of this physical server supports derived
tables
No data store within INT_1 is used as journalized
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Answer: e
Explanation: Make sure that your technology supports set-based operators. Check Support Set
Operators. Save before adding data sets. Share the same target structure. Select an IKM that
supports data sets.
Quiz
Which of the following is required for using set-based operators
successfully?
a. Your technology should support set-based operators.
b. Share the same target structure.
c. Select an IKM that supports data sets.
d. Save before adding data sets.
e. All of the above
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Summary
Implement lookups in ODI interfaces
Create ODI interfaces with multiple datasets
Use set-based operators in ODI interfaces
Use partitioning with ODI interfaces
Create ODI temporary interfaces
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Practice 5-1: Implementing ODI Integration with
Temporary Interfaces
In this practice, you create a temporary ODI interface for data
transformation. You also create another interface, which uses
your newly created temporary interface as a source for
transformation.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Using Variables in ODI
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Objectives
Describe variables scope
Use variables in object properties and in the resource
name of a datastore
Use variables within variables
Use variables as startup parameters
Use variables in topology
Perform tracking of variables
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Using Variables: Overview
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Using variables is highly recommended to create reusable packages or packages with a complex
conditional logic, interfaces, and procedures. Variables can be used everywhere within ODI. Their
values can be stored persistently in the ODI Repository if their Keep History parameter is set to All
values or Latest value. Otherwise, if their Keep History parameter is set to No History, their value
will only be kept in the memory of the agent during the execution of the current session.
Using Variables in ODI: Overview
Using variables is highly recommended for:
Creating reusable packages or packages with a complex
conditional logic
Interfaces
Procedures
Variables can be used everywhere within ODI.
Variables values can be stored:
Persistently in the ODI Repository if their Keep History
parameter is set to All values or Latest value
Only in the memory of the agent during the execution of the
current session if their Keep History parameter is set to No
History
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Variable Scope
Use the Expression Editor to refer to your variables in
packages, integration interfaces, and procedures.
When you use the Expression Editor, the variables are
retrieved directly from the repository.
When Expression Editor is not available:
You should manually prefix variable names with GLOBAL or
the PROJECT_CODE
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Referring to a Variable
Referring to variable MY_VAR in your objects should be done as
follows:
#MY_VAR: With this syntax, the variable must be in the
same project as the object referring to it. Its value will be
substituted.
To avoid ambiguity, consider using fully qualified syntax by
prefixing the variable name with the project code
#MY_PROJECT_CODE.MY_VAR: Using this syntax allows
you to use variables by explicitly stating the project that
contains the variable.
Note: It prevents ambiguity when two variables with the
same name exist, for example, at global and project level.
The value of the variable will be substituted at run time.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Referring to a Variable
# GLOBAL.MY_VAR: This syntax allows you to refer to a
global variable. Its value will be substituted in your code.
Note: You reference an ODI variable prefixed with the "#"
character. ODI substitutes the name of the variable by the
value before the code is executed (recommended).
The call of the variable works for OS commands, SQL, and
ODI API calls.
Example: TOWN = '#GLOBAL.VAR_TOWN
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
You can use the variable as a SQL bind variable by prefixing it with a colon rather than a hash.
However, this syntax is subject to restrictions because it applies only to SQL DML statements, not
for OS commands or ODI API calls and using the bind variable may result in performance loss. It
is advised to use ODI variables prefixed with the #character to ensure optimal performance at
run time.
When you reference an ODI Variable prefixed with the : character, the name of the variable is
NOT substituted when the RDBMS engine determines the execution plan. The variable is
substituted when the RDBMS executes the request. This mechanism is called Binding. If using the
binding mechanism, it is not necessary to enclose in delimiters (such as quotation marks) the
variables that store strings because the RDBMS is expecting the same type of data as specified
by the definition of the column for which the variable is used.
For example, if you use the variable TOWN_NAME = :GLOBAL.VAR_TOWN_NAME, the VARCHAR
type is expected.
Note: When you reference an ODI variable prefixed with the #" character, ODI substitutes the
name of the variable by the value before the code is executed by the technology. In expressions,
the variable reference needs to be enclosed in single quotation marks, for example TOWN =
'#GLOBAL.VAR_TOWN'. The call of the variable works for OS commands, SQL, and ODI API
calls.
Referring to a Variable: Using : Instead of #
You can use the variable as a SQL bind variable by prefixing it
with a colon : rather than a hash # (binding).
The variable is substituted when the RDBMS executes the
request (binding).
It is not necessary to enclose in quotation marks the
variables that store strings (RDBMS is expecting the same
type of data as the column for which the variable is used).
Example: TOWN_NAME = :GLOBAL.VAR_TOWN_NAME
Restrictions: This syntax applies only to SQL DML
statements, not for OS commands or ODI API calls.
Note: Using the bind variable may result in performance loss.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Using Variables in ODI Objects
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Variables are often used in packages. The four types of variable steps are as follows:
The Declare Variable step is often placed at the start of a package. It explicitly defines the
variable that is used in an interface, procedure, or other package step. In general, variables
are implicitly declared when they are referred to in a package. However, in certain complex
situations, this cannot be guaranteed. Using this kind of step also improves readability, as
you can visually display all the variables that are used in the package.
The Set Variable step is straightforward and can be used to perform two different actions.
You can assign a specific value to a variable. For example, you can set an error counter to
zero. You can also increment the value of a variable.
The Refresh Variable step updates the variable value by executing the SQL expression of
the variable. Note that variables are not automatically updated.
For example, you have a variable with an expression that retrieves the number of rows in a
table.
The Evaluate Variable step compares the value of the variable against a constant value or
another variable. This comparison is used to create branches and loops . You can use any
mathematical comparison, such as equals, greater than, less than, and so on. You can also
use the SQL membership operator IN. The execution path of the package then splits
according to the result of the comparison. Using the ok and ko tools, you define the true
and false paths.
Using Variables in Packages
The four types of variable steps in packages:
The Declare Variable step is often placed at the start of a
package. It explicitly defines the variable that is used in an
interface, procedure, or other package step.
Set Variable step is straightforward and can be used to
perform two different actions.
You can assign a specific value to a variable.
You can also increment the value of a variable.
The Refresh Variable step updates the variable value by
executing the SQL expression of the variable.
The Evaluate Variable step compares the value of the
variable against a constant value or another variable.
This comparison is used to create branches and loops.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
This slide shows an example of using variable steps in a package to control the workflow. The
Refresh Variable step updates the variable value by executing the SQL expression of the variable.
The Evaluate Variable step compares the value of the variable against a constant value to create
branches.
Using Variables in Packages: Example
Using variables to control the workflow:
The Refresh Variable step
updates the variable value
by executing the SQL
expression of the variable.
The Evaluate Variable
step compares the value
of the variable against a
constant value to create
branches.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Variables can be used in interfaces in two different ways:
As a value for a textual option of a Knowledge Module
In all Oracle Data Integrator expressions such as mappings, filters, joins, and constraints
To substitute the value of the variable into the text of an expression, precede its name by the #
character. The agent or the graphical interface will substitute the value of the variable in the
command before executing it.
The following example shows the use of a global variable named YEAR:
Update CLIENT set LASTDATE = sysdate where DATE_YEAR = '#GLOBAL.YEAR' /*
DATE_YEAR is CHAR type */
Update CLIENT set LASTDATE = sysdate where DATE_YEAR = #GLOBAL.YEAR /*
DATE_YEAR is NUMERIC type */
The bind variable mechanism of the SQL language can also be used. However, this is less
efficient, because the relational database engine does not know the value of the variable when it
constructs the execution plan for the query. To use this mechanism, precede the variable by the :
character, and make sure that the data type being searched is compatible with that of the variable.
For example:
update CLIENT set LASTDATE = sysdate where DATE_YEAR =:GLOBAL.YEAR
Using Variables in Interfaces
Variables can be used in interfaces in two different ways:
As a value for a textual option of a Knowledge Module
In all Oracle Data Integrator expressions such as
mappings, filters, joins, and constraints
To substitute the value of the variable into the text of an
expression, precede its name by the # character:
update CLIENT set LASTDATE = sysdate where DATE_YEAR =
'#GLOBAL.YEAR' /* DATE_YEAR is CHAR type */
You can also use bind variable mechanism (less efficient):
update CLIENT set LASTDATE = sysdate where DATE_YEAR
=:GLOBAL.YEAR
Bind mechanism must be used for Date type variables that
are used in a filter or join expressions:
SRC.END_DATE > :SYSDATE_VAR
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The bind variable mechanism must be used for date type variables that are used in a filter or join
expression. The following example shows a filter:
SRC.END_DATE > :SYSDATE_VAR where the variable SYSDATE_VAR is a date type variable
with the refresh query:
select sysdate into SYSDATE_VAR from dual
If the substitution method is used for a date variable, you need to convert the string into a date
format using the RDBMS-specific conversion function.
Note: You can drag a variable into most expressions with the Expression Editor.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Mapping: '#PRODUCT_PREFIX' || PR.PRODUCT_CODE
Concatenates the current projects product prefix variable with the product code. As the
value of the variable is substituted, you need to enclose the variable with single quotation
marks because it returns a string.
Join: CUS.CUST_ID = #DEMO.UID * 000 + FF.CUST_NO
Multiply the value of the UID variable of the DEMO project by 000 and add the CUST_NO
column before joining it with the CUST_ID column.
Filter: ORDERS.QTY between #MIN_QTY and #MAX_QTY
Filter orders according to the MIN_QTY and MAX_QTY thresholds.
Option Value: TEMP_FILE_NAME: #DEMO.FILE_NAME
Use the FILE_NAME variable as the value for the TEMP_FILE_NAME option.
Using Variables in Interfaces: Examples
Mapping: '#PRODUCT_PREFIX' || PR.PRODUCT_CODE
It concatenates the current projects product prefix variable
with the product code.
Join: CUS.CUST_ID = #DEMO.UID * 000 + FF.CUST_NO
Multiply the value of the UID variable of the DEMO project by
000 and add the CUST_NO column before joining it with the
CUST_ID column.
Filter: ORDERS.QTY between #MIN_QTY and #MAX_QTY
Filter orders according to the MIN_QTY and MAX_QTY
thresholds.
Option value: TEMP_FILE_NAME: #DEMO.FILE_NAME
Use the FILE_NAME variable as the value for the
TEMP_FILE_NAME option.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
It is also possible to use variables as substitution variables in graphical module fields such as
resource names or schema names in the topology. You must use the fully qualified name of the
variable (Example: #GLOBAL.MYTABLENAME) directly in the Oracle Data Integrator graphical
modules field.
Using this method, you can parameterize elements for execution, such as physical names of files
and tables (Resource field in the datastore) or their location (Physical Schema's schema [data] in
the topology), Physical Schema, and Data Server URL.
Using Variables in Object Properties
It is also possible to use variables as substitution variables in
graphical module fields such as resource names or schema
names.
You must use the fully qualified name of the variable
(Example: #GLOBAL.MYTABLENAME) directly in the Oracle
Data Integrator graphical modules field.
Using this method, you can parameterize elements for
execution, such as:
The physical names of files and tables (Resource field in the
datastore) or their location (Physical Schemas schema in
the topology)
Physical Schema
Data Server URL
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
You can use variables anywhere within your procedures code. Code examples are shown in this
slide.
Using Variables in Procedures
You can use variables anywhere within your procedures code.
Examples:
Step 1: SQL
Insert into #DWH.LOG_TABLE_NAME
Values (1, 'Loading Step Started', current_date)
Add a row to a log table that has a name only known at run
time.
Step 2: Jython
f = open('#DWH.LOG_FILE_NAME', 'w')
f.write('Inserted a row in table %s' %
('#DWH.LOG_TABLE_NAME') )
f.close()
Open file defined by the LOG_FILE_NAME variable and write
the name of the log table into which you have inserted a row.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
You should consider using options rather than variables whenever possible in procedures.
Options act like input parameters. Therefore, when executing your procedure in a package you
would set your option values to the appropriate values.
In this example you would write Step 1s code as follows:
Insert into <%=snpRef.getOption(LogTableName)%> Values (1, 'Loading
Step Started', current_date)
Then, when using your procedure as a package step, you would set the value of option
LogTableName to #DWH.LOG_TABLE_NAME.
Note that when using Groovy scripting (discussed in the lesson titled Using ODI Groovy Editor),
you need to enclose the variable name in double quotation marks ("), for example #varname
and #GLOBAL.varname; otherwise, the variables are not substituted with the ODI variable
value.
Using Variables in Procedures
You should consider using options rather than variables
whenever possible in procedures.
Options act like input parameters. When executing your
procedure in a package, you would set your option values to
the appropriate values.
Example: Adding a row to a log table that has a name only
known at run time
Using parameters:
Insert into #DWH.LOG_TABLE_NAME
Values (1, 'Loading Step Started', current_date)
Using options:
Insert into <%=snpRef.getOption(LogTableName)%> Values
(1, 'Loading Step Started', current_date)
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
It is sometimes useful to have variables depend on other variable values as shown in this slide.
Note: The bind variable mechanism must be used to define the refresh query for a date type
variable that references another date type variable. For example:
VAR1 date type variable has the refresh query select sysdate from dual.
VAR_VAR1 date type variable must have the refresh query select :VAR1 from dual.
Using Variables Within Variables
You may have variables that depend on other variable values,
for example:
STORE_ID: Alphanumeric variable. It is passed as a
parameter to a scenario.
Gives the ID of a store
STORE_NAME: Alphanumeric variable
The name of the current store is derived from the Stores
table filtered by the value returned by the concatenation of
the STORE_ID and STORE_CODE variables.
The code:
Select name From
<%=odiRef.getObjectName("L","STORES","D")%> Where
id='#DWH.STORE_ID'||'#DWH.STORE_CODE'
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
You may face some situations where the names of your source or target datastores are dynamic.
A typical example of this is when you need to load flat files into your Data Warehouse with a file
name composed of a prefix and a dynamic suffix such as the current date. For example, the order
file for March 26 would be named ORD2009.03.26.dat.
Note that you can only use variables in the resource name of a datastore in a scenario when the
variable has been previously declared.
Using Variables in the Resource Name of a
Data Store
Sometimes the names of your source or target datastores are
dynamic.
Example: You need to load flat files into your Data
Warehouse (DWH) with a file name composed of a prefix
and a dynamic suffix such as the current date:
ORD2009.03.26.dat
Method of implementation:
Create the FILE_SUFFIX variable in your DWH project and
set its SQL SELECT statement to select current_date (or
any appropriate date transformation to match the actual file
suffix format).
Note: You can only use variables in the resource name of a
datastore in a scenario when the variable has been previously
declared.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
To develop your loading interfaces, you should follow the steps shown in the slide.
Note: The variable in the datastore resource name must be fully qualified with its project code.
When using this mechanism, it is not possible to view the data of your datastore from within
Designer.
Using Variables in the Resource Name of a
Data Store
To develop your loading interfaces, you should follow these
steps:
1. Create the FILE_SUFFIX variable in your DWH project:
ORD2009.03.26.dat
2. Define your ORDERS file datastore in your model and set
its resource name to: ORD#DWH.FILE_SUFFIX.dat
3. Use your file datastore in your interfaces.
4. Design a package as follows:
1. Drag the FILE_SUFFIX variable to the package diagram.
2. Drag all interfaces that use the ORDERS datastore into the
package diagram.
At run time, the source file name will be substituted by the
appropriate value.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
In some integration projects, a flat file needs to be exported into a relational table. In this example:
1. The variable FileName is created and used as a dynamic name of a flat file to be exported
to a relational database table.
2. To reference the flat file dynamically, you edit the source datastore to point to the variable
rather than having the resource file name hardcoded: #FileName.
3. You create an interface to export a flat file to a relational table.
Using a Variable as a Startup Parameter: Example
The variable FileName is
to store the value of
dynamic name of the flat
file to be loaded into the
RDBMS table.
The variable
#FileName is used
a resource
parameter for the
datastore.
Create the
integration interface
for flat file-to-RDBMS
transformation.
1
2
3
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
4. Create a package with the variable and the interface.
5. From the package, you generate a scenario, and then specify that the variable be used as a
startup parameter.
Create a package with the
variable FileName and
the interface. Generate
the scenario.
From the package,
generate the scenario.
The variable is used
as the startup
parameter.
5
4
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
6. When you execute the scenario, the actual name of the flat file to be loaded into the
relational table will be set as a startup parameter.
The name of the file to be
used in the integration
process is defined as a
startup parameter when
you execute the scenario.
6
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Each server and each schema on the server have to be defined in the ODI topology. Now imagine
that you design processes that have to be run on a large number of systems. The exact same
code will be executed; only the physical location will be different. In this case, you have to either
create and maintain thousands of contexts or create an environment that will be more dynamic by
using variables in topology.
It is recommended to create at least two contexts:
A development context where all URLs in topology point to an actual server, not using the
variables. This will ensure that data can be viewed, and interfaces can be tested without any
concerns regarding the variables resolution.
The dynamic context will use the variables in topology to name the servers, port numbers,
usernames for the connections, and so on. The package will assign the appropriate values to
the variable and run the interfaces on the appropriate servers. This context will only be used
to validate that the processes defined in the Development context work properly when you
use the variables.
Independently from ODI, you will need a table to store the values that will be used for the topology
variables.
Using ODI Variables in Topology
If your processes run on a large number (hundreds) of
systems, the exact code will be executed on different
physical locations.
In this case, you have to either:
Create and maintain hundreds of contexts
Or
Create a dynamic environment by using variables in topology
In case of using variables in topology, it is recommended
to create at least two contexts:
Development context where all URL in topology point to an
actual server, not using the variables
Dynamic context that will use the variables in topology to
name the servers, port numbers, usernames for the
connections, and so on
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Define one physical data server and set its JDBC URL, such as shown in the slide.
Define the package for loading data for your store:
The input variable STORE_ID will be used to refresh the values for STORE_URL and
STORE_ACTIVE variables from the StoresLocation table.
If STORE_ACTIVE is set to YES, then the next steps will be triggered. The interfaces refer
to source datastores located according to the value of the STORE_URL variable .
Using ODI Variables in Topology: A Server URL
In some cases, you can refer to a variable in the URL of a
servers definition (rather than using contexts for different
locations):
When the number of sources is high (> 100)
When topology is defined externally in a separate table
For example:
jdbc:oracle:thin:@#EDW.STORE_URL
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
To connect to your stores using a variable in the URL of your data servers definition:
1. Create a StoresLocation table, such as the one shown in the slide.
2. Create three variables in your EDW project:
- STORE_ID: Takes the current store ID as an input parameter
- STORE_URL: Refreshes the current URL for the current store ID with the SELECT
statement: select StoreUrl from StoresLocation where StoreId =
#EDW.STORE_ID
- STORE_ACTIVE: Refreshes the current activity indicator for the current store ID with
the SELECT statement: select IsActive from StoresLocation where
StoreId = #EDW.STORE_ID
Using Variables in a Server URL: Example
1. Create a StoresLocation table as follows:
2. Create three variables in your Enterprise Data Warehouse
(EDW) project:
STORE_ID: Takes the current store ID as an input parameter
STORE_URL: Refreshes the URL for the current store ID with
the SELECT statement: select StoreUrl from
StoresLocation where StoreId = #EDW.STORE_ID
STORE_ACTIVE: Refreshes the current activity indicator for
the current store ID with SELECT statement: select
IsActive from StoresLocation where StoreId =
#EDW.STORE_ID
Store ID Store Name Store URL Is Active
1
Denver 10.21.32.198:1521:ORA1 YES
2
San Francisco 10.21.34.119:1525:SANF NO

124
New York 10.21.34.11:1521:NY YES
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
There are some cases where using contexts for different locations is less appropriate than using
variables in the URL definition of your data servers, for example, when the number of sources is
high (> 100) or when the topology is defined externally in a separate table. In these cases, you
can refer to a variable in the URL of a servers definition.
Suppose you want to load your warehouse from 250 source applicationshosted in Oracle
databasesused within your stores. Of course, one way to do it would be to define one context for
every store. However, doing so would lead to a complex topology that would be difficult to
maintain. Alternatively, you could define a table that references all the physical information to
connect to your stores and use a variable in the URL of your data servers definition.
Using Variables in a Server URL: Example
3. Define one physical data server for all your stores and set
its JDBC URL to:
jdbc:oracle:thin:@#EDW.STORE_URL
4. Define your package for loading data from your store.
The input variable STORE_ID will be used to refresh the
values for STORE_URL and STORE_ACTIVE variables from
the StoresLocation table.
If STORE_ACTIVE is set to YES, then the next steps will be
triggered. The interfaces refer to source datastores located
according to the value of the STORE_URL variable.
To start such a scenario for the New York store, you would
issue the following operating system command:
startscen.bat LOAD_STORE 1 PRODUCTION EDW.STORE_ID=3
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Tracking Variables
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Variable and Sequence tracking will record a history of variables and sequences participating in a
session. The values of all variables will be displayed in the Operator step in the section Variable
and Sequence Values.
With the variable tracking feature, you can also determine whether the variable was used in a
source/target operation or an internal operation such as an Evaluate step.
Tracking Variables
Tracking variables allows to determine:
The actual values of Oracle Data Integrator user variables
that were used during an executed session
Whether the variable was used in a source/target operation
or an internal operation such as an Evaluate step
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Variable tracking is configured at several levels:
When defining a variable, you can select Secure Value if you do not want the variable to be
recorded. This is useful when the variable contains passwords or other sensitive data. If
Secure Value is selected, the variable will never be tracked. It will be displayed as
unresolved in the source or target code, not be tracked in the repository, and it will not be
historized.
When executing or restarting a session, select Log Level 6 in the Execution or Restart
Session dialog box to enable variable tracking. Log level 6 has the same behavior as log
level 5, but with the addition of variable tracking.
When reviewing the execution results in Operator Navigator, you can:
View tracked variables and sequences in the Variables and Sequence Values section of the
Session Step or Session Task Editor.
Review the source/target operations of an execution in the Session Task Editor
Tracking Variables
Variables tracking is configured at several levels:
When defining a variable, you can select Secure Value if
you do not want the variable to be tracked.
Select Log Level 6 in the Execution or Restart Session
dialog box to enable variable tracking.
When reviewing the execution results in Operator Navigator,
you can view tracked variables in the Variables and
Sequence Values section of the Session Step.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
In the Code tab of the Session Task Editor, click Show/Hide Values to display the code with
resolved variable values. Note that only variables in substitution mode (#VARIABLE) can be
displayed with resolved variable values and that if the variable values are shown, the code
becomes read-only.
Tracking variables is useful for debugging purposes. See Section 22.2.3, Handling Failed
Sessions in Oracle Fusion Middleware Developer's Guide for Oracle Data Integrator 11g Release
1 (11.1.1) for more information on how to analyze errors in Operator Navigator and activate
variable tracking.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Variable tracking is available in ODI Studio and ODI Console sessions.
Note the following when tracking variables in Oracle Data Integrator:
Each value taken by a variable in a session can be tracked.
The values of all tracked variables can be displayed at step and task levels. This includes
when a variable is modified by a step or a task. The Step or Task Editor displays the name
and the new value of the variable.
The source and target code for a step or task can be viewed with either resolved variable
and sequence values or hidden variable values that display the variable and sequence
names. Note that if the variable values are shown, the code becomes read-only.
Variables, such as passwords, that are defined as Secure Value are never displayed in the
resolved code or variable list. A secure variable does not persist any value in the repository,
even if it is refreshed. Note also that the refresh of a secure variable does not work across
two sessions.
Tracking Variables: Notes
Note the following when tracking variables in Oracle Data
Integrator:
Each value taken by a variable in a session can be
tracked.
The values of all tracked variables can be displayed at
step and task levels.
Step or Task Editor displays the name and the new value of
the variable.
The source and target code for a step or task can be
viewed with either resolved variable values or hidden
variable values that display the variable names.
Variables, such as passwords, that are defined as Secure
Value are never displayed in the resolved code or variable
list.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Tracking Variables: Notes
When a session is purged, all variable values attached to
that session are purged along with the session.
Bind variables (:VARIABLE_NAME)cannot be tracked;
only substituted variables (#VARIABLE_NAME) can be
tracked.
Tracked values are exported and imported as part of a
session when the session is exported or imported.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Answer: b
Explanation: When you reference an ODI variable prefixed with the #" character, ODI substitutes
the name of the variable by the value before the code is executed by the technology.
Quiz
When you reference an ODI variable prefixed with the #"
character, ODI substitutes the name of the variable by the
value at run time.
a. True
b. False
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Summary
Describe variables scope
Use variables in object properties and in the resource
name of a datastore
Use variables within variables
Use variables as startup parameters
Use variables in topology
Perform tracking of variables
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Practice 6-1 Overview: Using Variables in ODI
Package
This practice covers creating an ODI procedure and a package
with variables to control execution of the workflow with
integration interfaces.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Practice 6-2 Overview: Using an ODI Variable as a
Startup Parameter
This practice covers creating an interface to export a flat file to
a relational table. You create a variable for the name of the flat
file. The actual name of the flat file to be loaded into the
relational table will be defined as a startup parameter at run
time.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Accelerating Development in ODI with Groovy
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Objectives
Describe ODI SDK
Use Groovy editor
Automate ODI development tasks with Groovy
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Interacting Programmatically with ODI
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The ODI SDK provides an interface to enable developers to leverage the ODI concepts through an
SDK instead of using the graphical interface.
ODI can now be embedded in other products that can drive ODI process creation and
execution from their own GUI.
Dynamic Mappings: The structure of the source and/or target systems is very dynamic.
That structure would not be easily implemented through a GUI that leverages fixed metadata
definitions.
An enhanced SDK enables the developer to execute virtually every ODI method through a Java
program that enables developers to leverage ODI concepts through an SDK instead of using ODI
11g Studio.
Note: ODI SDK and Public API are terms that are often used interchangeably. The Public API
mimics the usage of the graphical interface. For a better understanding of the logic implemented in
the SDK, ensure that you are familiar with the GUI. Classes, methods, and parameters will be
similar to the options available through the graphical interfaces. The valid code will match valid
objects in the ODI GUI.
Overview of ODI SDK
SDK enables developers to leverage ODI concepts
programmatically instead of using ODI 11g Studio.
SDK enables developers to execute ODI methods virtually,
through a Java program.
With SDK, ODI can be embedded in other products to
create and execute an ODI process from developers GUI.
SDK enables developers to implement dynamic mappings.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
This slide lists the ODI operations supported by SDK. The list is grouped by operations performed
in the Master Repository and operations performed in the Work Repository.
Master Repository:
- Creation of Master / Work Repositories
- Creation and management of the following objects:
Data servers, agents, contexts
Implementing mapping
Work Repository:
- Creation and management of metadata
Models and submodels
Datastores, columns, constraints
- Creation and management of ODI projects
Projects, Folders, Interfaces, Variables, Packages
- Retrieving values of flexfields
SDK-Supported ODI Operations
Master Repository:
Creation of Master / Work Repositories
Creation and management of the following objects:
Data servers, agents, contexts
Implementing mapping
Work Repository:
Creation and management of metadata
Models and submodels
Datastores, columns, constraints
Creation and management of ODI projects
Projects, Folders, Interfaces, Variables, Packages
Retrieving values of flexfields
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Master Repository
Modification of the data type conversion matrix
Creation of new technologies
Security management (Only users with the Supervisor role can use the SDK.)
Work Repository
Creation of new ODI procedures (though procedures can be imported and used in packages)
Creation of new knowledge modules
Creation of user functions
Locking/Unlocking of objects
Versioning of objects
No ability to duplicate an object in the repository (though this can be achieved through
code)
No markers or memos
Setting the value of flexfields
ODI Operations Not Supported by SDK
Master Repository:
Creation of new technologies
Security management
Work Repository:
Creation of new ODI procedures (though procedures can be
imported and used)
Creation of new knowledge modules
Creation of new user functions
Locking/unlocking of ODI objects
Versioning of ODI objects
Duplication of ODI objects in the repository
Working with markers
Setting values of flexfields
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Public API
- Developers relying on the SDK will be protected from repository evolutions over time.
Sunopsis APIs
- Each tool has a corresponding API version.
Substitution Methods
- They are used by developers to build generic code that will dynamically retrieve the
names of all tables, columns, and mappings during the process of building interfaces.
Combining Different APIs
Public API (or SDK):
Is a library of classes and methods for the manipulation of
objects in the ODI repository (without accessing the
repository tables directly)
Is protected from repository evolutions over time
Sunopsis APIs:
Are represented in the packages as Tools
Are used to send emails, automate the reverse-engineering
of models, invoke scenarios, detect changes, and so on
Can be leveraged by packages built through the public APIs
Substitution Methods:
Are used in knowledge modules and procedures
Enable dynamic retrieval of names (tables, columns),
mappings
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Using ODI Groovy Editor
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Groovy is a scripting language with Java-like syntax for the Java platform. The Groovy scripting
language simplifies the authoring of code by employing dot-separated notation, yet still supporting
syntax to manipulate collections, Strings, and JavaBeans. Groovy language expressions in ADF
Business Components differs from the Java code that you might use in a Business Components
custom Java class because Groovy expressions are executed at run time, whereas the strongly
typed language of Java is executed at compile time. Additionally, because Groovy expressions are
dynamically compiled, they are stored in the XML definition files of the business components
where you use it. ADF Business Components supports the use of the Groovy scripting language in
places where access to entity object and view object attributes is useful, including attribute
validators (for entity objects), attribute default values (for either entity objects or view objects),
transient attribute value calculations (for either entity objects or view objects), bind variable default
values (in view object query statements and view criteria filters), and placeholders for error
messages (in entity object validation rules). Additionally, ADF Business Components provides a
limited set of built-in keywords that can be used in Groovy expressions.
Note: Groovy can be used in ODI Procedures, so you could write a procedure with Groovy script
that does your work and calls the procedure through the regular channels, for example as a
scenario through web services.
Introduction to Groovy
Groovy is a scripting language with Java-like syntax for the
Java platform that:
Simplifies the authoring of code by employing dot-separated
notation
Supports syntax to manipulate collections, Strings, and
JavaBeans
Groovy scripting language expressions are:
Executed at run time (dynamically compiled)
Stored in the XML definition files of the business components
ADF Business Components:
Supports the use of the Groovy scripting language in places
where access to entity object and view object attributes is
useful
Provides a limited set of built-in keywords that can be used in
Groovy expressions
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The Groovy editor provides all standard features of a code editor such as syntax highlighting and
common code editor commands except for debugging. The following commands are supported
and accessed through the context menu or through the Source main menu:
Show Whitespace
Text Edits
- Join Line
- Delete Current Line
- Trim Trailing Whitespace
- Convert Leading Tabs to Spaces
- Convert Leading Spaces to Tabs
- Macro Toggle Recording
- Macro Playback
Indent Block
Unindent Block
Introduction to the Groovy Editor
The Groovy editor provides:
A single environment for creating, editing, and executing
Groovy scripts within the ODI Studio context
All standard features of a code editor such as syntax
highlighting and common code editor commands except for
debugging
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
You can execute one or several Groovy scripts at once and also execute one script several times
in parallel.
You can only execute a script that is opened in the Groovy editor. ODI Studio does not execute a
selection of the script, it executes the whole Groovy script.
To execute a Groovy script in ODI Studio, select the script that you want to execute in the Groovy
editor. Click Execute in the toolbar. The script is executed.
You can now follow the execution in the Log window.
Note that each script execution launches its own Log window. The Log window is named
according to the following format: Running <script_name>.
Executing Script with Groovy Editor
You can execute one or several Groovy scripts at once
and also execute one script several times in parallel.
You can only execute a script that is opened in the Groovy
editor.
ODI Studio does not execute a selection of the script; it
executes the whole Groovy script.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Performing SDK Tasks Using Java with Groovy
Editor
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
In this example, the ODI 11g SDK is used to create a new ODI Project within the Work Repository
that is connected within ODI Studio. A Knowledge Module (KM) is also to be imported into this
new project.
1. Open the native Groovy editor from the ODI Tools menu.
2. Enter the Groovy script in the Groovy console.
3. Verify that the ODI project is created.
Example of Performing an SDK Task Using Java
with Groovy Editor
2
3
1
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Performing Advanced Actions with Groovy
Accessing external libraries (for example, if an external
driver is needed) using custom libraries
Defining additional Groovy execution classpath (in addition
to all classpath entries available to ODI Studio)
Reading input with odiInputStream variable
Using the ODI instance:
ODI provides the variable odiInstance. It represents the
ODI instance, more precisely the connection to the
repository, in which the script is executed.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The Groovy editor is able to access external libraries; for example, if an external driver is needed.
To use external libraries, do one of the following:
Copy the custom libraries to the userlib folder. This folder is located:
On Windows operating systems: %APPDATA%/odi/oracledi/userlib
On UNIX operating systems: ~/.odi/oracledi/userlib
Add the custom libraries to the additional_path.txt file. This file is located in the userlib
folder and has the following content:
; Additional paths file
; You can add here paths to additional libraries
; Examples:
; C:\ java\libs\myjar.jar
; C:\ java\libs\myzip.zip
; C:\java\libs\*.jar will add all jars contained in the C:\java\libs\
directory
; C:\java\libs\**\*.jar will add all jars contained in the
C:\java\libs\ directory and subdirectories
Using Custom Libraries
To use external libraries, do one of the following:
Copy the custom libraries to the userlib folder. This
folder is located at: APPDATA%/odi/oracledi/userlib
Add the custom libraries to the additional_path.txt
file. This file is located in the userlib folder and has the
following content:
; Additional paths file
; You can add here paths to additional libraries
; Examples:
; C:\ java\libs\myjar.jar
; C:\ java\libs\myzip.zip
; C:\java\libs\*.jar will add all jars contained in
the C:\java\libs\ directory
; C:\java\libs\**\*.jar will add all jars contained
in the C:\java\libs\ directory and subdirectories
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
You can define a Groovy execution classpath in addition to all classpath entries available to ODI
Studio.
To define an additional Groovy execution classpath, perform the following steps:
1. Before executing the Groovy script, select Tools > Preferences.
2. In the Preferences dialog box, navigate to the Groovy Preferences page.
3. Enter the classpath and click OK.
Defining Additional Groovy Execution Classpath
1
2
3
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Oracle Data Integrator provides the odiInputStream variable to read input streams. This
variable is used as follows:
odiInputStream.withReader { println (it.readLine())}
When this feature is used, an Input text field is displayed at the bottom of the Log tab. Enter a
string text and press Enter to pass this value to the script. The script is exited after the value is
passed to it.
Read Input with the odiInputStream Variable
ODI provides the odiInputStream variable to read input
streams. This variable is used as follows:
odiInputStream.withReader { println
(it.readLine())}
Enter a string text in the bottom of the Log tab and press
Enter to pass this value to the script. The script is exited
after the value is passed to it.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
This slide provides an example that shows how to create an ODI Project with a Groovy script. In
createProject method, you define parameters for Project Name, Project Code, and Folder
Name of your interface.
Automating Development Tasks: Example 1
Creating an ODI project:
import
oracle.odi.core.persistence.transaction.support.DefaultTransactionDe
finition;
import oracle.odi.domain.project.OdiProject;
import oracle.odi.domain.project.OdiFolder;
def createProject(projectName, projectCode, folderName) {
txnDef = new DefaultTransactionDefinition();
tm = odiInstance.getTransactionManager()
txnStatus = tm.getTransaction(txnDef)
project = new OdiProject(projectName, projectCode)
folder = new OdiFolder(project, folderName)
odiInstance.getTransactionalEntityManager().persist
(project)
tm.commit(txnStatus)
}
createProject("DATA WAREHOUSING", "EDW", "INITIALIZATION")
In the
createProject
method, you
define the
parameters for
Project Name,
Project Code,
and Folder
Name.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
This slide provides an example of how to create an ODI model with the necessary topology using
the Groovy script.
To create the topology, enter:
lschema = createLogicalSchema("<Context>", "<Technology>", "<Logical
Schema Name>", "<Data Server Name>", "<Database User Name>",
ObfuscatedString.obfuscate("<Database User Password>"), "<URL>",
"<Driver>", "<DB Schema used for ODI Physical Schema>")
In this example, the following parameters are used:
lschema = createLogicalSchema("GLOBAL", "ORACLE", "ORACLE_EBS",
"ORACLE_HQ_DEV", "ODI", ObfuscatedString.obfuscate("ODI"),
"jdbc:oracle:thin:@localhost:1521:orcl", "oracle.jdbc.OracleDriver",
"ODI")
To create a model, enter:
createModel(lschema, "<Context>", "<Model Name>", "<Model Code>")
In this example, you use the following parameters:
createModel(lschema, "GLOBAL", "ORACLE_WH", "ORACLE_WH")
Creating an ODI topology and a model:
lschema = createLogicalSchema("<Context>", "<Technology>",
"<Logical Schema Name>", "<Data Server Name>", "<Database User Name>",
ObfuscatedString.obfuscate("<Database User Password>"),
"<Database Connection URL>", "<JDBC Driver>", "<Database Schema used for ODI
Physical Schema>")
createModel(lschema, "<Context>", "<MOdel Name>", "<Model Code>")
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Note the following methods and their parameters used to create the model and the topology:
createLogicalSchema:
contextCode: The ODI code for the context used to map the logical schema to the
physical
technologyCode: The ODI code for the technology
nameForLogicalSchema: The name for the logical schema to create
NameForDataserver: The name for the data server to create
userNameForAuthentication: The username for the connection to the data server
passwordForAuthentication: The password for the connection to the data server
urlForAuthentication: The URL for the connection to the data server
driverForAuthentication: The JDBC driver for the connection to the data server
schemaForAuthentication: The schema to use for the ODI physical schema
createModel:
logicalSchemaObject: The ODI logical schema object (instance of
ODILogicalSchema)
contextCode: The ODI context code for reverse engineering
nameForModel: The name for the model to create
codeForModel: The code for the model to create
Note: With these two methods or variations of them, you can easily construct your topology
objects and models that will allow going straight to reverse engineering when the script has been
run.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
This slide provides an example that shows how to create an ODI interface with a Groovy script.
To create an interface with Groovy, create a text file that contains a string (or strings) with the
following parameters separated by commas: interface name, source model, source datastore,
target model ,and target datastore separated by commas .
In the Groovy script, define the variables with the values for the ODI project, the project folder, and
the path to the text file you created.
Run the Groovy script to create the ODI interface.
Note: You can create more that one interface with your Groovy script if you define more than one
row in the text file (one line of parameters for each interface).
Creating an ODI interface:
<Interface Name>,<Source Model>,<Source Datastore>,<Target Model>,<Target
Datastore>
Create a text file
with interface name,
source model and
datastore, and
target model and
datastore separated
by commas.
1
2
In the Groovy script
you define the
variables for the ODI
project, the project
folder, and the path
to the text file you
created.
3
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
You can automate creating elements of ODI Studio user interface using Java with the Groovy
editor. This slide provides an example that shows an implementation of ODI Studio UI automation.
Studio UI Automation with Java:
import javax.swing.JMenuItem;
import javax.swing.JMenu;
import oracle.ide.Ide;
((JMenuItem)Ide.getMenubar().getGUI(false).getCompo
nent(4)).doClick();
JMenu mnu =
((JMenu)Ide.getMenubar().getGUI(false).getComponent
(4));
((JMenuItem)mnu.getMenuComponents()[0]).doClick()
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Answer: c and e
Explanation: Creation of user functions and knowledge modules are not supported by SDK.
Quiz
Which operations are not supported by SDK?
a. Creation of master/work repositories
b. Creation of models and submodels
c. Creation of user functions
d. Creation of packages
e. Creation of knowledge modules
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Summary
Describe ODI SDK
Use Groovy editor
Automate ODI development tasks with Groovy
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Practice 7-1 Overview: Automating ODI Tasks
with Groovy
This practice covers creating the ODI project, the topology and
model, and the ODI interface with Groovy.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Working with Complex Files in ODI
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Objectives
Describe complex files concept
Describe the technical requirements for using complex files
with ODI
Set the topology for complex files and create a complex
files model
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Complex Files Concept: Overview
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The ODI supports several files types:
Flat files (fixed or delimited): A File server corresponds to an Oracle Data Integrator data
server. In this File server, a directory containing files corresponds to a physical schema. A
group of flat files within a directory corresponds to an Oracle Data Integrator model, in which
each file corresponds to a datastore. The fields in the files correspond to the datastore
columns. Oracle Data Integrator provides the knowledge modules (KM) listed in this slide for
handling File data using the File driver.
XML files: An XML file corresponds to a data server in Oracle Data Integrator. Within this
data server, a single schema maps the content of the XML file. The Oracle Data Integrator
Driver for XML (XML driver) loads the hierarchical structure of the XML file into a relational
schema. ODI provides the IKM XML Control Append for handling XML data.
Complex files (files with native format): For these files, you have to read or write file
structures that are too complex to be parsed using the existing ODI File technology.
Types of Files in ODI
ODI supports the following types of files:
Fixed or delimited files (flat files) containing ASCII or
EBCDIC data (use File technology)
KMs to read from such files:
LKM File to SQL: Loads data from an ASCII or EBCDIC File to
any ANSI SQL-92compliant database used as a staging area
IKM SQL to File Append: Integrates data in a target file from any
ANSI SQL-92compliant staging area in replace mode
RKM File (FROM EXCEL): Retrieves metadata from Excel
XML files (use XML technology)
ODI XML driver loads the structure of the XML file into a
relational schema.
IKM XML Control Append KM is used for handling XML data.
Complex (or native format) files, that are too complex to be
parsed using the existing ODI File technology
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
For complex files it is possible to build a Native Schema description file that describes the file
structure. Using this native schema (nXSD) description and the Oracle Data Integrator Driver for
Complex Files, Oracle Data Integrator is able to reverse-engineer, read, and write information from
complex files. Oracle Data Integrator Driver for Complex Files (Complex File driver) converts
native format to a relational structure and exposes this relational structure as a data model in
Oracle Data Integrator.
The Complex File driver translates internally the native file into an XML (internal) structure, as
defined in the native schema (nXSD) description and from this XML file it generates a relational
schema that is used by Oracle Data Integrator. The overall mechanism is shown in this slide.
Most concepts and processes that are used for Complex Files are equivalent to those used for
XML files. The main difference is the step that transparently translates the Native File into an XML
structure that is used internally by the driver but never persisted.
A Complex File corresponds to an Oracle Data Integrator data server. Within this data server, a
single schema maps the content of the complex file.
The Oracle Data Integrator Driver for Complex File (Complex File driver) loads the complex
structure of the native file into a relational schema. This relational schema is a set of tables
located in the schema that can be queried or modified using SQL. The Complex File driver is also
able to unload the relational schema back into the complex file. The relational schema is
Complex Files Concept
For complex files, it is possible to build a Native Schema
description file that describes the file structure.
Using this native schema (nXSD) description and the
Oracle Data Integrator Driver for Complex Files, Oracle
Data Integrator is able to reverse-engineer, read, and write
information from complex files.
Native File
XML
(Internal)
Relational
Schema
Oracle Data
Integrator
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
reverse-engineered as a data model in ODI, with tables, columns, and constraints. This model is
used like a standard relational data model in ODI. If the modified data within the relational schema
needs to be written back to the complex file, the driver provides the capability to synchronize the
relational schema into the file.
Note: For simple flat files formats (fixed and delimited), File technology is recommended, and for
XML files, the XML technology.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
For more information on using KMs, see the lesson titled Generic SQL in Oracle Fusion
Middleware Connectivity and Knowledge Modules Guide for Oracle Data Integrator 11g Release 1
(11.1.1).
Knowledge Modules for Complex Files
You can use a Complex File data server as any SQL data
server.
Complex File data servers support both technology-
specific KMs sourcing or targeting SQL data servers, as
well as generic KMs.
You can also use the IKM XML Control Append when
writing to a Complex File data server.
This Knowledge Module implements specific option to
synchronize the data from the relational schema to the file,
which is supported by the Complex File driver.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Requirements for Working with Complex Files
There are no technology-specific requirements for using
Complex files in Oracle Data Integrator.
Connectivity requirements:
Complex files are accessed through the Oracle Data
Integrator Driver for Complex File. This JDBC driver is
installed with Oracle Data Integrator.
You must have the following connection information:
The location of the native schema (nXSD) file associated with your
native file
The location of the native complex file
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Complex Files: Configuring the Topology
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
You set up the topology for Complex files by creating a Data server and physical schema. Create
a data server for the Complex File technology using the standard procedure, as described in
"Creating a Data Server of the Oracle Fusion Middleware Developer's Guide for Oracle Data
Integrator 11gR1. This slide details only the fields required or specific for defining a Complex File
data server:
In the Definition tab:
Name: Name of the data server that will appear in Oracle Data Integrator
User/Password: These fields are not used for Complex File data servers
In the JDBC tab, enter the following values:
JDBC Driver: oracle.odi.jdbc.driver.file.complex.ComplexFileDriver
JDBC URL: jdbc:snps:complexfile?f=<native file location>&d=<native
schema>&re=<root element name>&s=<schema
name>[&<property>=<value>...]
Note: Creating a Complex File physical schema is standard procedure for any technology.
Setting Up the Topology
Setting up the topology comprises:
Creating a Complex File Data server, which corresponds
to one native file that is accessible to Oracle Data
Integrator
You create a data server for the Complex File technology
using the standard procedure.
The fields specific for defining a Complex File data server:
JDBC Driver:
oracle.odi.jdbc.driver.file.complex.ComplexFile
Driver
JDBC URL:
jdbc:snps:complexfile?f=<native file
location>&d=<native schema>&re=<root element
name>&s=<schema name>[&<property>=<value>...]
Creating a Complex File physical schema
Standard procedure for any technology
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The following are the key properties of the ODI Driver for Complex files:
f <native file name>: This is native file location. Use slash "/" in the path name instead
of back slash "\". It is possible to use an HTTP, FTP, or File URL to locate the file. Files
located by URL are read-only. This parameter is mandatory.
d <native schema>: This is the native schema (nXSD) file location. This parameter is
mandatory.
re <root element>: The element to take as the root table of the schema. This value is
case-sensitive. This property can be used for reverse-engineering, for example, a specific
section of the Native Schema. This parameter is mandatory.
s <schema name>: The name of the relational schema where the complex file will be
loaded. This parameter is mandatory. This schema will be selected when creating the
physical schema under the Complex File data server.
For the full set of ODI driver properties specified in the JDBC URL, refer to Appendix C
Oracle Data Integrator Driver for Complex Files Reference in Oracle Fusion Middleware
Connectivity and Knowledge Modules Guide for Oracle Data Integrator 11g Release 1
(11.1.1).
JDBC URL Properties
The key properties of the ODI Driver for Complex Files
(specified in the JDBC URL):
Property Value Notes
f
<native file
name>
Native file location. Use slash "/" in the
path name. It is possible to use HTTP,
FTP, or file URL to locate the file.
d
<native
schema>
Native schema (nXSD) file location.
This parameter is mandatory.
re <root element>
Name of the element to take as the
root table of the schema. This value is
case-sensitive. This parameter is
mandatory.
s <schema name>
Name of the relational schema where
the complex file will be loaded
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
JDBC URL: Example
The following example illustrates these URL properties:
JDBC URL:
jdbc:snps:complexfile?f=/infiles/PROD20100125_
001.csv&d=/infiles/products.nxsd&re=root&s=PRO
DUCTS
Connects to the PROD20100125_001.csv file described
by products.nxsd and expose this file as a relational
structure in the PRODUCT schema
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
This slide shows how to create a data server to implement Complex File technology. In ODI
Studio:
1. Open the Topology tab. In the Physical Architecture, expand the Complex File technology
node. Right-click Complex File and then select New Data Server.
2. Enter the name for this new data server, for example: PURCHASE_SAMPLE_CPLX_FILE,
and click the JDBC tab.
Complex File Data Server Definition: Example
1
2
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The Complex File technology uses a JDBC driver to read the original input file as well as the
metadata definition of this complex file generated using the Native Format Builder, a SOA
component. To select the driver:
3. Click the magnifying glass icon and then select the JDBC driver:
oracle.odi.jdbc.driver.file.complex.ComplexFileDriver.
4. Edit the URL to point to the input file and to the xsd file:
jdbc:snps:complexfile?f=C:\Labs\Files\Complex
Files\Purchase_sample.txt&d=C:\Labs\Files\Complex
Files\Purchase_schema.xsd&re=invoice
5. Test the connection.
Note: The native schema (nXSD) provided in the data server URL is used as the XSD file to
describe the XML structure. For more information, see Section 5.5.2 Reverse-Engineering an
XML Model in Oracle Fusion Middleware Connectivity and Knowledge Modules Guide for Oracle
Data Integrator 11g Release 1 (11.1.1).
Complex File Data Server Definition: Example
3
4
5
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
For Complex files technology, you define physical and logical schemas the same way as you did
for other technologies:
1. Create a new physical schema for this data server. Save physical schema.
2. Open Logical Architecture tab. Expand Technologies > Complex File and create a new
logical schema as shown in the slide. Connect this logical schema to the physical schema (in
this example PURCHASE_SAMPLE_CPLX_FILE. GEO) in all contexts. Save this logical
schema.
Definition of Physical and Logical Schema:
Example
1
2
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Complex Files in Integration Project
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
For setting up a project using the Complex File technology, you follow the standard procedure.
See Creating an Integration project" of Oracle Fusion Middleware Developer's Guide for Oracle
Data Integrator 11gR1.
It is recommended to import the following knowledge modules into your project for getting started:
LKM SQL to SQL
LKM File to SQL
IKM XML Control Append
A Complex File model groups a set of datastores. Each datastore typically represents an element
in the intermediate XML file generated from the native file using the native schema.
Complex File technology supports standard reverse-engineering, which uses only the abilities of
the Complex File driver (the same process as for XML Files).
Setting Up an Integration Project and Creating a
Complex File Model
Setting up a project using the Complex File technology
involves the standard procedure.
It is recommended to import the following knowledge
modules into your project for getting started:
LKM SQL to SQL
LKM File to SQL
IKM XML Control Append
A Complex File model groups a set of datastores. Each
datastore typically represents an element in the
intermediate XML file generated from the native file using
the native schema.
Complex File technology supports standard reverse-
engineering, which uses only the abilities of the Complex
File driver (the same process as for XML Files).
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
To perform a Standard Reverse- Engineering with a Complex File model use the usual procedure,
as described in Reverse-engineering a Model in the Oracle Fusion Middleware Developer's
Guide for Oracle Data Integrator 11gR1.
Setting Up an Integration project and Creating a
Complex File Model: Example
Follow the standard procedure:
1
2
3
4
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Designing an Interface with Complex File Model
You can use a complex file as a source or a target of an
integration interface.
Complex File data models are handled in interfaces
similarly to XML structures.
The same knowledge modules can be used for both
Complex files and XML technologies.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Note
BPEL is included in the ODI Suite, but not in the stand-alone ODI Enterprise Edition.
Creating an nXSD File
Complex file models depend on the nXSD schema for the
given file.
An nXSD file has to be created using a text editor or the
Native Format Builder Wizard that is part of Oracle BPEL
Process Manager.
This format extends the XML Schema standard by adding
additional parsing instructions to each element.
Using nXSD parsing technology, the native file is converted
into an internal XML format.
The nXSD format extends the standard XML Schema
Definition (XSD) format through nxsd attributes.
nXSD is a valid XML schema, because the XSD standard
allows extra attributes with their own namespaces.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Example of nXSD schema:
<?xml version="1.0"?>
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:nxsd="http://xmlns.oracle.com/pcbpel/nxsd"
elementFormDefault="qualified"
xmlns:tns="http://xmlns.oracle.com/pcbpel/demoSchema/csv"
targetNamespace="http://xmlns.oracle.com/pcbpel/demoSchema/csv"
attributeFormDefault="unqualified"
nxsd:encoding="US-ASCII" nxsd:stream="chars" nxsd:version="NXSD">
<xsd:element name="Root">
<xsd:complexType><xsd:sequence>
<xsd:element name="Header">
<xsd:element name="Branch" type="xsd:string"
nxsd:style="terminated" nxsd:terminatedBy=","/>
<xsd:element name="ListDate" type="xsd:string"
nxsd:style="terminated" nxsd:terminatedBy="${eol}"/>
Example of nXSD Schema
<?xml version="1.0"?>
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:nxsd="http://xmlns.oracle.com/pcbpel/nxsd"
elementFormDefault="qualified"
xmlns:tns="http://xmlns.oracle.com/pcbpel/demoSchema/csv"
targetNamespace="http://xmlns.oracle.com/pcbpel/demoSchema/csv"
attributeFormDefault="unqualified"
nxsd:encoding="US-ASCII" nxsd:stream="chars" nxsd:version="NXSD">
<xsd:element name="Root">
<xsd:element name="Header">
<xsd:element name="Branch" type="xsd:string"
<xsd:element name="ListDate" type="xsd:string"
nxsd:style="terminated" nxsd:terminatedBy="${eol}"/>
</xsd:sequence></xsd:complexType>
</xsd:element>
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
</xsd:element>
<xsd:element name="Customer" maxOccurs="unbounded">
<xsd:element name="Name" type="xsd:string"
<xsd:element name="Street" type="xsd:string"
nxsd:style="terminated" nxsd:terminatedBy="," />
<xsd:element name="City" type="xsd:string"
nxsd:style="terminated" nxsd:terminatedBy="${eol}" />
</xsd:element>
</xsd:element>
</xsd:schema>
Example of nXSD Schema
<xsd:element name="Customer" maxOccurs="unbounded">
<xsd:element name="Name" type="xsd:string"
<xsd:element name="Street" type="xsd:string"
nxsd:style="terminated" nxsd:terminatedBy="," />
<xsd:element name="City" type="xsd:string"
nxsd:style="terminated" nxsd:terminatedBy="${eol}" />
</xsd:element>
</xsd:element>
</xsd:schema>
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Note: The nXSD schema annotates elements to describe their position and delimiters within the
flat text file. The schema above uses almost exclusively the nxsd:terminatedBy instruction to
look for the next terminator characters. There are various constructs in nXSD to parse fixed-length
fields, look ahead in the document for string occurrences, perform conditional logic, use variables
to remember state, and many more.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The nXSD schema is created using the Native Format Builder Wizard in JDeveloper with the SOA
Extensions installed. The Oracle File and FTP Adapters are automatically integrated with Oracle
BPEL PM. When you drag File Adapter for FTP Adapter from the Component Palette of the
JDeveloper BPEL Designer to the design area, the Adapter Configuration Wizard starts with a
Welcome page.
When you click the Define Schema for Native Format button in the Messages page of the Adapter
Configuration Wizard, the Native Format Builder Wizard is displayed. The Messages page is the
last page that is displayed in the Adapter Configuration Wizard before the Finish page. For details,
refer to Oracle Fusion Middleware User's Guide for Technology Adapters 11g Release 1
(11.1.1.6.3).
Using Native Format Builder Wizard
Both Native Format Builder Wizard as well as the nXSD
language are described in the Oracle Fusion Middleware
User's Guide for Technology Adapters 11g Release 1
(11.1.1.6)
The way to start the Native Format Builder in BPEL is to
create a new File Adapter.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
You can create new nXSD schema for your complex file, or edit an existing native schema
generated using the Native Format Builder Wizard by sampling a delimited, fixed-length, or
complex-type file. To edit an existing native schema, select the Edit existing option in the Choose
Type page of the Native Format Builder Wizard, and click Browse to navigate to the location of the
existing schema file and then select the native schema file that must be edited. The Native Format
Builder Wizard guides you through the editing of the native schema file.
Before you edit a native schema file, you must ensure that the sample file specified in the
annotation within the schema exists. This annotation is automatically added when the native
schema is generated the first time from the sample file.
Note: If the format is complex, it is often a good idea to approximate it with a similar simple
format and then add the complex components manually. The resulting *.xsd file can be copied
and used as the format for ODI. Using this technique it is also possible to parse the same file
format in SOA Suite and ODI; for example, using SOA for small real-time messages, and ODI for
large batches.
You can create a new
schema using the Native
Format Builder Wizard for
a complex-type file.
To edit an existing native
schema. select the Edit
existing option.
Browse to navigate to the
location of the existing
schema file and then
select the native schema
file that must be edited.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
This slide shows the nXSD schema for a complex file generated by Native Format Builder Wizard.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Answer: b
Explanation: Complex File data servers support both technology-specific KMs sourcing or
targeting SQL data servers, as well as generic KMs.
Quiz
Complex File data servers support only technology-specific
KMs sourcing or targeting SQL data servers.
a. True
b. False
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Summary
Describe complex files concept
Describe the technical requirements for using complex files
with ODI
Set the topology and create a complex files model
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Practice 8-1 Overview: Configuring ODI Topology
and ODI Model with Complex Files
This practice covers exploring the Complex File technology in
ODI.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Integration of ODI in Enterprise Environment
and SOA
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Objectives
Describe the features of Java EE and Standalone Agents
Use different types of web services with ODI
Implement the integration of ODI with Service-Oriented
Architecture (SOA)
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
ODI Integration with Java EE
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
There are Standalone Agents and Standalone Agents in ODI 11g. Both types of agents are
multithreaded Java programs and can be configured for high availability. The main difference is
where and how you install them (in WebLogic Server or on top of a Java Virtual Machine [JVM])
and the benefits from this installation
The Standalone Agent is easier to deploy anywhere, but does not have clustering or connection
pooling. It can be monitored by Oracle Process Manager and Notification Server (OPMN).
The Java EE Agent is slightly more complex to set up (you need to install WLS first, set up a
domain, and so forth), but more preferable in terms of enterprise-scale deployment: clustering,
load-balancing, centralized monitoring (with Fusion Middleware Console) and so forth.
Note that the choice between the two types of agents is really a users choice, and it is easy to mix
both these types of agents seamlessly into an ODI architecture.
Java EE Agents require WebLogic Server whereas Standalone Agents run in their own JVM
container (no application server is required for Standalone Agents).
Understanding ODI Java EE Agent and
Standalone Agent
Both Java EE and Standalone Agents are multithreaded Java
programs and can be configured for high availability.
Standalone Agent:
Easier to deploy
Has built-in ODI Load Balancing feature
Does not have clustering or connection pooling
Can be monitored by Oracle Process Manager and
Notification Server (OPMN)
Java EE Agent:
Slightly more complex to set up (needs WLS domain)
Preferable for enterprise-scale deployment: clustering, load-
balancing, centralized monitoring (with Fusion Middleware
Console)
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The Runtime Agent can be deployed as a Java EE component within an application server. It
benefits in this configuration from the application server layer features such as clustering and
connection pooling for large configurations. This Java EE Agent exposes an MBeans interface,
enabling life-cycle operations (start/stop) from the application server console and metrics that can
be used by the application server console to monitor the agent activity and health.
Oracle WebLogic Server Integration
- Oracle Data Integrator components integrate seamlessly with the Java EE application
server.
Java EE Agent Template Generation
- Oracle Data Integrator provides a wizard to automatically generate templates for
deploying Java EE Agents in Oracle WebLogic Server. Such a template includes the
Java EE Agent and its configuration, and can optionally include the JDBC data sources
definitions required for this agent, as well as the drivers and libraries files for these data
sources to work.
By using the Oracle WebLogic Configuration Wizard, domain administrators can extend their
domains or create a new domain for the Oracle Data Integrator Java EE Runtime Agents.
Java EE Agent Deployment Features
The deployment features:
Oracle WebLogic Server Integration
Java EE Agent Template Generation
Oracle WebLogic Configuration Wizard
Automatic Data Source Creation for WebLogic Server
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Oracle Data Integrator provides an extension integrated into the Fusion Middleware Control
Console (Enterprise Manager). The Oracle Data Integrator components can be monitored as a
domain through this console, and administrators can have a global view of these components
along with other Fusion Middleware components from a single administration console. To
implement integration with Enterprise Manager:
ODI Java EE Agent must be deployed and configured with the existing WebLogic server
domain
Enterprise Manager and the ODI Enterprise Manager Plug-in must be deployed in the
WebLogic server domain that has the ODI Java EE Agent deployed and configured
Integration of ODI with Fusion Middleware Control
Standalone Agent
Java EE Agent
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The ODI Console provides web access to ODI repositories. It enables an ODI developer to browse
ODI objects (such as projects, models, logs, and so on) and manage the ODI environment through
the web service.
Business users, developers, operators, and administrators use their web browsers to access the
ODI Console. The ODI Console replaces the Metadata Navigator of ODI releases before ODI 11g.
Note that with the ODI Console, you also can perform executions.
Using ODI Console
Development
Agent
ODI
Console
Web access
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
To start the Oracle Data Integrator Console, in the browser enter:
http://<hostname>:Port/odiconsole. For example:
http://localhost:8002/odiconsole. To log in, enter SUPERVISOR for user ID, enter
SUNOPSIS for password. Browse ODI objects in the tree view, select the node, and then click the
View icon. The object or group of objects will be displayed in the tab.
Using ODI Console: Example
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Overview of Using Web Services with ODI
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
ODI Run-Time Web Services and Data Services are two different types of web services.
ODI Run-Time Web Services:
The Public Web Service connects to the repository to retrieve a list of contexts and
scenarios. This web service is deployed in a Java EE application server.
The Agent Web Service is built in the Java EE or Standalone Agent.
Types of Web Services
Data Services: Are generated by ODI to give
you access to your data through web services
Run-time web services:
Public Web Services: Access the ODI features through Web
services deployed in a Java EE application server
Agent Web Services:
Commands the Oracle Data Integrator Agent to start and
monitor a scenario, and to restart a session
Built in the Java EE or Standalone Agent
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Data services are specialized web services that provide access to data in datastores, and to the
captured changes in these datastores by using the CDC feature. These web services are
automatically generated by ODI and deployed to a web services containertypically, a Java
application server.
Overview of Data Services
Data services are specialized web services.
They provide access to:
Data in datastores
Captured changes in these datastores by using the Changed
Data Capture (CDC) feature
They are automatically generated by ODI and deployed to
a web services container.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
ODI automatically generates the Data Services from a datastore or a model and deploys to a web
services container in an application server.
This generation can be customized by using Service Knowledge Module (SKM). The resulting
data service is presented in the form of a Java package. You can compile and deploy these Java
classes to a web services container.
Data services can be generated and deployed into a web service stack implementing the Java API
for XML Web Services (JAX-WS), such as Oracle WebLogic Server.
Note: For more information about how to set up, generate, and deploy Data Services, refer to
Oracle Fusion Middleware Developer's Guide for Oracle Data Integrator 11g.
Generation of Data Services
Data services are automatically generated by ODI.
You can generate the Java classes that expose the
datastores data as a data service from a datastore or a
model, by using a Service Knowledge Module (SKM).
You can automatically compile and deploy these Java
classes to a web services container.
Web services
container
J2EE
Java classes
SKM
Datastore
Oracle DI
Generation Data Service Data Service
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
ODI Public Web Services provide operations such as starting a scenario.
A secured web service can be deployed in any web services container, typically a Java 2 Platform,
Enterprise Edition (J2EE) application server or ODI Standalone Agent.
Overview of Public Web Services
ODI Public Web Services are web services that enable
users to leverage ODI features in a Service-Oriented
Architecture (SOA).
A Public Web Service enables applications to trigger
transformation developed with ODI through a secured web
service. This web service:
Can be deployed in any web services container
Can be enabled with ODI Standalone Agent
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Note that the agent must be accessible from the web services container host, and it must have
access to the repositories.
The involved parameters are similar to the ones used when executing a scenario from an OS
command. The required parameters depend on the way you invoke the web service.
Using Public Web Service OdiInvoke
To execute a scenario by using a public Web service:
You can run a scenario from a Web service by using the
ODI Public Web Service OdiInvoke.
The OdiInvoke web service commands an agent to
connect to a given Work repository, and start a specific
scenario.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Public Web Services can be deployed in any web services container installed in your machine,
such as Java application server. Web services can also be provided by ODI Standalone Agent or
as an archive file.
Installing Public Web Services
Public Web Services can be:
Deployed in any web services container (typically, a Java
application server)
Provided by ODI Standalone Agent
Provided as an archive file (.aar)
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Invoking Web Services
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The response of the web service request is written and saved in an XML file that can be used in
ODI.
OdiInvokeWebService Tool
The OdiInvokeWebService tool can invoke any third-party
web service.
It can be used in the tool step of a package or in a
knowledge module.
It can save the response in an XML file that can be
processed with ODI.
The OdiInvokeWebService tool provides an Advanced
editor for generating its code. With this editor, you can:
Connect to the WSDL
Specify parameters for the tool
Select a specific operation and specify request parameters
Invoke a web service
View the web service response in the SOAP editor
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Web service
Operation
OdiInvokeWebService
Requests/invokes
Response
SOAP
SOAP
The OdiInvokeWebService tool
sends a client request to the
web service through HTTP or
HTTPS protocols.
1
The response is written
to a SOAP file.
3
OdiInvokeWebService in action
XML
The response is written to an XML file that
can be processed with ODI.
4
Port
OdiInvokeWebService invokes a specific
operation on a port of the web service.
The Web Services Description Language
(WSDL) file URL must be provided.
2
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Another way to invoke web services is by using the OdiInvokeWebService tool to create a tool
step in a package.
To create an OdiInvokeWebService tool step, perform the following steps:
1. Open the package where you want to create a tool step and click the Diagramtab.
2. From the toolbox, select the OdiInvokeWebService tool. Click the diagrama step
corresponding to your tool appears.
3. Click Free choice to be able to edit the step, then click the step icon in the diagram to open
the Properties panel.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The next section of this lesson explores ways to integrate ODI within a Service-Oriented
Architecture (SOA).
Integration of ODI with SOA
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Oracle Data Integrator can be integrated seamlessly in a Service-Oriented Architecture (SOA) in
several ways:
Data services that provide access to data stored in database tables. Coupled with the
Changed Data Capture (CDC) capability, Data Services can also provide access to the
changed records for a given subscriber. Data services are automatically generated by Oracle
Data Integrator and deployed as web services to a web container, usually a Java application
server.
Oracle Data Integrator can also expose its transformation processes as web services to
enable applications to use them as integration services. For example, a LOAD_SALES batch
process used to update the CRM application can be triggered as a web service from any
service-compliant application, such as Oracle BPEL, Oracle Enterprise Service Bus (OSB),
or Oracle Business Activity Monitoring (BAM). Transformations developed by using ODI can
therefore participate in the broader Service-Oriented Architecture initiative.
Third-party web services can be invoked as part of an ODI workflow and used as part of the
data integration processes. Requests are generated immediately and responses processed
through regular transformations. Suppose, for example, that your company subscribed to a
third-party service that exposes daily currency exchange rates as a web service.
If you want this data to update your multiple-currency data warehouse, ODI automates this task
with a minimum of effort. You would invoke the Web service from your data warehouse workflow
and perform any appropriate transformation on the incoming data.
ODI with SOA Integration Scenarios
Using Data Services to access data in the database tables
Generated by ODI and deployed as web services
Exposing ODI processes as web services
Triggered as web services from any service-compliant
application (Oracle BPEL, Oracle OSB, Oracle BAM)
Invoking third-party web services (for example, a BPEL
process) as part of ODI workflow and using them as part of
data integration processes
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
This slide shows a simple example with the Data Services, Run-Time Web services (Public Web
Service and Agent Web Service) and the OdiInvokeWebService tool.
The Data Services and Run-Time Web Services components are invoked by a third-party
application, whereas the OdiInvokeWebService tool invokes a third-party web service.
Data Services provides access to data in datastores (both source and target datastores), as well
as changes trapped by the Changed Data Capture framework. This web service is generated by
Oracle Data Integrator and deployed in a Java EE application server.
The Public Web Service connects to the repository to retrieve a list of context and scenarios. This
web service is deployed in a Java EE application server.
The Agent Web Service commands the Oracle Data Integrator Agent to start and monitor a
scenario and to restart a session. Note that this web service is built into the Java EE or
Standalone Agent.
The OdiInvokeWebService tool is used in a package and invokes a specific operation on a port
of the third-party web service, for example, to trigger a BPEL process.
Oracle Data Integrator Run-Time Web Services and Data Services are two different types of Web
services. Oracle Data Integrator Run-Time Web Services enable you to access the Oracle Data
Integrator features through web services, whereas Data Services are generated by Oracle Data
Integrator to give you access to your data through web services.
Integration of ODI Within SOA in Action
Integration
process
ESB
Third-party
application
Business
process
Public Web
Service
Java EE
Web services
container
Data Service
SOAP
Source Target
Agent Web
Service
Run-time
agent
Third-party
web service
Port
Operation
Start scenario
Restart sessions
Get sessions status
Repository
List contexts
List scenarios
SOAP
SOAP
SOAP
Data
Access
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
In this example, you see an ODI data load that is controlled by SOA processes and uses web
services as well as databases as sources.
1. In this example, a business process in the product management department is changing
data in the product table of the operational database. This data needs to be propagated to
the data warehouse with minimal delay.
2. The process calls an ODI bulk data service to initiate a load based on the changes since the
last load.
3. The ODI Agent executes a scenario based on a package that first calls a data service from
the finance department to obtain price discount information.
4. Then this service joins this information with the changed product data from the operational
database. The joined and transformed data is stored in the data warehouse.
Example 1: Using Data Services
Operational
database
products
Data
warehouse
WS, XML
SQL
Join
Call web
service
Update data.
1
2
3
4
Finance data
service discounts
(BPEL)
Product
mgmt process
(BPEL)
ODI bulk data
service
ODI
Agent
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
This slide shows how ODI can be integrated with a BPEL process for handling ODI errors using
BPEL Human Workflow.
In this example, you expose an ODI Transformation Process as a web service, which is called
from within a BPEL process:
1. Create a BPEL process, which will call your ODI scenario.
2. Create your ODI transformation interface.
3. Create a Package with the OdiInvokeWebService step connected to your transformation
interface. When adding the OdiInvokeWebService tool step, you should connect to the
appropriate BPEL process WSDL and specify the parameters for the tool.
4. Generate a scenario from the package.
5. Edit the BPEL process by adding the Partner Links for integrating with ODI and calling your
package.
Example 2: Exposing ODI Process as a Web
Service
1. Create the BPEL process, which will call your ODI
scenario.
2. Create your ODI integration interface.
3. Create the package with the OdiInvokeWebService
step connected to your transformation interface.
4. Generate a scenario from the package.
5. Edit the BPEL process to call your package.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
In JDeveloper, you may need to create the new application and the new project for your BPEL
process. The project type you create is BPEL Process Project. Create your BPEL process
according to your business project. This slide shows the example of a business process that
prepares the result that will be consumed and processed by ODI Interface. For details on creating
BPEL processes, refer to Oracle Fusion Middleware User's Guide for Oracle Business Process
Management 11g Release 1 (11.1.1.6.3).
Creating the BPEL process
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
This slide shows how to create an ODI interface and an ODI Package with the
OdiInvokeWebService step connected to your transformation interface:
1. Create your transformation interface.
2. Create the new package.
3. Open the Diagram tab and add a new OdiInvokeWebService step from the Internet
folder in the Toolbox panel. Also, add your interface step to the package.
Creating ODI Interface and ODI Package
1
2
3
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
4. Configure the OdiInvokeWebService step. Select the URL to connect to the BPEL
process WSDL. Enter other parameters for the OdiInvokeWebService tool as shown in
the slide. In the properties panel, fill the following parameters:
- Storage mode for Response File: New File
- Response File: /tmp/processResponse.xml
5. Connect the OdiInvokeWebService step to the Interface step in your diagram.
Creating ODI Interface and ODI Package
Enter the URL for connecting
to the BPEL process WSDL
and click the Connect to
WSDL button.
4
5
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
This slide shows the result of creating the ODI scenario for the package and the process of editing
the BPEL process for invoking the scenario.
1. Create the scenario for the ODI package.
2. Edit the BPEL process to ensure your invocation of your package.
Creating ODI Scenario and Editing the BPEL
Process
2
1
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
After the BPEL process is ready, it should be deployed to the application server and tested. To
deploy and test your BPEL process:
1. In JDeveloper, select your BPEL process, right-click and then select Deploy. Follow the
screens to deploy your process to the application server.
2. In Enterprise Manager, expand: SOA > soa-infra (AdminServer) > ODIInvoke. Select
ODIInvoke and click Test Service.
Deploying the BPEL Process to the Application
Server and Invoking from Enterprise Manager
2
1
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
A business process is a set of coordinated tasks and activities, involving both human and system
interactions, that leads to accomplishing a set of specific organizational goals. Business Process
Execution Language (BPEL) is a programming abstraction that enables developers to compose
multiple discrete web services into an end-to-end process flow. BPEL enables the top-down
realization of a Service-Oriented Architecture (SOA) through composition, orchestration, and
coordination of Web services.
In the BPEL process, the activities, called Partner links are used to integrate the business
process (BPEL) process with other applications within SOA. They link the BPEL process to
corresponding web services.
ODI can be integrated with BPEL process by using the ODIInvokeWebservice Tool. Thus, a
business process can be invoked from the ODI Package. After the execution, the BPEL process
can send a response back to ODI for subsequent data processing.
Example 3: Calling a Web Service from Within ODI
for Processing ODI Errors with BPEL Human
Workflow
ODI Package
Business Process
(BPEL Process)
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
For enterprise SOA deployments, there is almost always a need for enterprise data extraction,
loading, transformation, and validation. By leveraging the native SOA architecture within ODI, you
can take advantage of ODI to perform the ELT (Extract Load Transform). ODI provides the ability
to validate data during the load to the target by using ODI constraints or database constraints.
When this data is checked against a constraint by using Flow Control, any errors that are found
are not loaded to the target but are loaded to an errors table that is created and managed by ODI.
Each row of this table represents a record that did not pass a constraint. The row also has a
message column that explains why the record was rejected. This table can be edited within ODI
Designer or any other tool that can edit relational tables.
However, this is not always a convenient way for the end user or business user to edit the data.
Alternatively, the ODI Error Hospital can be created with BPEL Human Workflow. Any rows that
do not pass the constraint will be loaded to the error table on the target database. The ODI
scenario can be executed, and after the ODI ELT process is completed, the ODI scenario will then
call a BPEL web service to notify it of any errors during the load. The BPEL process will import the
errors and manage them by using BPEL human workflow tasks. A user can then use the BPEL
Worklist application to update bad records. On subsequent executions of the ODI scenario, the
updated records are recycled into the ELT process.
Processing ODI Errors with BPEL Human
Workflow
1. Develop an ODI transformation by using an interface with
ODI constraint and CKM.
2. Develop a BPEL process for invoking ODI from a SOA and
generating human workflow tasks.
3. Create an ODI package to execute the interface and
invoke the web service.
2
1
3 1
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The records that pass the constraint are loaded into the target, and the error record is in the error
table. In the slide, you see the steps to create such a process.
Because ODI manages the error table, you do not have to write any code to recycle the errors or
to delete the updated records when they are loaded successfully. Thus, you have to implement
the following steps:
1. First you need to build an ODI scenario that will validate data during the load to the target by
using ODI constraints or database constraints. This ODI scenario will then call a BPEL Web
service to notify it of any errors during the load.
2. You need to create a BPEL process that will import the errors and create BPEL human
workflow tasks. A user can then use BPEL Worklist to update records that contain such
errors.
3. You need to create an ODI package to execute the Interface and call a web service that
executes the BPEL process.
4. Finally, execute the ODI package, monitor execution of the BPEL process, and complete the
human workflow.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
1. Develop an ODI transformation by using an interface with an ODI constraint:
1. Create a model and set up an ODI constraint that you need to enforce.
2. Create a new interface to load the source file (XML) to the target (RDBMS).
Workflow
1. Develop an ODI transformation by using an interface with
an ODI constraint.
1
2
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
2. Create the BPEL process that will import data errors from the error table. The BPEL process
will then create the human workflow tasks from the error records. You build a new BPEL
process to track the errors for each execution of the ODI package and present them to a
user for review. The user will be able to update each of the fields and correct any errors so
that on the next execution of the ODI package, the corrected rows are inserted into the
target. In this example, JDeveloper is the tool that you use to build the BPEL process and
deploy it to the application server:
1. Create a New Project for the BPEL process.
2. Create and configure the connection to an ODI data source.
3. Invoke database adapters to read errors and write error corrections back to the error
table.
Workflow
2
1
3
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
4. Add Human Task to your BPEL process.
5. Configure the Human Task parameters.
Workflow
5
4
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
3. Deploy the BPEL process; create and execute the ODI package. Deploy the BPEL process
to the application server.
1. Deploy the BPEL process.
2. Create the ODI package to execute the interface and invoke the Web service.
3. Execute your ODI package.
Workflow
3. Deploy the BPEL process; create and execute the ODI
package.
2
1
3
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
4. Monitor execution of the BPEL process from the BPEL Console, and complete the human
task:
1. Using Oracle BPM Worklist application, perform an action on the human task to correct
errors.
2. Return to the BPEL Console page and refresh the browser. The process is now
updated and you see that it has progressed past the Error Hospital Human workflow
activity.
3. Click the Invoke Corrections activity to see the data that the error table has been
updated with.
Note: You should now have a fully functional ODI to BPEL Human Workflow Error Hospital. If you
rerun your ODI scenario now, the corrected errors are picked up and resubmitted to the target
table.
Workflow
4. Execute the ODI package, monitor execution of the BPEL
process, and complete the human task.
2
1
3
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Answer: a
Explanation: Both types of agents have web service capabilities and can be used within a SOA
environment.
Quiz
Both types of agents, Java EE and Standalone, can be used for
integration of ODI within a SOA.
a. True
b. False
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Summary
In this lesson, you should have learned how to:
Describe the features of Java EE and Standalone Agents
Use different types of web services with ODI
Implement the integration of ODI with Service-Oriented
Architecture (SOA)
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Practice 9-1 Overview: Integrating ODI in the
Enterprise Environment
This practice covers the following topics:
Performing the steps needed to configure the ODI Java EE
Agent with the existing WLS domain
Performing monitoring of your ODI environment using
Enterprise Manager 11g and ODI Console
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Practice 9-2 Overview: Exposing an ODI Scenario
as a Web Service
This practice covers the following topics:
Creating a new ODI scenario
Configuring a web service
Invoking the web service to execute the scenario
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Practice 9-3 Overview: Integrating ODI with a
BPEL Process Within SOA
This practice covers learning how to integrate ODI with a BPEL
process within SOA.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Enhancing ODI Security
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Objectives
Describe the methods of enhancing security in ODI
Use nongeneric profiles for implementing a strongly
secured approach
Describe the methods of integrating ODI security
Implement external authentication with LDAP server
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Oracle Data Integrator (ODI) Security: Overview
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Objects are the visible part of ODI object components (Java classes). It is necessary to
combine the notions of object and object instance (or instances), which in ODI are similar to
object-oriented programming notions.
An example of an instance, MY_PROJ_1, is an instance of the project-type object. Similarly,
another instance of a project-type object is YOUR_PROJ_2. Thus, an instance is a particular
occurrence of an object. For example, the Datawarehouse project is an instance of the
Project object.
Each object has a series of methods that are specific to it. The notion of a method in Data
Integrator is similar to the one in object-oriented programming.
A profile contains a set of privileges for working with ODI. One or more profiles can be
assigned to a user to grant the sum of these privileges to the user. An authorization by profile
is placed on an objects method for a given profile. It allows a user with this profile to be
given, either optionally or automatically, the right to this object through the method. The
presence of an authorization by a profile for a method, under an object in the tree of a profile,
shows that a user with this profile is entitled (either optionally or automatically) to this objects
instances through this method. The absence of authorization by a profile shows that a user
with this profile cannot, under any circumstance, invoke the method on an instance of the
object.
Note: Objects and methods are predefined in ODI and must not be changed.
Security Concepts: Overview
Object: The representation of an element that can be
handled through ODI (agents, models, datastores)
Instance (object instance): An instance is a particular
occurrence of an object.
For example, the Datawarehouse project is an instance of
the Project object.
Method: The type of action that can be performed on an
object
Profile: A generic rights model for working with ODI (set of
privileges)
An authorization by a profile is placed on an objects method
for a given profile.
One or more profiles can be assigned to a user to grant the
sum of these privileges to this user.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
A user in the Security Navigator module represents an ODI user and corresponds to the
login name used for connecting to a repository. A user inherits the following rights:
- All privileges granted to its various profiles
- Privileges on objects or instances given to this user
An authorization by the user is placed on a method of an object for a given user. It allows the
user to be given, either optionally or automatically, the right to this object through the
method.
A User Method is a privilege granted to a user on a method of an object type. Each granted
method allows the user to perform an action (edit, delete, and so forth) on instances of an
object type (project, model, datastore, and so forth). These methods are similar to the
Profiles Methods, applied to users.
Note: It is possible to grant users with privileges on instances on specific work repositories where
these instances exist. For example, you may grant a developer user with the edit privilege on the
LOAD_DATAWAREHOUSE scenario on the a DEVELOPMENT repository and not on a PRODUCTION
repository. An authorization by Object Instance is granted to a user on an object instance. It allows
to grant to this user certain methods of this object instance.
Security Concepts: Overview
A user is an ODI user who corresponds to the login name
used for connecting to a repository. A user inherits the
following rights:
All privileges granted to its various profiles
Privileges on objects or instances given to this user
An authorization by the user is placed on a method of an
object for a given user.
A User Method is a privilege granted to a user on a
method of an object type. It allows the user to perform an
action.
A Profile Method is an authorization granted to a profile on
a method of an object type.
Each granted method allows a user with this profile to
perform an action (edit, delete, and so forth) on an instance
of an object type (project, model, datastore, and so forth).
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
An authorization by Object Instance is granted to a user on an object instance. It allows to grant to
this user certain methods of this object instance.
The presence of an authorization by object instance in a user's tree for a given instance shows
that the user is granted specific privileges on the object methods for the given instance (these
privileges are specified in the Object Instance editor). If an instance is not visible in the tree of the
user instances, the User Method or Profile Method privileges for the object type apply.
As an instance may be replicated over the different work repositories that are attached to the
master repository, authorizations by object instances may be defined for one, multiple, or all your
work repositories attached to the master repository. For example, a LOAD_DATAWAREHOUSE
scenario instance may be replicated (using, for example, versioning) in the DEVELOPMENT, TEST,
and PRODUCTION repositories. Privileges on the instance will change depending on the
repository.
For example, it is common to replicate projects from a Development repository to a Test
repository. A developer may be granted edit privileges for his or her project on the Development
repository, but not on the Test repository. On the Test repository, the developer will only be
granted with view privileges on the project.
Authorizing by Object Instance
An authorization by Object Instance is granted to a user on
an object instance.
It allows to grant to this user methods of this object instance.
The presence of an authorization by object instance in a
user's tree shows that the user is granted specific privileges
on the object methods for the given instance.
If an instance is not visible in the tree of the user instances,
the User Method or Profile Method privileges for the object
type apply.
It is possible to grant users with privileges on instances on
specific work repositories where these instances exist.
Authorizations by object instances may be defined for one,
multiple, or all your work repositories attached to the master
repository.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
By using the Security Navigator tab, you can manage security in ODI. The Security Navigator
module allows ODI users and profiles to be created. It is used to assign user rights for methods
(edit, delete, and so on) on generic objects (data server, data types, and so on), and to fine-tune
these rights on the object instances (Server 1, Server 2, and so on).
The Security Navigator stores this information in a master repository. This information can be used
by all the other modules.
ODI Security Navigator: Overview
Create New Profile
ODI Objects
Create New User
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Security Navigator objects available to the current user are organized by these tree views:
- The objects, describing each ODI elements type (datastore, model, and so on)
- The users profiles, users, and their authorizations
You can perform the following operations in each tree view:
- Insert or import root objects to the tree view by clicking the appropriate button in the
frame title.
- Expand and collapse nodes by clicking them.
- Activate the methods associated with the objects (Edit, Delete, and so on) through the
pop-up menus.
- Edit objects by double-clicking them or by dragging them on the Workbench.
The windows for the object being edited or displayed appear in the Workbench.
Note: Each tree view appears in floatable frames that may be docked to the sides of the main
window. These frames can also be stacked up. When several frames are stacked up, tabs appear
at the bottom of the frame window to access each frame of the stack. Tree view frames can be
moved, docked, and stacked by selecting and dragging the frame title or tab. To lock the position
of views, select Lock window layout from the Windows menu. If a tree view frame does not appear
in the main window or has been closed, it can be opened by using the Windows > Show View
menu.
ODI Security Navigator: Overview
Import profile.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Generic profiles have the Generic privilege option selected for all object methods. This implies that
a user with such a profile is by default authorized for all methods of all instances of an object to
which the profile is authorized.
Nongeneric profiles are not by default authorized for all methods on the instances because the
Generic privilege option is not selected for all object methods. The administrator must grant the
user the rights on the methods for each instance.
If the security administrator wants a user to have the rights on all instances of an object type by
default, the user must be given a generic profile.
If the security administrator wants a user to have the rights on no instance by default, but wants to
grant the rights by instance, the user must be given a nongeneric profile.
Using Generic and Nongeneric Profiles
Generic profiles:
Have the Generic privilege option selected for all object
methods
A user with such a profile is by default authorized for all
methods of all instances of an object.
Nongeneric profiles:
Are not by default authorized for all methods of all
instances of an object
The administrator must grant the user the rights on the
methods for each instance.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
ODI has some built-in profiles that the security administrator can assign to the users he or she
creates. This slide shows some built-in profiles delivered with ODI. For a complete list of built-in
profiles, see the Oracle Fusion Middleware Developer's Guide for Oracle Data Integrator 11g
Release 1 (11.1.1).
CONNECT: Must be granted with another profile
DESIGNER: Use this profile for users who will work mainly on projects.
METADATA_ADMIN: Use this profile for users who work mainly on models.
NG_METADATA_ADMIN: Nongeneric version of the METADATA_ADMIN profile
OPERATOR: Use this profile for production users.
REPOSITORY_EXPLORER: Use this profile for users who do not need to modify objects.
SECURITY_ADMIN: Use this profile for security administrators.
Built-in Profiles
CONNECT: To connect to ODI
DESIGNER: To perform development operations
NG_DESIGNER: Nongeneric version of the DESIGNER
profile
METADATA_ADMIN: To manage metadata (models)
NG_METADATA_ADMIN: Nongeneric version of the
METATADA_ADMIN profile
OPERATOR: To manage run-time objects (for production
users)
REPOSITORY_EXPLORER: To view objects (no
modification)
SECURITY_ADMIN: To edit security (for security
administrators)
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
TOPOLOGY_ADMIN: Use this profile for system or ODI administrators
VERSION_ADMIN: Use this profile for those entitled to perform version management
operations
GN_VERSION_ADMIN: Nongeneric version of the VERSION_ADMIN profile.
Built-in Profiles
TOPOLOGY_ADMIN: To edit topology (for ODI
administrators)
VERSION_ADMIN: To create, restore, and edit versions
and solutions
NG_VERSION_ADMIN: Nongeneric version of the
VERSION_ADMIN profile
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Implementing a Strongly Secured Approach
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Security Policy Approach
There are two main approaches for defining security in ODI:
The generic approach, where users inherit the privileges of
the profiles they have
This policy is suitable for most cases, and is simple to
configure.
The strongly secured approach, where users have no
default authorizations on objects
This approach uses only nongeneric profiles.
The security administrator must grant the users
authorizations on object instances.
This policy is complex to configure because it requires to
manage privileges by instance.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Defining Security Policies: Using Generic Profiles
1. Create the users.
2. Give the users the generic profiles (built-in or customized).
3. Optionally, you can define a password policy to encourage
users to use a secured password.
1
2 2
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
To implement a strongly secured approach:
2. Give the users nongeneric profiles (built-in or customized).
3. Grant the users privileges on object instances after these are created. This operation must
be repeated for every new instance.
Note: Only certain objects support the authorization by object instance. These object types are
listed under the Instances node for each user. The methods for which the user has generic
privileges are not listed in the Object Instance Editor.
Defining Security Policies: Using Nongeneric
Profiles
2. Give the users the nongeneric profiles (built-in or
customized).
3. Grant the users with privileges on object instances after
these are created.
1
2
3
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
As shown in the previous slide, you have to grant users with authorizations to each object
instance. To grant an authorization by object instance to a user:
1. In Security Navigator, expand the Users accordion. In the Designer, Operator, or Topology
Navigator, expand the accordion containing the object onto which you want to grant
privileges. Select this object and then drag it on the user in the Users accordion. Click Yes.
The authorization by object instance editor appears.
2. This editor shows the list of methods available for this instance and the instances contained
into it. For example, if you grant privileges on a project instance, the folders, interfaces, and
so forth contained in this project will appear in the editor. Fine-tune the privileges granted per
object and method. You may want to implement the following simple privileges policies on
methods that you select from the list:
- To grant all these methods in all repositories, click Allow selected methods in all
repositories.
- To deny all these methods in all repositories, click Deny selected methods in all
repositories.
- To grant all these methods in certain work repositories, click Allow selected methods in
selected repositories and then select the repositories from the list. Click OK.
Using Nongeneric Profiles: Granting an
Authorization by Object Instance
2
Allow selected
methods in
selected
repositories.
1
Allow selected
methods in all
repositories. 2
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Unused Authorizations
The authorizations by object instance are stored in the
master repository.
If objects are deleted from all work repositories, the
authorizations are not necessarily deleted.
You may want to retain certain unused authorizations if they
refer, for example, to objects currently stored in an exported
file or in a stored solution.
The Security Clean-up Tool should be used periodically to
remove these unused authorizations from the master
repository.
Unused authorizations are removed if they refer to objects
that do not exist in the master repository or in any work
repository.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
To clean up unused authorizations:
1. From the Security Navigator toolbar menu, select Clean Up Security Settings... The Security
Clean-up Tool dialog appears.
2. On the Cleanable tab, select Clean for the cleanable security settings you want to remove.
The security settings that cannot be removed are shown on the Non-cleanable tab.
3. Click OK to clean up the selected security settings.
Cleaning Up Unused Authorizations
1
2
3
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
ODI Security Integration: Overview
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The ODI stores by default all the user information as well as the users privileges in the master
repository. A user who logs in to ODI logs against the master repository. This authentication
method is called Internal Authentication. ODI can optionally use Oracle Platform Security Services
(OPSS) to authenticate its users against an external Identity Store, which contains enterprise
users and passwords. Such an identity store is used at the enterprise level by all applications, to
have centralized username and password definitions and Single Sign-On (SSO). In such a
configuration, the repository contains only references to these enterprise users. This
authentication method is called External Authentication.
The ODI stores by default all security information in the master repository. This password storage
option is called Internal Password Storage. ODI can optionally use Java Provisioning Service
(JPS) for storing critical security information. If you are using JPS with ODI, the data server
passwords and contexts are stored in the JPS Credential Store Framework (CSF). This password
storage option is called External Password Storage.
Note: When using External Password Storage, other security details such as usernames,
password, and privileges remain in the master repository.
ODI Security Integration: Overview
Implementing External Authentication (OPSS)
Implementing External Password Storage
JPS Integration
External Directory
Server
Directory
Services Manager
External
Authentication
jps-config.xml
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Configuring ODI components for External Authentication
To use the External Authentication option, you need to configure an Enterprise Identity Store
(LDAP, Oracle Internet Directory, and so forth), and have this identity store configured for
each ODI component to refer by default to it. The configuration to connect and use the
identity store is contained in an OPSS configuration file called the jps-config.xml file.
Refer to the Oracle Fusion Middleware Security Guide, 11g Release 1 (11.1.1) for more
information. Copy this file into the ODI_HOME/client/odi/bin/ directory. The Studio
reads the identity store configuration and authenticates it against the configured identity
store.
ODI components deployed in a container (Java EE agent, Oracle Data Integrator Console)
do not require a specific configuration. They use the configuration of their container. Refer to
the Oracle Fusion Middleware Security Guide, 11g Release 1 (11.1.1) for more information
about an OPSS configuration in a Java EE context.
Implementing External Authentication (OPSS)
Configuring ODI components for External Authentication
Setting the authentication mode
Setting up when creating the master repository
Switching the authentication mode
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Setting the Authentication Mode:
You can set or modify the password storage in two ways:
- Creating the master repository enables you to define the authentication mode.
- Switching the Authentication Mode modifies the authentication mode for an existing
master repository.
Note: When you perform a password storage recovery, context and data server passwords are
lost and need to be reentered manually in the Topology Navigator. If you are using External
Authentication, usernames and passwords are externalized. ODI privileges remain within the
repository. Data servers and context passwords also remain in the master repository. You can
externalize data server and context passwords by using the External Password Storage feature.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
To use the External Authentication option, you need to configure an enterprise Identity Store
(LDAP, Oracle Internet Directory, and so on), and have this identity store configured for each ODI
component to refer by default to it. The configuration to connect and use the identity store is
contained in an OPSS Configuration file called the jps-config.xml file.
Copy this file into the ODI_HOME/client/odi/bin/ directory. The Studio reads the identity
store configuration and authenticates against the configured identity store. If you want to locate
this file in a different location, edit the ODI_HOME/client/odi/bin/odi.conf file and edit the
option that sets the location of the configuration file. This option is set in the following line:
AddVMOption -Doracle.security.jps.config=./jps-config.xml.
Copy the jps-config.xml file also to the ODI_HOME/agent/bin/ directory. The agent and
the command line scripts will authenticate against the configured identity store.
Configuring External Authentication
To use the External Authentication option, you need to:
Configure an enterprise Identity Store (Embedded
WebLogic LDAP, Oracle Internet Directory (OID), or other
LDAP)
Have this identity store configured for each ODI
component to refer by default to it
The configuration to connect and use the identity store is
contained in an OPSS Configuration file called the jps-
config.xml file.
Copy this file into the ODI_HOME/client/odi/bin/
directory.
Copy this file into the ODI_HOME/agent/bin/ directory.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
This graphic depicts the options of using either the Embedded LDAP Server, which runs on the
ODI WebLogic Server domain, or the External Directory Server.
The applications use the identity, policy, and credential stores configured in the domain in which
they run.
A credential store is a repository of security data (credentials) that certify the authority of users,
Java components, and system components. In a credential store, data is used during
authentication, and then, during authorization, when determining what actions the subject can
perform. For more information about using the policy and credential store, refer to Oracle Fusion
Middleware Security Guide11g Release 1 (11.1.1).
Using External LDAP Server
ODI can use the WebLogic Server Embedded LDAP Server or
an external enterprise-grade server, such as Oracle Internet
Directory (OID).
External Directory
Server (such as OID)
ODI
WebLogic
Server Domain
Embedded
LDAP Server
OR
Oracle Data
Integrator
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
OID is an LDAP v3 directory service that leverages the scalability, high availability, and security
features of Oracle Database. It serves as the central user repository for Oracle Access Manager
and other Oracle applications. This graphic depicts the high-level architecture of an OID
installation.
OID stores user data in an Oracle Database. It is recommended that a separate and dedicated
database for OID be used. The database may or may not be on the same host.
Oracle Directory Server is the component that actually services the directory requests. Directory
server instances listen to requests from the LDAP clients, fetch information from the database, and
return the data to the clients.
Oracle Directory Services Manager (ODSM) provides a GUI management application for OID.
This runs on a managed server within the WebLogic Server OID domain (called IDMDomain by
default). ODSM is the only OID component that runs on WebLogic Server.
Fusion Middleware Control can also be used to manage OID. Oracle Process Manager and
Notification Server (OPMN) manages and monitors OID.
For more information about OID, see the Oracle Fusion Middleware Administrator's Guide for
Oracle Internet Directory 11g Release 1 (11.1.1).
Oracle Internet Directory (OID): Architecture
Oracle
Internet
Directory
OPMN
Oracle
Database
WebLogic
Server Domain
Oracle
Directory
Services
Manager
Fusion
Middleware
Control
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Configuring ODI External Authentication
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Configuring External Authentication with OID
To configure the External Authentication option, you need to
perform the following steps:
1. Create a new OID user by using Directory Services
Manager.
2. Edit the ODI jps-config.xml file to point to your
External LDAP Server.
3. Run the script to set up the credentials to access LDAP in
the identity store.
4. Create a new ODI master repository referencing a user in
the external OID LDAP Server.
5. Create a new ODI connection referencing a user in the
external OID LDAP Server.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
To create a new OID LDAP user:
1. Connect to Directory Services Manager (DSM).
2. Expand the Data tree and select the existing administrator user node. Right-click the node
and select Create Like. You create the user, for example, SUPERVISOR with administrative
privileges.
3. Set the required (including the password) and the optional properties for the new user.
For details on creating users with Oracle Internet Directory and Directory Services Manager, refer
to the Adding Users and Groups to Oracle Internet Directory section in Oracle Fusion
Middleware Tutorial for Oracle Identity Management 11g Release 1 (11.1.1).
1. Creating a New OID User with Directory
Services Manager
1
2
3
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Copy the jps-config.xml file in the following folder:
C:\<ODI_HOME>\oracledi\client\odi\bin.
Verify that you have the following three sections in the file. View properties in these three
sections:
1. JPS OID LDAP Identity Store Service Instance
2. Basic Authenticator Login Module
3. <jpsContext name="default">
Edit the values in the JPS OID LDAP Identity Store Service Instance section in the jps-
config.xml file to point to your external OID LDAP Server.
Note: The idstore.type property points to the type of the LDAP identity store with no default
value. The following are the examples of some possible values for this parameter:
OID: Oracle Internet Directory 10g and 11g
OVD: Oracle Virtual Directory 10g and 11g
ACTIVE_DIRECTORY: Active Directory
IPLANET: Sun Java System Directory Server
WLS_OVD: WebLogic OVD (Embedded LDAP server)
2. Editing the jps-config.xml File to Point to
OID LDAP Server

<serviceInstance name="idstore.oid" provider="idstore.ldap.provider">
<property name="subscriber.name" value="dc=oracle" />
<property name="idstore.type" value="OID" />
<property name="cleartext.ldap.credentials" value="cn=orcladmin:welcome1"/>
<property name="ldap.url" value="ldap://adc2180116.us.oracle.com:20555" />
<extendedProperty>
<name>user.search.bases</name>
<values>
<value>cn=users,dc=oracle </value>
</values>
</extendedProperty>
<extendedProperty>
<name>group.search.bases</name>
<values>
<value>cn=groups,dc=oracle </value>
</values>
</extendedProperty>
<property name="username.attr" value="uid" />
<property name="groupname.attr" value="cn" />
</serviceInstance>
<serviceInstance name="idstore.loginmodule" provider=
"jaas.login.provider">
<description>Identity Store Login Module
</description>
<property name="loginModuleClassName" value=
"oracle.security.jps.internal.jaas.module.
idstore.IdStoreLoginModule"/>
<property name="jaas.login.controlFlag" value=
"REQUIRED"/>
</serviceInstance> <jpsContext name="default">
<serviceInstanceRef ref="idstore.oid"/>
<serviceInstanceRef ref="idstore.loginmodule"/>
</jpsContext>
1
3
2
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
To run the script (odi_credtool.cmd) to set up the credentials for the identity store, navigate to
the c:\<ODI_Home>\oracledi\client\odi\bin directory and then execute the
odi_credtool.cmd command. Ignore possible warnings. The input entries are provided in the
following:
Map: oracle.odi.credmap
Key: <OID Realm>
Username: <OID Administrator User Name>
Password: <OID Administrator Password>
3. Running the Script (odi_credtool.cmd) to Set
Up the Credentials for the Identity Store
Execute the odi_credtool.cmd file and provide the following
input:
Map: oracle.odi.credmap
Key: <OID Realm>
User: <OID Administrator User Name>
Password: <OID Administrator Password>
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
To create a new master repository referencing a user in the external LDAP Server:
1. For new master repository, create the new RDBMS schema/user (Oracle 11g) with Connect
and Resource privileges.
2. Start Master Repository Creation Wizard, and then enter the necessary Database
connection information.
4. Creating New ODI Master Repository
Referencing a User in the External LDAP Server
Database user
for new master
repository
2
1
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
3. Select Use External Authentication and provide new Supervisor username and password.
These are the User and Password created in OID (or other external LDAP server).
4. To specify the type password storage, select Internal or External Password storage. Click
Finish.
4. Creating New ODI Master Repository
Referencing a User in the External LDAP Server
3
This is a user
that is created
in the LDAP
server.
4
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
You need to create a new connection to your new master repository. Configure Repository
Connections with necessary parameters. In the Oracle Data Integrator Connection section, enter
the User and Password of the authenticated user in your external LDAP store.
5. Creating a New ODI Connection Referencing a
User in the External OID LDAP Server
A user created
in the LDAP
server
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Switching the authentication mode of the ODI repository changes the way users authenticate. This
operation must be performed by a Supervisor user.
When switching from an External to an Internal authentication, user passwords are not copied
from the identity store to the repository. The passwords are nullified. All the user accounts are
marked as expired and must be reactivated by a SUPERVISOR that is created during the switch.
When switching from an Internal to an External authentication, the users that exist in the repository
and match a user in the identity store are automatically mapped. Users that do not match a user in
the identity store are disabled. A Supervisor must edit the users so that their name has a match in
the identity store.
Switching the Master Repository Authentication
Mode
This operation must be performed by a Supervisor user.
When switching from an External to an Internal
authentication:
User passwords are not copied from the identity store to the
repository.
All the user expired accounts must be reactivated by a
SUPERVISOR that is created during the switch.
The context passwords are lost. Passwords for data servers
and the jdbc password of the work repository are removed
from their credential store.
When switching from an Internal to an External
authentication:
Users that exist in the repository and match a user in the
identity store are automatically mapped
Users that do not match a user in the identity store are
disabled
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Use the Switch Authentication Mode wizard to change the user authentication mode.
Note: Before launching the Switch Authentication Mode wizard, perform the following tasks:
Disconnect Oracle Data Integrator Studio from the repository.
Shut down every component by using the ODI repository.
Using Switch Authentication Mode Wizard
Use the Switch Authentication Mode Wizard to change the
user authentication mode.
Disconnect ODI Studio from the repository.
Before launching the Switch Authentication Mode Wizard:
Disconnect ODI Studio from the repository.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
To use the Switch Authentication Mode Wizard:
1. From the ODI main menu, select Switch Authentication Mode.... The Switch Authentication
Mode Wizard appears.
2. Specify the JDBC connectivity details of your ODI master repository as defined when
connecting to the master repository. Click Next.
1
2
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The next action varies depending on the current Authentication Mode in use:
If currently using Internal Authentication, you are prompted to switch to external
authentication.
If currently using External Authentication, you are prompted to switch to internal
authentication. You must provide and confirm a password for the SUPERVISOR user that the
wizard will automatically create in the repository.
Click Finish.
3
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
After switching to Internal authentication, you can reactivate all ODI users that have been
deactivated during the switch.
To reactivate a User:
1. In Security Navigator, expand the Users accordion. Select the user that you want to
reactivate from the list of users. Right-click and select Edit. The User editor appears.
2. Deselect Allow Expiration Date. If you want to set a password for this user, click Change
Password and enter the new password for this user.
3. From the File main menu, select Save.
Reactivating Users After Switching to Internal
Authentication
1
2
3
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
If you have switched from internal to external authentication, you can reconnect to the ODI
repository as one of the users with supervisor privileges and re-enable the ODI users that have
been disabled during the switch.
1. In Security Navigator, expand the Users accordion. Select the user that you want to re-
enable from the list of users. Right-click and select Open. The User editor appears.
2. In the Name field, enter a username that matches the login of an enterprise user in the
identity store. Click Retrieve GUID. If the username has a match in the identity store, this
external user's GUID appears in the External GUID field.
3. Click the Save button.
Re-enabling Users After Switching to External
Authentication
1
2
3
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
ODI stores by default all security information in the master repository. This is internal password
storage.
To use the external password storage option, you need to install a WebLogic Server instance
configured with JPS, and all ODI components (including the Runtime Agent) need to have access
to the remote credential store.
When using JPS with ODI, the data server passwords and contexts are stored in the JPS
Credential Store Framework (CSF).
To use the external password storage option, you need to install a WebLogic Server instance
configured with JPS and all ODI components (including the runtime Agent) need to have access to
the remote credential store. For more information, refer to Configuring Applications to Use OPSS
in the Oracle Fusion Middleware Application Security Guide 11g Release 1 (11.1.1).
External Password Storage
ODI stores by default all security information in the master
repositoryInternal Password Storage.
ODI can optionally use JPS for storing critical security
informationExternal Password Storage.
When using JPS with ODI, the data server passwords and
contexts are stored in the JPS Credential Store Framework
(CSF).
Other security details such as usernames, user passwords,
and privileges remain in the master repository.
It is possible to externalize the authentication and have users
and passwords stored in an Identity Store by using External
Authentication.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Setting Up External Password Storage
There are four ways to set or modify password storage:
Importing the master repository enables you to change the
password storage.
Creating the master repository enables you to define the
password storage.
Switching Password Storage modifies storage for an
existing master repository.
Recovering the Password Storage enables you to recover
from a credential store crash.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
To switch the password storage mode:
1. Disconnect ODI Studio from the repository. From the ODI main menu, select Password
Storage > Switch...
2. Specify the login details of your Oracle Data Integrator master repository, which are defined
when connecting to the master repository. Click Next.
Switching the Password Storage:
1. Disconnect ODI Studio from the repository. From the ODI
main menu, select Password Storage > Switch.
2. Specify the login details of your Oracle Data Integrator
master repository.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
3. Select External Password Storage.
4. With this mode, you have to provide the MBean Server Parameters to access the credential
store. These parameters are provided in the following. Click Finish. You can now reconnect
to the ODI repository.
- Host: MBeans Server Host (for example, abc115.oracle.com)
- Port: MBeans Server Port (for example, 7001)
- User: MBeans Server User Name (for example, weblogic)
- Password: MBeans Server Password (for example, weblogic)
Note: ODI offers a password recovery service that should be used only in case of an external
password storage crash. Using this procedure, password storage is forced to Internal Password
Storage as the external storage is no longer available. This operation should be performed by a
Supervisor user.
Switching the Password Storage:
3. Select the Password storage mode (Internal/External).
4. Specify the MBean server parameters. Click Finish.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Answer: d
Explanation: When switching from an External to an Internal authentication, user passwords are
not copied from the identity store to the repository. The passwords are nullified. All the user
accounts are marked as expired and must be reactivated by a SUPERVISOR that is created
during the switch.
Quiz
Which of the following is true about switching from an External
to an Internal authentication?
a. User passwords are not copied from the identity store to
the repository.
b. All the user accounts are marked as expired.
c. All the user accounts must be reactivated by a
SUPERVISOR user that is created during the switch.
d. All of the above
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Summary
Describe methods of enhancing security in ODI
Use nongeneric profiles for implementing a strongly
secured approach
Describe methods of integrating ODI security
Implement external authentication with LDAP server
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Practice 10-1: Implementing ODI External User
Authentication
In this practice, you implement ODI external authentication with
OID LDAP Server.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Choosing Integration Strategies: Best
Practices
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
This lesson provides an overview of managing ODI security. You also learn how to integrate
WebLogic Server and Enterprise Manager with ODI.
Objectives
Describe typical integration processes
Describe different integration strategies
Implement integration strategies with staging area on the
target
Implement integration strategies with staging area different
from the target
Maintain the integrity of data
Describe best practices for working on ODI projects
Implement real-time data integration by using ODI with
Oracle GoldenGate
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Defining an ODI Integration Strategy
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The following slides explain some of the integration strategies that are used in Oracle Data
Integrator. They are grouped into two families:
Strategies with Staging Area on the Target
Strategies with the Staging Area Different from the Target
Integration Strategies
They are grouped into two families:
Strategies with the Staging Area Different from the Target
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
An integration process uses an integration strategy that defines the steps required in the
integration process. Examples of integration strategies are:
Append: Optionally delete all records in the target datastore and insert all the flow into the
target.
Control Append: Optionally delete all records in the target datastore and insert all the flow
into the target. This strategy includes an optional flow control.
Incremental Update: Optionally delete all records in the target datastore. Identify new and
existing records by comparing the flow with the target, and then insert new records and
update existing records in the target. This strategy includes an optional flow control.
Slowly Changing Dimension: This strategy implements a Type 2 Slowly Changing
Dimension by identifying fields that require a simple update in the target record when
changed and fields that require to historize the previous record state.
This phase may involve one single server when the staging area and the target are located in the
same data server. It may involve two servers when the staging area and target are on different
servers.
The following strategies are used when the staging area
schema is located in the same data server as the target table
schema:
Append
Control Append
Incremental Update
Slowly Changing Dimension
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
This strategy inserts the incoming data flow into the target datastore, possibly deleting the content
of the target beforehand.
This integration strategy includes the following steps:
1. Delete (or truncate) all records from the target table. This step usually depends on a KM
option.
2. Transform and insert data from sources located on the same server and from loading tables
in the staging area. When dealing with remote source data, LKMs will have already prepared
loading tables. Sources on the same server can be read directly. The integration operation
will be a direct INSERT/SELECT statement leveraging, containing all the transformations
performed on the staging area in the SELECT clause and on all the transformations on the
target in the INSERT clause.
3. Commit the transaction. The operations performed on the target should be done within a
transaction and committed after they are all complete. Note that committing is typically
triggered by a KM option called COMMIT.
Note: The same integration strategy can be obtained by using the Control Append strategy and
not choosing to activate flow control.
Strategies with Staging Area on the Target:
Append
The Append strategy inserts the incoming data flow into the
target datastore, before possibly deleting the content of the
target.
1. Delete (or truncate) all records from the target table. This
step usually depends on a KM option.
2. Transform and insert data from sources located on the
same server and from loading tables in the staging area.
3. Commit the transaction. The operations performed on the
target should be done within a transaction and committed
after they are all complete.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
This approach can be improved by adding extra steps that will store the flow data in an integration
table ("I$"), and then call the CKM to isolate erroneous records in the error table ("E$").
The Control Append integration strategy includes the following steps:
1. Drop (if it exists) and create the integration table in the staging area. This is created with the
same columns as the target table so that it can be passed to the CKM for flow control.
2. Insert data into the loading table from the sources and loading tables by using a single
INSERT/SELECT statement similar to the one loading the target in the Append strategy.
3. Call the CKM for flow control. The CKM will evaluate every constraint defined for the target
table on the integration table data. It will create an error table and insert the erroneous
records into this table. It will also remove erroneous records from the integration table. After
the CKM completes, the integration table will contain only valid records. Inserting them into
the target table can then be done safely.
Control Append
The Control Append integration strategy includes the following
steps:
1. Drop (if it exists) and create the integration table in the
staging area.
2. Insert data into the loading table from the sources and
loading tables by using a single INSERT/SELECT
statement similar to the one loading the target in the
Append strategy.
3. Call the CKM for flow control. It will create an error table
and insert the erroneous records into this table. After the
CKM completes, the integration table will contain only valid
records.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Control Append
4. Remove all records from the target table. This step can be
made dependent on an option value that is set by the
designer of the interface.
5. Append the records from the integration table to the target
table in a single INSERT/SELECT statement.
6. Commit the transaction.
7. Drop the temporary integration table.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The Incremental Update strategy is used to integrate data in the target table by comparing the
records of the flow with existing records in the target according to a set of columns called the
"update key". Records that have the same update key are updated when their associated data is
not the same. Those that do not yet exist in the target are inserted. This strategy is often used for
dimension tables when there is no need to keep track of the records that have changed.
The challenge with such Integration Knowledge Modules (IKMs) is to use set-oriented SQL-based
programming to perform all operations rather than using a row-by-row approach that often leads to
performance issues. This method is described in the following:
1. Drop (if it exists) and create the integration table in the staging area. This is created with the
same columns as the target table so that it can be passed to the CKM for flow control. It also
contains an IND_UPDATE column that is used to flag the records that should be inserted ("I")
and those that should be updated ("U").
2. Transform and insert data in the loading table from the sources and loading tables by using a
single INSERT/SELECT statement. The IND_UPDATE column is set by default to "I".
3. Recycle the rejected records from the previous run to the integration table if the
RECYCLE_ERROR KM option is selected.
4. Call the CKM for flow control. The CKM will evaluate every constraint defined for the target
table on the integration table data. It will create an error table and insert the erroneous
records into this table. It will also remove erroneous records from the integration table.
Incremental Update
The Incremental Update strategy is used to integrate data in
the target table by comparing the records of the flow with the
existing records in the target according to the "update key".
staging area with the same columns as the target table so
that it can be passed to the CKM for flow control.
2. Transform and insert data in the loading table using a
single INSERT/SELECT statement. The IND_UPDATE
column is set by default to "I".
3. Recycle the rejected records from to the integration table if
the RECYCLE_ERROR KM option is selected.
4. Call the CKM for flow control. It will create an error table
and insert the erroneous records into this table.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
5. Update the integration table to set the IND_UPDATE flag to "U" for all the records that have
the same update key values as the target ones. Therefore, records that already exist in the
target will have a "U" flag. This step is usually an UPDATE/SELECT statement.
6. Update the integration table again to set the IND_UPDATE column to "N" for all records that
are already flagged as "U" and for which the column values are exactly the same as the
target ones. As these flow records match exactly the target records, they do not need to be
used to update the target data. After this step, the integration table is ready for applying the
changes to the target as it contains records that are flagged:
- "I": These records should be inserted into the target.
- "U": These records should be used to update the target.
- "N": These records already exist in the target and should be ignored.
7. Update the target with records from the integration table that are flagged as "U". Note that
the update statement is typically executed before the INSERT statement to minimize the
volume of data manipulated.
8. Insert records into the integration table that are flagged as "I" into the target.
Incremental Update
5. Update the integration table to set the IND_UPDATE flag to
"U" for all the records that have the same update key
values as the target ones.
6. Update the integration table again to set the IND_UPDATE
column to "N" for all records that are already flagged as
"U" and for which the column values are exactly the same
as the target ones.
7. Update the target with records from the integration table
that are flagged as "U".
8. Insert records into the integration table that are flagged as
"I" into the target.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Incremental Update: Optimization
This approach can be optimized depending on the
underlying database. The following examples illustrate
such optimizations:
With Teradata, it may be more efficient to use a left-outer join
between the flow data and the target table to populate the
integration table with the IND_UPDATE column already set
properly.
With Oracle, it may be more efficient in some cases to use a
MERGE INTO statement on the target table instead of an
UPDATE then INSERT.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Working with Slowly Changing Dimensions
Slowly Changing Dimension (SCD) is a dimension that
stores and manages both current and historical data over
time in a Data Warehouse
It is considered and implemented as one of the most
critical ETL (Extract, Transform, Load) tasks in tracking the
history of dimension records.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The three types of SCDs are:
Type 1 SCDs: Overwriting. In a Type 1 SCD, the new data overwrites the existing data.
Thus, the existing data is lost as it is not stored anywhere else. This is the default type of
dimension that you create. You do not need to specify any additional information to create a
Type 1 SCD.
Type 2 SCDs: Creating another dimension record. A Type 2 SCD retains the full history of
values. When the value of a chosen attribute changes, the current record is closed. A new
record is created with the changed data values and this new record becomes the current
record. Each record contains the effective time and expiration time to identify the time period
between which the record was active.
Type 3 SCDs: Creating a current value field. A Type 3 SCD stores two versions of values for
certain selected level attributes. Each record stores the previous value and the current value
of the selected attribute. When the value of any of the selected attributes changes, the
current value is stored as the old value and the new value becomes the current value.
Three Types of Slowly Changing Dimensions
The three types of SCDs are:
Type 1 SCD: Overwriting. The new data overwrites the
existing data (default type). The data is lost.
Type 2 SCD: Creating another dimension record.
It retains the history of values. When the value of an attribute
changes, the current record is closed.
A new record is created with the changed data values. It
contains the effective and expiration time.
Type 3 SCD: Creating a current value field.
It stores two versions of values for certain selected level
attributes.
Each record stores the previous value and the current value
of the selected attribute.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
A Type 2 SCD is one of the most well-known Data Warehouse loading strategies. It is often used
for loading dimension tables, to keep track of changes that occurred on some of the columns. A
typical SCD table would contain the following columns:
A surrogate key that is calculated automatically. This is usually a numeric column that
contains an autonumber such as an identity column, a rank function, or a sequence.
A natural key. List of columns that represent the actual primary key of the operational
system.
Columns that may be overwritten on change
Columns that require the creation of a new record on change
A start date column indicating when the record was created in the Data Warehouse
An end date column indicating when the record became obsolete (closing date)
A current record flag indicating whether the record is the actual one (1) or an old one (0)
Working with Type 2 Slowly Changing
Dimensions
It is often used for loading dimension tables, to keep track
of the changes that occurred on some of the columns.
A typical SCD table would contain the following columns:
A surrogate key calculated automatically
A natural key
Columns that may be overwritten on change
Columns that require the creation of a new record on change
A start date columnwhen the record was created.
An end date columnwhen the record closed.
A current record flag: The actual (1) or an old one (0)
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
This slide shows an example of the behavior of the product SCD. In the operational system, a
product is defined by its ID that acts as a primary key. Every product has a name, a size, a
supplier and a family. In the Data Warehouse, you need to store a new version of this product
whenever the supplier or the family is updated in the operational system.
In this example, the product dimension is first initialized in the Data Warehouse on March 12,
2006. All the records are inserted and are assigned a calculated surrogate key as well as the
ending date is set to January 1, 2400. As these records represent the current state of the
operational system, their current record flag is set to 1.
After the first load, the following changes happen in the operational system: The supplier is
updated for product P1; The family is updated for product P2; The name is updated for product P3;
Product P5 is added.
These updates have the following impact on the Data Warehouse dimension:
1. The update of the supplier of P1 is translated into the creation of a new current record
(Surrogate Key 5) and the closing of the previous record (Surrogate Key 1).
2. The update of the family of P2 is translated into the creation of a new current record
(Surrogate Key 6) and the closing of the previous record (Surrogate Key 2).
3. The update of the name of P3 updates the target record with Surrogate Key 3.
4. The new product P5 is translated into the creation of a new current record (Surrogate Key 7).
Type 2 SCDs: Example
1
Family is updated for product P2.
Name is updated for product P3.
Supplier is updated for product P1.
Product P5 is
added.
2
3
4
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
To create a Knowledge Module that implements this behavior, it is necessary to know which
columns act as a surrogate key, a natural key, a start date, and so on. Oracle Data Integrator
stores this information in the Slowly Changing Dimension Behavior field on the Description tab for
every column in the model.
When populating such a datastore in an interface, the IKM has access to this metadata by using
the SCD_xx selectors on the getColList() substitution method.
Implementing Type 2 SCDs
To create a Knowledge Module that implements the Type 2
SDC behavior, it is necessary to know which columns act as a
surrogate key, a natural key, a start date and so on.
ODI stores this information in the Slowly Changing
Dimension Behavior field on the Description tab for every
column in the model.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Implementing Type 2 SCDs: Steps
ODI implements Type 2 SCD as follows:
staging area.
2. Insert the flow data in the integration table by using only
mappings that apply to the Natural Key, Overwrite on
Change and Add Row on Change columns. Set the
starting timestamp to the current date and the ending
timestamp to a constant.
3. Recycle previous rejected records.
4. Call the CKM to perform a data quality check on the flow.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Note
There may be some cases where the SQL produced requires further tuning and optimization. For
more information, refer to Oracle Fusion Middleware Knowledge Module Developer's Guide for
Implementing Type 2 SCDs: Steps
5. Flag the records in the integration table as 'U' when the
natural key and the add row on change columns have not
changed compared to the current records of the target.
6. Update the target with the columns flagged overwrite on
change by using the integration table content filtered on
the 'U' flag.
7. Close old recordsthose for which the natural key exists
in the integration table.
Set the current record flag to 0 and the ending timestamp to
the current date
8. Insert the new changing records with their current record
flag set to 1.
9. Drop the integration table.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Strategies with Staging Area Different from the
Target
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
There are some cases when the source is a single file that can be loaded directly into the target
table by using the most efficient method. By default, ODI suggests to locate the staging area on
the target server, use a LKM to stage the source file in a loading table, and then use an IKM to
integrate the loaded data to the target table.
If the source data is not transformed, the loading phase is not necessary.
In this situation, you would use an IKM that directly loads the file data to the target: this requires
setting the staging area on the source file logical schema. By doing this, ODI will automatically
suggest to use a "Multi-Connection" IKM that moves data between a remote staging area and the
target.
Such an IKM would use a loader, and include the following steps:
1. Generate the appropriate load utility script.
2. Run the loader utility.
An example of such a KM is the IKM File to Teradata (TTU).
Target
These strategies are used when the staging area cannot be
located on the same data server as the target datastore (Files,
for example).
File to Server Append:
There are some cases where the source is a single file that
can be loaded directly into the target table.
If the source data is not transformed, the loading phase is
not necessary. In this situation, you would use an IKM that
directly loads the file data to the target.
This requires setting the staging area on the source file
logical schema.
ODI will automatically suggest to use a "Multi-Connection"
IKM that moves data between a remote staging area and the
target.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Server to Server Append
When using a staging area different from the target and when setting this staging area to an
RDBMS, it is possible to use an IKM that moves the transformed data from the staging area to the
remote target. This type of IKM is similar to a LKM and follows the same rules.
The steps when using the agent are usually:
1. Delete (or truncate) all records from the target table. This step usually depends on a KM
option.
2. Insert the data from the staging area to the target. This step has a SELECT statement in the
"Command on Source" tab that will be executed on the staging area. The INSERT statement
is written by using bind variables in the "Command on Target" tab and will be executed for
every batch on the target table.
The IKM SQL to SQL Append is a typical example of such a KM.
A variation of this strategy uses loaders or database-specific methods for loading data from the
staging area to the target, instead of the agent.
Target
Server to Server Append:
When using a staging area different from the target and
when setting this staging area to an RDBMS, it is possible to
use an IKM that moves the transformed data from the
staging area to the remote target.
This type of IKM is similar to a LKM and follows the same
rules.
Server to File or JMS Append:
When the target datastore is a file or JMS queue, the staging
area is set on a different location than the target.
You should use a "Multi-Connection" IKM that will integrate
the transformed data from your staging area to this target.
The method to perform this data movement depends on the
target technology
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Server to File or JMS Append
When the target datastore is a file, or a JMS queue or topic, the staging area is set on a different
location other than the target. Therefore, if you want to target a file or a queue datastore, you will
have to use a "Multi-Connection" IKM that will integrate the transformed data from your staging
area to this target. The method to perform this data movement depends on the target technology.
For example, it is possible to use the agent or specific features of the target (such as a Java API).
Typical steps of such an IKM will include:
1. Reset the target file or queue made dependent on an option.
2. Unload the data from the staging area to the file or queue.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Using ODI for Bulk Processing
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
A business process is a set of coordinated tasks and activities, involving both human and system.
Bulk data processing is the processing of a batch of discrete records between participating
applications. Oracle Application Integration Architecture (AIA) uses Oracle Data Integrator to
perform bulk data integrations.
Using ODI for Bulk Processing
AIA-approved design patterns for using ODI with Oracle
Application Integration Architecture (AIA):
Bulk data processing is the processing of a batch of
discrete records between participating applications.
Oracle AIA uses ODI to perform bulk data integrations.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
This slide provides an overview of AIA-approved design patterns for AIA ODI architecture.
Because ODI data transfer is always point-to-point, the source and target systems must be
capable of processing batch loads of data. An integration project should not adopt ODI as a
solution if there is a limitation in the number of rows that can be processed either on the source-
side or on the target-side application. There are four AIA-approved design patterns for using ODI
with AIA architecture:
Initial data loads: The initial set of data of a particular object is loaded from the source
database to the target database. For example, loading Customer Account information or
loading Invoice information into a new application database from an existing source
application database. In the process, XREF data may or may not get established depending
on the integration requirement.
High-volume transactions with XREF table: The interface tables on the source side have
a mechanism to indicate processed rows. Whenever a need arises for a high-volume data
transfer, AIA recommends using the Oracle Data Integrator solution for data transfer
between applications. Using this approach, the Oracle Data Integrator package transfers
data from source to target system on a regular basis. AIA recommends that the interface
tables on the source side have a mechanism to indicate processed rows.
Using ODI for Bulk Processing: Design Patterns
AIA-approved design patterns for using ODI with AIA
architecture:
Initial data loads: The initial set of data of a particular
object is loaded from the source database to the target
database.
High-volume transactions with XREF table
The interface tables on the source side have a mechanism to
indicate processed rows.
Intermittent high-volume transactions
If you have a requirement that batch loading co-exists with
regular online transactions, AIA recommends this approach.
Point-to-point integration with ODI
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Intermittent high-volume transactions: Batch loading coexists with regular online
transactions (AIA recommends this approach). If you have a requirement that batch loading
coexists with regular online transactions, AIA recommends this approach.
Point-to-point integration with ODI: For situations in which storing XREF data for high-
volume transactions does not make sense, AIA recommends using point-to-point integration
by using Oracle Data Integrator. For example, the headquarters of a retail store chain
receives data from individual retail stores every day at the close of business. In this scenario,
you need not store XREF data between each individual local store with HQ because there
are not any DML operations on those datasets. Local enterprise resource planning (ERP)
applications load their interface table and invoke an Oracle Data Integrator package to send
data to the HQ Interface table. After the data is processed, Oracle Data Integrator updates
the local ERP application's Interface table with a Transferred or Processed status for each
row.
For details, refer to Using Oracle Data Integrator for Bulk Processing in Oracle Fusion
Middleware Developer's Guide for Oracle Application Integration Architecture Foundation Pack
11g Release 1 (11.1.1.6.3).
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
When you create or update objects in one application, you may also want to propagate the
changes to the other applications. For example, when a new customer is created in an SAP
application, you may want to create a new entry for the same customer in your Oracle E-Business
Suite application named EBS. However, the applications that you are integrating may be using
different entities to represent the same information. For example, for each new customer in an
SAP application, a new row is inserted in its Customer database with a unique identifier such as
SAP_001. When the same information is propagated to Oracle E-Business Suite and Siebel
applications, the new row should be inserted with different identifiers such as EBS_1001 and
SBL001. In such cases, you need some type of functionality to map these identifiers with each
other so that they can be interpreted by different applications to be referring to the same entity.
This can be done by using cross-references.
The identifier mapping is also required when information about a customer is updated in one
application and the changes must be propagated to other applications. You can integrate different
identifiers by using a common value integration pattern, which maps to all identifiers in a cross-
reference table. For example, you can add one more column named Common to the cross-
reference table shown in the slide.
Cross-referencing is an Oracle Fusion Middleware function, available through the Mediator
component, and leveraged typically by any loosely coupled integration that is built on the Service
Oriented Architecture (SOA) principles. It is used to manage the run-time correlation between the
various participating applications of the integration.
Using Cross-Reference (XREF) Table: Overview
Cross-references enable you to dynamically map values
for equivalent entities created in different applications.
Cross-references are stored in the form of tables (XREF).
The identifier mapping is required when information about
a customer is updated in one application and the changes
must be propagated to the other applications (Common).
Add one more column: Common
Example:
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
In ODI, the creation of XREF data is a two-step process. Each step is an interface.
In the first interface, the user's source table is the source in Oracle Data Integrator and the user's
target table is the target in Oracle Data Integrator.
While transporting data from the source table to the target table, create the XREF data for the
source and common rows. In this process, if you want to populate any column of the target with
the COMMON identifier, the Oracle Data Integrator knowledge module takes care of that too.
In the second step, after data is posted from the interface table to the base table, the target
application must identify a datastore where the mapping between target identifier and common (or
source) identifier that you have sent during previous interface processing is available.
A second interface must be run in which this datastore is the source in Oracle Data Integrator and
the XREF table is the target. This creates the appropriate target columns in the XREF table.
Using the XREF Knowledge Module
In ODI, the creation of XREF data is a two-step process. Each
step is an interface:
First: The user's source table is the source in ODI and the
user's target table is the target in ODI. The XREF data for
the source and common rows is created. Any column of
the target is populated with the COMMON identifier.
Second: Target application identifies the datastore with the
mapping between the target identifier and the common (or
source) identifier.
In the second interface, this datastore is the source in ODI
and the XREF table is the target.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Note
Because XREF column names cannot be hardcoded, two variables must be defined to hold the
source and target column names. Normally, these column names are derived from the AIA
Configuration file. This section does not describe how to get that from the XML but rather it
describes how to refresh this from a SQL select statement.
Using the XREF Knowledge Module
1
2
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Real-Time Data Integration:
Using ODI with Oracle GoldenGate
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Various architectures for collecting transactional data from operational sources have been used to
populate Data Warehouses. These techniques vary mostly on the latency of data integration, from
daily batches to continuous real-time integration. The capture of data from sources is either
performed through incremental queries that filter based on a timestamp or flag, or through a CDC
mechanism that detects any changes as they are happening. Architectures are further
distinguished between pull and push operations, where a pull operation polls in fixed intervals for
new data, while in a push operation data is loaded into the target once a change appears.
A daily batch mechanism is most suitable if intraday freshness is not required for the data, such as
longer-term trends or data that is only calculated once daily (for example, financial close
information). Batch loads might be performed in a down time window, if the business model does
not require a 24-hour availability of the Data Warehouse.
Loading Data Patterns
Batch:
Data is loaded in full or incrementally using off-pick window
Capture: Filter Query
Mini-batch:
Data is loaded incrementally using intra-day loads
Capture: Filter Query
Micro-batch:
Source changes are captured and accumulated to be loaded
in intervals
Capture: CDC
Real-time:
Source changes are captured and applied immediately
Capture: CDC
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Oracle GoldenGate: Overview
Oracle GoldenGate provides low-impact capture, routing,
transformation, and delivery of database transactions across
heterogeneous environments in real time.
Real Time
Moves with sub-second latency
Heterogeneous
Moves changed data across
different databases and platforms
Transactional
Maintains transaction integrity
Key Capabilities:
Performance
Extensibility &
Flexibility
Reliability
Additional Differentiators:
Log-based capture
moves thousands of
transactions per second
with low impact
Meets a variety of
customer needs and data
environments with open,
modular architecture
Resilient against
interruptions and
failures
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Oracle GoldenGate provides the following data replication solutions:
High availability:
Live standby for an immediate failover solution that can later resynchronize with your primary
source
Active-active solutions for continuous availability and transaction load distribution between
two or more active systems
Zero-downtime upgrades and migrations:
Eliminates down time for upgrades and migrations
Live reporting:
Feeding a reporting database so that you do not burden your source production systems
Operational Business Intelligence (BI):
Real-time data feeds to operational datastores or Data Warehouses, directly or via ETL
tools.
Transactional data integration:
Real-time data feeds to messaging systems for business activity monitoring (BAM), business
process monitoring (BPM), and complex event processing (CEP)
Uses event-driven architecture (EDA) and service-oriented architecture (SOA)
Oracle GoldenGate Solutions: Overview
OLTP Systems
Log-based, Real-
Time Change Data
Capture
Standby
(Open & Active)
New DB/OS/HW
Reporting
Database
ODS EDW
EDW
ETL
ETL
Message
Bus
High Availability
Zero-Downtime
Upgrades and
Migrations
Live Reporting
Operational BI
Transactional
Data Integration
BPM
BAM
CEP
Oracle
GoldenGate
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Oracle GoldenGate consists of decoupled modules that are combined to create the best possible
solution for your business requirements.
On the source system(s):
Oracle GoldenGates Capture (Extract) process reads data transactions as they occur, by
reading the native transaction log, typically the redo log. Oracle GoldenGate moves only
changed, committed transactional data, which is only a percentage of all transactions
therefore operating with extremely high performance and very low impact on the data
infrastructure.
Filtering can be performed at the source or target: at table, column, or row level.
Transformations can be applied at the capture or delivery stages.
Advanced queuing (trail files):
To move the transactional data efficiently and accurately across systems, Oracle
GoldenGate converts the captured data into an Oracle Canonical Format in trail files. With
both source and target trail files, Oracle GoldenGates unique architecture eliminates any
single point of failure and ensures that data integrity is maintainedeven in the event of a
system error or outage.
Oracle GoldenGate Concepts
LAN/WAN
Internet
TCP/IP
Bi-directional
Capture
Trail
Pump
Delivery
Trail
Source
Database(s)
Target
Database(s)
Capture: Committed transactions are captured (and can be filtered) as
they occur by reading the transaction logs.
Trail: Stages and queues data for routing
Pump: Distributes data for routing to target(s)
Route: Data is compressed,
encrypted for routing to target(s)
Delivery: Applies data with transaction
integrity, transforming the data as required
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Using ODI with Oracle GoldenGate
Oracle Data Integrator can be combined with Oracle
GoldenGate (OGG), which provides a cross-platform data
replication and changed data capture.
Oracle GoldenGate works in a similar way to Oracles
asynchronous change data capture but handles greater
volumes and works across multiple database platforms.
GoldenGate uses its own data logging (or trails) to record
DML and DDL activities on a source database, which it
then replays on staging and target platforms to replicate
changes in real-time.
Using OGG technology with ODI is beneficial, especially if
you are hitting replication performance limits.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Oracle GoldenGate captures and replicates changes from the source system, typically into a
staging schema on the target Data Warehouse platform. This staging schema contains a
replicated copy of the source table and the structures that are used by ODIs own Changed Data
Capture (CDC) framework. ODI then picks up the changes in the structures of the ODI CDC
framework, transforms the data, perhaps joins it to look up other tables, and then loads it into the
target schema. This technology does not require ODI to be changed in any way, and use
GoldenGate CDC with the regular existing ODI CDC framework.
Using ODI and GoldenGate Together
Extract
Target DB
Target Tables
Source trail
files
Staging trail
files
Staging DB
Replicated
Source Tables
ODI CDC
Framework
WAN
Transactional RDBMS
Source Tables
ODI
Interfaces
Datapump
Replicat
Oracle GoldenGate
ODI
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
To initialize CDC process with Oracle GoldenGate, you have to create the package, which
consists of a perpetual loop that is triggered by the log entry of a new journal and processes new
changes with the CDC Load Customer interface. Execute this package to initialize ODI CDC with
GoldenGate.
Initialize CDC Process and Perform Change
Operations with Oracle GoldenGate
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Working on ODI Projects: Best Practices
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
In many ODI objects (procedure, variables, interfaces, and so on), you can add expressions and
SQL codes. A very common mistake is to enter qualified object names, as in the following
example that drops a TBL_001 table in the staging schema MY_SCHEMA:
DROP TABLE MY_SCHEMA.TBL_001
If you run this code in the production environment where the staging schema is called
PRD_MY_SCHEMA, your code does not work. Note that the schema names are defined in the
topology, and the contexts would handle getting the right schema name depending on the
execution context. The Substitution Methods (aka OdiRef API) exist to leverage in your code the
metadata stored in ODI and make the code context-independent.
In this case, it is assured that the qualified table name is generated according to the context you
are running your code into.
Using Substitution Methods, the generic code would be:
DROP TABLE <%odiRef.getObjectName("L", "TEMP_001","W")%>.
1. Use Context-Independent Design
Do not use qualified object names in expressions (DROP
TABLE MY_SCHEMA.TBL_001). This is a context-
dependent code.
Use Substitution Methods (for example, DROP
TABLE <%odiRef.getObjectName("L",
"TBL_001","W")%>).
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Procedures allow you to perform very complex actions, including SQL statements. In addition,
they allow you to use source and target connections, and support data binding. In short, you can
move data by using procedures.
Developers who feel at ease with SQL may be tempted to code their transformations and data
movements within procedures instead of using interfaces.
There are several issues with using procedures in such a way:
Procedures contain manual code that needs to be maintained manually.
Procedures do not maintain cross-references to other ODI artifacts such as datastores,
models, and so on, making their maintenance very complex compared to interfaces.
Procedures should never be used to move and transform data. These operations should be
performed by using interfaces.
2. Use Procedures Only when Needed
Procedures contain manual code that needs to
be maintained manually.
Procedures do not maintain cross-references to other ODI
artifacts such as datastores, models, and so on.
Their maintenance is complex compared to interfaces.
Procedures should never be used to move and transform
data. These operations should be performed by using
interfaces.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Project developers of data integration sometimes do not take into account the quality of their data.
This is a common mistake, as data integration itself may be to move and transform erroneous
data, and propagate it over the applications.
ODI allows you to enforce data quality of source data by using static checks as well as the quality
of the data before it is pushed into a target via flow checks. By using both these checks, you can
make sure that the quality of the data is improved or enforced when the data is moved and
transformed.
3. Always Enforce Data Quality
It is a common mistake to not take into account the quality
of data.
Enforce data quality of source data by using static checks.
Enforce quality data before it is pushed into a target by
using flow checks.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
In a package, you can sequence any number of steps. Every single step in a package may fail for
some reason (for example, target or source database is down). You must always consider these
different error cases when designing packages.
4. Handle Error Cases in Packages
Every single step in a package may fail for some reason.
For example, target or source database is down
You should always consider these different error cases
when designing packages.
You achieve this by having not only OK path (green arrows)
in the packages but also the KO path (red arrow)
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
The KM Choice is critical when designing interfaces. The KM Choice conditions the features
available and the performances of an interface.
Do not begin your interface with complex KMs. Choosing for example, technology-specific LKMs
by using loaders may lead to a nonworking interface because the loader configuration or the
installation is not correct. A safe choice for starting to work is to use generic KMs (usually SQL
KMs) that work in most cases. After designing your first interfaces with these KMs, you can start
switching to technology-specific KMs (read their descriptions first) and leverage all the features of
these KMs.
Avoid over-engineering interfaces: KMs with extra features have an extra cost in terms of
performance. For example, performing simple insert is faster than performing an incremental
update. If you are deleting the data in the target before integration, using incremental update is
over-engineering, and causes some performance loss. Use the KM that fits your needs.
Similarly, activating or deactivating some of the KM features may add extra steps that may
degrade the performance. Default KM options are sufficient for running the KM provided out of the
box. After running the KM with default options, it is always good to review the options and see
whether some of them can be changed for your needs. The KM and Option description are a good
documentation for understanding the optimization of KM usage.
5. Choose Right Knowledge Module
The KM Choice is critical when designing interfaces. The
KM Choice conditions the features available and the
performances of an interface.
Do not begin with complex KM
A safe choice for starting to work is to use generic KMs
(usually SQL KMs) that work in most cases.
When switching to technology-specific KMs, read their
descriptions first and leverage all the features.
KMs with extra features have an extra cost in terms of
performance.
Default KM options are sufficient for running the KM provided
out of the box.
Activating or deactivating some of the KM features may add
extra steps that may degrade the performance.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
6. Other Best Practices
Do not import all the KMs, only import as required.
Make sure that all repositories are created with strictly
different IDs (even on different sites). Document the
process for moving objects between repositories by using
import, export, or versioning.
Two repositories with the same ID may possibly contain
objects with the same internal ID, which mean the same
object for ODI.
Transferring objects between these repositories is similar to
copying files with same names across different directories,
and may lead to object replacement.
Add a description for each object (Variable, Interface, and
so on) for better understanding of the relationship and
usage of the object.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Answer: a
Explanation: This technology does not require ODI to be changed in any way, and uses
GoldenGate CDC with the regular existing ODI CDC framework.
Quiz
When using Oracle GoldenGate with ODI, you do not have to
change ODI CDC. You just use GoldenGate CDC with the
regular existing ODI CDC framework.
a. True
b. False
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
Summary
In this lesson, you should have learned how to:
Describe typical integration processes
Describe different integration strategies
Implement integration strategies with staging area on the
target
Implement integration strategies with staging area different
from the target
Maintain the integrity of data
Describe best practices for working on ODI projects
Implement real-time data integration by using ODI with
Oracle GoldenGate
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.
O
r
a
c
l
e

U
n
i
v
e
r
s
i
t
y

a
n
d

(
O
r
a
c
l
e

C
o
r
p
o
r
a
t
i
o
n
)

u
s
e

o
n
l
y
.
T
h
e
s
e

e
K
i
t

m
a
t
e
r
i
a
l
s

a
r
e

t
o

b
e

u
s
e
d

O
N
L
Y

b
y

y
o
u

f
o
r

t
h
e

e
x
p
r
e
s
s

p
u
r
p
o
s
e

S
E
L
F

S
T
U
D
Y
.

S
H
A
R
I
N
G

T
H
E

F
I
L
E

I
S

S
T
R
I
C
T
L
Y

P
R
O
H
I
B
I
T
E
D
.

D78191GC10 SG

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

D78191GC10 SG

Uploaded by

Copyright:

Available Formats

Oracle Data Integrator 11g:

Advanced Integration and

You might also like