ODI 12c Workshop Syd 20140820 Slides

1
Copyright 2013, Oracle and/or its affiliates. All rights reserved.
Oracle Data Integrator Overview

Eddie Ho,Senior Sales Consulting Manager, DIS
Prashant Kompella, Senior Sales Consultant
20th Aug, 2014
2
Agenda
8.45
Arrival, Registration & Housekeeping & Introductions
9.00
Overview of ODI, Whats New, Demo & introduction to the Lab environment
10.00
Labs session 1 Basic mapping, Distinct, Expression, Sort
10.45
<Break>
11.00
Labs session 2 Join, Filter, Lookup, Unions, Reusable mappings
12.30
< Lunch >
1.00
ODI Concepts & Product Update
2.00
Labs session 3 Aggregate, Split, Multiple targets and Parallel loads
3.00
< Break >
3.15
Labs session 4 Debugging, Standalone Agent, Scheduling
4.30
Q&A, / wrap-up / next steps
4.45
Close
Logistics
Clear_Guest
<#>
Introductions
Your name
Your company
Your role
Your objectives
<#>
Agenda
8.45
9.00
10.00
10.45
<Break>
11.00
12.30
< Lunch >
1.00
2.00
3.00
< Break >
3.15
4.30
4.45
Close
Oracle Data Integration

Overview
Data Integration Trends

Data Integration Tools Today Need to Do More
Big Data is the new normal
Real-time data jumps to the forefront

Application data now lives in the Cloud and
in a multitude of disparate environments

Conventional ETL tools are struggling to
meet performance demands
New Requirements for Data Integration
Integrated and
High Productivity
Tooling
Any Data Latency,

Real-time
High Performance,
High Availability
Cloud Ready
Any Data,
Any Source
Oracle Data Integration Solutions

Best-in-class Heterogeneous Platform for Data Integration
Oracle
Applications
Custom
Applications
MDM
Applications
Business
Intelligence
Activity
Monitoring
SOA
Platforms
Comprehensive Data Integration Solution

SOA Abstraction Layer
Process Manager
Data Services
Data Federation
Oracle GoldenGate
Oracle Data Integrator
Oracle Data Quality
Real-time Data
Log-based CDC
Data Replication
Data Verification
ELT/ETL
Data Transformation
Bulk Data Movement
Data Lineage
Data Profiling
Data Parsing
Data Cleansing
Match and Merge
Exadata
10
Service Bus
Data Warehouse/
Data Mart
OLTP
System
OLAP Cube
Flat Files
Web 2.0
Web and Event

Services, SOA
10
10

Overview
11
11
Oracle Data Integrator 12c

Bulk Data Processing and Data Transformation
Cloud
Apps

High Performance E-LT
Declarative Design
Database
Extensible Knowledge Modules
Certified for leading

technologies to deliver fast
time to value
High-performance, low cost
of ownership E-LT
architecture
Lightweight deployment
Big Data
CEP Data Services

Capable of Integrating Structured &
Unstructured Data
12
Flexible, easy to enrich

functionality
12
Industry Leading Performance of Oracle ODI

Extremely Fast Execution and Reduced Cost
Conventional ETL Architecture
Extract Transform
Load
Next Generation Architecture
E-LT
Extract
Load
Transform
13
ODI
Peformance
Transform
13
Declarative Design
Improved Developer Productivity For Faster Time to Solution
Easy and universal for designing
simple to complex mappings

Robust and Reusable: Create
ODI Declarative Design
Mappings once for any physical design

Quick to define and refactor maps,
high degree of automation

Extensible model for any data
integration mechanism
1
Define
What
You Want
2
Automatically
Generate
Dataflow
Benefits
Significantly reduces the learning curve
Define How: Knowledge Modules
Shortens implementation times
Simpler maintenance
14
14
Knowledge Modules
Simpler Physical Design and Shorter Implementation Time
Pluggable Knowledge Modules Architecture
Reverse
Engineer
Metadata
Journalize
(CDC)
Load from
Source to
Staging
Check
Constraint
Integrate,
Transform
Data
Service
Sample Out-of-the-Box Knowledge Modules (100+)

Oracle
Non-Oracle
GoldenGate
Oracle
Utilities
Oracle
DBLINK
Siebel CRM
E-Business
Suite
Oracle Merge
Oracle
Web Services
Oracle
Spatial
Hadoop
IBM DB2
SAP ABAP
SQL Server
Sybase
NoSQL
JMS
Teradata
Key Architecture
Benefits:
15
Faster development and simpler maintenance using templates

Easy to extend and add new best practices
Enforces predictability and reduces cost of ownership
15
Extended Connectivity
Leverages Existing IT, Faster Implementation
Big Data: Hadoop support
Best for Oracle: Merge, Spatial, MultiTable Insert, Optimizer Hints and more
Real-Time: CDC with GoldenGate
Applications: E-Business Suite, Siebel,
PeopleSoft, JD Edwards Enterprise
One, JD Edwards World, SAP ERP and
SAP BW
Heterogeneous: Optimizations for all
major RDBMS: IBM DB2, Microsoft SQL
Server, Teradata, Netezza etc.
16
Legacy
Applications
Databases
Unstructured
JMS
16

in Data Integration
Solutions
17
17
Oracle Real-time EDW Architecture

Application Databases
Staging
ODS
Data Mart
ETL
Real-time
Replication of
transactional
Changes
by GoldenGate
18
High frequency
micro-batches
of deltas by Oracle
Data Integrator
(ODI) + GoldenGate
integrated solution
Reporting/Analytics
Load
Incremental
updates
of Dimensions
and
Facts by ODI
CDC framework
18
EDQ and ODI: Overview

Sources
Oracle Data
Integrator
Enterprise Data
Quality
19
Data quality
firewall
Target(s)
E.g. Data
Warehouse
such as Exadata
Data Profiling
Automated Processes
Measure Ongoing Data Quality
Analyze and understand data

to build ODI mappings
De-duplication, complex
cleansing and parsing
invoked in ODI workflow
Assess quality of data

in target system. How well
is ETL working?
19
EDQ and ODI: Comprehensive Data Quality Process

1
Profile Data
- Continuous Quality
Monitoring
- Quality Alerts
Create new
Data Quality Rules

E-LT/ETL Process
Sources
Targets
- Add Data Quality

to E-LT/ETL Flow
Oracle Enterprise Data Quality
Parsing
20
Standardization
Cleansing
Matching
Merging
20
ODI Concepts & Product

Update
21
21
ODI EL-T Architecture Overview

Runtime (scheduling)
Design/Admin/Monitoring
ODI
Studio
ODI Agent
(Java process)
ODI
Studio
Repository
(design-time+
Run-time)
Flat
Files
Runtime (scheduling)
ODI Agent
(Java process)
Transformation
on DB (selected
By developer)
Flat
Files
22
22
ODI Modules
ODI Designer
Designer
Reverse-Engineer
Develop Projects
Release Scenarios
Operator
Operate production
Monitor sessions
Topology Manager
Define the infrastructure
of the IS
Security Manager
Manage user
privileges
Java Most Platforms

DB - Most ISO-92
Repository
23
<#>
23
Run-Time Components
Designer
Reverse-Engineer
Develop Projects
Release Scenarios
Java - Any Platform
Operator
Operate production
Monitor sessions
Monitor sessions
View Reports
Submit Jobs
Repository
Read sessions
Write reports
Any ISO-92 RDBMS

Agent
Handles schedules
Orchestrate sessions
Java - Any Platform
Return Code
Lightweight
Distributed Architecture
Execute Jobs
Information System
24
24
ODI 12c Mapping

Declarative and Flow combined
Easy for moderate use cases
Shows relationships of source
25
tables
Abstracts out physical
implementation from logical map
Quick to define and refactor
maps
Universal for big and complex
use cases
Extensible model for any data
integration mechanism
25
Mapping Components
ODI 12.1.3 Out-of-Box Components:
26
Aggregate
Dataset
Distinct
Expression
Filter
Join
Lookup
Pivot, Unpivot
Set
Sort
Split
Subquery
Table Functions
26
Multiple Targets
A mapping can contain multiple
targets
Any component can map result to
multiple downstream components

Split component allows to split input
logically based on expressions
27
27
Reusable Mappings
Capture common logic (sources, targets, transformations)
Reuse logic in mappings
28
28
Step by Step Debugger

Overview
Provide control to the user on the
Session Execution Flow
Pause, Resume, Run to next Step, etc.
Define breakpoints on Steps in

Packages
Toggle breakpoints in Sessions
Query data through Agent (gives
access to uncommitted data)
29
29
Types of ODI Knowledge Modules

There are six different types of knowledge modules:
KM Type
Models
Interfac
es
LKM
IKM
CKM
RKM
JKM
SKM
30
Description
Loading
Assembles data from source datastores to the staging area.
Integration
Uses a given strategy to populate the target datastore

from the staging area.
Check
Reverseengineering
Journalizing
Checks data in a datastore or during an integration process.
Web Services
<#>
Retrieves the structure of a data model from a database. Only

needed for customized reverse-engineering.
Sets up a system for Changed Data Capture to reduce the
amount of data that needs to be processed.
Defines the code that will be generated to create Data Web
Services (Exposing data as a web service)
30
E-LT Architecture with Data Integrator

Comprehensive, Lightweight Data Integration
Key Architecture Benefits: 100% Java, Open APIs, very fast E-LT
E-LT Architecture uses DBMS CPUs, not the Application CPUs
Open APIs, Open Metadata, Open XML Knowledge Modules
Apps
Business Intelligence
ODI Agent
ODI Agent may be deployed
in any part of the
architecture
LKM
JKM A
C$_0
LKM
C$_1
I$
E$ (Errors)
IKM
RKM
Transform
Extract-Load
31
IKM
D
File
C
ODI natively integrates with

all major Data Warehouses
<#>
CKM
Check-Load
31
Todays hands-on lab

environment
32
32
Hands-On Lab Scenario

Orders Application: An application for tracking
customer orders, hosted in Oracle and used as
a source.
Logical
1
Orders
Application
Lookup
Sales
Administration
Parameters
(Sales Source)
Join
Sales Administration: The administration or

tracking of sales, hosted in another supplied
database. This data warehouse is populated
with our transformations.
33
Physical
Parameters (File): Flat files (ASCII) issued from

the production system containing a list of sales
representatives and the segmentation of ages
into age ranges.
33

What you are about to see
ODI Studio
Easy to use Navigation
Centrally Managed
Metadata
Logically &
Physically
Abstracted
Orders
Application
34
Parameters
(Sales Source)
Sales
Administration
34

Drag and Drop Data

Stores
ODI Studio
Graphical Lookup
Definition
Graphical Join
Definition
35
35

ODI Studio
Session logging
Record tracking
Predefined Integration
via Knowledge Modules
36
36
Agenda
37
8.45
9.00
10.00
10.45
<Break>
11.00
12.30
< Lunch >
1.00
2.00
3.00
< Break >
3.15
4.30
4.45
Close
37
Hands-On Lab
Sessions 1 & 2
38
38
ODI (More) Advanced

Concepts
39
39
ODI Model & Logical/Physical Schema across

Environments
Project 1
Project 2
Project 2
Model A
Model B
Model C
Model A
Model B
Model C
LS X
LS Y
LS Z
LS X
LS Y
LS Z
Testing
40
Project 1
ODI Context
Production
40
Logical/Physical Views for Productivity Improvement
ETL jobs are

developed on
logical views
- Portable across
environments
Logical switching
to different
environments
41
41
ODI Tight Binding vs Loose Binding
Tight binding
Almost possible to change a mappings model
42
Loose binding
Change through Context
42
Types of ODI Knowledge Modules

There are six different types of knowledge modules:
KM Type
Models
Interfac
es
LKM
IKM
CKM
RKM
JKM
SKM
43
Description
Loading
Assembles data from source datastores to the staging area.
Integration
Uses a given strategy to populate the target datastore

from the staging area.
Check
Reverseengineering
Journalizing
Checks data in a datastore or during an integration process.
Web Services
Retrieves the structure of a data model from a database. Only

needed for customized reverse-engineering.
Sets up a system for Changed Data Capture to reduce the
amount of data that needs to be processed.
Defines the code that will be generated to create Data Web
Services (Exposing data as a web service)
43
Which KMs for What Flow?

When processing happens between two data servers, a data transfer
KM is required.
Before integration (Source Staging Area)
Requires an LKM, which is always multi-technology. E.g. SQL Server to
Oracle
At integration (Staging Area Target)
Requires a multi-technology IKM
When processing happens within a data server, it is entirely performed
by the server.
A single-technology IKM is required.
No data transfer is performed
44
44
Case #1
Using the Target as the Staging Area
Target (Oracle)
Source (MS SQL)
Staging Area
ORDERS
LKM_1
ORDER_LINES
LKM SQL to Oracle
LKM_2
PRODUCT
File
45
IKM_1
C$_0
IKM_1
C$_1
I$_SALES
SALES
IKM Oracle Incremental Update
IKM Oracle Incremental Update
LKM File to Oracle (SQLLDR)
45
Case #2
Using a third server as the Staging Area
Sunopsis Memory Engine
IKM SQL to SQL Append
Source (MS SQL)
Staging Area
ORDERS
LKM_1
IKM_1
SALES
C$_0
ORDER_LINES
IKM_1
LKM SQL to SQL
PRODUCT
File
LKM_2
C$_1
Target (Oracle)
I$_SALES
LKM File to SQL
46
46
Case #3
Using the Source as the Staging Area
Source (MS SQL)

ORDERS
IKM_1
ORDER_LINES
C$_0
IKM SQL to
SQL Append
Staging Area
IKM_1
IKM_1
Target (Oracle)
SALES
I$_SALES
C$_1
LKM_1
PRODUCT
File
47
LKM File to SQL
47
Which KMs for What Flow?

Four possible arrangements:
Source
Loading phase
Multi-tech LKM
Multi-tech LKM
(No LKM needed)
Staging area on source
(No LKM needed)
Staging area
Integration phase
Multi-tech IKM
Target
Single-tech IKM
Staging area on target
Multi-tech IKM
Single-tech IKM
Source, staging area and target in same location
48
48
ODI Check Knowledge Module (CKM)
Static Control Check Data in Table against Constraints
49
49
ODI Check Knowledge Module (CKM)
Static Control Check Transformed data Before Loading Target
Records in error table can be

fixed and merged back to next
ETL batch
50
50
ODI CDC Framework Overview (JKM)

Downstream
Data Consumers
Application Table XXX
Data mart 01
CDC Journal Table

J$<table name>
Primary Key
columns
window_id
123
44848
124
44848
Subscriber
Datamart01
Datamart02
Data mart 02
DML
Changes
51
CDC Journal View

JV$D<table name>
51
Comparison of 2 ODI CDC Frameworks

CDC Journal Table
J$<table name>
DML on table
(Classic)
Primary Key
columns
window_id
123
Empty
Insert by
per-row
table trigger
CDC Journal Table

J$<table name>
GoldenGate
replicat
Insert by OGG
52
Primary Key
columns
window_id
123
From
SEQ+RBA of
trail file
52
3 Simple Steps to Create ODI CDC Infrastructure
1. Create a new subscriber at ODI model
2. Add each table needed

to CDC
3. At last, Start Journal for a model
53
53
Each Table in ODI Mapping Needs 2 Parameters
54
54
How ODI Handles Delete Records at Source
JV$D<table>.JRN_FLAG = I
Insert/update at source
JV$D<table>.JRN_FLAG = D
Delete
Its up to data model design for synchronizing changes such as delete
records from ODS by ODI
55
55
ODI Job Packaging for CDC Framework

Process changed records in marked window
Light-weight check of data changes
(independent of change volume)
Lock/Extend
- Mark journal records
56
Unlock/Purge
- Purge processed journal records
56
Enterprise Data Quality: Governed Data Quality

Data
Stewards
Data
Analysts
57
Data
Stakeholders
57
EDQ Profiling Example
58
58
Build-out Full DQ Process
59
59
Key Feature: Pre-built processors

Comprehensive DQ Functionality with a Single User Interface and Repository
60
60
EDQ for DW/BI

1.
2.
3.
4.
Data in DW is Fit for Purpose
ODI or GoldenGate move data into DW

ODI maps data into Staging tables
EDQ cleanses data
ODI moves from staging into production tables
Golden Gate
BI/Analytics
Source of truth
Replica
source
tables
ODI
ODI
BI/analytics
Staging
tables
ODI
Data Sources
Production
Tables
Single source
of truth for
reference
EDQ
Parsing, Cleansing,
Standardization, Matching
61
61
Latest ODI Updates (12.1.3)
62
62
ODI 12.1.3 New Features

FIPS Compliance
XML Driver Enhancements (e.g. Recursion)

JSON Support
Hadoop Sqoop/Hbase Integration
Multi-Threaded Target Table Load in ODI Engine

Improved Control for Scenario/Load Plan Concurrent Execution
Create New Mode/Topology Objects (without building topology from
physical/logical to model)
More details can be found from viewlet
http://www.oracle.com/webfolder/technetwork/tutorials/obe/fmw/odi/odi_12c/odi_12.1.3_nf/presentation.
html
63
63
New Features in ODI 12.1.2

Improved User
Experience and
Productivity
Core Features for
Better Performance
Enterprise Deployment
Improved
Manageability
Deeper Oracle Stack
Integration
64
Declarative Flow-Based User Interface

Reusable Mappings
Step By Step Debugger
Parallel Target Table Load
In-Session Parallelism
Low Overhead Session
Integration with Common Admin Model

Integration with OPSS Enterprise Roles
Integration with Enterprise Manager Cloud Control
Integration with Oracle Warehouse Builder

Integration with Oracle GoldenGate
64
Oracle Warehouse Builder Integration/Migration

Overview
Goal
Provide a migration path to OWB users
Integrate OWB and ODI runtime for phased
migration
How
OWB repositories can now be managed in
Topology
OWB Jobs can be executed and monitored
in ODI
OWB to ODI migration utility available
(http://docs.oracle.com/middleware/1213/od
i/install-migrate-owb-odi/understanding.htm)
65
65
Management Pack for Oracle Data Integrator

Key Features
Discovery of ODI components
Agents: Standalone and JEE
Repositories
Source and Target databases
SLA Management
Alerts creation
Notifications: email, SNMP, etc.
Monitoring
Availability Monitoring
ODI homepage to quickly assess
infrastructure status
Performance Monitoring
ODI components performance
66
66
Management Pack for Oracle Data Integrator

Key Features
Performance Monitoring (continued)
Sessions execution statistics
Historical metrics
SQL Drill Down (requires Oracle DB Diagnostics and Tuning packs licenses)
Access database execution statistics
67
67
ODI Big Data SQOOP Load

Fast Load between
databases and Hadoop

Database can be source
or target
HDFS, Hive, or HBase
supported
Parallelized load using a
data partitioning scheme
68
Parallel
JDBC Queries
Source Database
SQOOP
HDFS
SQOOP
HDFS
SQOOP
HDFS
68
In-Session Parallelism
Overview
ODI 11g
Goal
Allow the execution of tasks and
steps in parallel rather than
sequentially
SrcSet1
SrcSet2
SrcSet3
Integration
Total Execution Time
How
Introduction of a new Runtime
concept: Execution Units
In 12.1.2 loading of C$ tables can
be executed in parallel
ODI 12.1.2
SrcSet1
SrcSet2
Integration
SrcSet3
Total Execution Time
69
69
Multi-Threaded Target Load (12.1.3)
Data Server settings:
Fast Load through parallel
JDBC connections
Configurable per Data
Server
Parallel
Load
Threads
JDBC
Source
70
ODI Agent
Target
Database
70
Enterprise Role Integration

Introduction
Goal
Tighter integration with LDAP security
How
Introduction of ODI Roles
ODI Roles can be mapped to Enterprise Roles
Enterprise Role inherits ODI Role privileges
71
71
Oracle Data Integration
Oracle Warehouse
Builder Product Update
72
72
OWB Support Policies

Customer Downloadable OWB Support Policies:
http://www.oracle.com/us/support/library/lifetime-support-technology-069183.pdf
Release
GA Date
Premier
Support Ends
Extended
Support Ends
Sustaining Support Ends
OWB 9.2
Jul 2003
Oct 2007
Not Available
Oct 2010
OWB 10.1
Apr 2004
Jul 2007
Not Available
Jul 2010
OWB 10.2
Jul 2005
Jul 2010
Jul 2013
Indefinite
OWB 11.1
Aug 2007
Aug 2012
Aug 2015
Indefinite
OWB 11.2
Sep 2009
Current Dates:
Jan 2015
Current Dates:
Jan 2018
Indefinite
Future Dates:
Same as DB12
Future Dates:
Same as DB12
For more Information about Support Levels:

http://www.oracle.com/us/support/lifetime-support/index.html
73
73
Best Practices for OWB to ODI Migrations

ODI 12c supports a phased migration approach
Execute and monitor existing OWB 11gR2 jobs from ODI 12c
Use the migration utility to convert existing OWB 11gR2 objects into
their ODI equivalents
Start new developments using ODI 12c
Understand that OWB and ODI are different ETL tools

There is no free lunch when migrating a technical migration
between different tools will never be 100% perfect or predictable
74
74
Invoking OWB 11.2 Processes from ODI 12c

New OWB Runtime Repository
technology available in Topology

OWB processes can be invoked
using the OdiStartOwbJob tool in

Packages
OWB audit information is available
in Operator, ODI Console and

Enterprise Manager
75
75
OWB to ODI Migration Utility

Overview
Versions supported
OWB 11.2.0.4 to ODI 12c
Design-time metadata migration only

Data Objects
Mappings
Separate command line utility

uses OWB SDK and ODI SDK
76
76
Hands-on Labs
77
77
78
78

ODI 12c Workshop Syd 20140820 Slides

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

ODI 12c Workshop Syd 20140820 Slides

Uploaded by

Copyright:

Available Formats

1

Copyright 2013, Oracle and/or its affiliates. All rights reserved.

Oracle Data Integrator Overview

Copyright 2013, Oracle and/or its affiliates. All rights reserved.

Arrival, Registration & Housekeeping & Introductions

Labs session 1 Basic mapping, Distinct, Expression, Sort

Labs session 2 Join, Filter, Lookup, Unions, Reusable mappings

< Lunch >

ODI Concepts & Product Update

Labs session 3 Aggregate, Split, Multiple targets and Parallel loads

< Break >

Labs session 4 Debugging, Standalone Agent, Scheduling

Q&A, / wrap-up / next steps

Copyright 2013, Oracle and/or its affiliates. All rights reserved.

Copyright 2013, Oracle and/or its affiliates. All rights reserved.

Copyright 2013, Oracle and/or its affiliates. All rights reserved.

Arrival, Registration & Housekeeping & Introductions

Labs session 1 Basic mapping, Distinct, Expression, Sort

Labs session 2 Join, Filter, Lookup, Unions, Reusable mappings

< Lunch >

ODI Concepts & Product Update

Labs session 3 Aggregate, Split, Multiple targets and Parallel loads

< Break >

Labs session 4 Debugging, Standalone Agent, Scheduling

Q&A, / wrap-up / next steps

Copyright 2013, Oracle and/or its affiliates. All rights reserved.

Oracle Data Integration

Copyright 2013, Oracle and/or its affiliates. All rights reserved.

Data Integration Trends

Big Data is the new normal

Real-time data jumps to the forefront

in a multitude of disparate environments

meet performance demands

Copyright 2013, Oracle and/or its affiliates. All rights reserved.

New Requirements for Data Integration

Any Data Latency,

Copyright 2013, Oracle and/or its affiliates. All rights reserved.

Oracle Data Integration Solutions

Comprehensive Data Integration Solution

Oracle Data Integrator

Oracle Data Quality

Copyright 2013, Oracle and/or its affiliates. All rights reserved.

Web and Event

Oracle Data Integrator

Copyright 2013, Oracle and/or its affiliates. All rights reserved.

Oracle Data Integrator 12c

Oracle Data Integrator

Extensible Knowledge Modules

Certified for leading

CEP Data Services

Copyright 2013, Oracle and/or its affiliates. All rights reserved.

Flexible, easy to enrich

Industry Leading Performance of Oracle ODI

Next Generation Architecture

Copyright 2013, Oracle and/or its affiliates. All rights reserved.

simple to complex mappings

ODI Declarative Design

Mappings once for any physical design

high degree of automation

Define How: Knowledge Modules

Shortens implementation times

Copyright 2013, Oracle and/or its affiliates. All rights reserved.

Sample Out-of-the-Box Knowledge Modules (100+)

Copyright 2013, Oracle and/or its affiliates. All rights reserved.