Oracle Data Integrator

Kiran Kumar R, PMP

INTRODUCTION

What is ODI?
Comprehensive data integration platform
Extraction, transformation and load of data, to and
from databases, events and Web services
Batch and real-time loading
Java-based, hot-pluggable, database independent
5
Data Movement and Transformation from Multiple
Sources to Heterogeneous Targets
BENEFITS KEY DIFFERENTIATED FEATURES
1. Performance Heterogeneous E-LT
2. Flexibility Event-Driven Platform
3. Productivity Declarative Design
4. Open 100% Java and SOA Native
5. Hot-Pluggable Knowledge Modules
Customer Value Proposition
Strategy: Best-of-Breed Core Architecture & Foundation
Previously Known as Sunopsis
Sunopsis were an ETL tool vendor originally from France,
with offices in the USA and UK
Two major products
Sunopsis Data Conductor (database integration)
Sunopsis Active Integration Platform (event and serviced-based
integration)
Sold on its platform independence, Java architecture and EAI
capabilities
Acquired by Oracle in 2006
ODI Positioning
Not a replacement for Oracle Warehouse Builder
OWB has Oracle data warehouse modeling features
OWB is the obvious choice when target is an Oracle database
ODI supplements OWB in the following scenarios:
Target database is non-Oracle (DB2, Teradata, XML, SQL Server etc)
Sources include Web Services, XML, SOAP etc
Data integration is complex / real-time
Multi-platform CDC / SCD / Bulk load and insert / MDM and Data Hubs
Batch windows require bad source data to be filtered out before loading
Adds functionality to the Oracle DW ETL stack
ODI and OWB likely to code-share/merge in future
In short term, possible replacement for OWB Enterprise ETL Option
ODI and OWB Similarities and Diffs
Both feature mappings (in ODI, interfaces) and process flows
(in ODI, packages)
Both are written in Java
Both use the target database to perform transformations (ELT)
But
ODI allows non-Oracle databases to be targeted
ODI extends integration capabilities to the SOA environment
ODI splits mappings into business rules and technical
implementation
OWB, however, has better support for DW and OLAP
And is (mostly) free, so is the default Oracle-Oracle choice
Market Position
Oracle Data Integrator Architecture
Overview of ODI Studio
17
Navigators
for each
functional
area.
ODI operations integrated in
IDE: Menu, Gallery, etc.
Intuitive multi-editors support
close all/save all, docking,
etc.
Property inspectors,
thumbnails and IDE windows
fully used as needed.
Accordions
for each
sub-area.
Popup menus
and tree
markers.
Interface Editor using
diagraming.
Interface Editor / Quick Edit
18
Same Interface
Two views
Interface Editor
19
Diagram zoom
and autolayout.
Datasets
management
Lookup
wizard
Errors and
Autofix.
Lookup
Datasets
appear here.
Properties
inspector for
selection.
Sortable and
Selectable columns.
DnD from sources to
mapping.
Thumbnail: birds
eye view of the
diagram.
22
Oracle Data Integrator
SOA Enabled
23
SOA Enabled: Data Access Services
Generate and share data
access services
1. Generate and deploy data services
2. Test data services
3. Leverage data services in your SOA
infrastructure
SOA Infrastructure
Services
Data Access
Transform
Business
Business
Processes
ESB
24
SOA Enabled: Data Flow Services

Expose transformations as
Web Services
1. Orchestrate data flows
2. Publish data flows as web services
in your SOA infrastructure
SOA Infrastructure
Services
Data Access
Transform
Business
Business
Processes
ESB
Bulk
Transf
.
Extended Capabilities
Same philosophy as OWB Use the Database as the ETL engine
ODI supports heterogeneous databases, not just Oracle
Built for SOA environments
Support for Web Services etc.,
Supports batch, event-based and real-time integration
Data Integrity Controls create a data firewall
Reduces data prep time by not processing erroneous data
Extensible through Knowledge Modules
Change Data Capture
Slowly Changing Dimensions
Bulk load
Data Integrator Overview








Solution Talk A Case Study


27
Solution Talk A Case Study
Industry
BFSI
Customer Profile
Our customer is the largest BFSI company in India, providing a wide range of
banking operations and property and casualty insurance products together with
accidental injury insurance and short-term health insurance products
With 31 sub-companies distributed across the country, this company has been
developing a complicated IT infrastructure and an analysis system with a huge
volume of data
All the database systems are Informix
Business Problem
KPI reports are significantly delayed due to poor system performance, leading to
the KPI system unreliable for decision-making
The current system is too overloaded to provide internal users with stable
service based on existing business, not mentioning supporting new business
Maintenance cost is augmenting due to sub-optimal DW development approach,
which has become a manifest issue during the economic downturn
Product Used
Oracle Database Enterprise Edition 11.1.0.6 with RAC, Partitioning Options
Oracle Data Integrator 10.1.3.4








Solution Talk What Challenges
Were Responded to ??
Customer Challenges -Existing System
Architecture
Distributed IT infrastructure
31 OLTP Informix DBs are
distributed across India, with a
central Informix DB

Performance-thirsty
transformation
Data is extracted (from 31 Informix
into ODS) and transformed (finally
into Report Data Layer) in real time
or periodically.

Non-standardized
implementation
Data extraction and transformation
are implemented via code such as
C,SQL, etc, no ETL tool here.







Solution Talk What did We
Achieve ??

Copyright 2008, Oracle and/or its affiliates. All rights reserved. 31
Achievements ODI Topology Manager
With ODI Topology Manager, we
can easily define all kinds of
data sources and target
technologies.
In our case, multiple Informix
databases can be defined as
data sources thereby UI friendly
data consolidation is achieved.
Here we define
Informix as data
source
Achievements ODI Designer
Here we design
ETL processing
ODI Designer is an integrated
GUI for users to design ETL
processes.
In our case, complicated
transformations can be defined
in a high efficient manner. Highly
optimized codes are generated
automatically by tool.
Achievements ODI Operator
With ODI Operator, we can
monitor and execute the ETL
processes conveniently.
In our case, ODI Operator helps
improve the maintenance
efficiency. For instance, users
can identify the ETL steps with
errors or warnings and pinpoint
the problems efficiently.
Key Benefits - Delivered
Key Focus Areas Benefits
High Performance
No analytical report is delayed
The overall performance is dramatically improved and the
hardware resources are utilized with more efficiency
Additional resource is available for processing new business
ETL Efficiency
New business can be developed timely
Distributed data can be easily consolidated
Data processing is easily and flexibly implemented
Transformation code is optimized with high performance









Solution Talk What did We
Propose ??


Oracle Proposed Solution System Architecture
Oracle is introduced as central DB
As the most powerful DB, Oracle
DW approach helps our customer
to solve current performance
problems and maximize system
availability.
ODI is introduced as ETL tool
As the most efficient ETL tool,
ODI integrates data from Informix
into Oracle, and subsequent data
transformation,with friendly UI
and efficient code.
OBIEE would be further introduced
Other Oracle DB for the
distributed sub-systems and BIEE
for reporting would be further
introduced, concerning that our
solution has successfully solved
current key problems.
Oracle Proposed Solution
DB EE with RAC and Partitioning for Central DB
Partitioning is used to manage huge
size of data efficiently and dramatically
improve performance

Compression is introduced to
dramatically decrease the size of data
and maximize the usage of storage.
RAC is used to support
complicated data processing with
high scalability and high
availability features.

Parallelism is introduced to
maximize the usage of available
resources so as to greatly shorten
response time for complicated
data processing
Oracle Proposed Solution
How ODI Integrates Data from Informix
ODI access Informix via JDBC using
Informix Driver. Connections are
managed within Topology, one
component of ODI.
Complicated transformations are
defined within Designer(the design-
time environment of ODI) ,
transformations such as extraction
from Informix(via JDBC), data
aggregation(via SQL).
Transformations are executed from
Agent(the run-time environment of
ODI), which calls Java,SQL,etc., to
run ETL processes using workflow.
Transformations are monitored from
Operator(the run-time logger of ODI),
which traces each step of the
transformation.






Solution Talk What Potential This
Solution Has ??









Solution Talk A Case Study


41
Copyright 2008, Oracle and/or its affiliates. All rights reserved.
4
2
iTech Solution Profile
Existing Pain

ALL RTOS
SCHEMAS
State
Consolidation
Register

Manual Data
Replication process

STATE LEVEL
RTOS
SCHEMAS
State Register

Manual Data
Replication process
National Level
RTO Schema

Manual Data Replication
process for Huge volumes
of data








Solution Talk A Case Study


43
Copyright 2008, Oracle and/or its affiliates. All rights reserved.
4
4
Existing Data Flow @ HDFC Std Life
Problems with current data flow
Performance is poor
Multi step integration CL + FTP + OWB
More Maintenance effort
Difficult to incorporate development Best Practices
No Unified monitoring and scheduling
Current Integration Flow
ODI can make this step as
straight through
Copyright 2008, Oracle and/or its affiliates. All rights reserved.
4
6
The proposed data integration solution is based on Oracle Data
Integrator (ODI) product.
Oracle Data Integrator provided a unique data integration platform, that
was designed for performance with a high degree of flexibility and
modularity.


Benefits
Performance Leveraged a high performance E-LT architecture.:
Productivity- The in-built declarative design was utilised which in-turn
drives reusability, accelerates development and maintenance.
Flexibility A single platform was presented that natively supports
several data integration styles and latencies.
Modularity Hot-pluggable knowledge modules were used to better
support application and database platform diversity.
Oracles Data Integration Solution
47
Advantages of using ODI
High Performance using Bulk set operations
End-to-End integration within ODI
Easy to maintain because of Repository based architecture and
Impact analysis features
Best Practices can be productized into Knowledge Modules
End-to-End data lineage features to enable report to source data
lineage
CDC mechanism for DB2/400, SQL Server and Oracle

Proposed Data Integration Flow
Copyright 2008, Oracle and/or its affiliates. All rights reserved.
4
8

ODI Screenshots
Proof of concept: Results
Performance Improvement








NOTE- Execution Times mentioned for OWB does not include the time taken
to extract file from DB2 and FTP the same to Oracle Data Warehouse Server.
This currently takes 45 minutes for all mappings put together. ODI accesses
DB2/400 tables directly and does not require an FTP step.
Copyright 2008, Oracle and/or its affiliates. All rights reserved.
4
9
No. of
Records
ODI
(Secs)
No. of
Records
OWB
(Secs)
%
Improvement
ACTIVITY 82911 36 83561 118 ~ 320 %
POLICY_AC
COUNT_DD
533833 718 545028 1693 ~ 230%
Summary
Oracle Data Integrator is the new addition to Oracles Middleware toolset.
ODI-EE is the only product that combines the best performance, flexibility
and cost-savings alternatives for data integration business needs. The E-
LT foundation provided by ODI-EE offers Oracle customers a way to
leverage existing database
(Oracle and non-Oracle) investments without sacrificing world-class
functionality that they would expect from a best-of-breed data integration
platform.
Complements OWB by adding advanced data integration
Functions
Works in the staging/integration layer
Pre-built modules for CDC, bulk loading etc
Supports event-based, service-based and real-time integration
Adds the ability to target heterogeneous platforms
Demonstration