Oracle Data Integrator Building First Interface PTS Workshop: ODI 11g Copyright 2010, Oracle. All rights reserved. <Insert Picture Here> Agenda
Key Concepts Staging Area and Execution Location Business Example Building the First Interface Lab Overview Q&A
Copyright 2010, Oracle. All rights reserved. Key Concepts Copyright 2010, Oracle. All rights reserved. What Is an Interface? An interface is an ODI object that loads one target datastore with data from one or more source data stores.
An interface defines the loading and integration strategies from sources to target based on the selection of Knowledge Modules
An interface implements business rules as mappings, joins, filters, and constraints. Copyright 2010, Oracle. All rights reserved. What Is a Mapping? A mapping is a business rule implemented as a SQL clause.
It is a transformation rule that maps columns in source data stores onto one of the target datastore columns.
It is executed by a relational database server at run time. Copyright 2010, Oracle. All rights reserved. What Is a Join? In relational models, a join operation links records in several tables, from any technologies.
Interfaces use joins to link multiple source data stores.
A join is implemented as a SQL clause that relates the columns of two or more data stores. Copyright 2010, Oracle. All rights reserved. What Is a Filter? A filter is a SQL clause applied to the columns of one source data store. Only records matching this filter are processed by the interface. Copyright 2010, Oracle. All rights reserved. Business Rules for Interfaces Business rules are implemented in interfaces as: Mappings Filters Joins Constraints Lookups They are implemented within Designer in: Data models Interfaces They are stored in the Work repository. Copyright 2010, Oracle. All rights reserved. Where Are the Rules Defined? The business rules implemented in an interface are located as follows: In target datastore definition in the model. Constraints to be enforced are selected on the interfaces Control tab. Constraints
Joins Filters Mapping In the interface diagram. Filter may be inherited from the data store definition in the model or created manually. In the interface diagram. Joins may be inherited from the model definition or created manually. In the interface diagram. Each column of the target has at most one mapping.
Lookups
In the interface editor. Create lookups using a source as the driving table and a lookup datastore or interface. Copyright 2010, Oracle. All rights reserved. How does ODI Implement Business Rules? ODI is based on business rules. First, define the business rules. Next, implement them in SQL. ODI does not require a proprietary engine. Exploits in-house RDBMS engines to perform the required processes Generates native or standard SQL code ODI separates the rules from the processes. Business rules are defined on an interface. Processing instructions are implemented in knowledge modules. Copyright 2010, Oracle. All rights reserved. Staging Area and Execution Location Copyright 2010, Oracle. All rights reserved. What Is the Staging Area? The staging area is a separate, dedicated area in an RDBMS where ODI creates its temporary objects and executes some of your transformation rules. The staging area is usually located on the target data server Copyright 2010, Oracle. All rights reserved. Execution Location In an ODI Interface, each rule may be executed on the: Source Target Staging area The execution location is specified at design time How a rule gets implemented depends on the technology of its execution location Copyright 2010, Oracle. All rights reserved. Business Example Copyright 2010, Oracle. All rights reserved. A Business Problem Source (MySQL) ORDERS LINES
CORRECTIONS file Target (Oracle) SALES ERRORS Business rules Filter - Take into account only ORDERS marked as closed. Join - A row from LINES has a matching ORDER_ID in ORDERS. Mapping - Targets SALES is the sum of the order lines AMOUNT grouped by sales rep. and the corrections applied. Sales Rep = Sales Rep ID from ORDERS Constraints -ID cannot be null and must be unique. The Sales Rep. ID should exist in the Target sales Rep table. Copyright 2010, Oracle. All rights reserved. Implementing the Rules Business rules implemented in SQL Source (MySQL) ORDERS LINES
Mapping SALES = SUM(LINES.AMOU NT) + CORRECTION.VAL UE SALES_REP = ORDERS.SALES_ REP_ID Constraints ID is flagged not null in the model. Unique index UK_ID is declared on the SALES table. Filter ORDERS.STATUS=CLOSED
Copyright 2010, Oracle. All rights reserved. Integration Process These business rules form the basis of the integration process. Integration Process ExtractLoadTransform (check) Source (MySQL) ORDERS LINES
CORRECTIONS File Target (Oracle) SALES ERRORS Copyright 2010, Oracle. All rights reserved. Process Details Detailed sequence of operations for the process Target (Oracle) SALES ERRORS Transform and integrate TEMP_ SALES
CORRECTIONS File TEMP_1 Extract/Join/ Transform TEMP_2 Extract/Transform Join/Transform 1 2 3 4 5 Copyright 2010, Oracle. All rights reserved. Process Implementation: Example 1 Using a proprietary engine or application Proprietary Engine - Specific Language Target (Oracle) SALES ERRORS Source (MySQL) ORDERS LINES
Check constraints/ Isolate errors 1 2 3 4 5 Copyright 2010, Oracle. All rights reserved. Process Implementation: Example 2 Using an RDBMS data server: DB2/UDB DB2/UDB Staging AREA The language is SQL Target (Oracle) SALES ERRORS Source (MySQL) ORDERS LINES
Check constraints/ Isolate errors 1 2 3 4 5 Copyright 2010, Oracle. All rights reserved. Process Implementation: Example 3 Using the Oracle target data server Target (Oracle) Staging Area The language is SQL SALES ERRORS Source (MySQL) ORDERS LINES
Check constraints/ Isolate errors 1 2 3 4 5 Copyright 2010, Oracle. All rights reserved. Building the First Interface Copyright 2010, Oracle. All rights reserved. Quick Overview of Designer Editor Object Tree Toolbar Main Tabs Navigators Accordions Sub-Tabs (Finger Tabs) Editor Tabs Copyright 2010, Oracle. All rights reserved. Terminology ETL/ELT projects are designed in the Designer Navigator Transformations in ODI are defined in objects called Interfaces. Interfaces are stored into Projects Interfaces are sequenced in a Package that will be ultimately compiled into a Scenario for production execution
Copyright 2010, Oracle. All rights reserved. Metadata in ODI Metadata in ODI are available in the Model navigator. Each Model will contain the tables from database schema. A model can contain all tables from a schema, or only a subset of the tables of the schema Models can contain sub models for an easier organization of the tables from a schema
Copyright 2010, Oracle. All rights reserved. Interface An Interface will define Where the data are sent to (the Target) Where the data are coming from (the Sources) How the data are transformed from the Source format to the target format (the Mappings) How the data are physically transferred from the sources to the target (the data Flow) Source and target are defined using Metadata imported from the databases and other systems Mappings are expressed in SQL Flows are defined in Templates called Knowledge Modules (KMs) Copyright 2010, Oracle. All rights reserved. Requirements To run an interface, you need at least the following: A target table An Integration Knowledge Module (selected in the Flow tab) A Loading Knowledge Module if there is a remote source.
If you have all the prerequisites, you are ready to execute the interface. Copyright 2010, Oracle. All rights reserved. Important Note An interface populates only a single target datastore.
To populate several targets, you need several interfaces. Copyright 2010, Oracle. All rights reserved. Lab Overview Copyright 2010, Oracle. All rights reserved. Lab 3: Creating a One-to-One Interface Overview 1. Create and name the interface. 2. Define the target datastore. 3. Define the source datastore. 4. Define the mappings. 5. Save the interface. Copyright 2010, Oracle. All rights reserved. Creating and Naming an Interface 1. Navigate to the project and folder where you want to create your interface. 2. Right-click the Interfaces node and select New Interface. 3. Enter the name. 4. Enter the description. Copyright 2010, Oracle. All rights reserved. Defining the Target Datastore 1. Select the Models view. 2. Expand the model containing the target datastore. 3. Click the interfaces Mapping tab. 4. Drag the data store from the Models view to the Target Datastore area. Copyright 2010, Oracle. All rights reserved. Defining the Source Data Store 1. Expand the model containing the source data store. 2. Drag the data store from the Models view to the Diagram zone in the interface. Copyright 2010, Oracle. All rights reserved. Defining the Mappings 1. Select the column of the target datastore that you want to map. 2. Click the Implementation tab in the Mapping pane, then drag columns from the source table directly into the mapping area. You may use the Expression Editor. 3. If possible, check the code. Copyright 2010, Oracle. All rights reserved. Saving the Interface Click the Save button to save the interface. You can click Yes to save and close the interface. The Cancel button closes the interface without saving. Interfaces are saved in the Work repository. Copyright 2010, Oracle. All rights reserved. Executing the Interface Copyright 2010, Oracle. All rights reserved. Copyright 2010, Oracle. All rights reserved. Backup Slides Copyright 2010, Oracle. All rights reserved. Creating, Naming a New Interface Interfaces are created in Projects To create any object in ODI, right-click on the parent node and select New xyz This is true for interfaces as well: On the projects Interfaces entry, select Right-Click/New Interface. Copyright 2010, Oracle. All rights reserved. Selection of Sources and Target Source Tables Target Table (single target) Metadata Drag and drop the Metadata from the Models tree into the interface to make these sources or targets Copyright 2010, Oracle. All rights reserved. What Is a Mapping? A mapping is a business rule implemented as a SQL clause.
It is a transformation rule that maps columns in source data stores onto one of the target datastore columns.
It is executed by a relational database server at run time. Copyright 2010, Oracle. All rights reserved. Automatic Mappings
Automatic Mapping creates mappings by matching column names automatically.
ODI will prompt you before doing so: you have the option to disable this feature. Copyright 2010, Oracle. All rights reserved. Mappings in the Interface Target Columns (click here to open the mapping field) Mapping expressions (read only) Type or edit your mapping expressions here Expression Editor button Copyright 2010, Oracle. All rights reserved. Using the Expression Editor 1. Click the expression editor button ( ) in the mapping window 2. Build your SQL expressions from the SQL help at the bottom, and from the Columns at the left Copyright 2010, Oracle. All rights reserved. Valid Mapping Types The following type of clauses may be used in the mappings: Value Source Column DBMS Function DBMS Aggregate Combination String values should be enclosed in single quotes: SQL', '5 but 10.3 Drag and drop the column or use the expression editor. It is prefixed by the datastores alias. E.g.: SRC_SALES.PROD_ID Use the expression editor for the list of supported functions and operators MAX(), MIN(), etc. ODI automatically generates the GROUP BY clause. Any combination of clauses is allowed: SRC_SALES_PERSON.FIRST_NAME || ' ' || UCASE(SRC_SALES_PERSON.LAST_NAME) Copyright 2010, Oracle. All rights reserved. Filtering Data SQL filter expression Execution location Expression editor Save expression Check expression. Drag and drop a column on the background area Then type the filter expression Copyright 2010, Oracle. All rights reserved. Saving the Interface Click the Save icon ( ) to save the interface Click the Save All icon ( ) to save all open editors An editor title in italics shows an unsaved interface. Close editor to exit interface without saving
Interfaces are saved in the Work Repository. Copyright 2010, Oracle. All rights reserved. Graphical Representation of the Flow Source and target systems are graphically represented in the Flow tab This is where KM are chosen, and KM options are set Copyright 2010, Oracle. All rights reserved. Running an Interface Simply click the Execute toolbar button Copyright 2010, Oracle. All rights reserved. Code Generation When we ask ODI to Execute the transformations, ODI will generate the necessary code for the execution (usually SQL code) The code is stored in the repository The execution details are available in the Operator Navigator: Statistics about the jobs (duration, number of records processed, inserted, updated, deleted) Actual code that was generated and executed by the database Error codes and error messages returned by the databases if any Copyright 2010, Oracle. All rights reserved. The Operator Interface Switch to the Operator Navigator tab Copyright 2010, Oracle. All rights reserved. Refresh the Logs Display By default, ODI will not refresh the logs. There are two ways to refresh the logs:
Manual refresh: click on this icon in the toolbar:
Automatic refresh: Set the refresh rate (in seconds) in the toolbar and click on this icon in the toolbar:
Copyright 2010, Oracle. All rights reserved. Multiple Levels of Details Job level details Specific step in the job Actual code sent to the systems (SQL or other) Copyright 2010, Oracle. All rights reserved. Errors Reporting The red icon in the tree indicates the steps that failed
Error Codes and Error Messages are reported at all levels Copyright 2010, Oracle. All rights reserved. Information Available for each Level Session: Session-level statistics and time information Step: Interface-level statistics and execution details Task: Generated code and statistics Copyright 2010, Oracle. All rights reserved. Understanding the Operator Icons Running