Designing EIM Interfaces from a Performance Perspective

DRAFT 1 Published by BT Retail T&T

IN CONFIDENCE

Siebel EIM Best Practice

IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE

About this document...

Author
The author of this document may be contacted at:

Paul Duncan (MEH4) Alexander Bain House 2nd Floor (pp 2DAQ) 15 York Street GLASGOW G2 8LA Tel: Mobile: Email: 0141 220 8704 07939 215 541 Paul.3.Duncan@bt.com

Distribution List
BT Siebel EIM Interface Development Teams: High Level Designers Low Level Designers Development Teams BT Siebel Performance Testing and Management

Intended Audience
This document contains detailed technical information relating to Siebel Enterprise Integration Manager (EIM) and Oracle. A level of understanding of these technologies is assumed.

Filing
The filing reference for this document is: TBD

Page 2 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE

History
Issue 0.1 Date 08-AUG-2004 Author Paul Duncan Reason Initial Creation

Page 3 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE

Contents
1 OVERVIEW.......................................................................................................................................................................5 1.1 PURPOSE..........................................................................................................................................................................5 1.2 WHATS NOT INCLUDED......................................................................................................................................................5 1.3 DOCUMENT CONVENTIONS..................................................................................................................................................6 2 EIM BEST PRACTICE....................................................................................................................................................7 2.1 INTRODUCTION..................................................................................................................................................................7 2.2 EIM CONFIGURATION........................................................................................................................................................7 2.2.1 Overview................................................................................................................................................................7 2.2.2 IFB Generic Parameters.......................................................................................................................................9
2.2.2.1 Only Base Tables.............................................................................................................................................. ...............9 2.2.2.2 Ignore Base Tables..................................................................................................................................................... ......9 2.2.2.3 Ignore Base Columns............................................................................................................................................... ......10 2.2.2.4 Default Columns..................................................................................................................................................... .......10 2.2.2.5 Fixed Columns............................................................................................................................................................ ...10 2.2.2.6 Use Index Hints, Use Essential Index Hints.......................................................................................................... .........11 2.2.2.7 Session SQL............................................................................................................................................... ...................12 2.2.2.8 Using Synonyms..................................................................................................................................................... .......12 2.2.2.9 Rollback On Error, Commit Each Pass, Commit Each Table............................................................................ .............13 2.2.2.10 Update After Failed Insert...................................................................................................................................... ......14

2.2.3 IFB Import Process Specific Parameters............................................................................................................15

2.2.3.1 Only Base Columns........................................................................................................................................ ...............15 2.2.3.2 Insert Rows, Update Rows........................................................................................................................................ .....15 2.2.3.3 Net Change....................................................................................................................................................... .............16 2.2.3.4 Oracle Insert Append Mode............................................................................................................................. ..............17 2.2.3.5 Trim Spaces................................................................................................................................................... ................18

2.2.4 EIM Delete Process Configuration ....................................................................................................................19

2.2.4.1 Delete All Rows...................................................................................................................................................... .......19 2.2.4.2 Delete Exact................................................................................................................................................................. ..19 2.2.4.3 Delete Matches............................................................................................................................................. .................20

2.2.5 EIM Merge Process Configuration.....................................................................................................................22 2.2.6 Other Parameters................................................................................................................................................24 3 RUNNING EIM...............................................................................................................................................................26 3.1 DATA.............................................................................................................................................................................26 3.2 IFB FILE AND EIM PROCESSES........................................................................................................................................27 3.3 SCHEDULING...................................................................................................................................................................28 3.3.1 Degree of Parallelism/Layout.............................................................................................................................28 3.3.2 Batchsize..............................................................................................................................................................28 3.3.3 EIM Streaming Configurations...........................................................................................................................28 3.3.4 Control-M Scheduling.........................................................................................................................................29 3.4 EIM TABLE MAINTENANCE.............................................................................................................................................31 4 VOLUME 4 LESSONS LEARNED...............................................................................................................................32 4.1 ENHANCEMENTS TO EIM PROCESSES.................................................................................................................................32 4.1.1 Redundant Updates.............................................................................................................................................32 4.1.2 Splitting Updates.................................................................................................................................................33 4.1.3 Hints and Optimiser Mode..................................................................................................................................35 4.2 DATABASE CONFIGURATION...............................................................................................................................................35 4.2.1 Indexes.................................................................................................................................................................35 4.3 ODM ADAPTER FUNCTIONALITY......................................................................................................................................36

Page 4 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE

1 Overview
1.1 Purpose
This document is intended to be used as a guide for the design and deployment of EIM interfaces to support the delivery of Siebel based projects at BT. It will be of particular relevance to the delivery of OneSiebel (aka Account Managed Platform, AMP) and OneView (aka Volume Siebel Release 6). This document is split into two main sections as follows: 1. Siebel EIM Best Practice This section details techniques to be adopted to ensure that EIM interfaces and their configurations adhere to Siebel and BT Best Practices. 2. OneView Release 4 data load Lessons Learned This section details the key lessons learned from the OneView (aka Volume Siebel, Big Data, CCC) bulk data load of Billing Accounts, Assets and Consumer Updates into the OneView Siebel instance. This document has been assembled from experience from BTs eVolve and Volume platforms. These applications are fed data via ODM controlled data loads. Where processes other than ODM will drive EIM interfaces (e.g. Converge to OneSiebel migration jobs) then some ODM related content will still be relevant and so all sections should be read.

1.2 Whats not included

This document is intended to share Best Practice and Lessons Learned and is neither a High Level nor a Low Level Design. This document should be used to feed into the above mentioned designs. Details of Oracle-Level tuning is not included although some techniques may be mentioned in the context of other EIM specific parameters or configurations. This is not a One Stop Shop from which to learn EIM. Instead, this document is intended to communicate EIM Best Practice.

Page 5 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE

1.3 Document Conventions

Key points within this document have been included in call-out boxes as follows: This indicates a convention that should be followed by Development Teams within BT. A warning sign indicates that specific care should be taken, that a piece of work is required to determine the best configuration or that a convention or parameter should not be followed. An information symbol is used to indicate additional points of interest, not necessarily relevant to specific implementations at BT. These Document Conventions appear for most key point (in the EIM Best Practice section) and so the documents recommendations could be summarised by reading just the illustrated points.

Page 6 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE

2 EIM Best Practice

2.1 Introduction
Siebels Enterprise Integration Manager (EIM) operation is determined by a configuration file. This file is called the IFB file and, coupled with loading data into a Siebel EIM/Interface table (a staging table that Siebel uses so that it can build the logical data relationships as defined within Siebels Repository) determines what operations EIM carries out. As a generic, out-of-the-box interface mechanism, EIMs performance can be streamlined by reducing the amount of work it has to do. This is done by imposing strict IFB guidelines. Not all parameters are discussed in detail. A single Interface Table maps to multiple Base Tables (to closely follow the nature of Siebel Business Objects and Business Components). All tables and related foreign keys are processed by default unless parameters are specified in the ifb file. This document describes how to alter EIMs default operation by specifying parameters in the ifb file. Some of these changes make functional impacts (i.e. change the way in which how EIM processes the data) whereas others can improve performance (normally by limiting the amount of work EIM and the underlying database has to do).

2.2 EIM Configuration

2.2.1 Overview
EIM supports the following 4 modes of operation: o o o o Import Delete Merge Export

The EIM Import job is actually an Upsert operation and so this document will refer to both Import (Insert) and Update EIM operations. There are a number of generic ifb parameters that are valid for all modes of operation as well as those that are operation specific. This document will explore Import, Delete and Merge jobs in more detail. EIM Exports will not be covered, as they are not (generally) used within BT instead, PL/SQL (often controlled by Ab Initio graphs) is used to query the Siebel data tables directly in order to export information.

Page 7 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE So that some of the terminology within this document is clear, heres an example IFB file ( indicates lines removed for clarity): [Siebel Interface Manager] LOG TRANSACTIONS TO FILE = FALSE USING SYNONYMS = FALSE [Update_Volume_Bill_Accnt] TYPE = IMPORT BATCH = $BatchRange TABLE = EIM_ACCOUNT ONLY BASE TABLES = S_PARTY, S_ORG_EXT INSERT ROWS = S_PARTY, FALSE UPDATE ROWS = S_PARTY, FALSE INSERT ROWS = S_ORG_EXT, FALSE NET CHANGE = FALSE ONLY BASE COLUMNS = S_PARTY.PARTY_UID, \ S_PARTY.PARTY_TYPE_CD, \ S_ORG_EXT.NAME, \ S_ORG_EXT.LOC, \ S_ORG_EXT.BU_ID, \ Note: a backslash (\) should be used to escape new lines for readability. Many parameters can be specified in the Header section, the Process section or both. The rules of inheritance are as follows: If a parameter is set in the Header section, it is included in all Process sections. If a parameter is used in a Shell process, it is included in all Shell Sub processes (detailed later). If a parameter is in the Header section and also in the Process section, then the value in the Process section over-rides that of the Header section. The same is true for Shell process and Shell Sub process. If a parameter exists only in a Process section and not in the Header section, then it is only used in that Process. header section

Process section

Page 8 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE

2.2.2 IFB Generic Parameters

The parameters here apply to all EIM job types, with process specific (i.e. Import, Delete or Merge) parameters and techniques being specified in subsequent sections. 2.2.2.1 Only Base Tables This is a Process section parameter. Usage: ONLY BASE TABLES = BASE_TABLE1, BASE_TABLE2 where BASE_TABLE1 & BASE_TABLE2 are tables mapped to by the EIM table in question. By only including the Base Table relationships that you need to resolve, EIMs work is reduced and so Performance can be improved. Example: In OneView, a Consumer Update job incorrectly carried out updates to S_PARTY, S_ORG_EXT, S_ACCNT_POSTN and S_ORG_BU despite the fact that the data in S_ACCNT_POSTN and S_ORG_BU was being defaulted by the Consumer Adapter (Ab Initio) each time. By removing these tables from the ONLY BASE TABLES parameter, the performance of the Volume Siebel Release 4 Consumer Update job was greatly improved. So that performance gains such as these can be made, detailed functional analysis of the EIM interface must be carried out in order to fully understand the data source.

Usage:

This parameter should be used ONLY BASE TABLES

2.2.2.2 Ignore Base Tables This is a Process section parameter.

INGORE BASE TABLES = BASE_TABLE1, BASE_TABLE2 where BASE_TABLE1 & BASE_TABLE2 are tables mapped to by the EIM table in question. As an alternative to ONLY BASE TABLES, this parameter can be used to detail specific tables that should be ignored by the EIM process. For clarity, the only instance in which this parameter would be used instead of ONLY BASE TABLES is where the majority of the tables must be included. By using this parameter, it will be clear which tables will not be used.

This parameter will generally not be used IGNORE BASE TABLES

Page 9 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE 2.2.2.3 Ignore Base Columns This is a Process section parameter. Usage: IGNORE BASE COLUMNS = BASE_TABLE1.BASE_COLUMN1,\ BASE_TABLE1.BASE_COLUMN2,\ BASE_TABLE2.BASE_COLUMN1 where BASE_TABLE1 and BASE_TABLE2 are mapped to by the EIM table in question and BASE_COLUMN1 & BASE_COLUMN2 are columns on each Base Table. This parameter can be used to instruct EIM to ignore the processing of columns specified. This is most useful for ignoring unused foreign key. This would result in EIM not attempting to resolve the foreign key in question. This parameter would generally not be used on OneSiebel with the use of ONLY BASE COLUMNS being preferred.

This parameter will generally not be used IGNORE BASE COLUMNS Not supported for Merge processes.

2.2.2.4 Default Columns There are a number of EIM columns where it may be desirable to set a default value for a column, if no value is specified. There is an option to specify these via the ifb file (DEFAULT COLUMN = COLUMN_NAME, value) however this gives EIM an additional SQL statement to execute. In tuning EIM, the objective is to reduce the work that EIM has to do, therefore fixed & default columns should be set by the external interface architecture i.e. Ab Initio.

This parameter should not be used DEFAULT COLUMN

2.2.2.5 Fixed Columns There are a number of EIM columns where it may be desirable to set a fixed value for a column. There is an option to specify these via the ifb file (FIXED COLUMN = COLUMN_NAME, value) however this gives EIM an additional SQL statement to execute. In tuning EIM, the objective is to reduce the work that EIM has to do, therefore fixed & default columns should be set by the external interface architecture i.e. Ab Initio.

This parameter should not be used DEFAULT COLUMN

Page 10 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE 2.2.2.6 Use Index Hints, Use Essential Index Hints These are Process section ifb file parameter. Usage: USE INDEX HINTS = TRUE|FALSE USE ESSENTIAL INDEX HINTS = TRUE|FALSE The default values of these parameters are indicated in bold, meaning that EIM adds index hints to certain SQL statements. There are two levels of granularity available. By specifying USE INDEX HINTS = TRUE, then additional index hints are used from the default. If no hints are desired, then set both parameters to FALSE. EIM does not always suggest the most appropriate execution path and so to determine whether the (hard coded) hints improve performance or not, EIM jobs should be performance tested with and without this set to True in order to gauge performance characteristics. Within Volume most interfaces (but not all) have been found to perform better with no index hints in place when the Oracle Cost Based Optimizer (CBO) is used. The actual index hints themselves cannot be specified just an ability to turn on|off index hints is provided. In Volume Release 4 this parameter was used in conjunction with the SESSION SQL = "Alter session set optimizer_mode = all_rows" although as OneSiebel have chosen the Siebel 7.7 version, then the database will already be running in Cost Based mode. It is important that OneSiebel performance testing explores this area. As Siebel 7.7s now supports (in fact, mandates) the Oracle Cost Based Optimizer, then the USE INDEX HINTS, USE ESSENTIAL INDEX HINTS must be thoroughly examined by Performance Testing. On Volume it has been observed that Siebel often adds inappropriate index hints such as hinting the use of S_ORG_EXT_U2 to resolve foreign keys to S_ORG_EXT. As this index is led by BU_ID which is non-selective in the Volume application (all Accounts belong to the BT Volume organisation), this is an inappropriate index to use. By turning hints off, the more appropriate S_ORG_EXT_U1 index (or better still a custom index) may be chosen by Oracle.

Page 11 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE 2.2.2.7 Session SQL This parameter has already been introduced in the section on USE INDEX HINTS, USE ESSENTIAL INDEX HINTS. Usage: SESSION SQL = sql statement to be executed Note that no terminating semi-colon (;) should be used eg. SESSION SQL = "Alter session set optimizer_mode = all_rows" This parameter can be used to execute custom SQL at the beginning of EIMs session (EIM is a single threaded, single database connection component). Only a single statement can be issued. Session SQL should NOT be used to modify data in the Siebel schema directly as Siebel does not support direct SQL (except for querying). Session SQL should NOT be used to modify data in the Siebel schema directly as Siebel does not support direct SQL (except for querying). As Siebel 7.7s now supports (in fact, mandates) the Oracle Cost Based Optimizer, then setting of optimiser mode via SESSION SQL may no longer be appropriate. The Performance Team will investigate this area. 2.2.2.8 Using Synonyms This is an EIM Process section parameter. Usage: USING SYNONYMS = TRUE|FALSE EIMs default operation is to look up Account Synonyms in order to resolve foreign keys to S_ORG_EXT. This requires that EIM does an additional lookup and so setting this parameter to False can reduce overall run time. The default is True. Although OneSiebel may have a requirement to support Synonyms (Trading As names), ODM will supply Siebel User Keys of the Account as defined in S_ORG_EXT and so setting this parameter to FALSE will not impact this functionality.

This parameter should be used USING SYNONYMS = FALSE

Page 12 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE 2.2.2.9 Rollback On Error, Commit Each Pass, Commit Each Table These parameters determine what happens when a low level EIM error occurs (such as an Oracle error). By default, EIM commits each table and each pass, meaning that if an error occurs, the result may be partially imported data. Example The Volume Consumer entity consists of an entry in S_PARTY, S_ORG_EXT, S_ACCNT_POSTN, S_ORG_BU. If a Low Level occurs during EIM Import, an entry may be inserted into S_PARTY, S_ORG_EXT but not into other tables, making this record logically invalid within OneView. In order to properly rollback transactions in the event of a Low Level error, then this parameter should be used in conjunction with COMMIT EACH PASS = FALSE, COMMIT EACH TABLE = FALSE. Usage: ROLLBACK ON ERROR = TRUE|FALSE COMMIT EACH PASS = TRUE|FALSE COMMIT EACH TABLE = TRUE|FALSE The defaults are indicated in bold. As per Siebel Alert 290, these settings should be set to perform a single commit (ROLLBACK ON ERROR = TRUE, COMMIT EACH PASS = FALSE, COMMIT EACH TABLE = FALSE) for all Merge and Delete Operations. Whilst it may immediately appear beneficial to use these settings for Insert and Update operations, there is a high risk that this would impact competing processes (like Siebel Analytics) by causing an Oracle Snapshot too old error. This combination of parameters should, therefore, only be used for EIM Merge and Delete processes. As it does not make sense to use ROLLBACK ON ERROR without using both COMMIT EACH PASS = FALSE and COMMIT EACH TABLE = FALSE, then this parameter should not be included in the header section of the ifb file (as is the case currently on Volume).

Page 13 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE 2.2.2.10Update After Failed Insert If single commit is not implemented (and it should generally not be used for EIM Import processes) in OneView, ODMs TrueDiff process should check all tables in a multi-table entity to ensure that the Update after failed Insert operation performs properly. EIM Import jobs intended to Insert-only into the Base Tables are not generally configured with UPDATE ROWS = FALSE, which may otherwise be expected (to avoid a Data Quality Issue where an anomaly has occurred). This is so that if the initial insert fails, then sending the record again will result in an update to a partial record. In Volume at present, this is not implemented properly because the TrueDiff process (the ODM process that identifies which records to insert, update or delete from Siebel based on comparing the data within Siebel with that within ODMs datastores) does not compare columns in all tables for every entity. As per Siebel Alert 290, these settings should be set to perform a single commit (ROLLBACK ON ERROR = TRUE, COMMIT EACH PASS = FALSE, COMMIT EACH TABLE = FALSE) for all Merge and Delete Operations. If single commit is not used (and it should not be), ODMs TrueDiff process should be capable of including all columns in all Base tables involved in each EIM process.

Page 14 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE

2.2.3 IFB Import Process Specific Parameters

This section details those parameters that are specific to EIM jobs of type Import (both Insert and Update operations). As well as the parameters detailed within this section, those parameters in the Generic section can also be applied. 2.2.3.1 Only Base Columns This is a Process section parameter. Usage: IGNORE BASE COLUMNS = BASE_TABLE1.BASE_COLUMN1,\ BASE_TABLE1.BASE_COLUMN2,\ BASE_TABLE2.BASE_COLUMN1 where BASE_TABLE1 and BASE_TABLE2 are mapped to by the EIM table in question and BASE_COUMN1 & BASE_COLUMN2 are columns on each Base Table. Whereas the generic parameter Ignore Base Columns instructs EIM to ignore specific columns, Only Base Columns instructs EIM to only process those columns specifically listed. It does not make sense to use both Ignore Base Columns and Only Base Columns in the same job. Either specify one or the other. ONLY BASE COLUMNS should be used for every interface to achieve best performance You must include Foreign Key columns (Insert and Update), required fields (Insert operation) and any data columns to be loaded.

Usage:

ONLY BASE COLUMNS should be used for every interface to achieve best performance.

2.2.3.2 Insert Rows, Update Rows These two Process section parameters impact upon both Data Quality and Performance.

INSERT ROWS = BASE_TABLEN, TRUE|FALSE UPDATE ROWS = BASE_TABLEN, TRUE|FALSE As has already been stated, an EIM table maps to multiple base tables. The use of the above parameters controls what type of operation is allowed on each specified table. The default for both parameters is True. As an EIM Import job is actually an Upsert operation. i.e. if the User Keys populated in the EIM table matches those of an existing row in the Base Tables, then the operation is an Update. If the User Keys populated in the EIM table do not match those of an existing row in the Base Tables, then the operation is an Insert operation. BT has invested heavily in the ODM layers capability of creating separate Insert and Update transactions, based around the unload functionality which is necessary due to multi-mastering. As such, ODM determines whether the operation should be an insert or an update to Siebels data. Therefore, the author believes that every EIM job should specifically use INSERT ROWS Page 15 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE and UPDATE ROWS to specify the operations that EIM is allowed to perform. This will prevent any rows from being updated (where User Keys match in error) or inserted (when User Keys are not located) unintentionally. Performance can be improved by preventing redundant updates. INSERT ROWS and UPDATE ROWS should be used to tighten Data Quality and improve performance. If single commit is not used, ODMs TrueDiff process should be capable of checking all tables involved in an EIM interface. This would mean, optimally, that an Insert interface could specify UPDATE ROWS = FALSE on base tables so that no erroneous update would be allowed. If TrueDiff is not fully aware of each Siebel entity, then Insert jobs would need to be set to allow updates although this wouldnt always fix records as intended (self-healing capability). Although INSERT ROWS = TRUE is the default, it is a good idea to state this specifically so that the interfaces intention is not ambiguous. Full use of INSERT ROWS and UPDATE ROWS will mean that any occurrences of PARTIALLY_IMPORTED should be treated as rightful errors. 2.2.3.3 Net Change This parameter is used to specify whether columns can be updated to NULL. Usage: NET CHANGE = TRUE|FALSE The default value is shown in bold. When set to FALSE, Base Table columns will be updated to NULL if the corresponding EIM table column is NULL and ONLY BASE COLUMNS is used. This is the only way to update a column to NULL. EIMs default behaviour is to skip processing of NULL columns in the EIM table (except where ONLY BASE COLUMNS is used). Because of the way in which ODM adapters are designed and owing to the fact that ONLY BASE COLUMNS is used throughput, NET CHANGE = FALSE is used for all Update processes. Whilst the underlying SQL generated by NET CHANGE = FALSE implements this by use of a large NVL based statement, this has not been seen to cause a significant processing overhead.

Page 16 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE 2.2.3.4 Oracle Insert Append Mode This is a new feature in Siebel 7.5x. Usage: ORACLE INSERT APPEND MODE = TRUE|FALSE The default is shown in bold. Using this means that when inserting data into the Base tables, Oracles Insert Append Mode is used. This inserts data into new blocks above the High Water Mark. Whilst this may improve general insert performance, use of this parameter will lead to increased fragmentation and increased storage requirements. The Volume Production DBA team do not support use of this parameter. Do not use this parameter as it can lead to increased Base Table fragmentation as well as increased storage requirements.

Page 17 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE 2.2.3.5 Trim Spaces This is an EIM Process section parameter. Usage: TRIM SPACES = TRUE|FALSE EIMs default operation is to right trim white space around data fields:
UPDATE SET SIEBEL.EIM_ACCOUNT IF_ROW_STAT_NUM = 0, IF_ROW_STAT = 'IN_PROGRESS', T_ORG_EXT__STA = 0, T_ORG_EXT__RID = NULL, T_ORG_EXT__EXS = 'N', T_ORG_EXT__UNQ = 'N', T_ORG_EXT_BU_ID = NULL, T_ORG_EXT_CURRPR = NULL, ... ... END_DT = DECODE(END_DT,NULL,TO_DATE(NULL), TO_DATE(TO_CHAR(END_DT,'YYYY-MM-DD')||' 00:00:00','YYYY-MM-DD HH24:MI:SS')), START_DT = DECODE(START_DT,NULL,TO_DATE(NULL), TO_DATE(TO_CHAR(START_DT,'YYYY-MM-DD')||' 00:00:00','YYYY-MM-DD HH24:MI:SS')), NAME = RTRIM(NAME), LOC = DECODE(LOC,NULL,NULL,RTRIM(LOC)), ALIAS_NAME = DECODE(ALIAS_NAME,NULL,NULL,RTRIM(ALIAS_NAME)), ASGN_PH_AC = DECODE(ASGN_PH_AC,NULL,NULL,RTRIM(ASGN_PH_AC)), ASGN_PH_CC = DECODE(ASGN_PH_CC,NULL,NULL,RTRIM(ASGN_PH_CC)), BASE_CURCY_CD = DECODE(BASE_CURCY_CD,NULL,NULL,RTRIM(BASE_CURCY_CD)), ... ... URL = DECODE(URL,NULL,NULL,RTRIM(URL)), VAT_REGN_NUM = DECODE(VAT_REGN_NUM,NULL,NULL,RTRIM(VAT_REGN_NUM)), ROW_STATUS = DECODE(ROW_STATUS,NULL,NULL,RTRIM(ROW_STATUS)), COMMENTS = DECODE(COMMENTS,NULL,NULL,RTRIM(COMMENTS)), STATUS = DECODE(STATUS,NULL,NULL,RTRIM(STATUS)) WHERE (IF_ROW_BATCH_NUM = 1)

Not all columns are trimmed but this cannot be controlled. By setting this parameter to False, then this operation is not performed during the Initialize Interface Tables EIM pass. Whilst this theoretically slims down EIM processing, the gains to by had by setting TRIM SPACES = FALSE will be negligible when compared to the impact of other EIM parameters. Additionally, both the eVolve and Volume development teams have expressed that they prefer to retain the default of TRUE.

TRIM SPACES should be left at the default value of TRUE. It adds no value to specify this explicitly in the ifb.

Page 18 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE

2.2.4 EIM Delete Process Configuration

This section details parameters and techniques as well as important Technical Information from Siebel Support, which is specific to EIM jobs of type Delete and Merge. This section is structured differently than other like sections within this document as this walks the reader through each type of Delete job, namely: o o o Delete All Rows Delete Exact Delete Matches

2.2.4.1 Delete All Rows This Process section parameter is used to specify that all rows mapped to by the interface table should be deleted. This parameter should therefore be used with extreme caution! Usage: DELETE ALL ROWS = TRUE|FALSE The default value is False. No other parameters are really be valid as all data in the specified Base Table will be deleted. Once run, the User Keys of records deleted by the process will be loaded into the EIM tables and can be archived or deleted. This parameter should be used with extreme caution! It is not particularly relevant to Production Processes. 2.2.4.2 Delete Exact This Process section parameter is used to specify that EIM should delete rows in the Base Tables whose User Keys match those of records loaded into the Interface Tables. Usage: DELETE EXACT = TRUE|FALSE As EIM only needs the User Keys, then only the User Keys should be loaded. This operation is primarily concerned with deleting rows from the primary Base Tables mapped to by the given EIM table. Deletion of Child Rows can be controlled by use of ONLY BASE TABLES. DELETE ROWS can be used to specify from which tables deletes are allowed. The default value is True. Usage: DELETE ROWS = BASE_TABLEN, TRUE|FALSE where BASE_TABLEN is a Base Table mapped to by the specific EIM table. UPDATE ROWS is also valid for this job. Page 19 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE Setting DELETE ROWS and/or UPDATE ROWS to FALSE should be done with extreme caution as it may result in dangling foreign keys i.e. foreign keys that no longer point to a valid record. Once run, no additional data is populated into the interface tables and so the original rows that were loaded into the EIM table may be archived or deleted. Setting DELETE ROWS and/or UPDATE ROWS to FALSE should be done with extreme caution as it may result in dangling foreign keys. DELETE EXACT is the safest method of deleting data in bulk from Siebel and so is the adopted delete method at BT.

2.2.4.3 Delete Matches This Process section parameter is used to specify that EIM should delete data from the Base Tables that matches the where clause specified in the ifb. e.g. DELETE MATCHES = EIM_ACCOUNT, (CUST_STAT_CD = Inactive) In the example above, all rows whose CUST_STAT_CD = Inactive would be deleted. Once run, this EIM job will have written the User Keys of the records deleted into the EIM tables. This data can then be either archived or deleted. There is no concept of Batchsize in a Delete Matches job all rows that meet the criteria are processed in a single run. There is no concept of batchsize in a Delete Matches job. Therefore, several jobs may be required to delete the intended records, perhaps grouping them by a data range of range of Unique Identifiers.

Page 20 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE

2.2.4.3.1Important Information from Siebel Support

There are two important Siebel Alerts need to be kept in mind for all Delete and Merge jobs. These Alerts should be implemented (subject to confirmation/approval from Siebel) in all Delete and Merge jobs. o There is a Siebel Alert (Alert 306: Specify $NUM_IFTABLE_LOAD_CUTOFF = 0 for EIM merge and delete) that states that you must specify a parameter ($NUM_IFTABLE_LOAD_CUTOFF=0) otherwise EIM will be blind to some metadata relating to foreign keys. This could clearly cause DQ issues as it may leave dangling foreign keys. Whilst this parameter should be able to be specified in the IFB Header section as detailed within the Alert, this was found to repeatedly cause a core dump on the Solaris Application Server. A Service Request was logged on Siebel SupportWeb and the workaround of specifying this via Extended Parameters on the command line was found. Orphaned records may be created in the current configuration in the event that an Oracle error (or other similarly severe error) occurs whilst the EIM job is running. This is as a result of another Siebel Alert (Alert 209: EIM Merge and Delete processes should use only one commit per batch to avoid data integrity issues) and, again, may cause DQ issues. To prevent orphans from being created the following settings should be specified in the Process section: COMMIT EACH TABLE = FALSE COMMIT EACH PASS = FALSE ROLLBACK ON ERROR = TRUE (If a severe error does occurs, the entire batch is rolled back but records are left in the EIM table with an IF_ROW_STAT of IN_PROGRESS because Siebel does a commit after preparing the interface tables (see Trim Spaces section within this document for an example EIM log file showing this). Therefore, if records are left with an IF_ROW_STAT = IN_PROGRESS after all EIM jobs have completed, then it is clear that a low-level error has occurred.) Use the following parameters for all Delete and Merge jobs:

COMMIT EACH TABLE = FALSE COMMIT EACH PASS = FALSE ROLLBACK ON ERROR = TRUE $NUM_IFTABLE_LOAD_CUTOFF = 0 should be specified for all Delete and Merge jobs. ODMs GSL (or any other calling process) must be able to specify this parameter and, in fact, it would be wise that the GSL could build a flexible Server Manager command line mode call so that Extended Parameters and other specialised attributes can be included. This would require that the current ODM EIM_DETAIL table and surrounding functionality be enhanced. Including $NUM_IFTABLE_LOAD_CUTOFF = 0 within the IFB files Header section may cause core dumps on the Solaris Application Server. To workaround this, specify the parameter via Extended Parameters on the command line.

Page 21 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE

2.2.5 EIM Merge Process Configuration

In a Merge job, victim and survivor records are specified in the EIM tables. The IF_ROW_MERGE_ID on the victim record is populated with the row_id (in the Interface Table) of the survivor record e.g. IF_ROW_BATCH_NUM 100 100 100 100 100 In the example above: OneSiebel and OneView are survivor records (no IF_ROW_MERGE_ID specified). Volume and Big Data are merged into OneView. Account Managed Platform is merged into OneSiebel. ROW_ID 1 2 3 4 5 NAME OneSiebel OneView Volume Account Managed Platform Big Data IF_ROW_MERGE_ID

2 1 2

A Merge job does not have any specific parameters, however there are a number of recommendations and observations: o o o A Merge job only needs the User Keys and IF_ROW_MERGE_ID on the victim record to be specified. Siebel Alerts 306 and 209 (as detailed in the section entitled Important Information from Siebel Support) should be implemented. UPDATE ROWS is the only other relevant setting. This controls whether foreign keys in a child table are updated to point towards the survivor or not. Setting this parameter to FALSE can cause dangling foreign keys and is not recommended. The batchsize used should be considerably smaller than the batchsize usually adopted e.g. on Volume the standard EIM batchsize is 10,000 but a batchsize of just 2000 is used for Address Merge operations. Whilst EIM Merge can theoretically be run in parallel if records are victim records are mutually exclusive across all batches, this is not recommended Oracle locks occurred during Performance Testing on a number of occasions and as database resource is considerable even for a single EIM stream, then running EIM merge in parallel is likely to saturate IO resources very quickly and result in an own goal. As in the example above if more than 1 victim is to be merged into survivor, this should be done in the same batch (if mutual exclusivity can be guaranteed). Merge operations should NOT be split into batches of 2 containing just a survivor and a victim (as was the case on Volume at one point).

Page 22 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE o Volume is currently experiencing difficulties with EIM Merge jobs due to Performance and Data Quality Issues. o Performance Due to the large number of relationships within the Siebel database, not all relevant foreign key columns on the database have been indexed, out of the box. This invokes Full Table Scans which are not viable against OneViews largest tables. Siebel Expert services are currently engaged to resolve the issues and a resolution is forthcoming. o Data Quality A Siebel documented issue whereby duplicate columns are left behind in intersection tables was recently experienced by the Volume team. Whilst this issue is reportedly resolved in 7.7, it merits that due care be taken. Use the following parameters for all Delete and Merge jobs:

COMMIT EACH TABLE = FALSE COMMIT EACH PASS = FALSE ROLLBACK ON ERROR = TRUE $NUM_IFTABLE_LOAD_CUTOFF = 0 should be specified for all Delete and Merge jobs. ODMs GSL must be able to specify this parameter and, in fact, it would be wise that the GSL could build a flexible Server Manager command line mode call so that Extended Parameters and other specialised attributes can be included. This would require that the current Volume EIM_DETAIL table and surrounding functionality be enhanced. The Volume team encountered Data quality issues due to an out-of-the-box Siebel issue. Whilst this has been reportedly fixed in 7.7, due care needs to be taken.

Page 23 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE

2.2.6 Other Parameters

The following table includes details on other EIM parameters for completeness. Parameters requiring attention include description text in bold. N.B. This guide was prepared with reference to both Siebel 7.5.2 and Siebel 7.7. Parameter Description Header Section Only Parameters Other parameters can generally be specified in either the header or the process section. CONNECT ODBC data source not required as this is set using data retrieved from the Gateway Server LOG TRANSACTIONS TO A Siebel Remote related parameter this should be set to FILE FALSE in the header section of all IFB files whilst Siebel Remote is not being used. All Siebel Remote related parameters will need to be reviewed if Siebel Remote is to be deployed. USERNAME This parameter should not be set. Rather this information is retrieved from the Gateway. PASSWORD This parameter should not be set. Rather this information is retrieved from the Gateway. TABLEOWNER This parameter should not be set. Rather this information is retrieved from the Gateway. PROCESS This defines the process to invoke when EIM is started with the IFB file. This should not be used as it is preferred that the process name be specified externally i.e. specified on the command line when invoking EIM. Generic Process Parameters BATCH The batch or batch range to be processed. INCLUDE Used in Shell Processes to automate the running of a number of processes automatically. Shell Processes should generally not be used, as scheduling of jobs is generally mastered outwith the EIM environment and specific EIM processes to invoke are specified on the Server Manager command line. Default.ifb, which can be found within the Application Servers admin directory, contains examples of calling Shell Processes. LOG TRANSACTIONS A Siebel Remote related parameter. This parameter is only relevant if Docking:Transaction Logging System Preference is set to True. Not relevant to either OneSiebel r1.0 or OneView r6.0. SKIP BU_ID DEFAULT The default value of False should be retained to specify that EIM uses the default value defined within the Repository when setting virtual null keys. TABLE This parameter specifies which EIM table is to be processed by the EIM job. TRANSACTION SQL The optional parameter can be used (similarly to SESSION SQL) to run ad-hoc SQL after each EIM SQL statement is committed. To date no specific use has been found for this parameter. TYPE Specifies whether the EIM process is of type IMPORT (includes BT Update and Import processes), EXPORT, DELETE, MERGE or SHELL. Shell processes should not generally be used. Default.ifb, which can be found within the Application Servers admin directory, contains examples of calling Shell Processes. Page 24 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE Parameter Description Header Section Only Parameters Other parameters can generally be specified in either the header or the process section. UPDATE STATISTICS As this parameter only applies to DB2, it should not be used within BT (BT uses Oracle). Import Process Parameters COMMIT OPERATIONS A Siebel Remote related parameter. Not relevant to either OneSiebel r1.0 or OneView r6.0. FILTER QUERY This option can be used to specify which rows in the EIM table are to be processed e.g. FILTER QUERY = (CUST_STAT_CD = Active). As BT has invested heavily in external preparation of data for EIM (Ab Initio graphs), then there should be no need to use a parameter like this. MISC SQL This parameter should be used for initial loads only the author will review its applicability for OneSiebel at a later date. ATTACHMENT DIRECTORY This parameter is used to specify the location of attachments to be loaded into Siebel. EIM must have both read and write access to this directory. (Default = SIEBEL_HOME/INPUT) Export Process Parameters Note: The EIM Export process is not generally used within BT but CLEAR INTERFACE TABLE The default for this parameter is True meaning that the EIM export process deletes all existing rows in the interface table prior to starting the export process. Use of this parameter is only possible where no other users/processes are sharing the specific EIM table to be used. EXPORT ALL ROWS Synonymous with DELETE ALL ROWS EXPORT MATCHES Synonymous with DELETE MATCHES Delete Process Parameters CASCADE DELETE ONLY Used to specify whether child records are deleted (TRUE) or their FK is updated to NULL (TRUE). The default is False. DELETE SKIP PRIMARY This parameter specifies whether EIM should perform a cascade update to the primary child column. The Default value is TRUE. Merge Process Parameters SET BASED LOGGING A Siebel Remote related parameter. This parameter is only relevant if Docking:Transaction Logging System Preference is set to True. Not relevant to either OneSiebel r1.0 or OneView r6.0.

Page 25 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE

3 Running EIM
3.1 Data
Reviews of several Siebel implementations at BT has revealed that there has, on occasion, been some confusion as to what data needs to be populated for each different job type. This section clarifies what data is required for each type of EIM job. Import (Insert) To insert a new record (into Primary or Child Base Tables), User Keys, Required Fields, Data columns and System Columns need to be populated. Import (Update) To update an existing record just the User Keys, specific Data Columns to update and System Columns need to be populated. Delete To Delete a record using Delete Exact, just the User Keys and System Columns need to be populated. Merge To Merge records, just User Keys and System fields need be populated.

The following definitions apply to the above: User Keys the combination of columns used by EIM to uniquely identify a row (synonymous with Compound Keys). Required Fields those fields that must be populated when creating a new record. Data Columns regular columns in the EIM table that map to columns in the Base Tables. System Columns CREATED, CREATED_BY, LAST_UPD, LAST_UPD_BY, ROW_ID, IF_ROW_BATCH_NUM, IF_ROW_MERGE_ID, IF_ROW_STAT, MODIFICATION_NUM, CONFLICT_ID.

The following are mandatory System Columns: IF_ROW_BATCH_NUM controls the Batch to which the record belongs ROW_ID used to uniquely identify records within a batch. The combination of IF_ROW_BATCH_NUM and ROW_ID must be unique. IF_ROW_MERGE_ID (for EIM Merge processes only) set this to the ROW_ID within the batch that the record will be merged into. IF_ROW_STAT set with a meaningful status, such as FOR_IMPORT, FOR_UPDATE. This will be set to IN_PROGRESS by EIM during processing.

The other System Columns are defaulted on a table level (Oracle default value) in some instances not all instances. It is, however, good practice to set these fields explicitly so that records in EIM tables can be easily identified. Note that the values populated in these other System Columns will not influence the corresponding Base table records these columns are set by the EIM process.

Only load the data required by the interface.

Page 26 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE

3.2 IFB File and EIM Processes

This section details guidelines that should be adopted by BT Siebel interface development teams relating to how to logically collate multiple EIM processes into ifb files. In general, there should be a single ifb file per entity, with multiple processes being maintained within that ifb file e.g.: A file called onesiebel_customer.ifb should contain all Customer related EIM processes for OneSiebel, such as: Import_OneSiebel_Customer Update_OneSiebel_Customer Cease_OneSiebel_Customer Delete_OneSiebel_Customer Merge_OneSiebel_Customer

Note: the Cease process is a logical delete of the Customer whilst the Delete process is a physical delete. By including multiple EIM processes for a given entity within a single IFB file, then configuration management can be controlled at an appropriate level. The calling process (Ab Initio adapter, script etc) should specify the process to run (and other parameters) via the command line e.g.: $SIEBEL_ROOT/bin/srvrmgr -g $GATEWAY:$GPORT -e $ENTERPRISE -s $APPSERVER -u $SUSER -p $SPASS -c "run task for component eim with config="$IFB", traceflags=3, errorflags=1, SQLFlags=8, ExtendedParams="BatchRange=$BATCH", Process="$PNAME"" (UNIX escape sequences have been omitted for clarity). $SIEBEL_ROOT the path where the Siebel Application Server is installed $GATEWAY the IP address (or hostname) of the Siebel Gateway Server $GPORT the port on which the Gateway Server is running $ENTERPRISE the name of the (Logical) Siebel Enterprise Server $APPSERVER the name of the (Logical) Siebel Application Server $SUSER - the Siebel application user to be used to run the EIM process under $SPASS the password of the aforementioned user $IFB the full path and file name of the IFB file to be used $BATCH the batch or batch range to be processed by this EIM process (the use of batch ranges minimizes the overheads associated with starting an EIM process) $PNAME the name of the EIM Process within the IFB file to invoke

As EIM name-value parameter pairs and extended parameters may need to be controlled on an ad-hoc basis, then the calling process should be configurable in the way in which the command line is structured and sent to Server Manager. By including multiple EIM processes for a given entity within a single IFB file, then configuration management can be controlled at an appropriate level. This guideline should be adopted.

Page 27 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE

3.3 Scheduling
This section discusses some factors effecting the running of EIM as follows: As EIM operates on batches of data, then use of database server resources (in terms of redo, archiving and undo) can be optimised by use of different batch sizes. EIM runs as a single threaded process against the database server, therefore it can be run in parallel to increase throughput. Different scheduling options.

3.3.1 Degree of Parallelism/Layout

The GSL supports the running of EIM batches in parallel. This means that concurrent EIM streams are fired off at the same time, working against different ranges of batches simultaneously. During Volume 4 Performance Testing determined the optimal degree of parallelism by varying the number of concurrent streams and analysing throughput. A limit comes about when the competition for I/O results in a degraded throughout as more and more streams are added. If a job is more CPU bound than IO bound, then parallelism can generally be increased (e.g. to 8way). By determining the optimal degree of parallelism, then better EIM throughput can be achieved although competing processes also need to be catered for.

Performance Testing should vary the Layout to achieve the highest throughput.

3.3.2 Batchsize
Similarly to Layout, EIMs best throughput can be found by varying batchsize during testing. For Volume, the optimal batchsize is generally 10,000 records per batch although smaller batches must be used for Merge and Delete jobs to reduce the risk of causing competing process to fail (e.g. Snapshot too old errors). Such a large optimal batchsize is possible when the buffer cache is large.

Performance Testing should vary the Batchsize to achieve the highest throughput.

3.3.3 EIM Streaming Configurations

Whilst Converge determined that when spawning more than 5 EIM stream this needed to be spread across multiple Siebel Application Servers, such behaviour has yet to be exhibited on Volume. Volume runs all EIM processes from a single Siebel Application Server.

Page 28 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE

Volume 4s Performance was consistent in determining that I/O on the database server was the single biggest bottleneck (for loads into tables not populated, the single biggest bottleneck is generally with CPU). This therefore did not give rise to performing testing the spawning of EIM from multiple Siebel Application Servers. Performance Testing should determine whether increasing the number of physical Siebel Application Servers enhances EIM performance. If this behaviour is not verified, the multi-server EIM streaming configuration should not be adopted.

3.3.4 Control-M Scheduling

In Volume, the current schedule (maintained and operated from Control-M) is based upon 3 streams and looks something like this:

Stream 1 Process 1

Stream 2 Process 4

Stream 3

Process 2

Process 5

Unload Entity 1 Unload Entity 2 Unload Entity n

Process 3

Process n

Process n

In this schedule, EIM table loads happen at the tail end of the on-line day (8pm onwards) this has not yet caused any performance impact upon the on-line processes. In the excerpt above, multiple streams run in the schedule: Stream 1 runs EIM processes (each of which is running in parallel). There are dependencies between the processes between the streams Stream 2 runs EIM processes as in Stream 1. Stream 3 runs unloads with dependencies with the EIM processes in the other two streams a table isnt unloaded until all EIM processing against that table is complete (S_ORG_EXT is, therefore, unloaded last in Stream 3 as it is accessed by most EIM jobs).

In this approach jobs that process different EIM and Base tables can be run simultaneously, with unloads going on in parallel dependencies between EIM streams and unloads must be enforced. This maximises use of the batch window, as long as the database server can cope with the demands put upon it.

Page 29 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE

Where a process is determined to run in isolation with maximum throughput at a certain degree of parallelism during testing, this may need to be reduced if run in parallel with other EIM processes and alongside unloads.

Page 30 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE

3.4 EIM Table Maintenance

Due to the number of operations executed against the EIM tables, then the following table maintenance is recommended: Truncate EIM tables regularly This has many advantages including releasing unused storage and removing fragmentation. Analyse EIM tables after each table load As EIM executes many SQL statements in Cost Based Mode, then table (and index) statistics assist Oracle in choosing an appropriate execution path. Regular statistics gathering on the base tables should also be done for the same reasons. Storage Parameters given that Row Chaining can occur on some EIM tables, then the project DBAs should consider setting a high PCTFree value.

Page 31 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE

4 Volume 4 Lessons Learned

This section outlines some of the key lessons learned during Performance Test and Deployment of Volume 4 as well as from an exercise to overhaul all EIM interfaces in Production on Volume. Changes made to the ODM Ab Initio components have already been integrated into Volume 6 and AMP High Level Designs and so are only mentioned here for completeness.

4.1 Enhancements to EIM Processes

This section contains some of the enhancements made to EIM Processes that should be adopted by subsequent BT Siebel projects.

4.1.1 Redundant Updates

Removal of redundant operations within the EIM process is the single biggest opportunity (with possible exception of ensuring best possible sorting) to make performance improvements This is best illustrated by example. The original Update_Volume_Consumer process looked similar to that shown below:
[Update_Volume_Consumer] TYPE = IMPORT BATCH = $BatchRange TABLE = EIM_ACCOUNT ONLY BASE TABLES = S_PARTY, S_ORG_EXT, S_ORG_BU, S_ACCNT_POSTN INSERT ROWS = S_PARTY, FALSE INSERT ROWS = S_ORG_EXT, FALSE INSERT ROWS = S_ORG_BU, FALSE INSERT ROWS = S_ACCNT_POSTN, FALSE NET CHANGE = FALSE ONLY BASE COLUMNS = S_PARTY.PARTY_UID, \ S_PARTY.PARTY_TYPE_CD, \ S_PARTY.NAME, \ S_PARTY.ROOT_PARTY_FLG, \ S_PARTY.GROUP_TYPE_CD, \ S_ORG_EXT.NAME, \ S_ORG_EXT.LOC, \ S_ORG_EXT.BU_ID, \ S_ORG_EXT.PAR_ROW_ID, \ S_ORG_EXT.ACCNT_FLG, \ S_ORG_EXT.ACTIVE_FLG, \ ... S_ORG_EXT.ACCNT_TYPE_CD, \ S_ORG_EXT.OU_TYPE_CD, \ S_ORG_EXT.INTEGRATION_ID, \ S_ORG_EXT.OU_NUM, \ S_ORG_EXT.MASTER_OU_ID, \ S_ORG_EXT.BASE_CURCY_CD, \ S_ORG_BU.ORG_ID, \ S_ORG_BU.ORG_LOC, \ S_ORG_BU.ORG_NAME, \ S_ORG_BU.BU_ID, \ S_ORG_EXT.CURR_PRI_LST_ID, \ S_ACCNT_POSTN.OU_EXT_ID, \ S_ACCNT_POSTN.POSITION_ID, \ S_ACCNT_POSTN.ROW_STATUS, \ S_ORG_EXT.PR_POSTN_ID, \ S_ORG_EXT.X_BT_FOLLOW_UP_RATING, \

Page 32 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE

The following redundant updates were removed: S_ORG_BU, S_ACCNT_POSTN these tables are populated with default relationships by the adapter. As this data never changes, then redundant updates to these large tables were removed, a large improvement was made. This was done by removing the aforementioned tables from the ONLY BASE TABLES parameter. S_ORG_EXT.MASTER_OU_ID as the Consumers Master account is itself (a self-reference), and this can never change, then the Master_Ou_Id column was able to be removed. Being a Foreign Key to S_ORG_EXT, gains in the end to end EIM runtime for Update_Volume_Consumer will be able to be almost halved (this change is yet to be implemented in Production). This was done by removing S_ORG_EXT.MASTER_OU_ID from the ONLY BASE COLUMNS parameter.

The following index changes were made: Siebel was found to hint the inappropriate S_ORG_EXT_U2 index (led by BU_ID which is non-selective in Volume) to resolve relationships to S_ORG_EXT. By removing index hints (and invoking CBO), Oracle was able to start using the more appropriate S_ORG_EXT_U1 index (led by name). A marginal gain was realised through this change. A new index has been designed that will further improve this interface. Two indexes were also identified to reduce IO. Detailed Oracle configuration is not included within this document.

Once all these changes have been implemented in live, then this processs typical throughput for bulk update operations will have increased several folds. Ensure that update processes do not update tables or resolve foreign keys unnecessarily.

4.1.2 Splitting Updates

Another way to yield big gains in to split update records out into multiple update processes as opposed to a single update process. All (previous) ODM EIM update jobs simply detected that something within the record had changed and provided a full update record to EIM. It is inefficient to resolve foreign keys columns that have not changed redundant updates. Changes to the Update_Volume_Asset are currently in progress on Volume and this serves as a good way to illustrate the concept of split updates: The original Asset Update process looked like this: [Update_Volume_Asset] TYPE = IMPORT TABLE = EIM_ASSET BATCH = $BatchRange USE INDEX HINTS = FALSE USE ESSENTIAL INDEX HINTS = FALSE SESSION SQL = "Alter session set optimizer_mode = all_rows" ONLY BASE TABLES = S_ASSET INSERT ROWS = S_ASSET, FALSE NET CHANGE = FALSE ONLY BASE COLUMNS = S_ASSET.ASSET_NUM, \ Page 33 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE

S_ASSET.PROD_ID, \ S_ASSET.BU_ID, \ S_ASSET.AUTO_SR_FLG, \ S_ASSET.COMP_UND_WRNTY_FLG, \ S_ASSET.PER_ADDR_ID, \ S_ASSET.PAR_ASSET_ID, \ S_ASSET.ROOT_ASSET_ID, \ S_ASSET.BILL_ACCNT_ID, \ S_ASSET.X_BT_HIERARCHY_CD Examination reveals that there are 4 FKs to heavily populated tables that are resolved by this update process, namely, PER_ADDR_ID (FK to S_ADDR_PER), PAR_ASSET_ID and ROOT_ASSET_ID (FKs to S_ASSET) and BILL_ACCNT_ID (FK to S_ORG_EXT). Functional analysis determined that the Asset Hierarchy would never change and that Bill Account and Address would only ever rarely change. By drilling down in the ODM TrueDiff process to produce separate individual outputs in the event that one of these FKs actually does change, then performance can be dramatically improved. The result is that a single Asset Update process is being split to produce the following EIM update processes: [Update_Volume_Asset] this process is used where only attributes have changed. The 4 FKs were removed from this process. [Update_Volume_Asset_Address] specifically designed for Volume 5 where a large number of Assets addresses would change, this process only updates Asset Attributes and Asset Address. The Asset Hierarchy and Bill Account FKs have been removed from this process. [Update_Volume_Asset_All] as it was determined that Asset Hierarchy should never change and that Bill Account would only change rarely, these changes were grouped together. Due to the way in which the TrueDiff code works with split updates, then this process would update Asset Attributes and all FKs (read on).

The Ab Initio code for the TrueDiff Update splits works as follows: 1. If there are changes to Asset Attributes, then the record is flagged for [Update_Volume_Asset]. 2. If there is a change to the Asset Address, then the record is flagged for [Update_Volume_Asset_Address]. By flagging the record for this process, any flag that was set previously is over-written. 3. If there is a change in the Asset Hierarchy or the Bill Account, then the record is flagged for the [Update_Volume_Asset_All] process. Any previous flag that was set is over-written. In this case all attributes and FKs are updated. In general EIM update performance can be improved by only updating FKs that actually need to be updated. This is particularly prevalent when the foreign key is to a heavily loaded Base table such as S_ORG_EXT.

Split out update processes into separate processes so that foreign keys are not resolved unnecessarily.

Page 34 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE

4.1.3 Hints and Optimiser Mode

Loads into the already heavily populated S_ORG_EXT table were improved by turning of index hints and altering the sessions database optimiser to Cost Based mode (from the default of Rules based): USE INDEX HINTS = FALSE USE ESSENTIAL INDEX HINTS = FALSE SESSION SQL = "Alter session set optimizer_mode = all_rows" This technique was used so that Oracle would pick a custom index (described in the next section) using RBO to improve throughput. As OneSiebel has opted to use Siebel 7.7, then execution path optimisation will work differently. The Performance Test team will investigate this.

Analyse EIM performance using different Oracle optimizer modes with and without Siebel generated index hints.

4.2 Database Configuration

4.2.1 Indexes
Most of the User Key indexes in Siebel do not contain the ROW_ID column meaning that 2 I/Os are often required where just one could suffice. As I/O was consistently the major contributing factor to less-than-desired throughput, then this technique can be used to reduce I/O intensive operations. e.g. UPDATE siebel.EIM_ACCOUNT IT SET ... T_ORG_EXT_MASTER = (SELECT MIN(BT.ROW_ID) FROM siebel.S_ORG_EXT BT WHERE (BT.BU_ID = IT.MASTER_ACCNT_BI AND BT.NAME = IT.MASTER_ACCNT_NAME AND BT.LOC = IT.MASTER_ACCNT_LOC)), ... WHERE (IF_ROW_BATCH_NUM = 21 AND IF_ROW_STAT_NUM = 0 AND T_ORG_EXT__STA = 0); In the SQL above, the S_ORG_EXT_U1 index (S_ORG_EXT(NAME, LOC, BU_ID, CONFLICT_ID)) is chosen by the Cost Based Optimizer. This is an improvement on the default Siebel execution path where the S_ORG_EXT_U2 index (S_ORG_EXT(BU_ID, NAME, LOC, BU_ID, CONFLICT_ID)) is hinted, but still not good enough. In this situation an IO must be made to the S_ORG_EXT_U1 index in order to locate the matching row in Oracle. The index returns the internal Oracle RowId of the row in question, but Page 35 of 36 IN CONFIDENCE

Siebel EIM Best Practice IN CONFIDENCE

then has to do another IO on the S_ORG_EXT table to get the Siebel Row_id required by the statement. By creating a custom index on S_ORG_EXT(NAME, LOC, BU_ID, ROW_ID), then the query is optimised as it need only make a single IO on S_ORG_EXT and its indexes. Also, as this new index simply overlays an existing Siebel Ux index (but replaces the unnecessary CONFLICT_ID with the more useful ROW_ID) the risk to abnormal GUI operation is very low.

Analyse long running EIM SQL statements and add/remove indexes as appropriate. Experienced DBAs are required to assist with this task. Full regression testing is required for any changes to indexes. Indexes should only be altered after verification by experienced DBAs.

4.3 ODM Adapter Functionality

ODM enhancements have already been incorporated into OneView 6 and OneSiebel ODM High Level Designs and are not discussed in detail here. Several performance enhancements were made to the ODM Ab Initio adapters during Release 4 deployment. These changes were as follows: Sorting by sorting the data at the input of the adapter (before any throttle functionality) as well as within the GSL itself, then the data was able to be sorted as best as possible. Data sorting was the single biggest win (with the exception of removing redundant updates) during Release 4. Serial Batch Calculation although creating batches within the adapter serially as opposed to in parallel increases processing time in the Ab Initio environment, the gains observed within EIM are worth the sacrifice. These gains are a result the improved sorting that serial calculation provides. Filter before throttling the filter functionality was used so that throttled data one night would continue where it left off the previous night. This meant that the data was sorted more tightly. < End of Document >

Page 36 of 36 IN CONFIDENCE