2011 IBM Corporation

Campaign Flowchart Design For Optimal Performance

Mark Griner

EMM Managing Consultant, Professional Services

Marketing Innovation Summit 2011

Agenda
IBM Unica Campaign Fundamental Features
Table mapping Flowchart operation Custom macros

Design for Performance

Design process Flowchart building blocks Campaign server throughput

Q&A

2011 IBM Corporation

Marketing Innovation Summit 2011

Target Audience
Organization Roles:
Application owner Technical application lead Application administrator

Skill Sets:
OS scripting SQL Advanced IBM Unica Campaign features

Responsibilities:
Application configuration Advanced design and performance tuning Creating reusable application objects (catalogs, custom macros, etc)

2011 IBM Corporation

IBM Unica Campaign Fundamental Features

Marketing Innovation Summit 2011

Table Mapping: Terminology

In Data Marts Dimension tables contain normalized lists of values (States, Zips, Products, etc) Fact tables contain quantitative characteristics of business transactions with foreign key for dimensions A table is either a fact or a dimension In IBM Unica Campaign: Base tables contain audience keys
For a normalized base table audience column is the primary key Denormalized base tables contain multiple records with the same audience key

Dimension tables could be joined to base tables or other dimensions A table could be mapped in multiple times in different roles
2011 IBM Corporation

Marketing Innovation Summit 2011

The Role of Table Mappings

Providing IBM Unica Campaign with metadata about audience levels and table relationships Controlling the complexity of user queries Mapping at the right audience level(s) simplifies flowchart logic Defining the structure of SQL generated by IBM Unica Campaign:
Joins between base and dimension tables are executed in the database Joins between base tables are performed on IBM Unica Campaign server Types of join (inner, outer or automatic) between two tables

2011 IBM Corporation

Marketing Innovation Summit 2011

Table Mappings: Example

Normalized Base Table Dimension of Transactions

Audience Level: Customer

De-normalized Base Table Dimension of Customer

Customer
CustID Name Address (PK)

Channel
ChannelID (PK) Description Dimension

Transactions
TxnID CustID ProdID TimeID ChannelID Quantity Amount Discount (PK) (FK) (FK) (FK) (FK)

Product
ProdID SKU Brand Dimension (PK)

Time
TimeID Date Dimension (PK)

2011 IBM Corporation

Marketing Innovation Summit 2011

Table Mapping: Example 2

Normalized Base Table

Household
HH ID (PK) Address Ciy State ZIP Dimension of HH

Audience Level: Household

Dimension of Transaction Dimension of Customer

Customer
CustID (PK) First Name Last name HH ID

Channel
ChannelID (PK) Description

Transactions
TxnID CustID ProdID TimeID ChannelID Quantity Amount Discount (PK) (FK) (FK) (FK) (FK)

Product Brand
BrandID (PK) Brand Name Description Dimension of Product ProdID (PK) SKU Brand ID Dimension of Transaction

Time
TimeID Date Dimension of Transaction (PK)

2011 IBM Corporation

Marketing Innovation Summit 2011

Understanding Flowchart Operation (No in-db)

Campaign Server
ID Lists Retrieval ID List Merge

Database
Process Select Queries

Universe and Suppression Queries

Temp Table Creation ID List Upload Process Select Query

DF Calc ID List Segment

Query for Derived Field Data Data Retrieval Temp Table Creation ID List Upload

Sorting File Output

Query for Output Fields Data Retrieval

Process Select Query

2011 IBM Corporation

Marketing Innovation Summit 2011

Understanding Flowchart Operation (With in-db)

Campaign Server Database
Process Insert into Select Queries Process Insert into Select Queries Process Select Query

Universe and Suppression Queries

Merge Query Query for Derived Field Data

DF Calc ID List Segment

Data Retrieval

Temp Table Creation ID List Upload Query for Output Fields

Process Insert into Select Query

2011 IBM Corporation

Marketing Innovation Summit 2011

Campaign Macros vs. Database Functions

Campaign GROUPBY Macros SQL Server 2005 Oracle DB2 Netezza Teradata

Aggregation Functions
AvgOf CountOf DistinctOf MaxOf MinOf SumOf StdDevOf IndexOf MedianOf RankOf
AVG COUNT COUNT (DISTINCT) MAX MIN SUM STDDEV AVG COUNT COUNT (DISTINCT) MAX MIN SUM STDDEV AVG COUNT COUNT (DISTINCT) MAX MIN SUM STDDEV AVG COUNT COUNT (DISTINCT) MAX MIN SUM STDDEV AVG COUNT COUNT (DISTINCT) MAX MIN SUM STDDEV_POP

Analytic Functions
ROW_NUMBER ROW_NUMBER / COUNT = 0.5 RANK ROW_NUMBER PERCENTILE_DISC (0.5) DENSE_RANK ROW_NUMBER ROW_NUMBER / COUNT = 0.5 DENSE_RANK ROW_NUMBER ROW_NUMBER / COUNT = 0.5 DENSE_RANK ROW_NUMBER ROW_NUMBER / COUNT = 0.5 RANK *)

2011 IBM Corporation

Marketing Innovation Summit 2011

Custom Macro Definition Example

Tools-> Custom Macros Name and parameter list Description/Help Expression
SQL that selects audience keys and optional value Includes TEMPTABLE token and <parameters> Uses {} for conditional query generation

Expression type Data source Return value type

2011 IBM Corporation

Marketing Innovation Summit 2011

Custom Macro Usage Example

Used in query definitions Listed in Query Helper Syntax help is available Top level Select SQL:
select a.CustomerID, sum(a.UnitPrice*a.Quantity a.Discount) from dbo.vOrderDetails a group by a.CustomerID

Mid stream SQL:

select a.CustomerID, sum(a.UnitPrice*a.Quantity a.Discount) from dbo.vOrderDetails a , UAC_34 b where a.CustomerID=b.CustomerID group by a.CustomerID

2011 IBM Corporation

Marketing Innovation Summit 2011

Using SQL Custom Macros with In DB Processing

03/31/2008 20:42:31.756 (4312) [I] [PROCESS] ONE_PROCESS_RUN_START Select2 ... 03/31/2008 20:42:31.776 (4312) [I] [DB QUERY] [Select2] FIONA: DROP TABLE UAC_116_4 ... 03/31/2008 20:42:35.211 (4312) [I] [DB QUERY] [Select2] FIONA (thread 00000D08): CREATE TABLE UAC_116_4 NOLOGGING AS SELECT AUTODCC.ACCT.INDIV_ID INDIV_ID, sum ( ACCT_BALANCE) Val Calculating FROM AUTODCC.ACCT , UAC_116_3 WHERE AUTODCC.ACCT.INDIV_ID = UAC_116_3.INDIV_ID GROUP BY AUTODCC.ACCT.INDIV_ID [sdbtacc:2387] 03/31/2008 20:42:48.280 (4312) [I] [DB QUERY] [Select2] FIONA (thread 00000D98): CREATE TABLE UAC_116_5 NOLOGGING AS SELECT DISTINCT UAC_116_4.INDIV_ID Using FROM UAC_116_4 INNER JOIN UAC_116_3 ON (UAC_116_4.INDIV_ID=UAC_116_3.INDIV_ID) WHERE ((UAC_116_4.VAL > 5000)) [sdbtacc:2387] 03/31/2008 20:42:54.168 (4312) [I] [DB QUERY] [Select2] FIONA (thread 00000B88): SELECT COUNT (*) FROM UAC_116_5 [sdbtacc:2387]

macro values

macro values

2011 IBM Corporation

Design For Performance

Marketing Innovation Summit 2011

Design for Performance Process

Communicate

Design/Config Design Change Execute Flowchart

Analyze

Collect Data
2011 IBM Corporation

Marketing Innovation Summit 2011

Design Best Practices

When designing individual flowcharts
Distribute workload between AC and Database
Perform data intensive operations where they can be done faster
Table joins Complex calculations

Minimize transfer of data between Campaign and databases

Leverage available resources

Campaign
Virtual memory Connections and query threads

Database
SQL structure (select distinct, order by) Query optimizer (by collecting statistics, creating indices) Bulk loader Temp table use thresholds and temp table pooling

For improving overall Campaign throughput

Optimize frequently scheduled and long running flowcharts first Consider consolidation of large number of similar user flowcharts into a system flowchart
2011 IBM Corporation

Flowchart Building Blocks

Marketing Innovation Summit 2011

Select Process
Process Configuration
Query with a single base table and dimensions Query with multiple base tables Derived fields Persisted derived fields SQL custom macros Raw SQL Using Max Cell Size

In-DB Eligibility
Yes Yes
(Full outer join)

Yes
(single SQL macro)

No Yes Yes
(with <outputtemptable> token)

No

2011 IBM Corporation

Marketing Innovation Summit 2011

Select Process Recommendations

Control table joins with appropriate table mappings If selection based on calculated fields is required
Avoid calculations in top level selects Use SQL custom macros to keep calculations in the database Use Campaign macros to calculate on Unica server

Beware of performance impact of limiting cell size

2011 IBM Corporation

Marketing Innovation Summit 2011

Extract Process
Use Extract when you need records not audience keys. For example:
Pick up new transactions/responses from a touch point system Enrich, derive, aggregate data Move data from one location/format to another

Process Configuration
Extract within one data source and no derived fields Extract from one data source landing in a file or another data source Derived fields De-duping, sorting

In DB Eligibility
Yes No No No

2011 IBM Corporation

Marketing Innovation Summit 2011

Example of Extract

2011 IBM Corporation

Marketing Innovation Summit 2011

Extract Log
Creating extract table
04/06/2011 13:26:13.081 (9532) [I] [DB QUERY] [Extract2] NorthWind (thread 00001C04): CREATE TABLE UAC_EX_1_8 (CustomerID nchar(5), OrderDate datetime, UnitPrice decimal(19,4), Quantity smallint, Discount real, Amount real) [sdbtacc:2663] ... 04/06/2011 13:26:14.094 (9532) [I] [DB QUERY] [Extract2] NorthWind (thread 0000219C): INSERT INTO UAC_EX_1_8(CustomerID, OrderDate, UnitPrice, Quantity, Discount, Amount) SELECT dbo.vOrderDetails.CustomerID, dbo.vOrderDetails.OrderDate, dbo.vOrderDetails.UnitPrice, dbo.vOrderDetails.Quantity, dbo.vOrderDetails.Discount, dbo.vOrderDetails.Amount FROM dbo.vOrderDetails WHERE (dbo.vOrderDetails.OrderDate > '1998-01-01') [sdbtacc:2663]

Populating extract table

2011 IBM Corporation

Marketing Innovation Summit 2011

Extract Process Recommendations

Perform data selection and calculations in separate Extracts If the size of extract is small use Unica macros for calculations If the size of extract is large
Write extract results to a working table Create SQL macros to do calculations on data in the working table Use Select with Derived Fields to complete calculations

When extract source does not have a common audience key

Land extract results in the marketing data mart Join extract to Customer table as a dimension (on email and last name, for example) Snapshot extract results plus an audience key from dimension to a work table Map work table at a primary audience level

2011 IBM Corporation

Marketing Innovation Summit 2011

Merge Process
Merge is always eligible for in db processing Merge(OR) is implemented as UNION of input cells temp tables Match (AND) is implemented as JOIN of input cells Exclude by default is implemented using NOT EXISTS construct Exclude could also be performed by Except and Not In queries Data source level parameters that control Exclude query structure:
UseExceptForMerge=TRUE | FALSE UseNotInForMerge=TRUE | FALSE

2011 IBM Corporation

Marketing Innovation Summit 2011

Segment Process
Process Configuration
Segment by Field Segment by Query Segment by Query with Mutual Exclusivity

In-DB Eligibility
No Same as Select Same as Select

2011 IBM Corporation

Marketing Innovation Summit 2011

Segment Process Recommendation Use process configuration eligible for in db optimization Make sure the number of segmentation cells does not exceed MaxQueryThreads setting When using queries with a single base table in a mutually exclusive Segment, control the structure of de-duping SQL with parameters:
UseExceptForMerge UseNotInForMerge

2011 IBM Corporation

Marketing Innovation Summit 2011

Sample Process
Not eligible for In DB Optimization For random sampling in database use Segment with SQL custom macro that invokes random numbers generator in the database

2011 IBM Corporation

Marketing Innovation Summit 2011

Snapshot Process
Process Configuration Options Output from a single base table and dimensions Output from multiple base tables and dimensions Output of Unica Generated Fields Output of derived fields Output de-duping In DB Eligibility Yes Yes Yes Yes (single SQL macro) No

2011 IBM Corporation

Marketing Innovation Summit 2011

Audience Process
Not eligible for in database optimization Convert processing to a single audience level by using basedimension table mappings and GROUPBY Campaign or SQL macros Or Replace Audience by a Snapshot to write out both audience keys for the current selection Map in Snapshot output table at a different audience level Use Select to pick the right keys from the output table

2011 IBM Corporation

Marketing Innovation Summit 2011

Mail List Process

Mail List process generates output list and logs to Contact History
Generation of output lists is consistent with Snapshot:
When data source and target are in the same data source, list processing happens in database When SQL Macros are used to derive output fields, the processing happens in database

Contact History logging will extract data to Campaign Server Minimize the number of cells Be cognizant of the effect that data driven offer personalization could have on performance
2011 IBM Corporation

Marketing Innovation Summit 2011

Response Process
Response process reads data from an action table and logs responses to Response History
Response action table data will be extracted to Campaign Server Response process will create and index temp tables in the system tables data source The complexity of queries that Response generates depends on the codes that are used for response attribution Using treatment codes is the most efficient and precise way to track responses
2011 IBM Corporation

Managing IBM Unica Campaign Server Throughput

Marketing Innovation Summit 2011

Key Steps to Better Campaign Server Performance

1. Monitor Campaign server workload and performance 2. Analyze workload and identify flowcharts with highest impact (frequency, run time, resource consumption) 3. Schedule and stagger flowchart runs to avoid resource contention 4. Re-design individual flowcharts to improve performance 5. Aggregate processing performed by multiple similar user flowcharts (e.g. run by UDM) into one system flowchart
2011 IBM Corporation

Questions?

Thank you!