Performance Tuning Overview

The goal of performance tuning is to optimize session performance by eliminating performance

bottlenecks. To tune session
performance, first identify a performance bottleneck, eliminate it, and then identify the next performance
bottleneck until
you are satisfied with the session performance. You can use the test load option to run sessions when you
tune session
performance.
If you tune all the bottlenecks, you can further optimize session performance by increasing the number of
pipeline partitions
in the session. Adding partitions can improve performance by utilizing more of the system hardware while
processing the
session.
ecause determining the best way to improve performance can be complex, change one variable at a time,
and time the session
both before and after the change. If session performance does not improve, you might want to return to the
original
configuration.
Complete the following tasks to improve session performance:
Optimize the target. !nables the Integration "ervice to write to the targets efficiently.
Optimize the source. !nables the Integration "ervice to read source data efficiently.
Optimize the mapping. !nables the Integration "ervice to transform and move data efficiently.
Optimize the transformation. !nables the Integration "ervice to process transformations in a mapping
efficiently.
Optimize the session. !nables the Integration "ervice to run the session more #uickly.
Optimize the PowerCenter components. !nables the Integration "ervice and $epository "ervice to
function optimally.
Optimize the system. !nables %ower&enter service processes to run more #uickly.
Identifying Bottlenecks
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
This chapter includes the following topics(
Identifying ottlenecks )verview
*sing Thread "tatistics
Identifying Target ottlenecks
Identifying "ource ottlenecks
Identifying +apping ottlenecks
Identifying "ession ottlenecks
Identifying "ystem ottlenecks
Identifying ottlenecks )verview
The first step in performance tuning is to identify performance bottlenecks. %erformance bottlenecks can
occur in the source and target databases, the mapping, the session, and the system. ,enerally, look for
performance bottlenecks in the following order(
Target
ource
!apping
ession
ystem
The strategy is to identify a performance bottleneck, eliminate it, and then identify the next performance
bottleneck until you are satisfied with the performance. To tune session performance, you can use the test
load option to run sessions.
"se the following methods to identify performance #ottlenecks:
$un test sessions. You can configure a test session to read from a flat file source or to write to a flat file
target to identify source and target bottlenecks.
"tudy performance details and thread statistics. You can analyze performance details to identify session
bottlenecks.
%erformance details provide information such as readfromdisk and writetodisk counters. -or more
information about performance details, see .+onitoring /orkflows. in the /orkflow Administration ,uide.
-or more information about thread statistics, see
"sing Thread tatistics.
+onitor system performance. You can use system monitoring tools to view the percentage of &%* usage,
I0) waits, and paging to identify system bottlenecks.
)nce you determine the location of a performance bottleneck, use the following guidelines to eliminate the
bottleneck(
!liminate source and target database bottlenecks. 1ave the database administrator optimize database
performance by optimizing the #uery, increasing the database network packet size, or configuring index
and key constraints.
!liminate mapping bottlenecks. -ine tune the pipeline logic and transformation settings and options in
mappings to eliminate
mapping bottlenecks.
!liminate session bottlenecks. )ptimize the session strategy and use performance details to help tune
session configuration.
!liminate system bottlenecks. 1ave the system administrator analyze information from system monitoring
tools and improve &%*
and network performance.
*sing Thread "tatistics
You can use thread statistics to identify source, target, or transformation bottlenecks. y default, the
Integration "ervice
uses one reader thread, one transformation thread, and one writer thread to process a session.
The session log provides the following thread statistics(
$un time. Amount of time the thread was running.
Idle time. Amount of time the thread is idle. It includes the time the thread is waiting for other thread
processing within
the application. Idle time includes the time the thread is blocked by the Integration "ervice, but it does not
include the
time the thread is blocked by the operating system.
usy. %ercentage of the run time the thread is not idle. %ower&enter uses the following formula to
determine the percent of
time a thread is busy(
2run time ' idle time3 0 run time 4 566
If a transformation thread is 5667 busy, consider adding a partition point in the segment. /hen you add
partition points to
the mapping, the Integration "ervice increases the number of transformation threads it uses for the
session. 1owever, if the
machine is already running at or near full capacity, then do not add more threads.
If the reader or writer thread is 5667 busy, consider using string datatypes in the source or target ports.
8on'string ports
re#uire more processing.
You can identify which thread the Integration "ervice uses the most by reading the thread statistics in the
session log.
!xample
/hen you run a session, the session log lists run information and thread statistics similar to the following
text(
99999 $*8 I8-) -)$ T,T :)A; )$;!$ ,$)*% <5=, "$& %I%!:I8! <5= 99999
+A8A,!$> %!T:?@A65B Thread <$!A;!$?5?5?5= created for the read stage of partition point
<"C?8et;ata= has completed( Total $un
Time D <EFG.G@6HBB= secs, Total Idle Time D <EIE.G65F@H= secs, usy %ercentage D <F.55IFI6=.
+A8A,!$> %!T:?@A65H Thread <T$A8"-?5?5?5?5= created for the transformation stage of partition
point <"C?8et;ata= has
completed( Total $un Time D <EFF.I65I5B= secs, Total Idle Time D <6.666666= secs, usy %ercentage D
<566.666666=.
+A8A,!$> %!T:?@A6@@ Thread </$IT!$?5?5?5= created for the write stage of partition point2s3
<;AI:Y?"TAT*"= has completed(
Total $un Time D <EFF.IGIHIA= secs, Total Idle Time D <AH@.G6@IFA= secs, usy %ercentage D <5A.GB6FBE=.
+A8A,!$> %!T:?@A6@5 99999 !8; $*8 I8-) 99999
8otice the Total $un Time and usy %ercentage statistic for each thread. The thread with the highest usy
%ercentage
identifies the bottleneck in the session.
In this session log, the Total $un Time for the transformation thread is EFF seconds and the usy
%ercentage is 5667. This
means the transformation thread was never idle for the EFF seconds. The reader and writer busy
percentages were significantly
smaller, about F7 and 5A7. In this session, the transformation thread is the bottleneck in the mapping.
You can determine which transformation takes more time than the others by adding pass'through partition
points to all
transformations in the mapping. /hen you add partition points, more transformation threads appear in the
session log. The
bottleneck is the transformation thread that takes the most time.
8ote( You can ignore high busy percentages when the total run time is short, for example, under G6
seconds. This does not
necessarily mean the thread is a bottleneck.
Identifying Target ottlenecks
The most common performance bottleneck occurs when the Integration "ervice writes to a target
database. To identify a target
bottleneck, configure a copy of the session to write to a flat file target. If the session performance increases
significantly when you write to a flat file, you have a target bottleneck. If a session already writes to a flat
file target,
you probably do not have a target bottleneck.
You can also read the thread statistics in the session log to determine if the target is the bottleneck. /hen
the Integration
"ervice spends more time on the writer thread than the transformation or reader threads, you have a target
bottleneck. -or
more information about using thread statistics to identify bottlenecks, see *sing Thread "tatistics.
&auses of target bottlenecks may include small check point intervals, small database network packet sizes,
or problems during
heavy loading operations. -or more information about eliminating a target bottleneck, see )ptimizing the
Target.
Identifying "ource ottlenecks
%erformance bottlenecks can occur when the Integration "ervice reads from a source database. You can
read the thread
statistics in the session log to determine if the source is the bottleneck. /hen the Integration "ervice
spends more time on
the reader thread than the transformation or writer threads, you have a source bottleneck. -or more
information about using
thread statistics to identify bottlenecks, see *sing Thread "tatistics.
If the session reads from a flat file source, you probably do not have a source bottleneck. You can improve
session
performance by setting the number of bytes the Integration "ervice reads per line if you read from a flat file
source.
If the session reads from a relational source, use the following methods to identify source bottlenecks(
*se a -ilter transformation.
*se a read test mapping.
*se a database #uery.
*sing a -ilter Transformation
You can use a -ilter transformation in the mapping to measure the time it takes to read source data.
Add a -ilter transformation in the mapping after each source #ualifier. "et the filter condition to false so that
no data is
processed passed the filter transformation. If the time it takes to run the new session remains about the
same, then you have
a source bottleneck.
*sing a $ead Test +apping
You can create a read test mapping to identify source bottlenecks. A read test mapping isolates the read
#uery by removing
the transformation in the mapping.
&omplete the following steps to create a read test mapping(
+ake a copy of the original mapping.
In the copied mapping, keep only the sources, source #ualifiers, and any custom Joins or #ueries.
$emove all transformations.
&onnect the source #ualifiers to a file target.
$un a session against the read test mapping. If the session performance is similar to the original session,
you have a source
bottleneck.
*sing a ;atabase Cuery
You can identify source bottlenecks by executing the read #uery directly against the source database.
&opy the read #uery directly from the session log. !xecute the #uery against the source database with a
#uery tool such as
is#l. )n /indows, you can load the result of the #uery in a file. )n *8I4 systems, you can load the result
of the #uery in
0dev0null.
+easure the #uery execution time and the time it takes for the #uery to return the first row. If there is a long
delay
between the two time measurements, you can use an optimizer hint to eliminate the source bottleneck.
&auses of source bottlenecks may include an inefficient #uery or small database network packet sizes. -or
more information
about eliminating source bottlenecks, see )ptimizing the "ource.
Identifying +apping ottlenecks
If you determine that you do not have a source or target bottleneck, you might have a mapping bottleneck.
You can identify
mapping bottlenecks by adding a -ilter transformation before each target definition. "et the filter condition
to false so
that no data is loaded into the target tables. If the time it takes to run the new session is the same as the
original
session, you have a mapping bottleneck.
You can also identify mapping bottlenecks by using performance details. 1igh errorrows and
rowsinlookupcache counters
indicate a mapping bottleneck. -or more information about performance counters, see %erformance
&ounters.
To determine which transformation in the mapping is the bottleneck, add pass'through partition points to all
transformations
possible and read the thread statistics in the session log. /hen the Integration "ervice spends more time
on one
transformation thread than the reader, writer, or other transformation threads, that transformation has a
bottleneck. -or
more information about using thread statistics to identify bottlenecks, see *sing Thread "tatistics.
-or more information about eliminating mapping bottlenecks, see )ptimizing +appings.
Identifying "ession ottlenecks
If you do not have a source, target, or mapping bottleneck, you may have a session bottleneck. You can
identify a session
bottleneck by using the performance details. The Integration "ervice creates performance details when you
enable &ollect
%erformance ;ata in the %erformance settings on the session properties.
%erformance details display information about each transformation. All transformations have some basic
counters that indicate
the number of input rows, output rows, and error rows. -or more information about performance details,
see the /orkflow
Administration ,uide.
"mall cache size, low buffer memory, and small commit intervals can cause session bottlenecks. -or more
information about
eliminating session bottlenecks, see )ptimizing "essions.
Identifying "ystem ottlenecks
After you tune the source, target, mapping, and session, you may consider tuning the system. You can
identify system
bottlenecks by using system tools to monitor &%* usage, memory usage, and paging.
The Integration "ervice uses system resources to process transformations, run sessions, and read and
write data. The
Integration "ervice also uses system memory for other data such as aggregate, Joiner, rank, and cached
lookup tables. You can
use system performance monitoring tools to monitor the amount of system resources the Integration
"ervice uses and identify
system bottlenecks.
)n /indows, you can use system tools in the Task +anager or Administrative Tools.
)n *8I4 systems, you can use system tools such as vmstat and iostat to monitor system performance.
-or more information about eliminating system bottlenecks, see )ptimizing the "ystem.
Identifying "ystem ottlenecks on /indows
)n /indows, you can view the %erformance and %rocesses tab in the Task +anager. To access the Task
+anager, press use
&trlKAltK;el, and click Task +anager. The %erformance tab in the Task +anager provides an overview of
&%* usage and total
memory used. You can view more detailed performance information by using the %erformance +onitor on
/indows. To access the
%erformance +onitor, click "tart > %rograms > Administrative Tools, and choose %erformance +onitor.
*se the /indows %erformance +onitor to create a chart that provides the following information(
%ercent processor time. If you have more than one &%*, monitor each &%* for percent processor time. If
the processors are
utilized at more than B67, you may consider adding more processors.
%ages0second. If pages0second is greater than five, you may have excessive memory pressure 2thrashing3.
You may consider
adding more physical memory.
%hysical disks percent time. The percent of time that the physical disk is busy performing read or write
re#uests. If the
percent of time is high, tune the cache for %ower&enter to use in'memory cache instead of writing to disk. If
you tune the
cache, re#uests are still in #ueue, and the disk busy percentage is at least E67, add another disk device or
upgrade to a
faster disk device. You can also use a separate disk for each partition in the session.
%hysical disks #ueue length. The number of users waiting for access to the same disk device. If physical
disk #ueue length is
greater than two, you may consider adding another disk device or upgrading the disk device. You also can
use separate disks
for the reader, writer, and transformation threads.
"erver total bytes per second. This is the number of bytes the server has sent to and received from the
network. You can use
this information to improve network bandwidth.
Identifying "ystem ottlenecks on *8I4
You can use *8I4 tools to monitor user background process, system swapping actions, &%* loading
process, and I0) load
operations. /hen you tune *8I4 systems, tune the server for a maJor database system.
*se the following *8I4 tools to identify system bottlenecks on the *8I4 system(
lsattr '! 'I sys6. *se this tool to view current system settings. This tool shows maxuproc, the maximum
level of user
background processes. You may consider reducing the amount of background process on the system.
iostat. *se this tool to monitor loading operation for every disk attached to the database server. Iostat
displays the
percentage of time that the disk was physically active. 1igh disk utilization suggests that you may need to
add more disks.
If you use disk arrays, use utilities provided with the disk arrays instead of iostat.
vmstat or sar 'w. *se this tool to monitor disk swapping actions. "wapping should not occur during the
session. If swapping
does occur, you may consider increasing the physical memory or reduce the number of memory'intensive
applications on the
disk.
sar 'u. *se this tool to monitor &%* loading. This tool provides percent usage on user, system, idle time,
and waiting time.
If the percent time spent waiting on I0) 27wio3 is high, you may consider using other under'utilized disks.
-or example, if
the source data, target data, lookup, rank, and aggregate cache files are all on the same disk, consider
putting them on
different disks.
Optimizing the Target
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
This chapter includes the following topics(
)ptimizing the Target )verview
;ropping Indexes and Ley &onstraints
Increasing ;atabase &heckpoint Intervals
*sing ulk :oads
*sing !xternal :oads
+inimizing ;eadlocks
Increasing ;atabase 8etwork %acket "ize
)ptimizing )racle Target ;atabases
)ptimizing the Target )verview
%ou can optimize the following types of targets:
-lat file
$elational
-lat -ile Target
If you use a shared storage directory for flat file targets, you can optimize session performance by ensuring
that the shared
storage directory is on a machine that is dedicated to storing and managing files, instead of performing
other tasks.
If the Integration "ervice runs on a single node and the session writes to a flat file target, you can optimize
session
performance by writing to a flat file target that is local to the Integration "ervice process node.
$elational Target
If the session writes to a relational target, you can perform the following tasks to increase performance(
;rop indexes and key constraints. -or more information, see ;ropping Indexes and Ley &onstraints.
Increase checkpoint intervals. -or more information, see Increasing ;atabase &heckpoint Intervals.
*se bulk loading. -or more information, see *sing ulk :oads.
*se external loading. -or more information, see *sing !xternal :oads.
+inimize deadlocks. -or more information, see +inimizing ;eadlocks.
Increase database network packet size. -or more information, see Increasing ;atabase 8etwork %acket
"ize.
)ptimize )racle target databases. -or more information, see )ptimizing )racle Target ;atabases.
;ropping Indexes and Ley &onstraints
/hen you define key constraints or indexes in target tables, you slow the loading of data to those tables.
To improve
performance, drop indexes and key constraints before running the session. You can rebuild those indexes
and key constraints
after the session completes.
If you decide to drop and rebuild indexes and key constraints on a regular basis, you can use the following
methods to
perform these operations each time you run the session(
*se pre'load and post'load stored procedures.
*se pre'session and post'session "C: commands. -or more information on pre'session and post'session
"C: commands, see
./orking with "essions. in the /orkflow Administration ,uide.
8ote( To optimize performance, use constraint'based loading only if necessary.
Increasing ;atabase &heckpoint Intervals
The Integration "ervice performance slows each time it waits for the database to perform a checkpoint. To
increase
performance, consider increasing the database checkpoint interval. /hen you increase the database
checkpoint interval, you
increase the likelihood that the database performs checkpoints as necessary, when the size of the
database log file reaches
its limit.
-or more information about specific database checkpoints, checkpoint intervals, and log files, consult the
database
documentation.
*sing ulk :oads
You can use bulk loading to improve the performance of a session that inserts a large amount of data into a
;@, "ybase A"!,
)racle, or +icrosoft "C: "erver database. &onfigure bulk loading in the session properties.
/hen bulk loading, the Integration "ervice bypasses the database log, which speeds performance. /ithout
writing to the
database log, however, the target database cannot perform rollback. As a result, you may not be able to
perform recovery.
/hen you use bulk loading, weigh the importance of improved session performance against the ability to
recover an incomplete
session.
/hen bulk loading to +icrosoft "C: "erver or )racle targets, define a large commit interval to increase
performance.
+icrosoft "C: "erver and )racle start a new bulk load transaction after each commit. Increasing the
commit interval reduces
the number of bulk load transactions, which increases performance. -or more information about commit
intervals, see
Increasing the &ommit Interval.
-or more information about configuring bulk loading, see ./orking with Targets. in the /orkflow
Administration ,uide
*sing !xternal :oads
You can use an external loader to increase session performance. If you have a ;@ !! or ;@ !!! target
database, you can use
the ;@ !! or ;@ !!! external loaders to bulk load target files. The ;@ !! external loader uses the
Integration "ervice
db@load utility to load data. The ;@ !!! external loader uses the ;@ Autoloader utility.
If you have a Teradata target database, you can use the Teradata external loader utility to bulk load target
files. To use
the Teradata external loader utility, set up the attributes, such as !rror :imit, Tenacity, +ax"essions, and
"leep, to
optimize performance.
If the target database runs on )racle, you can use the )racle "C:9:oader utility to bulk load target files.
/hen you load
data to an )racle database using a pipeline with multiple partitions, you can increase performance if you
create the )racle
target table with the same number of partitions you use for the pipeline.
If the target database runs on "ybase IC, you can use the "ybase IC external loader utility to bulk load
target files. If the
"ybase IC database is local to the Integration "ervice process on the *8I4 system, you can increase
performance by loading
data to target tables directly from named pipes. If you run the Integration "ervice on a grid, configure the
Integration
"ervice to check resources, make "ybase IC a resource, make the resource available on all nodes of the
grid, and then, in the
/orkflow +anager, assign the "ybase IC resource to the applicable sessions.
-or more information about configuring the Integration "ervice to check resources, see .&reating and
&onfiguring the
Integration "ervice. in the Administrator
+inimizing ;eadlocks
If the Integration "ervice encounters a deadlock when it tries to write to a target, the deadlock only affects
targets in the
same target connection group. The Integration "ervice still writes to targets in other target connection
groups.
!ncountering deadlocks can slow session performance. To improve session performance, you can
increase the number of target
connection groups the Integration "ervice uses to write to the targets in a session. To use a different target
connection
group for each target in a session, use a different database connection name for each target instance. You
can specify the
same connection information for each connection name.
-or more information about deadlocks, see ./orking with Targets. the /orkflow Administration ,uide.
Increasing ;atabase 8etwork %acket "ize
If you write to )racle, "ybase A"! or, +icrosoft "C: "erver targets, you can improve the performance by
increasing the
network packet size. Increase the network packet size to allow larger packets of data to cross the network
at one time.
Increase the network packet size based on the database you write to(
)racle. You can increase the database server network packet size in listener.ora and tnsnames.ora.
&onsult your database
documentation for additional information about increasing the packet size, if necessary.
"ybase A"! and +icrosoft "C:. &onsult your database documentation for information about how to
increase the packet size.
-or "ybase A"! or +icrosoft "C: "erver, you must also change the packet size in the relational
connection obJect in the
/orkflow +anager to reflect the database server packet size.
)ptimizing )racle Target ;atabases
If the target database is )racle, you can optimize the target database by checking the storage clause,
space allocation, and
rollback or undo segments.
/hen you write to an )racle database, check the storage clause for database obJects. +ake sure that
tables are using large
initial and next values. The database should also store table and index data in separate tablespaces,
preferably on different
disks.
/hen you write to )racle databases, the database uses rollback or undo segments during loads. Ask the
)racle database
administrator to ensure that the database stores rollback or undo segments in appropriate tablespaces,
preferably on
different disks. The rollback or undo segments should also have appropriate storage clauses.
You can optimize the )racle database by tuning the )racle redo log. The )racle database uses the redo
log to log loading
operations. +ake sure the redo log size and buffer size are optimal. You can view redo log properties in the
init.ora file.
If the Integration "ervice runs on a single node and the )racle instance is local to the Integration "ervice
process node,
you can optimize performance by using I%& protocol to connect to the )racle database. You can set up
)racle database
connection in listener.ora and tnsnames.ora.
-or more information about optimizing )racle databases, see the )racle documentation
Optimizing the ource
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
This chapter includes the following topics(
)ptimizing the "ource )verview
)ptimizing the Cuery
*sing &onditional -ilters
Increasing ;atabase 8etwork %acket "ize
&onnecting to )racle ;atabase "ources
*sing Teradata -ast!xport
*sing tempdb to Moin "ybase or +icrosoft "C: "erver Tables
)ptimizing the "ource )verview
If the session reads from a relational source, review the following suggestions for improving performance(
)ptimize the #uery. -or example, you may want to tune the #uery to return rows faster or create indexes
for #ueries that
contain )$;!$ Y or ,$)*% Y clauses. -or more information, see )ptimizing the Cuery.
*se conditional filters. You can use the %ower&enter conditional filter in the "ource Cualifier to improve
performance. -or
more information, see *sing &onditional -ilters.
Increase database network packet size. You can improve the performance of a source database by
increasing the network packet
size, which allows larger packets of data to cross the network at one time. -or more information, see
Increasing ;atabase
8etwork %acket "ize.
&onnect to )racle databases using I%& protocol. You can use I%& protocol to connect to the )racle
database to improve
performance. -or more information, see &onnecting to )racle ;atabase "ources.
*se the -ast!xport utility to extract Teradata data. You can create a %ower&enter session that uses
-ast!xport to read
Teradata sources #uickly. -or more information, see *sing Teradata -ast!xport.
&reate tempdb to Join "ybase A"! or +icrosoft "C: "erver tables. /hen you Join large tables on a
"ybase A"! or +icrosoft "C:
"erver database, it is possible to improve performance by creating the tempdb as an in'memory database.
-or more information,
see *sing tempdb to Moin "ybase or +icrosoft "C: "erver Tables.
)ptimizing the Cuery
If a session Joins multiple source tables in one "ource Cualifier, you might be able to improve performance
by optimizing the
#uery with optimizing hints. Also, single table select statements with an )$;!$ Y or ,$)*% Y clause
may benefit from
optimization such as adding indexes.
*sually, the database optimizer determines the most efficient way to process the source data. 1owever,
you might know
properties about the source tables that the database optimizer does not. The database administrator can
create optimizer
hints to tell the database how to execute the #uery for a particular set of source tables.
The #uery that the Integration "ervice uses to read data appears in the session log. You can also find the
#uery in the
"ource Cualifier transformation. 1ave the database administrator analyze the #uery, and then create
optimizer hints and
indexes for the source tables.
*se optimizing hints if there is a long delay between when the #uery begins executing and when
%ower&enter receives the first
row of data. &onfigure optimizer hints to begin returning rows as #uickly as possible, rather than returning
all rows at
once. This allows the Integration "ervice to process rows parallel with the #uery execution.
Cueries that contain )$;!$ Y or ,$)*% Y clauses may benefit from creating an index on the )$;!$
Y or ,$)*% Y columns.
)nce you optimize the #uery, use the "C: override option to take full advantage of these modifications. -or
more information
about using "C: override, see ."ource Cualifier Transformation. in the Transformation ,uide.
You can also configure the source database to run parallel #ueries to improve performance. -or more
information about
configuring parallel #ueries, see the database documentation.
*sing &onditional -ilters
A simple source filter on the source database can sometimes negatively impact performance because of
the lack of indexes. You
can use the %ower&enter conditional filter in the "ource Cualifier to improve performance.
/hether you should use the %ower&enter conditional filter to improve performance depends on the
session. -or example, if
multiple sessions read from the same source simultaneously, the %ower&enter conditional filter may
improve performance.
1owever, some sessions may perform faster if you filter the source data on the source database. You can
test the session with
both the database filter and the %ower&enter filter to determine which method improves performance.
Increasing ;atabase 8etwork %acket "ize
If you read from )racle, "ybase A"! or, +icrosoft "C: "erver sources, you can improve the performance
by increasing the
network packet size. Increase the network packet size to allow larger packets of data to cross the network
at one time.
Increase the network packet size based on the database you read from(
)racle. You can increase the database server network packet size in listener.ora and tnsnames.ora.
&onsult your database
documentation for additional information about increasing the packet size, if necessary.
"ybase A"! and +icrosoft "C:. &onsult your database documentation for information about how to
increase the packet size.
-or "ybase A"! or +icrosoft "C: "erver, you must also change the packet size in the relational
connection obJect in the
/orkflow +anager to reflect the database server packet size.
&onnecting to )racle ;atabase "ources
If you are running the Integration "ervice on a single node and the )racle instance is local to the
Integration "ervice
process node, you can optimize performance by using I%& protocol to connect to the )racle database. You
can set up an )racle
database connection in listener.ora and tnsnames.ora.
*sing Teradata -ast!xport
-ast!xport is a utility that uses multiple Teradata sessions to #uickly export large amounts of data from a
Teradata
database. You can create a %ower&enter session that uses -ast!xport to read Teradata sources #uickly.
To use -ast!xport, create a mapping with a Teradata source database. In the session, use -ast!xport
reader instead of $elational reader. *se a -ast!xport connection to the Teradata tables that you want to
export in a session. -or more information about how to use the Integration "ervice with the Teradata
-ast!xport utility, see ./orking with "ources. in the /orkflow Administration ,uide.
Optimizing !appings
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
This chapter includes the following topics(
)ptimizing +appings )verview
)ptimizing -lat -ile "ources
&onfiguring "ingle'%ass $eading
)ptimizing "imple %ass Through +appings
)ptimizing -ilters
)ptimizing ;atatype &onversions
)ptimizing !xpressions
)ptimizing !xternal %rocedures
)ptimizing +appings )verview
+apping'level optimization may take time to implement, but it can significantly boost session performance.
-ocus on
mapping'level optimization after you optimize the targets and sources.
,enerally, you reduce the number of transformations in the mapping and delete unnecessary links between
transformations to
optimize the mapping. &onfigure the mapping with the least number of transformations and expressions to
do the most amount of
work possible. ;elete unnecessary links between transformations to minimize the amount of data moved.
You can also perform the following tasks to optimize the mapping(
)ptimize the flat file sources. -or example, you can improve session performance if the source flat file does
not contain
#uotes or escape characters. -or more information, see )ptimizing -lat -ile "ources.
&onfigure single'pass reading. You can use single'pass readings to reduce the number of times the
Integration "ervice reads
sources. -or more information, see &onfiguring "ingle'%ass $eading.
)ptimize "imple %ass Through mappings. You can use simple pass through mappings to improve session
throughput. -or more
information, see )ptimizing "imple %ass Through +appings.
)ptimize filters. You can use "ource Cualifier transformations and -ilter transformations to filter data in the
mapping. To
maximize session performance, only process records that belong in the pipeline. -or more information, see
)ptimizing -ilters.
)ptimize datatype conversions. You can increase performance by eliminating unnecessary datatype
conversions. -or more
information, see )ptimizing ;atatype &onversions.
)ptimize expressions. -or example, minimize aggregate function calls and replace common expressions
with local variables. -or
more information, see )ptimizing !xpressions.
)ptimize external procedures. If an external procedure needs to alternate reading from input groups, you
can block input data
to increase session performance. -or more information, see )ptimizing !xternal %rocedures.
)ptimizing -lat -ile "ources
&omplete the following tasks to optimize flat file sources(
)ptimize the line se#uential buffer length.
)ptimize delimited flat file sources.
)ptimizing 4+: and flat file sources.
)ptimizing the :ine "e#uential uffer :ength
If the session reads from a flat file source, you can improve session performance by setting the number of
bytes the
Integration "ervice reads per line. y default, the Integration "ervice reads 56@A bytes per line. If each line
in the source
file is less than the default setting, you can decrease the line se#uential buffer length in the session
properties.
)ptimizing ;elimited -lat -ile "ources
If a source is a delimited flat file, you must specify the delimiter character to separate columns of data in the
source
file. You must also specify the escape character. The Integration "ervice reads the delimiter character as a
regular
character if you include the escape character before the delimiter character. You can improve session
performance if the
source flat file does not contain #uotes or escape characters.
-or more information about using delimited flat file sources, see ./orking with "ources. the /orkflow
Administration ,uide.
)ptimizing 4+: and -lat -ile "ources
4+: files are usually larger than flat files because of the tag information. The size of an 4+: file depends
on the level of
tagging in the 4+: file. +ore tags result in a larger file size. As a result, the Integration "ervice may take
longer to read
and cache 4+: sources.
&onfiguring "ingle'%ass $eading
"ingle'pass reading allows you to populate multiple targets with one source #ualifier. &onsider using
single'pass reading if
you have multiple sessions that use the same sources. You can combine the transformation logic for each
mapping in one
mapping and use one source #ualifier for each source. The Integration "ervice reads each source once
and then sends the data
into separate pipelines. A particular row can be used by all the pipelines, by any combination of pipelines,
or by no
pipelines.
-or example, you have the %urchasing source table, and you use that source daily to perform an
aggregation and a ranking. If
you place the Aggregator and $ank transformations in separate mappings and sessions, you force the
Integration "ervice to
read the same source table twice. 1owever, if you include the aggregation and ranking logic in one
mapping with one source
#ualifier, the Integration "ervice reads the %urchasing source table once, and then sends the appropriate
data to the two
separate pipelines.
/hen changing mappings to take advantage of single'pass reading, you can optimize this feature by
factoring out common
functions from mappings. -or example, if you need to subtract a percentage from the %rice ports for both
the Aggregator and
$ank transformations, you can minimize work by subtracting the percentage before splitting the pipeline.
You can use an
!xpression transformation to subtract the percentage, and then split the mapping after the transformation.
-igure E'5 shows the single'pass reading, where the mapping splits after the !xpression transformation(
-igure E'5. "ingle'%ass $eading

)ptimizing "imple %ass Through +appings
You can optimize performance for "imple %ass Through mappings. To pass directly from source to target
without any other
transformations, connect the "ource Cualifier transformation directly to the target. If you use a mapping
wizard to create a
"imple %ass Through mapping, the wizard creates an !xpression transformation between the "ource
Cualifier transformation and
the target.
-or more information about creating "imple %ass Through mappings with a mapping wizard, see .*sing
the +apping /izards. in
the ;esigner ,uide.
)ptimizing -ilters
*se one of the following methods to filter data(
*se a "ource Cualifier transformation. The "ource Cualifier transformation filter rows from relational
sources.
*se a -ilter transformation. The -ilter transformation filters data within a mapping. The -ilter transformation
filters rows
from any type of source.
If you filter rows from the mapping, you can improve efficiency by filtering early in the data flow. *se a filter
in the
"ource Cualifier transformation to remove the rows at the source. The "ource Cualifier transformation
limits the row set
extracted from a relational source.
If you cannot use a filter in the "ource Cualifier transformation, use a -ilter transformation and move it as
close to the
"ource Cualifier transformation as possible to remove unnecessary data early in the data flow. The -ilter
transformation limits the row set sent to a target.
Avoid using complex expressions in filter conditions. You can optimize -ilter transformations by using
simple integer or true0false expressions in the filter condition.
8ote( You can also use a -ilter or $outer transformation to drop reJected rows from an *pdate "trategy
transformation if you do not need to keep reJected rows.
-or more information about the "ource Cualifier transformation, see ."ource Cualifier Transformation. in
the Transformation
,uide. -or more information about the -ilter transformation, see .-ilter Transformation. in the
Transformation ,uide.
)ptimizing ;atatype &onversions
You can increase performance by eliminating unnecessary datatype conversions. -or example, if a
mapping moves data from an
Integer column to a ;ecimal column, then back to an Integer column, the unnecessary datatype conversion
slows performance.
/here possible, eliminate unnecessary datatype conversions from mappings.
*se the following datatype conversions to improve system performance(
*se integer values in place of other datatypes when performing comparisons using :ookup and -ilter
transformations. -or example, many databases store *.". zip code information as a &har or Narchar
datatype. If you convert the zip code data to an Integer datatype, the lookup database stores the zip code
HAI6I'5@IA as HAI6I5@IA. This helps increase the speed of the
lookup comparisons based on zip code.
&onvert the source dates to strings through port'to'port conversions to increase session performance. You
can either leave the ports in targets as strings or change the ports to ;ate0Time ports.
-or more information about converting datatypes, see .;atatype $eference. in the ;esigner ,uide.
)ptimizing !xpressions
You can also optimize the expressions used in the transformations. /hen possible, isolate slow
expressions and simplify them.
&omplete the following tasks to isolate the slow expressions(
$emove the expressions one'by'one from the mapping.
$un the mapping to determine the time it takes to run the mapping without the transformation.
If there is a significant difference in session run time, look for ways to optimize the slow expression.
-actoring )ut &ommon :ogic
If the mapping performs the same task in multiple places, reduce the number of times the mapping
performs the task by moving
the task earlier in the mapping. -or example, you have a mapping with five target tables. !ach target
re#uires a "ocial
"ecurity number lookup. Instead of performing the lookup five times, place the :ookup transformation in
the mapping before
the data flow splits. 8ext, pass the lookup results to all five targets.
+inimizing Aggregate -unction &alls
/hen writing expressions, factor out as many aggregate function calls as possible. !ach time you use an
aggregate function call, the Integration "ervice must search and group the data. -or example, in the
following expression, the Integration
"ervice reads &):*+8?A, finds the sum, then reads &):*+8?, finds the sum, and finally finds the sum
of the two sums(
"*+2&):*+8?A3 K "*+2&):*+8?3
If you factor out the aggregate function call, as below, the Integration "ervice adds &):*+8?A to
&):*+8?, then finds the
sum of both.
"*+2&):*+8?A K &):*+8?3
$eplacing &ommon !xpressions with :ocal Nariables
If you use the same expression multiple times in one transformation, you can make that expression a local
variable. You can use a local variable only within the transformation. 1owever, by calculating the variable
only once, you speed performance.
-or more information, see ./orking with Transformations. in the Transformation ,uide.
&hoosing 8umeric Nersus "tring )perations
The Integration "ervice processes numeric operations faster than string operations. -or example, if you
look up large amounts of data on two columns, !+%:)Y!!?8A+! and !+%:)Y!!?I;, configuring the
lookup around !+%:)Y!!?I; improves performance.
)ptimizing &har'&har and &har'Narchar &omparisons
/hen the Integration "ervice performs comparisons between &1A$ and NA$&1A$ columns, it slows
each time it finds trailing blank spaces in the row. You can use the Treat &1A$ as &1A$ )n $ead option
when you configure the Integration "ervice in the
Administration &onsole so that the Integration "ervice does not trim trailing spaces from the end of &har
source fields. -or more information about the Treat&1A$as&1A$on$ead option, see .&reating and
&onfiguring the Integration "ervice. in the
Administrator ,uide.
&hoosing ;!&);! Nersus :))L*%
/hen you use a :))L*% function, the Integration "ervice must look up a table in a database. /hen you
use a ;!&);! function, you incorporate the lookup values into the expression so the Integration "ervice
does not have to look up a separate table.
Therefore, when you want to look up a small set of unchanging values, using ;!&);! may improve
performance. -or more
information about using the ;!&);! function, see .-unctions. in the Transformation :anguage $eference.
*sing )perators Instead of -unctions
The Integration "ervice reads expressions written with operators faster than expressions with functions.
/here possible, use
operators to write expressions. -or example, you have the following expression that contains nested
&)8&AT functions(
&)8&AT2 &)8&AT2 &*"T)+!$".-I$"T?8A+!, O P3 &*"T)+!$".:A"T?8A+!3
You can rewrite that expression with the QQ operator as follows(
&*"T)+!$".-I$"T?8A+! QQ O P QQ &*"T)+!$".:A"T?8A+!
)ptimizing II- !xpressions
II- expressions can return a value and an action, which allows for more compact expressions. -or
example, you have a source
with three Y08 flags( -:,?A, -:,?, -:,?&. You want to return values based on the values of each flag.
You use the following expression(
II-2 -:,?A D PYP and -:,? D PYP A8; -:,?& D PYP,
NA:?A K NA:? K NA:?&,
II-2 -:,?A D PYP and -:,? D PYP A8; -:,?& D P8P,
NA:?A K NA:? ,
II-2 -:,?A D PYP and -:,? D P8P A8; -:,?& D PYP,
NA:?A K NA:?&,
II-2 -:,?A D PYP and -:,? D P8P A8; -:,?& D P8P,
NA:?A ,
II-2 -:,?A D P8P and -:,? D PYP A8; -:,?& D PYP,
NA:? K NA:?&,
II-2 -:,?A D P8P and -:,? D PYP A8; -:,?& D P8P,
NA:? ,
II-2 -:,?A D P8P and -:,? D P8P A8; -:,?& D PYP,
NA:?&,
II-2 -:,?A D P8P and -:,? D P8P A8; -:,?& D P8P,
6.6,
33333333
This expression re#uires B II-s, 5G A8;s, and at least @A comparisons.
If you take advantage of the II- function, you can rewrite that expression as(
II-2-:,?ADPYP, NA:?A, 6.63K II-2-:,?DPYP, NA:?, 6.63K II-2-:,?&DPYP, NA:?&, 6.63
This results in three II-s, two comparisons, two additions, and a faster session.
!valuating !xpressions
If you are not sure which expressions slow performance, evaluate the expression performance to isolate
the problem.
&omplete the following steps to evaluate expression performance(
Time the session with the original expressions.
&opy the mapping and replace half of the complex expressions with a constant.
$un and time the edited session.
+ake another copy of the mapping and replace the other half of the complex expressions with a constant.
$un and time the edited session.
)ptimizing !xternal %rocedures
You might want to block input data if the external procedure needs to alternate reading from input groups.
/ithout the blocking functionality, you would need to write the procedure code to buffer incoming data. You
can block input data instead of buffering it which usually increases session performance.
-or example, you need to create an external procedure with two input groups. The external procedure
reads a row from the first input group and then reads a row from the second input group. If you use
blocking, you can write the external procedure code to block the flow of data from one input group while it
processes the data from the other input group. /hen you write the external procedure code to block data,
you increase performance because the procedure does not need to copy the source data to a buffer.
1owever, you could write the external procedure to allocate a buffer and copy the data from one input
group to the buffer until it is ready to process the data. &opying source data to a buffer decreases
performance.
-or more information about blocking input data, see .&ustom Transformation. in the Transformation ,uide.
Optimizing Transformations
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
This chapter includes the following topics(
)ptimizing Transformations )verview
)ptimizing Aggregator Transformations
)ptimizing &ustom Transformations
)ptimizing Moiner Transformations
)ptimizing :ookup Transformations
)ptimizing "e#uence ,enerator Transformations
)ptimizing "orter Transformations
)ptimizing "ource Cualifier Transformations
)ptimizing "C: Transformations
!liminating Transformation !rrors
)ptimizing Transformations )verview
You can further optimize mappings by optimizing the transformations contained in the mappings.
You can optimize the following transformations in a mapping(
Aggregator transformations. -or more information, see )ptimizing Aggregator Transformations.
&ustom transformations. -or more information, see )ptimizing &ustom Transformations.
Moiner transformations. -or more information, see )ptimizing Moiner Transformations.
:ookup transformations. -or more information, see )ptimizing :ookup Transformations.
"e#uence ,enerator transformations. -or more information, see )ptimizing "e#uence ,enerator
Transformations.
"orter transformations. -or more information, see )ptimizing "orter Transformations.
"ource Cualifier transformations. -or more information, see )ptimizing "ource Cualifier Transformations.
"C: transformations. -or more information, see )ptimizing "C: Transformations.
)ptimizing Aggregator Transformations
Aggregator transformations often slow performance because they must group data before processing it.
Aggregator transformations need additional memory to hold intermediate group results.
You can use the following guidelines to optimize the performance of an Aggregator transformation(
,roup by simple columns.
*se sorted input.
*se incremental aggregation.
-ilter data before you aggregate it.
:imit port connections.
-or more information about the Aggregator transformation, see .Aggregator Transformation. in the
Transformation ,uide.
,roup y "imple &olumns
You can optimize Aggregator transformations when you group by simple columns. /hen possible, use
numbers instead of string and dates in the columns used for the ,$)*% Y. Avoid complex expressions in
the Aggregator expressions.
*se "orted Input
You can increase session performance by sorting data for the Aggregator transformation. *se the "orted
Input option to sort data.
The "orted Input option decreases the use of aggregate caches. /hen you use the "orted Input option,
the Integration "ervice assumes all data is sorted by group. As the Integration "ervice reads rows for a
group, it performs aggregate calculations. /hen necessary, it stores group information in memory.
The "orted Input option reduces the amount of data cached during the session and improves performance.
*se this option with the "ource Cualifier 8umber of "orted %orts option or a "orter transformation to pass
sorted data to the Aggregator transformation.
You can benefit from better performance when you use the "orted Input option in sessions with multiple
partitions.
-or more information about using "orted Input in the Aggregator transformation, see .Aggregator
Transformation. in the Transformation ,uide.
*se Incremental Aggregation
If you can capture changes from the source that affect less than half the target, you can use Incremental
Aggregation to optimize the performance of Aggregator transformations.
/hen using incremental aggregation, you apply captured changes in the source to aggregate calculations
in a session. The Integration "ervice updates the target incrementally, rather than processing the entire
source and recalculate the same calculations every time you run the session.
You can increase the index and data cache sizes to hold all data in memory without paging to disk. -or
more information, see Increasing the &ache "izes.
-or more information about using Incremental Aggregation, see .*sing Incremental Aggregation. in the
/orkflow Administration ,uide.
-ilter ;ata efore You Aggregate
-ilter the data before you aggregate it. If you use a -ilter transformation in the mapping, place the
transformation before the Aggregator transformation to reduce unnecessary aggregation.
:imit %ort &onnections
:imit the number of connected input0output or output ports to reduce the amount of data the Aggregator
transformation stores in the data cache.
)ptimizing &ustom Transformations
The Integration "ervice can pass a single row to a &ustom transformation procedure or a block of rows in
an array. You can write the procedure code to specify whether the procedure receives one row or a block of
rows. You can increase performance when the procedure receives a block of rows(
You can decrease the number of function calls the Integration "ervice and procedure make. The
Integration "ervice calls the input row notification function fewer times, and the procedure calls the output
notification function fewer times.
You can increase the locality of memory access space for the data.
You can write the procedure code to perform an algorithm on a block of data instead of each row of data.
)ptimizing Moiner Transformations
Moiner transformations can slow performance because they need additional space at run time to hold
intermediary results. You can view Moiner performance counter information to determine whether you need
to optimize the Moiner transformations. -or more information about performance tuning counters, see
%erformance &ounters.
*se the following tips to improve session performance with the Moiner transformation(
;esignate the master source as the source with fewer duplicate key values. /hen the Integration "ervice
processes a sorted Moiner transformation, it caches rows for one hundred keys at a time. If the master
source contains many rows with the same key value, the Integration "ervice must cache more rows, and
performance can be slowed.
;esignate the master source as the source with the fewer rows. ;uring a session, the Moiner
transformation compares each row of the detail source against the master source. The fewer rows in the
master, the fewer iterations of the Join comparison occur, which speeds the Join process.
%erform Joins in a database when possible. %erforming a Join in a database is faster than performing a Join
in the session. The type of database Join you use can affect performance. 8ormal Joins are faster than
outer Joins and result in fewer rows. In some cases, you cannot perform the Join in the database, such as
Joining tables from two different databases or flat file systems.
To perform a Join in a database, use the following options(
&reate a pre'session stored procedure to Join the tables in a database.
*se the "ource Cualifier transformation to perform the Join.
Moin sorted data when possible. You can improve session performance by configuring the Moiner
transformation to use sorted input. /hen you configure the Moiner transformation to use sorted data, the
Integration "ervice improves performance by minimizing disk input and output. You see the greatest
performance improvement when you work with large data sets. -or an unsorted Moiner transformation,
designate the source with fewer rows as the master source.
)ptimizing :ookup Transformations
If the lookup table is on the same database as the source table in your mapping and caching is not
feasible, Join the tables in the source database rather than using a :ookup transformation.
If you use a :ookup transformation, perform the following tasks to increase performance(
*se the optimal database driver.
&ache lookup tables.
)ptimize the lookup condition.
Index the lookup table.
)ptimize multiple lookups.
-or more information about :ookup transformations, see .:ookup Transformation. in the Transformation
,uide.
*sing )ptimal ;atabase ;rivers
The Integration "ervice can connect to a lookup table using a native database driver or an );& driver.
8ative database drivers provide better session performance than );& drivers.
&aching :ookup Tables
If a mapping contains :ookup transformations, you might want to enable lookup caching. /hen you enable
caching, the Integration "ervice caches the lookup table and #ueries the lookup cache during the session.
/hen this option is not enabled, the Integration "ervice #ueries the lookup table on a row'by'row basis.
The result of the :ookup #uery and processing is the same, whether or not you cache the lookup table.
1owever, using a lookup cache can increase session performance for smaller lookup tables. In general,
you want to cache lookup tables that need less than I66 +.
&omplete the following tasks to further enhance performance for :ookup transformations(
*se the appropriate cache type.
!nable concurrent caches
)ptimize :ookup condition matching
$educe the number of cached rows.
)verride the )$;!$ Y statement.
*se a machine with more memory.
-or more information about performance tuning tips for caches, see )ptimizing &aches. -or more
information about lookup caches, see .:ookup &aches. in the Transformation ,uide.
Types of &aches
*se the following types of caches to increase performance(
"hared cache. You can share the lookup cache between multiple transformations. You can share an
unnamed cache between transformations in the same mapping. You can share a named cache between
transformations in the same or different mappings.
%ersistent cache. If you want to save and reuse the cache files, you can configure the transformation to use
a persistent cache. *se this feature when you know the lookup table does not change between session
runs. *sing a persistent cache can improve performance because the Integration "ervice builds the
memory cache from the cache files instead of from the database.
-or more information about lookup caches, see .:ookup &aches. in the Transformation ,uide.
!nable &oncurrent &aches
/hen the Integration "ervice processes sessions that contain :ookup transformations, the Integration
"ervice builds a cache in memory when it processes the first row of data in a cached :ookup
transformation. If there are multiple :ookup transformations in a mapping, the Integration "ervice creates
the caches se#uentially when the first row of data is processed by the :ookup transformation. This slows
:ookup transformation processing.
You can enable concurrent caches to improve performance. /hen the number of additional concurrent
pipelines is set to one or more, the Integration "ervice builds caches concurrently rather than se#uentially.
%erformance improves greatly when the sessions contain a number of active transformations that may take
time to complete, such as Aggregator, Moiner, or "orter transformations. /hen you enable multiple
concurrent pipelines, the Integration "ervice no longer waits for active sessions to complete before it builds
the cache. )ther :ookup transformations in the pipeline also build caches concurrently.
-or more information about configuring concurrent caches, see .:ookup &aches. in the Transformation
,uide.
)ptimize :ookup &ondition +atching
/hen the :ookup transformation matches lookup cache data with the lookup condition, it sorts and orders
the data to determine the first matching value and the last matching value. You can configure the
transformation to return any value that matches the lookup condition. /hen you configure the :ookup
transformation to return any matching value, the transformation returns the first value that matches the
lookup condition. It does not index all ports as it does when you configure the transformation to return the
first matching value or the last matching value. /hen you use any matching value, performance can
improve because the transformation does not index on all ports, which can slow performance. -or more
information about configuring the lookup condition, see .:ookup Transformation. in the Transformation
,uide.
$educing the 8umber of &ached $ows
You can reduce the number of rows included in the cache to increase performance. *se the :ookup "C:
)verride option to add a /1!$! clause to the default "C: statement.
)verriding the )$;!$ Y "tatement
y default, the Integration "ervice generates an )$;!$ Y statement for a cached lookup. The )$;!$
Y statement contains all lookup ports. To increase performance, you can suppress the default )$;!$ Y
statement and enter an override )$;!$ Y with fewer columns.
The Integration "ervice always generates an )$;!$ Y statement, even if you enter one in the override.
%lace two dashes O''P after the )$;!$ Y override to suppress the generated )$;!$ Y statement.
-or example, a :ookup transformation uses the following lookup condition(
IT!+?I; D I8?IT!+?I;
%$I&! RD I8?%$I&!
The :ookup transformation includes three lookup ports used in the mapping, IT!+?I;, IT!+?8A+!, and
%$I&!. /hen you enter the )$;!$ Y statement, enter the columns in the same order as the ports in the
lookup condition. You must also enclose all database reserved words in #uotes.
!nter the following lookup #uery in the lookup "C: override(
"!:!&T IT!+"?;I+.IT!+?8A+!, IT!+"?;I+.%$I&!, IT!+"?;I+.IT!+?I; -$)+
IT!+"?;I+ )$;!$ Y IT!+"?;I+.IT!+?I;, IT!+"?;I+.%$I&! ''
-or more information about overriding the )rder y statement, see .:ookup Transformation. in the
Transformation ,uide.
*sing a +achine with +ore +emory
You can also increase session performance by running the session on an Integration "ervice machine with
a large amount of memory. Increase the index and data cache sizes as high as you can without straining
the machine. If the Integration "ervice machine has enough memory, increase the cache so it can hold all
data in memory without paging to disk.
)ptimizing the :ookup &ondition
If you include more than one lookup condition, place the conditions with an e#ual sign first to optimize
lookup performance.
Indexing the :ookup Table
The Integration "ervice needs to #uery, sort, and compare values in the lookup condition columns. The
index needs to include every column used in a lookup condition.
You can improve performance for the following types of lookups(
&ached lookups. To improve performance, index the columns in the lookup )$;!$ Y statement. The
session log contains the )$;!$ Y statement.
*ncached lookups. To improve performance, index the columns in the lookup condition. The Integration
"ervice issues a "!:!&T statement for each row that passes into the :ookup transformation.
)ptimizing +ultiple :ookups
If a mapping contains multiple lookups, even with caching enabled and enough heap memory, the lookups
can slow performance. Tune the :ookup transformations that #uery the largest amounts of data to improve
overall performance.
To determine which :ookup transformations process the most data, examine the
:ookup?rowsinlookupcache counters for each :ookup transformation. The :ookup transformations that
have a large number in this counter might benefit from tuning their lookup expressions. If those expressions
can be optimized, session performance improves. -or information about tuning expressions, see
)ptimizing !xpressions.
)ptimizing "e#uence ,enerator Transformations
You can optimize "e#uence ,enerator transformations by creating a reusable "e#uence ,enerator and
using it in multiple mappings simultaneously. You can also optimize "e#uence ,enerator transformations
by configuring the 8umber of &ached Nalues property.
The 8umber of &ached Nalues property determines the number of values the Integration "ervice caches at
one time. +ake sure that the 8umber of &ached Nalue is not too small. You may consider configuring the
8umber of &ached Nalues to a value greater than 5,666.
If you do not have to cache values, set the 8umber of &ache Nalues to 6. "e#uence ,enerator
transformations that do not use cache are faster than those that re#uire cache.
)ptimizing "orter Transformations
&omplete the following tasks to optimize a "orter transformation(
Allocate enough memory to sort the data.
"pecify a different work directory for each partition in the "orter transformation.
Allocating +emory
If the Integration "ervice cannot allocate enough memory to sort data, it fails the session. -or best
performance, configure "orter cache size with a value less than or e#ual to the amount of available
physical $A+ on the Integration "ervice machine. Informatica recommends allocating at least B +
2B,IBB,G6B bytes3 of physical memory to sort data using the "orter transformation. "orter cache size is set
to B,IBB,G6B bytes by default.
If the amount of incoming data is greater than the amount of "orter cache size, the Integration "ervice
temporarily stores data in the "orter transformation work directory. The Integration "ervice re#uires disk
space of at least twice the amount of incoming data when storing data in the work directory. If the amount
of incoming data is significantly greater than the "orter cache size, the Integration "ervice may re#uire
much more than twice the amount of disk space available to the work directory.
*se the following formula to determine the size of incoming data(
S input rows 2<"um2column size3= K 5G3
-or more information about sorter caches, see ."orter Transformation. in the Transformation ,uide.
/ork ;irectories for %artitions
The Integration "ervice creates temporary files when it sorts data. It stores them in a work directory. You
can specify any directory on the Integration "ervice machine to use as a work directory. y default, the
Integration "ervice uses the value specified for the T%+Temp;ir server variable.
/hen you partition a session with a "orter transformation, you can specify a different work directory for
each partition in the pipeline. To increase session performance, specify work directories on physically
separate disks on the Integration "ervice nodes.
)ptimizing "ource Cualifier Transformations
*se the "elect ;istinct option for the "ource Cualifier transformation if you want the Integration "ervice to
select uni#ue values from a source. *se "elect ;istinct option to filter unnecessary data earlier in the data
flow. This can improve performance.
-or more information about "ource Cualifier transformations, see ."ource Cualifier Transformation. in the
Transformation ,uide.
)ptimizing "C: Transformations
/hen you create an "C: transformation, you configure the transformation to use external "C: #ueries or
#ueries that you define in the transformation. /hen you configure an "C: transformation to run in script
mode, the Integration "ervice processes an external "C: script for each input row. /hen the
transformation runs in #uery mode, the Integration "ervice processes an "C: #uery that you define in the
transformation.
!ach time the Integration "ervice processes a new #uery in a session, it calls a function called
"C:%repare to create an "C: procedure and pass it to the database. /hen the #uery changes for each
input row, it has a performance impact.
/hen the transformation runs in #uery mode, you can improve performance by constructing a static #uery
in the transformation. A static #uery statement does not change, although the data in the #uery clause
changes. To create a static #uery, use parameter binding instead of string substitution in the "C: !ditor.
/hen you use parameter binding you set parameters in the #uery clause to values in the transformation
input ports.
/hen an "C: #uery contains commit and rollback #uery statements, the Integration "ervice must recreate
the "C: procedure after each commit or rollback. To optimize performance, do not use transaction
statements in an "C: transformation #uery.
/hen you create the "C: transformation, you configure how the transformation connects to the database.
You can choose a static connection or you can pass connection information to the transformation at run
time.
/hen you configure the transformation to use a static connection, you choose a connection from the
/orkflow +anager connections. The "C: transformation connects to the database once during the
session. /hen you pass dynamic connection information, the "C: transformation connects to the
database each time the transformation processes an input row.
!liminating Transformation !rrors
In large numbers, transformation errors slow the performance of the Integration "ervice. /ith each
transformation error, the Integration "ervice pauses to determine the cause of the error and to remove the
row causing the error from the data flow. 8ext, the Integration "ervice typically writes the row into the
session log file.
Transformation errors occur when the Integration "ervice encounters conversion errors, conflicting
mapping logic, and any condition set up as an error, such as null input. &heck the session log to see where
the transformation errors occur. If the errors center around particular transformations, evaluate those
transformation constraints.
If you need to run a session that generates a large number of transformation errors, it is possible to
improve performance by setting a lower tracing level. 1owever, this is not a recommended long'term
solution to transformation errors. -or more information about error tracing and performance, see $educing
!rror Tracing.
Optimizing essions
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
This chapter includes the following topics(
)ptimizing "essions )verview
*sing a ,rid
*sing %ushdown )ptimization
$un &oncurrent "essions and /orkflows
Allocating uffer +emory
)ptimizing &aches
Increasing the &ommit Interval
;isabling 1igh %recision
$educing !rror Tracing
$emoving "taging Areas
)ptimizing "essions )verview
)nce you optimize the source database, target database, and mapping, you can focus on optimizing the
session. You can perform the following tasks to improve overall performance(
*se a grid. You can increase performance by using a grid to balance the Integration "ervice workload. -or
more information, see *sing a ,rid.
*se pushdown optimization. You can increase session performance by pushing transformation logic to the
source or target database. -or more information, see *sing %ushdown )ptimization.
$un sessions and workflows concurrently. You can run independent sessions and workflows concurrently
to improve session and workflow performance. -or more information, see $un &oncurrent "essions and
/orkflows.
Allocate buffer memory. You can increase the buffer memory allocation for sources and targets that re#uire
additional memory blocks. If the Integration "ervice cannot allocate enough memory blocks to hold the
data, it fails the session. -or more information, see Allocating uffer +emory.
)ptimize caches. You can improve session performance by setting the optimal location and size for index
and data caches. -or more information, see )ptimizing &aches.
Increase the commit interval. !ach time the Integration "ervice commits changes to the target,
performance slows. You can increase session performance by increasing the interval at which the
Integration "ervice commits changes. -or more information, see Increasing the &ommit Interval.
;isable high precision. %erformance slows when the Integration "ervice reads and manipulates data with
the high precision datatype. You can disable high precision to improve session performance. -or more
information, see ;isabling 1igh %recision.
$educe errors tracing. To improve performance, you can reduce the error tracing level, which reduces the
number of log events generated by the Integration "ervice. -or more information, see $educing !rror
Tracing.
$emove staging areas. /hen you use a staging area, the Integration "ervice performs multiple passes on
the data. You can eliminate staging areas to improve session performance. -or more information, see
$emoving "taging Areas.
*sing a ,rid
You can use a grid to increase session and workflow performance. A grid is an alias assigned to a group of
nodes that allows you to automate the distribution of workflows and sessions across nodes.
/hen you use a grid, the Integration "ervice distributes workflow tasks and session threads across
multiple nodes. $unning workflows and sessions on the nodes of a grid provides the following performance
gains(
alances the Integration "ervice workload.
%rocesses concurrent sessions faster.
%rocesses partitions faster.
*sing %ushdown )ptimization
You can increase session performance by pushing transformation logic to the source or target database.
ased on the mapping and session configuration, the Integration "ervice executes "C: against the source
or target database instead of processing the transformation logic within the Integration "ervice.
$un &oncurrent "essions and /orkflows
If possible, run sessions and workflows concurrently to improve performance. -or example, if you load data
into an analytic schema, where you have dimension and fact tables, load the dimensions concurrently.
Allocating uffer +emory
/hen the Integration "ervice initializes a session, it allocates blocks of memory to hold source and target
data. The Integration "ervice allocates at least two blocks for each source and target partition. "essions
that use a large number of sources and targets might re#uire additional memory blocks. If the Integration
"ervice cannot allocate enough memory blocks to hold the data, it fails the session.
You can configure the amount of buffer memory, or you can configure the Integration "ervice to
automatically calculate buffer settings at run time. -or information about configuring these settings, see
./orking with "essions. in the /orkflow Administration ,uide.
You can increase the number of available memory blocks by adJusting the following session parameters(
;T+ uffer "ize. Increase the ;T+ buffer size on the %roperties tab in the session properties.
;efault uffer lock "ize. ;ecrease the buffer block size on the &onfig )bJect tab in the session
properties.
To configure these settings, first determine the number of memory blocks the Integration "ervice re#uires
to initialize the session. Then, based on default settings, calculate the buffer size and0or the buffer block
size to create the re#uired number of session blocks.
If you have 4+: sources or targets in a mapping, use the number of groups in the 4+: source or target in
the calculation for the total number of sources and targets.
-or example, you create a session that contains a single partition using a mapping that contains E6
sources and E6 targets. Then you make the following calculations(
You determine that the session re#uires a minimum of @66 memory blocks(
<2total number of sources K total number of targets39 @= D 2session buffer blocks3
566 9 @ D @66
ased on default settings, you determine that you can change the ;T+ uffer "ize to 5E,666,666, or you
can change the ;efault uffer lock "ize to EA,666(
2session uffer locks3 D 2.H3 9 2;T+ uffer "ize3 0 2;efault uffer lock "ize3 9 2number of partitions3
@66 D .H 9 5A@@@@@@ 0 GA666 9 5
or
@66 D .H 9 5@666666 0 EA666 9 5
8ote( -or a session that contains n partitions, set the ;T+ uffer "ize to at least n times the value for the
session with one partition. The :og +anager writes a warning message in the session log if the number of
memory blocks is so small that it causes performance degradation. The :og +anager writes this warning
message even if the number of memory blocks is enough for the session to run successfully. The warning
message also gives a suggestion for the proper value.
If you modify the ;T+ uffer "ize, increase the property by multiples of the buffer block size.
Increasing ;T+ uffer "ize
The ;T+ uffer "ize setting specifies the amount of memory the Integration "ervice uses as ;T+ buffer
memory. The Integration "ervice uses ;T+ buffer memory to create the internal data structures and buffer
blocks used to bring data into and out of the Integration "ervice. /hen you increase the ;T+ buffer
memory, the Integration "ervice creates more buffer blocks, which improves performance during
momentary slowdowns.
Increasing ;T+ buffer memory allocation generally causes performance to improve initially and then level
off. /hen you increase the ;T+ buffer memory allocation, consider the total memory available on the
Integration "ervice process system.
If you do not see a significant increase in performance, ;T+ buffer memory allocation is not a factor in
session performance.
8ote( $educing the ;T+ buffer allocation can cause the session to fail early in the process because the
Integration "ervice is unable to allocate memory to the re#uired processes.
To increase the ;T+ buffer size, open the session properties and click the %roperties tab. !dit the ;T+
uffer "ize property in the %erformance settings.
Increase the property by multiples of the buffer block size, and then run and time the session after each
increase.
)ptimizing the uffer lock "ize
;epending on the session source data, you might need to increase or decrease the buffer block size.
If the machine has limited physical memory and the mapping in the session contains a large number of
sources, targets, or partitions, you might need to decrease the buffer block size. -or more information, see
Allocating uffer +emory.
If you are manipulating unusually large rows of data, you can increase the buffer block size to improve
performance. If you do not know the approximate size of the rows, you can determine the configured row
size by completing the following steps.
To evaluate needed buffer block size(
In the +apping ;esigner, open the mapping for the session.
)pen the target instance.
&lick the %orts tab.
Add the precision for all columns in the target.
If you have more than one target in the mapping, repeat steps @ to A for each additional target to calculate
the precision for each target.
$epeat steps @ to E for each source definition in the mapping.
&hoose the largest precision of all the source and target precisions for the total precision in the buffer block
size calculation.
The total precision represents the total bytes needed to move the largest row of data. -or example, if the
total precision e#uals II,666, then the Integration "ervice re#uires II,666 bytes in the buffers to move that
row. If the buffer block size is GA,666 bytes, the Integration "ervice can move only one row at a time.
Ideally, a buffer accommodates at least 566 rows at a time. "o if the total precision is greater than I@,666,
increase the size of the buffers to improve performance.
To increase the buffer block size, open the session properties and click the &onfig )bJect tab. !dit the
;efault uffer lock "ize property in the Advanced settings.
Increase the ;T+ buffer block setting in relation to the size of the rows. As with ;T+ buffer memory
allocation, increasing buffer block size should improve performance. If you do not see an increase, buffer
block size is not a factor in session performance.
)ptimizing &aches
The Integration "ervice uses the index and data caches for Aggregator, $ank, :ookup, and Moiner
transformations. The Integration "ervice stores transformed data in the data cache before returning it to the
pipeline. It stores group information for those transformations in the index cache. Also, the Integration
"ervice uses a cache to store data for "orter transformations and 4+: target definitions.
You can configure the amount of cache memory, or you can configure the Integration "ervice to
automatically calculate cache memory settings at run time. -or information about configuring these
settings, see ./orking with "essions. in the /orkflow Administration ,uide.
If the allocated data and index cache is not large enough to store the data, the Integration "ervice stores
the data in a temporary disk file as it processes the session data. %erformance slows each time the
Integration "ervice pages to a temporary file. !xamine the performance details to determine how often the
Integration "ervice pages to a file. -or more information on performance details, see %erformance
&ounters.
%erform the following tasks to optimize caches(
:imit the number of connected input0output and output only ports.
"elect the optimal cache directory location.
Increase the data and index cache sizes.
*se the GA'bit version of %ower&enter to run large cache sessions.
:imiting the 8umber of &onnected %orts
-or transformations that use data cache, limit the number of connected input0output and output only ports.
:imiting the number of connected input0output or output ports reduces the amount of data the
transformations store in the data cache.
&ache ;irectory :ocation
If the Integration "ervice runs on a single node and you want to increase performance, ensure that the
cache directory is on the same machine as the Integration "ervice. You may encounter performance
degradation when you cache large #uantities of data on a mapped or mounted drive.
-or more information about choosing the cache directory, see ."ession &aches. in the /orkflow
Administration ,uide.
Increasing the &ache "izes
If the allocated data or index cache is not large enough to store the data, the Integration "ervice stores the
data in a temporary disk file as it processes the session data. !ach time the Integration "ervice pages to
the temporary file, performance slows.
You can examine the performance details to determine when the Integration "ervice pages to the
temporary file. The Transformation?readfromdisk or Transformation?writetodisk counters for any
Aggregator, $ank, :ookup, or Moiner transformation indicate the number of times the Integration "ervice
must page to disk to process the transformation. "ince the data cache is typically larger than the index
cache, increase the data cache more than the index cache.
If the session contains a transformation that uses a cache and you run the session on a machine with
ample memory, increase the data and index cache sizes so all data can fit in memory.
-or more information about calculating the index and data cache size for Aggregator, $ank, :ookup, or
Moiner transformations, see ."ession &aches. in the /orkflow Administration ,uide. -or more information
about calculating the "orter cache size, see ."orter Transformation. in the Transformation ,uide. -or more
information about the 4+: target cache, see ./orking with 4+: "essions. in the 4+: ,uide.
*sing the GA'bit version of %ower&enter
If you process large volumes of data or perform memory'intensive transformations, you can use the GA'bit
%ower&enter version to increase session performance. The GA'bit version provides a larger memory space
that can significantly reduce or eliminate disk input0output.
This can improve session performance in the following areas(
&aching. /ith a GA'bit platform, the Integration "ervice is not limited to the @ , cache limit of a I@'bit
platform.
;ata throughput. /ith a larger available memory space, the reader, writer, and ;T+ threads can process
larger blocks of data.
Increasing the &ommit Interval
The commit interval setting determines the point at which the Integration "ervice commits data to the
targets. !ach time the Integration "ervice commits, performance slows. Therefore, the smaller the commit
interval, the more often the Integration "ervice writes to the target database, and the slower the overall
performance.
If you increase the commit interval, the number of times the Integration "ervice commits decreases and
performance improves.
/hen you increase the commit interval, consider the log file limits in the target database. If the commit
interval is too high, the Integration "ervice may fill the database log file and cause the session to fail.
Therefore, weigh the benefit of increasing the commit interval against the additional time you would spend
recovering a failed session.
&lick the ,eneral )ptions settings in the session properties to review and adJust the commit interval.
;isabling 1igh %recision
If a session runs with high precision enabled, disabling high precision might improve session performance.
The ;ecimal datatype is a numeric datatype with a maximum precision of @B. To use a high precision
;ecimal datatype in a session, configure the Integration "ervice to recognize this datatype by selecting
!nable 1igh %recision in the session properties. 1owever, since reading and manipulating the high
precision datatype slows the Integration "ervice, you can improve session performance by disabling high
precision.
/hen you disable high precision, the Integration "ervice converts data to a double. The Integration "ervice
reads the ;ecimal row IH666EBA55IB@6IEI5FAEEEI6@B@ as IH666EBA55IB@6I x 565I. -or more
information about handling high precision data, see ./orking with "essions. in the /orkflow Administration
,uide.
&lick the %erformance settings in the session properties to enable high precision.
$educing !rror Tracing
To improve performance, you can reduce the number of log events generated by the Integration "ervice
when it runs the session. If a session contains a large number of transformation errors, and you do not
need to correct them, set the session tracing level to Terse. At this tracing level, the Integration "ervice
does not write error messages or row'level information for reJect data.
If you need to debug the mapping and you set the tracing level to Nerbose, you may experience significant
performance degradation when you run the session. ;o not use Nerbose tracing when you tune
performance.
The session tracing level overrides any transformation'specific tracing levels within the mapping. This is not
recommended as a long'term response to high levels of transformation errors.
$emoving "taging Areas
/hen you use a staging area, the Integration "ervice performs multiple passes on the data. /hen
possible, remove staging areas to improve performance. The Integration "ervice can read multiple sources
with a single pass, which may alleviate the need for staging areas. -or more information about single'pass
reading, see &onfiguring "ingle'%ass $eading.
Optimizing the PowerCenter Components
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
This chapter includes the following topics(
)ptimizing the %ower&enter &omponents )verview
)ptimizing %ower&enter $epository %erformance
)ptimizing Integration "ervice %erformance
)ptimizing the %ower&enter &omponents )verview
You can optimize performance of the following %ower&enter components(
%ower&enter repository
Integration "ervice
If you run %ower&enter on multiple machines, run the $epository "ervice and Integration "ervice on
different machines. To load large amounts of data, run the Integration "ervice on the higher processing
machine. Also, run the $epository "ervice on the machine hosting the %ower&enter repository.
)ptimizing %ower&enter $epository %erformance
&omplete the following tasks to improve %ower&enter repository performance(
!nsure the %ower&enter repository is on the same machine as the $epository "ervice process.
)rder conditions in obJect #ueries.
*se a single'node tablespace for the %ower&enter repository if you install it on a ;@ database.
:ocation of the $epository "ervice %rocess and $epository
You can optimize the performance of a $epository "ervice that you configured without the high availability
option. To optimize performance, ensure that the $epository "ervice process runs on the same machine
where the repository database resides.
)rdering &onditions in )bJect Cueries
/hen the $epository "ervice processes a parameter with multiple conditions, it processes them in the
order you enter them. To receive expected results and improve performance, enter parameters in the order
you want them to run.
-or more information about ordering conditions in obJect #ueries, see ./orking with )bJect Cueries. in the
$epository ,uide.
*sing a "ingle'8ode ;@ ;atabase Tablespace
You can optimize repository performance on I+ ;@ !!! databases when you store a %ower&enter
repository in a single'node tablespace. /hen setting up an I+ ;@ !!! database, the database
administrator can define the database on a single node.
/hen the tablespace contains one node, the %ower&enter &lient and Integration "ervice access the
repository faster than if the repository tables exist on different database nodes.
If you do not specify the tablespace name when you create, copy, or restore a repository, the ;@ system
specifies the default tablespace for each repository table. The ;@ system may or may not specify a
single'node tablespace.
)ptimizing Integration "ervice %erformance
&omplete the following tasks to improve Integration "ervice performance(
*se native drivers instead of );& drivers for the Integration "ervice.
$un the Integration "ervice in A"&II data movement mode if character data is F'bit A"&II or !&;I&.
&ache %ower&enter metadata for the $epository "ervice.
$un Integration "ervice with high availability.
8ote( /hen you configure the Integration "ervice with high availability, the Integration "ervice recovers
workflows and sessions that may fail because of temporary network or machine failures. To recover from a
workflow or session, the Integration "ervice writes the states of each workflow and session to temporary
files in a shared directory. This may decrease performance.
*sing 8ative and );& ;rivers
The Integration "ervice can use );& or native drivers to connect to databases. *se native drivers to
improve performance.
$unning the Integration "ervice in A"&II ;ata +ovement +ode
/hen all character data processed by the Integration "ervice is F'bit A"&II or !&;I&, configure the
Integration "ervice to run in the A"&II data movement mode. In A"&II mode, the Integration "ervice uses
one byte to store each character. /hen you run the Integration "ervice in *nicode mode, it uses two bytes
for each character, which can slow session performance.
&aching %ower&enter +etadata for the $epository "ervice
You can use repository agent caching to improve ;T+ process performance. /hen you enable repository
agent caching, the $epository "ervice caches metadata re#uested by the Integration "ervice. /hen you
cache metadata, the Integration "ervice reads the cache for subse#uent runs of the task rather than
fetching the metadata from the repository. )nly metadata re#uested by the Integration "ervice is cached.
-or example, you run a workflow with 5,666 sessions. The first time you run a workflow with caching
enabled, the Integration "ervice fetches the session metadata from the repository. ;uring subse#uent runs
of the workflow, the $epository "ervice fetches the session metadata from the cache. This increases ;T+
process performance.
Optimizing the ystem
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
This chapter includes the following topics(
)ptimizing the "ystem )verview
Improving 8etwork "peed
*sing +ultiple &%*s
$educing %aging
*sing %rocessor inding
)ptimizing the "ystem )verview
)ften performance slows because the session relies on inefficient connections or an overloaded Integration
"ervice process system. "ystem delays can also be caused by routers, switches, network protocols, and
usage by many users.
"low disk access on source and target databases, source and target file systems, and nodes in the domain
can slow session performance. 1ave the system administrator evaluate the hard disks on the machines.
After you determine from the system monitoring tools that you have a system bottleneck, make the
following global changes to improve the performance of all sessions(
Improve network speed. "low network connections can slow session performance. 1ave the system
administrator determine if the network runs at an optimal speed. ;ecrease the number of network hops
between the Integration "ervice process and databases. -or more information, see Improving 8etwork
"peed.
*se multiple &%*s. You can use multiple &%*s to run multiple sessions in parallel and run multiple pipeline
partitions in parallel. -or more information, see *sing +ultiple &%*s.
$educe paging. /hen an operating system runs out of physical memory, it starts paging to disk to free
physical memory. &onfigure the physical memory for the Integration "ervice process machine to minimize
paging to disk. -or more information, see $educing %aging.
*se processor binding. In a multi'processor *8I4 environment, the Integration "ervice may use a large
amount of system resources. *se processor binding to control processor usage by the Integration "ervice
process. Also, if the source and target database are on the same machine, use processor binding to limit
the resources used by the database. -or more information, see *sing %rocessor inding.
Improving 8etwork "peed
The performance of the Integration "ervice is related to network connections. A local disk can move data E
to @6 times faster than a network. &onsider the following options to minimize network activity and to
improve Integration "ervice performance.
If you use flat file as a source or target in a session and the Integration "ervice runs on a single node, store
the files on the same machine as the Integration "ervice to improve performance. /hen you store flat files
on a machine other than the Integration "ervice, session performance becomes dependent on the
performance of the network connections. +oving the files onto the Integration "ervice process system and
adding disk space might improve performance.
If you use relational source or target databases, try to minimize the number of network hops between the
source and target databases and the Integration "ervice process. +oving the target database onto a
server system might improve Integration "ervice performance.
/hen you run sessions that contain multiple partitions, have the network administrator analyze the network
and make sure it has enough bandwidth to handle the data moving across the network from all partitions.
*sing +ultiple &%*s
&onfigure the system to use more &%*s to improve performance. +ultiple &%*s allow the system to run
multiple sessions in parallel as well as multiple pipeline partitions in parallel.
1owever, additional &%*s might cause disk bottlenecks. To prevent disk bottlenecks, minimize the number
of processes accessing the disk. %rocesses that access the disk include database functions and operating
system functions. %arallel sessions or pipeline partitions also re#uire disk access.
$educing %aging
%aging occurs when the Integration "ervice process operating system runs out of memory for a particular
operation and uses the local disk for memory. You can free up more memory or increase physical memory
to reduce paging and the slow performance that results from paging. +onitor paging activity using system
tools.
You might want to increase system memory in the following circumstances(
You run a session that uses large cached lookups.
You run a session with many partitions.
If you cannot free up memory, you might want to add memory to the system
*sing %rocessor inding
In a multi'processor *8I4 environment, the Integration "ervice may use a large amount of system
resources if you run a large number of sessions. As a result, other applications on the machine may not
have enough system resources available. You can use processor binding to control processor usage by the
Integration "ervice process node. Also, if the source and target database are on the same machine, use
processor binding to limit the resources used by the database.
In a "un "olaris environment, the system administrator can create and manage a processor set using the
psrset command. The system administrator can then use the pbind command to bind the Integration
"ervice to a processor set so the processor set only runs the Integration "ervice. The "un "olaris
environment also provides the psrinfo command to display details about each configured processor and the
psradm command to change the operational status of processors. -or more information, see the system
administrator and "un "olaris documentation.
In an 1%'*4 environment, the system administrator can use the %rocess $esource +anager utility to
control &%* usage in the system. The %rocess $esource +anager allocates minimum system resources
and uses a maximum cap of resources. -or more information, see the system administrator and 1%'*4
documentation.
In an AI4 environment, system administrators can use the /orkload +anager in AI4 E: to manage system
resources during peak demands. The /orkload +anager can allocate resources and manage &%*,
memory, and disk I0) bandwidth. -or more information, see the system administrator and AI4
documentation
"sing Pipeline Partitions
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
This chapter includes the following topics(
*sing %ipeline %artitions )verview
)ptimizing the "ource ;atabase for %artitioning
)ptimizing the Target ;atabase for %artitioning
*sing %ipeline %artitions )verview
After you tune the application, databases, and system for maximum single'partition performance, you may
find that the system is under'utilized. At this point, you can configure the session to have two or more
partitions.
You can use pipeline partitioning to improve session performance. Increasing the number of partitions or
partition points increases the number of threads. Therefore, increasing the number of partitions or partition
points also increases the load on the nodes in the Integration "ervice. If the Integration "ervice node or
nodes contain ample &%* bandwidth, processing rows of data in a session concurrently can increase
session performance.
8ote( If you use a single'node Integration "ervice and you create a large number of partitions or partition
points in a session that processes large amounts of data, you can overload the system.
If you have the partitioning option, perform the following tasks to manually set up partitions(
Increase the number of partitions.
"elect the best performing partition types at particular points in a pipeline.
*se multiple &%*s.
-or more information about pipeline partitioning, see .*nderstanding %ipeline %artitioning. in the /orkflow
Administration ,uide.
Increasing the 8umber of %artitions
You can increase the number of partitions in a pipeline to improve session performance. Increasing the
number of partitions allows the Integration "ervice to create multiple connections to sources and process
partitions of source data concurrently.
/hen a session uses a file source, you can configure it to read the source with one thread or multiple
threads. &onfigure the session to read file sources with multiple threads to increase session performance.
The Integration "ervice creates multiple concurrent connections to the file source.
/hen you create a session, the /orkflow +anager validates each pipeline in the mapping for partitioning.
You can specify multiple partitions in a pipeline if the Integration "ervice can maintain data consistency
when it processes the partitioned data.
*se the following tips when you add partitions to a session(
Add one partition at a time. To best monitor performance, add one partition at a time, and note the session
settings before you add each partition.
"et ;T+ uffer +emory. /hen you increase the number of partitions, increase the ;T+ buffer size. If the
session contains n partitions, increase the ;T+ buffer size to at least n times the value for the session with
one partition. -or more information about the ;T+ uffer +emory, see Allocating uffer +emory.
"et cached values for "e#uence ,enerator. If a session has n partitions, you should not need to use the
.8umber of &ached Nalues. property for the "e#uence ,enerator transformation. If you set this value to a
value greater than 6, make sure it is at least n times the original value for the session with one partition.
%artition the source data evenly. &onfigure each partition to extract the same number of rows.
+onitor the system while running the session. If there are &%* cycles available 2@6 percent or more idle
time3, it is possible to improve session performance by adding a partition.
+onitor the system after adding a partition. If the &%* utilization does not go up, the wait for I0) time goes
up, or the total data transformation rate goes down, then there is probably a hardware or software
bottleneck. If the wait for I0) time goes up by a significant amount, then check the system for hardware
bottlenecks. )therwise, check the database configuration.
-or more information about partitioning sessions, see *sing %ipeline %artitions.
"electing the est %erforming %artition Types
You can specify different partition types at different points in the pipeline to increase session performance.
You can optimize session performance by using the database partitioning partition type for source and
target databases. You can use database partitioning for )racle and I+ ;@ sources and I+ ;@
targets. /hen you use source database partitioning, the Integration "ervice #ueries the database system
for table partition information and fetches data into the session partitions. /hen you use target database
partitioning, the Integration "ervice loads data into corresponding database partition nodes.
You can use multiple pipeline partitions and database partitions. 1owever, to improve performance, ensure
the number of pipeline partitions e#uals the number of database partitions.
To increase performance, specify partition types at the following partition points in the pipeline(
"ource Cualifier transformation. To read data from multiple flat files concurrently, specify one partition for
each flat file in the "ource Cualifier transformation. Accept the default partition type, pass'through.
-ilter transformation. "ince the source files vary in size, each partition processes a different amount of
data. "et a partition point at the -ilter transformation, and choose round'robin partitioning to balance the
load going into the -ilter transformation.
"orter transformation. To eliminate overlapping groups in the "orter and Aggregator transformations, use
hash auto'keys partitioning at the "orter transformation. This causes the Integration "ervice to group all
items with the same description into the same partition before the "orter and Aggregator transformations
process the rows. You can delete the default partition point at the Aggregator transformation.
Target. "ince the target tables are partitioned by key range, specify key range partitioning at the target to
optimize writing data to the target.
*sing +ultiple &%*s
The Integration "ervice performs read, transformation, and write processing for a pipeline in parallel. It can
process multiple partitions of a pipeline within a session, and it can process multiple sessions in parallel.
If you have a symmetric multi'processing 2"+%3 platform, you can use multiple &%*s to concurrently
process session data or partitions of data. This provides increased performance, as true parallelism is
achieved. )n a single processor platform, these tasks share the &%*, so there is no parallelism.
The Integration "ervice can use multiple &%*s to process a session that contains multiple partitions. The
number of &%*s used depends on factors such as the number of partitions, the number of threads, the
number of available &%*s, and amount or resources re#uired to process the mapping.
-or more information about partitioning, see .*nderstanding %ipeline %artitioning. the /orkflow
Administration ,uide
)ptimizing the "ource ;atabase for %artitioning
You can add partitions to increase the speed of the #uery. *sually, each partition on the reader side
represents a subset of the data to be processed.
&omplete the following tasks to optimize the source database for partitioning,
Tune the database. If the database is not tuned properly, creating partitions may not make sessions
#uicker. -or more information, see Tuning the ;atabase.
!nable parallel #ueries. "ome databases may have options that must be set to enable parallel #ueries.
&heck the database documentation for these options. If these options are off, the Integration "ervice runs
multiple partition "!:!&T statements serially.
"eparate data into different tables spaces. !ach database provides an option to separate the data into
different tablespaces. If the database allows it, use the %ower&enter "C: override feature to provide a
#uery that extracts data from a single partition.
,roup the sorted data. You can partition and group source data to increase performance for a sorted Moiner
transformation. -or more information, see ,rouping "orted ;ata.
+aximize single'sorted #ueries. -or more information, see +aximizing "ingle'"orted Cueries.
Tuning the ;atabase
If the database is not tuned properly, the results may not make the session any #uicker. You can test the
database to ensure it is tuned properly.
To verify that the database is tuned properly(
&reate a pipeline with one partition.
+easure the reader throughput in the /orkflow +onitor.
Add the partitions.
Nerify that the throughput scales linearly.
-or example, if the session has two partitions, the reader throughput should be twice as fast. If the
throughput does not scale linearly, you probably need to tune the database.
,rouping "orted ;ata
You can also partition and group the source data to increase performance for the sorted Moiner
transformation. %lace the partition point before the "orter transformation to maintain grouping and sort the
data within each group.
To group data, ensure that rows with the same key value are routed to the same partition. The best way to
ensure that data is grouped and distributed evenly among partitions is to add a hash auto'keys or key'
range partition point before the sort origin.
+aximizing "ingle'"orted Cueries
To maximize a single'sorted #uery on the database, consider the following tuning options that enable
parallelization(
&heck for configuration parameters that perform automatic tuning. -or example, )racle has a parameter
called parallel?automatic?tuning.
+ake sure intra'parallelism is enabled. Intra'parallelism is the ability to run multiple threads on a single
#uery. -or example, on )racle, look at parallel?adaptive?multi?user. )n ;@, look at intra?parallel.
Nerify the maximum number of parallel processes that are available for parallel executions. -or example,
on )racle, look at parallel?max?servers. )n ;@, look at max?agents.
Nerify the sizes for various resources used in parallelization. -or example, )racle has parameters such as
large?pool?size, shared?pool?size, hash?area?size, parallel?execution?message?size, and
optimizer?percent?parallel. ;@ has configuration parameters such as dft?fetch?size, fcm?num?buffers,
and sort?heap.
Nerify the degrees of parallelism. You may be able to set this option using a database configuration
parameter or an option on the table or #uery. -or example, )racle has parameters
parallel?threads?per?cpu and optimizer?percent?parallel. ;@ has configuration parameters such as
dft?prefetch?size, dft?degree, and max?#uery?degree.
Turn off options that may affect database scalability. -or example, disable archive logging and timed
statistics on )racle.
8ote( -or a comprehensive list of tuning options available, see the database documentation.
)ptimizing the Target ;atabase for %artitioning
If a session contains multiple partitions, the throughput for each partition should be the same as the
throughput for a single partition session. If you do not see this correlation, then the database is probably
inserting rows into the database serially.
To ensure that the database inserts rows in parallel, check the following configuration options in the target
database(
"et options in the database to enable parallel inserts. -or example, set the db?writer?processes and ;@
has max?agents options in an )racle database to enable parallel inserts. "ome databases may enable
these options by default.
&onsider partitioning the target table. If possible, try to have each partition write to a single database
partition using a $outer transformation to do this. Also, have the database partitions on separate disks to
prevent I0) contention among the pipeline partitions.
"et options in the database to enhance database scalability. -or example, disable archive logging and
timed statistics in an )racle database to enhance scalability.
%erformance &ounters
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
This appendix includes the following topics(
%erformance &ounters )verview
$eadfromdisk and /ritetodisk &ounters
$owsinlookupcache &ounter
!rrorrows &ounter
%erformance &ounters )verview
All transformations have counters. -or example, each transformation tracks the number of input rows,
output rows, and error rows for each session. "ome transformations also have performance counters. You
can use the performance counters to increase session performance.
This appendix discusses the following performance counters(
$eadfromdisk
/ritetodisk
$owsinlookupcache
!rrorrows
-or more information about performance counters, see .+onitoring /orkflows. in the Administrator ,uide.
$eadfromdisk and /ritetodisk &ounters
If a session contains Aggregator, $ank, or Moiner transformations, examine each
Transformation?readfromdisk and Transformation?writetodisk counters.
If these counters display any number other than zero, you can improve session performance by increasing
the index and data cache sizes. The Integration "ervice uses the index cache to store group information
and the data cache to store transformed data, which is typically larger. Therefore, although both the index
cache and data cache sizes affect performance, you will most likely need to increase the data cache size
more than the index cache size. -or more information about configuring cache sizes, see ."ession
&aches. in the /orkflow Administration ,uide.
If the session performs incremental aggregation, the Integration "ervice reads historical aggregate data
from the local disk during the session and writes to disk when saving historical data. As a result, the
Aggregator?readtodisk and Aggregator?writetodisk counters display numbers besides zero. 1owever, since
the Integration "ervice writes the historical data to a file at the end of the session, you can still evaluate the
counters during the session. If the counters show numbers other than zero during the session run, you can
increase performance by tuning the index and data cache sizes. Increase the index and data cache sizes
as high as you can without straining the Integration "ervice machine.
To view the session performance details while the session runs, right'click the session in the /orkflow
+onitor and choose %roperties. &lick the %roperties tab in the details dialog box.
$owsinlookupcache &ounter
+ultiple lookups can decrease session performance. It is possible to improve session performance by
locating the largest lookup tables and tuning those lookup expressions. -or more information, see
)ptimizing +ultiple :ookups.
!rrorrows &ounter
Transformation errors impact session performance. If a session has large numbers in any of the
Transformation?errorrows counters, it is possible to improve performance by eliminating the errors. -or
more information, see !liminating Transformation !rrors.