Transaction Router Performance Issues [ID 476759.

1]
Modified 20-JAN-2011 Type TROUBLESHOOTING Status PUBLISHED

In this Document
Purpose
Last Review Date
Instructions for the Reader
Troubleshooting Details
Symptom or Error Message
Cause
Diagnostic Steps
Solution
Applies to:
Siebel Remote Server - Version: 7.5.2.104 [15258] to 8.1 [21039] - Release: V7 t
o V8
Information in this document applies to any platform.
Area(s):Remote/Replication/Anywhere
Release(s):V7 (Enterprise), V7 (Professional), V8 (Enterprise), V8 (Professional
), V6 (Siebel 2000-Enterprise)
Database(s):All Supported Databases
App Server OS(s):All Supported Platforms
Latest release tested against:V8 (Enterprise)
Keywords:Remote, Transaction, Router, slow, performance, ORA-01000, Vis Rel Even
t Opers, visibility
This document was previously published as Siebel Troubleshooting Steps 8.
Purpose
Transaction Router Performance Issues.
Last Review Date
21-Dec-2007
Instructions for the Reader
A Troubleshooting Guide is provided to assist in debugging a specific issue. Whe
n possible, diagnostic tools are included in the document to assist in troublesh
ooting.
Troubleshooting Details
Symptom or Error Message
The Transaction Router Siebel Remote server task may fail or exhibit slow perfor
mance under certain conditions which will cause a backlog in the transactions to
be routed to Siebel Remote mobile users and regional nodes.
Symptoms of a Transaction Router performance issue may include:

Mobile users or connected users working on a regional node report that they
are not seeing up to date data.

There are a large number of .dx files in the SIEBEL_ROOT\Siebsrvr\Docking\tx

nproc folder.

Increased memory usage of the Transaction Router process may eventually caus
e it to error.
Cause
At a high level there are three main causes of apparent Transaction Router perfo
rmance problems:

Visibility related transactions. These are most commonly created by position

changes made via the user interface or EIM, and running Generate Reporting Rela
tionships.

Docking rules and data distribution

Slow running queries

NOTE: If you are using MS SQL Server 7.0, setting maximum degree of parallelism
to a value greater than 1 may also cause Transaction Router to hang. Refer to Do
cument 475779.1 for more information.
Diagnostic Steps
Monitoring Performance
The first step in diagnosing a Transaction Router performance issue is to determ
ine the severity of the problem. Determine if transactions are just being routed
slowly, or if mobile users and regional nodes are receiving no transactions at
all
To monitor Transaction Router performance, run the following SQL statement every
30 minutes on the Siebel server. This SQL provides, most importantly, the numbe
r of outstanding transactions to be routed to each node, along with the time whe
n a particular node was last routed to:
For Oracle
SELECT NAME NODE, LAST_TXN_NUM, A.MAX_TXN_ID - LR.LAST_TXN_NUM TXNSTOROUTE, TO_
CHAR (LR.LAST_UPD, 'DD-MON-YY HH24:MI:SS') LASTROUTED
FROM S_NODE N, S_DOCK_STATUS LR,
(SELECT MAX(TXN_ID) MAX_TXN_ID FROM S_DOCK_TXN_LOG) A
WHERE N.EFF_END_DT IS NULL
AND LR.NODE_ID = N.ROW_ID AND LR.TYPE = 'ROUTE' AND
LR.LOCAL_FLG = 'Y' AND LR.LAST_TXN_NUM != -1
ORDER BY LR.LAST_TXN_NUM;
For MS SQL Server and DB2
SELECT NAME NODE, LAST_TXN_NUM,
A.MAX_TXN_ID - LR.LAST_TXN_NUM TXNSTOROUTE,
LR.LAST_UPD LASTROUTED
FROM S_NODE N,
S_DOCK_STATUS LR,
(SELECT MAX(TXN_ID) MAX_TXN_ID FROM S_DOCK_TXN_LOG) A
WHERE N.EFF_END_DT IS NULL
AND LR.NODE_ID = N.ROW_ID AND LR.TYPE = 'ROUTE' AND LR.LOCAL_FLG = 'Y' AND LR.LA
ST_TXN_NUM != -1
ORDER BY LR.LAST_TXN_NUM
Review the results returned by the above SQL in conjunction with all the Transac
tion Router log files. In Siebel applications version 6.x, 7.x and 8.x, the log
files are TxnRoute*.log; in Siebel applications version 5.x, they are TxnRoute*.
trc. The following are examples of the diagnostic information that can be gather
ed to help in identifying the causes listed above:

Visibility Related Transactions

If the cause is visibility related transactions, the Transaction Router log file
s will not normally contain any errors. Mobile users will be receiving transacti
ons but most likely at a reduced rate, for example:
2002-02-07 15:44:48 Client: DALEYJ, Read 49 files (102448-102496), Wrote 3 files
(71905-71907)
2002-02-07 15:44:48 Txns Read: 16505, Opers Read: 18290, Vis Opers Written: 5
32
2002-02-07 15:44:48 Found 5994 Vis Event Opers, Found 4188 Vis Rel Event Oper
s
2002-02-07 16:13:55 2002-02-07 16:13:55 Client: COWANC, Read 0 files (102448-10
2447)
This log file shows this particular Transaction Router task processing 49 .dx fi
les (00102448.dx to 00102496.dx) in the SIEBEL_ROOT\Siebsrvr\Docking\txnproc fol
der for one node named DALEYJ. It took nearly 30 minutes to write 3 .dx files to
this node's \outbox directory. Note that there a number of Visibility Related E
vent Operations (4188) that cause the Transaction Router to perform additional v
isibility checks against the database and can subsequently cause additional data
downloads to the node.
The next step is to check the contents of the .dx files being read by the Transa
ction Router using txnutl; in the above example you would check files 00102448.d
x to 00102496.dx. For further instructions on using this utility, refer to Docum
ent 475688.1. Once the .dx files have been translated with txnutl, check the con
tents for insert, update, and delete transactions on the following tables:
S_NODE_REL
S_NODE_EMP
S_POSTN_RPT_REL
S_POSTN
S_EMP_POSTN
S_SHARED_EMP
S_ORG_INT
Transactions on the above tables can significantly affect Transaction Router per
formance; this is expected behavior. Note that transactions on other tables may
cause Transaction Router to perform additional visibility checks against the dat
abase, but the above tables are known to have a significant impact on performanc
e.

Docking Rules and Data Distribution

If the cause is docking rules and data distribution, the Transaction Router log
files may appear to hang at Iteration 0 where the last entry in the Transaction
Router log files details the .dx file and transaction being processed, for examp
le:
2002-02-01 18:12:57 ***** Iter: 0
2002-02-01 18:12:57 TxnProc lroute: file 3321, txn id 7264908
The Transaction Router may then continue to process without errors or may fail w
ith error messages including:
SBL-CSC-00106: Invalid file header expecting Siebel Tall/Skinny File
SBL-CSC-00120: Not enough space in buffer cache
SBL-CSC-00200: Cannot acquire spinlock hashWrite
SBL-CSC-00301: Invalid value 0 for argument pVisDataInfo->createLatch
The above error messages indicate that that the shared visibility database visda
ta.dbf is corrupt. If there are multiple Transaction Router tasks running on the
Siebel server, all the log files should be reviewed for the above errors, becau
se it is most likely that only one of them will contain the above error while th
e others will contain the following:
SBL-CSC-00213: Invalid visibility database. Shutdown and restart *all* routers t
o rebuild.

If the above symptoms are seen, it is likely that mobile users and regional node
s will be receiving very few or no transactions, and docking rules and data dist
ribution are the most likely cause. To confirm this, gather the following:

Start a single Transaction Router task and have a DBA perform an RDBMS t
race on the Transaction Router task as detailed in one of the following document
s:

Document 476768.1 explains how to trace a Siebel server task with SQ

L Profiler on MS SQL Server

Document 476760.1 explains how to trace a Siebel server task with SQ

L Trace and TKPROF on Oracle

Document 476767.1 explains how to trace a Siebel server task with a

DB2 Event Monitor on DB2

Run the visrule script for the appropriate version of Siebel against the
server database, following the directions below appropriate to your version of
Siebel applications:

Siebel 7 & 8
For Siebel 7 and 8 there is a utility that can be used to generate these scripts
automatically. The utility is called visrulediag. To generate a script using th
is utility, open a command prompt and navigate to the SIEBEL_ROOT\SiebSrvr\bin d
irectory. Run siebenv set to set the environment parameters and issue the follow
ing command:
>visrulediag /U username /P password /C datasource /D tableowner /M G
The /M G specifies to generate a SQL script. By default the threshold value /T w
ill be 1000. At this point the utility will prompt for the dock objects. For the
purpose of this Troubleshooting step enter:
>all
This will generate the script to a file called visrulediag.sql in the SIEBEL_ROO
T\SiebSrvr\bin directory. Run visrulediag.sql against the server database and se
nd the results to Technical Support for review.
Siebel 2000
The following are the visrule scripts for Siebel applications version 6.x:
Application
Visrule Script Files
Siebel eBusiness Applications
Attachment 476759.1:1
Attachment 476759.1:2
Attachment 476759.1:3
Siebel Financial Services 6.0.2
Attachment 476759.1:4
Siebel Financial Services 6.2.1
Attachment 476759.1:5
Siebel Financial Services 6.3
Attachment 476759.1:6
Siebel Consumer Sector 6.0.2
Attachment 476759.1:7
Siebel Consumer Sector 6.0.3
Attachment 476759.1:8
Siebel Consumer Sector 6.0.4
Attachment 476759.1:9
Siebel eCommunications 6.0.1
Attachment 476759.1:10
Siebel eCommunications 6.3
Attachment 476759.1:11
Siebel Automotive 6.0.1
Attachment 476759.1:12
Siebel Automotive 6.3
Attachment 476759.1:13
Life Sciences 6.0.1
Attachment 476759.1:14
Life Sciences 6.0.2
Attachment 476759.1:15
Life Sciences 6.3
Attachment 476759.1:16

Review the output of both the RDBMS trace and the visrule check script f
rom above for queries that return greater than 100,000 rows. If a particular doc
king rule returns a large amount of data to the Transaction Router, it may be ne
cessary to deactivate a particular docking rule with the assistance of Siebel Ex
pert Services.

Slow Running Queries

If the cause is slow running queries, the Transaction Router log files will not
normally contain any errors and mobile users will be receiving transactions but
most likely at a reduced rate, for example:
2001-12-11 16:03:17 Client: MENZIEJ2, Read 0 files (18354-18353), Wrote 1 files
(7426-7426)
2001-12-11 16:03:17 Txns Read: 24, Opers Read: 1047, Vis Opers Written: 3
2001-12-11 16:03:17 Found 0 Vis Event Opers, Found 0 Vis Rel Event Opers
2001-12-11 16:08:18 Client: COLLIEJ2, Read 0 files (18354-18353), Wrote 1 files
(5750-5750)
This shows this particular Transaction Router task processing two .dx files (001
8354.dx to 0018353.dx) in the SIEBEL_ROOT\Siebsrvr\Docking\txnproc folder, for o
ne node named MENZIEJ2. It took over 5 minutes to write one .dx file to this nod
e's \outbox directory. Note there are no Visibility Related Event Operations, so
this indicates that there may be a performance issue with a query being execute
d by the Transaction Router.
Queries can run slowly for many reasons including missing or corrupt indexes, ta
ble fragmentation, insufficient resources, and underlying RDBMS problems. To inv
estigate further, start a single Transaction Router task and have a DBA perform
an RDBMS trace on the Transaction Router task as in 2.a. above. Review the RDBMS
trace to determine if there are any long running queries that could be impactin
g the Transaction Router's performance, for example:

Be suspicious of long running queries that return few rows as opposed to

long running queries that return many rows.

If there are several long running queries, be suspicious if all these qu

eries involve a common table.

Solution
To improve Transaction Router performance there are some initial steps that shou
ld be taken.
First, check that the paging file size on all Siebel server machines is 1.5 time
s the amount of physical RAM available. Also monitor the CPU and memory usage on
 the Siebel server and database server machines for high utilization of either r
esource.
Second, set the following Component parameters at the Transaction Router Compone
nt level:
Id Db Size : 409600
Node Division Factor : X
Id Db Size specifies the size of the shared visibility database, and the maximum
is 409600. Do not increase the size of Id Db Size above 409600. X equals the nu
mber of Transaction Router tasks that will be running on that application server
. The first Transaction Router task should be started with the additional parame
ter:
Id Db Recreate : TRUE
This will recreate visdata.dbf with the increased size above and can improve Tra
nsaction Router performance. Subsequent Transaction Router tasks can be started
once the defaults have been set at the component level.

Note. Starting in Siebel Version 8.x, both Transaction Processor and Transaction
Router use the shared visibility database (visdata.dbf) so please make sure th
at the Id Db Size parameter is the same for both components
If the above steps do not help improve Transaction Router performance, the follo
wing options are available for each of the identified causes:

Visibility Related Transactions

If the cause of the Transaction Router performance problem is visibility related

transactions then there are two possible solutions:

Re-extract all mobile users and regional nodes.

Allow the Transaction Router server tasks to continue processing until t

he backlog is cleared. The backlog of transactions to route to each node will mo
st likely increase until the Transaction Router has processed all the visibility
related transactions, and then the backlog will reduce more quickly. Starting a
dditional Transaction Router tasks will also increase performance, but be carefu
l not to overload the Siebel server or database server machine by starting too m
any Transaction Router tasks.

Docking Rules and Data Distribution

If the cause of the Transaction Router performance problem appears to be docking

rule related, log a Service Request with Siebel Technical Support and send the
following diagnostic information in for review:

RDBMS trace of the Transaction Router task

 Transaction Router log file

.dx files the Transaction Router is processing from the SIEBEL_ROOT\sieb

srvr\Docking\txnproc directory

The results of the visrule script

In Siebel applications version 6.x, there were a few specific examples of dockin
g rules causing Transaction Router performance problems:

In Siebel 2000, the Product Dock Object was changed to be of type Limite
d visibility rather than Enterprise. This has caused some customers to hit Trans
action Router performance issues particularly if there are a large number of Ass
ets associated to Products. Siebel Systems has recommended in some instances tha
t the Product Dock Object should be set to Enterprise visible, which means that
all Products will be downloaded to mobile users and regional nodes. In other ins
tances, specific docking rules have been de-activated. The appropriate action is
dependent on a particular customer requirements and data distribution.

Transactions which update the table S_SRC, which is part of the Source D
ock Object, are incorrectly treated as a visibility related events causing the T
ransaction Router to perform unnecessary visibility checks. Change Request 12-4B
VBP9 has been logged to address this product defect and has been fixed in Siebel
7.

Other docking rules have caused Transaction Router performance problems

caused by data distribution including rules for the Order and Organization Dock
Objects.

Slow Running Queries

If the cause of the Transaction Router appears to be a poor performing query the
n work with the DBA to see if there are any obvious underlying RDBMS reasons:

Check that all the indexes are present and valid on the tables involved
in the poor performing queries. Check the relevant version of the Siebel Data Mo
del to determine what indexes should be present.

Check that the tables and indexes involved in the poor performing querie
s are not heavily fragmented.
NOTES:

The causes described above are not an exhaustive list of potential Transacti
on Router performance issues.

The causes described above are not mutually exclusive. If you diagnose one c
ause, you should continue to investigate to ensure there are no additional issue
s.

You should also try to eliminate any other potential environment issues incl
uding operating system and network problems.

Siebel Systems strongly recommends that customers fully benchmark the Transa
ction Router performance with realistic data quantities before going into produc
tion with a new release or making significant changes in the amount or type of d
ata to be routed to mobile users and regional nodes.

For example, in production, you may plan to have 1,000 mobile clients and to reg
ularly load 500,000 accounts via EIM, followed by a run Batch Assignment. Fully
test this scenario out in a test environment prior to rolling it out to your pro
duction environment. Similar testing should be carried out in a test environment
when modifying the Organizational hierarchy to avoid encountering a Transaction
Router performance issue in your production environment.

Show Attachments Attachments

vr_check_6_0_1.sql (465.41 KB)
vr_check_6_1.sql (494.96 KB)
vr_check_6_3.sql (559.92 KB)
vr_check_CS_6_0_2.sql (603.75 KB)
vr_check_CS_6_0_3.sql (603.75 KB)
vr_check_am_6_0_1.sql (603.21 KB)
vr_check_am_6_3.sql (698.09 KB)
vr_check_cs_6_0_4.sql (611.83 KB)
vr_check_ec_6_0_1.sql (598.36 KB)
vr_check_ec_6_3.sql (693.23 KB)
vr_check_fn.sql (616.69 KB)
vr_check_fn_6_2_1.sql (740.87 KB)
vr_check_fs_6_3.sql (828.54 KB)
vr_check_ls_6_0_1.sql (619.92 KB)
vr_check_ls_6_0_2.sql (619.92 KB)
vr_check_ls_6_3.sql (717.49 KB)
Show Related Information Related
Products
Siebel > Customer Relationship Management > CRM - Enterprise Edition > Siebe
l Remote Server
Errors
ORA-1000; SBL-CSC-00213; SBL-CSC-00200; SBL-CSC-00120; SBL-CSC-00106; SBL-CSC-00
301
Back to topBack to top