Understanding

CUCM Traces CUCM Traces

AKHIL BEHL
NCE, GDC UC
CCIE (Voice, Security)
2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces 1
Session Objective
CUCM Traces Introduction
SDI t SDI traces
SDL traces
Explain types of traces p yp
Trace configuration
Collecting trace files from CUCM
Common trace file locations and collection
Trace file reading and analysis
Best practices for collecting and configuring Best practices for collecting and configuring
traces
2 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
CUCM Traces
Introduction
3 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
CUCM Trace Introduction
CUCM has its own logging system. These are in form of txt files
These txt files are called Traces. They help in troubleshooting and
are a very effective way to solve an issue
Th t fil id i d th d t il f i t th The trace files can provide in depth details of various events as they
occur with time stamp, event id, trace handles (TCP handle) etc
4 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
System Diagnostic Interface (SDI) Traces
CUCM (also known as System Diagnostic Interface or SDI) traces
are the user friendly call processing trace files we have available
Traces can be set at different levels to capture different activities
D t il d P id d t il d d b i f ti d hi hl Detailed : Provides detailed debug information and highly
repetitive messages that are primarily used for debugging,
including Keepalives and responses
Arbitrary : Provides low-level debug traces. This level is best
suited for debugging difficult problems. This level includes nearly
everything that is included in Detailed with the exception of everything that is included in Detailed with the exception of
KeepAlives
5 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
Error : Provides traces generated in abnormal conditions, such as
coding errors or other errors that normally should not occur
S i l P id t f ll i f ti l titi Special : Provides traces for all informational, non-repetitive
messages such as process startup messages, registration
messages, and so on. All system and device initialization traces
are at this level are at this level
State Transition : Provides traces for call processing events or
normal events traced for the subsystem(signaling layers) normal events traced for the subsystem (signaling layers)
Significant : Provides trace for Media Layer events
SCCP
Signaling
RTP
6 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
RTP
Voice Stream
System Distribution Layer : SDL Traces
An SDL Trace log file contains call-processing information from
services such as Cisco CallManager, Cisco CTIManager, and Cisco
TFTP TFTP
Its important to understand how the SDL files are named
SDL trace filenames will be in the format: SDLxxx_yyy_zzzzzz.txt
x is the node id , y is the application id, z is the trace file number
The node id is unique to each node in the cluster and is assigned at
the time the server is added to the System->Communications Manager
page in CCMAdmin
100 - CallManager (ccm), 200 - CTI Manager (ctimanager) etc
SDL
7 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
SDL
SDL Trace Contents
SDL trace contents
Line Number | date +timestamp | node id | type of signal | signal name | state of Line Number | date + timestamp | node id | type of signal | signal name | state of
receiving process | receiving process | sending process | tag field | signal queue
depth | additional info
Line Number starts at 000000000when the service starts It keeps incrementing Line Number starts at 000000000 when the service starts. It keeps incrementing
until the service is restarted and the value is preserved across trace files
The date/timestamp is the date and time when the trace file was written
Node id is simply a reference within each SDL trace line indicating the id of the
node that wrote the trace line
The type of signal has multiple values The type of signal has multiple values
Signal - the name of the signal
8 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
SDL Contents (Contd.)
SDL trace file contents
Line Number | date +timestamp | node id | type of signal | signal name | state of Line Number | date + timestamp | node id | type of signal | signal name | state of
receiving process | receiving process | sending process | tag field | signal queue
depth | additional info
The state of the receiving process is the state before the signal is processed The state of the receiving process is the state before the signal is processed
The receiving process is the PID of the process recieving the SDL signal
The sending process is the PIDof the process sending the SDL signal The sending process is the PID of the process sending the SDL signal
The tag field is miscellaneous information that is not commonly used in
troubleshooting
The signal queue depth outlines the number of signals waiting to be processed
in each queue
The additional info field is typically the data associated with the signal being sent
9 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
The additional info field is typically the data associated with the signal being sent
to the receiving process. For an error it will contain details about the error. It is
valid for this field to be empty
T f SDI T Types of SDI Traces
10 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
Callmanager : Types of Traces and their
meaningg
Cisco CallManager Trace Fields
Enable H245 Message Trace : This is used for H 245 signaling for Enable H245 Message Trace : This is used for H.245 signaling for
H.323 calls, including the media processing messages.
Enable PRI Trace : This activates a trace of Primary Rate Interface
(PRI) devices.
Enable ISDN Translation Trace : This is used for a Layer 3 trace of
Q.931 (ISDN messages).
Enable H225 & Gatekeeper Trace : This is used to trace the H.225
signaling messaging for H 323 calls signaling messaging for H.323 calls.
Enable Miscellaneous Trace : Activates a trace of miscellaneous
devices.
E bl C f B id T U d t f t f th Enable Conference Bridge Trace : Used to perform trace of the
conference bridges.
Enable CM Real-Time Information Server Trace : This is used for
CallManager RIS traces - the real-time information server.
11 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
CallManager RIS traces the real time information server.
Enable CDR Trace : This enables tracing of call detail record (CDR)
processing.
Callmanager : Types of Traces and their
meaning (Contd.) g ( )
Enable Analog Trunk Trace : This is used for a trace of all MGCP-based
devices using an analog interface.
Enable All Phone Device Trace : To get a trace of phone devices, including
Cisco IP SoftPhones, and shows events such as on-hook, off-hook, key
presses, and so on.
Enable MTP Trace : This trace if for media termination point devices and
transcoders.
Enable All Gateway Trace : This activates a trace of all analog and digital Enable All Gateway Trace : This activates a trace of all analog and digital
gateways.
Enable Forward and Miscellaneous Trace : Used for trace of call forwarding
and all subsystems not covered by another checkbox and all subsystems not covered by another checkbox.
Enable MGCP Trace : For MGCP messages for MGCP-based devices.
Enable Media Resource Manager Trace : This is trace for media resource
12 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
g
manager.
Other Traces
There are other service traces as well in
addition to CUCM traces
CTI traces to debug and troubleshoot CTI CTI traces to debug and troubleshoot CTI
related issues
CDR To troubleshoot CDR related issues
DB and Admin To troubleshoot DB sync DB and Admin To troubleshoot DB sync
issues, MLA logging etc
Performance and Monitoring Same as
windows Perfmon
Security services CAPF related traces
Directory services DirSync (LDAP trace)
traces
Backup and Restore Related to DRS traces
System Services CCMWeb service, common
interface etc traces
13 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
SOAP Services AXL SOAP related traces
Platform services OS Admin interface traces
CUCM Trace
Configuration
14 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
CUCM Trace Configuration g
CUCM Trace Configuration ->
Ci U ifi d i bilit T T fi ti Cisco Unified serviceability -> Trace -> Trace configuration
15 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
CUCM Trace Configuration (Contd.)
16 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
Trace Collection
17 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
CUCM Trace Collection
CUCM Traces from CUCM 5.x onwards are collected using
RTMT
RTMT is a plugin available in CUCM plugins
Download RTMT for windows and install it on your pc
Login with CUCM admin credentials with Pubs IP
Login with desired profile (default is CM_Default)
G t >T l >t d l t l Go to -> Tools -> trace and log central
18 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
CUCM Trace Collection Procedure (Step
by Step) by Step)
Step 1: Turn on Traces in CallManager
Open up CallManager Admin page and navigate to Serviceability Then go to Open up CallManager Admin page and navigate to Serviceability. Then go to
Trace Configuration, select publisher and choose the CallManager Service.
Make sure Trace is on, and that debug trace level is set to Detailed. We will
also see a lot of options with checkboxes. Make sure that the apply to all
d i l h k d d li k nodes is also checked and click save
Step 2: Download and Install RTMT (Real Time Monitoring Tool) ( g )
This tool is accessed from the normal CallManager Administration page. It
can be found under Application Plugins. Download and install this tool. Make
sure we can run the RTMT tool and log in using the admin credentials you
t l i t th b d i Ch T l T T &L use to log into the web admin page. Choose Tools ->Trace ->Trace & Log
Central (ensure that is theres a firewall between management PC and
CUCM, the ports for RTMT are opened 1090 and 1099 TCP )
19 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
CUCM Trace Collection Procedure (Step
by Step) Contd. by Step) Contd.
Step 3. Place Your Test Call
Once we have RTMT loaded up we need to place your test call or create the Once we have RTMT loaded up, we need to place your test call or create the
condition we are trying to capture. Try to repeat the problem re creation twice
so that, we can be sure that we can find it
Step 4. Collect Trace Files
Now go to RTMT window that we opened in step 2, and double click Collect
Files Find CallManager in the list and click the checkbox for All Servers Files . Find CallManager in the list, and click the checkbox for All Servers
since we may have a situation where a phone is registered to one server and
the gateway is registered to another. This is good info to have beforehand so
the search for the trace is a bit easier. Now for the file options, choose relative
d b k 5 10 i t M k h t h i range and go back 5 or 10 minutes. Make sure whatever we choose is
sufficient to capture the test call or error condition
20 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
CUCM Trace Collection
Click Collect files
21 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
CUCM Trace Collection (Contd.)
Choose CUCM traces (and any other traces desired)
22 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
CUCM Trace Collection (Contd.)
Accept the defaults for save (except delete collected log files from
server and also select Zip Files)
S l t th d t d ti f hi h t t b l d Select the date and time from which traces are to be analyzed
23 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
CUCM Trace Collection (Contd.)
The RTMT tool will collect the files for the selected time period
The files will be saved in selected location (default is C:\Program
Fil \Ci \C llM S i bilit \J Rt t\ Files\Cisco\CallManager Serviceability\J Rtmt\
<CUCMname_CiscoCallManager_sdi_2009-06-15_12-46-57.zip>
24 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
Common Trace
locations and
collection
25 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
CUCM frequently used traces and file
location location
SDI traces - /var/log/active/cm/trace/ccm/sdi
SDL traces - /var/log/active/cm/trace/ccm/sdl
Perfmon logs - /var/log/active/cm/log/ris/csv and
/var/log/active/cm/log/amc/PerfMonLog
Callmanager app log /var/log/active/syslog/CallManager* * Callmanager app log - /var/log/active/syslog/CallManager*.*
Note to get the files use the syntax - file get activelog cm/trace/ccm/sd*/*
Equivalent to CLI trace collection, in RTMT this can be done as follows
For UCM sdl and sdi traces, check the "Cisco CallManager" box for each server
For RIS Perfon logs, check the box "Cisco RIS Data Collector PerfMonLog" for
each server you are interested in
For the SystemEvent Logs and Application logs, check the box for "Event Viewer -
26 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
For the System Event Logs and Application logs, check the box for Event Viewer
System Log and Application log" for each server you are interested in
Reading CUCM
Traces
27 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
Reading CallManager SDI Traces g g
How to Read a Trace ?
Stationinit is a message from the end point to the CallManager.
StationD is the message from CallManager to endpoint.
Start finding dd=xxx, where dd is Dialed Digits and xxx is number
dialled
When you get the Dialed String, make sure to note the TCP
Handle because it doesnt changes for a Phone unless its reset.
28 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
Reading CallManager Traces (Contd.) Reading CallManager Traces (Contd.)
Now followthe tcp handle to go with flowof call Now, follow the tcp handle to go with flow of call
Alternatively (less preferred method) you can follow the Device ID y ( p ) y
(Mac address) or the IP of the endpoint
If f i i l d t h dl i t d S If a conference is involved, a new tcp handle is generated. So,
one now has to follow this new tcp handle till call lasts or
conference lasts
The dialed string space also gives a clear picture of CSS, PSS,
translation pattern hit, Route Pattern hit etc
29 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
Example Trace File p
Trace File
30 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
An Example of CFB Trace An Example of CFB Trace
In this trace we can see how the CFB Resources are configured on
the CallManager Also we can see the error of resource unavailable the CallManager. Also we can see the error of resource unavailable.
CallManager|UnicastBridgeControl::StationRegister MaxStream=0, ActiveStream=0|<CLID::CALLMANAGER01-
Cluster><NID::10.7.0.25><CT::1,100,95,1.
7687829><IP::10.7.0.25><DEV::CFB CALLMANAGER> _
04/25/2002 16:41:58.948 Cisco CallManager|UnicastBridgeControl::StationRegister CurrentAvail=0
MaxAvail=0|<CLID::CALLMANAGER01-Cluster><NID::10.7.0.25><CT::1,100,95,1.7687
829><IP::10.7.0.25><DEV::CFB_CALLMANAGER>
04/25/2002 16:41:58.838 Cisco IP Voice Media Streaming App|kRequestedCFBStreamsFailed- CFB requested streams
failure. Requested failure. Requested
streams:48Allocated streams:0App ID:Cisco IP Voice Media Streaming App Cluster ID:CALLMANAGER01-Cluster Node
ID:10.7.0.25|<CLID::CALLMANAGER01-Cluster><NID::10.7.0.25><CT::Alarm>
04/25/2002 16:41:34.306 Cisco IP Voice Media Streaming App|CIpVMSMgr::ReadConfigurationCFB DeviceName =
CFB_CALLMANAGER|<CLID::CALLMANAGER01-Cluster><NID::10.7.0.25>
04/25/200216:41:34.322Cisco IP Voice Media StreamingApp|CIpVMSMgr::ReadConfigurationCFB ServerName[0] = 04/25/2002 16:41:34.322 Cisco IP Voice Media Streaming App|CIpVMSMgr::ReadConfigurationCFB ServerName[0]
10.7.0.25, Port =
2000|<CLID::CALLMANAGER01-Cluster><NID::10.7.0.25>
04/25/2002 16:41:34.338 Cisco IP Voice Media Streaming App|CIpVMSMgr::ReadConfigurationCFB Number of Parties =
48|<CLID::CALLMANAGER01-Cluster><NID::10.7.0.25>
04/25/2002 16:41:34.338 Cisco IP Voice Media Streaming App|CIpVMSMgr::ReadConfigurationCFB Run Flag
=1|<CLID::CALLMANAGER01-Cluster><NID::10 7 0 25>
31 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
1|<CLID::CALLMANAGER01 Cluster><NID::10.7.0.25>
An Example of AC Client trace p
Call control initialization failed error message comes on ac.
1 09/07 07:55:25.953 DEBUG New trace level: DEBUG
2 09/07 07:55:25.953 INFO
3 09/07 07:55:25.953 INFO **************************************
4 09/07 07:55:25.953 INFO ****** Attendant Console Client ******
5 09/07 07:55:25.953 INFO * Version : 1.2(1 ES7)
6 09/07 07:55:25.953 INFO * Built on : Thu Nov 6 13:17:48 PST 2003
7 09/07 07:55:25.953 INFO **************************************
8 09/07 07:55:25.953 INFO
9 09/07 07:55:25.953 DEBUG Command Argument 0: C:\Program Files\Cisco\Call Manager Attendant
Console\bin\ACClient
12 09/07 07:55:26.140 WARN Line state socket creation failed on /199.42.66.144:1105:
java.net.BindException: Cannot assign requested address: Cannot bind
The issue was related to the XP ghost and on reinstalling afresh,
th ll t l
32 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
the call control came up
An Example of TFTP Trace p
TFTP trace showing an issue with file not being found in TFTP
The phone is unable to register and keeps on rebooting The phone is unable to register and keeps on rebooting
02/20/2008 17:34:15.144
TFTP|CTftpConnect::checkAndSetIsStatic()[0268F1F8][3976:132 143 64 20:25000] TFTP|CTftpConnect::checkAndSetIsStatic()[0268F1F8][3976:132.143.64.20:25000],
[sep00506003606c.cnf.xml] is [not a] Static File|<CLID::StandAloneCluster><NID::CM-1>
02/20/2008 17:34:15.144
TFTP|-->CTftpConnect::tftp[3976:132.143.64.20:25000] |<CLID::StandAloneCluster><NID::CM-1>
02/20/2008 17:34:15.144
TFTP|CTftpConnect::checkAndHandleDiagReq[0268F1F8],[3976:132.143.64.20:25000],
File[SEP00506003606C.cnf.xml]|<CLID::StandAloneCluster><NID::CM-1>
02/20/2008 17:34:15.144 TFTP|
CTftpConnect::tftp[3976:132.143.64.20:25000] [SEP00506003606C.cnf.xml] not found in bin/config cache Looking
at disk|<CLID::StandAloneCluster><NID::CM-1>

33 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
This issue was resolved by rebuilding the CNF files from TFTP service
parameters
An Example of PRI Trace p
PRI Trace showing issue with an incoming call on MGCP gateway
03/09/2010 11:26:46.940 CCM|MGCPpn9d - S0/SU1/DS1-1@FFX-BW-7-VoIP-GW - Collision Detected in PriNi2
Interface, callphase=9|<CLID::StandAloneCluster><NID::162.70.58.2><CT::2,100,135,1.2928815><IP::1
62.70.58.4><DEV::><LVL::State Transition><MASK::2000>
03/09/2010 11:26:46.940 CCM|Entering default,restarting | g , g
channel|<CLID::StandAloneCluster><NID::162.70.58.2><CT::2,100,135,1.2928815 >
<IP::162.70.58.4><DEV::><LVL::State Transition><MASK::2000>03/09/2010 11:26:46.941
CCM|MGCPpn9d::conflictingChannelHandler - Don't Restart this channel since it is marked OOS_NE under
service parameter:S0/SU1/DS1-1@FFX-BW-7-VoIP-GW=or due to MGCP gateway hardware
failure|<CLID::StandAloneCluster><NID::162.70.58.2><CT::2,100,135,1.2928815 >
This was a Hardware issue as all channels were in service
This issue was resolved by upgrading DSP firmware
34 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
Best Practices for est act ces o
Trace reading
35 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
Best Practices for Trace Readingg
Always, have an Idea about the time period when issue occurs. It
becomes very very difficult to go through hundreds of Trace files becomes very very difficult to go through hundreds of Trace files
Always Understand the Call Flow before troubleshooting y g
Information about network setup and topology always helps
d t d th fl f i li d di understand the flow of signaling and media
Always tryto save older trace files and begin afresh if debugging a Always try to save older trace files and begin afresh if debugging a
new issue
36 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
Q & A ?
37 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
Th k Y Thank You
38 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces
39 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential AKHIL BEHL CUCM Traces