You are on page 1of 92

Cisco Day at the Movies

Embedded Event Manager and IP


SLA

Dave Broache
Systems Engineer
July 2009

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 1
What Can EEM Do for YOU
Auto Fault Detection and Recovery Automated Troubleshooting
Automatically Monitors Network Automate Best Practice
Status, Generate Alerts on Network Troubleshooting Steps and Collect
Failures or Switch to a Better Link Critical Information in Time

EEM

Periodic Automatic Data Collection and Perform Automatic Configuration


Reporting Improves Productivity and When Network Devices Are Connected
Visibility for Network Managers or Removed

Auto Data Collection and Reporting Automatic Device Configuration

BRKSEC-2005
BRKDEV-1191_c2 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 2
BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 3
BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 4
BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 5
BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 6
BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 7
EEM Event Detectors
  EEM 1.0   EEM 2.2
SNMP ED Embedded Object Tracking (EOT)
Syslog ED ED
Resource ED
  EEM 2.0
Redundancy framework ED
Application ED
Interface ED   EEM 2.4
Counter ED SNMP notification ED
Timer ED XML-RPC ED
Watchdog ED   EEM 3.0
  EEM 2.1 Custom CLI ED
CLI ED Routing ED
None ED NetFlow ED
Object Insertion Removal (OIR) ED IP SLA ED
GOLD ED

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 8
EEM Platform Support Matrix

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 9
BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 10
BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 11
BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 12
BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 13
BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 14
BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 15
BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 16
BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 17
BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 18
BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 19
BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 20
BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 21
BRKSEC-2005
BRKRST-3467_c1 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 22
BRKSEC-2005
BRKRST-3467_c1 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 23
BRKSEC-2005
BRKRST-3467_c1 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 24
BRKSEC-2005
BRKRST-3467_c1 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 25
BRKSEC-2005
BRKRST-3467_c1 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 26
BRKSEC-2005
BRKRST-3467_c1 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 27
BRKSEC-2005
BRKRST-3467_c1 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 28
BRKSEC-2005
BRKRST-3467_c1 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 29
BRKSEC-2005
BRKRST-3467_c1 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 30
BRKSEC-2005
BRKRST-3467_c1 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 31
BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 32
BRKSEC-2005
BRKRST-3467_c1 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 33
BRKSEC-2005
BRKRST-3467_c1 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 34
BRKSEC-2005
BRKRST-3467_c1 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 35
BRKSEC-2005
BRKRST-3467_c1 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 36
BRKSEC-2005
BRKRST-3467_c1 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 37
BRKSEC-2005
BRKRST-3467_c1 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 38
BRKSEC-2005
BRKRST-3467_c1 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 39
BRKSEC-2005
BRKRST-3467_c1 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 40
BRKSEC-2005
BRKRST-3467_c1 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 41
BRKSEC-2005
BRKRST-3467_c1 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 42
BRKSEC-2005
BRKRST-3467_c1 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 43
BRKSEC-2005
BRKRST-3467_c1 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 44
BRKSEC-2005
BRKRST-3467_c1 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 45
BRKSEC-2005
BRKRST-3467_c1 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 46
BRKSEC-2005
BRKRST-3467_c1 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 47
BRKSEC-2005
BRKRST-3467_c1 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 48
BRKSEC-2005
BRKRST-3467_c1 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 49
BRKSEC-2005
BRKRST-3467_c1 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 50
BRKSEC-2005
BRKRST-3467_c1 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 51
BRKSEC-2005
BRKRST-3467_c1 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 52
BRKSEC-2005
BRKRST-3467_c1 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 53
EEM Site @ cisco.com—
Product Information
  New EEM release
announcements
  General product information
  EEM configuration guide
  EEM policy design guides
  Whitepaper and use
case studies

http://www.cisco.com/go/eem
BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 54
Cisco Beyond — Product Extension
Community EEM Scripting Community
  Open source scripts share,
upload, download, learn by
example
  Categories include: network
management, diagnostics,
routing, QoS, high availability,
user interface, security, etc.
  User comments and ratings

Example:

http://www.cisco.com/go/ciscobeyond
BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 55
Trivia

Q and A
1) Question: What are the 2 types of EEM policies
a user can write?

Answer: Applet, TCP script

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 56
BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 57
Cisco IOS IP SLA

Todd McCree
Cisco SE

Presentation_ID © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 58
Agenda

  SLA – The basics


  Architecture
  Configuration
  Monitoring and Debugging
  Refernce and Summary

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 59
What’s an SLA?

“A Service Level Agreement is the formalization of


the ‘Quality of the Service’ in a contract between
the Customer and the Service Provider.”

Fred Baker, Fellow of Cisco Systems

If You Can‘t Measure It—Don‘t Negotiate It...

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 60
SLA Criteria

  Easy to understand
  Attainable
  Meaningful
  Controllable
  Application/service driven

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 61
SLA Parameters

  Latency (Delay) - Propagation delay, Serialization


delay, and Queuing delay
  Jitter
  Packet Loss
  Burst Loss (multiple packets)
  Packet Reordering

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 62
The Concept of Cisco IP SLA

  If you have a running Cisco IOS® router,


turn it into an active probing device:
Synthetic Probe
Core technology in IOS
Available on most Cisco platforms from 12.0(5)T or later

  Reuse your current equipment and enhance


existing network management applications

IP SLA

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 63
IP SLA Technical Overview

  Wide measurement capabilities (UDP, TCP, ICMP)


  Near millisecond precision
  Accessible using CLI and SNMP
  Proactive notification
  Historical data storage
  Flexible scheduling options
  Already in Cisco IOS (available on most platforms)
  Almost all interfaces supported, physical, and logical

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 64
Comprehensive Hardware Support
Enterprise and Aggregation/Edge Core

Cisco IOS Software Release 12.2S

Cisco CRS-1

Cisco 12000
Cisco Catalyst 6500; Series
Cisco 10000 Cisco 7600 Series
Cisco 7300
Cisco 7200 Series
Series
Series

Access

Cisco IOS Software Releases 12.3T and 12.4

Cisco 7200 Cisco 2900,


Cisco 3800 and 3560, and
Cisco 3700
Cisco 2600/ Series 3750 Series
Cisco 1700/ Series 7300 Series
Cisco 800 2800 Series
1800 Series
Series

(Responder Only)

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 65
Agenda

  SLA – The basics


  Architecture
  Configuration
  Monitoring and Debugging
  Refernce and Summary

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 66
How Does It Work?
  Hop-by-hop analysis
  Edge-to-edge measurement

  Proactive notification Management


Application
Rising and falling thresholds
Robust threshold definition
for SLAs
SNMP traps generated when IP Host
SLA violated Configure
Collect
Thresholds can trigger SA Present

operation activation for further analysis

Cisco IOS Device


IP SLA IP SLA
Measure

Measure
(IP SLA Responder)

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 67
IP SLA Sender

  Cisco IOS device that sends probe packets


  Operation configuration takes place on the
sender only
  Once the operation is finished, all the results are to be
polled off the sender
  Target is another host (IP Host, or IP SLA Responder)
  Some operations require the target to run the IP SLA
responder (Jitter for instance), some other are working
with a simple IP Host (ICMP Ping)

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 68
IP SLA Responder

  Runs on Cisco IOS


  Configure ‘ip sla monitor responder’, or set
rttMonApplResponder.0=1 with SNMP
  Sender uses the IP SLA Control Protocol to
communicate with responder before sending the
test packets
  Responder knows the type of operation, the port used,
the duration
  Responder inserts in/out timestamps in packet payload
(measures CPU time spent)

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 69
IP SLA Operation with Responder

Control Message Ask Receiver to


IP SLA Sender Open Port 2020 on UDP IP SLA Responder

IP SLA-Control

UDP, 1967
Responder Says OK
Control
Start Listening on
Phase
UDP Port 2020

Sending Test Packets…

IP SLA-Test

UDP, 2020

Probing
Phase

Done: Stop Listening

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 70
Agenda

  SLA – The basics


  Architecture
  Configuration
  Monitoring and Debugging
  Refernce and Summary

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 71
Configuring an Operation

R3(config)#ip sla 1
R3(config-ip-sla)# ?
IP SLAs entry configuration commands:
dhcp DHCP Operation
dlsw DLSW Operation
dns DNS Query Operation
exit Exit Operation Configuration
frame-relay Frame-relay Operation
ftp FTP Operation
http HTTP Operation
icmp-echo ICMP Echo Operation
icmp-jitter ICMP Jitter Operation
mpls MPLS Operation
path-echo Path Discovered ICMP Echo Operation
path-jitter Path Discovered ICMP Jitter Operation
slm SLM Operation
tcp-connect TCP Connect Operation
udp-echo UDP Echo Operation
udp-jitter UDP Jitter Operation
voip Voice Over IP Operation

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 72
ICMP Echo Operation

  “Ping” test
  Target can be any IP host
  Response time is computed by measuring the time
taken between sending an ICMP echo request
message to the destination and receiving an ICMP
echo reply
  Processing delays on the source router is subtracted

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 73
ICMP Echo Operation (Example)

ip sla 2
icmp-echo 10.32.130.2
tos 32
frequency 120
ip sla schedule 2 life forever start-time now

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 74
ICMP Echo Operation (Output)

R3#show ip sla stat 2

Round Trip Time (RTT) for Index 2


Latest RTT: 100 ms
Latest operation start time: *17:32:53.315 CET Tue Feb 21 2006
Latest operation return code: Timeout
Number of successes: 0
Number of failures: 1
Operation time to live: Forever

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 75
UDP Jitter Operation

  Measures the delay, delay variance (jitter) and


packet loss by generating periodic UDP traffic
  Measures: per-direction jitter, per-direction packet-loss,
and round trip time
  Detect and report out-of-sequence and
corrupted packets
  One-way delay requires Cisco IOS 12.2(2)T or
later and clock synchronization between source
and destination
  Always requires IP SLA responder
  Starting Cisco IOS 12.3(4)T, the operation can measure
MOS and ICPIF scores for VoIP
BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 76
UDP Voice Jitter Operation (Example)
  Simulating G.711 VoIP call
  Use RTP/UDP ports 16384 and above, the packet size is 172 bytes
(160 bytes of payload, 12 bytes for RTP header)
  Packets are sent every 20 milliseconds (interval)
  Runs every minute (frequency)

ip sla 1
udp-jitter 10.0.0.2 5556
num-packets 1000
request-data-size 172
tos 32
ip sla schedule 1 life forever start-time now

B C
A A = 20 ms
B = 20 s (1000 x 20 ms)
C = 40 s (60 s – 20 s)

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 77
Schedule and Stop

  To schedule operation <n> to start:

ip sla schedule <n> [life seconds] [start-time {pending | now | hh:mm [month day
|day month]}][ageout seconds]

  To stop a running operation <n>:

no ip sla schedule <n>

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 78
Scheduling Caveat

  If you configure multiple operations to start ‘now’,


all will start at the same time after a router reload
  Consider using the option ‘after’ instead of ‘now’
  Example, new operations are started every second:

ip sla schedule <n> start-time


after 00:01:00
ip sla schedule <n+1> start-time
after 00:01:01
ip sla schedule <n+2> start-time
after 00:01:02

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 79
Agenda

  SLA – The basics


  Architecture
  Configuration
  Monitoring and Debugging
  Refernce and Summary

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 80
IP SLA Application Version

R3#show ip sla application


IP Service Level Agreements
Version: Round Trip Time MIB 2.2.0, Infrastructure Engine-II
Time of last change in whole IP SLAs: *17:46:22.215 CET Tue Feb 21 2006
Estimated system max number of entries: 10852

Estimated number of configurable operations: 10847


Number of Entries configured : 5
Number of active Entries : 2
Number of pending Entries : 0 Maximum Number
Number of inactive Entries : 3 of Configurable Operations
Supported Operation Types
Type of Operation to Perform: dhcp
Type of Operation to Perform: dns
Type of Operation to Perform: echo
Type of Operation to Perform: frameRelay
Type of Operation to Perform: ftp
Type of Operation to Perform: http
Type of Operation to Perform: jitter
Type of Operation to Perform: pathEcho Supported Operations
Type of Operation to Perform: pathJitter
Type of Operation to Perform: tcpConnect
Type of Operation to Perform: udpEcho
Type of Operation to Perform: voip

IP SLAs low memory water mark: 14976312


Memory Limit

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 81
Display the Configuration State

R3#show ip sla configuration 1


IP SLAs, Infrastructure Engine-II.
Entry number: 1
Owner:
Tag:
Type of operation to perform: udp-jitter
Target address/Source address: 1.1.1.1/0.0.0.0
Target port/Source port: 1000/0
Request size (ARR data portion): 32
Operation timeout (milliseconds): 5000
Packet Interval (milliseconds)/Number of packets: 20/10
Type Of Service parameters: 0x0
Verify data: No
Vrf Name:
Control Packets: enabled
Schedule:
Operation frequency (seconds): 60 (not considered if randomly scheduled)
Next Scheduled Start Time: Pending trigger
Group Scheduled : FALSE
Randomly Scheduled : FALSE
Life (seconds): 3600
Entry Ageout (seconds): never
Recurring (Starting Everyday): FALSE
Status of entry (SNMP RowStatus): notInService
Threshold (milliseconds): 5000
Distribution Statistics:
Number of statistic hours kept: 2
Number of statistic distribution buckets kept: 1
Statistic distribution interval (milliseconds): 20
Enhanced History:

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 82
Deleting Operations

  To delete one operation <n>:

router(config)# no ip sla <n>

  To delete all operations:

router(config)# ip sla reset

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 83
Debugging an Operation’s Activities

  To debug operation <n> activity:

debug ip sla <n>

  To debug activity of the responder:

debug ip sla 0

The Responder Is the Equivalent


of Operation Zero

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 84
Debugging an Operation’s Errors

  To debug errors for operation <n>:

router# debug ip sla error <n>

  To debug errors the responder:

router# debug ip sla error 0

The Responder Is the Equivalent


of Operation Zero

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 85
Debug Sample Output

16:27:45.402: ip sla 1: Starting An Echo Operation - IP sla Probe 1


16:27:45.406: source=10.52.132.69(49175) dest-ip=10.52.132.68(9999)
16:27:45.406: sending control msg:
16:27:45.406: Ver: 1 ID: 144 Len: 52
16:27:45.406: cmd: command: RTT_CMD_JITTER_PORT_ENABLE, ip: 10.52.132.68,
port: 9999, duration: 5200
16:27:45.414: receiving reply
16:27:45.414: Ver: 1 ID: 144 Len: 8
16:27:45.422: sdTime: 2104279296 dsTime: -2017879294
16:27:45.422: responseTime (1): 2
16:27:45.442: sdTime: 2104279296 dsTime: -2017879295
16:27:45.442: jitterOut: 0
16:27:45.442: jitterIn: -1
16:27:45.442: responseTime (2): 1
<. . .>

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 86
Agenda

  SLA – The basics


  Architecture
  Configuration
  Monitoring and Debugging
  Reference and Summary

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 87
References

  Cisco IOS IP SLA Web site on CCO:


http://www.cisco.com/go/ipsla
This page contains links to executive and technical
documents, documentation, and white papers

  Suggested reading:
Cisco IOS IP Service Level Agreements User Guide
Cisco IOS IP Service Level Agreements for Voice over IP

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 88
A Complementary Solution

NetFlow IP SLA
From where? Latency
To Who? Loss
When? Jitter
How Much? Server Delay
Which Apps? (HTTP, DNS,
What ToS? TCP Connect)

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 89
Conclusion

  IP SLA is the integrated Cisco IOS feature to actively


measure and report applications and network
performance
  It offers a broad set of measurement functions
  Several network management applications support it

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 90
Trivia

Q and A
1) Question: How much does IPSLA cost on a
Cisco 3845 router

Answer: It is free on the 3845


(and all other platforms)

BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 91
BRKSEC-2005 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 92

You might also like