You are on page 1of 108

Deploying Tight-SLA

services on an IP
Backbone
Clarence Filsfils – cf@cisco.com

NANOG 25 © 2001, Cisco Systems, Inc. All rights reserved. 1


Objective

• To present design & deployment good


practices to enable tight SLAs to be
offered
– when to use what and how
– validation results
– operational guidelines
– deployment experience
• Focus on the backbone design

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 2
An overview of the Analysis

LLJ:Loss/Latency/Jitter Convergence
DiffServ ISIS Sub-Second
TE FRR Sub-100ms
DSTE

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 3
Further information

• “Engineering a Multiservice IP backbone to


support tight SLAs”, Computer Networks
Special Edition on the New Internet
Architecture
• Full-Day Tutorial
–RIPE41, APRICOT 2002:
www.ibb.net/~filsfils
• Low-Level Design Guides, Validation Results

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 4
Agenda

• Introduction and SLA


• Sub-Second IGP Convergence
• Backbone Diffserv Design
• Conclusion

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 5
Typical Core Per Class SLA
Characteristics

Through- Avail- Loss


Class Delay Jitter
put ability rate
VoIP     
Bus    ?
BE  

Typically more Classes at the Edge


Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 6
One-Way Jitter
• Delay variation generally computed as the
variation of the delay for two consecutive
packets
• Due to variation of
– Propagation delay
– Switching / processing delay
– Queuing / scheduling delay
• Jitters buffers remove variation but
contribute to delay

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 9
Backbone VoIP Jitter Budget
• Typical jitter budget:
– Mouth to ear budget 100ms
– Backbone propagation – 30ms
– Codec delay – ~35ms
– Jitter Budget = 35ms
> 30ms for the access
> 5ms for the core
> 10 hops => 500 µs/hop

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 10
Per flow sequence preservation

• Best-practise IP Design: per-flow loadbalacing!


• Re-ordering Impact on Service Perception
– Long-Lived TCP: degraded goodput
– Real-time video: loss rate += OOS_rate
– VoIP: jitter

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 15
Re-ordering Impact on Service
Server to Multiple Clients
100
Pewrcentage of applicatiosn

80
Linux 15ms
throughput

60 Unix 15 ms
Linux 35ms
40 Unix 35ms

20

0
0.01% 0.10% 1.00% 10.00 100.00
% %
Rate of packets reordered

• [LAOR01]: “Results show that packet reordering, by at least three packet locations,
of only a small percentage of packets in the backbone link can cause a significant
degradation of applications throughput. Long flows are affected the most. Due to the
potential effect, minimizing packet reordering, as well as mitigating its effect
algorithmically, should be considered”.
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 16
Loss of Connectivity / Convergence

• Incentive to reduce the loss of


connectivity (LoC)
• Availability
– 99.999% per day  0.9sec of downtime
• VoIP
– 40msec LoC: glitch
– 1, 2 sec LoC: call drop

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 17
How to specify the target for the
metric
• SLA statistical definitions do matter
– min/avg/max versus percentile
– Measured time interval…
• SLAs definitions today tend to be loose
– averaged over a month
– averaged over many POP-to-POP pairs
(temptation to add short pairs to reduce
average…)
• IP Performance Metrics IETF WG
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 18
Optimizing the IP Infrastructure

• Loss, Latency, Jitter: iif Demand < Offer


– OverProvisioned Backbone
– Differentiated Services
– Capacity Planning
– TE and DS-TE
• Loss of connectivity due to link/node
failure
– IGP Convergence
– MPLS FRR Protection
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 19
Agenda

• Introduction and SLA


• Sub-Second IGP Convergence
• Backbone Diffserv Design
• Conclusion

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 20
Loss of Connectivity

• IGP Backbone Convergence:


– the time it takes for connectivity to be
restored upon link/node failure/addition for
an IP flow starting on an edge access
router and ending on another edge access
router, excluding any variation of BGP
routes.
• For this session, IGP = ISIS

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 21
Historical ISIS Convergence

• 10 to 30 seconds
• Not excellent
• In the past, focus has been more on
stability than on fast convergence
– typical trade-off

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 22
What this presentation will explain
IGP Backbone Convergence

9000
8000
7000
6000
5000
ms

4000
3000
2000
1000
0
default fast isis

• ISIS Convergence in 1 or 2 second is


conservative
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 23
Link-State protocol
overview

NANOG 25 © 2001, Cisco Systems, Inc. All rights reserved. 24


An example network
H
3 G
5
5
3 F
12 12
4 2
C D
E 3
3 7 3
8 S2
B 4 S3
S1
A
3
S0

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 25
The Final SPT rooted at A
G: oif so & s3, Cost 13
5

F: oif so & s3, Cost 8

2
C: oif so & s3, Cost 6 D: oif s3, Cost 3
E: oif so, Cost 11
3
3
8 3
S3
B: oif so, Cost 3

A: oif null, Cost 0


3 S0

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 26
G: oif so & s3, Cost
5 13

F: oif so & s3, Cost 8

2
D: oif s3,
C: oif so & s3, Cost 6
E: oif so, Cost 11 Cost 3
3
G
3
5
8 3
5 B: oif so, Cost 3
S3
F 12 A: oif null, Cost 0
12
4 2 3 S0
C D
E 3
3 7 3
8 G: oif s3, Cost 13
4 S
B
S
2 S
3
5
3 1 A
F: oif s3, Cost 8
S
0
2 D: oif s3,
E: oif s1 & C: oif s3, Cost 6 Cost 3
s3, Cost 12 3
8 3
B: oif s1, Cost 4
S3
A: oif null, Cost 0
4 S1
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 27
The RIB construction
Lo0: 1.1.1.1/32, C=0 Pos1: 2.0.0.1/30, C=2

C: oif so & s3, Cost 6 D: oif s3, Cost 3


3
3
3
S3
B: oif so, Cost 3
A: oif null, Cost 0
3 S0

• ISIS adds the following paths to the RIB:


– 1.1.1.1/32: OIF = S0 or S3 with Metric 6 (6+0)
– 2.0.0.1/30: OIF = S0 or S3 with Metric 8 (6+2)
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 28
LSDB, RIB and FIB

sh isis data Static


ISIS LSDB Routes BGP table

Best

Control RIB sh ip route

Data Plane
FIB & dFIB sh ip cef

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 29
SPF optimisations

NANOG 25 © 2001, Cisco Systems, Inc. All rights reserved. 30


SPF Optimizations
• Most Basic Implementation
– Any change (link, node, leave)
 recompute the whole SPT and the whole RIB
• Optimization 1: decouple SPT and RIB
– If any topology change (node, link)
Called “SPF”
 recompute SPT and the RIB
– If only a leave change (IP prefix)
Called “PRC”  keep the SPT, just update the RIB for the
nodes whose leaves have changed

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 31
PRC
G Cost: 13, NH: D
Int lo 0: 65.1.1.1/32

F Cost: 8, NH: D, B

C Cost: 6, NH: D, B D Cost: 3, NH: D


E Cost: 11, NH: B

S2
B Cost: 3, NH: B S3
S1

A Cost: 0, NH: --
S0

• PRC here consists in just adding 65.1.1.1/32


in the RIB. The SPT is not affected.
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 32
Incremental-SPF

• Optimization 2
• When the topology has changed, instead
of building the whole SPT from scratch
just fix the part of the SPT that is affected
• Only the leaves of the nodes re-analyzed
during that process are updated in the RIB

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 33
Incremental-SPF
G Cost: 13, NH: D

F Cost: 8, NH: D, B

C Cost: 6, NH: D, B D Cost: 3, NH: D


E Cost: 11, NH: B

S2
B Cost: 3, NH: B S3
S1
C-G link is down.
A Cost: 0, NH: --
S0 C-G link was not used in SPT
anyway, therefore there is no
need to run SPF.

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 34
Incremental-SPF
G Cost: 13, NH: D
H

F Cost: 8, NH: D, B

C Cost: 6, NH: D, B D Cost: 3, NH: D


E Cost: 11, NH: B

F reports a new neighbor.


The SPT need only to be
S2
extended behind F. There is
B Cost: 3, NH: B S3
no need for router A to S1
recompute the whole SPT
Router A will compute SPF A Cost: 0, NH: --
from node F S0

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 35
Incremental-SPF

• More information is kept in the SPT


–Parents list
–Neighbors list
• Based on the changed information, the
SPT is “modified” in order to reflect the
changes

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 36
Incremental-SPF

• The further away from the root the change,


the higher the gain

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 37
SPF, PRC, I-SPF: summary

• Only a leaf change


– PRC
• Graph impacted
– normal-SPF: recompute the full SPT and
hence reinserts all the ISIS routes in the
RIB
– I-SPF: only recomputes the part of the
SPT that is affected. Only the leaves from
that part are affected.

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 38
Topology and Leaf
Optimizations

NANOG 25 © 2001, Cisco Systems, Inc. All rights reserved. 39


Parallel point-to-point adjacencies
C D
E 3
3 7 3
8 S2
B 4 S3
LSP B S1 LSP A
IS: 3 A A IS: 3 B
IS: 4 A
3 IS: 4 B
S0
IS: 3 C IS: 7 C
IS: 8 E IS: 3 D

• Only best parallel adjacency is reported

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 40
P2P mode for back-to-back GE
Rtr-A Rtr-B

Rtr-A Rtr-B Rtr-A Rtr-B

interface fastethernet1/0
Pseudonode isis network point-to-point

• No DIS election
• No CSNP transmission
• No Pseudo-node and extra link
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 41
Speeding up route installation

• Limit the # of leaves in the IGP


– only the BGP speakers are needed ( )
– rest: I-BGP

router isis
advertise passive-only
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 42
SPF, PRC and LSP-gen
Exponential BackOff
Timers

NANOG 25 © 2001, Cisco Systems, Inc. All rights reserved. 43


Backoff timer algorithm
• IS-IS throttles it main events
– SPF computation
– PRC computation
– LSP generation
• Throttling slows down convergence
• Not throttling can cause melt-downs
• The scope is to react fast to the first
events but, under constant churn, slow
down to avoid to collapse
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 44
Backoff timer algorithm

spf-interval <Max> [<Init> <Inc>]

• Maximum interval: Maximum amount of time the


router will wait between consecutives executions
• Initial delay: Time the router will wait before
starting execution
• Incremental interval: Time the router will wait
between consecutive execution. This timer is
variable and will increase until it reaches
Maximum-interval

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 45
spf-interval 10 100 1000
E1 E2 E3 E4 E5 E6 E7
Event1

SPF SPF SPF


100ms 1000ms 2000ms 4000ms

• Then 8000ms
• Then maxed at 10sec
• 20s without Trigger is required before resetting
the SPF timer to 100ms

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 46
Default Values
• Maximum-interval:
– SPF: 10 seconds
– PRC: 5 seconds
• Incremental-interval:
– LSP-Generation: 5
seconds – SPF: 5.5 seconds
– PRC: 5 seconds
• Initial-wait:
– LSP-Generation: 5
– SPF: 5.5 seconds seconds
– PRC: 2 seconds
– LSP-Generation: 50
milliseconds

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 47
Two-Way Connectivity Check

E B

LSP
LSP
F

• For propagating Bad News,


1! LSP is enough
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 48
Timers for Fast Convergence
router isis
spf-interval 1 1 50
• Init Wait: 1ms prc-interval 1 1 50

– 5.5 sec faster than default reaction!


– Optimized for the going down mode
• Exp Increment ~ S ms
• Max Wait ~ n * S ms
– CPU utilization < 1/n
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 49
Timer for Fast Convergence
router isis
lsp-gen-interval 5 1 50

• The timers are designed to optimize the


propagation of the information to other
nodes.
– Init-Wait = 1ms, 49ms faster than
default
– Exp-Inc = S, eg. 50ms

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 50
LSP Pacing and Flooding

NANOG 25 © 2001, Cisco Systems, Inc. All rights reserved. 53


LSP Pacing and Flooding
Int pos x/x
isis lsp-interval <>

• Pacing:
– Default: 33msecs inter-LSP gap
– backoff protection
– full database download
– suggest to keep the default
• Flooding
– flood/SPF trade-off
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 54
Link Protocol Properties

NANOG 25 © 2001, Cisco Systems, Inc. All rights reserved. 55


Link Protocol Properties

• Link Failure Detection


– the faster and more reliable, the better
• Dampening flapping links
– Fast signalling of a Down information
– Stable signalling of an UP information
– Freeze a flapping link in Down status

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 56
POS – Detection of a link failure
• Pos delay trigger line:
– hold time before reacting to a line alarm
– default is: immediate reaction
• Pos delay trigger path:
– hold time before reacting to a path alarm
– default is: no reaction
• Carrier-delay
– hold time between the end of the pos delay
holdtime and the bring down of the IOS interface
– default: 2000 msec

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 57
POS – Detection of a link failure
int pos 1/0
carrier-delay msec 8

• Redundant for POS interfaces

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 58
POS – Detection of a link failure
int pos 1/0
carrier-delay msec 8
pos delay triggers line 60
pos delay triggers path 60

R1-ADM--PROTECTED_SONET_net--ADM-R2

• Should delay a little to allow for SONET


protection. Suggestion: 60msec
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 59
POS – Detection of a link failure
int pos 1/0
carrier-delay msec 8
pos delay triggers line 0
pos delay triggers path 0

R1-ADM--UNprotected_SONET_net--ADM-R2

• Should react as fast possible


– line default ok
– path default not ok
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 60
POS – Detection of a link failure
int pos 1/0
carrier-delay msec 8
pos delay triggers line 60

R1-DWDM--PROTECTED_DWDM_net--DWDM-R2

• Should delay for DWDM protection


– Suggestion: 60msec
– Alarm will be section or line
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 61
POS – Detection of a link failure
int pos 1/0
carrier-delay msec 8
pos delay triggers line 0

R1-DWDM--UNPROTECTED_DWDM_net--DWDM-R2

• Should react asap


– line: default ok
– path: not needed: default ok
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 62
POS – Bringing a down link back up

• Upon alarm clearance, POS Driver will wait


10seconds + <Carrier-Delay> before
turning the interface back up, hence
before triggering ISIS convergence

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 63
POS – Best for Convergence

• Very fast Link failure detection


– no need to tune the ISIS hello/holdtime
• Native anti-flap property of POS
– down info is signalled very fast
– up info is confirmed for 10s before
relaying to interface

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 64
Other types of Links

• Link Failure Detection


– If the native mode is too slow or if the link
has no failure detection capability
– ISIS Hello/Holdtime tuning
• Interface Dampening
– New feature to provide same Dampening
capability as BGP to the generic Interface
(applies to all types of interfaces)

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 65
Fast Hello’s
int serial0
isis hello-interval minimal
isis hello-multiplier 4

• Fast hello’s allow a dead timer of 1


second
• POS much faster/reliable
• Only useful when layer1/2 can’t help!

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 66
Operating this Design

NANOG 25 © 2001, Cisco Systems, Inc. All rights reserved. 67


ISIS Fast Convergence
Design

NANOG 25 © 2001, Cisco Systems, Inc. All rights reserved. 71


Design Tips
int pos 1/0
• POS as link type carrier-delay msec 8
pos delay trigger …

–Do not tune ISIS hello’s and LSP-interval

• Design to minimize ISIS nodes,


links, prefixes
• Optimization: PRC, I-SPF, Flooding,
Parallel adjacencies, p2p GE
• SPF, PRC, LSP-Gen timers router isis
spf-interval 1 1 50
prc-interval 1 1 50
lsp-gen-interval 5 1 50
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 72
Test Results

NANOG 25 © 2001, Cisco Systems, Inc. All rights reserved. 73


Test Scenari

• 12.0(19)S
• Carrier-delay configured to 8ms
• SPF, PRC, LSP-Gen Timers
– Default Timers
– Fast ISIS Configuration

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 74
ISIS: 1200
Nodes, 4000
Leaves
BGP: 144000
prefixes 1200 Nodes
4000 prefixes

Agilent:
• A-B & B->A
• 10000 pps
• accuracy:
0.1ms
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 75
A
ISIS: 1200
Nodes, 4000
Leaves
BGP: 144000
prefixes 10000pps 10000pps

Agilent:
• A-B & B->A
• 10000 pps
• accuracy:
0.1ms
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. B 76
ISIS: 1200
Nodes, 4000
Leaves
BGP: 144000
prefixes
ais
Agilent:
• A-B & B->A
• 10000 pps
• accuracy:
0.1ms
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 77
ISIS: 1200
Nodes, 4000
Leaves
BGP: 144000
prefixes
Ais
Agilent: cleared

• A-B & B->A


• 10000 pps
• accuracy:
0.1ms
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 78
SPF Duration

1200 SPF duration


nodes
140
4000
120
prefixes
100 1
80 2
60 3
40 8
20
0
10-1 e5-1 e7-1 e3-1 f11-1

• SPF duration: ~ 100ms

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 79
Down
Link goes down
ms
6000.0

5000.0

4000.0
ab fast
ba fast
3000.0
ab def
ba def
2000.0

1000.0

0.0
1 2 3 4 5 6 7 8 9 10

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 80
Carrier-Delay
ISIS A->B 12.0(18)ST down event - AVG

3000

2500

2000

0ms

• Graph from Iain


8ms
msec

12ms
1500
16ms
50ms
2000ms
1000

500

0
1
carrier delay ms

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 81
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 82
Convergence

500

450
• 500 ISIS n
400

350

300
• 1000 ISIS p
250

200
• 80000 BGP p
• Accuracy:
150

100

50 0.1 ms
0

• 10 iterations
ISIS-NH A->B ISIS-LNE A->B BGP1 A->B BGP2 A->B BGP3 A->B

Average [ms] Std-dev [ms]

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 83
Conclusion

NANOG 25 © 2001, Cisco Systems, Inc. All rights reserved. 84


Conclusion

• IGP convergence needs to be optimized


for Tight-SLA Services
• New development speed up convergence
without stability compromise
• Test results indicate that sub-second
convergence is realistic
• For sub-100ms Convergence, local action
based on precomputed tables might be
required

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 85
Agenda

• Introduction and SLA


• Sub-Second IGP Convergence
• Backbone Diffserv Design
• Conclusion

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 86
OverProvisioned
Backbone
DiffServ with a single class!

NANOG 25 © 2001, Cisco Systems, Inc. All rights reserved. 87


The Key is OverProvisioning
Offer must be higher than Demand
• The service that traffic receives is
dependent upon the ratio of traffic load to
available capacity
• More Bandwidth (offer) than traffic
(demand) means
– Low loss
– Low Latency
– Low Jitter
• Refs: [ROBERTS], [CHARNY], [BONALD]

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 88
Over-Provisioned Backbone
• A simple rule of design:

95-Percentile (5-min average Load) <= 50%


Link

which means

OverProvisioning (OP) > 2

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 89
Over-provisioning
(Source: Stephen Casner, Packet Design, NANOG 22)

Jitter Measurement Summary


for the Week
69 million packets transmitted
Zero packets lost
100% jitter < 700s

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 90
Drawback

• Risk related to provisioning failure


• Fate Sharing!
– No isolation between VPN, VoIP, Internet
• Expensive
– design for the aggregate!

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 91
Provisioning failure

• Capacity planning failures


– Small overprovisioning ratio: 2 vs 16
• Unexpected traffic demands
• Network failure situations
• Bandwidth unavailability
• Internet DoS Attack

FATE SHARING: Internet affects VoIP


Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 92
“Not every week is like this”
(Source: Stephen Casner, Packet Design, NANOG 22)

99.99%

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 93
Recommendation: use DiffServ!

• Higher Availability of SLA


– Higher overprovisioning ratio (4 and
more)
– Service Isolation
• Cheaper
– Overprovisioning per Class!
• Mature Technology

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 94
Service Isolation

DSCP ECN

• DiffServ Per-Hop Behavior


– Expedited Forwarding
>Low-latency/jitter scheduler (often a PQ)
– Assured Forwarding
>Bandwidth allocation and Multi-level
Congestion avoidance (RED)

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 95
Backbone Diffserv Design

• 2 or 3 Aggregate classes
• Edge DSCP marking policy to indicate class

Class DSCP IP Prec Binary PHB


VoIP 40 5 101 000 EF
Bus 32 4 100 000 AF1
Network 48 6 110 000 AF1
BE 0 0 000 000 Default

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 96
Backbone Diffserv Design

• VoIP
– EF PHB (a strict PQ)
– OP(V) = 4
• Business
– AF1 PHB: 90% of the remaining BW
– OP(V+B) = 2.25 = 1/0.45
• Internet
– AF2 PHB: 10% of the remaining BW
– OP(Aggr) = 1.25 = 1/0.8
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 97
Aggr Over-provisioning is
Expensive
• 1! DS Class:
D
3C/1C – Agg <= 0.5
8
7
• 3 DS Classes
6
5 – V <= 0.25
4
3 – V+B <= 0.45
2
1
0
– Agg <= 0.8
0.2
0.02
0.05
0.08
0.11
0.14
0.17

0.23
0.26
0.29
0.32
0.35
0.38
0.41
0.44
V+B

Ex: V = 0.1, B =0.25


1 DS: D = 0.15
3 DS: D = 0.45
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 98
WRED Tuning

Link speed P Min. Th Max. Th.


OC3/STM-1 1292 194 1218
OC12/STM-4 5184 778 2826
OC48/STM-16 20000 3000 19384

• Based on simulations for at least 85%


utilisation with a mean queue size below
20msec

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 99
Typical Backbone Diffserv Design
class-map match-any VOIP
match ip precedence 5 PE1 P1 PE2
P2
class-map match-any BUS
match ip precedence 4
match ip precedence 6
!
policy-map OC3_POLICY
class VOIP
priority P3 P4 PE4
PE3
class BUS
bandwidth percent remaining 90
random-detect prec 4 97 609 1
random-detect prec 6 97 609 1 Static!
class class-default
No inbound DiffServ Policy!
bandwidth percent remaining 10
random-detect prec 0 97 609 1 No marking, policing,
! shaping in the core!
interface POS0/1
ip address 10.0.1.1 255.255.255.252 RED as congestion
service-policy output OC3_POLICY avoidance for each Data
(TCP) Class
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 105
Provisioning is simple

• Same as ISIS, OSPF


• Configuration is done once and then it
remains static

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 106
Capacity Planning
• Aggregate Based
– DiffServ Isolation – risk hedging
• Per-Class Based
– OP per link/class
– Traffic Matrix per Class
– Better network utilization
• Significant edge qos deployment over last
24 months contribute to better NMS
support for QoS
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 107
Mature Technology

• EF: jitter due to non-EF


• AF: accuracy of BW allocation
• AF: latency as a function of AF load
• Even the rare cases are dealt with

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 108
EF: jitter charecteristic
voice packet latency on eng2 OC48

135000
130000
51v(200), 45bu,
125000
150be
120000
30v(200), 45bu,
115000 150be
ns

110000 15v(200), 45bu,


150be
105000
9v(200), 45bu,
100000
150be
95000
90000
1
14

27
40
53

66
79
92

percentile

E2b-OC48: Five times better than 500µs budget


Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 109
EF: jitter characteristic
100
90
80
70
v=30%, bus=45%,
delay (µs)

60
be=150%
50
v=75%, bus=45%,
40 be=150%
30
20
10
0
1

E4-QOC48: 7 times better than 500us


Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 110
AF: Bandwidth Allocation Accuracy

Accuracy of BUS bandwidth allocation

0.15
Measured Accuracy of BUS BW

0.1

0.05
[% of linerate]

-0.05

-0.1

-0.15
Expected BUS BW [% of linerate]

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 111
AF: Latency = f(load)
Latency in business class on OC48 with IMIX

1000000
Max latency (us)

100000

10000

1000

100
70 100 130 160 190 220 250
business load ratio

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 112
Optimised for even rare/corner-
cases
Input Output
Ports Rx (rx-cos) side of i/p line card Tx side (tx-cos) of o/p line card Ports

E2: 2048 tofab VOQs E2: 128 fromfab Qs

Crossbar Switch Fabric


16x16x8 16x8

CEF

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 113
Capacity Planning and
Monitoring

NANOG 25 © 2001, Cisco Systems, Inc. All rights reserved. 117


Capacity Planning and Monitoring

• A number of tools exist for capacity


planning:
–Per link statistics
–Core traffic matrices
–Active SLA monitoring

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 119
Link statistics

• packets and bytes through the class


• packets random-dropped
• packets forced-dropped
• no-buffer drops
• ignores

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 120
Core Traffic Matrix

• TMS: FIB accounting per non-recursive entry


• NetFlow v9 aggregated per BGP next-hop
• TE tunnel statistics (full-mesh req.)
• Reverse inference (research)

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 121
IPPM Infrastructure
POP2

POP1 POP3

POP4 SLA probes

PE P

PE Ie. SAA
PE
Shadow Router
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 123
MPLS-based
Technologies
TE, DS-TE, FRR

NANOG 25 © 2001, Cisco Systems, Inc. All rights reserved. 124


TE and SLA’s

• TE allows for the routing based on


constraints other than shortest-path
– bandwidth availability
– propagation latency
• DS-TE allows this for the aggregate and at
least one additional class-type

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 125
When TE is justified

• Drivers for MPLS TE deployment:


– Network asymmetry
– Unexpected demand
– Long bandwidth lead-times
• Drivers for DS-TE:
– above for aggregate; AND
– EF Load > 25% due to unoptimized
classic routing

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 126
MPLS FRR

• Link/Node Local Protection


– Pre-established and pre-computed
– Requires MPLS TE deployment
• When sub-second convergence is not
enough, but 50ms is required

Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 127
Tight-SLA IP Backbone
Conclusion

Clarence Filsfils - cfilsfil@cisco.com

NANOG 25 © 2001, Cisco Systems, Inc. All rights reserved. 128


An overview of the Analysis

LLJ:Loss/Latency/Jitter Convergence
DiffServ: likely a Must
- EF(jitter) < 50us ISIS Sub-Second: Likely a Must
- AF: 99.95% accuracy,
160us latency
More Assurance, Cheaper
MPLS FRR: for <100ms
TE: if asymetric topology,
unexpected growth, long lead
times

DS-TE: if TE and EF
utilization per link risks to
be –too
Clarence Filsfils high
Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved. 129

You might also like