Professional Documents
Culture Documents
services on an IP
Backbone
Clarence Filsfils – cf@cisco.com
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 2
An overview of the Analysis
LLJ:Loss/Latency/Jitter Convergence
DiffServ ISIS Sub-Second
TE FRR Sub-100ms
DSTE
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 3
Further information
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 4
Agenda
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 5
Typical Core Per Class SLA
Characteristics
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 9
Backbone VoIP Jitter Budget
• Typical jitter budget:
– Mouth to ear budget 100ms
– Backbone propagation – 30ms
– Codec delay – ~35ms
– Jitter Budget = 35ms
> 30ms for the access
> 5ms for the core
> 10 hops => 500 µs/hop
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 10
Per flow sequence preservation
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 15
Re-ordering Impact on Service
Server to Multiple Clients
100
Pewrcentage of applicatiosn
80
Linux 15ms
throughput
60 Unix 15 ms
Linux 35ms
40 Unix 35ms
20
0
0.01% 0.10% 1.00% 10.00 100.00
% %
Rate of packets reordered
• [LAOR01]: “Results show that packet reordering, by at least three packet locations,
of only a small percentage of packets in the backbone link can cause a significant
degradation of applications throughput. Long flows are affected the most. Due to the
potential effect, minimizing packet reordering, as well as mitigating its effect
algorithmically, should be considered”.
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 16
Loss of Connectivity / Convergence
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 17
How to specify the target for the
metric
• SLA statistical definitions do matter
– min/avg/max versus percentile
– Measured time interval…
• SLAs definitions today tend to be loose
– averaged over a month
– averaged over many POP-to-POP pairs
(temptation to add short pairs to reduce
average…)
• IP Performance Metrics IETF WG
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 18
Optimizing the IP Infrastructure
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 20
Loss of Connectivity
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 21
Historical ISIS Convergence
• 10 to 30 seconds
• Not excellent
• In the past, focus has been more on
stability than on fast convergence
– typical trade-off
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 22
What this presentation will explain
IGP Backbone Convergence
9000
8000
7000
6000
5000
ms
4000
3000
2000
1000
0
default fast isis
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 25
The Final SPT rooted at A
G: oif so & s3, Cost 13
5
2
C: oif so & s3, Cost 6 D: oif s3, Cost 3
E: oif so, Cost 11
3
3
8 3
S3
B: oif so, Cost 3
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 26
G: oif so & s3, Cost
5 13
2
D: oif s3,
C: oif so & s3, Cost 6
E: oif so, Cost 11 Cost 3
3
G
3
5
8 3
5 B: oif so, Cost 3
S3
F 12 A: oif null, Cost 0
12
4 2 3 S0
C D
E 3
3 7 3
8 G: oif s3, Cost 13
4 S
B
S
2 S
3
5
3 1 A
F: oif s3, Cost 8
S
0
2 D: oif s3,
E: oif s1 & C: oif s3, Cost 6 Cost 3
s3, Cost 12 3
8 3
B: oif s1, Cost 4
S3
A: oif null, Cost 0
4 S1
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 27
The RIB construction
Lo0: 1.1.1.1/32, C=0 Pos1: 2.0.0.1/30, C=2
Best
Data Plane
FIB & dFIB sh ip cef
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 29
SPF optimisations
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 31
PRC
G Cost: 13, NH: D
Int lo 0: 65.1.1.1/32
F Cost: 8, NH: D, B
S2
B Cost: 3, NH: B S3
S1
A Cost: 0, NH: --
S0
• Optimization 2
• When the topology has changed, instead
of building the whole SPT from scratch
just fix the part of the SPT that is affected
• Only the leaves of the nodes re-analyzed
during that process are updated in the RIB
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 33
Incremental-SPF
G Cost: 13, NH: D
F Cost: 8, NH: D, B
S2
B Cost: 3, NH: B S3
S1
C-G link is down.
A Cost: 0, NH: --
S0 C-G link was not used in SPT
anyway, therefore there is no
need to run SPF.
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 34
Incremental-SPF
G Cost: 13, NH: D
H
F Cost: 8, NH: D, B
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 35
Incremental-SPF
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 36
Incremental-SPF
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 37
SPF, PRC, I-SPF: summary
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 38
Topology and Leaf
Optimizations
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 40
P2P mode for back-to-back GE
Rtr-A Rtr-B
interface fastethernet1/0
Pseudonode isis network point-to-point
• No DIS election
• No CSNP transmission
• No Pseudo-node and extra link
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 41
Speeding up route installation
router isis
advertise passive-only
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 42
SPF, PRC and LSP-gen
Exponential BackOff
Timers
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 45
spf-interval 10 100 1000
E1 E2 E3 E4 E5 E6 E7
Event1
• Then 8000ms
• Then maxed at 10sec
• 20s without Trigger is required before resetting
the SPF timer to 100ms
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 46
Default Values
• Maximum-interval:
– SPF: 10 seconds
– PRC: 5 seconds
• Incremental-interval:
– LSP-Generation: 5
seconds – SPF: 5.5 seconds
– PRC: 5 seconds
• Initial-wait:
– LSP-Generation: 5
– SPF: 5.5 seconds seconds
– PRC: 2 seconds
– LSP-Generation: 50
milliseconds
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 47
Two-Way Connectivity Check
E B
LSP
LSP
F
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 50
LSP Pacing and Flooding
• Pacing:
– Default: 33msecs inter-LSP gap
– backoff protection
– full database download
– suggest to keep the default
• Flooding
– flood/SPF trade-off
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 54
Link Protocol Properties
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 56
POS – Detection of a link failure
• Pos delay trigger line:
– hold time before reacting to a line alarm
– default is: immediate reaction
• Pos delay trigger path:
– hold time before reacting to a path alarm
– default is: no reaction
• Carrier-delay
– hold time between the end of the pos delay
holdtime and the bring down of the IOS interface
– default: 2000 msec
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 57
POS – Detection of a link failure
int pos 1/0
carrier-delay msec 8
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 58
POS – Detection of a link failure
int pos 1/0
carrier-delay msec 8
pos delay triggers line 60
pos delay triggers path 60
R1-ADM--PROTECTED_SONET_net--ADM-R2
R1-ADM--UNprotected_SONET_net--ADM-R2
R1-DWDM--PROTECTED_DWDM_net--DWDM-R2
R1-DWDM--UNPROTECTED_DWDM_net--DWDM-R2
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 63
POS – Best for Convergence
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 64
Other types of Links
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 65
Fast Hello’s
int serial0
isis hello-interval minimal
isis hello-multiplier 4
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 66
Operating this Design
• 12.0(19)S
• Carrier-delay configured to 8ms
• SPF, PRC, LSP-Gen Timers
– Default Timers
– Fast ISIS Configuration
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 74
ISIS: 1200
Nodes, 4000
Leaves
BGP: 144000
prefixes 1200 Nodes
4000 prefixes
Agilent:
• A-B & B->A
• 10000 pps
• accuracy:
0.1ms
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 75
A
ISIS: 1200
Nodes, 4000
Leaves
BGP: 144000
prefixes 10000pps 10000pps
Agilent:
• A-B & B->A
• 10000 pps
• accuracy:
0.1ms
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. B 76
ISIS: 1200
Nodes, 4000
Leaves
BGP: 144000
prefixes
ais
Agilent:
• A-B & B->A
• 10000 pps
• accuracy:
0.1ms
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 77
ISIS: 1200
Nodes, 4000
Leaves
BGP: 144000
prefixes
Ais
Agilent: cleared
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 79
Down
Link goes down
ms
6000.0
5000.0
4000.0
ab fast
ba fast
3000.0
ab def
ba def
2000.0
1000.0
0.0
1 2 3 4 5 6 7 8 9 10
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 80
Carrier-Delay
ISIS A->B 12.0(18)ST down event - AVG
3000
2500
2000
0ms
12ms
1500
16ms
50ms
2000ms
1000
500
0
1
carrier delay ms
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 81
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 82
Convergence
500
450
• 500 ISIS n
400
350
300
• 1000 ISIS p
250
200
• 80000 BGP p
• Accuracy:
150
100
50 0.1 ms
0
• 10 iterations
ISIS-NH A->B ISIS-LNE A->B BGP1 A->B BGP2 A->B BGP3 A->B
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 83
Conclusion
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 85
Agenda
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 86
OverProvisioned
Backbone
DiffServ with a single class!
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 88
Over-Provisioned Backbone
• A simple rule of design:
which means
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 89
Over-provisioning
(Source: Stephen Casner, Packet Design, NANOG 22)
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 90
Drawback
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 91
Provisioning failure
99.99%
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 93
Recommendation: use DiffServ!
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 94
Service Isolation
DSCP ECN
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 95
Backbone Diffserv Design
• 2 or 3 Aggregate classes
• Edge DSCP marking policy to indicate class
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 96
Backbone Diffserv Design
• VoIP
– EF PHB (a strict PQ)
– OP(V) = 4
• Business
– AF1 PHB: 90% of the remaining BW
– OP(V+B) = 2.25 = 1/0.45
• Internet
– AF2 PHB: 10% of the remaining BW
– OP(Aggr) = 1.25 = 1/0.8
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 97
Aggr Over-provisioning is
Expensive
• 1! DS Class:
D
3C/1C – Agg <= 0.5
8
7
• 3 DS Classes
6
5 – V <= 0.25
4
3 – V+B <= 0.45
2
1
0
– Agg <= 0.8
0.2
0.02
0.05
0.08
0.11
0.14
0.17
0.23
0.26
0.29
0.32
0.35
0.38
0.41
0.44
V+B
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 99
Typical Backbone Diffserv Design
class-map match-any VOIP
match ip precedence 5 PE1 P1 PE2
P2
class-map match-any BUS
match ip precedence 4
match ip precedence 6
!
policy-map OC3_POLICY
class VOIP
priority P3 P4 PE4
PE3
class BUS
bandwidth percent remaining 90
random-detect prec 4 97 609 1
random-detect prec 6 97 609 1 Static!
class class-default
No inbound DiffServ Policy!
bandwidth percent remaining 10
random-detect prec 0 97 609 1 No marking, policing,
! shaping in the core!
interface POS0/1
ip address 10.0.1.1 255.255.255.252 RED as congestion
service-policy output OC3_POLICY avoidance for each Data
(TCP) Class
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 105
Provisioning is simple
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 106
Capacity Planning
• Aggregate Based
– DiffServ Isolation – risk hedging
• Per-Class Based
– OP per link/class
– Traffic Matrix per Class
– Better network utilization
• Significant edge qos deployment over last
24 months contribute to better NMS
support for QoS
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 107
Mature Technology
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 108
EF: jitter charecteristic
voice packet latency on eng2 OC48
135000
130000
51v(200), 45bu,
125000
150be
120000
30v(200), 45bu,
115000 150be
ns
27
40
53
66
79
92
percentile
60
be=150%
50
v=75%, bus=45%,
40 be=150%
30
20
10
0
1
0.15
Measured Accuracy of BUS BW
0.1
0.05
[% of linerate]
-0.05
-0.1
-0.15
Expected BUS BW [% of linerate]
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 111
AF: Latency = f(load)
Latency in business class on OC48 with IMIX
1000000
Max latency (us)
100000
10000
1000
100
70 100 130 160 190 220 250
business load ratio
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 112
Optimised for even rare/corner-
cases
Input Output
Ports Rx (rx-cos) side of i/p line card Tx side (tx-cos) of o/p line card Ports
CEF
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 113
Capacity Planning and
Monitoring
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 119
Link statistics
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 120
Core Traffic Matrix
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 121
IPPM Infrastructure
POP2
POP1 POP3
PE P
PE Ie. SAA
PE
Shadow Router
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 123
MPLS-based
Technologies
TE, DS-TE, FRR
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 125
When TE is justified
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 126
MPLS FRR
Clarence Filsfils – Nanog 25 © 2002, Cisco Systems, Inc. All rights reserved. 127
Tight-SLA IP Backbone
Conclusion
LLJ:Loss/Latency/Jitter Convergence
DiffServ: likely a Must
- EF(jitter) < 50us ISIS Sub-Second: Likely a Must
- AF: 99.95% accuracy,
160us latency
More Assurance, Cheaper
MPLS FRR: for <100ms
TE: if asymetric topology,
unexpected growth, long lead
times
DS-TE: if TE and EF
utilization per link risks to
be –too
Clarence Filsfils high
Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved. 129