Troubleshooting MPLS

Vinit Jain
CCIE# 22854
Twitter: @vinugenie
ATE-CL032
Agenda
Fundamentals
Troubleshooting MPLS Control Plane
Troubleshooting MPLS Forwarding Plane

ATE-CL032
Fundamentals

ATE-CL032
 MPLS has two major components:
Control plane: Exchanges Layer 3 routing information and labels
Forwarding plane: Forwards packets based on labels

Control plane contains complex mechanisms to exchange routing

information, such as OSPF, EIGRP, IS-IS, and BGP, and to
exchange labels, such as TDP, LDP, BGP, and RSVP.
Forwarding plane forwards packets based on CEF

ATE-CL032
 RIB is the Routing Information Base that is analogous to the
IP routing table.
FIB aka CEF is Forwarding information base that is derived
from the IP routing table.
LIB is Label Information Base that contains all the label
bindings learned via LDP
LFIB is Label Forwarding Information Base that is derived
from FIB entries and corresponding LIB entries.

ATE-CL032
 FEC ( Forwarding Equivalence Class)
Group of IP packets forwarded in the same manner (e.g. over same forwarding path)

A FEC can represent a: Destination IP prefix, VPN ID, ATM VC,

VLAN ID, Traffic Engineering tunnel, Class of Service.

ATE-CL032
 Control plane
Routing updates from
routing protocols database
other routers
IP routing table (RIB)
Label bindings
Label Information Base (LIB) learned via LDP from
Forwarding plane other routers

Incoming IP Packet
IP forwarding table (FIB)

Incoming MPLS Outgoing MPLS/IP

Packet
Label forwarding table (LFIB)
Packet

ATE-CL032
 IGP domain with a label IGP domain with a label
distribution protocol distribution protocol

LSP follows IGP shortest path LSP diverges from IGP shortest path

LSPs are derived from IGP routing information

LSPs may diverge from IGP shortest path
LSP tunnels (explicit routing) with TE

LSPs are unidirectional

ATE-CL032
 MPLS uses a 32-bit label field that is inserted between Layer 2
and Layer 3 headers(frame-mode).

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

Label COS S TTL

Label = 20 bits
COS/EXP = Class of Service, 3 bits
S = Bottom of Stack, 1 bit
TTL = Time to Live (Loop detection)

ATE-CL032
 Ethertype 0x0800 refers to IP
Ethertype 0x8847 refers to MPLS
Based on the Ethertype, the packet is handed over to the
appropriate processing engine in the router

ATE-CL032
 An MPLS packet may have more than one label
Frame Mode can handle a stack of two or more labels, depending
on the platform
Bottom most label has the S-bit set to 1
LSRs label switch packets based ONLY on the label at the top of
the stack
ATE-CL032
 The following scenarios may produce more than one
label:

MPLS VPNs (two labels: The top label points to the egress router and
the second label identifies the VPN.)
MPLS TE with Fast Reroute (two or more labels: The top label is for
the backup tunnel and the second label points to the primary tunnel
destination.)
MPLS VPNs combined with MPLS TE (three or more labels.)

ATE-CL032
Troubleshooting MPLS
Control Plane

ATE-CL032
 LDP_ID should be hardcoded via
mpls ldp router-ID <interface>

The above wont do any good unless

<interface> is UP when LDP gets started
Existing LDP_ID (usually an interface) is shut

Following avoids both shortcomings

mpls ldp router-ID <interface> force
ATE-CL032
 Use
the same Loopback0 as the router-ID for
LDP, IGP, BGP, etc.
Assignan IP address to the Loopback0 from
the separate IP address subnet (or space)
Avoidthe IGP summarization of prefixes that
correspond to the router-ids
ATE-CL032
 sh mpls ldp discovery [detail]
Must show xmit/recv on LDP enabled interface

PE1#sh mpls ldp discovery

Local LDP Identifier:
10.13.1.61:0 Local LDP_ID
Discovery Sources: Xmited and Recvd
Eth0/0 is Interfaces:
configured Ethernet0/0 (ldp): xmit/recv
Hellos on that interface
with LDP LDP Id: 10.13.1.101:0 Discovered Neighbors
Ethernet1/0 (ldp): xmit/recv
LDP_ID
LDP Id: 10.13.1.101:0

debug mpls ldp transport connections

Should give information regarding whether the HELLOS are advertised/received
ATE-CL032
 sh mpls interface [detail]
Lists whether MPLS is enabled and the application that enabled MPLS
on the interface
PE2#sh mpls interface Serial2/0
Interface IP Tunnel Operational
Serial2/0 Yes (ldp) No Yes PE2 P1

PE2#sh mpls interface ser2/0 detail MPLS Enabled !

Interface Serial2/0: interface Serial2/0
IP labeling enabled (ldp) LDP Enabled description To P1 ser2/0
LSP Tunnel labeling not enabled ip address 10.13.2.6/30
BGP tagging not enabled mpls label protocol ldp
Tagging operational mpls ip
Fast Switching Vectors: mpls mtu 1508
IP to MPLS Fast Switching Vector
!
MPLS Turbo Vector
MTU = 1508 MPLS MTU
ATE-CL032
PE1#sh mpls ldp neighbor
Peer LDP Ident: 10.13.1.101:0; Local LDP Ident 10.13.1.61:0
TCP connection: 10.13.1.101.11031 - 10.13.1.61.646
State: Oper; Msgs sent/rcvd: 58/60; Downstream
Up time: 00:39:27
LDP discovery sources:
Ethernet0/0, Src IP addr: 10.13.1.5
Ethernet1/0, Src IP addr: 10.13.1.9
Addresses bound to peer LDP Ident:
10.13.1.9 10.13.1.5 10.13.2.5 10.13.1.101

PE1#sh tcp brief| i 646

43ABB020 10.13.1.101.11031 10.13.1.61.646 ESTAB
PE1#

ATE-CL032
 sh mpls ldp neighbor [neighbor]
Shows LDP neighbor and relevant info

sh mpls ldp neighbor [interface]

LDP neighbors discovered over this interface

debug mpls ldp session io|state

Useful when the session doesnt come up

debug mpls ldp messages sent|receive

Shows all the LDP messages sent or received
ATE-CL032
 Label binding => prefix + Label
Label bindings are stored in the LIB
LIB => Label Information Base
Label exchange

PE1 P1
10.13.1.61/32 10.13.1.101/32

ATE-CL032
 sh mpls ip binding <prefix> <mask> det
Lists all the LDP neighbors and their labels for that prefix

sh mpls ip binding advertisement-acls

Lists LDP filter, if there is any, on the first line. Prefixes followed by
Advert acl(s): are advertised via LDP, others are not.

ATE-CL032
 Be Careful on the production routers
debug mpls ldp advertisements
Useful to see label bindings that are advertised
debug mpls ldp binding
Useful to see label bindings that are received
debug mpls ldp message sent|received
Useful for the protocol understanding purposes
ATE-CL032
Troubleshooting MPLS
Forwarding Plane

ATE-CL032
 With MPLS, the idea is to de-couple the forwarding from the IP
header
The forwarding decision is based on the MPLS header, not the IP
header
The above is true once the packet is inside the MPLS network
Forwarding is still based on the IP header at the edge where the
packet first enters the MPLS network

ATE-CL032
 CEF must be configured on all the routers in a MPLS network.
CEF takes care of the crucial recursion and resolution
operations
CEF is must for the MPLS

ATE-CL032
 PE-RTR#sh mpls forwarding 10.13.1.11
Local Outgoing Prefix Bytes tag Outgoing Next Hop
tag tag or VC or Tunnel Id switched interface
59 46 10.13.1.11/32 0 Se10/0/0 point2point
PE-RTR#

Outgoing label also conveys what treatment the

packet is going to get. It could also be:
I. Pop - Pops the topmost label
II. Untagged - Untag the incoming MPLS packet
III. Aggregate - Untag and then do a FIB lookup

Label values 0-15 are reserved.

ATE-CL032
PE1#sh mpls forwarding-table
Local Outgoing Prefix Bytes tag Outgoing Next Hop
tag tag or VC or Tunnel Id switched interface
16 2002 10.13.1.22/32 0 Et0/0 10.13.1.5
2002 10.13.1.22/32 0 Et1/0 10.13.1.9
18 Pop tag 10.13.1.101/32 0 Et1/0 10.13.1.9
Pop tag 10.13.1.101/32 0 Et0/0 10.13.1.5
19 Pop tag 10.13.2.4/30 0 Et1/0 10.13.1.9
Pop tag 10.13.2.4/30 0 Et0/0 10.13.1.5
20 Untagged 5.5.5.5/32[V] 0 Se2/0 point2point
21 Pop tag 10.13.21.4/30 0 Et1/0 10.13.1.9
Pop tag 10.13.21.4/30 0 Et0/0 10.13.1.5
24 Aggregate 200.1.61.4/30[V] 0
26 Untagged 30.30.30.1/32[V] 0 Se2/0 point2point
PE1#

ATE-CL032
 Untagged
Convert the incoming MPLS packet to an IP packet and forward it.

Pop
Pop the top label from the label stack present in an incoming MPLS
packet and forward it as an MPLS packet. If there was only one label in
the stack, then forward it as an IP packet. SAME as imp-null label.
Aggregate
Convert the incoming MPLS packet to an IP packet and then do a FIB
lookup for it to find out the outgoing interface.

ATE-CL032
 Three cases in the MPLS forwarding:
1) Label Imposition - IP to MPLS conversion
2) Label swapping - MPLS to MPLS
3) Label disposition - MPLS to IP conversion

So, depending upon the case, we need to check:

1) FIB - For IP packets that get forwarded as MPLS
2) LFIB - For MPLS packets that get fwded as MPLS
3) LFIB - For MPLS packets that get fwded as IP
ATE-CL032
 MPLS Loadsharing (due to multiple paths to a prefix) is no
different from that of IP
Hashing-algorithm is still the typical FIB based i.e per-dest
loadsharing by default **
So the show commands are still relevant
Sh ip cef exact-route <source> <dest> etc.
But the dest must be known in the FIB table, otherwise the
command wont work.
Wont work on P routers for the VPN prefixes.

ATE-CL032
 mpls mtu <bytes> can be applied to an interface to
change the MPLS MTU size on the interface
MPLS MTU size is checked by the router
while converting an IP packet into a labeled packet or
transmitting a labelled packet
Label imposition(s) increases the packet size by 4
bytes/label, hence the outgoing packet size may exceed
interface MTU size, hence the need to tune MTU

ATE-CL032
Remember that :
mpls mtu <bytes> command has no effect on interface
or IP MTU size.
By default, MPLS MTU = interface MTU
MPLS MTU setting doesnt affect MTU handling for IP-to-
IP packet switching

ATE-CL032
 If the label imposition makes the packet bigger than the
MPLS MTU size of an outgoing interface, then:
- If the DF bit set, then discard the packet and send ICMP reply
back (with code=4)
- If the DF bit is not set, then fragment the IP packet (say, into 2
packets), and then impose the same label(s) on both the packets,
and then transmit MPLS packets

ATE-CL032
 show mpls forwarding
Shows all LFIB entries (vpn, non-vpn, TE etc.)

show mpls forwarding <prefix>

LFIB lookup based on a prefix

show mpls forwaring label <label>

LFIB lookup based on an incoming label

show mpls forwarding <prefix> detail

Shows detailed info such as L2 encap etc
ATE-CL032
R2#show mpls forwarding 10.13.1.11 detail
Local Outgoing Prefix Bytes tag Outgoing Next Hop
tag tag or VC or Tunnel Id switched interface
45 51 10.13.1.11/32 0 Fa1/1/1 10.13.7.33
MAC/Encaps=14/18, MRU=1500, Tag Stack{51}
0003FD1C828100044E7548298847 00033000
No output feature configured
Per-packet load-sharing
R2#

14/18 means that the L2 header is of 14 bytes, but

L2+label header is 18 bytes (one label is 4 bytes)

ATE-CL032
Be Careful on the production routers
debug mpls lfib cef
Useful for seeing FIB and LFIB interaction when a label is
missing for a prefix

debug mpls lfib struct

Shows changes in the LFIB structures when label is
allocated/deallocated

ATE-CL032
 Label imposition is always done using FIB

Label swapping and disposition is always done using LFIB

Increase the MPLS MTU to accommodate the largest packet

payload size

Make sure that baby giant/jumbo is enabled on the Ethernet

switches
ATE-CL032
 show mpls forwarding vrf <vrf>
Lists labels for the VPN prefixes learned from the CE(s)
PE1#show mpls forwarding vrf ABC

Local Outgoing Prefix Bytes Label Outgoing Next Hop

Label Label or Tunnel Id Switched interface

19 No Label 100.1.1.1/32[V] 2136 Et0/0 192.168.10.2

20 No Label 192.168.10.0/30[V] 570 aggregate/ABC

ATE-CL032
Case Study

ATE-CL032
 Customer reported traffic forwarding issue to the VRFs attached to a newly
configured PE2 router
The PE1 router has the VPN label which is being shared with the remote PE2
router

On PE1, the CEF shows the correct forwarding output.

ATE-CL032
 The first step in MPLS deployment is to verify if the LSP is complete or not.
Use ping mpls ipv4 <dest-pe-loopback> <subnet_mask> to verify LSP Path
Use traceroute mpls ipv4 <dest-pe-loopback> <subnet_mask> to verify what is the
path and see the point where MPLS packet is getting dropped
The other option is to check the labeling and LFIB information hop by hop or at
least on the node where the MPLS trace is dropped.

ATE-CL032
 The MPLS PING failed
MPLS Trace dropped on P-1 router
Show mpls forwarding <PE2-loopback> output shows no label as outgoing label

P-1# show mpls forwarding 3.3.3.3

Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
17 No Label 3.3.3.3/32 476193 Et0/0 23.23.23.2

Verified that LDP was enabled between the two routers but there was no
bindings

ATE-CL032
 The P-1 router had an ACL to limit the allocation of labels for certain prefixes
Sometimes, there are too many prefixes in the core due to which the labels get
exhausted
To prevent such situations, LDP is configured to allocate labels for certain prefixes but
not all.
PE2 loopback address was added in the ACL which fixed the problem

P-1(config)#no mpls ldp advertise-labels

P-1(config)#mpls ldp advertise-labels for LOOPBACK_ACL

ATE-CL032
References
Cisco Support Community Blog Post
- https://supportforums.cisco.com/blog/12599296/configuring-and-troubleshooting-basic-
mpls-layer3-vpn

ATE-CL032
Thank you

ATE-CL031