RST-3320 (Usa, 2006) TS BGP

Troubleshooting BGP
RST-3320
RST-3320
12552_04_2006_c2
2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
Your Speaker
Kawika Chetron
Cisco IOS-XR BGP Developer
San Jose, CA
kawika@cisco.com
RST-3320
12552_04_2006_c2
Cisco Public
Prerequisites
BGP operational experience
Basic configuration
Show commands
Understand BGP attributes
RST-3320
12552_04_2006_c2
Cisco Public
This Is an Interactive Session

Dont Hesitate to Stop Me to Ask a Question
RST-3320
12552_04_2006_c2
Cisco Public
Overview
BGP peering in excruciating detail
TCP details
BGP
TCP interaction
Fast keepalives and holdtimers
BGP memory analysis

Why does BGP consume so much
memory?
BGP processes
Explanation of what BGP-related
processes do
BGP table version

RST-3320
12552_04_2006_c2
Cisco Public
BGP/TCP Agenda
The Basics
Source/Destination Addresses and Ports
Active vs. Passive Sessions
TCP Connection Collisions
TTLTime to Live
MSSMax Segment Size
Security
RST-3320
12552_04_2006_c2
Cisco Public
BGP/TCP Agenda
EBGP Multihop Peering
Peering State Machine
Common Mistakes
Troubleshooting Commands
Faster Peer Establishment
Flapping Peers
Keepalives and Holdtimers
RST-3320
12552_04_2006_c2
Cisco Public
The Basics
BGP uses TCP port 179
Relies on TCP to successfully deliver all BGP messages
Peers exchange OPEN messages which contain basic info
such as:
Router ID
AS #
Capabilities
FSM (Finite State Machine) enforces rules for state

transitions; exchange of routing information requires being in
the established state
Keepalive/holdtime mechanism ensures peering validity
BGP attempts to open a TCP session to every peer
RST-3320
12552_04_2006_c2
Cisco Public
TCPSource/Destination Parameters
IP Addresses
Destination IP is specified via neighbor x.x.x.x
Source IP is outbound interface by default
Source IP may be specified via
neighbor x.x.x.x update-source interface
TCP port numbers

Destination will be port 179
Source port is ephemeral
RST-3320
12552_04_2006_c2
Cisco Public
TCPSource/Destination Addresses
1.1.1.1
R1
2.2.2.2
10.1.1.1
10.1.1.2
R2
Both sides must agree on source/destination addresses

R1 to R2 connection
neighbor 2.2.2.2 remote-as 100
neighbor 2.2.2.2 update-source loopback 0
R2 to R1 connection
R1 and R2 do not agree on what addresses to use

BGP will tear down the TCP session due to the conflict
Strict enforcement of neighbor addresses helps identify
misconfigurations and adds security
RST-3320
12552_04_2006_c2
Cisco Public
10
R2 attempts to open a session to R1
BGP: 10.1.1.1 open active, local address 2.2.2.2
R1 denies the session because of the address

mismatch
debug ip bgp (Cisco IOS) or debug bgp io
(Cisco IOS-XR) on R1 shows
BGP: 2.2.2.2 passive open to 10.1.1.1
BGP: 2.2.2.2 passive open failed - 10.1.1.1 is
not update-source Loopback0's address (1.1.1.1)
RST-3320
12552_04_2006_c2
Cisco Public
11
1.1.1.1
R1
2.2.2.2
10.1.1.1
10.1.1.2
R2
R1 to R2 connection
R2 to R1 connection
Routers agree on source/destination address

BGP will accept this TCP session
RST-3320
12552_04_2006_c2
Cisco Public
12
TCPActive vs. Passive Session

R1 Opens TCP Session to R2
R1
R2
Active sessionIf the TCP session initiated by R1 is the one used

between R1 and R2 then R1 actively established the session
Passive sessionFor the same scenario R2 passively established
the session
R1 actively opened the session
R2 passively accepted the session
Can be configured
IOS: neighbor x.x.x.x transport connection-mode [active|passive]
IOS-XR: neighbor x.x.x.x session-open-mode {active-only |
passive-only}
RST-3320
12552_04_2006_c2
Cisco Public
13
TCPActive vs. Passive Session

Use show ip bgp neighbor to determine if a
router actively or passively established a session
(Cisco IOS only)
R1#show ip bgp neighbors 2.2.2.2
BGP neighbor is 2.2.2.2, remote AS 200, external link
BGP version 4, remote router ID 2.2.2.2
[snip]
Local host: 1.1.1.1, Local port: 12343
Foreign host: 2.2.2.2, Foreign port: 179
TCP open from R1 to R2s port 179 established

the session
Tells us that R1 actively established the session
RST-3320
12552_04_2006_c2
Cisco Public
14
TCPConnection Collision
1.1.1.1
R1
2.2.2.2
R2
It is possible for both sides to initiate a session at the

same time
The session that is actively established by the peer with the
highest router-ID is the winner
Rarely happens
Not an issue if this does occur
Connection collisions and graceful restart
RST-3320
12552_04_2006_c2
Cisco Public
15
TTL of TCP PacketsTime to Live

TTL 1default
AS 100 TTL 2ebgp-multihop 2

R1
R2
AS 200
R3
BGP uses a TTL of 1 for eBGP peers

For eBGP peers that are more than 1 hop away a larger TTL
must be used
neighbor x.x.x.x ebgp-multihop
[2-255]
RST-3320
12552_04_2006_c2
Cisco Public
16
TCP MSSMax Segment Size

MSSLimit on the largest packet that can traverse a TCP session
Anything larger must be fragmented and re-assembled at the TCP layer
MSS is 536 bytes by default
536 bytes is inefficient for Ethernet (MTU of 1500) or POS (MTU of

4470) networks
TCP is forced to break large packets into 536 byte chunks
Adds overheads
Slows BGP convergence and reduces scalability
ip tcp path-mtu-discovery (Cisco IOS) tcp path-mtudiscovery (Cisco IOS-XR)

MSS = Lowest MTU between destinationsIP overhead (20 bytes)TCP overhead
(20 bytes)
1460 bytes for Ethernet network
4430 bytes for POS network
Will be enabled by default for BGP sessions in the future

New knob will allow you to enable/disable per peer
RST-3320
12552_04_2006_c2
Cisco Public
17
TCP MSSMax Segment Size
BGP Messages
TCP Packets
MSS of 536
TCP Packets
MSS of 1460
KA
Update
KA
Update
536 bytes
1460 bytes
BGP KAs (keepalives) are 19 bytes

BGP updates vary is size up to 4096 bytes
The larger the TCP MSS the fewer TCP packets required
Fewer packets means less overhead and faster convergence
RST-3320
12552_04_2006_c2
Cisco Public
18
TCP Security
Minimal built in security
Random source port #s
Strict source/destination IP agreement
TCPs MD5 authentication should be used

neighbor x.x.x.x password FOO
MD5 + BTSH (BGP TTL Security Hack) provides

protection with minimal CPU cost
RST-3320
12552_04_2006_c2
Cisco Public
19
BTSHBGP TTL Security Hack

Hacker
ISP
R1
AS 100
R2
Valid BGP Messages

Spoofed BGP Messages
Hackers spoof BGP messages to R1 as if they are R2

R1 must use MD5 to filter out the bogus messages
MD5 validation must be done on the RP (route processor)
RST-3320
12552_04_2006_c2
Cisco Public
20

Provides a lightweight mechanism to defend
against most BGP spoof attacks
Does not replace the need for MD5 authentication
Sender sets the TTL to 255

Receiver checks for a TTL of 254 for directly
connected neighbors
A lower acceptable TTL value must be configured for
multihop neighbors
RST-3320
12552_04_2006_c2
Cisco Public
21

Hacker
ISP
TTL 255
R1
AS 100
R2
TTL 254
TTL 255
R1 and R2 both use BTSH

Both sides must configure the feature
neighbor x.x.x.x ttl-security 255 (IOS)
neighbor x.x.x.x ttl-security
(IOS-XR)
May use BTSH instead of ebgp-multihop if you control both ends
of the session
Packets from R2 will have a TTL of 255

Packets generated by hackers will have a TTL that is less than 255
Easy to compare the TTL value vs. the 255 threshold and discard spoofed packets
Discards can be done at the linecard
TTL check is much cheaper than MD5
RST-3320
12552_04_2006_c2
Cisco Public
22

Attack scope is reduced to directly
connected devices
MD5 should still be used to authenticate any
message that makes it past BTSH
RST-3320
12552_04_2006_c2
Cisco Public
23
eBGP Disable-Connected-Check
eBGP peers must meet one of the following criteria
Are directly connected which is verified by comparing the
eBGP peers address with our connected subnets
Are configured for ebgp-multihop which disables the
connected subnet check
Single hop eBGP loopback peering does not fit

either rule very well
Default TTL (Time to Live) is one so neighbor x.x.x.x
ebgp-multihop 1 is silently ignored by the parser
neighbor x.x.x.x ebgp-multihop 2
must be used here
RST-3320
12552_04_2006_c2
Cisco Public
24
R1 and R3 are eBGP peers
that are loopback peering
Older code must use the
following in R1 and R3
neighbor x.x.x.x ebgpmultihop 2
R1
R3
AS 100
AS 200
Small security hole

If the R1 to R3 link goes down
the session could establish
via R2
R2
Desired Path
Used Path
RST-3320
12552_04_2006_c2
Cisco Public
25
New code does not need an
ebgp-multihop statement;
Instead use:
neighbor x.x.x.x
disable-connectedcheck
TTL is one
Session cannot establish
via R2
R1
R3
AS 100
AS 200
R2
If R1 to R3 link is down so is
the BGP session
Closes security hole
Available in Cisco IOS, but
not in Cisco IOS-XR
RST-3320
12552_04_2006_c2
Cisco Public
26

Enough TCP talk
Assume we can successfully establish TCP
sessions with our peers
BGP peers use a finite state machine to reach
established state
Possible peering states:
Idle
Connect
Active
Open sent
Open confirm
Established
RST-3320
12552_04_2006_c2
Cisco Public
27

Step #1Idle stateBGP is currently not attempting to
establish a session with the peer
Step #2Active stateBGP is actively trying to open a TCP
session with the peer
Step #3Open sentBGP sent an OPEN message to the peer
An OPEN message contains
Versionshould always be version 4
AS#The AS# of the router generating the OPEN message
HoldTimer180 seconds by default but can be modified via
neighbor x.x.x.x timers X Y; X is the keepalive timer, Y is the holdtimer
Router-IDIf unspecified, some valid router-id will be chosen;
bgp router-id x.x.x.x overrides default ID selection
Optional parametersCapabilities such as route refresh, IPv6 support,
VPNv4 support, etc.
RST-3320
12552_04_2006_c2
Cisco Public
28
OPEN Message
0
10
11 12 13 14 15 16 17 18 19
20
21 22 23 24 25 26 27 28 29
30
31
Version
My Autonomous System
Hold Time
BGP Identifier
Opt. Parm. Len.
Optional Parameters
RST-3320
12552_04_2006_c2
Cisco Public
29

Upon receiving an OPEN message
Validate AS#
Check for invalid router IDs
Lowest holdtimer will be used
Parse capabilities
Step #4Open confirmSend a keepalive as an

acknowledgement to the received OPEN message
Step #5EstablishedOnce you have received a
keepalive to acknowledge your OPEN message go
to established state
RST-3320
12552_04_2006_c2
Cisco Public
30

Active
OpenSent
Idle
Valid Condition
Error Condition
OpenConfirm
Established
If everything is okay, proceed to the next state

If not, reset back to Idle state
RST-3320
12552_04_2006_c2
Cisco Public
31

debug ip bgp (Cisco IOS) or debug bgp (Cisco IOS-XR) shows
the state transitions
BGP:
BGP:
BGP:
BGP:
BGP:
BGP:
BGP:
BGP:
BGP:
BGP:
BGP:
BGP:
BGP:
BGP:
BGP:
BGP:
BGP:
BGP:
BGP:
BGP:
BGP:
RST-3320
12552_04_2006_c2
2.2.2.2
2.2.2.2
2.2.2.2
2.2.2.2
2.2.2.2
2.2.2.2
2.2.2.2
2.2.2.2
2.2.2.2
2.2.2.2
2.2.2.2
2.2.2.2
2.2.2.2
2.2.2.2
2.2.2.2
2.2.2.2
2.2.2.2
2.2.2.2
2.2.2.2
2.2.2.2
2.2.2.2
went from Idle to Active

open active, local address 1.1.1.1
read request no-op
went from Active to OpenSent
sending OPEN, version 4, my as: 100, holdtime 180 seconds
send message type 1, length (incl. header) 45
rcv message type 1, length (excl. header) 26
rcv OPEN, version 4, holdtime 180 seconds
rcv OPEN w/ OPTION parameter len: 16
rcvd OPEN w/ optional parameter type 2 (Capability) len 6
OPEN has CAPABILITY code: 1, length 4
OPEN has MP_EXT CAP for afi/safi: 1/1
OPEN has ROUTE-REFRESH capability(old) for all address-families
OPEN has ROUTE-REFRESH capability(new) for all address-families
rcvd OPEN w/ remote AS 200
went from OpenSent to OpenConfirm
went from OpenConfirm to Established
Cisco Public
32
Peer EstablishmentCommon Mistakes

There are dozens of configuration mistakes that can prevent peer
establishment
No IP connectivity
Can you ping your peer?
If loopback peering, use extended ping to set the source address correctly
AS numbersare they correct?

router bgp X
neighbor 1.1.1.1 remote-as X
neighbor 1.1.1.1 local-as X
bgp confederation identifier X
bgp confederation peers X Y Z
Peering addressesare both sides in agreement?

Multihop eBGP peeringare you using one of the following?
neighbor x.x.x.x ebgp-multihop [2-255]
neighbor x.x.x.x disable-connected-check
neighbor x.x.x.x ttl-security
RST-3320
12552_04_2006_c2
Cisco Public
33
Troubleshooting Commands
debugs and show commands are your friends
Use them to get more information on why a peer
will not establish
TCP
show tcp brief all (Cisco IOS) show tcp brief
(Cisco IOS-XR)
show tcp statistics
debug ip tcp transactions (Cisco IOS only)
BGP
debug ip bgp
debug ip bgp events
show ip bgp neighbor
RST-3320
12552_04_2006_c2
Cisco Public
34
show tcp brief all

Display a list of addresses/port numbers that the
router will accept TCP sessions on
R1 TCP info
R1#show tcp brief all
TCB
Local Address
Foreign Address
(state)
64316F14
1.1.1.1.12345
2.2.2.2.179
ESTAB
6431BA8C
*.179
2.2.2.2.*
LISTEN
62FFDEF4
*.*
*.*
LISTEN
R1#
R2 TCP info
R2#show tcp brief all
TCB
Local Address
Foreign Address
(state)
641606FC
2.2.2.2.179
1.1.1.1.12345
ESTAB
6415B49C
*.179
1.1.1.1.*
LISTEN
638471A4
*.*
*.*
LISTEN
R2#
RST-3320
12552_04_2006_c2
Cisco Public
35
show tcp statistics

Output has rcvd and sent sections
Key rcvd stats
Checksum errors
Out-of-order packets
# bytes acknowledged
R1show tcp statistics

Rcvd: 7005 Total, 10 no port
0 checksum error, 0 bad offset, 0 too short
1166 packets (67334 bytes) in sequence
0 dup packets (0 bytes)
0 partially dup packets (0 bytes)
0 out-of-order packets (0 bytes)
0 packets (0 bytes) with data after window
0 packets after close
0 window probe packets, 0 window update packets
0 dup ack packets, 0 ack packets with unsend data
4186 ack packets (73521 bytes)
RST-3320
12552_04_2006_c2
Cisco Public
36
show tcp statistics

Key sent stats
Bytes sent
Average packet size = (bytes sent)/(# data packets)
Is average packet size roughly MSS?
# connections initiated, accepted, closed, etc.
R1 sent stats
Sent: 9150 Total, 0 urgent packets
4810 control packets (including 127 retransmitted)
2172 data packets (71504 bytes)
0 data packets (0 bytes) retransmitted
0 data packets (0 bytes) fastretransmitted
2168 ack only packets (160 delayed)
0 window probe packets, 0 window update packets
1346 Connections initiated, 10 connections accepted, 1014 connections
established
1790 Connections closed (including 0 dropped, 443 embryonic dropped)
127 Total rxmt timeout, 0 connections dropped in rxmt timeout
0 Keepalive timeout, 0 keepalive probe, 0 Connections dropped in keepalive
RST-3320
12552_04_2006_c2
Cisco Public
37
debug ip tcp transactions

debug ip tcp transactions
Can be very chatty
Use with care
Output when R1 and R2 create a session

R1#sh log | i TCP0:
TCP0: state was ESTAB -> FINWAIT1 [12345 -> 2.2.2.2(179)]
TCP0: sending FIN
TCP0: state was FINWAIT1 -> FINWAIT2 [12345 -> 2.2.2.2(179)]
TCP0: FIN processed
TCP0: state was FINWAIT2 -> TIMEWAIT [12345 -> 2.2.2.2(179)]
TCP0: Connection to 2.2.2.2:179, advertising MSS 1460
TCP0: state was CLOSED -> SYNSENT [12346 -> 2.2.2.2(179)]
TCP0: state was SYNSENT -> ESTAB [12346 -> 2.2.2.2(179)]
TCP0: tcb 6430DCDC connection to 2.2.2.2:179, received MSS 1460, MSS is 1460
R1#
RST-3320
12552_04_2006_c2
Cisco Public
38

Four sections of output
AFI independent BGP info
AFI specific BGP info
BGP specific TCP info
Generic TCP info
Section 1AFI independent BGP info

BGP state = Established, up for 00:02:07
Last read 00:00:06, last write 00:00:13, hold time is 180, keepalive interval is 60 seconds
Neighbor capabilities:
Route refresh: advertised and received(new)
Address family IPv4 Unicast: advertised and received
Message statistics:
InQ depth is 0
OutQ depth is 0
Sent
Rcvd
Opens:
6
6
Notifications:
0
0
Updates:
4
0
Keepalives:
175
177
Route Refresh:
0
0
Total:
185
183
Default minimum time between advertisement runs is 30 seconds
RST-3320
12552_04_2006_c2
Cisco Public
39

Section 2AFI specific BGP info
For address family: IPv4 Unicast
BGP table version 2, neighbor version 2/0
Output queue size : 0
Index 1, Offset 0, Mask 0x2
1 update-group member
Sent
Rcvd
Prefix activity:
------Prefixes Current:
1
0
Prefixes Total:
1
0
Implicit Withdraw:
0
0
Explicit Withdraw:
0
0
Used as bestpath:
n/a
0
Used as multipath:
n/a
0
Outbound
Inbound
Local Policy Denied Prefixes:
-------------Total:
0
0
Number of NLRIs in the update sent: max 1, min 1
RST-3320
12552_04_2006_c2
Cisco Public
40

Section 3BGP specific TCP info
Connections established 6; dropped 5
Last reset 00:02:09, due to User reset
External BGP neighbor may be up to 255 hops away.
Connection state is ESTAB, I/O status: 1, unread input bytes: 0
Local host: 1.1.1.1, Local port: 12348
Foreign host: 2.2.2.2, Foreign port: 179
RST-3320
12552_04_2006_c2
Cisco Public
41

Section 4Generic TCP info
Enqueued packets for retransmit: 0, input: 0
Event Timers (current time is 0x5817B38):
Timer
Starts
Wakeups
Retrans
5
0
TimeWait
0
0
AckHold
4
3
SendWnd
0
0
KeepAlive
0
0
GiveUp
0
0
PmtuAger
0
0
DeadWait
0
0
iss: 3541899715
irs: 2288128196
snduna: 3541899871
rcvnxt: 2288128318
mis-ordered: 0 (0 bytes)
Next
0x0
0x0
0x0
0x0
0x0
0x0
0x0
0x0
sndnxt: 3541899871
rcvwnd:
16263
sndwnd:
delrcvwnd:
16229
121
SRTT: 146 ms, RTTO: 1283 ms, RTV: 1137 ms, KRTT: 0 ms
minRTT: 0 ms, maxRTT: 300 ms, ACK hold: 200 ms
Flags: higher precedence, retransmission timeout, nagle, path mtu capable
Datagrams (max data segment is 1460 bytes):
Rcvd: 7 (out of order: 0), with data: 4, total data bytes: 121
Sent: 10 (retransmit: 0), with data: 5, total data bytes: 155
RST-3320
12552_04_2006_c2
Cisco Public
42

RFC 4271 ConnectRetry Timer
If a connection fails, wait ConnectRetry seconds before attempting
another connection
Used to insert a delay between connection attempts
Avoids DOSing a router with aggressive connection attempts
RFC suggests 120 seconds
Old behavior
OPEN delays were jittered
eBGP peer OPEN delay35 to 50 seconds
iBGP peer OPEN delay25 to 30 seconds
Times have changed

120 seconds is an eternity in the sub-second convergence world
Even 25 seconds is too long
RST-3320
12552_04_2006_c2
Cisco Public
43

New behavior
Delay OPENs when you need to
OPEN immediately for all other cases
Still use jitter as a safety mechanism
If an OPEN fails, increase the delay exponentially
Open delays are good when:

A peer is unreachable
A peer is misconfigured such that the session is torn down immediately;
Examples: Incorrect AS# or duplicate router ID
An error occurred that we likely cannot recover from quickly;
Examples: Hold time expired or max-prefix exceeded
Open delays are bad when:

A peer is reachable and configured correctly
RP failover
User interaction tears down an established peer
clear ip bgp *
Changing the BGP router ID or cluster ID
RST-3320
12552_04_2006_c2
Cisco Public
44

647 msecs to re-establish peer after clear ip bgp *
R1#show log | include ADJCHANGE
*May 2 14:40:50.551: %BGP-5-ADJCHANGE: neighbor 2.2.2.2 Down User reset
*May 2 14:40:51.198: %BGP-5-ADJCHANGE: neighbor 2.2.2.2 Up
R1#
Assume we misconfigure R1 with an incorrect remote-as

The session will constantly bounce so we want a larger OPEN delay
OPEN delay is larger and is jittered

debug ip bgp shows
R1#show log | include Idle to Active|delayed
BGP: 2.2.2.2 went from Idle to Active
BGP: 2.2.2.2 open active delayed 33704ms (35000ms
RST-3320
12552_04_2006_c2
max, 28% jitter)

max, 28% jitter)
max, 28% jitter)
max, 28% jitter)
Cisco Public
45
Flapping Peers
Peers may flap for several reasons
OPEN parameters prevent peer establishment
Bad message could trigger a NOTIFICATION
Holdtime expires
RST-3320
12552_04_2006_c2
Cisco Public
46
Flapping PeersOPEN Parameters

Peers must exchange OPEN messages
Peering may fail if a field is incorrect
Peering routers share the same router ID
Incorrect configuration of an AS#
Easy to figure out with debug ip bgp

R1#show log | i NOTIFICATION
%BGP-3-NOTIFICATION: sent to neighbor 2.2.2.2 2/2 (peer
in wrong AS) 2 bytes 00C8 FFFF FFFF FFFF FFFF FFFF FFFF
FFFF FFFF 002D 0104 00C8 00B4 0202 0202 1002 0601 0400
0100 0102 0280 0002 0202 00
RST-3320
12552_04_2006_c2
Cisco Public
47
Flapping PeersBad Messages

BGP message could be bad for two reasons
Doesnt meet protocol standards
Parameters do not agree with configuration
If a bad message is received

Store a copy of the message for troubleshooting purposes
Send the peer a NOTIFICATION
Tear down the session
Example
[snip]
Connections established 4; dropped 4
Last reset 00:07:40, due to BGP Notification sent, peer in wrong AS
Message received that caused BGP to send a Notification:
FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
002D0104 00C800B4 02020202 10020601
04000100 01020280 00020202 00
RST-3320
12552_04_2006_c2
Cisco Public
48

Holdtime is negotiated via OPEN messages
In seconds
Lowest holdtime requested by the two peers wins
0 seconds means infinite holdtime
3 seconds is the lowest non-zero setting
180 seconds by default
Keepalive (KA) timer must be lower than the holdtimer

In seconds
Is usually 1/3 of the holdtime
1 second is the lowest setting
60 seconds by default
RST-3320
12552_04_2006_c2
Cisco Public
49

Both timers are configurable
neighbor x.x.x.x timers X Y Z
X (0-65535) is keepalive
Y (0-65535) is holdtime
Z (0-65535) minimum acceptable holdtime
A per-peer holdtime expire timer tracks last

received KA
Cisco treats all BGP messages as KAs
Receiving an UPDATE will reset a peers holdtime expired
timer back to zero
Promotes stability by honoring more than just KAs
RST-3320
12552_04_2006_c2
Cisco Public
50

If a peers holdtime expire timer reaches zero, tear
down the session
If we send a holdtime expired NOTIFICATION
We did not receive a KA for holdtime from the peer
If we receive a holdtime expired NOTIFICATION

Our peer did not receive a KA from us for holdtime
RST-3320
12552_04_2006_c2
Cisco Public
51

Holdtime NOTIFICATION
R1
R2
Assume R2 sends a holdtime expired NOTIFICATION to R1

R2 did not receive a KA from R1 for holdtime seconds
%BGP-5-ADJCHANGE: neighbor 1.1.1.1 Down BGP
Notification sent
%BGP-3-NOTIFICATION: sent to neighbor
1.1.1.1 4/0 (hold time expired) 0 bytes
Troubleshooting steps
RST-3320
12552_04_2006_c2
Cisco Public
52

Did R1 build and transmit a keepalive for R2?
debug ip bgp keepalive
When did we last send or receive data with

the peer?
BGP state = Established, up for 00:12:49
Last read 00:00:45, last write 00:00:44, hold time is 180, keepalive interval is 60 seconds
If R1 did not build and transmit a KA

How is R1 on memory?
What is the R1s CPU load?
Is R2s TCP window open?
RST-3320
12552_04_2006_c2
Cisco Public
53

Were the KAs from R1 to R2 lost in transit?
Is routing correct?
Can you successfully ping from R1 to R2 while the session
is established?
Extended ping using update-source as source address if
applicable
Check outbound interface drops on R1

Check inbound interface drops on R2
Check for drops on transit devices between
R1and R2
RST-3320
12552_04_2006_c2
Cisco Public
54

Common reasons
BGP speaker is out of memory
BGP speaker has extremely high CPU
KAs are dropped on overrun interface queues
KAs are disappearing somewhere in the network
Black holes
Routing loops
QoS drops
RST-3320
12552_04_2006_c2
Cisco Public
55
Flapping PeersCase Study
AS 1
AS 2
eBGP
R1
R2
Layer 2 Cloud
Small Packets
Large Packets
Small packets are ok

Large packets are lost in the cloud
BGP session flaps
RST-3320
12552_04_2006_c2
Cisco Public
56
Flapping Peer
bgp log-neighbor-changes will generate an
ADJCHANGE message when a peer flaps
R2#
%BGP-5-ADJCHANGE: neighbor 1.1.1.1 Down BGP Notification sent
%BGP-3-NOTIFICATION: sent to neighbor 1.1.1.1 4/0 (hold time expired) 0 bytes
R2#show ip bgp neighbor 1.1.1.1 | include Last reset
Last reset 00:01:02, due to BGP Notification sent,hold time expired
R2 is sending a NOTIFICATION to R1
R1 has a problem sending keepalives?
The keepalives are lost in the cloud?
R2 has a problem receiving the keepalive?
RST-3320
12552_04_2006_c2
Cisco Public
57
Flapping Peer
R1#show ip bgp sum | begin Neighbor
Neighbor
V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
2.2.2.2
4
2
53
284
10167 0
97
00:02:15
0
R1#show ip bgp summary | begin Neighbor
Neighbor
V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
2.2.2.2
4
2
53
284
10167 0
98 00:03:04
0
At 2 min 15 seconds 284 messages have been delivered

A keepalive is generated, OutQ increases from 97 to 98
At 3 min 4 seconds we still have only delivered 284 BGP
messages
Notice that the MsgSent counter has not moved
Hellos are stuck in OutQ behind update packets
RST-3320
12552_04_2006_c2
Cisco Public
58
Flapping Peer
R1#ping 2.2.2.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 16/21/24 m
R1#ping ip
Target IP address: 2.2.2.2
Repeat count [5]:
Datagram size [100]: 1500
Timeout in seconds [2]:
Extended commands [n]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 5, 1500-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
Normal pings work but a ping of 1500 bytes fails?

RST-3320
12552_04_2006_c2
Cisco Public
59
Flapping Peer
Things to check
MTU values
Traffic shaping
Rate-limiting parameters
Looks like a Layer 2 problem

At this point we have verified that BGP
is not at fault
Next step is to troubleshoot Layer 2
RST-3320
12552_04_2006_c2
Cisco Public
60
Flapping Peer
AS 1
AS 2
eBGP
R1
R2
Layer 2
ATM or FR
Cloud
Small Packets
Large Packets
Large packets are ok now

BGP session is stable
RST-3320
12552_04_2006_c2
Cisco Public
61
Timer Tuning
KA and holdtime tuning
KA of one and holdtime of three is the minimum
Timers of 3/9 are used by customers today
Low timers
Detect failures quickly
Less stable during network turbulence
Could make turbulence more severe
High timers
Detect failures slowly
Much more stable during network turbulence
How low should you go? It depends

How stable is your network?
How critical is fast convergence?
How much available CPU do you have?
Will you loose your job if a peer flaps?
RST-3320
12552_04_2006_c2
Cisco Public
62
Timer TuningCase Study

7200 with NPE-400 running 12.2(28)S
1000 BGP peers
3/9 timers
CPU at 19% in steady state
1/3 timers
CPU at 31% in steady state
Sort the show proc cpu output to see the most

active processes (top can be used in IOS-XR)
dw-7204a#sh proc cpu sorted 5sec
CPU utilization for five seconds: 31%/10%; one minute: 29%; five minutes: 22%
PID Runtime(ms)
Invoked
uSecs
5Sec
1Min
5Min TTY Process
135
74460
3473
21439 11.27% 10.34% 7.85%
0 BGP I/O
46
36620
6309
5804 6.31% 6.28% 4.81%
0 IP Input
79
6276
6918
907 1.75% 1.20% 0.78%
0 TCP Timer
134
13984
4773
2929 1.59% 1.82% 1.82%
0 BGP Router
136
44
37
1189 0.07% 0.00% 0.00%
0 BGP Scanner
RST-3320
12552_04_2006_c2
Cisco Public
63
Timer TuningCase Study

Peers were stable but
Zero BGP routes present, no IGP, basic config, etc.
Your mileage may vary in the real world
31% of the CPU just for keepalives is significant

Will you always have 31% available?
CPUHOGs, temporary network congestion, etc. are
more likely to result in neighbor flaps
RST-3320
12552_04_2006_c2
Cisco Public
64
Timer TuningAlternate Solutions

Fast keepalives do not scale very well
IGPs have been using them for a while
They work but it isnt the ideal solution
Polling models in general can get expensive
There are two event driven (instead of polling)

solutions
bgp fast-external-fallover
FSD (Fast Session Deactivation) via ATF (Address
Tracking Feature) (Cisco IOS only)
RST-3320
12552_04_2006_c2
Cisco Public
65
If the interface used to reach an eBGP peer goes
down, tear down the session
Only works for single hop eBGP peers
Enabled by default
Avoid waiting for holdtime to expire
RST-3320
12552_04_2006_c2
Cisco Public
66
ATFAddress Tracking Filter

ATF is a middle man between clients that use the
RIB and the RIB
Clients could be BGP, OSPF, EIGRP, etc.
A client tells ATF what prefixes he is interested in

ATF tells the client when one of these prefixes has
a RIB change
RST-3320
12552_04_2006_c2
Cisco Public
67
BGP FSD
BGP FSDFast Session Deactivation
Register peers addresses with ATF
ATF will let BGP know if there is a change to a
peers address
If we lose our route to the peer, tear down
the session
No need to wait for the holdtimer to expire
RST-3320
12552_04_2006_c2
Cisco Public
68
BGP FSD
Ideal for IBGP peers and multihop eBGP peers
Can tear down BGP sessions at IGP
convergence speed
Off by default
neighbor x.x.x.x fall-over
RST-3320
12552_04_2006_c2
Cisco Public
69
Memory Agenda
BGPs use of memory
Memory loss after a Cisco IOS upgrade
Memory loss after multipath
RST-3320
12552_04_2006_c2
Cisco Public
70
Memory
Why does a BGP router use so much memory?
Every BGP customer
Typically BGP stores 100x the number of routes as

your IGP
Every prefix must be stored in multiple places
BGP has a data structure
The RIB has a data structure
CEF has a data structure
RST-3320
12552_04_2006_c2
Cisco Public
71
Memory
Each Prefix Must Be Stored by
Three Separate Databases
R1#show ip bgp 22.2.2.2
BGP routing table entry for 22.2.2.2/32, version 3
Paths: (1 available, best #1, table Default-IP-Routing-Table)
Flag: 0x820
Not advertised to any peer
200
2.2.2.2 from 2.2.2.2 (2.2.2.2)
Origin IGP, metric 0, localpref 100, valid, external, best
R1#show ip route 22.2.2.2
Routing entry for 22.2.2.2/32
Known via "bgp 100", distance 20, metric 0
Tag 200, type external
Last update from 2.2.2.2 00:00:15 ago
Routing Descriptor Blocks:
* 2.2.2.2, from 2.2.2.2, 00:00:15 ago
Route metric is 0, traffic share count is 1
AS Hops 1
Route tag 200
R1#show ip cef 22.2.2.2
22.2.2.2/32
nexthop 2.2.2.2 FastEthernet0/1
RST-3320
12552_04_2006_c2
Cisco Public
72
Memory
I upgraded from release X to release Y and my free memory
decreased by 8 meg.
Mr. Angry Customer
New features come at a cost

Example
Feature FOO forces us to store an additional 12 bytes for
every prefix and 8 bytes for every path
A network has 150k prefixes and 400k prefixes
(150k * 12) + (400k * 8) = 5 meg increase in memory
consumption
Other 3 meg could be due to
Other new features in Cisco IOS
Increased image size
RST-3320
12552_04_2006_c2
Cisco Public
73
Memory
I carry a full Internet feed, I enabled BGP multipath
and BGP Router began consuming tons of
memory.
Mr. Brave Customer
CEF needs to store a loadinfo structure for every

multipath prefix
Loadinfo structures are normal for IGP routes
Without multipath, you wont see loadinfos for BGP routes
Multipath is designed for a handful of routes

The full Internet table is way more than a handful
Use with caution
RST-3320
12552_04_2006_c2
Cisco Public
74
Cisco IOS BGP Processes

Four BGP processes
BGP Open
In charge of accepting/rejecting TCP sessions for a peer
Each peer has their own BGP Open process
Process dies after peer is established
BGP IO
Interacts with TCP
InboundParse data from TCP into BGP messages
InboundOnce assembled place the message on a
peers InQ
OutboundPull BGP messages off of each peers OutQ
and deliver to TCP
RST-3320
12552_04_2006_c2
Cisco Public
75

BGP Scanner
Import scanner runs once every 15 seconds
Imports VPNv4 routes into vrfs
bgp scan-time import X
Full scanner run happens every 60 seconds

bgp scan-time X
Lowering this value is not recommended
Full scan performs multiple housekeeping tasks

Validate nexthop reachability
Validate bestpath selection
Route redistribution and network statements
Conditional advertisement
Route dampening
BGP database cleanup
RST-3320
12552_04_2006_c2
Cisco Public
76

BGP Router
The core of Cisco IOSs BGP implementation
Originate routes
Network statements
Route redistribution
Aggregate statements
Process inbound messages

Remove messages of off peers InQs
Parse the messages
Evaluate inbound policies
RST-3320
12552_04_2006_c2
Cisco Public
77

Calculate bestpath for all routes
BGP has as 12-step program to determine the bestpath
Keep the RIB up to date

Add/delete routes to reflect bestpath changes
Modify current RIB entry in place when possible
Generate updates to converge peers

Most stressful task for BGP router
Evaluate each prefix against the outbound policies of peers
Generate updates/withdraws as needed
A prefix must be reevaluated when its bestpath changes
RST-3320
12552_04_2006_c2
Cisco Public
78
Cisco IOS-XR BGP Processes

BGP (speaker) process
Establish connections, send/receive routes
BPM
Configuration, starting of other processes
bRIB
Central collection of partial bestpaths
Run in distributed mode only
RST-3320
12552_04_2006_c2
Cisco Public
79
BGP Scanner
CPU spike is normal when scanner runs
Is a low priority process
Scanner spike shouldnt adversely effect other processes
Scanning a full table of internet routes is a big job

debug ip bgp events will show you when scanner ran for each
address-family
BGP: Performing BGP general scanning
BGP(0): scanning IPv4 Unicast routing tables
BGP(IPv4 Unicast): Performing BGP Nexthop scanning for general scan
BGP(0): Future scanner version: 7, current scanner version: 6
BGP(1): scanning IPv6 Unicast routing tables
BGP(IPv6 Unicast): Performing BGP Nexthop scanning for general scan
BGP(2): scanning VPNv4 Unicast routing tables
BGP(VPNv4 Unicast): Performing BGP Nexthop scanning for general scan
BGP(4): scanning IPv4 Multicast routing tables
BGP(IPv4 Multicast): Performing BGP Nexthop scanning for general scan
BGP(5): scanning IPv6 Multicast routing tables
BGP(IPv6 Multicast): Performing BGP Nexthop scanning for general scan
RST-3320
12552_04_2006_c2
Cisco Public
80
BGP Scanner
Improvements have been made to reduce
CPU impact of scanner
Route redistribution is now fully event-driven
Network statements are now fully event-driven
Nexthop Tracking (NHT)

NHT detects that our route to one of our BGP nexthops
has changed
NHT triggers a lightweight scanner run that only validates
nexthop reachability and recalculates bestpaths
Nexthop and bestpath validation no longer happens in
scanner every 60 seconds
RST-3320
12552_04_2006_c2
Cisco Public
81
BGP Scanner
BGP Scanner Improvements
80%
12.0S
12.2S
70%
CPU Load
60%
50%
40%
30%
20%
10%
0%
27
46
66
85 110 132 152 171 193 219 239 258 277 305 326 346 366
Time (Seconds)
7200 with NPE-G1

900k routes in the BGP table
BGP Scanner in 12.2S uses much less CPU
RST-3320
12552_04_2006_c2
Cisco Public
82
BGP Table Version

Lots of things must happen when bestpaths change
RIB must be notified
Peers must be informed
Must have a way to track who has been informed of which
bestpath changes
Prefix table version

Each prefix has a 32 bit number that is its table version
A prefixs table version is bumped for every bestpath change
Bumped means the table version is changed from the current
version to the next available version #
Assume 10.0.0.0/8 has a table version of 27 and the highest table
version used by any prefix is 30; If 10.0.0.0/8 has a bestpath
change his table version will be bumped to 31
RST-3320
12552_04_2006_c2
Cisco Public
83
BGP Table Version

show ip bgp x.x.x.x
will show you a prefixs table version
R1#sh ip bgp 10.0.0.0
BGP routing table entry for 10.0.0.0/8, version 31
Paths: (1 available, best #1, table Default-IP-Routing-Table)
Flag: 0x820
Not advertised to any peer
200
2.2.2.2 from 2.2.2.2 (2.2.2.2)
Origin IGP, metric 0, localpref 100, valid, external, best
R1#
RST-3320
12552_04_2006_c2
Cisco Public
84
BGP Table Version

RIB and peer table versions
We have a table version for the RIB
Also have a table version for each peer
Used to keep track of which bestpath
changes have been propagated to whom
If peer 1.1.1.1 has a table version of 60 this tells us we have

informed 1.1.1.1 of all bestpath changes for prefixes with a
table version of <= 60
If any prefix has a table version > 60 then we need to inform
1.1.1.1 of that prefixs bestpath
Once 1.1.1.1 has been updated his table version will be
updated accordingly
Same concept for the RIB and its table version
RST-3320
12552_04_2006_c2
Cisco Public
85
BGP Table Version

show ip bgp summary is best for viewing RIB and
peer version #s
R2#show ip bgp summ
BGP router identifier 2.2.2.2, local AS number 200
BGP table version is 13, main routing table version 13
3 network entries using 351 bytes of memory
3 path entries using 156 bytes of memory
Neighbor
State/PfxRcd
1.1.1.1
R2#
V
4
AS MsgRcvd MsgSent
100
4386
4388
TblVer
13
InQ OutQ Up/Down

0
0 01:20:24
Highest table version of any prefix = main routing

table version
RIB is converged
1.1.1.1 is converged
RST-3320
12552_04_2006_c2
Cisco Public
86
BGP Table Version

Example
Assume the highest table version of any prefix is 10
The RIB has a table version of 10
The RIB is up to date for all prefixes
All peers have a table version of 10

Our peers are currently converged
Five prefixes experience a bestpath change

Highest table version is now 15
Inform the RIB of these five changes
Do RIB adds, deletes, and/or modifies
When complete, set the RIB table version to 15
Inform our peers of these five changes

Build updates and/or withdraws for each peer
When complete, set our peers table versions to 15
RST-3320
12552_04_2006_c2
Cisco Public
87
BGP Table Version

Why am I babbling about this?
Gives you a way to know who has been informed
about what
Provides a way to tell how many bestpath changes your
network is experiencing
You have 150k routes and see the table version increase by 150k every
minute something is wrong
You have 150k routes and see the table version increase by 300 every
minute sounds like normal network churn
You should monitor the table version in your network to

determine what is normal for you
If the table version is increasing rapidly then that could
explain why BGP is busy
RST-3320
12552_04_2006_c2
Cisco Public
88
Troubleshooting BGPSummary
The more you know about a subject the easier it is to troubleshoot
Understand what BGP should be doing
Use the troubleshooting tools below to determine where the behavior is breaking
debugs
Can be useful but can also be chatty
Limit output using access-lists or RPL policy when possible
Configure no logging console to protect the CPU
show commands
Provide useful data
Low CPU impact
Show commands are your friend
Track BGP table version changes

Every bestpath change leads to a table version bump
Can be used to identify potential problems
RST-3320
12552_04_2006_c2
Cisco Public
89
The End
Questions?
Comments?
Thank you
RST-3320
12552_04_2006_c2
Cisco Public
90
Complete Your Online Session Evaluation

Win fabulous prizes; Give us your
feedback
Receive ten Passport Points for each
session evaluation you complete
Go to the Internet stations located
throughout the Convention Center to
complete your session evaluation
Drawings will be held in the
World of Solutions
Tuesday, June 20 at 12:15 p.m.
Wednesday, June 21 at 12:15 p.m.
Thursday, June 22 at 12:15 p.m. and 2:00 p.m.
RST-3320
12552_04_2006_c2
Cisco Public
91
Recommended Reading
Continue your Cisco Networkers
learning experience with further
reading from Cisco Press
Check the Recommended
Reading flyer for suggested
books
Available Onsite at the

Cisco Company Store
RST-3320
12552_04_2006_c2
Cisco Public
92
RST-3320
12552_04_2006_c2
Cisco Public
93

RST-3320 (Usa, 2006) TS BGP

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

RST-3320 (Usa, 2006) TS BGP

Uploaded by

Copyright:

Available Formats

Troubleshooting BGP

2006 Cisco Systems, Inc. All rights reserved.

2006 Cisco Systems, Inc. All rights reserved.

Understand BGP attributes

2006 Cisco Systems, Inc. All rights reserved.

This Is an Interactive Session

2006 Cisco Systems, Inc. All rights reserved.

Fast keepalives and holdtimers

BGP memory analysis

BGP table version

2006 Cisco Systems, Inc. All rights reserved.

2006 Cisco Systems, Inc. All rights reserved.

2006 Cisco Systems, Inc. All rights reserved.

FSM (Finite State Machine) enforces rules for state

2006 Cisco Systems, Inc. All rights reserved.

TCP port numbers

2006 Cisco Systems, Inc. All rights reserved.

Both sides must agree on source/destination addresses

R1 and R2 do not agree on what addresses to use

2006 Cisco Systems, Inc. All rights reserved.

R1 denies the session because of the address

2006 Cisco Systems, Inc. All rights reserved.

Routers agree on source/destination address

2006 Cisco Systems, Inc. All rights reserved.

TCPActive vs. Passive Session

Active sessionIf the TCP session initiated by R1 is the one used

2006 Cisco Systems, Inc. All rights reserved.

TCPActive vs. Passive Session

TCP open from R1 to R2s port 179 established

2006 Cisco Systems, Inc. All rights reserved.

It is possible for both sides to initiate a session at the

2006 Cisco Systems, Inc. All rights reserved.

TTL of TCP PacketsTime to Live

AS 100 TTL 2ebgp-multihop 2

BGP uses a TTL of 1 for eBGP peers

2006 Cisco Systems, Inc. All rights reserved.

TCP MSSMax Segment Size

536 bytes is inefficient for Ethernet (MTU of 1500) or POS (MTU of

ip tcp path-mtu-discovery (Cisco IOS) tcp path-mtudiscovery (Cisco IOS-XR)

Will be enabled by default for BGP sessions in the future

2006 Cisco Systems, Inc. All rights reserved.

TCP MSSMax Segment Size

BGP KAs (keepalives) are 19 bytes

2006 Cisco Systems, Inc. All rights reserved.

TCPs MD5 authentication should be used

MD5 + BTSH (BGP TTL Security Hack) provides

2006 Cisco Systems, Inc. All rights reserved.

BTSHBGP TTL Security Hack

Valid BGP Messages

Hackers spoof BGP messages to R1 as if they are R2

2006 Cisco Systems, Inc. All rights reserved.

BTSHBGP TTL Security Hack

Sender sets the TTL to 255

2006 Cisco Systems, Inc. All rights reserved.

BTSHBGP TTL Security Hack

R1 and R2 both use BTSH

Packets from R2 will have a TTL of 255

2006 Cisco Systems, Inc. All rights reserved.

BTSHBGP TTL Security Hack

2006 Cisco Systems, Inc. All rights reserved.

Single hop eBGP loopback peering does not fit

2006 Cisco Systems, Inc. All rights reserved.

Small security hole