Professional Documents
Culture Documents
BRKRST-3067
www.ciscolivevirtual.com
Agenda
Troubleshooting Catalyst 4500
Architecture
Interface Packet Drops
High CPU
IOS and IOS-XE Crashes
IOS-XE Licenses
Catalyst 4500 Series Switches
BRKRST-3067
Cisco Public
Architecture Overview
Intelligent Supervisors
Shared Packet
Memory
CPU
TCAMs1
Switching ASICs
Packet
Processor
Packet Processor
NFL2
Forwarding
Engine
Forwarding Engine
Supervisor
Specialised Hardware
Line Card
Stub ASICs
BRKRST-3067
Cisco Public
Agenda
Troubleshooting Catalyst 4500
Architecture
High CPU
IOS and IOS-XE Crashes
IOS-XE Licenses
BRKRST-3067
Cisco Public
24Gbps
46xx
3G
E Series Linecard
TX queue is full
Stub ASICs
Oversubscription
Pause frames received
48Gbps
Packet Processing
Engine
47xx
6G
E Series Linecard
Stub ASICs
BRKRST-3067
Cisco Public
Front
Panel Ports
24Gbps
Next steps
Cisco Public
PauseFramesDrop
0
BRKRST-3067
Cisco Public
Next steps
Disable flow control receive on the local interface
Disable flow control send on the peer
SUP6-E# show interfaces g2/47 flowcontrol
Port
Send FlowControl Receive FlowControl
admin
oper
admin
oper
--------- -------- -------- -------- -------Gi2/47
on
off
off
off
BRKRST-3067
RxPause TxPause
------- ------130
0
Cisco Public
interface
errors, 0
interface
errors, 0
gi1/13
CRC, 0
gi1/13
CRC, 0
| include overrun
frame, 86432 overrun, 0 ignored
| include overrun
frame, 206658 overrun, 0 ignored
PauseFramesDrop
0
Next steps
Enable flow control send on the local interface
Cisco Public
10
Agenda
Troubleshooting Catalyst 4500
Architecture
High CPU
IOS and IOS-XE Crashes
IOS-XE Licenses
BRKRST-3067
Cisco Public
11
Supervisor 6-E
CPU
TCAMs
Packet
Processor
Forwarding
Engine
Tx Queue
Memory
Telnet, SSH
Line Card
Stub ASICs
BRKRST-3067
Cisco Public
12
Cisco Public
13
Dual core
Cisco Public
14
Cisco Public
15
Next steps
Identify the event that is dropping packets so that we know what type
of packet to look for in further CPU debugging
BRKRST-3067
Cisco Public
16
Cisco Public
17
Next steps
Is there a pattern
and is this legitimate traffic?
2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
BRKRST-3067
18
Agenda
Troubleshooting Catalyst 4500
Architecture
High CPU
IOS and IOS-XE Crashes
IOS-XE Licenses
BRKRST-3067
Cisco Public
19
BRKRST-3067
Cisco Public
20
BRKRST-3067
Cisco Public
21
(not supported)
Cisco Public
22
Agenda
Troubleshooting Catalyst 4500
Architecture
High CPU
IOS and IOS-XE Crashes
IOS-XE Licenses
BRKRST-3067
Cisco Public
23
Feature Sets
Entservices
Entservices
Ipbase
Lanbase
Ipbase
Lanbase
IPbase
Lanbase
BRKRST-3067
Entservices
Ipbase
Lanbase
Lanbase
Cisco Public
24
BRKRST-3067
Cisco Public
25
license
License Store: Dynamic License Storage
StoreIndex: 0 Feature: entservices Version: 1.0
License Type: Evaluation
Evaluation total period: 8 weeks 4 days
Evaluation period left: 4 weeks
3 days
License State: Inactive
Temporary License
BRKRST-3067
Cisco Public
26
BRKRST-3067
Cisco Public
27
BRKRST-3067
Cisco Public
28
BRKRST-3067
Cisco Public
29
Agenda
Sup720 Architecture
Sup720 Layer 2 and Layer 3 Unicast Troubleshooting
BRKRST-3067
Cisco Public
31
Layer3 Control-plane
E.g., OSPF, BGP, SNMP
Replication engine
E.g., Multicast, SPAN
L3/L4 forwarding
SFP /
SFP GETX
MSFC 3
Flash
DRAM
Flash
DRAM
ACE
Counter
RP
CPU
1 Gbps
SP
CPU
1 Gbps
Port ASIC
L3/4 Engine
MET
Switch Fabric
20 Gbps
16 x 20G Traces
Traces # 1 to 16
Integrated Switch
Fabric
BRKRST-3067
QoS
Adj
FIB ACL
NetFlow TCAM TCAM TCAM
TCAM
EOBC
Fabric Interface
and
Replication Engine
L2 forwarding
L2 Engine
PFC3
DBUS
RBUS
L2 CAM (64K)
16 Gbps Bus
Cisco Public
32
Agenda
Sup720 Architecture
Sup720 Layer 2 and Layer 3 Unicast Troubleshooting
BRKRST-3067
Cisco Public
33
Port
ASIC
Layer 2 Layer 2
Engine
Engine
L3/4
DFC3 Engine
Port
ASIC
Fabric
Interface &
Replication
Engine
WS-X6704
Module 8
MET
Switch Fabric
MET
Fabric
Interface &
Replication
Engine
Port
ASIC
Port
ASIC
Layer 2 Layer 2
Engine
Engine
L3/4
DFC3 Engine
Fabric
Interface &
Replication
Engine
Port
Port
ASIC ASIC
WS-X6748
Module 7
MET
Cisco Public
34
Entries are stored based on result of hash done on MAC and VLAN
MAC learning is done per PFC or DFC
Each PFC/DFC maintains separate L2 CAM table
PFC and DFCs age entries independently
Refreshing of entries based on seeing traffic from specific host
New learns on one forwarding engine communicated to other
engines via MAC-Sync process
MAC table size:
64K entries on PFC3A / 3B / 3BXL and DFC3A / 3B / 3BXL
96K entries on PFC3C / 3CXL and DFC3C / 3CXL
BRKRST-3067
Cisco Public
35
BRKRST-3067
ARPA
Cisco Public
Te8/3
0011.bced.e400
Te8/3
Type
Po1
Cat6K
Vlan700
Host1
7.0.1.1
Gig7/4
Hardware Addr
Vlan700
Te8/1
ARPA
Gig8/1
000b.fca2.fe0a
Gig7/3
Age (min)
Type
Gig8/2
Hardware Addr
Gig7/5
Age (min)
Gig8/3
Host2
7.0.1.2
Te8/1
R2
Po2
Po1
R1
36
L2 Unicast Traffic
Te8/3
Te8/3
Host2
7.0.1.2
Te8/1
Te8/1
R2
vlan
mac address
type
learn
age
ports
------+----------------+--------+-----+----------+-------------------
dynamic
Yes
170
Po2
vlan
mac address
type
learn
age
BRKRST-3067
dynamic
Yes
170
Po2
ports
Gig7/4
Cat6K
Gig8/1
Po2
Gig7/3
50
Gig8/2
Yes
Gig7/5
dynamic
Gig8/3
Po1
Po1
Cisco Public
Host1
7.0.1.1
Po1
R1
37
L2 Unicast Traffic
BRKRST-3067
Cisco Public
Te8/1
Te8/3
Te8/3
Po1
Host1
7.0.1.1
Gig7/3
Gig7/4
Gig8/2
Gig8/1
Cat6K
Gig7/5
Primary
Entry
Host2
7.0.1.2
Gig8/3
Te8/1
R2
Po2
Po1
R1
38
Te8/3
Host2
7.0.1.2
Te8/3
R2
Te8/1
Te8/1
L2 Unicast Traffic
dst-ip
mpls label-ip
Cat6K
BRKRST-3067
Gig7/3
Gig7/4
Gig8/2
Gig8/1
Gig7/5
Gig8/3
MPLS: Label or IP
R1#show etherchannel load-balance hash-result interface po1 ip 7.0.1.2
Po1
Po2
Cisco Public
Host1
7.0.1.1
Po1
R1
39
Layer 2 Layer 2
Engine
Engine
L3/4
DFC3 Engine
Port
Port
ASIC
ASIC
Fabric
Interface &
Replication
Engine
WS-X6704
Module 8
MET
Switch Fabric
MET
Fabric
Interface &
Replication
Engine
Port
ASIC
BRKRST-3067
Port
ASIC
Layer 2 Layer 2
Engine
Engine
L3/4
DFC3 Engine
Fabric
Interface &
Replication
Engine
Port
ASIC
Port
ASIC
Cisco Public
WS-X6748
Module 7
MET
FIB/Adjacency Tables
L3 FIB Table Programming Flow
show ip route (RIB)
Verify Layer 3
IOS Routing Table (RP)
show ip arp
show ip cef
remote
command
module <mod>
show ip cef
show ip cef
adjacency
remote
command
module <mod>
show adjacency
detail
41
L3 Unicast Traffic
Cat6K
Cisco Public
Gig7/3
Gig7/4
Gig8/1
Gig8/2
BRKRST-3067
Te8/3
Te8/3
SW
Gig7/5
Vlan702
Vlan705
Vlan704
Vlan703
Vlan701
Te8/1
Te8/1
Host2
9.0.1.2
Gig8/3
R2
Po2
HW
Host1
8.0.1.1
Po1
R1
42
L3 Unicast Traffic
Te8/3
Te8/3
Te8/1
Host2
9.0.1.2
Te8/1
R2
Cat6K
BRKRST-3067
Cisco Public
Host1
8.0.1.1
Gig7/5
Gig7/3
Gig7/4
Gig8/2
Gig8/1
Gig8/3
R1
43
L3 Unicast Traffic
SW
Host2
9.0.1.2
Te8/3
R2
Te8/3
Te8/1
9.0.0.0/8
Te8/1
7.6.1.2/32
Cat6K
attached to TenGigabitEthernet8/1
9.0.0.0/8
Cat6K#show
Te8/1
Te8/3
Gig7/4
108749 9.0.0.0/8
Adjacency
Gig8/1
Prefix
Gig7/3
Index
HW
Gig8/2
Gig7/5
Gig8/3
Interface: Te8/3, Next Hop: 7.7.1.2, Vlan: 1090, Destination Mac: 000f.f8e4.d000
BRKRST-3067
Cisco Public
Host1
8.0.1.1
R1
44
L3 Unicast Traffic
Cat6K#show adjacency ten 8/3 7.7.1.2 detail
Protocol Interface
Address
IP
7.7.1.2(17)
TenGigabitEthernet8/3
Aggregate HW adjacency
statistics (SW collects it from all
DFC/PFCs for all prefixes linked
to this adjacency)
Rewrite information
(Dmac|Smac|0800): verify it is
conform with next hop rewrite info
Encap length 14
000FF8E4D0000050F0F874000800
ARP
RVTEN - RPF Vlan table enable, RVTSEL - RPF Vlan table select
Format: IPV4_DA - (8 | xtag vpn pi cr recirc tos prefix)
Format: IPV4_SA - (9 | xtag vpn pi cr recirc prefix)
M(108749 ): E | 1 FFF
0 0 0 0
255.0.0.0
V(108749 ): 8 | 1 0
0 0 0 0
9.0.0.0
BRKRST-3067
(A:294933 ,P:1,D:0,m:14,B:0)
Cisco Public
45
L3 Unicast Traffic
Cat6K#show mls cef adjacency entry 294933 to 294947 mod 7
Index: 294933
Index: 294947
294933
294934
294935
294936
294937
294938
294939
packets: 0, bytes: 0
packets: 0, bytes: 0
packets: 0, bytes: 0
packets: 2001, bytes: 236118
packets: 0, bytes: 0
packets: 0, bytes: 0
packets: 0, bytes: 0
packets: 0, bytes: 0
BRKRST-3067
Cisco Public
46
Agenda
Sup720 Architecture
Sup720 Layer 2 and Layer 3 Unicast Troubleshooting
BRKRST-3067
Cisco Public
47
MSFC 3
Flash
DRAM
RP
CPU
Flash
SP
CPU
DRAM
1 Gbps
Inband
C
Port ASIC
C
1 Gbps
Inband
Sup720
SP: show ibc
SP: show process cpu
BRKRST-3067
Cisco Public
48
Next Steps
Process: recurring events, control plane process etc.
Interrupts: incorrect switching path, system exceeding hardware
resources
BRKRST-3067
Cisco Public
49
BRKRST-3067
Cisco Public
50
Broadcast Storm
Traffic with IP-Options enabled
Traffic to which ICMP redirect or Unreachable required e.g., TTL=1, ACL Deny
Traffic that needs further CPU processing e.g., ACL Logging
Consider Optimised ACL
Logging (OAL)
BRKRST-3067
Cisco Public
51
Next steps
Check if any debug is enabled via show debug. Issue undebug all if it is
not needed
Disable logging by no logging console or no logging terminal
BRKRST-3067
Cisco Public
52
Enc
1
1
Flags
200
200
Input
Vl100
Vl100
Output
None
None
Cisco Public
Packet details
53
Cisco Public
54
Cisco Public
Source/Destination
55
Agenda
Sup720 Architecture
BRKRST-3067
Cisco Public
56
Switch 1
BRKRST-3067
Switch 2
Cisco Public
57
BRKRST-3067
Cisco Public
58
Cisco Public
Switch 1 is active,
switch 2 is standby,
both are up
Te1/3/6(P)
Te2/2/8(P)
Switch 1
side of the
VSL
Switch 2
side of the
VSL
59
Cisco Public
60
Indicates
problem
sending LMP
packet to the
VSL peer
Rx error details
Interface My info
My info
Bad MAC
Bad switch Domain id
Peer info
mismatch
absent
Address
id
mismatch
mismatch
------------------------------------------------------------------------------Te1/5/4
0
1
0
0
0
0
Te1/3/3
0
0
0
0
0
0
These errors usually indicate a
Te1/3/4
0
0
0
0
0
0
misconfiguration on one of the peers
Te1/3/6
0
0
0
0
0
0
1 unidirectional packet when
first link in VSL came up
BRKRST-3067
Cisco Public
61
Cisco Public
62
Gig5/9
Gig4/16
Gig2/2
Gig5/2
Gig2/4
Physical
BRKRST-3067
Host1
8.0.1.1
R1
2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Host1
8.0.1.1
Ten1/1
Ten1/4
Ten2/2/7
Cat6K
Gig2/6/12
Gig1/9/36
Gig2/9/15
Gig1/6/2
Gig1/5/1
Logical
VSL
Gig5/9
Gig4/16
Gig2/2
Gig5/2
Gig2/4
Host2
9.0.1.2
Host2
9.0.1.2
Ten1/3/2
R2
R2
Po2
Po1
R1
63
MEC
Cisco Public
64
9.0.1.2
Cisco Public
65
Looking at the HW table shows next hop directly attached to local switch is preferred
Cat6K#show mls cef lookup 9.0.1.0 switch 1 mod 3
Packet coming in on switch 1 module 3, for 9.0.0.0/8
Codes: decap - Decapsulation, + - Push Label
prefers next hop attached to local switch id 1
Index Prefix
Adjacency
108775 9.0.0.0/8
Te1/3/2
, 000f.35ed.7c00
Cat6K#show mls cef lookup 9.0.1.0 switch 2 mod 2
Packet coming in on switch 2 module 2, for 9.0.0.0/8
Codes: decap - Decapsulation, + - Push Label
prefers next hop attached to local switch id 2
Index Prefix
Adjacency
108775 9.0.0.0/8
Te2/2/7
, 000f.35ed.7c00
Cat6K#show mls cef exact-route 8.0.1.1 0 9.0.1.2 0 switch 1 mod 3
Interface: Te1/3/2, Next Hop: 7.6.1.2, Vlan: 4064, Destination Mac: 000f.35ed.7c00
Cat6K#show mls cef exact-route 8.0.1.1 0 9.0.1.2 0 switch 2 mod 2
Interface: Te2/2/7, Next Hop: 7.7.1.2, Vlan: 4056, Destination Mac: 000f.35ed.7c00
BRKRST-3067
Cisco Public
66
Im active
Im active
VSL
Standby
BRKRST-3067
Cisco Public
67
Enhanced PAgP
Uses new Type-Length-Value (TLV) field in PAgP packets. contains the ID of
the active switch
Requires Multi-chassis EtherChannel (MEC) switch support ePAgP
BFD
Uses a dedicated connection between VSL peers
Requires IP addresses in different subnet on the interfaces in the BFD link
BRKRST-3067
Cisco Public
68
Logical
VSL
Physical
Po1
Po2
Po1
Ten1/4
Ten2/2/7
R1
R1
BRKRST-3067
Gig1/6/1
Gig5/9
Gig4/16
Gig2/2
Gig5/2
Gig2/4
Gig5/9
Gig4/16
Gig2/2
Gig5/2
Gig2/4
Po2
Cat6K
Gig2/6/12
Gig1/9/36
Gig2/9/15
Gig1/6/2
Gig1/5/1
Gig1/6/1
Gig2/9/1
Fast Hello
Ten1/1
R2
Ten1/3/2
R2
Cisco Public
69
Cisco Public
70
Partner
Partner
Partner
Port
Detect Capable
Name
Port
Version
Gi1/5/1
Yes
R1
Gi2/4
1.1
Gi1/6/2
Yes
R1
Gi5/2
1.1
BRKRST-3067
Cisco Public
71
SW1
SW2
Active
Standby
Gig1/6/1
Gig2/9/1
Fast Hello
Po2
I lost my VSL.
Let me take over as Active!
ePAgP
Gig5/9
Gig4/16
Gig2/2
Gig5/2
Gig2/4
Po1
R1
BRKRST-3067
Cisco Public
72
Enhanced PAGP
detected both switch ids
were active at the same
time
Cisco Public
73
Switch Status
Preempt
Priority
Role
Number
Oper(Conf) Oper(Conf)
Session ID
Local
-----------------------------------------------------------------LOCAL
UP
TRUE (Y*)
200(200)
ACTIVE
Switch
Switch Status
Preempt
Priority
Role
Number
Oper(Conf) Oper(Conf)
Session ID
Local
Remote
-----------------------------------------------------------------LOCAL
UP
TRUE (Y*)
100(100)
ACTIVE
Cisco Public
*Apr 1 12:49:52.140 CET: %VSLP-SW2_SP-5-RRP_MSG: Peer Switch with unsaved configurations needs
to be reloaded. Please save relevant configurations on the peer switch and reload it.
Cisco Public
75
Agenda
Sup720 Architecture
Sup720 Layer 2 and Layer 3 Unicast Troubleshooting
BRKRST-3067
Cisco Public
76
NetFlow
Hash &
Data
tables for
stats &
features
1GE /
10GE
Uplinks
MSFC 5
Central
Management
Processor
NetFlow
2 x 1 Gbps
LIF Map
contains port
mappings
CL1
ADJ
FIB
CL2
TCAM TCAM TCAM TCAM
Port ASIC
CPU
DRAM
CL1 TCAM
contains
Security &
Feature ACL
entries
LIF
MAP
L3/4 Engine
RPF
Table
Flash
MET
Switch Fabric
26 x 40G Traces
Switch Fabric
Traces # 1 to 26
BRKRST-3067
40 Gbps
EOBC
Fabric Interface
&
Replication Engine
DBUS
RBUS
LIF Table
LIF Stats
L2 Engine
L2 CAM (128K)
PF
C4
ACE
Counter
Cisco Public
Logical
Interface (LIF)
Table contains
entries
Reverse Path
Forwarding info
Hardware
for ACL
TCAM
counters
L2 CAM
contains
MAC
entries
77
Sup2T
Control-Plane
Data-Plane
BRKRST-3067
Dual Core
Processor
MSFC 3
MSFC 5
PFC3
PFC4
Cisco Public
78
Sup2T Overview
Integrated with Control Management Processor (CMP)
USB-based console and flash support
Key improvement: Single-Cycle Input & Output Processing
Has superior Control-Plane protection.
Supports all Sup720 features, including VSS
BRKRST-3067
Cisco Public
79
Agenda
Sup720 Architecture
Sup720 Layer 2 and Layer 3 Unicast Troubleshooting
BRKRST-3067
Cisco Public
80
R2
.2
Gig3/25
.5
.6
Gig3/26
Sup2T
BRKRST-3067
Gig3/1
192.168.100.4/30
.1
192.168.100.0/30
Host2 Gig3/2
192.168.200.9
.1
Gig3/1
Cisco Public
Host1
192.168.200.5
Gig1/26
192.168.25.4/30
.2
192.168.25.0/30
Gig1/25
.6
.5
Gig3/2
R1
81
192.168.200.8/30
R2
.1
.2
192.168.100.0/30
Host2 Gig3/2
192.168.200.9
Gig3/1
192.168.100.4/30
Gig3/25
.5
.6
Gig3/26
Sup2T
attached to GigabitEthernet3/25
192.168.200.8/30
nexthop 192.168.100.1 GigabitEthernet3/25
Index Prefix
219265 192.168.200.8/30
Adjacency
Gi3/25
Gi3/26
.2
.1
Gig3/1
192.168.200.9 0
Interface: Gi3/26, Next Hop: 192.168.100.5, ifnum: 0x7E, Destination Mac:
001d.e656.cc00
LIF: 0x2000411B
BRKRST-3067
Cisco Public
Host1
192.168.200.5
Gig1/26
192.168.25.4/30
Gig1/25
HW
192.168.25.0/30
.6
.5
Gig3/2
R1
82
R2
SW
.2
Cisco Public
.6
Gig3/26
.2
Host1
192.168.200.5
Gig1/26
192.168.25.4/30
Gig1/25
Gig3/1
.5
Sup2T
.1
BRKRST-3067
Gig3/1
Gig3/25
192.168.25.0/30
.1
192.168.100.0/30
Host2 Gig3/2
192.168.200.9
192.168.100.4/30
.6
.5
Gig3/2
R1
83
R2
HW
.2
Gig3/25
Codes: M - mask entry, V - value entry, A - adjacency index, NR- no_route bit
LS - load sharing count, RI - router_ip bit, DF: default bit
CP - copy_to_cpu bit, AS: dest_AS_number, DGTv - dgt_valid bit
DGT: dgt/others value
Format:IPV4 (valid class vpn
prefix)
M(54794 ):
1
F
3FFF 255.255.255.255
V(54794 ):
1
0
0
192.168.100.5
(A:376833, LS:0, NR:0, RI:0, DF:0 CP:0 DGTv:1,
DGT:0)
.2
.1
Gig3/1
Cisco Public
.6
Gig3/26
Gig1/25
BRKRST-3067
.5
Sup2T
192.168.25.0/30
Gig3/1
192.168.100.4/30
.1
192.168.100.0/30
Host2 Gig3/2
192.168.200.9
Host1
192.168.200.5
Gig1/26
192.168.25.4/30
.6
.5
Gig3/2
R1
84
R2
.2
Gig3/1
192.168.100.4/30
.1
192.168.100.0/30
Host2 Gig3/2
192.168.200.9
Gig3/25
.5
.6
Gig3/26
Sup2T
.2
.1
192.168.25.0/30
Gig1/25
Gig3/1
Gig1/26
192.168.25.4/30
.6
.5
Gig3/2
. . . Continued . . .
BRKRST-3067
Cisco Public
Host1
192.168.200.5
R1
85
BRKRST-3067
Cisco Public
192.168.100.4/30
.2
Gig3/1
Gig3/25
.5
.6
Gig3/26
Sup2T
Gig1/25
.2
Rewrite MAC info
.1
Gig3/1
Host1
192.168.200.5
Gig1/26
192.168.25.4/30
.1
192.168.100.0/30
Host2 Gig3/2
192.168.200.9
Continued . . .
192.168.25.0/30
. .
R2
.6
.5
Gig3/2
R1
86
Session Summary
Understanding of the architecture of Sup6-E Sup7-E, Sup-720 and
Sup-2T
Input and output buffers of 4500 line cards
Tools to troubleshoot high CPU
Understanding of the IOS-XE architecture
Verifying L2 and L3 forwarding path through Sup-720 and Sup-2T
Troubleshooting all that is VSS
..Ensuring packets get to their destination as efficiently as possible
BRKRST-3067
Cisco Public
87
Q&A
BRKRST-3067
89
BRKRST-3067
Cisco Public
90
Appendix
Catalyst 4500
Command Summaries
Chassis, Supervisor, Line Card comparison
HA Redundancy
ISSU
Unicast Forwarding
Catalyst 6500
GOLD/ EEM Health Monitoring
BRKRST-3067
Cisco Public
91
Commands
show module
Verify PoE line card supports enough power per port, per slot
Verify whether problem persists after changing ports, stub ASICs, and/or line
cards
BRKRST-3067
Cisco Public
92
Commands
BRKRST-3067
Cisco Public
93
Troubleshooting Licensing
Command Summary
Troubleshooting Steps
Commands
BRKRST-3067
Cisco Public
94
Troubleshooting QoS
Command Summary
Troubleshooting Steps
Commands
show running-config
BRKRST-3067
Cisco Public
95
Commands
show platform hardware flow label
show platform hardware flow table label <value>
BRKRST-3067
Cisco Public
96
Appendix
Catalyst 4500
Command Summaries
Chassis, Supervisor, Line Card comparison
HA Redundancy
ISSU
Unicast Forwarding
Catalyst 6500
GOLD/ EEM Health Monitoring
BRKRST-3067
Cisco Public
97
4503-E
4507R+E
4510R+E
4506-E
See the appendix for supervisor, line card, and chassis product and compatibility details.
BRKRST-3067
Cisco Public
98
Catalyst 4948E
Catalyst 4900M
Catalyst 4948-10GE
Catalyst 4948
BRKRST-3067
Cisco Public
99
Supervisor Comparison
Layer 2/3/4
Services
Supervisor 7-E
Supervisor 6-E
Supervisor 6L-E
Bandwidth3 &
Throughput
Full L2/3/4,
enhanced routing1,
IPv6
848 Gbps
Full L2/3/4,
enhanced routing1,
IPv6
320 Gbps
280 Gbps
Chassis Support
225 mpps
Supervisor V10GE
Full L2/3/4,
enhanced routing,
IPv62
136 Gbps
Supervisor V
96 Gbps
Uplinks
NetFlow
-E/+E Chassis
E-series and
classic
4x10GE
Yes
E-Series &
Classic
4 x GE or 2 x 10
GE (TwinGig)
No
E-Series &
Classic
2 x 10 GE, 4 x GE,
10 GE and 2 x GE,
or 2 x GE and
10GE
No
Classic
4 x GE and 2 x
10GE
Yes
Classic
2 x GE
Optional
Classic
2 x GE
Optional
See Sup IV
Classic
4 x GE and 2 x
10GE
No
See Sup IV
Classic
2 x GE
No
4503-E 4503
Classic
8 x GE & 12
10/100/1000 PoE
No
250 mpps
250 mpps
Line Cards
102 mpps
72 Mpps
Supervisor IV
64 Gbps
48 mpps
Supervisor IIPlus-10GE
Supervisor IIPlus
Supervisor IIPlus-TS
1. EIGRP,
BRKRST-3067
108 Gbps
81 mpps
64 Gbps
48 mpps
64 Gbps
48 mpps
3. Bi-directional.
Cisco Public
100
Supervisor 7-E
256,000
128,000
12,000
Supervisor 6-E
256,000
128,000
12,000
Supervisor 6L-E
57,000
32,000
3072
Supervisor V10GE
128,000
64,000
6,000
Supervisor V
128,000
64,000
3,000
Supervisor IV
128,000
64,000
3,000
Supervisor IIPlus-10GE
32,000
32,000
3,000
Supervisor II-Plus
32,000
32,000
3,000
Supervisor IIPlus-TS
32,000
32,000
3,000
BRKRST-3067
Cisco Public
101
Line Card
BRKRST-3067
Speed
Port Type
Wire-rate or
Oversubscribed?
WS-4748-UPOE+E
48
10/100/1000
RJ45
Wire-rate
WS-4748-RJ45V+E
48
10/100/1000
RJ45
Wire-rate
12
10GBASE-X
SFP+
2.5:1 oversubscribed
WS-X4648-RJ45-E
48
10/100/1000
RJ45
2:1 oversubscribed
WS-X4640-CSFP-E
80
1000BASE-X
CSFP
4:1 oversubscribed
WS-X4624-SFP-E
24
1000BASE-X
SFP
Wire-rate
WS-X4612-SFP-E
12
1000BASE-X
SFP
Wire-rate
WS-X4712-SFP+E
E-Series
Ports
Cisco Public
102
Classic FE
over Fibre
Classic FE
over Copper
Classic FE
PoE
BRKRST-3067
Line Card
Ports
Speed
Port Type
Wire-rate or
Oversubscribed?
WS-X4648-RJ45V-E
48
10/100/1000
RJ45
2:1 oversubscribed
WS-X4648-RJ45V+E
48
10/100/1000
RJ45
2:1 oversubscribed
WS-X4606-X2-E
10GBASE-X
X2 or SFP w/
TwinGig
WS-X4248-FE-SFP
48
1000BASE-X
SFP
Wire-rate
WS-X4124-FX-MT
24
100BASE-FX
MM MT-RJ
Wire-rate
WS-X4148-FX-MT
48
100BASE-FX
MM MT-RJ
Wire-rate
WS-X4148-FE-BD-LC
48
100BASE-BX10-D
SMF Single LC
Wire-rate
WS-X4124-RJ45
24
10/100
RJ45
Wire-rate
WS-X4148-RJ
48
10/100
RJ45
Wire-rate
WS-X4148-RJ21
48
10/100
RJ45
Wire-rate
WS-X4224-RJ45V
24
10/100
RJ45
Wire-rate
WS-X4248-RJ45V
48
10/100
RJ45
Wire-rate
WS-X4248-RJ21V
48
10/100
RJ21
Wire-rate
Cisco Public
103
Classic GE
over Copper
Classic GE
over Copper
PoE
BRKRST-3067
Line Card
Ports
Speed
Port Type
Wire-rate or
Oversubscribed?
WS-X4302-GB
1000BASE-X
GBIC
Wire-rate
WS-X4306-GB
1000BASE-X
GBIC
Wire-rate
WS-X4418-GB
18
1000BASE-X
GBIC
2 ports Wire-rate
4:1 oversubscribed
WS-X4448-GB-LX
48
1000BASE-LX
SFP
8:1 oversubscribed
WS-X4448-GB-SFP
48
1000BASE-X
SFP
8:1 oversubscribed
WS-X4506-GB-T
6+6
10/100/1000 and
1000BASE-X
Wire-rate
WS-X4424-GB-RJ45
24
10/100/1000
RJ-45
4:1 oversubscribed
WS-X4448-GB-RJ45
48
10/100/1000
RJ-45
8:1 oversubscribed
WS-X4548-GB-RJ45
48
10/100/1000
RJ-45
8:1 oversubscribed
WS-X4524-GB-RJ45V
24
10/100/1000
RJ-45
4:1 oversubscribed
WS-X4548-GB-RJ45V
48
10/100/1000
RJ-45
8:1 oversubscribed
Cisco Public
16 ports
104
Catalyst 4900M
Catalyst 4948-10GE
Catalyst 4948
Bandwidth1
176 Gbps
320 Gbps
136 Gbps
96 Gbps
Throughput
102 mpps
72 mpps
IPv6
In hardware
In hardware
In software
In software
Height
1 RU
2 RU
1 RU
1RU
48
40
48
48
24
32 with TwinGig
17.5MB
16MB
16MB
16MB
VLANs
4096
4096
2048
2048
Multicast entries
128,000
32,000
32,000
MAC addresses
55,000
55,000
55,000
32,000
1. Bi-directional.
BRKRST-3067
Cisco Public
105
Description
Mitigation
CrcAlign-Err
Swap out cable with known good cable. Test with known good switch
port. Test with known good NIC card or other peer port. Check for
duplex mismatch.
Symbol-Err
Swap out cable with known good cable. Test with known good switch
port. Test with known good NIC card or other peer port.
Undersize
Oversize
Fragments
Swap out cable with known good cable. Test with known good switch
port. Test with known good NIC card or other peer port.
Single-Col
Normal for half-duplex ports. Abnormal for full-duplex ports. Check for
duplex mismatch. Check if link is over-utilized.
Multi-Col
Normal for half-duplex ports. Abnormal for full-duplex ports. Check for
duplex mismatch. Check if link is over-utilized.
Late-Col
BRKRST-3067
Cisco Public
106
Description
Mitigation
Faulty
Authfail
Offline
Occurs when switch brings the line card online. This should be a transient state.
In Reset
PwrOver
Determine if connected devices are receiving the right amount of inline power according to
show power inline. If correct, disconnect phones one at a time, noting the inline power
utilization change. If one or more devices trigger a change greater than what is listed in show
power inline, those devices may be faulty.
PwrMax or
PwrFault
Determine if connected devices are receiving the right amount of inline power according to
show power inline. If correct, disconnect phones one at a time, noting the inline power
utilization change. If one or more devices trigger a change greater than what is listed in show
power inline, those devices may be faulty.
PwrDeny
BRKRST-3067
Cisco Public
107
Appendix
Catalyst 4500
Command Summaries
Chassis, Supervisor, Line Card comparison
HA Redundancy
ISSU
Unicast Forwarding
Catalyst 6500
GOLD/ EEM Health Monitoring
BRKRST-3067
Cisco Public
108
SSO
12.2(12c)EW onward
12.2(20)EWA onward
ISSU
NSF
Requires SSO
Requires SSO
109
Dropped Tx packets: 5
Internal error count: 0
Interrupt count: 35055
Link events: 1
Count of Phy polls: 35327
Count of postprocessing: 35042
Phy status: 100Mb Half
BRKRST-3067
Continuously incrementing?
Initiate a switchover to reset
EOBC MAC
Cisco Public
110
Troubleshooting SSO
Sanity Check
Must Match
Should Match
Supervisor type
Rommon version
Supervisor Hw revision
Memory1
Boot variables
Optical modules
IOS version2
May Match
Configuration register
value
Model
Serial No.
---+-----+--------------------------------------+------------------+----------1
2
Mod
WS-X4516-10GE
JAB09160071
WS-X4516-10GE
JAE1008W6KF
Redundancy role
Operating mode
Redundancy status
----+-------------------+-------------------+---------------------------------1
Standby Supervisor
SSO
Standby hot
Active Supervisor
SSO
Active
1. Supervisors II-Plus and II-Plus-10GE can be configured with 256 or 512 MB. Use show version to verify capacity.
2. Different IOS versions accepted (and expected!) during software upgrade or downgrade using ISSU (12.2(37)SG and beyond).
BRKRST-3067
Cisco Public
111
Model
Serial No.
---+-----+--------------------------------------+------------------+----------1
2
WS-X4516-10GE
JAB09160071
Supervisor
M MAC addresses
Hw
Fw
Sw
Supervisor present,
Status
but in ROMMON
12.2(37)SG
Ok
2 Unknown
Unknown
Other
Mod
Redundancy role
Unknown
Operating mode
Redundancy status
----+-------------------+-------------------+---------------------------------1
Active Supervisor
SSO
Active
Standby Supervisor
SSO
Disabled
Catalyst-4507#
BRKRST-3067
Cisco Public
112
Standby failures = 1
Last switchover reason = none1
Hardware Mode = Duplex
Configured Redundancy Mode = Stateful Switchover
Operating Redundancy Mode = Stateful Switchover
Maintenance Mode = Disabled
If mismatched, super-visor
type, IOS version, or
memory do not match
If Down, check if standby supervisor is
present and in ROMMON
Communications = Up
Cisco Public
113
Match!
BRKRST-3067
Cisco Public
114
BRKRST-3067
Cisco Public
115
%C4K_REDUNDANCY-2-IOS_VERSION_CHECK_FAIL: IOS version mismatch. Active supervisor version is 12.2(37)SG. Standby supervisor version is
12.2(31)SG. Redundancy feature may not work as expected.
4 %C4K_REDUNDANCY-6-MODE:
5
6
1
BRKRST-3067
Cisco Public
116
NSF-capable router
NSF-capable router
RP Switchover!
Route
updates
Route
updates
Route
updates
Supervisor
Switchover!
Catalyst 4510R
NSF-aware switch
L3 traffic flowing
Catalyst 4510R
NSF-aware switch
L3 traffic flowing
Catalyst 4510R
NSF-capable switch
L3 traffic flowing
Supervisor switchover
Supervisor switchover
Cisco Public
Supervisor
Switchover!
Router forwards
routing in
updates
NSF-capability
introduced
12.2(31)SG
for
classic
and 12.2(44)SG
for
No
routesupervisors
flaps! L3 traffic
flowing!
Supervisor 6-E.
117
Troubleshooting NSF
Sanity Check
Is NSF configured?
show running-config | begin router
Cisco Public
118
Troubleshooting NSF
Digging Deeper: OSPF
Switch-1# show running-config | begin router
router ospf 1
log-adjacency-changes
nsf
network 10.0.0.0 0.255.255.255 area 0
network 20.0.0.0 0.255.255.255 area 0
Switch-1
Switch-2
yes
BRKRST-3067
enabled/running
Cisco Public
119
Appendix
Catalyst 4500
Command Summaries
Chassis, Supervisor, Line Card comparison
HA Redundancy
ISSU
Unicast Forwarding
Catalyst 6500
GOLD/ EEM Health Monitoring
BRKRST-3067
Cisco Public
120
ISSU Overview
Performing In-Service Software Upgrades
ACTIVE
OLD
STANDBY
OLD
BRKRST-3067
load
version
ACTIVE
OLD
run
version
STANDBY
NEW
Cisco Public
STANDBY
OLD
commit
version
STANDBY
NEW
ACTIVE
ACTIVE
NEW
NEW
121
Troubleshooting ISSU
Sanity Check
Is autoboot configured?
show bootvar
Configuration register ends in 2
Cisco Public
122
Troubleshooting ISSU
Interpreting Error Messages
issu loadversion
issu runversion
issu acceptversion
Cisco Public
123
BRKRST-3067
Cisco Public
124
Troubleshooting ISSU
Debugging Supervisor Sync Failures
Occurs when command is not supported by standbys IOS version
Results in RPR mode
Unconfigure mismatched commands to synchronise supervisors and
return to SSO mode
Catalyst-4507# issu loadversion 1 bootflash:cat4500-ipbase-mz.122-31.SGA3 2
slavebootflash:cat4500-ipbase-mz.122-31.SGA3
5 23:29:33.127: Config Sync: Bulk-sync failure due to Servicing Incompatibility. Please check full list of mismatched commands via:
interface GigabitEthernet7/1
! <submode> "interface"
- dot1x host-mode multi-domain
! </submode> "interface
BRKRST-3067
Cisco Public
125
Appendix
Catalyst 4500
Command Summaries
Chassis, Supervisor, Line Card comparison
HA Redundancy
ISSU
Unicast Forwarding
Catalyst 6500
BRKRST-3067
Cisco Public
126
show
show
show
show
show
Si
Si
show
show
show
show
show
show
interface
ip route
arp
ip cef
adjacency
platform hardware ip route [ipv4] network/host
Si
Si
Access
L2
Distribution
L2/L3
Core
L3
Distribution
L2/L3
Access
L2
172.16.1.1
1. Cisco Catalyst 4900M and 4948E also uses TCAM4, so you can use the same troubleshooting commands.
BRKRST-3067
Cisco Public
127
Appendix
Catalyst 4500
Command Summaries
Chassis, Supervisor, Line Card comparison
HA Redundancy
ISSU
Unicast Forwarding
Catalyst 6500
GOLD/ EEM Health Monitoring
BRKRST-3067
Cisco Public
128
Si
BRKRST-3067
Cisco Public
129
Line
Card
Fabric
Forwarding
Engine
Active Supervisor
Standby Supervisor
CPU
Line
Card
Cisco Public
130
Cisco Public
131
Boot-Up Diagnostics
Switch(config)# diagnostic bootup level complete
Runtime Diagnostics
Health-Monitoring
On-Demand
Scheduled
BRKRST-3067
Cisco Public
Cisco Public
133
Test Name
Attributes
(day hh:mm:ss.ms)
=================
000 00:00:30.00
000 00:00:15.00
not configured
not configured
not configured
not configured
not configured
not configured
not configured
not configured
not configured
not configured
not configured
not configured
not configured
not configured
not configured
000 00:00:05.00
not configured
BRKRST-3067
<snip>
Cisco Public
134
not configured
n/a
not configured
n/a
not configured
n/a
not configured
n/a
not configured
n/a
not configured
n/a
not configured
n/a
not configured
n/a
not configured
n/a
not configured
n/a
not configured
n/a
not configured
n/a
000 00:00:15.00 10
Cisco Public
135
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
----------------------------------------------------------------------------
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
2) TestLoopback:
Port
---------------------------------------------------------------------------.
BRKRST-3067
Cisco Public
136
BRKRST-3067
Cisco Public
137
On demand diagnostics:
Use as a pre-deployment tool: run complete diagnostics
before putting hardware into production environment
Use as a troubleshooting tool when suspecting
hardware failure
Si
Scheduled diagnostics:
Schedule key diagnostics tests periodically
Schedule all non-disruptive tests periodically
Health-monitoring diagnostics:
Key tests running by default
Enable additional non-disruptive tests for specific functionalities
enabled in your network: IPv6, MPLS, NAT
BRKRST-3067
Cisco Public
138
BRKRST-3067
Cisco Public
139