Nexus 7000 FabricPath

askorokh@cisco.com +7(495)789-8615
2010 Cisco and/or its affiliates. All rights reserved.

Agenda
FabricPath introduction FabricPath technical review FabricPath design considerations

2010 Cisco and/or its affiliates. All rights reserved.

FabricPath Introduction

2010 Cisco and/or its affiliates. All rights reserved.

Introducing Cisco FabricPath

An NX-OS Innovation for Layer 2 Networks

Layer 2 strengths
licity p Sim

Resilience

Flex ib

Simple configuration Flexible provisioning Low cost

Fabric Path

Layer 3 strengths
Leverage bandwidth Fast convergence Highly scalable
ility

Simplicity

Flexibility

Bandwidth

Availability

Cost

"The FabricPath capability within Cisco's NX-OS offers dramatic increases in network scalability and resiliency for our service delivery data center. FabricPath extends the benefits of the Nexus 7000 in our network, allowing us to leverage a common platform, simplify operations, and reduce operational costs. Mr. Klaus Schmid, Head of DC Network & Operating, T-Systems International GmbH its affiliates. All rights reserved. 2010 Cisco and/or
4

Architecture Flexibility Through NX-OS

Spanning-Tree vPC FabricPath
16 Switches

Active Paths Pod Bandwidth

Single Up to 10 Tbps

Dual Up to 20 Tbps

16 Way Up to 160 Tbps

Layer 2 Scalability Infrastructure Virtualization and Capacity

2010 Cisco and/or its affiliates. All rights reserved.

FabricPath Simplicity from the Outside

Multi-Domain Silos FabricPath Any App, Anywhere!

Fabric

Benefits server team by providing a network Fabric that looks like a single switch Breaks down silos, permits workload mobility, provides maximum flexibility Lowers OPEX by simplifying server team operation Reduces dependency on/interaction with network team
2010 Cisco and/or its affiliates. All rights reserved.

FabricPath Simplicty from the Inside

Benefits network team by: Reducing number of switches
Higher port density Lower oversubscription

Isolating network from the users

No impact due to topology changes Fabric can be upgraded/reconfigured live

Utilizing an open protocol

Unicast, multicast, broadcast, VLAN pruning all controlled by single control protocol Maintenance and troubleshooting equivalent to L3 network Easy to extend, providing standards-compliance with Cisco value-add

2010 Cisco and/or its affiliates. All rights reserved.

Nexus 7000 F-Series Module

First FabricPath-capable hardware platform from Cisco

Scalable 512 ports per system High-performance 320/230 Gbps (switching/ backplane), 5s latency Investment Protection Seamless Upgrade and Interoperability Standards Based TRILL and DCB support Flexible 1/10G ports auto-sensing

10K ports shipped!

Energy Efficient ~10W per 10GbE port

The F-Series modules on the Cisco Nexus 7000 series are currently deployed in LLNLs high performance computing infrastructure, offering us a high density 10GE and low latency networking solution. This technology has enabled LLNL to build large storage network fabrics to support the world class supercomputing systems vital to the laboratory's national security research and development missions
Matt Leininger, Deputy for Advanced Technology Projects at Lawrence Livermore National Laboratory

2010 Cisco and/or its affiliates. All rights reserved.

Cisco FabricPath enables faster, simpler, flatter data center networks

[W]e assessed FabricPath in terms of its ability to boost bandwidth, reroute around trouble, and simplify network management. In all three areas, FabricPath delivered [T]he switches forwarded all traffic with zero frame loss, validating FabricPath's ability to load-share across 16 redundant connections. FabricPath converges far faster than spanning tree. [T]here's no question it represents a significant advancement in the state of the networking art

http://www.networkworld.com/reviews/2010/102510-cisco-fabricpath-test.html

2010 Cisco and/or its affiliates. All rights reserved.

FabricPath Technical Review

2010 Cisco and/or its affiliates. All rights reserved.

10

FabricPath IS-IS
FabricPath IS-IS replaces STP as control-plane protocol in FabricPath network Introduces link-state protocol with support for ECMP for Layer 2 forwarding Exchanges reachability of Switch IDs and builds forwarding trees
STP BPDU STP BPDU

Improves failure detection, network reconvergence, and high availability Minimal IS-IS knowledge required no user configuration by default
Maintains plug-and-play nature of Layer 2

FabricPath IS-IS

2010 Cisco and/or its affiliates. All rights reserved.

11

Why IS-IS?
A few key reasons: Has no IP dependency no need for IP reachability in order to form adjacency between devices Easily extensible Using custom TLVs, IS-IS devices can exchange information about virtually anything Provides SPF routing Excellent topology building and reconvergence characteristics

2010 Cisco and/or its affiliates. All rights reserved.

12

FabricPath versus Classic Ethernet Interfaces

Classic Ethernet (CE) Interface Interfaces connected to existing NICs and traditional network devices Send/receive traffic in 802.3 Ethernet frame format Participate in STP domain Forwarding based on MAC table
FabricPath interface CE interface

Ethernet

Ethernet

FabricPath Header

FabricPath Interface Interfaces connected to another FabricPath device Send/receive traffic with FabricPath header No spanning tree!!! No MAC learning Exchange topology info through L2 ISIS adjacency Forwarding based on Switch ID Table
2010 Cisco and/or its affiliates. All rights reserved.

13

FabricPath versus CE VLANs

In FabricPath system, each VLAN identified as either a CE VLAN (default) or a FabricPath VLAN Only traffic in FabricPath VLANs can traverse FabricPath domain Bridging between M1 and F1 ports possible only on CE VLANs
n7k(config)# vlan 10 n7k(config-vlan)# mode ? ce Classical Ethernet VLAN mode fabricpath FabricPath VLAN mode n7k(config-vlan)# mode
2010 Cisco and/or its affiliates. All rights reserved.

VLAN Mode
CE VLAN FabricPath VLAN

M1 Ports F1 Ports
CE mode

F1 Ports
CE mode FP mode

F1 port in FabricPath VLAN can run in CE mode or in FabricPath mode!

14

Basic FabricPath Data Plane Operation

DSID20 SSID10 DMACB SMACA DSID20 SSID10 DMACB SMACA Payload

FabricPath interface CE interface

S10
Ingress FabricPath Switch

Payload

S20
Egress FabricPath Switch
Payload

DMACB SMACA Payload

SMACA DMACB

DMACB SMACA Payload

Payload SMACA DMACB

MAC A

MAC B

Ingress FabricPath switch determines destination Switch ID and imposes FabricPath header Destination Switch ID used to make routing decisions through FabricPath core No MAC learning or lookups required inside core Egress FabricPath switch removes FabricPath header and forwards to CE
2010 Cisco and/or its affiliates. All rights reserved.

15

FabricPath Encapsulation
16-Byte MAC-in-MAC Header

Classical Ethernet Frame

DMAC

SMAC

802.1Q

Etype

Payload

CRC

Original CE Frame

Cisco FabricPath Frame

6 bits Endnode ID (5:0) 1
U/L

Outer DA (48)

Outer SA (48)

FP Tag (32)

DMAC

SMAC

802.1Q

Etype

Payload

CRC (new)

1
I/G

2 bits Endnode ID (7:6)

1
RSVD

1
OOO/DL

12 bits Switch ID

8 bits Sub Switch ID

16 bits Port ID

16 bits Etype

10 bits Ftag

6 bits TTL

Switch ID Unique number identifying each FabricPath switch Sub-Switch ID Identifies devices/hosts connected via VPC+ Port ID Identifies the destination or source interface Ftag (Forwarding tag) Unique number identifying topology and/or multidestination distribution tree TTL Decremented at each switch hop to prevent frames looping infinitely
2010 Cisco and/or its affiliates. All rights reserved.

16

FabricPath MAC Table

Edge switches maintain both MAC address table and Switch ID table Ingress switch uses MAC table to determine destination Switch ID Egress switch uses MAC table (optionally) to determine output switchport
S10 S20 S30 S40

FabricPath MAC Table on S100

MAC IF/SID e1/1 e1/2 S101 S200

Local MACs point to switchports Remote MACs point to Switch IDs

A B C D

S100

S101

S200

MAC A
2010 Cisco and/or its affiliates. All rights reserved.

MAC B

MAC C

MAC D 17

show mac address-table dynamic

S100# sh mac address-table dynamic Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen,+ - primary entry using vPC Peer-Link VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID ---------+-----------------+--------+---------+------+----+-----------------* 10 0000.0000.0001 dynamic 0 F F Eth1/15 * 10 0000.0000.0002 dynamic 0 F F Eth1/15 * 10 0000.0000.0003 dynamic 0 F F Eth1/15 * 10 0000.0000.0004 dynamic 0 F F Eth1/15 * 10 0000.0000.0005 dynamic 0 F F Eth1/15 * 10 0000.0000.0006 dynamic 0 F F Eth1/15 * 10 0000.0000.0007 dynamic 0 F F Eth1/15 * 10 0000.0000.0008 dynamic 0 F F Eth1/15 * 10 0000.0000.0009 dynamic 0 F F Eth1/15 * 10 0000.0000.000a dynamic 0 F F Eth1/15 10 0000.0000.000b dynamic 0 F F 200.0.30 10 0000.0000.000c dynamic 0 F F 200.0.30 10 0000.0000.000d dynamic 0 F F 200.0.30 10 0000.0000.000e dynamic 0 F F 200.0.30 10 0000.0000.000f dynamic 0 F F 200.0.30 10 0000.0000.0010 dynamic 0 F F 200.0.30 10 0000.0000.0011 dynamic 0 F F 200.0.30 10 0000.0000.0012 dynamic 0 F F 200.0.30 10 0000.0000.0013 dynamic 0 F F 200.0.30 10 0000.0000.0014 dynamic 0 F F 200.0.30 S100#
S100 S200

S10

S20

S30

S40

po1 po2 po3 po4

2010 Cisco and/or its affiliates. All rights reserved.

18

FabricPath Routing Table

FabricPath IS-IS manages Switch ID (routing) table All FabricPath-enabled switches automatically assigned Switch ID (no user configuration required) Algorithm computes shortest (best) paths to each Switch ID based on link metrics Equal-cost paths supported between FabricPath switches
S10 S20 S30 S40

FabricPath Routing Table on S100

One best path to S10 (via L1)
Switch S10 S20 S30 S40 IF L1 L2 L3 L4 L1, L2, L3, L4 L1, L2, L3, L4 L1 L2 L3 L4

Four equal-cost paths to S101

S101 S200

2010 Cisco and/or its affiliates. All rights reserved.

S100

S101

S200

19

Building the FabricPath Routing Table

Switch S20 S30 S40 S100 S101 S200 IF L1,L5,L9 L1,L5,L9 L1,L5,L9 L1 L5 L9 L5 L1 L2 L3 L6 L4 L7 L9 L8 L10 L11 L12 Switch S10 S20 IF L4,L8,L12 L4,L8,L12 L4,L8,L12 L4 L8 L12

S10

S20

S30

S40

S30 S100 S101 S200

S100
Switch S10 S20 S30 S40 S101 S200 IF L1 L2 L3 L4 L1, L2, L3, L4 L1, L2, L3, L4

S101

S200
Switch S10 S20 S30 IF L9 L10 L11 L12 L9, L10, L11, L12 L9, L10, L11, L12

MAC A

MAC B

MAC C

MAC D

S40 S100 S101

2010 Cisco and/or its affiliates. All rights reserved.

20

show fabricpath route

S100# sh fabricpath route FabricPath Unicast Route Table 'a/b/c' denotes ftag/switch-id/subswitch-id '[x/y]' denotes [admin distance/metric] ftag 0 is local ftag subswitch-id 0 is default subswitch-id FabricPath Unicast Route Table for Topology-Default 0/100/0, number of next-hops: 0 via ---- , [60/0], 5 day/s 1/10/0, number of next-hops: 1 via Po1, [115/10], 0 day/s 1/20/0, number of next-hops: 1 via Po2, [115/10], 0 day/s 1/30/0, number of next-hops: 1 via Po3, [115/10], 2 day/s 1/40/0, number of next-hops: 1 via Po4, [115/10], 2 day/s 1/200/0, number of next-hops: 4 via Po1, [115/20], 0 day/s via Po2, [115/20], 0 day/s via Po3, [115/20], 2 day/s via Po4, [115/20], 2 day/s S100# 18:38:46, local 04:15:58, isis_l2mp-default 04:16:05, isis_l2mp-default 08:49:51, isis_l2mp-default 08:47:56, isis_l2mp-default 04:15:58, 04:15:58, 08:49:51, 08:47:56, isis_l2mp-default isis_l2mp-default isis_l2mp-default isis_l2mp-default

S10

S20

S30

S40

po1 po2 po3 po4

S100

S200

2010 Cisco and/or its affiliates. All rights reserved.

21

FabricPath ECMP
When multiple forwarding paths available, path selection based on ECMP hash function Up to 16 next-hop interfaces for each destination Switch ID Number of next-hops installed in U2RIB controlled by maximum-paths command under FabricPath IS-IS process (default is 16) Path selection based on hash function
S1

S100

S16

2010 Cisco and/or its affiliates. All rights reserved.

22

Conversational MAC Learning

MAC learning method designed to conserve MAC table entries on FabricPath edge switches
FabricPath core switches do not learn MACs at all

Each forwarding engine distinguishes between two types of MAC entry:

Local MAC MAC of host directly connected to forwarding engine Remote MAC MAC of host connected to another forwarding engine or switch

Forwarding engine learns remote MAC only if bidirectional conversation occurring between local and remote MAC
MAC learning not triggered by flood frames

Conversational learning enabled in all FabricPath VLANs

2010 Cisco and/or its affiliates. All rights reserved.

23

Conversational MAC Learning

FabricPath MAC Table on S300
MAC B C IF/SID S200 (remote) e7/10 (local)

S300

FabricPath MAC Table on S100

MAC A B IF/SID e1/1 (local) S200 (remote)

S100

MAC C

FabricPath MAC Table on S200

MAC IF/SID S100 (remote) e12/1(local) S300 (remote)

S200 MAC A

A B C

MAC B

2010 Cisco and/or its affiliates. All rights reserved.

24

FabricPath Multidestination Trees

Root for Tree 1 S10 S20 S30 Root for Tree 2 S40

Multidestination traffic constrained to loop-free trees touching all FabricPath switches Root switch assigned for each multidestination tree in FabricPath domain Loop-free tree built from each Root and assigned a network-wide identifier (Ftag) Support for multiple multidestination trees provides multipathing for multi-destination traffic

S100

S101

S200

Two trees supported in NX-OS release 5.1

S100

S20

S100

S10

S10

S101

S30

S40

S101

S20

Root

S200

S40

Root

S200

S30

Logical Tree 1
2010 Cisco and/or its affiliates. All rights reserved.

Logical Tree 2
25

Multidestination Trees and Role of the Ingress FabricPath Switch

Ingress FabricPath switch determines which tree to use for each flow
Other FabricPath switches forward based on tree selected by ingress switch
Root for Tree 1 S10 S20 S30 Root for Tree 2 S40

Broadcast and unknown unicast typically use first tree Hash-based tree selection for multicast, with several configurable hash options
L1 L5 L2 L3 L6 L4 L7 L9 L8 L10 L11 L12

Multidestination Trees on Switch 100

Tree 1 2 IF L1,L2,L3,L4 L4

S100

S101

S200

2010 Cisco and/or its affiliates. All rights reserved.

26

Putting It All Together Host A to Host B

(1) Broadcast ARP Request
Multidestination Trees on Switch 10
Tree IF L1,L5,L9 L9 DSIDFF Ftag1 SSID100 DMACFF SMACA L5 L1 L2 L3 L6 L4 L7 L9 L8 L10 L11 L12 DSIDFF Ftag1 SSID100 DMACFF SMACA Payload

Root for Tree 1 S10

S20

S30

Root for Tree 2 S40

Ftag

1 2

Multidestination Trees on Switch 100

Tree IF L1,L2,L3,L4 L4

Payload

Broadcast

1 2

S100

S101

S200

Multidestination Trees on Switch 200

DMACFF SMACA Payload Tree IF L9 L9,L10,L11,L12 Payload SMACA DMACFF

FabricPath MAC Table on S100

MAC A IF/SID e1/1 (local)

Ftag
MAC A

1 2

MAC B

FabricPath MAC Table on S200

MAC IF/SID

Learn MACs of directly-connected devices unconditionally

Dont learn MACs in flood frames

2010 Cisco and/or its affiliates. All rights reserved.

27

Putting It All Together Host A to Host B

(2) Unicast ARP Reply
Multidestination Trees on Switch 10
Tree IF L1,L5,L9 L9 DSIDMC1 Ftag1 SSID200 L5 L1 L2 L3 L6 L4 L7 L9 L8 L10 L11 L12 DSIDMC1 Ftag1 SSID200 DMACA SMACB Payload

S10

S20

S30

S40

Ftag

1 2

Multidestination Trees on Switch 100

Tree IF L1,L2,L3,L4 L4

DMACA SMACB Payload

Ftag

1 2

S100

S101

S200

Multidestination Trees on Switch 200

Payload SMACB DMACA Tree IF L9 L9,L10,L11,L12 DMACA SMACB Payload

FabricPath MAC Table on S100

MAC IF/SID e1/1 (local) S200 (remote)

Unknown
MAC A

1 2

A B

MAC B

FabricPath MAC Table on S200

MAC IF/SID

If DMAC is known, then learn remote MAC

2010 Cisco and/or its affiliates. All rights reserved.

A
B e12/2 (local)

28

Putting It All Together Host A to Host B

(3) Unicast Data
FabricPath Routing Table on S30
Switch IF L11 DSID200 Ftag1 SSID100 L5 L1 L2 L3 L6 L4 L7 L9 L8 L10 L11 L12 DSID200 Ftag1 SSID100 DMACB SMACA

S10

S20

S30

S40

S200

S200

FabricPath Routing Table on S100

Switch S10 S20 S30 S40 S101 IF L1 L2 L3 L4 L1, L2, L3, L4 L1, L2, L3, L4

DMACB SMACA Payload

Hash S101

Payload

S100

S200

FabricPath Routing Table on S30

DMACB SMACA Payload Switch IF Payload SMACA DMACB

S200

S200
MAC A

S200

S200

MAC B

FabricPath MAC Table on S100

MAC A IF/SID e1/1 (local) S200 (remote)
2010 Cisco and/or its affiliates. All rights reserved.

FabricPath MAC Table on S200

MAC A IF/SID S100 (remote) e12/2 (local)

29

S3

FabricPath
L2

Introducing VPC+
S1

L1

F1 F1 F1

VPC+ F1

F1 S2

CE

VPC+ allows dual-homed connections from edge ports into FabricPath domain with active/active forwarding
CE switch, Layer 3 router, dualhomed server, etc. Physical

F1
po3

Host A

VPC+ requires F1 modules with FabricPath enabled in the VDC

Peer-link and all VPC+ connections must be to F1 ports Logical
F1 S1 F1 F1 S3
L1 L2

Host AS4L1,L2
F1 F1 F1 S2

VPC+ creates virtual FabricPath switch for each VPC+-attached device to allow load-balancing within FabricPath domain

VPC+

Virtual Switch 4 becomes next-hop for Host A in FabricPath domain

S4
po3

Host A
2010 Cisco and/or its affiliates. All rights reserved.

30

VPC vs. VPC+

A given VDC can be part of VPC domain, or VPC+ domain, but not both VPC+ only works on F1 modules with FabricPath enabled in the VDC Conversion between VPC and VPC+ is disruptive
VPC Peer-link Member ports VLANs M1 ports or F1 ports M1 ports or F1 ports CE or FabricPath VLANs VPC+ F1 ports F1 ports FabricPath VLANs only FabricPath core port

Peer-link switchport mode CE trunk port

2010 Cisco and/or its affiliates. All rights reserved.

31

VPC+ Physical Topology

Peer link and PKA required Peer link runs as FabricPath core port VPCs configured as normal VLANs must be FabricPath VLANs
S10 S20 S30 S40

No requirements for attached devices other than channel support

S100

S200

MAC A

MAC B

MAC C 32

2010 Cisco and/or its affiliates. All rights reserved.

VPC+ Logical Topology

S10

S20

S30

S40

Virtual switch introduced

S1000

S100

S200

MAC A

MAC B

MAC C 33

2010 Cisco and/or its affiliates. All rights reserved.

Remote MAC Entries for VPC+

S200# sh mac address-table dynamic Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen,+ - primary entry using vPC Peer-Link VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID ---------+-----------------+--------+---------+------+----+-----------------* 10 0000.0000.000c dynamic 1500 F F Eth1/30 10 0000.0000.000a dynamic 1500 F F 1000.11.4513 S200#

S10

S20

S30

S40

S1000
po1 po2

S100

S200

1/30

MAC A

MAC B

MAC C

2010 Cisco and/or its affiliates. All rights reserved.

34

FabricPath Routing for VPC+

S200# sh fabricpath route topology 0 switchid 1000 FabricPath Unicast Route Table 'a/b/c' denotes ftag/switch-id/subswitch-id '[x/y]' denotes [admin distance/metric] ftag 0 is local ftag subswitch-id 0 is default subswitch-id FabricPath Unicast Route Table for Topology-Default 1/1000/0, number of next-hops: 2 via Po1, [115/10], 0 day/s 01:09:56, isis_l2mp-default via Po2, [115/10], 0 day/s 01:09:56, isis_l2mp-default S200#

S10

S20

S30

S40

S1000
po1 po2

S100

S200

1/30

MAC A

MAC B

MAC C

2010 Cisco and/or its affiliates. All rights reserved.

35

VPC+ and Active/Active HSRP

With VPC+ and SVIs in mixed-chassis, HSRP Hellos sent with VPC+ virtual switch ID FabricPath edge switches learn HSRP MAC as reached through virtual switch Traffic destined to HSRP MAC can leverage ECMP if available Either VPC+ peer can route traffic destined to HSRP MAC

HSRP Active
DSIDMC SSID1000 DMAC0002 SMACHSRP Payload S1000

HSRP Standby

SVI
S10 S20

SVI
S30 S40

po1

po2

S100

S200

1/30

MAC A

MAC B

MAC C

2010 Cisco and/or its affiliates. All rights reserved.

36

HSRP MAC on Edge Switches

HSRP Active HSRP Standby

SVI
S10 S20

SVI
S30 S40

S1000
po1 po2

S100

S200

MAC A

MAC B

MAC C

S200# sh mac address-table dynamic address 0000.0c07.ac0a Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen,+ - primary entry using vPC Peer-Link VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID ---------+-----------------+--------+---------+------+----+-----------------10 0000.0c07.ac0a dynamic 0 F F 1000.0.1054 S200#

2010 Cisco and/or its affiliates. All rights reserved.

37

Active/Active HSRP for FabricPath with VPC+

Programs gateway MAC on both active and standby devices Requires VPC+ peer link on F1 modules
L3
Active Standby HSRP

FabricPath CE

SVI GWY MACrouter MAC S1 GWY MACproxy L3 port-channel

F1

M1 F1

VPC+
F1

M1

SVI GWY MACrouter MAC S2

F1

F1

F1

GWY MACproxy L3 port-channel

L1

F1

F1 L2

po3

GWY MACL1,L2

GWY MACpo3

2010 Cisco and/or its affiliates. All rights reserved.

38

Active/Active Gateway MAC for FabricPath with VPC+

External HSRP routers connected via VPC+ Gateway MAC advertised into FabricPath domain
FabricPath

L3
Active HSRP Standby

CE

po1

po2

F1

F1 F1

GWY MACpo1 S1
F1

VPC+

F1 F1

F1

GWY MACpo1 S2
F1

F1

F1

L1

F1

F1 L2

po3

GWY MACL1,L2

GWY MACpo3

2010 Cisco and/or its affiliates. All rights reserved.

39

e1/1-4

e1/1-4

2010 Cisco and/or its affiliates. All rights reserved.

40

po3 L1

2010 Cisco and/or its affiliates. All rights reserved.

41

 2010 Cisco and/or its affiliates. All rights reserved.

42