BRKDCT 2218 PDF

Data Centre Design For The Small
to Medium Enterprise
Nic Rouhotas Technology Solutions Architect

BRKDCT-2218
Agenda
Design Evolution
Optics & Cabling considerations
STP, vPC and Spine/Leaf
Scaling Brownfield and Greenfield Data Center Networks
Building Blocks, Availability Zones and Segmentation
Data Centre Interconnect
Programmability, Automation and Orchestration
Summary
LOTS of Related Sessions
Session ID Title Presenter Date / Time
BRKDCT-2375 End-to-End Application-Centric Data Center Carlos Pereira Check on CiscoLive App
BRKDCT-2334 Data Centre Deployments and Best Practices with Brenden Buresh Check on CiscoLive App
NX-OS
BRKDCT-3378 Building Data Center networks with VXLAN BGP-EVPN Lukas Krattiger Check on CiscoLive App
BRKDCT-2134 Data Centre Interconnect with OTV and Other Solutions Dave Jansen Check on CiscoLive App
BRKDCT-2615 How to Achieve True Active-Active Data Centre John Schaper Check on CiscoLive App
Infrastructures
BRKACI-2001 Integration and Interoperation of Existing Nexus Azeem Suleman Check on CiscoLive App
Networks into an ACI Architecture
BRKDCN-2025 Maximizing Network Programmability and Automation Nicolas Delecroix Check on CiscoLive App
with Open NX-OS
BRKACI-2004 How to Setup an ACI Fabric from Scratch Camillo Rossi Check on CiscoLive App
BRKDCT-2218 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Design Evolution
Small and Medium Business
What is a Small to Medium Business for an Australian market?

According to the ASIC a small business entity has an annual
revenue of less than $25m and fewer than 50 employees.*
A medium business has 200 or fewer employees with an annual 1SMB
revenue of around $150m. 2Large
Australian SMBs make 97% of all Australian businesses,

produced one third of total GDP, and employ 4.7 million people*
BRKDCT-2218 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Many different aspects to todays Data Centers?
O/S, Server Virtualisation & Containers
Automation/Orchestration/Mngmt Operating Systems
Device-by-Device vs System Level Hypervisors; Virtual Machine Managers
Containers (e.g. Linux; Docker)
Manual vs Scripted vs Programmable
Applications Applications Applications Applications Applications Applications Applications Applications Microservices
Applications Applications Applications
SDN: none; s/w Add-on; Built-In
Applications Applications Applications Applications Applications Applications Applications Applications Applications Applications Applications
Management: CLI; GUI; API
Cloud, Service, Infrastructure Orchestration
Applications Applications Applications Applications Applications Applications Applications Compute Type & Form Factor
Applications Applications Applications Applications
Applications Applications Applications Applications Applications Applications Applications Applications
Network Centric > Policy Centric Applications Applications Applications
X86 Rack & Blade Servers
Network/Fabric Build UCS Managed (Blade and/or Rack)
Applications Applications Applications Applications Applications Non X86; Legacy (P-Series, Mainframe etc.)
Applications Applications Applications Applications Applications Applications
Type (Traditional; CLOS)
Compute Connectivity
Capabilities; Ease of Integration
Facility & Cabling Scale & Efficiency
Applications Applications Applications Applications Applications
Varying Speed 10M/100M; 1G/10G/25G
Applications Applications Applications Applications Applications Applications
Power/Cooling (Per Rack/Row/Hall)
Applications Applications Applications Applications Applications NIC/HBA/CNA # interfaces per-server
L2 vs L3 Based; Central vs Distr. Applications Applications Applications
Applications
Flood & Learn vs Control PlaneApplications NIC Teaming models
Applications Applications Applications Applications
Cabling Style: Oversubscription 25:1 to 1:1
End of Row (EoR); Middle of Row (MoR); Applications Applications Applications Applications Applications
Cabling to ToR; MoR; EoR
Speed (10M-100G+)
Top of Rack (ToR) Applications Applications Applications Applications Applications Applications
Intra Rack: Copper, Direct Attach Network Services/Peripheral Storage & Protocols
Inter Rack: AOC; MMF; SMF NAS; SAN
Firewalls; LBs
WAN; Campus; OOB IP (File, Block), Fibre Channel,
FCoE
Brownfields/Legacy DC network
Distributed, HyperConverged
DC Interconnect (DCI)
Public Cloud BRKDCT-2218 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Why Data Center Fabrics?
Definition: ensemble of switches that behave and
get configured like a single giant switch
Internet
Public Offsite DC
Flexibility: allows workload mobility, VLANs Cloud Enterprise
everywhere Network Site B
Robustness: reduce failure domains, L2/L3 Mobile
NORTH - SOUTH TRAFFIC

boundary on leafs, anycast gateway
FC
(Virtual) Network services move out to border
leafs, policy-based service chaining DATA FCoE
API CENTER iSCSI / NAS
Performance: full cross sectional bandwidth FABRIC Storage

(any-to- any) with ECMP, avoid oversubscription Orchestration/
Latency: deterministic at scale, single hop away Monitoring
Scalability: add end nodes, maintain
oversubscription
Cost: fixed switches vs modular switches Services
Server/Compute
EAST WEST TRAFFIC
Designing Small to Medium Sized Data Centers
Defining Small to Medium Client Access
WAN / DCI
Require Dedicated DC Switches, moving away
from collapsed core with campus Campus
Mostly virtualized, some baremetal servers

Scalability L3
-----------
L2
Size for current needs, reuse components in FC
larger designs
FCoE
Design Options iSCSI / NAS
Feature choice + priority = tradeoffs

Where the industry is going:
Programmability, Automation (SDN buzz)
Virtual Network Services
Network Topologies
Ring Star Full Mesh Hub
Tree N-Tiered Spine Leaf
Examples of DC Networking Topologies
Switching Architecture Changes
Shifting of Internal Architecture
DBUS SOC SOC SOC SOC

RBUS
EOBC CROSSBAR
Linecard Linecard Linecard Linecard Linecard

SOC SOC SOC SOC
Design Shifts Resulting from Increasing Gate Density and Bandwidth
10/100M 100M/1G 1G/10G 10G/100G

Fabric Evolution
Shift from scale up to scale out
STP VPC Fabric Path VXLAN
L3
L3
Traditional Overlay Based
Which network model would YOU choose?
Example: 350 VMs; 25 Baremetal Servers @ 1/10GE, (HA), IP Based Storage
L3
OVERLAY e.g. VXLAN, ACI

L3
96 front L2
facing
ports
96 front L2
facing
ports
Collapsed/Single Tier Spine Leaf
Scalability?
1RU
Fixed vs Modular switches?
Oversubscription? Downlink ports: 48p Uplink/Breakout ports: 6-12p Cost (today vs tomorrow)?
Management and Automation?

Optics & Cabling Considerations
Distance and speed dependent on cable medium and type
Cabling the Network Topology

Twinax= Direct Attach Copper (CX1)
AOC= Active Optical Cable

RJ-45 = Cat5e, Cat6a, Cat7/7a, Cat8
MMF= Multi Mode Fiber

SMF=Single Mode Fiber
Traditional Physical Layer Connectivity

Spine Leaf
Type/Distance/Speed
(examples)
L3
Core
10/40/100G Twinax; AOC (10G <15m; 40G<15m; 100G <30m)
L3/L2
Spine
Distribution MMF (10G <400m; 40G <150m; 100G <100m)
VXLAN
SMF (40G LR4 <10km; 40G ER4 <40km; 100G
1/10/40G LR4 <10k, 100G ER4-L <25k) 40G/100G L3
Leaf
Access L2
L2
Twinax Copper; AOC (10/40G <15m)

1/10/25G Copper RJ-45 (10G <100m; 40G <30m) 1/10/25G
MMF (<400m)
Host to Switch Cabling Style Switches
Servers
Interconnects
Top of Rack Middle of Row End of Row

ToR MoR EoR
Intra- Rack Inter Rack Inter Rack

Not all options listed
Twinax Direct Attach Copper (Passive CX1 10G 1,3,5m, 40G 1,3,5m; 100G 1-3m); (Active 10G 7-10m ; 40G 7-15m) Copper
Active Optical Cable (AOC) ( 10G 7,10M ; 40G 1-15m; 100G 1-30m) Fiber
Copper Base-T/RJ-45 Connector (10G Cat6A <100m; 40G Cat7/7A 30-60m; 100G Cat8 <30m) Copper
Fabric Extender Transceiver (FET)* (FET-10G ; FET-40G <150m with MPO-12) *Requires N2k and supporting parent switch Fiber
MMF (10G BiDi <150m, 40G SR4 <300m; 100G SR10, SR4 <100m) Fiber
SMF (10G ER <40km; 10G LR; 40G LR <2km; 100G ER4 <40km; 100G LR4 <10km) Fiber
Pictures from jimenez_3bq_01_0711.pdf, 802.3bq Ethernet Alliance

* 10/100M not always available on newer 1/10/25G models
ToR Access/Leaf Switch Model Selection

UPLINKS
SFP+ Models:
10G
QSFP+ Models:
Example Nexus 9300 series switch 10G (Breakout, QSA), 40G
QSFP28 Models:
10, 25,40,50,100G combinations
Downlink ports: 48p Uplink/Breakout ports: 6-12p
DOWNLINKS
RJ-45 Models (combo examples):
10M/100M/1000M*, 1G/10G, 100M/1G/10G
SFP+ Models (combo examples):
10G, 1/10G, 10/25G, 1/10/25G
Multi-Medium at the Row level
Leaf pair with SFP+ downlinks
Leaf pair with RJ-45 downlinks
SPINE SPINE
SFP+ Pluggable Optics

(e.g. Twinax DAC; AOC; MMF)
1/10GBase-T
(e.g. Cat 6A RJ-45)
Direct Attached Cabling - Whats Better?
- Twinax Direct Attach Cabling (passive or active) vs AOC
QSFP 40G CABLE DIAMETER
AOC Active Optical Cables

SFP+ AOC 10G SFP+ Twinax AOC cables are thinner than Twinax Copper cables
copper cable
AOC Enable improved air Flow vs Twinax Copper
Primarily used for ToR to server or ToR to Leaf connectivity
No Electromagnetic Interference with AOC
Reach up to 30m (Copper cables have reach limitations with increasing speeds)
Less weight and Flexible: Significantly simplifies Cable management
3mm 10mm
AOC Twinax copper

cable
10G AOC vs 10G Twinax CX1

Network fiber migration with increase in data rates
Multimode Fiber Single Mode Fiber
1G
1km 10km
10G
400m 10km
40G
150m 2km 10km
100G
100m 500m 2km 10km
Shorter link distances with higher data

rates on OM4 multimode fiber.
Short reach optics for single mode fiber at higher data rates.
Multi-Mode Fiber (MMF) Fiber Core implications
10G vs 40G vs 100G
IEEE
10G Optical Link (10GBase-SR)
LC Fiber Cores LC
Rx Tx LC connector SC connector
Rx (SFP+) (X2, XENPAK)
Tx
2 fiber cores (i.e. duplex) 1 Tx/Rx pair
40G Optical Link (40GBase-SR4)

MPO-12 MPO-12
Rx Tx
Tx
Rx MPO -12 has 4 fiber cores un-used MPO-12
8 fiber cores (used) 4 Tx/Rx pairs
100G Optical Link (100GBase-SR10)

MPO-24 MPO-24
Rx Tx
MPO-24
Tx
Rx
MPO-24 has 4 fiber cores un-used
20 fiber cores (used) 10 Tx/Rx pairs

Parallel vs BiDi Duplex
QSFP-40G-SR4 vs. QSFP-40G-SR-BD BiDi Optics
QSFP-40G-SR4 QSFP-40G-SR-BD
QSFP SR QSFP SR QSFP-BIDI
TX/RX
TX
4 x 10G 2 x 20G
12-Fiber 12-Fiber Duplex Duplex
infrastructure infrastructure Multimode Multimode
4 x 10G Fiber 2 x 20G Fiber
RX TX/RX
12-Fiber ribbon cable with Duplex multimode fiber

MPO connectors at both with Duplex LC connectors
ends at both ends
Higher cost to upgrade from 10G Use of duplex multimode fiber lowers cost of
to 40G due to 12-Fiber upgrading from 10G to 40G by leveraging
Infrastructure per interface pair existing 10G multimode infrastructure
150m (OM4); 100m (OM3)
For your
Transceiver form factors reference
10G, 40G, 100G
18
72
Max
outer
10G, 40G Transceivers 100G Transceivers
Optics Pluggable Interfaces For your
reference
SFP & QSFP, QSFP28 SLIC= Subscriber line interface Converter (QSFP to SFP;
QSFP to QSFP)
BiDi= Bidirectional Optics (<150m)
QSA = QSFP to SFP Adapter
* * = Roadmap
SFP SFP+ QSFP+ QSFP28

(H x W x D): 8.5 x 13.4 x 56.5mm
(H x W x D) 13.5 x 18.4 x 72.4 mm.
Pluggable Options Pluggable Options

1G SFP 1G SFP (via QSA)
10G SFP+ 10G SFP+ (via QSA)
25G SFP28 25G SFP+ (via QSFP to SFP+ SLIC)
40G QSFP+, BiDi
50G (via SLIC) **
100G QSFP28, Bidi**
Host: 1/10/25G Network: 10/40/50/100G

Refer to the Compatibility Matrix
http://www.cisco.com/c/en/us/products/collateral/interfaces-modules/transceiver-modules/datasheet-c78-736282.html
http://www.cisco.com/c/en/us/td/docs/interfaces_modules/transceiver_modules/compatibility/matrix/10GE_Tx_Matrix.html
Breakout Support NOTE: Breakout support is platform hardware and software dependent
QSFP to SFP for high density SFP

QSFP40G 40G-SR
Cisco NX-OS supports breakout interfaces. The breakout command works at the MPO-LC break-out cable
module level and splits the 40G interface of a module into four 10G interfaces. The SR
module is reloaded and the configuration for the interface is removed when the SFP+
command is executed. snippet from NX-OS configuration guide QSFP+
SR
SFP+
40G-SR4/CSR4 SR
SFP+
SR
SFP+
LC-Duplex
QSFP 4x10G-LR
MPO-LC break-out cable
LR
SFP+
QSFP+
LR
SFP+
Cisco QSFP to SFP Breakout Examples: 4x10G LR LR

SFP+
Breakout 40G QSFP to 4x10G SFP+ LR or Active Copper or AOC LR
SFP+
Breakout 100G QSFP to 10x10G SFP+ LR or ER-L
Breakout 100G QSFP to 4x25G SFP+ Passive Copper LC-Duplex
Impact of Link Speed
Three non-oversubscribed topologies:
2010Gbps 540Gbps 2100Gbps
Uplinks Uplinks Uplinks
2010Gbps 2010Gbps 2010Gbps

Downlinks Downlinks Downlinks
http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=6627738
Impact of link Speed Intuition
Higher speed links improve ECMP efficiency
2010Gbps 540Gbps 2100Gbps 20x10G Probability of 100% throughput 3%

Uplinks Uplinks Uplinks
1 2 20
5x40G Probability of 100% throughput 75%
1 2 3 4 5
2x100G Probability of 100% throughput 99%
1110Gbps flows
(55% load)
1 2
Design Considerations
Increased BW Utilization due to 40G / 100G speedup
2x
40/100G
8 x 10G
8 x 10G 8 x 10G
Expected Max Effective Throughput = 86.33% Expected Max Effective Throughput = 65.6%
Uplink speed ups avoid hashing collisions to the provide effective utilization of available capacity
A speedup of 40G on uplinks for 10G attached servers results in FCT that are 1240% lower than
that of a 10G fabric
For your
reference
Data Center Cabling Architecture Considerations

Optics will drive physical architecture
Employ a main cross-connect at a MDA, for best management, scalability and growth
Plan for future device optics density and reach; fiber density parallel (SM/MM); <300m
(MM)
Use low loss connectors (<0.35dB/fiber) to enable MDA
Allocate rack space for orderly fiber count growth and network device interface scale
Plan for future high density 10G/25Gserver access to be addressed by 25/40/100GE
break out cables
Switch Architecture Type for Access Modular or ToR
Choice of SM or SM, Duplex or Parallel adoption time frame will determine choice
MDA: Main Distribution Area

STP, vPC and Spine/Leaf
History Lesson: Spanning tree
Spanning Tree introduced around 1985,

prevents loops
32 years ago, we also saw:
Windows 1.0
DNS come out of academia
First Nintendo Entertainment System
Host or
People searching for Turbo button on i286 PC Switch
Since then, most DC Designs built to

work around STP
2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Virtual Port Channel (VPC)
vPC Domain
VPC invented to overcome STP limitations

IEEE standard in 2000 (802.3ad)
Not perfect, but a good workaround
STP is still there on every link
Host or
Human error, misconfiguration, bug can Switch
still cause issues
Virtual Port Channel (vPC) Fabric
vPC Domain 1
vPC Northbound & Southbound

More efficient than native STP
STP is still running
Dual sided vPC
Back-to-Back vPC
Another good workaround

mini-fabric
Configuration can become complex as

switch counts grow vPC Domain 2
vPC makes two switches look as

one.but what about 4 switches?
Host or
Switch
Why Spine-Leaf Design? Flexibility and Efficiency
To speed up flow
completion times, add
Need
Need even
moremore
host more backplane,
Utilization
Per Spine
host ports?
ports? spread load across
AddAdd
another
a leafleaf more spines
Per Spine
Utilization
40G fabric ports
FCT
FCT
FCT
FCT
FCT
FCT
* FCT = Flow Completion Times 10G host ports
Lower FCT = FASTER
96 ports APPLICATIONS
144 ports
2x48 10G192
(960
ports
Gbps total)
3x48 10G (1440 Gbps total)
4x48 10G (1920 Gbps total)
2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
Multi-path Fabric Based Designs - FabricPath
Natural migration from vPC

MAC in MAC encapsulation
Easy to turn on (Nexus 5/6/7k)
No STP within Fabric; BPDUs dont FP
traverse fabric L3
Distributed L3 gateway to edge, VLAN L2

Border
MAN/WAN
anywhere notion
TRILL=Standard based, limited
capabilities
Fabric Path = Cisco proprietary features
L3-Based Fabrics
Every link forwarding
L3 routing fast convergence
VXLAN overlay (MAC in UDP) VXLAN
Flood and learn vs VXLAN L3

BGP-EVPN control plane L2
MAN/WAN Border
STP might still exist on edges,
but not within the fabric
VPC still needed at edge
Spine/Leaf:
Flexible and efficient design
Consistent hop count & latency
Scaling Brownfield and Greenfield
Data Center Networks
Scaling a VPC-based DC design
Access L3
Layer L2
VLANs
100-150 Host Host Host
Consolidated
Core/Agg
Layer
Access L3 Access
Layer L2 Layer
VLANs VLANs
100-150 Host Host Host Host Host Host 151-200
Distributed or
Consolidated
Core/Agg
Layer
L3
L2
Access Access
Layer Layer
VLANs VLANs
100-150 Host Host Host Host Host Host 151-200
Integrating ACI with an existing network
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/migration_guides/migrating_existing_networks_to_aci.html
ACI Pod
New DC
Row Upgrade
New Application
Distributed or
Consolidated
Core/Agg Spine
Layer Layer
L3
ACI Fabric
L2 (VXLAN based)
Dual sided
vPC
ACI Border Leafs

Access Access Access
Layer Layer Layer
VLANs VLANs VLAN
100-150 151-200
Host
201-250
Host Host
ACI Pod
New DC
Row Upgrade
New Application
Distributed or
Consolidated
Core/Agg Spine
Layer Layer
L3
ACI Fabric
L2 (VXLAN based)
Access ACI
Layer
VLANs
Leafs
100-150
Host and Border Leafs
Host Host
Building Blocks, Availability Zones
and Segmentation
WAN Campus
Building Block Design Zone Zone
Campus
WAN Building Building Block DC
Block Zone Site
Layer 3
Routing for
segmentation
Si Si
Demarcation L3 Routing only
Border
Leaf/Transit
switches
Connect OOB
Management
into Campus
Core or WAN
Data Center OOB

Building Block Management
Segmentation Granularity
DEV WEB
PRODUCTION WEB
DMZ VLAN 1 VXLAN 2
POD
TEST APP
SHARED WEB
VLAN 3
SERVICES
PROD DB
Basic DC Network Segment by Application Network centric Per Application-tier / Intra-EPG Container Security
Segmentation Lifecycle Segmentation Service Level Micro-Segmentation
Micro-Segmentation
Coarse Fine
Level of Segmentation/Isolation/Visibility
Network Centric Deployment
Leverage known networking constructs.
VLANs, IP address / subnets, Flood domains etc.
Existing network Nexus, Catalyst, etc
VRF VNI: 300100 VRF VNI: 300100
MAC_A,IP_A: 10.1.1.2 MAC_B,IP_B:20.1.1.2

VNI L2: 100100 VNI L2: 100200
VLAN 100:
10.1.1.0/24 VLAN 100 VLAN 200
VLAN 200:
20.1.1.0/24 VXLAN EVPN Fabric
Existing network Nexus, Catalyst, etc TENANT
PRIVATE NETWORK (VRF)
BD: Blue BD: Red

10.1.1.0/24 20.1.1.0/24
VLAN 100:
10.1.1.0/24 EPG: Blue-100 EPG: Red-200
VLAN 200:
20.1.1.0/24 ACI Fabric
Policy Centric Deployment (with ACI)
Leverage grouped constructs Network Level; Application Policy Level
Abstraction with Application network profiles, Policy Oriented
Automation of network services
APIC Controller: declaration of intention, translate network constructs via policy
Application
External Profile
Connectivity L3 Outside
EPG EPG
EPG EPG EPG

Web App DB
Web App DB
EP EP EP EP
FW / SLB FW / SLB
App Profile EPs are devices which attach to the network either virtually or physically, e.g:
Virtual Machine
Physical Server (running Bare Metal or Hypervisor)
External Layer 2 device
External Layer 3 device
Combined Network Model: Virtual Ports, Physical Ports, External L2 VLAN, External L3 subnet VLAN
Subnet
Firewall
Load balancer
Data Center Design Requirements
Things will fail, so how can we protect ourselves
* http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html
DCN Building Blocks with PoDs Spine Node
Leaf Node
PoD = Point of Delivery Layer 3
Collection of compute, storage, and network PoD

resources that conform to a standard operating
footprint sharing the same failure domain Data Center
DCI
RJ-45 SFP+
Campus
Hosts Hosts WAN
Fabric
DCN Building Blocks with Availability Zones
Using Amazon Web Services terms
Global Multi-Site Data Center Deployment
Region Region
Avail
Pod
Zone
Avail Avail
Pod Pod
Zone Zone
Multi-Pod
Smaller instances of fabrics with distinct control plane
but shared policy and management planes Transit Leaf Node /Inter POD
Network (IPN) node
Spine Node
PoD = Point of Delivery
Collection of compute, storage, and network Leaf Node
resources that conform to a standard operating

footprint sharing the same failure domain Layer 3
PoD 1 PoD 2
Within or Across DC locations using:

VXLAN EVPN Multi-Pod
PoD 3
ACI Multi-Pod
Creative" use of Multi-Pod
Smaller instances of fabrics with distinct control plane but shared policy and
management planes
Development Cycle Evergreen Concept
Pre-Prod
Ver n Ver n+1
(Stage)
Stack Stack
Test/Dev Prod
Test
Multi-Pod Multi-Pod
Data Center Interconnect
IT Trends - Distributed Data Centers
Building the Data Center Cloud
Distributed Data Center Goals:

Seamless workload mobility between multiple
datacenters
Distributed applications closer to end users
Pool and maximize global compute resources
Ensure business continuity and disaster
avoidance with workload mobility, distributed
deployments and clustered applications
Small to Medium Enterprises often rely on two
datacenters
NX-OS Overlay Transport Virtualization Technology
Extend VLANs Across Datacenters
OTV
Classical Pod Scalable Pod Leaf Spine Pod

Spanning Tree Protocol vPC, N-tier design VXLAN, ACI
Feature: Benefits: In the news:

Simplified Layer 2 / VLAN Extensions Field proven, very mature M3 linecards with NX OS 8.0
Restricted fault domain (STP domain), Enable L2 elasticity across DCs OTV Loopback Join Interface for
loop prevention Multicast-based OTV control-plane
Simplify Virtual Machine Mobility
Optimized Multicast replication Wire-rate 256-bit AES MACsec on all
Extend Layer 2 without the risks of
ports at all speeds along with OTV
Dual homing large fault domain
Works over dark fibre, MPLS or IP Simple 3 easy commands !
Data Centre Interconnect Options with OTV Client Access
Options for L2 Interconnect ASR1000

WAN / DCI
Client Access Campus
WAN / DCI
Campus
ASR1000
L3
-----------
L3 L2
-----------
L2 N7K
Virtual DC Virtual DC
Services in Services in
Software VM VM VM VM VM VM VM VM VM VM VM VM Software
Virtualised Servers with Nexus CSR1000v Virtualised Servers with Nexus

1000v, vPath, CSR 1000v 1000v, vPath, CSR 1000v
VXLAN as a Data Center Interconnect?
Building
DCI is an architectural discussion
VXLAN is an encapsulation technique
DCI has always been an architectural

discussion, but recently some
vendors have made it become an
encapsulation discussion
VXLAN can absolutely fit into a DCI
architecture if you handle it
CAREFULLY
VXLAN as a Data Center Interconnect?
Enablers to VXLAN in an DCI implementation?
VXLAN with BGP/EVPN fabric end-to-end
Host Reachability (MAC/IP) information is managed and distributed end-to-end
When to use VXLAN/EVPN Stretched Fabric ?

Across short distances (Metro), Private L3 DCI, IP Multicast available end-to-end
Multiple Greenfield DC with VXLAN/EVPN, continuity of control/data plane
Host Reachability is End-to-End
Caveats:
VXLAN for DCI not mature yet
VXLAN
DCI Leaf nodes
No architectural protection mechanisms at edge tunnel
Use hardware protection mechanisms:

Storm Control, BPDU Guard, HMM Route Tracking
Control-Plane with MAC-learning, ARP suppress VXLAN Leaf VXLAN Stretched Fabric
Bud Node
Preference for OTV or ACI MultiPOD (safer approach) Traffic is encapsulated & de-encapsulated on each far end side
DCI with ACI Multi-Fabric options
Single APIC Cluster/Single Domain (BC-DA) Multiple APIC Clusters/Multiple Domains (DR)
Stretched Fabric Dual-Fabric Connected (L2 and L3 Extension)
ACI Fabric ACI Fabric 1 ACI Fabric 2
Site 1 Site 2
L2/L3
More Scale
Multi-Pod (6 pods today; 12+ pods roadmap) More Flexibility Multi-Site (Future)
More Automation
Pod A IP Network Pod n Site A IP Network Site n
MP-BGP - EVPN
MP-BGP - EVPN
APIC Cluster
VXLAN for DCI: Final Considerations
VXLAN is an encapsulation, not an architecture

VXLAN for DCI is no more in its infancy... but still be
very careful when using VXLAN for DCI, you have to
take care of many moving parts
Its like building your own custom animal
o Looks cool
...... but takes special care
Prefer OTV or ACI MultiPOD
... or rely on Cisco AS experience
Programmability, Automation and
Orchestration
Why Network Programmability?
More Speed More Repeatability More Flexibility More Innovation
Fewer Mistakes
Programmability, Automation, Orchestration
Network-Enabled Applications
Orchestration
CloudCenter
UCSD
Cloud | On-Prem
Open North Bound RESTful APIs
Automation
NFM
Controllers, Configuration Management Tools

Open & Programmable | Standards-Based South Bound APIs
Open Device Programmability
Physical and Virtual Network Infrastructure

Controlling, Managing & Automating the Fabric
IT Service Management (e.g. Service Now)
Multi Cloud (e.g. Cisco CloudCenter) Application Policy

Fabric Manager Infrastructure
(i.e. DCNM, NFM) Orchestration & Workflow Automation Controller
Connection Private cloud: compute | network | storage
Creation Expansion e.g. Cisco UCS Director, Openstack, Puppet, Chef, Ansible, Salt etc. APIC APIC
APIC
Using APIs
Reporting Fault Mgmt
Network Centric
Network Centric Policy Centric
CAMPUS CAMPUS Application
WAN / DCI WAN / DCI
Centric
Spine
Spine
L3 with Overlay ACI Fabric
Leaf Leaf
Standalone NX-OS Application Centric Infrastructure
Device by Device or Do it Yourself Fabric (ACI)
Choice of NMS, Fabric Manager Turnkey System
Choice in Automation and Programmability
Open NX-OS
Application Centric Programmable Fabric Programmable Network

Application Optimized Networks
Turnkey integrated solution with BGP EVPN standard-based
w/Segment Routing
security, centralized management,
compliance and scale 3rd party controller support
Modern NX-OS with
Nexus Fabric Manager / DCNM 10 for NX-API REST/YANG/OpenConfig
Automated application centric-policy
model with embedded security automation and management across
DevOps toolset used for
N2K-N9K
Network Mgmt
Broad and deep ecosystem
(Puppet, Chef, Ansible etc.)
Fabric
Management
DB DB
Web Web App Web App
Automation, APIs, Controllers and Tool-chains
Nexus Programmability Nexus 7K Nexus 5K / 6K Nexus 9K
Puppet/Chef/Ansible Shipping Shipping Shipping
Provisioning &
PoAP Shipping Shipping Shipping
Orchestration
OpenStack Shipping Shipping Shipping
XMPP Shipping Shipping Shipping

NetConf(SSH/XML) Shipping Shipping Shipping
NX API CLI (JSONXML) Shipping Shipping Shipping
Protocols and
NX API REST Future Future Shipping
Data Models
YANG Future Future Shipping
RESTconf/gRPC Future Future Shipping
Streaming Telemetry Future No Shipping
Native Python Shipping Shipping Shipping
Programmatic Linux container Shipping Shipping Shipping

Interfaces Guest Shell Future Future Shipping
OpenFlow Shipping Future Shipping
Open Fabric Programming Git Hub Repository For Your
Reference
https://github.com/datacenter/
USE
Enterprise Customer Igniting The Programmable Network CASE
Fabric Provisioning Automation with Open NX-OS Ignite and Puppet
Programmable
Network Customer Benefits
Reduced Operational Costs
Fully Automate day zero fabric provisioning with

Ignite
Automate full day one configuration

management with Puppet and Open NX-OS
Eliminate errors
Use repeatable process for consistent

provisioning
Enable idempotent configurations, push

changes only when necessary
https://github.com/datacenter/ignite
https://github.com/cisco/cisco-network-puppet-module
Commercial Cloud Provider USE
Tenant Onboarding Automation with Ansible and Open NX-OS CASE
Customer Benefits
Fast and secure tenant onboarding
Time reduced from days to minutes with Open

NX-API
NX-OS and Ansible OE
Ansible secure agentless architecture ensures

no agents to exploit or update
Drive Compliance and Consistency
Eliminate configuration drift across network
Continuously check network for standards and

model compliance
https://developer.cisco.com/opennxos
http://github.com/datacenter
Programming a Fabric For Your
Reference
A lot of work is being done to provide customers maximum flexibility in

programming & automation interfaces
Free Open Programmability book:
http://www.cisco.com/c/dam/en/us/td/docs/switches/datacenter/nexus9000/sw/open_n
xos/programmability/guide/Programmability_Open_NX-OS.pdf
New community site dedicated to NXOS programmability:
https://opennxos.cisco.com
A lot of work has been done to increase available knowledge on network

programming across all Cisco products
DevNet: If you havent visited, please do
https://devnet.cisco.com
SANDBOX! FREE 24 X 7 hosted labs
Open NX-OS

w/Segment Routing
Modern NX-OS with
N2K-N9K
Network Mgmt
Fabric
Management
DB DB
Web Web App Web App
Along the Spectrum from CLI to ACI
A New Way To Do Fabric Management
Scripting to the
CLI and/or API
Basic
Element
Manager
CLI ACI
Fabric Management functionality
Common Customer Asks
Fabric management automation High interest

Many Fabrics are based multiple protocols such
as VXLAN, BGP/EVPN, IS-IS, Multicast
New protocols, new configurations, new things to
learn
Simple tool to ease burden of adoption
CVD/Best practices done for you!
Cisco Nexus Fabric Manager (NFM)
Fabric aware tool that simplifies and automates VXLAN
deployments
Connection
Intelligent fabric lifecycle management
Automates via knowledge of
underlying fabric
Creation Expansion
Simplifies fabric management via GUI
NFM Actively builds & manages: NFM
VXLAN/EVPN Underlay Fabric

Overlay connectivity Reporting Fault Mgmt
Manages Nexus 9000 switches

Spine Leaf Only, Max 50 leafs (current) Intelligent Fabric Management
Lifecycle
Data Center Network Manager (DCNM) 10
Extensible, Automatable Network
Management System tool for Storage, LAN
and Programmable (NX-OS Mode) Fabrics
Advanced Feature support including VPC, VXLAN, POAP,
Storage Zoning, Slowdrain, OTV, Topology, S/W Image,
Configuration
and Template management
Aids troubleshooting through monitoring and reporting and
Topology views
Cisco Nexus (2k, 5k, 7k, 9k) and Cisco MDS switching families
Multi-Site, Multi-DC capable
FP
VXLAN
Underlay+Overlay
Solution
Templates
Multi-Fabric
Multi-Site
Financial Customer USE
Open NX-OS Network Analytics with Splunk Forwarder CASE
Customer Requirements
Monitor CPU, disks, fans, temperature sensors
Monitor network statistics (packets, drops, errors etc.)
Extensible framework to include additional data

sources/attributes on-demand
Customer Benefits
Time Scale
Network visibility and operational intelligence at scale
(minutes)
Pro-active problem identification
Splunk Cisco Network (incl. Nexus)
Add-on Dash Board Faster MTTR (Mean Time To Repair)
https://github.com/inspired/cisco_ios One Splunk dashboard for network monitoring along with

applications, servers and services
Open NX-OS

w/Segment Routing
Modern NX-OS with
N2K-N9K
Network Mgmt
Fabric
Management
DB DB
Web Web App Web App
FCAPS and Automation
Application Centric
Programmable Fabric Programmable Network
Infrastructure
Connection
Creation Expansion
VTS
Reporting Fault Mgmt
Turnkey integrated solution with security, centralized

management, compliance and scale Integrated stack Modern NX-OS with enhanced NX-APIs
Or
Automated application centric-policy model with A-la-carte Automation DevOps toolset used for Network Management
embedded security (Puppet, Chef, Ansible etc.)
Streamlined Workflow Management
Customer Script based Operations and Workflows
Security
External
Performance Tools
External
Accounting
Integrated Tools
Tools
Fault Integrated
Tools
Configuration
ACI Manages Infrastructure through Abstraction
External Connectivity
SLB Configuration
SLB Configuration
Host Connectivity
FW Configuration
FW Configuration
Host Connectivity
Host Connectivity
ACL
ACL
QoS
QoS
QoS
ACL
Network Path
Forwarding QoS QoS QoS
ACI Manages Infrastructure through Abstraction
External Connectivity
SLB Configuration
SLB Configuration
Host Connectivity
FW Configuration
FW Configuration
Host Connectivity
Host Connectivity
ACL
ACL
QoS
QoS
QoS
ACL
Network Path
Forwarding QoS QoS QoS
Application Network Profile abstracts the

logical state from the physical infrastructure
ACI APIC Interfaces
CLI
GUI
REST API ACI Toolkit
REST API
Full Object Model exposed Cobra
Objects
Attributes
Children
Relationships
Relative and Distinguished Names
JSON or XML Ruby
https://github.com/datacenter
Different People, Different Tools
Powerful/Complex
SDK
REST
API
APIC
GUI
APIC
CLI
Simple/Rigid
BRKDCT-2218
2016
2017 Cisco
Ciscoand/or its its
and/or affiliates. All All
affiliates. rights reserved.
rights CiscoCisco
reserved. PublicPublic 21
Custom Dashboards with Ruby SDK
https://github.com/datacenter/aci
Common Ask: Simplify Layer 2 integration
ACI
Brownfields Network
VLAN 100
Fabric
VLAN 200
Cat3750 Cat3750
VLAN 100 VLAN 200
Open
Network Centric Application Plus (NCAplus) Source
Tool to deploy classic VLANs into ACI tailored to customer process Tool
Deploy L2 Networks easily without having to view ACI GUI
Networks
Access
Definitions
Monitoring
https://github.com/datacenter/NCAplus
ACI Network Automation for NetOps
Cisco Open Source Projects for the Data Center
https://github.com/datacenter
ACI SDK Cobra
The complete API SDK for the Cisco APIC
Acitoolkit
A basic toolkit for accessing the Cisco APIC
https://github.com/datacenter/acitoolkit
NCAplus
An application designed for ACI to simplify the creation of L2 networks.
ACI App Centerhttps://aciappcenter.cisco.com/
App Center: New Apps Contract Viewer
VisuDash
Programmability
Think different!
When the shoe doesnt fit, see if its
possible to customize the shoe.
Key Takeaways
Cisco has many options for building data
center networks, you own the key
All solutions can start small and grow
No Cisco solution has to be a rip and
replace
Spine-Leaf does not have to be expensive
Programmable fabrics provide new tools for
simplified operations
Automated fabrics provide new methods of
managing DC Networking
Call to Action
www.ciscolive.com for presentations and recordings

Visit World of Solutions and look for Data Center
Meet the Engineer
Visit DevNet Zone and see what they have to offer!
Q&A
Complete Your Online Session Evaluation
Give us your feedback and receive a
Cisco Live 2017 Cap by completing the
overall event evaluation and 5 session
evaluations.
All evaluations can be completed via

the Cisco Live Mobile App.
Caps can be collected Friday 10 March Learn online with Cisco Live!
at Registration. Visit us online after the conference
for full access to session videos and
presentations.
www.CiscoLiveAPAC.com
Thank you

BRKDCT 2218 PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKDCT 2218 PDF

Uploaded by

Copyright:

Available Formats

Data Centre Design For The Small

Nic Rouhotas Technology Solutions Architect

What is a Small to Medium Business for an Australian market?

Australian SMBs make 97% of all Australian businesses,

NORTH - SOUTH TRAFFIC

Performance: full cross sectional bandwidth FABRIC Storage

EAST WEST TRAFFIC

Mostly virtualized, some baremetal servers

Design Options iSCSI / NAS

Feature choice + priority = tradeoffs

Ring Star Full Mesh Hub

Tree N-Tiered Spine Leaf

DBUS SOC SOC SOC SOC

Linecard Linecard Linecard Linecard Linecard

Design Shifts Resulting from Increasing Gate Density and Bandwidth

10/100M 100M/1G 1G/10G 10G/100G

STP VPC Fabric Path VXLAN

Traditional Overlay Based

OVERLAY e.g. VXLAN, ACI

Collapsed/Single Tier Spine Leaf

Management and Automation?

Cabling the Network Topology

AOC= Active Optical Cable

MMF= Multi Mode Fiber

Traditional Physical Layer Connectivity

Twinax Copper; AOC (10/40G <15m)

Top of Rack Middle of Row End of Row

Intra- Rack Inter Rack Inter Rack

Pictures from jimenez_3bq_01_0711.pdf, 802.3bq Ethernet Alliance

ToR Access/Leaf Switch Model Selection

Downlink ports: 48p Uplink/Breakout ports: 6-12p

Leaf pair with SFP+ downlinks

Leaf pair with RJ-45 downlinks

SFP+ Pluggable Optics

AOC Active Optical Cables

AOC Twinax copper

10G AOC vs 10G Twinax CX1

Shorter link distances with higher data

40G Optical Link (40GBase-SR4)

8 fiber cores (used) 4 Tx/Rx pairs

100G Optical Link (100GBase-SR10)

20 fiber cores (used) 10 Tx/Rx pairs

12-Fiber ribbon cable with Duplex multimode fiber

10G, 40G, 100G

10G, 40G Transceivers 100G Transceivers

SFP SFP+ QSFP+ QSFP28

Pluggable Options Pluggable Options

Host: 1/10/25G Network: 10/40/50/100G

QSFP to SFP for high density SFP

Cisco QSFP to SFP Breakout Examples: 4x10G LR LR

2010Gbps 2010Gbps 2010Gbps

2010Gbps 540Gbps 2100Gbps 20x10G Probability of 100% throughput 3%

5x40G Probability of 100% throughput 75%

2x100G Probability of 100% throughput 99%

Data Center Cabling Architecture Considerations

MDA: Main Distribution Area

Spanning Tree introduced around 1985,

Since then, most DC Designs built to

VPC invented to overcome STP limitations

still cause issues

vPC Northbound & Southbound

Another good workaround

Configuration can become complex as