Redefining Data Centre Switching

Enabling Stateful vMotion with VXLAN

Marcin Mazur
Sales Manager – CEE
CCIE #24210

+48 601 51 3331

mmazur@aristanetworks.com
A bit about Arista Networks

10GbE Switches for the

Virtualized Datacenter

>1300 Customers
>400 Employees
Profitable, self-funded, pre-IPO
network infrastructure provider
Open Linux-based OS
Fully automated testing, and
SW development
NETWORKING
VIRTUAL
NETWORKING
 ECMP VXLAN OVF
802.1q Flat
 Why vMotion?
•  Conduct maintenance without business disruption

•  Migrate VMs away from failing or underperforming servers

•  Align resources with business priorities

•  Optimize loading

•  Follow the sun?

 Traditional Stateful vMotion

128.218.10.x

128.218.10.4 128.218.10.4
Non-Stateful vMotion Across L3 Subnets

128.218.10.x 128.218.11.x

•  Breaks TCP Sockets

•  NFS Mounts Go Away

•  Reachability?

128.218.10.4 128.218.11.4
So Today, We Build Large L2 Networks!

vswitch
vswitch
vswitch

l  Use VLAN tags to segregate customer traffic It Doesn’t

l  Use Spanning Tree to create loop-free topologies
Scale!

l  Multi-vendor, standards-based, proven technology

l  What could go wrong?

So Today, We Build Large L2 Networks!

vswitch
vswitch
vswitch

l  Number of customer domains limited to 4094

l  Number of VMs constrained by MAC table limits
It Doesn’t
l  Limited multipathing support
Scale!

l  Difficult to stretch across data centers
So Today, We Build Large L2 Networks!

vswitch
vswitch
vswitch

What about Trill? Fabrics?

l  Forklift Upgrades!
l  Single-vendor Proprietary Solutions

l  Doesn’t Solve Some Of The Key Problems

Quick Recap So Far
There is a new technology
called VXLAN that we will get to
eventually (promise!)
Smart network people hate large
layer-2 networks
Some vendors want to push
large layer-2 networks so they
can force equipment refresh
vMotion is best if it can preserve
connection state
What Virtualization Admins Really Want!

128.218.10.x 128.218.11.x

128.218.10.4 128.218.10.4
Why L3 Stateful vMotion Won t Work - SORRY!
You must preserve the IP
Address to preserve the TCP
sockets

DNS ‘tricks’ don’t cut it - DNS

is cached on most clients

Mobile IP Solutions may help

solve outbound traffic, but how
do you know where to send
return path?
 Virtual eXtensible LAN

128.218.10.x 128.218.11.x

Outer: 128.218.11.2

128.218.12.4 128.218.12.5 128.218.12.6

Inner: 128.218.12.1
 VXLAN Logical View 128.218.11.1

128.218.11.0/24

Outer: 128.218.11.2

128.218.12.4 128.218.12.5 128.218.12.6

Inner: 128.218.12.1

128.218.12.0/24
 How does unicast forwarding work?

11.2 MAC&IP are UDP Encapsulated 10.67

128.218.12.4 128.218.12.6
11.1 9.1
11.2 L2 9.2 10.1 10.67

Encapsulation is transparent to traditional switch/router nodes

 What about broadcasts/unknown?

Outer: 128.218.11.2

128.218.12.4 128.218.12.5 128.218.12.6

Inner: 128.218.12.1

128.218.12.0/24 = 224.0.0.12

Multicast has several advantages - more groups, no new

hardware, well proven in last decade
 VXLAN Framing Format

Outer MAC Header

Outer DMAC
Outer SMAC
Ethertype (Opt)
OVLAN
Ethertype 0x800

Outer IP Header
Fragment Header
Version
IHL
TOS
Length
ID
Flags
TTL
Protocol
Outer SA
Outer DA

Offset
Checksum

Outer UDP Header

DST Port =
SRC Port = xxxx
UDP Length
UDP Checksum

VXLAN Port

VXLAN Header
8 Reserved Flags
Reserved (24)
24-Bits VXLAN Network Identifier : (VNI)
Reserved (8)

VXLAN Interesting Points
Uses UDP to encapsulate, inner
protocol controls reliable delivery
Uses Multicast to replicate for
broadcast/unknown forwarding -
leverages PIM and IGMP pruning for
traffic management
Segment ID is 24 bits, supports
16.7M unique groups
Overcomes MAC and VLAN table
limitations on ToR switches
Works with most/all existing
switching implementations from
major vendors
 With VXLAN... It’s a L3 World!
L3
L3
L3
L3

L3
L3
L3

vswitch
vswitch
vswitch

l  Number of customer domains limited to 4094 16,777,216

Number of VMs NOT constrained by MAC table limits
l 
It Scales!

l  Limited FULL multipathing support
l  Difficult Easy to stretch across data centers
 VXLAN means I can
put any VM, on any
server, in seconds,
software provisioned,
without forklifting my
network
 What do Cloud Networks Look Like?
Choice of Large Deep Buffers for
All Links Active L2 or Massive L3 Congestion
Scale Management

Cabinet/Subnet 1 Cabinet/Subnet 2 Cabinet/Subnet 94

New Cabinets
VMware
Server 1
Server 1 can be1 deployed
Server
Wire Speed for
Mgmt Node
with Zero Touch
Integration Intra-Rack
Server 2 Server 2 Server 2
Performance

Servers can be
bare-metal
Scale
Server 20
from 10 Server 48
provisioned
Server 48
from
Dual-Homed servers to network
Host Support for 100,000 Servers
Resilience
How do we
troubleshoot a
tunneled, encapsulated,
multicast
environment???
VM Tracer for VXLAN
Full physical to virtual visibility

Network audit to ensure

reachability

Automated provisioning

Workflow without finger

pointing

Other awesome capabilities

 VM Tracer - Multi-Tenancy
Arista EOS can be connected to multiple vCenter
instances

Each vCenter instance can be assigned a non-

overlapping VLAN range that is supported for
Adaptive Segmentation Pvt Cloud

Lab
Prod

The attempted creation of a VM outside of the
allowed VLAN range creates an alert/alarm in
vCenter RANGE 100-199 RANGE 200-399 RANGE 400-499

esx1 esx2 esx3

 VM Tracer - Host Discovery
show vmtracer interface host

Ethernet46 : esx-1.aristanetworks.com

Manufacturer: Dell Inc.

Model: PowerEdge 2950

CPU type: Intel(R) Xeon(R) CPU 5110 @ 1.60GHz

CPUs : 1

CPU Cores: 2

NIC Manufacturer: NetXen

NIC Model: NetXen NX3031 Dual Port SFP+ 10GbE Server Adapter

Service Tag: ABCDEF1234

Eth47
esx1 esx2 esx3
 VM Tracer - VM Discovery

show vmtracer interface Ethernet48

Ethernet48: esx1.aristanetworks.com/ndsTest/dvuplink1!
!
Switchport
Host/Domain
vSwitch/Uplink



!
VM Name        Network Adapter     VLAN Status State

-------------------------------------------------------

Eth47

Eth48
Exchange       Network adapter 4   7    up/up --

Apache      Network adapter 3   6    up/up vMotion

MySQL          Network adapter 1   5    up/up FT-A! dvuplink0
dvuplink1

vSwitch

VM Name
Adapter Name
VLAN/Status/State

Exchange Apache MySQL

VLAN 5 VLAN 6 VLAN 7

esx1
 VM Tracer - VM Adaptive Segmentation
VM Tracer automatically creates, prunes, and un-prunes VLANs on 802.1q
VLAN Trunks to ESX Hosts. Routed subnets are not auto-created for IP
stability.
VM Tracer Operation

VLAN
VLAN2,6,7
2,7 VLAN 2,5,6,7

PERMITVLAN
PERMIT VLAN2,6,7
2,7 PERMIT VLAN 2,5,6,7

VMK0 - VLAN2 VMK0 - VLAN2

MySQL Exchange Apache Apache MySQL

VLAN 7 VLAN 5 VLAN 6 VLAN 6 VLAN 7

 Automated Provisioning of VXLAN
Create new VNI Segment

Add Multicast Route

IVMWARE
Setup Rendezvous Point

128.218.10.x 128.218.11.x

VNI Test: 224.0.0.12

 spine0: show vmtracer vxlan all

Visibility - so where
is my VM now?
leaf1 : Ethernet 47:dvUplink0/vxtest/esx10

!VNI:Test : mcast-addr 224.0.0.12!
vm-blue/vnic1: 128.218.12.7!
: Ethernet 48:dvUplink1/vxtest/esx10!
VNI:Test : mcast-addr 224.0.0.12!
vm-orange/vnic0 : 128.218.12.5

leaf2 : Ethernet 24:dvuplink1/vxtest/esx11!
VNI:Test : mcast-addr 224.0.0.12

!vm-tiger/vnic1 : 128.218.12.4!
: Ethernet 64:dvuplink0/vxtest/esx11!
spine0 VNI:Test : mcast-addr 224.0.0.12!
vshield/vnic0 : 128.218.12.1!
leaf1 Global Address! leaf2
vshield/vnic1 : 128.218.11.2

128.218.10.x 128.218.11.x

esx10 esx11
VNI ‘Test’: 224.0.0.12

vm-blue vm-orange vshield vm-tiger

 The Role of the Physical Switch
l  Network virtualization works with your exiting layer-3
switches!

l  Provide tunnel termination for non-virtualized endstations and/
or for highest performance

l  Provide tunnel-aware QOS, ACL, visibility

l  Integrate with virtualization manager for improved manageability

l  Provide the best physical infrastructure for
virtualized environments

Summary
Arista simply provides the best network for VMware

Native vSphere integration into the physical network

to automate, and operationalize the infrastructure

Simplified systems architecture to reduce

complexity and improve TCO

Optimized power and efficiency to maximize

compute density and enable profitable public, or
cost-effective private cloud deployments

Co-authors and innovators of VXLAN to enable

stateful L3 vMotion