BRKDCT 2868

Network Integration of
Server Virtualization
with LAN and Storage
Network Implications
& Best Practices
BRKDCT-2868
Bjrn R. Martinussen
BRKDCT-2868
2009 Cisco Systems, Inc. All rights reserved.
Cisco Public
Housekeeping
We value your feedback- don't forget to complete your
online session evaluations after each session & complete
the Overall Conference Evaluation which will be available
online from Thursday
Visit the World of Solutions
Please remember this is a 'non-smoking' venue!
Please switch off your mobile phones
Please make use of the recycling bins provided
Please remember to wear your badge at all times
including the Party
BRKDCT-2868
Cisco Public
Session Objectives
At the end of the session, the participants should
be able to:
Objective 1: Understand key concepts of server
virtualization architectures as they relate to the network.
Objective 2: Explain the impact of server virtualization
on DC network design (Ethernet & Fiber Channel)
Objective 3: Design Cisco DC networks to support
server virtualization environments
BRKDCT-2868
Cisco Public
Network Implications of Server Virtualization &

Best Practices
BRKDCT-2868
Cisco Public
Virtualization
App
App
App
Guest OS
Guest OS
Mofied Stripped
Down OS with
Hypervisor
Hypervisor
Host OS
Guest OS
Modified OS
VMware
Microsoft
Cisco Public
App
VM
Modified OS
Mofied Stripped
Down OS with
Hypervisor
CPU
CPU
CPU
BRKDCT-2868
App
VM
VM
Guest OS
App
XEN aka
Paravirtualization
BRKDCT-2868
Cisco Public
BRKDCT-2868
Cisco Public
VMware Architecture in a Nutshell

Mgmt
Network
App.
App.
Production
Network
VM Kernel
Network
App.
Console
OS
OS
OS
OS
VM Virtualization Layer
Virtual
Machines
Physical Hardware
CPU
m
BRKDCT-2868
em
y
or
ESX Server Host

Cisco Public
VMware HA Clustering
App2
App1
App1
App2
App3
App4
Guest OS
Guest OS
Guest OS
Guest OS
Hypervisor
BRKDCT-2868
em
y
or
App5
Guest OS
Guest OS
Hypervisor
Hypervisor
ESX Host 1
CPU
Guest OS
ESX Host 3
ESX Host 2
CPU
m
Cisco Public
em
y
or
CPU
m
em
y
or
Application-level HA clustering
(Provided by MSCS, Veritas etc)
App1
App2
App3
App4
Guest OS
Guest OS
Guest OS
Guest OS
Hypervisor
BRKDCT-2868
em
y
or
Guest OS
App5
App2
Guest OS
Guest OS
Hypervisor
Hypervisor
ESX Host 1
CPU
App1
ESX Host 3
ESX Host 2
CPU
m
Cisco Public
em
y
or
CPU
m
em
y
or
10
HA + DRS
HA takes care of Powering on
VMs on available ESX hosts in
the least possible time (regular
migration, not VMotion based)
DRS takes care of migrating
the VMs over time to the most
appropriate ESX host based
on resource allocation
(VMotion migration)
BRKDCT-2868
Cisco Public
11
Agenda
VMware LAN Networking
vSwitch Basics
NIC Teaming
vSwitch vs LAN Switch
Cisco/VMware DC DESIGNS
Vmware SAN Designs

VMware Virtual Networking
BRKDCT-2868
Cisco Public
12
VMware Networking Components

Per ESX-server configuration
vSwitch VMNICS = uplinks
VMs
vNIC
vSwitch0
VM_LUN_0007
vmnic0
VM_LUN_0005
vNIC
vmnic1
Virtual Ports
BRKDCT-2868
Cisco Public
13
vNIC MAC Address

VMs MAC address
automatically generated
Mechanisms to avoid MAC
collision
VMs MAC address doesnt
change with migration
VMs MAC addresses can be
made static by modifying the
configuration files
ethernetN.address =
00:50:56:XX:YY:ZZ
BRKDCT-2868
/vmfs/volumes/46b9d79a2de6e23e-929d001b78bb5a2c/VM_LUN_0005
/VM_LUN_0005.vmx
ethernet0.addressType = "vpx"
ethernet0.generatedAddress =
"00:50:56:b0:5f:24
ethernet0.addressType =
static
ethernet0.address =
"00:50:56:00:00:06
Cisco Public
14
vSwitch Forwarding Characteristics

Forwarding based on MAC address (No Learning):
If traffic doesnt match a VM MAC is sent out to vmnic
VM-to-VM traffic stays local
vSwitches TAG traffic with 802.1q VLAN ID
vSwitches are 802.1q Capable
vSwitches can create Etherchannels
BRKDCT-2868
Cisco Public
15
BRKDCT-2868
Cisco Public
16
BRKDCT-2868
Cisco Public
17
BRKDCT-2868
Cisco Public
18
VM Port-Group vSwitch
BRKDCT-2868
Cisco Public
19
VLANs - External Switch Tagging - EST

VM1
VLAN tagging and

stripping is done by the
physical switch
Service
Console
VM2
Virtual NICs
VMkernel
NIC
VSwitch B
VSwitch A
VMkernel
ESX
Server
No ESX configuration
required as the server is
not tagging
The number of VLANs
supported is limited to
the number of physical
NICs in the server
Physical NICs
Physical
Switches
BRKDCT-2868
VLAN 100
VLAN 200
Cisco Public
20
VLANs - Virtual Switch Tagging - VST

VM1
The vSwitch tags

outgooing frames with
the VLAN Id
Service
Console
VM2
Virtual NICs
VMkernel
NIC
VSwitch A
VMkernel
Physical NICs
Physical
Switches
BRKDCT-2868
ESX
Server
The vSwitch strips any

dot1Q tags before
delivering to the VM
Physical NICs and
switch port operate as a
trunk
Number of VLANs are
limited to the number of
vNICs
dot1Q
VLAN 100
VLAN 200
Cisco Public
No VTP or DTP. All

static config. Prune
VLANs so ESX doesnt
process broadcasts
21
VLANs - Virtual Guest Tagging - VGT

VM1
Service
Console
VM2
dot1Q
VM applied
Virtual NICs
VMkernel
NIC
VSwitch A
VMkernel
Physical NICs
Physical
Switches
BRKDCT-2868
Portgroup VLAN Id set to

4095
Guest can send/receive

any tagged VLAN frame
Number of VLANs per
guest are not limited to
the number of VNICs
VMware does not ship
with the driver:
dot1Q
VLAN 100
ESX
Server
Tagging and stripping of

VLAN ids happens in the
guest VM requires an
802.1Q driver
VLAN 200
Windows E1000
Linux dot1q module
Cisco Public
22
Agenda
vSwitch Basics
NIC Teaming
BRKDCT-2868
Cisco Public
23
Meaning of NIC Teaming in VMware (1)

ESX server NIC cards
vSwitch Uplinks
vmnic1
vmnic0
vmnic2
vmnic3
NIC Teaming
NIC Teaming
THIS IS NOT NIC Teaming

vNIC
vNIC
vNIC
vNIC
vNIC
ESX Server Host
BRKDCT-2868
Cisco Public
24
Meaning of NIC Teaming in VMware (2)
This is NOT Teaming
Teaming is Configured at
The vmnic Level
BRKDCT-2868
Cisco Public
25
Design Example
2 NICs, VLAN 1 and 2, Active/Standby
Port-Group 1
VLAN 2
802.1q
Vlan 1,2
802.1q
Vlan 1,2
vmnic1
vmnic0
ESX Server
vSwitch0
Port-Group 2
VLAN 1
VM1
BRKDCT-2868
VM2
Service Console
Cisco Public
26
Active/Standby per-Port-Group
CBS-right
CBS-left
VMNIC1
VMNIC0
Port-Group2
Port-Group1
vSwitch0
VM5
.5
BRKDCT-2868
VM7
.7
ESX Server
Cisco Public
VM4
VM6
.4
.6
27
Port-Group Overrides vSwitch Global

Configuration
BRKDCT-2868
Cisco Public
28
Active/Active
ESX server NIC cards
ESX server
vmnic0
vmnic1
vSwitch
Port-Group
VM1
BRKDCT-2868
VM2
VM3
VM4
VM5
Cisco Public
29
Active/Active
IP-Based Load Balancing
Works with Channel-Group
mode ON
LACP is not supported
(see below):
Port-channeling
9w0d: %LINK-3-UPDOWN: Interface

GigabitEthernet1/0/14, changed state
to up
ESX server
vmnic0
9w0d: %LINK-3-UPDOWN: Interface

GigabitEthernet1/0/13, changed state
to up
vmnic1
vSwitch
Port-Group
9w0d: %EC-5-L3DONTBNDL2:
Gi1/0/14 suspended: LACP currently
not enabled on the remote port.
9w0d: %EC-5-L3DONTBNDL2:
Gi1/0/13 suspended: LACP currently
not enabled on the remote port.
VM1
BRKDCT-2868
Cisco Public
VM2
VM3
VM4
30
Agenda
vSwitch Basics
NIC Teaming
BRKDCT-2868
Cisco Public
31
All Links Active, No Spanning-Tree

Is There a Loop?
CBS-right
CBS-left
NIC3
NIC2
NIC1
NIC4
Port-Group1
Port-Group2
vSwitch1
VM5
.5
BRKDCT-2868
VM7
.7
ESX Server
Cisco Public
VM4
VM6
.4
.6
32
Broadcast/Multicast/Unknown Unicast
Forwarding in Active/Active (1)
802.1q
Vlan 1,2
vmnic0
802.1q
Vlan 1,2
vmnic1
vSwitch0
Port-Group 1
VLAN 2
ESX Server
BRKDCT-2868
VM1
VM2
Cisco Public
33
Broadcast/Multicast/Unknown Unicast
Forwarding in Active/Active (2)
802.1q
Vlan 1,2
ESX Host
NIC1
802.1q
Vlan 1,2
NIC2
vSwitch
VM1
BRKDCT-2868
VM2
VM3
Cisco Public
34
Can the vSwitch Pass Traffic Through?
E.g. HSRP?
NIC1
NIC2
vSwitch
VM1
BRKDCT-2868
VM2
Cisco Public
35
Is This Design Possible?

Catalyst1
Catalyst2
802.1q
802.1q
ESX server1
VMNIC1
VMNIC2
vSwitch
VM5
.5
BRKDCT-2868
VM7
.7
Cisco Public
36
vSwitch Security
Promiscuous mode Reject
prevents a port from
capturing traffic whose
address is not the VMs
address
MAC Address Change,
prevents the VM from
modifying the vNIC
address
Forget Transmits prevents
the VM from sending out
traffic with a different MAC
(e.g NLB)
BRKDCT-2868
Cisco Public
37

Similarly to a LAN Switch:
Forwarding based on MAC
address
VM-to-VM traffic stays local
Vswitches TAG traffic with
802.1q VLAN ID
vSwitches are 802.1q Capable
vSwitches can create
Etherchannels
Preemption Configuration
(similar to Flexlinks, but no delay
preemption)
BRKDCT-2868
Cisco Public
Differently from a LAN Switch

No Learning
No Spanning-Tree protocol
No Dynamic trunk negotiation
(DTP)
No 802.3ad LACP
2 Etherchannel backing up each
other is not possible
No SPAN/mirroring capabilities:
Traffic capturing is not the
equivalent of SPAN
Port Security limited
38
Agenda
vSwitch Basics
NIC Teaming
BRKDCT-2868
Cisco Public
39
vSwitch and NIC Teaming Best Practices

Q: Should I use multiple vSwitches
or multiple Port-Groups to isolate
traffic?
A: We didnt see any advantage in
using multiple vSwitches, multiple
Port-Groups with different VLANs
give you enough flexibility to isolate
servers
Q: Which NIC Teaming

configuration should I use?
A: Active/Active, Virtual Port-ID
based
Q: Do I have to attach all NICs in
the team to the same switch or to
different switches?
Q: Should I use EST or VST?
A: with Active/Active Virtual Port-ID

based, it doesnt matter
A: Always use VST, i.e. assign the

VLAN from the vSwitch
Q: Should I use Beaconing?
Q: Can I use native VLAN for VMs?

A: Yes you can, but to make it
simple dont. If you do, do not TAG
VMs with the native VLAN
BRKDCT-2868
Cisco Public
A: No
Q: Should I use Rolling Failover (i.e.
no preemption)
A: No, default is good, just enable
trunkfast on the Cisco switch
40
Cisco Switchport Configuration

Make it a Trunk
interface GigabitEthernetX/X
Enable Trunkfast
description <<** VM Port **>>
Can the Native VLAN be used for

VMs?
no ip address
Yes, but IF you do, you have 2

options
switchport
switchport trunk encapsulation dot1q
Configure VLAN ID = 0 for the VMs

that are going to use the native VLAN
(preferred)
switchport trunk native vlan <id>
Configure vlan dot1q tag native on

the 6k (not recommended)
switchport mode trunk
switchport trunk allowed vlan xx,yy-zz
Do not enable Port Security

(see next slide)
switchport nonegotiate
Make sure that teamed NICs are in

the same Layer 2 domain
spanning-tree portfast trunk
no cdp enable
Provide a Redundant Layer 2 path
!
Typically: SC, VMKernel, VM Production

BRKDCT-2868
Cisco Public
41
Configuration with 2 NIC

SC, VMKernel, Production Share NICs
Trunks
802.1q
NIC teaming
Active/Active
ESX Server
VMNIC2
VMNIC1
Port-Group
1
802.1q: Production VLANs,

Service Console, VM Kernel
Port-Group Port-Group
3
2
vSwitch 0
Global
Active/Active
VST
VM1
HBA1
BRKDCT-2868
Cisco Public
Service
Console
VM2
VM Kernel
HBA2
Active/Standby
Vmnic1/vmnic2
Active/Standby
Vmnic2/vmnic1
42
Configuration with 2 NICs

Dedicated NIC to SC, VMKernel, Separate NIC for Production
Trunks
802.1q
NIC teaming
Active/Active
ESX Server
VMNIC2
VMNIC1
Port-Group
1
802.1q: Production VLANs,

Service Console, VM Kernel
Port-Group Port-Group
3
2
vSwitch 0
Global
Active/Standby
Vmnic1/vmnic2
VST
VM1
HBA1
BRKDCT-2868
Cisco Public
Service
Console
VM2
VM Kernel
HBA2
Active/Standby
Vmnic2/vmnic1
Active/Standby
Vmnic2/vmnic1
43
Network Attachment (1)

Secondary
root
Rapid PVST+
root
802.1q:
Production,
SC, VMKernel
Trunkfast
BPDU guard
No Blocked Port,
No Loop
Catalyst1
Catalyst2
802.1q:
Production,
SC, VMKernel
802.1q
1
VMNIC1
ESX server1
BRKDCT-2868
2
VMNIC2
vSwitch
VMNIC1
vSwitch
Cisco Public
All NICs are used

Traffic distributed
On all links
VMNIC2
ESX server 2
44

802.1q:
Production, SC, VMKernel
root
Secondary
root
Rapid PVST+
Trunkfast
BPDU guard
Typical Spanning-Tree
V-Shape Topology
802.1q:
Production,
SC, VMKernel
802.1q
1
VMNIC1
2
VMNIC2
vSwitch
VMNIC1
VMNIC2
ESX server 2 vSwitch
ESX server1
BRKDCT-2868
All NICs are used

Traffic distributed
On all links
Cisco Public
45
BRKDCT-2868
Cisco Public
46
BRKDCT-2868
Cisco Public
47
BRKDCT-2868
Cisco Public
48
BRKDCT-2868
Cisco Public
49
BRKDCT-2868
Cisco Public
50
BRKDCT-2868
Cisco Public
51
BRKDCT-2868
Cisco Public
52
BRKDCT-2868
Cisco Public
53
BRKDCT-2868
Cisco Public
54
BRKDCT-2868
Cisco Public
55
BRKDCT-2868
Cisco Public
56
BRKDCT-2868
Cisco Public
57

Secondary
root
root
802.1q:
Production,
SC, VMKernel
Trunkfast
BPDU guard
No Blocked Port,
No Loop
Catalyst1
Catalyst2
802.1q:
SC and VMKernel
802.1q:
Production
6
4
ESX server1
5
ESX server 2
vSwitch
BRKDCT-2868
Rapid PVST+
vSwitch
Cisco Public
58

802.1q:
root
Secondary
root
Rapid PVST+
Trunkfast
BPDU guard
V-Shape Topology
Catalyst2
Catalyst1
802.1q:
Production
802.1q:
SC and VMKernel
BRKDCT-2868
6
4
ESX server1
ESX server 2
vSwitch
vSwitch
Cisco Public
59
How About?
802.1q:
root
Secondary
root
Trunkfast
BPDU guard
V-Shape Topology
Catalyst2
Catalyst1
802.1q:
Production
802.1q:
SC and VMKernel
1
ESX server1
8
ESX server 2
vSwitch
vSwitch
BRKDCT-2868
6
4
Cisco Public
60
4 NICs with Etherchannel

Clustered switches
802.1q:
Production
7
4
vSwitch
vSwitch
ESX server1
BRKDCT-2868
802.1q:
SC, VMKernel
Cisco Public
ESX server 2
61
VMotion Migration Requirements
BRKDCT-2868
Cisco Public
62
VMKernel Network can be routed

VM Kernel
Network
Mgmt VM Kernel Production
Network Network
Network
Virtual
Machines
ESX Server Host

BRKDCT-2868
Cisco Public
63
VMotion L2 Design
Rack1
Rack10
vmnic0
vmnic0
vSwitch0
vmkernel
BRKDCT-2868
vSwitch0
vSwitch2
vSwitch1
vSwitch2
vmkernel
Service
console
ESX Host 2
vmnic2
vmnic2 vmnic3
vmnic1
VM4
Cisco Public
VM5
VM6
ESX Host 1
64
HA clustering (1)
EMC/Legato AAM based
Recommendations:
HA Agent runs in every host
Have 2 Service Console on

redundant paths
Heartbeats Unicast UDP port

~8042 (4 UDP ports opened)
Avoid losing SAN access (e.g. via

iSCSI)
Hearbeats run on the Service

Console ONLY
Make sure you know before hand

if DRS is activated too!
When a Failure Occurs, the ESX

Host pings the gateway (on the
SERVICE CONSOLE ONLY) to
verify Network Connectivity
If ESX Host is isolated, it shuts
down the VMs thus releaseing
locks on the SAN
BRKDCT-2868
Cisco Public
Caveats:
Losing Production VLAN
connectivity only, ISOLATES VMs
(theres no equivalent of uplink
tracking on the vswitch)
Solution:
NIC TEAMING
65
HA clustering (2)
iSCSI access/VMkernel
10.0.200.0
COS 10.0.2.0
Prod 10.0.100.0
vmnic0
vmnic0
VM1
VM2
VM1
ESX2 Server Host
ESX1 Server Host

BRKDCT-2868
VM2
Cisco Public
66
Agenda
VMware SAN Designs
Storage Fundamentals
Storage Protocols
BRKDCT-2868
Cisco Public
67
Multiple ESX ServersShared Storage
BRKDCT-2868
Cisco Public
68
VMFS
VMFS Is High Performance Cluster File System
for Virtual Machines
Stores the entire virtual machine
state in a central location
Virtual Machines
Supports heterogeneous storage

arrays
ESX
Server
ESX
Server
ESX
Server
ESX
Server
VMFS
VMFS
VMFS
VMFS
Adds more storage to a VMFS

volume dynamically
Servers
Allows multiple ESX Servers to

access the same virtual machine
storage concurrently
Storage
Enable virtualization-based
distributed infrastructure services
such as VMotion, DRS, HA
A.vmdk
BRKDCT-2868
Cisco Public
69
Three Layers of the Storage Stack

Virtual
disks
(VMDK)
Virtual Machine
Datastores
VMFS Vols
(LUNs)
Physical
disks
BRKDCT-2868
ESX Server
Storage Array
Cisco Public
70
ESX Server View of SAN

FibreChannel disk arrays appear as SCSI targets
(devices) which may have one or more LUNs
On boot, ESX Server scans for all LUNs by sending
inquiry command to each possible target/LUN number
Rescan command causes ESX Server to scan again,
looking for added or removed targets/LUNs
ESX Server can send normal SCSI commands to any
LUN, just like a local disk
BRKDCT-2868
Cisco Public
71
ESX Server View of SAN (Cont.)

Built-in locking mechanism to ensure multiple hosts can
access same disk on SAN safely
VMFS-2 and VMFS-3 are distributed file systems, do
appropriate on-disk locking to allow many ESX Server servers
to access same VMFS
Storage is a resource that must be monitored and

managed to ensure performance of VMs
Leverage 3rd-party systems and storage management tools
Use VirtualCenter to monitor storage performance from virtual
infrastructure point of view
BRKDCT-2868
Cisco Public
72
Choices in Protocol
FC, iSCSI or NAS?
Best practice to leverage the existing infrastructure
Not to introduce too many changes all at once
Virtual environments can leverage all types
You can choose what fits best and even mix them
Common industry perceptions and trade offs still apply in the
virtual world
What works well for one does not work for all
BRKDCT-2868
Cisco Public
73
Which Protocol to Choose?

Leverage the existing infrastructure when possible
Consider customer expertise and ability to learn
Consider the costs (Dollars and Performance)
What does the environment need in terms of throughput
Size for aggregate throughput before capacity
What functionality is really needed for Virtual Machines

Vmotion, HA, DRS (works on both NAS and SAN)
VMware Consolidated Backup (VCB)
ESX boot from disk
Future scalability
DR requirements
BRKDCT-2868
Cisco Public
74
FC SANConsiderations
Leverage multiple paths for high availability
Manually distribute I/O intensive VMs on
separate paths
Block access provides optimal performance for large
high transactional throughput work loads
Considered the industrial strength backbone for most
large enterprise environments
Requires expertise in storage management team
Expensive price per port connectivity
Increasing to 10 Gb throughput (Soon)
BRKDCT-2868
Cisco Public
75
iSCSIConsiderations
Uses standard NAS infrastructure
Best Practice to
Have dedicated LAN/VLAN to isolate from other network
traffic
Use GbE or faster network
Use multiple NICs or iSCSI HBAs
Use iSCSI HBA for performance environments
Use SW initiator for cost sensitive environments
Supports all VI 3 features

Vmotion, DRS, HA
ESX boot from HW initiator only
VCB is in experimental support today full support shortly
BRKDCT-2868
Cisco Public
76
NFSConsiderations
Has more protocol overhead but less FS overhead than VMFS as
the NAS FS lives on the NAS Head
Simple to define in ESX by providing
Configure NFS server hostname or IP
NFS share
ESX Local datastore name
No tuning required for ESX as most are already defined

No options for rsize or wsize
Version is v3,
Protocol is TCP
Max mount points = 8 by default

Can be increase to hard limit of 32
Supports almost all VI3 features except VCB

BRKDCT-2868
Cisco Public
77
Summary of Features Supported

Protocol
Vmotion,
VCB
DRS & HA
ESX boot
from disk
FC SAN
Yes
Yes
Yes
Yes
Soon
Yes
Yes
Soon
No
Yes
No
No
iSCSI SAN
HW init
iSCSI SAN
SW init
NFS
BRKDCT-2868
Cisco Public
78
Choosing Disk Technologies

Traditional performance factors
Capacity / Price
Disk types (SCSI, FC, SATA/SAS)
Access Time; IOPS; Sustained Transfer Rate
Drive RPM to reduce rotational latency
Seek time
Reliability (MTBF)
VM performance gated ultimately by IOPS density and

storage space
IOPS Density -> Number of read IOPS/GB
Higher = better
BRKDCT-2868
Cisco Public
79
The Choices One Needs to Consider

FS vs. Raw
VMFS vs. RDM (when to use)
NFS vs. Block

NAS vs. SAN (why use each)
iSCSI vs. FC
What is the trade off?
Boot from SAN

Some times needed for diskless servers
Recommended Size of LUN

it depends on application needs
File system vs. LUN snapshots (host or array vs. Vmware VMFS
snapshots) which to pick?
Scalability (factors to consider)
# hosts, dynamic adding of capacity, practical vs. physical limits
BRKDCT-2868
Cisco Public
80
Trade Offs to Consider

Ease of provisioning
Ease of on-going management
Performance optimization
Scalability Head room to grow
Function of 3rd Party services
Remote Mirroring
Backups
Enterprise Systems Management
Skill level of administration team

How many shared vs. isolated storage resources
BRKDCT-2868
Cisco Public
81
Isolate vs. Consolidate

Storage Resources
RDMs map a single LUN to one VM
One can also dedicate a single VMFS Volume
to one VM
When comparing VMFS to RDMs both the above
configurations are what should be compared
The bigger question is how many VM can share a
single VMFS Volume without contention causing pain
The answer is that it depends on many variables
Number of VMs and their workload type
Number of ESX servers those VM are spread across
Number of concurrent request to the same disk sector/platter
BRKDCT-2868
Cisco Public
82
Isolate vs. Consolidate
Poor utilization
Islands of allocations
More management
BRKDCT-2868
Increased utilization
Easier provisioning
Less management
Cisco Public
83
Where Have You Heard This Before

Remember the DAS SAN migration
Convergence of LAN and NAS
All the same concerns have been raised before
What if the work load of some cause problems for all?
How will we know who is taking the lions share of resource?
What if it does not work out?
Our Biggest Obstacle Is Conventional Wisdom!

The Earth Is Flat!
If Man Were Meant to fly He Would Have Wings
BRKDCT-2868
Cisco Public
84
VMFS vs. RDMRDM Advantages

Virtual machine partitions are stored in the native
guest OS file system format, facilitating layered
applications that need this level of access
As there is only one virtual machine on a LUN, you
have much finer grain characterization of the LUN,
and no I/O or SCSI reservation lock contention.
The LUN can be designed for optimal performance
With Virtual Compatibility mode, virtual machines
have many of the features of being on a VMFS, such
as file locking to allow multiple access, and snapshots
BRKDCT-2868
Cisco Public
85
VMFS vs. RDMRDM Advantages

With Physical Compatibility mode, it gives a virtual
machine the capability of sending almost all lowlevel SCSI commands to the target device, including
command and control to a storage controller, such as
through SAN Management agents in the virtual
machine.
Dynamic Name Resolution: Stores unique information
about LUN regardless of changes to physical address
changes due to hardware or path changes
BRKDCT-2868
Cisco Public
86
VMFS vs. RDMRDM Disadvantages

Not available for block or RAID devices that do not
report a SCSI serial number
No snapshots in Physical Compatibility mode, only
available in Virtual Compatibility mode
Can be very inefficient, in that, unlike VMFS, you can
only have one VM access a RDM
BRKDCT-2868
Cisco Public
87
RDMs and Replication

RDMs mapped RAW LUNs can be replicated to the
Remote Site
RDMs reference the RAW LUNs via
the LUN number
LUN ID
VMFS3 Volumes on Remote site will have unusable

RDM configuration if either properties change
Remove the old RDMs and recreate them
Must correlate RDM entries to correct RAW LUNs
Use the same RDM file name as old one to avoid editing the
vmx file
BRKDCT-2868
Cisco Public
88
StorageType of Access
RAW
VMFS
RAW may give better

performance
Leverage templates and

quick provisioning
RAW means more LUNs
Fewer LUNs means you

dont have to watch Heap
More provisioning time
Advanced features still

work
Scales better with

Consolidated Backup
Preferred Method
BRKDCT-2868
Cisco Public
89
StorageHow Big Can I Go?

One Big Volume or Individual?
Will you be doing replication?
More granular slices will help
High performance applications?
Individual volumes could help
With Virtual Infrastructure 3
VMDK, swap, config files, log files, and snapshots all live
on VMFS
BRKDCT-2868
Cisco Public
90
What Is iSCSI?
A SCSI transport protocol, enabling access to storage
devices over standard TCP/IP networks
Maps SCSI block-oriented storage over TCP/IP
Similar to mapping SCSI over Fibre Channel
Initiators, such as an iSCSI HBA in an ESX Server,

send SCSI commands to targets, located in iSCSI
storage systems
Block storage
IP
BRKDCT-2868
Cisco Public
91
VMware iSCSI Overview

VMware added iSCSI as a supported option in VI3
Block-level I/O over TCP/IP using SCSI-3 protocol
Supporting both Hardware and Software Initiators
GigE NiCs MUST be used for SW Initiators (no 100Mb NICs)
Support iSCSI HBAs (HW init) and NICs for SW only today
Check the HCL for supported HW Initiators and SW NICs
What we do not support in ESX 3.0.1

10 gigE
Jumbo Frames
Multi Connect Session (MCS)
TCP-Offload Engine (TOE) Cards
BRKDCT-2868
Cisco Public
92
VMware ESX Storage Options

FC
VM
FC
VM
iSCSI/NFS
VM
FC
VM
DAS
VM
VM
SCSI
80%+ of install base

uses FC storage
iSCSI is popular in SMB
market
DAS is not popular
because it prohibits
VMotion
BRKDCT-2868
Cisco Public
93
Virtual Servers Share a Physical HBA
HW
Hypervisor
Virtual
Servers
A zone includes the physical hba and

the storage array
Access control is demanded to storage
array LUN masking and mapping, it is
based on the physical HBA pWWN and
it is the same for all VMs
The hypervisor is in charge of the
mapping, errors may be disastrous
MDS9000
Mapping
pWWN-P
Zone
BRKDCT-2868
Storage Array
(LUN Mapping and Masking)
FC
FC
pWWN-P
Single Login on a Single Point-to-Point Connection

Cisco Public
FC Name Server
94
NPIV Usage Examples

Intelligent Pass-thru
Virtual Machine Aggregation
FC
FC
FC
FC
FC
FC
FC
Switch becomes an HBA

concentrator
FC
FC
NP_Port
NPIV enabled HBA

F_Port
F_Port
BRKDCT-2868
Cisco Public
95
Raw Device Mapping

RDM allows direct
read/write access
to disk
VM1
Block mapping is still

maintained within a
VMFS file
Rarely used but
important for clustering
(MSCS supported)
FC
VM2
FC
RDM
Mapping
FC
VMFS
Used with NPIV

environments
BRKDCT-2868
Cisco Public
96
Storage Multi-Pathing
No storage load balancing, strictly failover
Two modes of operation dictate behavior
(Fixed and Most Recent)
Fixed Mode
VM
VM
Allows definition of preferred paths

If preferred path fails a secondary path is used
If preferred path reappears it will fail back
FC
FC
Most Recently Used

If current path fails a secondary path is used
If previous path reappears the current path is still used
Supports both Active/Active and Active/Passive

arrays
Auto detects multiple paths
BRKDCT-2868
Cisco Public
97
Q and A
BRKDCT-2868
Cisco Public
98
Recommended Reading
BRKDCT-2868
Cisco Public
99
Agenda
VMware SAN Designs
BRKDCT-2868
Cisco Public
100
Cisco Nexus 1000v Virtual Switch
BRKDCT-2868
Cisco Public
101
VI 3.5 Network Configuration
BRKDCT-2868
Cisco Public
102
BRKDCT-2868
Cisco Public
103
BRKDCT-2868
Cisco Public
104
BRKDCT-2868
Cisco Public
105
BRKDCT-2868
Cisco Public
106
Nexus 1000V Virtual Chassis Model

One Virtual Supervisor Module managing
multiple Virtual Ethernet Modules
Dual Supervisors to support HA environments
A single Nexus 1000V can span multiple ESX

Clusters
SVS-CP# show module

Mod Ports Module-Type
--- ----- --------------------------------1
1
Supervisor Module
2
1
Supervisor Module
3
48
Virtual Ethernet Module
4
48
Virtual Ethernet Module
BRKDCT-2868
Cisco Public
Model
Status
------------------ ---------Cisco Nexus 1000V
Cisco Nexus 1000V
active *
standby
ok
ok
107
Single Chassis Management

A single switch from control plane and management plane perspective
Protocols such as CDP operates as a single switch
XML API and SNMP management appears as a single virtual
chassis
Upstream-4948-1#show cdp neighbor
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone
Device ID
Platform
Holdtme
Capability
Gig 1/5
136
Nexus
Gig 1/10
136
Nexus
Gig 1/12
136
Nexus
Port ID
N1KV-Rack10
1000V
Eth2/2
N1KV-Rack10
1000V
Eth3/5
N1KV-Rack10
1000V
Eth21/2
BRKDCT-2868
Local Intrfce
Cisco Public
108
BRKDCT-2868
Cisco Public
109
BRKDCT-2868
Cisco Public
110
BRKDCT-2868
Cisco Public
111
BRKDCT-2868
Cisco Public
112
BRKDCT-2868
Cisco Public
113
BRKDCT-2868
Cisco Public
114
BRKDCT-2868
Cisco Public
115
Deploying the Cisco Nexus 1000V

Collaborative Deployment Model
Server N
1.
VMW Virtual Center &

Cisco Nexus 1000V
relationship established
2.
Network Admin configures

Nexus 1000V to support
new ESX hosts
3.
4.
Nexus 1000VNexus
DVS1000V
Nexus
1000V -VEM
-VEM
Nexus 1000V
1000V -VEM
-VEM
Nexus
VMW ESX
ESX
VMW
Server Admin plugs new

ESX host into network &
adds host to Cisco DVS
with Virtual Center
Repeat step 3 to add
another host & extend
DVS configuration
VMW ESX
ESX
VMW
4.
Nexus 1000V
1000V
Nexus
Virtual Center
BRKDCT-2868
Server 1
Cisco Public
VSM
VSM
116
Introduction to Port Profiles

Port Profiles are a collection interface commands
switchport mode access
switchport access vlan 57
no shutdown
Applied at the interface level using to either physical or virtual
interfaces
Dynamic configuration
Port Profile changes are propagated immediately to all ports
using that profile
Interfaces can be configured manually in conjunction with a profile
BRKDCT-2868
Cisco Public
117
Port Profiles Propagation

Port profiles are pushed via the Virtual Center API
Upon connection/reconnection with Virtual Center the
VSM re-verifies the correct port profile configuration
exists within Virtual Center
Port profile state and type must be set for
propagation to occur
N1K-CP(config-port-prof) state enable
N1K-CP(config-port-prof) vmware port-group (optional name)
BRKDCT-2868
Cisco Public
118
Policy Based VM Connectivity

What can a policy do?
Policy definition supports:
VLAN, PVLAN settings
ACL, Port Security, ACL
Redirect
Server
VM
VM
#1
#1
VM
VM
#2
#2
VM
VM
#3
#3
VM
VM
#4
#4
Nexus 1000 DVS
NetFlow Collection
VMW ESX
ESX
VMW
Rate Limiting
QoS Marking (COS/DSCP)
Remote Port Mirror (ERSPAN)
Virtual Center
BRKDCT-2868
Cisco Public
Nexus 1000
119
BRKDCT-2868
Cisco Public
120
BRKDCT-2868
Cisco Public
121
BRKDCT-2868
Cisco Public
122
VMware Administrator View

Consistent Workflow:
Continue to select Port
Groups when configuring
a VM in VMware Virtual
Infrastructure Client
BRKDCT-2868
Cisco Public
123
BRKDCT-2868
Cisco Public
124
BRKDCT-2868
Cisco Public
125
BRKDCT-2868
Cisco Public
126
Mobility of Security & Network Properties

1.
2.
3.
Virtual Center kicks off

a Vmotion
(manual/DRS) &
notifies Nexus 1000V
During VM replication,
Nexus 1000V copies
VM port state to new
host
VM
VM
#1
#1
VM
VM
#2
#2
VM
VM
#4
#4
VM
VM
#5
#5
VM
VM
#6
#6
VM
VM
#7
#7
VM
VM
#8
#8
Nexus 1000V
1000V DVS
DVS
Nexus
Nexus 1000 -VEM
Nexus 1000 -VEM
VMW ESX
ESX
VMW
Once VMotion
completes, port on new
ESX host is brought up
& VMs MAC address
is announced to the
network
VM
VM
#3
#3
Nexus 1000V
1000V -VEM
-VEM
Nexus
VMW ESX
ESX
VMW
3.
Nexus 1000V
1000V
Nexus
Network Update
Virtual Center
BRKDCT-2868
Server 2
Server 1
Cisco Public
ARP for VM1 sent to

network
Flows to VM1 MAC
redirected to Server 2
VSM
VSM
127
BRKDCT-2868
Cisco Public
128
Nexus 1000V Deployment Scenarios

Rack Optimized
Servers
1. Works with all types of

servers (rack
optimized, blade
servers, etc.)
Blade Servers
2. Works with any type of

upstream switch
(Blade,
Top or Rack, Modular)
3. Works at any speed
(1G or 10G)
4. Nexus 1000V VSM can
be deployed as a VM or
a physical appliance
Nexus 1000V
Virtual Center
BRKDCT-2868
Cisco Public
VSM
129
BRKDCT-2868
Cisco Public
130
BRKDCT-2868
Cisco Public
131
BRKDCT-2868
Cisco Public
132
Virtual Supervisor Options

Server 1
VM
#1
VM
#2
VM
#3
Server 2
VM
#4
VM
#5
VM
#6
VM
#7
Server 3
VM
#8
VM
#9
VM
#10
VM
#11
VEM
VEM
VEM
VMW ESX
VMW ESX
VMW ESX
VM
#12
VSM
VSM
VSM
VSM Virtual Appliance

ESX Virtual Appliance

Special dependence on CPVA
server
Supports up to 64 VEMs
BRKDCT-2868
VSM Physical Appliance
Cisco Public

Cisco branded x86 server

Runs multiple instances of the
VSM virtual appliance
Each VSM managed

independently
133
Distributed Switching
Each Virtual Ethernet Module behaves like an independent switch
No address learning/synchronization across VEMs
No concept of Crossbar/Fabric between the VEMs
Virtual Supervisor is NOT in the data path
No concept of forwarding from an ingress linecard to an egress
linecard (another server)
No Etherchannel across VEMs
Nexus 1000V
VSM
BRKDCT-2868
VEM
VEM
VEM
VMW ESX
VMW ESX
VMW ESX
Cisco Public
134
Switching Interface Types

Physical Ethernet Ports
NIC cards on each server
Appears as Eth interface on a specific module in NX-OS
Example Eth10/7
Static assignment as long as the module ID does not change
Up to 32 per host
Virtual Ethernet Ports

Virtual Machine facing ports
Appears as Veth within NX-OS.
Not assigned to a specific module to simplify VMotion
BRKDCT-2868
Cisco Public
135
Tracing Virtual Ethernet Ports

show interface VEthernet
Vethernet2 is up
Hardware is Virtual, address is 0050.5675.26c5
Owner is VMware VMkernel, adapter is vmk0
Active on module 8, host tc-esx05.cisco.com
VMware DVS port 16777215
Port-Profile is Vmotion
Port mode is access
Rx
444385 Input Packets 444384 Unicast Packets
0 Multicast Packets 1 Broadcast Packets
572675241 Bytes
Tx
687655 Output Packets 687654 Unicast Packets
0 Multicast Packets 1 Broadcast Packets 1 Flood Packets
592295257 Bytes
0 Input Packet Drops 0 Output Packet Drops
BRKDCT-2868
Cisco Public
136
Manageability and Scalability Details

RBAC
Cisco Nexus 1000V Virtual

Supervisor Module: Virtual
appliance in VMDK or ISO
image, supports up to 64
VMware ESX or ESXi
Wireshark
ERSPAN
LLDP, CDP
Cisco Nexus 1000V Virtual

Ethernet Module: maximum
256 ports
EEM
Rollback
BRKDCT-2868
Cisco Public
137
Server Virtualization Key Takeaways

What you should takeaway from this session:
Ability to explain the key concepts of server
virtualization and know the key players in the market.
Ability to explain to customers key network design
criteria which must be considered when deploying
server virtualization
Ability to recommend network and storage best
practices associated with deploying server virtualization
technologies.
BRKDCT-2868
Cisco Public
138
Meet The Expert

To make the most of your time at Cisco Networkers 2009,
schedule a Face-to-Face Meeting with a top Cisco Expert.
Designed to provide a "big picture" perspective as well as
"in-depth" technology discussions, these face-to-face
meetings will provide fascinating dialogue and a wealth of
valuable insights and ideas.
Visit the Meeting Centre reception desk located in the
Meeting Centre in World of Solutions
BRKDCT-2868
Cisco Public
139
Whitepapers and blogs

http://www.cisco.com/go/nexus1000v/
http://www.cisco.com/en/US/products/ps9670/prod_whi
te_papers_list.html
The Role of 10 Gigabit Ethernet in Virtualized Server
Environments
http://blogs.vmware.com/networking/
http://www.cisco.com/go/datacenter/ look for VMware
Infrastructure 3 in a Cisco Network Environment
http://www.cisco.com/en/US/docs/solutions/Enterprise/
Data_Center/vmware/vmware.html
BRKDCT-2868
Cisco Public
140
BRKDCT-2868
Cisco Public
141

BRKDCT 2868

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKDCT 2868

Uploaded by

Copyright:

Available Formats

Network Integration of

2009 Cisco Systems, Inc. All rights reserved.

2009 Cisco Systems, Inc. All rights reserved.

2009 Cisco Systems, Inc. All rights reserved.

Network Implications of Server Virtualization &

2009 Cisco Systems, Inc. All rights reserved.

2009 Cisco Systems, Inc. All rights reserved.

2009 Cisco Systems, Inc. All rights reserved.

2009 Cisco Systems, Inc. All rights reserved.

VMware Architecture in a Nutshell

2009 Cisco Systems, Inc. All rights reserved.

ESX Server Host

2009 Cisco Systems, Inc. All rights reserved.

2009 Cisco Systems, Inc. All rights reserved.

2009 Cisco Systems, Inc. All rights reserved.

Vmware SAN Designs

2009 Cisco Systems, Inc. All rights reserved.

VMware Networking Components

vSwitch VMNICS = uplinks

2009 Cisco Systems, Inc. All rights reserved.

vNIC MAC Address

2009 Cisco Systems, Inc. All rights reserved.

vSwitch Forwarding Characteristics

2009 Cisco Systems, Inc. All rights reserved.

2009 Cisco Systems, Inc. All rights reserved.

2009 Cisco Systems, Inc. All rights reserved.

2009 Cisco Systems, Inc. All rights reserved.

2009 Cisco Systems, Inc. All rights reserved.

VLANs - External Switch Tagging - EST

VLAN tagging and

2009 Cisco Systems, Inc. All rights reserved.

VLANs - Virtual Switch Tagging - VST

The vSwitch tags

The vSwitch strips any

2009 Cisco Systems, Inc. All rights reserved.

No VTP or DTP. All

VLANs - Virtual Guest Tagging - VGT

Portgroup VLAN Id set to

2009 Cisco Systems, Inc. All rights reserved.

Guest can send/receive

Tagging and stripping of

2009 Cisco Systems, Inc. All rights reserved.

Meaning of NIC Teaming in VMware (1)

THIS IS NOT NIC Teaming

ESX Server Host

2009 Cisco Systems, Inc. All rights reserved.

Meaning of NIC Teaming in VMware (2)

This is NOT Teaming

2009 Cisco Systems, Inc. All rights reserved.

2009 Cisco Systems, Inc. All rights reserved.

2009 Cisco Systems, Inc. All rights reserved.

Port-Group Overrides vSwitch Global

2009 Cisco Systems, Inc. All rights reserved.

ESX server NIC cards

2009 Cisco Systems, Inc. All rights reserved.

9w0d: %LINK-3-UPDOWN: Interface

9w0d: %LINK-3-UPDOWN: Interface

2009 Cisco Systems, Inc. All rights reserved.

2009 Cisco Systems, Inc. All rights reserved.

All Links Active, No Spanning-Tree

2009 Cisco Systems, Inc. All rights reserved.

2009 Cisco Systems, Inc. All rights reserved.

2009 Cisco Systems, Inc. All rights reserved.

description << VM Port >>