VMware & Storage

VMware&Storage

Presented By:
Marc Staimer, President & CDS
Dragon Slayer Consulting
marcstaimer@mac.com
503-579-3763
 DragonSlayerConsultingIntro

M
MarcStaimer
St i P id t & CDS
President&CDS
12+years
Storage,SANS,SW,Networks,Servers
Consultsvendors(>100)
Consults end users (> 400)
Consultsendusers(>400)
Analysisattradeshows marcstaimer@mac.com
503-579-3763
PublishesconsistentlywithTechTarget
Publishes consistently with Tech Target
Periodicallypublishedfortrademagazines
30+yearsindustryexperience
y y p

Oct10 NSG 2
 PictureisWorth
a 1000 Words
a1000Words In the course of my consultancy practice Im often asked
H
How do
d I define
d fi a sonofabitch?
f bit h?

I wax philosophic with metaphysical postulations,

incomplete aphorisms, & inconsistent sophisms that make
me absolutely positive that a picture is truly worth a
thousand words.

So, in this photo, the guy on the right is a member of a

bomb squad
q in the midst of a deactivation.

The guy behind him, well hes a sonofabitch.

Oct10 NSG 3
 Agenda

RealWorldLevelSetting
Real World Level Setting
HowServerVirtualizationWorkswithStorage
Server Virtualization Storage Gotcha Issues
ServerVirtualizationStorageGotchaIssues
ServerVirtualizationDataProtectionIssues
Howtoresolve

Oct10 NSG 4
 WhyServerVirtualization?

GreaterApplicationAvailability
F t /i l
Faster/simplerservermanagement t
Faster/simplerDR&BusContinuity
Serverconsolidation

Oct10 NSG 5
 GreaterApplicationAvailabilityVMware
Reducesoreliminatesscheduleddowntime
Vmotion,StorageVmotion,ThinProvisioning

Oct10 NSG 6
 FasterSimplerServerManagement
Easiertoimplement,provision,operate,manage

Oct10 NSG 7
 FasterSimplerDR&BusContinuity

Betterabletohandleunscheduleddowntime
LocalorRemote(SRM)

Oct10 NSG 8
 ServerConsolidation
Businesscasetojustifytheabove
Less
Servers
NICsand/orHBAs
Cables
bl
Switchports
Switches
Rackspace
Floorspace
p
Power
Cooling
LessCapEx&OpEx
Oct10 NSG 9
 vSphere4.1=CloudScale
32hosts/cluster
320VMs/host
3,000VMs/cluster(2x)
1,000hosts/vCenter(>3x)
10,000VMs/vCenter(>3x)
15,000registerVMs/vCenter(>3x)

Oct10 NSG 10
 SeriouslyEnhancedScalabilityinOtherWaysToo

vSphere4
vSphere 4 vSphere 4 1
vSphere4.1 Ratio
ConcurrentVIClients 30 120 4x
HostsperDC
p 100 500 5x
VMsperDC 2,500 5,000 2x
LinkedMode 10,000 30,000 3x
ConcurrentvMotionsperhost 2@upto2.67GBps 8@8GBps ~4x
Hostedfootprint&memconsumption NA 40% less 3x>IOPS
Transparent memory compression
Transparentmemory NA > 15% perf increase
>15%perfincrease >1.15x

Oct10 NSG 11
11
 TheSecretofLife

Oct10 NSG 12
 SignificantVMwarevStorageImprovements
VMwarevSpherereallyenhancedvStorage
VirtualDiskThinProvisioning
Vi l Di k Thi P i i i
ImprovediSCSISoftwareInitiatorEfficiency
vCenterServerStorageEnhancements
EnhancedStorageVmotion
DynamicExpansionofVMFSVolumes
vStorageAPIsforMultipathing

Oct10 NSG 13
 VMwareStorageSystemOptions

Rawdevicemapping
R d i i SAN
VMFSsharedblock DASorSAN
SharedNFSstorage NAS

14 NSG
Oct10
 RawDeviceMapping(RDM)
GuestVMsaccessstoragedirectlyoveriSCSIorFC
VMscanevenbootfromrawdevices
Great
Perserverqueuesforperformance
Easier measurement
Easiermeasurement
Methodologyforclustering
Virtualtovirtualorvirtualtophysical
B t
But
TrickyVMotionandDRS
NostorageVmotion
NoSRM
Alotmoremanuallyintensivemgmtoverhead
Limitedto256LUNsperdatacenter
p

15 NSG
Oct10
 VMFS(VirtualMachineFileSystem)
Shared Block Storage
SharedBlockStorage

Sharedstorage common/workstationapproach
StoresVMDKimageinVMFSdatastores
DASorFC/iSCSISAN
DominantVMwareStorage
VMFSisVMwaresownFS
Develops1stonVMFS
Traditional,familiar,mature,robust
VM
VMwareadminindependence
d i i d d
Primefeatures(StorageVMotion,etc)
Multipathing,loadbalancing,failover
Multipathing load balancing failover
16 NSG
Oct10
 HowServerVirtualizationWorksw/SANStorage
StorageLUNsare
St LUN
VirtualizedandsharedbyVirtualServer

Oct10 NSG 17
 LUNOversubscription
StoragesystemscantdistinguishbetweenVMs
EachVMcontestsforsameHDDs
Contentionseriouslydecreasesperformance
ThereisnoVMQoStoday

Traditional SAN Storage

Virtual Storage

Oct10 NSG 18
 Consequences EspeciallyWithSATAHDDs
SlowerSATAdrivesdonthandlecontentionwell
Nominalbuffersorques=higherresponsetimes

FC/SAS

Que depth of 0 to 32
SATA
Usually 0
7,200 RPM Que depth of 256 to 512
5,400 RPM 15,000 RPM
10,000 RPM
,
7,200 RPM

Ends up consuming
the controllers que

Oct10 NSG 19
 vSphere4.1StorageIOControl(a.k.a.QoS)

Solves some IO contention issues

SolvessomeIOcontentionissues
UserdefinableQoSprioritization
Enables/disabled per specific datastore
Enables/disabledperspecificdatastore
Tiedtodatastorelatencythresholds
Exceeds threshold then congestion is assumed
Exceedsthreshold,thencongestionisassumed
AtthatpointVMsareprioritizedbypolicy
AdminscanalsodefineamtofIOPSreachable/VM
/

Oct10 NSG 20
 OtherProblematicVMwareVMFSStorageIssues

StorageProvisioning
St P ii i
Toomuchoversubscription
Painfulscaling
DataProtection

Oct10 NSG 21
 StorageProvisioning

VM&applicationdisruptiveonstorageside
VM & li ti di ti t id
Timeconsuming
Manuallyintensive
ThecompleteoppositeofVMprovisioning
p pp p g
Requiresknowledge,expertise,&knowhow
RAIDgroupsetup,DP,snapshots,replication,etc.
g p p, , p , p ,

Oct10 NSG 22
 KeyIssue:PerformanceTroubleshooting
VMAppperformancedeclinesnoticeably
Whenmovingfromphysicaltovirtualservers
When moving from physical to virtual servers
OftencausingfruitlessVMmigrations
Lotsofadminfrustrationlookingforrootcause
Lots of admin frustration looking for root cause
Evenmoretroublingwhenproblemisintermittent
Significant potential for VM SCSI timeouts
SignificantpotentialforVMSCSItimeouts
Missioncritical VMsrequireownuniqueLUNs

Oct10 NSG 23
 OftenComesFromTooMuchOversubscription

Toomuchwithinthe:
Hypervisor
LUN
HDDs
SANfabric
TargetStorageports
C t iti l
Createspositiveloop
Problemsfeedonthemselves
Findingrootcausecanbeahairpullingexperience
d g oot cause ca be a a pu g e pe e ce
Especiallyintermittent excessivetimeconsumption
Mustcapture/trapproblemwhileitsoccurring
Which means it has to occur again
Whichmeansithastooccuragain

Oct10 NSG 24
 StorageSystemScalingStillPainful

ManuallyIntensiveplanning
M ll I t i l i
Storagetypicallyscalesup,notout
Ripoutandreplace
Ongoingdatamigration
Appdisruptive

Oct10 NSG 25
 AMajorP.I.T.R.forLargeEnvironments

SpecificallywhenVM#exceedsdoubledigits
Timeconsumingpedanticmanuallyintensivetasks
gp y
SettingLUNassignments&RAIDlevels
LUNmanagement
MostSANstorageLUNexpansionismanuallyintensive

Oct10 NSG 26
 WaystoAvoidTheseVMFSStorageIssues

Storage that
Storagethat
Isaseasytoprovision,change,expandasVMs
Makes troubleshooting easy
Makestroubleshootingeasy
Requiresminimaltonoexpertise
To implement operate and manage
Toimplement,operate,andmanage
HelpsavoidLUNoversubscriptionissues
There are 2 types
Thereare2types
VerysimpleiSCSIstorage
Andthemostoverlooked&incrediblyeasyNFS
y y

Oct10 NSG 27
 VMwareSharedNFSStorage
IncrediblyeasyprovisioningsinceallVMDKsarejustfiles
Incredibly easy provisioning since all VMDKs are just files
VMwareVMDKthinprovisioning
vSphere4defaultsetting
Si l
SimpleonlineNFSdatastore&perf.scaling
li NFS d & f li
Effectivew/simpledatastorerefreshbutton
Perf.Increasesw/easytrunkingIEEE802.3ad
0mgmtororneedforVMFSorRDM
EliminatesconstrainedorsingleIOqueues
Performanceltdonlybyserver,NAS,orBWIO
Greatlysimplifiesmgmt,ops,&changemgmt
EliminatesFCswitch,zone,hardzone,HBAmgmt
EliminatesLUNmasking,oridenticalLUNIDs
g,
ImprovesdataprotectionVMBUR
Increasedgranularity(wholeVMsorfileswithinVMs)
OptimizedrandomIOresponsetime
Optimized random IO response time idealforESX&vSphere
ideal for ESX & vSphere
SinglemountacrosspointacrossmultipleIPaddresses
28 NSG
Oct10
 HowTheDifferentStorageOptionsCompare
Whichonelookseasiest?

Shared Storage Local Storage

(Clustered) (Stand Alone)

VMFS
(Formatted)

NAS/NFS( VMFS VMFS RDMs

File Based) (On iSCSI) (On FC) (On FC or iSCSI)

Physical Mode Virtual

iSCSI HBA SW Initiator Pass--Through
Pass Mode

Persistent Undoable
29 NSG
Oct10
 HowNFSComparestoVMFS

Set upNFSdatastoreonNAS
upNFSdatastoreonNAS
AssigntoVCenter Set upLUN
S
Setuptrunkingonserver
ki Set upRAIDset
Andnetworkswitch AssignLUNtoVMFS
Virtualize LUN
AssignvirtualLUNstoVMs
Thin provision virtual LUNs
ThinprovisionvirtualLUNs
SetupFCoriSCSImultipathing
SetupFCswitchzones
SetupFCswitchhardzones
SetuptwoSANs
For load balancing
Forloadbalancing
Forchangemgmt

30 NSG
Oct10
 NFSDatastoresSimplerCapacityExpansion

IncreasestorageonNFSside HaltIOtotheLUN
Refresh each VMssstorage
RefresheachVM storage I t SAN id
IncreasestorageonSANside
Usenewlyprovisionedstorage Increasedatastoresize
Includingunderlyingpartition
RescaneachVMconnected
T SAN t
ToSANstorage
ResumeIOtoLUN
Usenewlyprovisionedstorage

31 NSG
Oct10
Oct10 NSG 32
VMwareInfrastructureNFSMyths
VMwaredoesntfullysupportNFS
O ll f
OrallfunctionsonNFS
i NFS
NFSistooslowforVMware
F
FewuseNFSw/VMwareinproduction
NFS /VM i d ti
NFSVMwareCPUloadisheavier
VM
VMwarelimitedto8NFSdatastores
li it d t 8 NFS d t t
NFSdatastoreslimitedto16TB
NFS thi
NFSthinprovisionedVMDKsautorehydrate
ii d VMDK t h d t
Whenmovedorcloned
Windows VMs cant boot or utilize NFS datastores
WindowsVMscantbootorutilizeNFSdatastores
33 NSG
Oct10
 VMwarevSphere4.1NFSSupportImproved

NFS th
NFSthroughputover10Gimprovedsignificantly
h t 10G i d i ifi tl
12 40%forReads&32 124%forWrites
Vs iSCSI improvements over 10G of only
Vs.iSCSIimprovementsover10Gofonly
623%forReads&819%forWrites
Plus better performance monitoring
Plusbetterperformancemonitoring
Providinglatency&throughputstatisticsfor
HostperNFSdatastore
VMsperNFSdatastore
NAS
AndVMsperVMDK

Oct10 NSG 34
 SomeVMwareStorageConclusions

RDMsrequiredforclusters
q
VMFSSANstorageisthemostcommon
Has some issues to watch out for
Hassomeissuestowatchoutfor
NFSstorageistheeasiest
i i
iSCSIstorageisVMFSSANstorage
Andprettydarneasyaswell
Dependingonvendor

Oct10 NSG 35
 VMwareDPMajorIssues

Consumesmajorserverresources
Significantlymorethancomparablephysicalservers
Reducingserverconsolidationbenefits
Difficulttomanage
Why????

Oct10 NSG 36
 DefinitionofInsanityisDoingtheSameThing
Over&Over&ExpectingDifferentResults

Oct10 NSG 37
 KeyDPVmware Problem Agents
Agent

A
Agentsaresoftwarew/adminprivileges
t ft / d i i il
A.k.a.plugins,liteagents,clientsoftware
R l i
Roleiscollectdata&sendittoabackupormediaserver
ll d & di b k di
Completefilesandongoingincrementalchanges
/
Separateagentstypical/OS,database,ERP,&emailapp
AswellasforBU,CDP,&Archiving/app
Canbemorethanoneagent/server(OS,DBMS,Email,etc.)
Whenagentsdeduplicateand/orencrypt=moreresources

Oct10 NSG 38
 ComparedtoPhysicalServers
Insteadof1or2agentsperphysicalserver
Therearelotsofagentsperphysicalserver
Wastingunderutilizedserverresourcesisonething
Itsquiteanotherwhenthatserverisoversubscribed

Oct10 NSG 39
 ReducesServerVirtualizationConsolidation
AgentslimitVMs/physicalserver
Reduceseffectiveconsolidationbenefits
Decreasesfinancialsavings,payback,&ROI
VMbackupsbottlenecktheIO
Simultaneousbackupshavebandwidthconstraints
Backupsmustbemanuallyscheduledserially

Oct10 NSG 40
 AgentsHaveOtherFrustrationsAsWell

Agentscompromisesecurity
A i i
Agentsareverydifficulttoadmin&manage
Especiallyasservers&appsproliferate

Oct10 NSG 41
 AgentsCompromiseSecurity
Afirewallportmustbeopenedperagent
Agentshaveadminprivileges
Createsabackdooraccesstoeverythingontheserver
Hackerstargetagents BUdatamustbeimportant
Agentsarelisteningonaportjustwaitingtobehacked
Hackercantrytohackdozensthousandsofservers
Often w/o being detected >agents=>attackpoints
Oftenw/obeingdetected > agents = > attack points
Lackofencryptioninflightputstransmitteddataatrisk
Agentencryptionwastesevenmoreserverresources
Anowinsituation

Oct10 NSG 42
 AgentsareDifficulttoAdmin&Manage
Installinganagentcanbemaddeninglyfrustrating
Requiresanappdisruptivesystemreboottoinitialize
R i di i b i i i li
Upgradespatcheshotfixesrequireappdisruptivesystemreboot1
Agents upgraded when SW is upgraded
AgentsupgradedwhenSWisupgraded

1Some BU software have an automated upgrade process; however, the reboots are still disruptive

Oct10 NSG 43
 Agent=TooManyRestoreFailures
Limitedrollbackcapabilityifatall
Notimebasedversioning
N i b d i i
Multisteprestores
Data has to be restored from backup media
Datahastoberestoredfrombackupmedia
Tomediaorbackupserverbeforerestoredtoserver
Usuallyrequiresmultiplesteps&passes
I f t t l it i d f il
Infrastructurecomplexity=increasedfailures
Moreagentsoftwarepartsincreasesfailureprobability
LotsofagentflavorsPlatforms,OS,DBMS(allkinds),&email
ComplicatedTroubleshooting
Particularlyaggravatingwhenanagentstopsworking
Th l
Thelargertheenvironment,themorecomplicatedthetroubleshooting
th i t th li t d th t bl h ti
Oct10 NSG 44
 ThreeWaystoMitigateVMwareDPIssues

VADPbasedDataProtection
VADP b d D t P t ti
Storagebasedsnapshots
CloudbasedBUR

Oct10 NSG 45
 VADP vStorageAPIforDataProtection
SignificantlyreducesagentsonVMs
Allows BU software to collect data through VADP
AllowsBUsoftwaretocollectdatathroughVADP
UtilizingVMwaresnapshots&VSSforWindows
BestforWindows&Linuxguests
est o do s & u guests
Allstoragearchitectures FC,ISCSI&NFS
FasterincrementalBU
FasterdifferentialBU
FasterfullVMimageBU
Filelevelbackup&restore
AvailableonvSphere4&ESX3.5(latest)

Oct10 NSG 46
 VCB(EOL)
BU
Agent

Mount
M

Snapshot
p
Snapshot
Snapshot

Oct10 NSG 47
 VADP BU
Agent

Snapshot
Snapshot
Snapshot

Oct10 NSG 48
 VADPvsVCB

VADP VCB
Requiresaddldownload& No,builtintothedata Yes
install protectionsoftware
p
FullVMimagebackup Yes,singlestepcopy No
sourcetotarget
I
IncrementalVMimageBU
t l VM i BU Y
Yes N
No
Filelevel BU Yes,Windows &Linux Yes,Windowsonly
Full VM image restore
FullVMimagerestore Yes Yes,byusingVMware
Yes, by using VMware
Converter
IncrementalVMimagerestore Yes No
Filelevelrestore Yes,usingrestoreagents Yes,usingrestoreagents
Oct10 NSG 49
 ButVADPHasLimitations

J t1 i
Just1pieceofthepuzzle
f th l
Requiresintegrationw/BUsoftware
VMwareslowendVDRisanexample
NoUnixfilelevelbackuporrestores
ll d l h
StillneedsatleastoneagentonvSphere
Notapplicationaware
NeedsagentsforstructuredappsnotonWindows
OronWindowsbutnotVSSsupported
Requires physical server resources
Requiresphysicalserverresources
Oct10 NSG 50
 StorageBasedSnapshots

Snapshot
Snapshot
Snapshot

Oct10 NSG 51
 StorageSnapshotsAreGenerallyPrettyEasy

Noagentsonserversorapps
N t
MediumtofinegranularityRPO&RTO
S
Snapshotssenttoothersite
h t t t th it
Withpossibilityofbidirectional
Snap restores mount data point & done
Snaprestores=mountdata,point&done
RemoteSnapshotpromotabletoproductionvolume
Fast virtuallyinstantaneouswithnoBUWindows
Fast virtually instantaneous with no BU Windows
Centrallyadministerw/storage
In some cases even deduped
Insomecasesevendeduped
Oct10 NSG 52
 StorageSnapshotsHasItsIssues

Snapsoftennotstructureddatacrashconsistent
Mostnotintegratedw/WindowsVSS
Most not integrated w/Windows VSS
Structuredappsrequiringcrashconsistency=agents
Requiresintegrationw/BUSWconsole
AgentsquiesceDBMS,providingwriteconsistency
BUsoftwaretellsstoragesystemtosnapshot
Snapshots/system
h /
Highcostw/Capacitybasedlicensing
Storagesystemtendstobehighercost
Oct10 NSG 53
 SnapshotWorkarounds BUSW+Agents
Integratew/BUSoftware&BUagents
Toquiescestructuredapps
q pp

Snapshot
Snapshot
Snapshot

Oct10 NSG 54
 StorageSnapshotsw/VMwareShould

Beintegratedw/VSS
Have nominal operational burden
Havenominaloperationalburden
Eliminatebackupwindowissues
Provide minimal app disruptions
Provideminimalappdisruptions
Beautomated
Allowforextensivenumbersofsnapshots

Oct10 NSG 55
 CloudBasedBUR(Backup&Restore)
Agentlessoragentlimited
Appnondisruptive
Implementations,ops,mgmt

Localrecoveries
Onsitephysicalorvirtualappliance
VADPintegrated
CBTaplus
l
Disasterrecoveranywhere
Off it d t
Offsitedata
FromfinetocoarsegrainRPO&RTO
Limitedonsitestorage&infrastructurerequirements
Li it d it t &i f t t i t
Oct10 NSG 56
 CloudBUR
VerylowTCO

Oct10 NSG 57
 NotAllCloudBURisCreatedEqual
Make sure Cloud BUR service provider meets your requirements:
MakesureCloudBURserviceprovidermeetsyourrequirements:
Multitenant
Storesyourdataencrypted
Givesyoukeymgmt
IntegrateswithVMwareVADP
CBTisaplus
Isasagentlessaspossible
Timebasedversioning,evenCDP
Providesbothlocalrecoveries&DR
FilelevelforallVMs(notjustLinux&Windows)
Iseasytorecoveratanygranularity
One pass recoveries preferred
Onepassrecoveriespreferred
ProvideswrittenSLARPO&RTOguarantees
Makesitfairlysimpletomovetoanotherprovider
Orinhouse

Oct10 NSG 58
 VMwareDPConclusions

Th
Thereare3verygoodalternatives
3 d lt ti
VADP
Anenablingtool
StorageSnapshot
M h j l
Morethanjustatool
CloudbasedBUR
Oft
Oftenusesbothplusinfinitelyscalable
b th l i fi it l l bl

Oct10 NSG 59
1MoreThing VDI&Storage

Oct10 NSG 60
 NowYouKnowWheretheRocksAre

marcstaimer@mac.com
503579
503 5793763
Oct10 NSG 61