Vi3!35!25 Resource MGMT

Resource Management Guide
ESX Server 3.5, ESX Server 3i version 3.5 VirtualCenter 2.5
Resource Management Guide Revision: 20090612 Item: VI-ENG-Q407-439
You can find the most up-to-date technical documentation on the VMware Web site at: http://www.vmware.com/support/ The VMware Web site also provides the latest product updates. If you have comments about this documentation, submit your feedback to: docfeedback@vmware.com
20062009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware, the VMware boxes logo and design, Virtual SMP, and VMotion are registered trademarks or trademarks of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
VMware, Inc. 3401 Hillview Ave. Palo Alto, CA 94304 www.vmware.com

2 VMware, Inc.
Contents
AboutThisBook
1 GettingStartedwithResourceManagement 13
ViewingHostResourceInformation 14 UnderstandingVirtualMachineResourceAllocation 18 ReservingHostResources 20 VirtualMachineAttributes:Shares,Reservation,andLimit AdmissionControl 22 ChangingVirtualMachineAttributes 23 CreatingandCustomizingResourcePools 25 UnderstandingExpandableReservation 29 CreatingandCustomizingClusters 30
20
2 ResourceManagementConcepts 33
WhatAreResources? 33 ResourceProvidersandConsumers 33 HowESXServerManagesResources 35 HowAdministratorsConfigureResources 35 ResourceUtilizationandPerformance 36 UnderstandingESXServerArchitecture 36 VMkernel 37 VMkernelResourceManager 37 VMkernelHardwareInterfaceLayer 37 VirtualMachineMonitor 37 ServiceConsole 38 HowAdministratorsAffectCPUManagement 38 HowAdministratorsCanAffectMemoryManagement UnderstandingCPUandMemoryVirtualization 39 CPUVirtualizationBasics 39 MemoryVirtualizationBasics 40
39
VMware, Inc.
3 UnderstandingandManagingResourcePools 43
WhatAreResourcePools? 44 WhyUseResourcePools? 45 HostResourcePoolsandClusterResourcePools 46 ResourcePoolAdmissionControl 47 CreatingResourcePools 48 UnderstandingExpandableReservations 50 ViewingResourcePoolInformation 50 ResourcePoolSummaryTab 51 ResourcePoolResourceAllocationTab 52 ChangingResourcePoolAttributes 54 MonitoringResourcePoolPerformance 54 AddingVirtualMachinestoResourcePools 55 RemovingVirtualMachinesfromResourcePools 56 ResourcePoolsandClusters 56 ClustersEnabledforDRS 57 ClustersNotEnabledforDRS 58
4 UnderstandingClusters 59
IntroductiontoClusters 59 VMwareDRS 60 VMwareHA 60 ClustersandVirtualCenterFailure 61 UnderstandingVMwareDRS 62 InitialPlacement 62 LoadBalancingandVirtualMachineMigration 66 DistributedPowerManagement 68 DRSClusters,ResourcePools,andESXServer 70 HostMaintenanceandStandbyModes 71 UnderstandingVMwareHA 72 TraditionalandHAFailoverSolutions 72 VMwareHAFeatures 74 FailoverCapacity 75 PlanningforHAClusters 75 VMwareHAandSpecialSituations 77 PrimaryandSecondaryHosts 78 HAClustersandMaintenanceMode 78 HAClustersandDisconnectedHosts 78 HAClustersandHostNetworkIsolation 79 UsingHAandDRSTogether 80
VMware, Inc.
Contents
Valid,Yellow,andRedClusters 81 ValidCluster 81 YellowCluster 84 RedCluster 85
5 CreatingaVMwareCluster 89
ClusterPrerequisites 89 ClustersEnabledforHA 90 VirtualCenterVMotionRequirements 91 ClusterCreationOverview 92 CreatingaCluster 93 ChoosingClusterFeatures 93 SelectingAutomationLevel 93 SelectingHAOptions 94 SelectingaVirtualMachineSwapfileLocation 95 FinishingClusterCreation 95 ViewingClusterInformation 96 SummaryPage 96 DRSResourceDistributionCharts 98 DRSRecommendationsPage 99
6 ManagingVMwareDRS 101
CustomizingDRS 101 AddingHoststoaDRSCluster 102 AddingManagedHoststoaCluster 102 AddingUnmanagedHoststoaCluster 103 RemovingHostsfromClusters 104 HostRemovalandResourcePoolHierarchies HostRemovalandVirtualMachines 105 HostRemovalandInvalidClusters 105 ApplyingDRSRecommendations 105 RecommendationGrouping 106 UsingtheDRSRecommendationsPage 107 ReconfiguringDRS 107 UsingDRSAffinityRules 108 UnderstandingRuleResults 110 DisablingorDeletingRules 110
104
VMware, Inc.
7 ClustersandVirtualMachines 111
AddingVirtualMachinestoaCluster 111 AddingaVirtualMachineDuringCreation 111 MigratingaVirtualMachinetoaCluster 112 AddingaHostwithVirtualMachinestoaCluster 112 PoweringOnVirtualMachinesinaCluster 112 DRSEnabled 112 HAEnabled 113 RemovingVirtualMachinesfromaCluster 113 MigratingVirtualMachinesoutofaCluster 113 RemovingaHostwithVirtualMachinesfromaCluster 114 CustomizingDRSforVirtualMachines 114 CustomizingHAforVirtualMachines 115
8 ManagingVMwareHA 117
CustomizingHA 117 AddingHoststoanHACluster 118 AddingManagedHoststoaCluster 118 AddingUnmanagedHoststoaCluster 118 ResultsofAddingHoststoaCluster 119 ConfiguringandUnconfiguringHAonaHost WorkingwithVMwareHA 121 SettingAdvancedHAOptions 121
120
9 AdvancedResourceManagement 123
CPUVirtualization 124 SoftwareCPUVirtualization 124 HardwareAssistedCPUVirtualization 124 VirtualizationandProcessorSpecificBehavior 125 PerformanceImplications 125 UsingCPUAffinitytoAssignVirtualMachinestoSpecificProcessors 126 MulticoreProcessors 128 Hyperthreading 129 EnablingHyperthreading 130 HyperthreadingandESXServer 130 AdvancedServerConfigurationforHyperthreading 131 Quarantining 133 HyperthreadingandCPUAffinity 133 MemoryVirtualization 133 SoftwareMemoryVirtualization 133
6 VMware, Inc.
Contents
HardwareAssistedMemoryVirtualization 134 PerformanceImplications 135 UnderstandingMemoryOverhead 136 MemoryAllocationandIdleMemoryTax 138 HowESXServerHostsAllocateMemory 138 HowHostMemoryIsUsed 139 MemoryTaxforIdleVirtualMachines 140 HowESXServerHostsReclaimMemory 140 MemoryBalloon(vmmemctl)Driver 141 SwapSpaceandGuestOperatingSystems 142 Swapping 142 SwapSpaceandMemoryOvercommitment 144 SwapFilesandESXServerFailure 144 SharingMemoryAcrossVirtualMachines 144 AdvancedAttributesandWhatTheyDo 145 SettingAdvancedHostAttributes 145 SettingAdvancedVirtualMachineAttributes 149
10 UsingNUMASystemswithESXServer 151
IntroductiontoNUMA 152 WhatIsNUMA? 152 NUMAChallengesforOperatingSystems 152 ESXServerNUMAScheduling 153 VMwareNUMAOptimizationAlgorithms 154 HomeNodesandInitialPlacement 154 DynamicLoadBalancingandPageMigration 155 TransparentPageSharingOptimizedforNUMA 156 ManualNUMAControls 156 IBMEnterpriseXArchitectureOverview 157 AMDOpteronBasedSystemsOverview 158 ObtainingNUMAConfigurationInformationandStatistics 159 CPUAffinityforAssociatingVirtualMachineswithaSingleNUMANode 159 MemoryAffinityforAssociatingMemoryAllocationswithaNUMANode 160
11 BestPractices 163
ResourceManagementBestPractices 163 CreatingandDeployingVirtualMachines 164 Planning 164 CreatingVirtualMachines 164 DeployingtheGuestOperatingSystem 165
VMware, Inc. 7
DeployingGuestApplications 165 ConfiguringVMkernelMemory 165 VMwareHABestPractices 166 NetworkingBestPractices 166 SettingUpNetworkingRedundancy 167 OtherVMwareHAClusterConsiderations 170
A PerformanceMonitoringUtilities:resxtopandesxtop 171
DecidingtoUseresxtoporesxtop 171 UsingtheresxtopUtility 171 UsingtheesxtopUtility 172 UsingtheUtilitiesinInteractiveMode 173 InteractiveModeCommandLineOptions 173 CPUPanel 176 MemoryPanel 179 StoragePanels 184 NetworkPanel 193 UsingtheUtilitiesinBatchMode 194 UsingtheUtilitiesinReplayMode 195
Index 197
VMware, Inc.
About This Book
Thismanual,theResourceManagementGuide,discussesresourcemanagementfor VMwareVirtualInfrastructureenvironments.Itsfocusisonthefollowingmajor topics: Resourceallocationandresourcemanagementconcepts Virtualmachineattributesandadmissioncontrol Resourcepoolsandhowtomanagethem Clusters,VMwareDistributedResourceScheduler(DRS), VMwareHighAvailability(HA),andhowtoworkwiththem Advancedresourcemanagementoptions Performanceconsiderations ResourceManagementGuidecoversbothESXServer3.5andESXServer3iversion3.5. Foreaseofdiscussion,thisbookusesthefollowingproductnamingconventions: FortopicsspecifictoESXServer3.5,thisbookusesthetermESXServer3. FortopicsspecifictoESXServer3iversion3.5,thisbookusesthetermESXServer 3i. Fortopicscommontobothproducts,thisbookusesthetermESXServer. Whentheidentificationofaspecificreleaseisimportanttoadiscussion,thisbook referstotheproductbyitsfull,versionedname. WhenadiscussionappliestoallversionsofESXServerforVMwareInfrastructure 3,thisbookusesthetermESXServer3.x.
VMware, Inc.
Intended Audience
Thismanualisforsystemadministratorswhowanttounderstandhowthesystem managesresourcesandhowtheycancustomizethedefaultbehavior.Itsalsoessential foranyonewhowantstounderstandanduseresourcepools,clusters,DRS,orHA. ThismanualassumesyouhaveaworkingknowledgeofESXServerandofthe VirtualCenterServer.
Document Feedback
VMwarewelcomesyoursuggestionsforimprovingourdocumentation.Ifyouhave comments,sendyourfeedbackto: docfeedback@vmware.com
VMware Infrastructure Documentation

TheVMwareInfrastructuredocumentationconsistsofthecombinedVMware VirtualCenterandESXServerdocumentationset.
Abbreviations Used in Figures

ThefiguresinthismanualusetheabbreviationslistedinTable 1. Table 1. Abbreviations
Abbreviation database datastore dsk# hostn RP SAN tmplt user# VC VI VM# Description VirtualCenterdatabase Storageforthemanagedhost Storagediskforthemanagedhost VirtualCentermanagedhosts Resourcepool Storageareanetworktypedatastoresharedbetweenmanagedhosts Template Userwithaccesspermissions VirtualCenter VMwareInfrastructureClient Virtualmachinesonamanagedhost
10
VMware, Inc.
About This Book
Technical Support and Education Resources

Thefollowingsectionsdescribethetechnicalsupportresourcesavailabletoyou.You canaccessthemostcurrentversionsofthismanualandotherbooksbygoingto: http://www.vmware.com/support/pubs
Online and Telephone Support

Useonlinesupporttosubmittechnicalsupportrequests,viewyourproductand contractinformation,andregisteryourproducts.Goto http://www.vmware.com/support. Customerswithappropriatesupportcontractsshouldusetelephonesupportforthe fastestresponseonpriority1issues.Goto http://www.vmware.com/support/phone_support.html.
Support Offerings
FindouthowVMwaresupportofferingscanhelpmeetyourbusinessneeds.Goto http://www.vmware.com/support/services.
VMware Professional Services

VMwareEducationServicescoursesofferextensivehandsonlabs,casestudy examples,andcoursematerialsdesignedtobeusedasonthejobreferencetools. Coursesareavailableonsite,intheclassroom,andliveonline.Foronsitepilot programs andimplementationbestpractices,VMwareConsultingServicesprovides offeringsto helpyouassess,plan,build,andmanageyourvirtualenvironment.To accessinformationabouteducationclasses,certificationprograms,andconsulting services,gotohttp://www.vmware.com/services.
VMware, Inc.
11
12
VMware, Inc.
Getting Started with Resource Management
Thischapterintroducesbasicresourcemanagementconceptsusingasimpleexample. Thechapterstepsyouthroughresourceallocation,firstinasinglehostenvironment, andtheninamorecomplexmultihostenvironment. Thischapterdiscussesthefollowingtopics: ViewingHostResourceInformationonpage 14 UnderstandingVirtualMachineResourceAllocationonpage 18 ChangingVirtualMachineAttributesonpage 23 CreatingandCustomizingResourcePoolsonpage 25 UnderstandingExpandableReservationonpage 29 CreatingandCustomizingClustersonpage 30
VMware, Inc.
13
Viewing Host Resource Information

Inthissection,youexploreahostsresourcesandlearnhowtodeterminewhouses them. NOTEYoucanalsoperformmanyofthetasksinthischapterusingaVIClient connectedtoanESXServersystemoraVIWebAccessClientconnectedtoaserver. Assumethatasystemadministratorforasmallcompanyhassetuptwovirtual machines,VMQAandVMMarketing,onanESXServerhost.SeeFigure 11. Figure 1-1. Single Host with Two Virtual Machines
ESX Server host
VM-QA
VM-Marketing
To view information about a host 1 2 StartaVMwareInfrastructureClient(VIClient)andconnecttoaVirtualCenter Server. Intheinventorypanelontheleft,selectthehost. WiththeSummarytabselected,thepanelsdisplaythefollowinginformation aboutthehost.

Summary Panel General panel Commands panel Resources panel Information Shown Showsinformationaboutprocessors,processortype,andsoon. Allowsyoutoselectcommandstoexecutefortheselectedhost. Showsinformationaboutthetotalresourcesoftheselected host.Thispanelincludesinformationaboutthedatastores connectedtothehost.
14
VMware, Inc.
Chapter 1 Getting Started with Resource Management
Fordetailedinformationaboutavailablememory,clicktheConfigurationtab,and selectMemory. Thepanelliststotalresources,howmuchisusedbyvirtualmachines,andhow muchisusedbytheserviceconsole(ESXServer3only).
Theamountofphysicalmemorythevirtualmachinescanuseisalwayslessthan whatisinthephysicalhostbecausethevirtualizationlayertakesupsome resources.Forexample,ahostwithadual3.2GHzCPUand2GBofmemorymight
VMware, Inc.
15
make6GHzofCPUpowerand1.5GBofmemoryavailableforusebyvirtual machines. 4 Fordetailedinformationabouthowthetwovirtualmachinesusethehosts resources,clicktheResourceAllocationtab.
YouseetheCPUReservationandMemoryReservation,howmuchofthe reservationisused,andhowmuchisavailable. NOTEIntheResourceAllocationtabshown,novirtualmachinesarerunning,so noCPUormemoryisused.Yourevisitthistabafterpoweringonavirtual machine. Thefieldsdisplaythefollowinginformation.

Field CPUReservation CPUReservationUsed Description TotalCPUresourcesavailableforthishost. TotalCPUresourcesofthishostthatarereservedby runningvirtualmachines. Note:Virtualmachinesthatarenotpoweredondonot consumeCPUresources.Forpoweredonvirtualmachines, thesystemreservesCPUresourcesaccordingtoeachvirtual machinesReservationsetting. CPUReservationUnused TotalCPUresourcesofthishostthatarenotcurrently reserved. Consideravirtualmachinewithreservation=2GHzthatis totallyidle.Ithas2GHzreserved,butitisnotusinganyof itsreservation. Othervirtualmachinescannotreservethese2GHz. Othervirtualmachinescanusethese2GHz,thatis,idle CPUreservationsarenotwasted.
16
VMware, Inc.
Field MemoryReservation
Description Totalmemoryresourcesavailableforthishost. Ifavirtualmachinehasamemoryreservationbuthasnot yetaccesseditsfullreservation,theunusedmemorycanbe reallocatedtoothervirtualmachines.
MemoryReservation Used
Totalmemoryresourcesofthishostthatarereservedbya runningvirtualmachineandvirtualizationoverhead. Note:Virtualmachinesthatarenotpoweredondonot consumememoryresources.Forpoweredonvirtual machines,thesystemreservesmemoryresourcesaccording toeachvirtualmachinesReservationsettingandoverhead. Afteravirtualmachinehasaccesseditsfullreservation, ESXServerallowsthevirtualmachinetoretainthismuch memory,andwillnotreclaimit,evenifthevirtualmachine becomesidleandstopsaccessingmemory.
MemoryReservation Unused
Totalmemoryresourcesofthishostthatarenotcurrently reserved.
ClicktheMemoryorCPUbuttondependingontheinformationyouwant.
VMware, Inc.
17
Table 1-1. Virtual Machine Attributes

Field Name Reservation MHz/MB Limit Description Nameofthevirtualmachine. AmountofCPUormemoryreservedforthisvirtualmachine. Bydefault,noreservationisspecifiedand0isdisplayed.See Reservationonpage 21. AmountofCPUormemoryspecifiedastheupperlimitforthis virtualmachine. Bydefault,nolimitisspecifiedandUnlimitedisdisplayed.See Limitonpage 22. Shares Sharesspecifiedforthisvirtualmachine.Eachvirtualmachineis entitledtoresourcesinproportiontoitsspecifiedshares, boundedbyitsreservationandlimit.Avirtualmachinewith twiceasmanysharesasanotherisentitledtotwiceasmany resources. SharesdefaulttoNormal.SeeSharesonpage 20. SharesValue %Shares Type Numberofsharesallocatedtothisvirtualmachine. Percentageofsharesallocatedtothisvirtualmachine. Forresourcepools,eitherExpandableorFixed.See UnderstandingExpandableReservationonpage 29.
Understanding Virtual Machine Resource Allocation

Whenyoucreateavirtualmachine,theNewVirtualMachinewizardpromptsyoufor thememorysizeforthisvirtualmachine.Thisamountofmemoryisthesameasthe amountofmemoryyouinstallinaphysicalmachine. NOTETheESXServerhostmakesthismemoryavailabletovirtualmachines.Thehost allocatesthenumberofmegabytesspecifiedbythereservationdirectlytothevirtual machine.Anythingbeyondthereservationisallocatedusingthehostsphysical resourcesor,whenphysicalresourcesarenotavailable,handledusingspecial techniquessuchasballooningorswapping.SeeHowESXServerHostsReclaim Memoryonpage 140.
18
VMware, Inc.
Figure 1-2. Virtual Machine Memory Configuration
Thesystemalsopromptsforthenumberofvirtualprocessors(CPUs)iftheoperating systemyouhavechosensupportsmorethanone. Figure 1-3. Virtual CPU Configuration
WhenCPUresourcesareovercommitted,theESXServerhosttimeslicesthephysical processorsacrossallvirtualmachinessoeachvirtualmachinerunsasifithasthe specifiednumberofprocessors. WhenanESXServerhostrunsmultiplevirtualmachines,itallocateseachvirtual machineashareofthephysicalresources.Withthedefaultresourceallocationsettings, allvirtualmachinesassociatedwiththesamehostreceive: AnequalshareofCPUpervirtualCPU.Thatmeanssingleprocessorvirtual machinesareassignedonlyhalfoftheresourcesofadualprocessorvirtual machine. AnequalshareperMBofvirtualmemorysize.Thatmeansan8GBvirtualmachine isentitledtoeighttimesasmuchmemoryasa1GBvirtualmachine.
VMware, Inc.
19
Reserving Host Resources

Insomesituations,systemadministratorswanttoknowthatacertainamountof memoryforavirtualmachinecomesdirectlyfromthephysicalresourcesofthe ESXServermachine.Similarly,theadministratormightwanttoguaranteethatacertain virtualmachinealwaysreceivesahigherpercentageofthephysicalresourcesthan othervirtualmachines. Youcanreservephysicalresourcesofthehostusingeachvirtualmachinesattributes, discussedinthenextsection. NOTEInmostcases,usethedefaultsettings.SeeChapter 11,BestPractices,on page 163forinformationonhowtobestusecustomresourceallocations.
Virtual Machine Attributes: Shares, Reservation, and Limit

Foreachvirtualmachine,youcanspecifyshares,reservation(minimum),andlimit (maximum).Thissectionexplainswhatitmeanstospecifytheseattributes.
Shares
Sharesspecifytherelativepriorityorimportanceofavirtualmachine.Ifavirtual machinehastwiceasmanysharesofaresourceasanothervirtualmachine,itisentitled toconsumetwiceasmuchofthatresource.SharesaretypicallyspecifiedasHigh, Normal,orLowandthesevaluesspecifysharevalueswitha4:2:1ratio,respectively. YoucanalsochooseCustomtoassignaspecificnumberofshares(whichexpressesa proportionalweight)toeachvirtualmachine. Specifyingsharesmakessenseonlywithregardtosiblingvirtualmachinesorresource pools,thatis,virtualmachinesorresourcepoolswiththesameparentintheresource poolhierarchy.Siblingsshareresourcesaccordingtotheirrelativesharevalues, boundedbythereservationandlimit.SeeWhatAreResourcePools?onpage 44for anexplanationofthehierarchyandsiblingconcepts. Whenyouassignsharestoavirtualmachine,youalwaysspecifytherelativepriority forthatvirtualmachine.
20
VMware, Inc.
CPUandmemorysharevalues,respectively,defaultto: High2000sharespervirtualCPUand20sharespermegabyteofvirtual machinememory Normal1000sharespervirtualCPUand10sharespermegabyteofvirtual machinememory Low500sharespervirtualCPUand5sharespermegabyteofvirtualmachine memory YoucanalsospecifyaCustomsharevalue. Forexample,anSMPvirtualmachinewithtwovirtualCPUsand1GBRAMwithCPU andmemorysharessettoNormalhas2x1000=2000sharesofCPUand10x1024=10240 sharesofmemory. NOTEVirtualmachineswithmorethanonevirtualCPUarecalledSMP(symmetric multiprocessing)virtualmachines. Theamountofresourcesrepresentedbyeachsharechangeswhenanewvirtual machineispoweredon.Thisaffectsallvirtualmachinesinthesameresourcepool.For example: Twovirtualmachinesrunonahostwith8GHz.TheirCPUsharesaresetto Normalandget4GHzeach. Athirdvirtualmachineispoweredon.ItsCPUsharesvalueissettoHigh,which meansitshouldhavetwiceasmanysharesasthemachinessettoNormal.Thenew virtualmachinereceives4GHzandthetwoothermachinesgetonly2GHzeach. Notethatthesameresultoccursiftheuserspecifiesacustomsharevalueof2000 forthethirdvirtualmachine.
Reservation
Reservationspecifiestheguaranteedreservationforavirtualmachine.Theserver allowsyoutopoweronavirtualmachineonlyiftheCPUandmemoryreservationis available.Theserverguaranteesthatamountevenwhenthephysicalserverisheavily loaded.Thereservationisexpressedinconcreteunits(megahertzormegabytes).When resourcesarenotused,theESXServerhostmakesthemavailabletoothervirtual machines.
VMware, Inc.
21
Forexample,assumeyouhave2GHzavailableandspecifyareservationof1GHzfor VM1and1GHzforVM2.Noweachvirtualmachineisguaranteedtoget1GHzifit needsit.However,ifVM1isusingonly500MHz,VM2canuse1.5GHz. Reservationdefaultsto0.Itisagoodideatospecifyareservationtoguaranteethatthe necessaryCPUormemoryarealwaysavailableforthevirtualmachine.
Limit
LimitspecifiestheupperlimitforCPUormemoryforavirtualmachine.Aservercan allocatemorethanthereservationtoavirtualmachine,butneverallocatesmorethan thelimit,evenifthereisunutilizedCPUormemoryonthesystem.Thelimitis expressedinconcreteunits(megahertzormegabytes). CPUandmemorylimitdefaulttounlimited.Whenthememorylimitisunlimited,the amountofmemoryconfiguredforthevirtualmachinewhenitwascreatedbecomesits implicitlimitinmostcases. Inmostcases,itisnotnecessarytospecifyalimit.Therearebenefitsanddrawbacks: BenefitsAssigningalimitisusefulifyoustartwithasmallnumberofvirtual machinesandwanttomanageuserexpectations.Performancewilldeteriorateas youaddmorevirtualmachines.Youcansimulatehavingfewerresourcesavailable byspecifyingalimit. DrawbacksYoumightwasteidleresourcesifyouspecifyalimit.Thesystem doesnotallowvirtualmachinestousemoreresourcesthanthelimit,evenwhen thesystemisunderutilizedandidleresourcesareavailable.Specifythelimitonly ifyouhavegoodreasonsfordoingso.
Admission Control
Whenyoupoweronavirtualmachine,thesystemcheckstheamountofCPUand memoryresourcesthathavenotyetbeenreserved.Basedontheavailableunreserved resources,thesystemdetermineswhetheritcanguaranteethereservationforwhich thevirtualmachinehasbeenconfigured(ifany).Thisprocessiscalledadmissioncontrol. IfenoughunreservedCPUandmemoryareavailable,orifthereisnoreservation,the virtualmachineispoweredon.Otherwise,anInsufficient Resourceswarning appears. NOTEInadditiontotheuserspecifiedmemoryreservation,foreachvirtualmachine thereisalsoanamountofoverheadmemory.Thisextramemorycommitmentis includedintheadmissioncontrolcalculation.SeeUnderstandingMemoryOverhead onpage 136.
22
VMware, Inc.
WhentheexperimentalDistributedPowerManagementfeatureisenabled,hostsmay beplacedinstandbymode(thatis,poweredoff)toreducepowerconsumption.The unreservedresourcesprovidedbythesehostsareconsideredavailableforthepurpose ofadmissioncontrol.Ifavirtualmachinecannotbepoweredonwithoutthese resources,arecommendationtopoweronsufficientstandbyhostsismade.See DistributedPowerManagementonpage 68.
Changing Virtual Machine Attributes

Earlierinthischapter,youviewedhostsandvirtualmachinesandtheirresource allocation.Youdidnotspecifyshares,reservation,andlimitforthevirtualmachines. Inthisexample,assume: TheQAvirtualmachineismemoryintensive.Youwanttospecifythat,when systemmemoryisovercommitted,VMQAcanusetwiceasmuchmemoryand CPUastheMarketingvirtualmachine.SetmemorysharesandCPUsharesto High. MakesurethattheMarketingvirtualmachinehasacertainamountofguaranteed CPUresources.YoucandosousingaReservationsetting. To edit a virtual machines resource allocation 1 2 3 4 StartaVIClientandconnecttoaVirtualCenterServer. SelectthehostintheinventorypanelandclicktheResourceAllocationtab. RightclickVMQA,thevirtualmachineforwhichyouwanttochangeshares,and chooseEditResourceSettings. IntheCPUResourcespanel,chooseHighfromtheSharesdropdownmenu.
VMware, Inc.
23
RepeatthesestepsintheMemoryResourcespanel,andclickOK.
6 7
Rightclickthemarketingvirtualmachine(VMMarketing). ChangethevalueintheReservationfieldtothenumber,andclickOK.
ClickOKwhenyouredone.
24
VMware, Inc.
SelectthehostsResourceAllocationtabandclickCPU,youseethatsharesfor VMQAaretwicethatoftheothervirtualmachine.
Becausethevirtualmachineshavenotbeenpoweredon,theReservationUsed fieldshavenotchanged. 10 PoweronVMMarketingandseehowtheCPUReservationUsedandCPU Unreservedfieldschange.
Creating and Customizing Resource Pools

Asorganizationsgrow,theycanaffordfasterandbettersystemsandallocatemore resourcestothedifferentdepartments.Inthissection,youlearnhowtouseresource poolstodivideahostsresources.Youcanalsouseresourcepoolsinconjunctionwith VMwareclusters,wheretheyallowyoutomanagetheresourcesofallhostsinacluster asonepoolofresources. Whenyoucreatearesourcepool,specifythefollowingattributes: Reservation,limit,andsharesbehavejustastheydoforvirtualmachines.See ChangingVirtualMachineAttributesonpage 23. TheReservationTypeattributeallowsyoutosetuptheresourcepoolsothatthe poolcanreserveavailableresourcesfromitsparentifitdoesnothaveenough resourcesavailablelocally.SeeUnderstandingExpandableReservationon page 29.
VMware, Inc. 25
Continuingwiththeexampleabove,assumethatyounolongerwanttoassignone virtualmachineeachtoyourQAandMarketingdepartmentsbutwanttogiveeach departmentapredefinedamountofresources.Dependingondepartmentalneeds,the departmentadministratorcancreatevirtualmachinesforthedepartment. Forexample,ifyoustartedwithahostthatprovides6GHzofCPUand3GBofmemory, youcanchooseshareallocationsofHighforRPQAandsharesallocationsofNormal forRPMarketing.Thatresultsinapproximately4GHzand2GBofmemoryforRPQA, and2GHzand1GBforRPMarketing.Thoseresourcesarethenavailabletothevirtual machinesintherespectiveresourcepools.SeeFigure 14. Figure 1-4. ESX Server Host with Two Resource Pools
ESX Server host 6GHz, 3GB
RP-QA
4GHz, 2GB
RPMarketing 2GHz, 1GB
VM-QA 1
VM-QA 2
VM-Marketing 1
VM-Marketing 2
VM-Marketing 3
To create and customize resource pools 1 2 3 StartaVIClientandconnecttoaVirtualCenterServer. Intheinventorypanelontheleft,selectahostandchooseNewResourcePoolin theCommandspanelontheright. IntheCreateResourcePooldialogbox,typetheresourcepoolname(forexample, RPQA).
26
VMware, Inc.
SpecifySharesofHighfortheCPUandmemoryresourcesofRPQA.
Createasecondresourcepool,RPMarketing: a b c LeaveSharesatNormalforCPUandmemory. SpecifyaReservationforCPUandmemory. ClickOKtoexit.
SelectthehostintheinventorypanelandclicktheResourceAllocationtab. Theresourcepoolshavebeenaddedtothedisplay.Inthetoppanel,the Reservationforthesecondresourcepoolhasbeensubtractedfromtheunreserved resources.Inthesecondpanel,resourcepoolinformation,includingtheresource pooltype,isnowavailable.
VMware, Inc.
27
Table 12isasummaryofthevaluesthatyoucanspecifyforaresourcepool. Table 1-2. Resource Pool Attributes

Field CPUShares MemoryShares Reservation Description Allowsyoutospecifythesharesforthisresourcepool.Thebasicprinciples arethesameasforvirtualmachines,discussedinSharesonpage 20. DisplaystheamountofCPUormemorythehostreservesforthisresource pool.Defaultsto0. Anonzeroreservationissubtractedfromtheunreservedresourcesofthe parent(hostorresourcepool).Theresourcesareconsideredreserved, regardlessofwhethervirtualmachinesareassociatedwiththeresource pool. Expandable reservation Ifthischeckboxisselected(thedefault),andiftheresourcepoolneedsto makeareservationthatishigherthanitsownreservation(forexample,to poweronavirtualmachine),theresourcepoolcanuseresourcesofthe parentresourcepoolandreservethoseresources. SeeUnderstandingExpandableReservationonpage 29. Limit DisplaystheupperlimitontheCPUormemorythatthehostallocatesto theselectedresourcepool.Defaultisunlimited.Thisdefaultavoids wastingidleresources. DeselecttheUnlimitedcheckboxtospecifyadifferentlimit. Resourcepoollimitsareuseful,forexample,ifyouwanttoassignacertain amountofresourcestoagroupadministrator.Thegroupadministrator cancreatevirtualmachinesforthegroupasneeded,butneverusemore resourcesthanspecifiedbythelimit.
Aftertheresourcepoolshavebeencreated,addvirtualmachinestoeachresourcepool. Avirtualmachinessharesarerelativetoothervirtualmachines(orresourcepools) withthesameparentresourcepool. NOTEAfteryouaddvirtualmachinestotheresourcepool,selecttheresourcepools ResourceAllocationtabforinformationonreservedandunreservedresources.
28
VMware, Inc.
Understanding Expandable Reservation

Howexpandablereservationsworkiseasiesttounderstandusinganexample. Assumethefollowingscenario(showninFigure 15): 1 2 3 4 5 ParentpoolRPMOMhasareservationof6GHzandonerunningvirtualmachine VMM1thatreserves1GHz. YoucreateachildresourcepoolRPKIDwithareservationof2GHzandwith ExpandableReservationselected. Youaddtwovirtualmachines,VMK1andVMK2,withreservationsof2GHz eachtothechildresourcepoolandattempttopowerthemon. VMK1canreservetheresourcesdirectlyfromRPKID(whichhas2GHz). NolocalresourcesareavailableforVMK2,soitborrowsresourcesfromtheparent resourcepool,RPMOM.RPMOMhas6GHzminus1GHz(reservedbythevirtual machine)minus2GHz(reservedbyRPKID),whichleaves3GHzunreserved. With3GHzavailable,youcanpoweronthe2GHzvirtualmachine.
Figure 1-5. Admission Control with Expandable Resource Pools, Example 1
6GHz
RP-MOM
VM-M1, 1GHz 2GHz RP-KID
VM-K1, 2GHz
VM-K2, 2GHz
Now,consideranotherscenariowithVMM1andVMM2(showninFigure 16): 1 2 3 PowerontwovirtualmachinesinRPMOMwithatotalreservationof3GHz. YoucanstillpoweronVMK1inRPKIDbecause2GHzareavailablelocally. WhenyoutrytopoweronVMK2,RPKIDhasnounreservedCPUcapacitysoit checksitsparent.RPMOMhasonly1GHzofunreservedcapacityavailable(5GHz ofRPMOMarealreadyinuse3GHzreservedbythelocalvirtualmachinesand 2GHzreservedbyRPKID).Asaresult,youcannotpoweronVMK2,which requiresa2GHzreservation.
VMware, Inc.
29
Figure 1-6. Admission Control with Expandable Resource Pools, Example 2
6GHz
RP-MOM
VM-M1, 1GHz 2GHz RP-KID
VM-M2, 2GHz
VM-K1, 2GHz
VM-K2, 2GHz
Creating and Customizing Clusters

Intheprevioussection,yousetuptworesourcepoolsthatsharedtheresourcesofa singlehost.Aclusterconsistsofasetofhosts.IfVMwareDRS(DistributedResource Scheduling)isenabled,theclustersupportssharedresourcepoolsandperforms placementanddynamicloadbalancingforvirtualmachinesinthecluster.The experimentalDistributedPowerManagementfeaturecanalsobeenabledwithDRS.It reducesaclusterspowerconsumptionbyprovidingrecommendationsforplacing hostsintostandbypowermodewhensufficientexcesscapacityexists.IfVMwareHA (HighAvailability)isenabled,theclustersupportsfailover.Whenahostfails,all associatedvirtualmachinesarerestartedondifferenthosts. NOTEYoumustbelicensedtouseclusterfeatures. Thissectionstepsyouthroughcreatingaclusterandexplainsbasiccluster functionality.Thefocusisonthedefaultbehaviorofbasicclusters. Assumeyouhaveaclusterthatconsistsofthreephysicalhosts.Eachhostprovides 3GHzand1.5GB,withatotalof9GHzand4.5GBavailable.Ifyouenabletheclusterfor DRS,youcancreateresourcepoolswithdifferentreservationorsharestocontrol aggregateallocationsforgroupsofvirtualmachines,forexample,bydepartment, project,oruser. ForDRSenabledclusters,thesystemplacesvirtualmachinesonthemostsuitable physicalhosts(ormakesrecommendationsforplacement)whenvirtualmachinesare poweredon.Theexactbehaviordependsonthedefaultautomationlevelofthecluster ortheautomationmodeofspecificvirtualmachines.
30
VMware, Inc.
To create and customize a cluster 1 2 3 4 5 6 7 StartaVIClientandconnecttoaVirtualCenterServer. Intheinventorypanelontheleft,rightclickadatacenterandchooseNewCluster. NametheclusterandenableitforHAandDRS. Keepthedefault,fullyautomated,forDRS. KeepthedefaultsforhostfailuresandadmissioncontrolforHA. SelecttheappropriateoptionforSwapfilePolicyforVirtualMachines. ClickFinish. TheVirtualCenterServercreatesanewclusterwiththespecifiedattributes.
ForinformationonDRS,HA,andavailableattributes,seeChapter 5,Creatinga VMwareCluster,onpage 89. Thenexttaskistoaddanumberofhoststothecluster.UsingclustersenabledforDRS makessenseevenifyouhaveonlytwohostsinthecluster. AclusterenabledforHAcansupportamaximumoffourconcurrenthostfailures.In thefollowingsteps,youaddahosttotheclusterthatismanagedbythesame VirtualCenterServer. To add a host to the cluster 1 IntheleftpaneloftheVIClient,selectthehostanddragitovertheclustersicon. IftheclusterisenabledforDRS,youarepromptedwhetheryouwanttoaddthe hostsvirtualmachinesdirectlytotheclusters(invisible)rootresourcepoolor whetheryouwanttocreateanewresourcepooltorepresentthathost.Theroot resourcepoolisatthetoplevelandisnotdisplayedbecausetheresourcesare ownedbythecluster.
VMware, Inc.
31
IftheclusterisnotenabledforDRS,allresourcepoolsareremoved.
Choosetheappropriateoption. Ifyouchoosethefirstoption,theresourcepoolhierarchythatwasonthehostyou areaddingtotheclusteriscollapsedandallresourceswillbemanagedbythe cluster.Choosethesecondoptionifyoucreatedresourcepoolsforthehost. NOTEIfyouareusingaclusterenabledforHA,thatclustermightbemarkedwith aredwarningiconuntilyouhaveaddedenoughhoststosatisfythespecified failovercapacity.SeeValid,Yellow,andRedClustersonpage 81.
SelecttheclusterandchooseitsResourceAllocationtabtoaddmorehostsand lookattheresourceallocationinformationforthecluster.
32
VMware, Inc.
Resource Management Concepts
Thischapterdiscussesthefollowingtopics: WhatAreResources?onpage 33 UnderstandingESXServerArchitectureonpage 36 UnderstandingCPUandMemoryVirtualizationonpage 39
What Are Resources?

ResourcesincludeCPU,memory,power,disk,andnetworkresources.Thismanual focusesprimarilyonCPUandmemoryresources.Powerresourcescanbeadministered withtheexperimentalDistributedPowerManagementfeature.SeeDistributedPower Managementonpage 68.Forinformationaboutdiskandnetworkresources,seethe ESXServerConfigurationGuide.
Resource Providers and Consumers

Withinavirtualinfrastructureenvironment,itishelpfultothinkofresourceproviders andconsumers. Hostsandclustersareprovidersofphysicalresources. Forhosts,availableresourcesarethehostshardwarespecification,minustheresources usedbythevirtualizationsoftware.
VMware, Inc.
33
Aclusterisagroupofhosts.YoucancreateaclusterusingVMwareVirtualCenter,and addmultiplehoststothecluster.VirtualCentermanagesthesehostsresourcesjointly: theclusterownsalloftheCPUandmemoryofallhosts.Youcanenabletheclusterfor jointloadbalancingorfailover.SeeChapter 4,UnderstandingClusters,onpage 59 foranintroductiontoclusters. Resourcepoolsarealogicalabstractionforflexiblemanagementofresources.Resource poolscanbegroupedintohierarchies.Theycanbeconsideredbothresourceproviders andconsumers.Resourcepoolsprovideresourcestochildresourcepoolsandvirtual machines.Resourcepoolsarealsoresourceconsumersbecausetheyconsumetheir parentsresources.SeeChapter 3,UnderstandingandManagingResourcePools,on page 43. Virtualmachinesareresourceconsumers.Thedefaultresourcesettingsassignedduring creationworkwellformostmachines.Youcanlatereditthevirtualmachinesettingsto allocateasharebasedpercentageofthetotalCPUandmemoryoftheresource provideroraguaranteedreservationofCPUandmemory.Whenyoupoweronthat virtualmachine,theservercheckswhetherenoughunreservedresourcesareavailable andallowspowerononlyifthereareenoughresources.(Thisprocessiscalled admissioncontrol.) Toseehowclusters,resourcepools,andvirtualmachinesaredisplayedintheVIClient, seeFigure 21. Figure 2-1. Clusters, Resource Pools, and Virtual Machines in VI Client
Cluster Resource pool
Virtual machine
34
VMware, Inc.
Chapter 2 Resource Management Concepts
How ESX Server Manages Resources

EachvirtualmachineconsumesaportionoftheCPU,memory,networkbandwidth, andstorageresourcesoftheESXServerhost.Thehostguaranteeseachvirtualmachine itsshareoftheunderlyinghardwareresourcesbasedonanumberoffactors: AvailableresourcesfortheESXServerhost(orthecluster). Reservation,limit,andsharesofthevirtualmachine.Theseattributesofavirtual machinehavedefaultvaluesthatyoucanchangetocustomizeresourceallocation. SeeUnderstandingVirtualMachineResourceAllocationonpage 18. Numberofvirtualmachinespoweredonandresourceutilizationbythosevirtual machines. Reservation,limit,andsharestheadministratorassignedtotheresourcepoolsin theresourcepoolhierarchy. Overheadrequiredtomanagethevirtualization. Theservermanagesdifferentresourcesdifferently.TheservermanagesCPUand memoryresourcesbasedonthetotalavailableresourcesandthefactorslistedabove. Theservermanagesnetworkanddiskresourcesonaperhostbasis.AVMwareserver: Managesdiskresourcesusingaproportionalsharemechanism. Controlsnetworkbandwidthwithnetworktrafficshaping. NOTETheESXServerConfigurationGuideisthebestresourceforinformationondisk andnetworkresources.TheFibreChannelSANConfigurationGuideand iSCSISANConfigurationGuidegivebackgroundandsetupinformationforusingESX ServerwithSANstorage.
How Administrators Configure Resources

Inmanycases,thedefaultsthesystemuseswhenyoucreateavirtualmachineare appropriate.Insomecases,youmightfinditusefultocustomizevirtualmachinesso thatthesystemallocatesmoreorfewerresourcestothem. Virtualmachineandresourcepoolattributes,andhowtocustomizethem,are discussedthroughoutthisguide.SeeHowAdministratorsAffectCPUManagement onpage 38andHowAdministratorsCanAffectMemoryManagementonpage 39 foranintroduction.
VMware, Inc.
35
Resource Utilization and Performance

Resourceutilizationisthekeytoperformance.Thebestwaytogetthehighest performancefromyourvirtualinfrastructurecomponentsistomakesurenoresource isabottleneck.SeeChapter 11,BestPractices,onpage 163.SeeAppendix A, PerformanceMonitoringUtilities:resxtopandesxtop,onpage 171forinformation ontheresxtopandesxtopperformancemeasurementtools.
Understanding ESX Server Architecture

ThedifferentcomponentsofanESXServersystemworktogethertorunvirtual machinesandgivethemaccesstoresources.Thissectionbrieflydescribesthe ESXServerarchitecture. NOTESkipthissectionifyourinterestisthepracticalapplicationofresource management. Figure 22showsthemaincomponentsofanESXServerhost. NOTETheserviceconsolecomponentshowninFigure 22isapplicableonlywhen ESXServer3isused.ESXServer3idoesnotprovideaserviceconsole.
Figure 2-2. ESX Server Host Components

virtual machines applications guest operating system resource manager VMkernel hardware interface layer x86 hardware CPU memory disk network service console
Virtual Machine Monitor (VMM)
36
VMware, Inc.
VMkernel
TheVMkernelisahighperformanceoperatingsystemdevelopedbyVMwarethat runsdirectlyontheESXServerhost.TheVMkernelcontrolsandmanagesmostofthe physicalresourcesonthehardware,including: Memory Physicalprocessors Storageandnetworkingcontrollers TheVMkernelincludesschedulersforCPU,memory,anddiskaccess,andhas fullfledgedstorageandnetworkstacks.ItalsoincludestheVirtualMachineFile System(VMFS).VMFSisadistributedfilesystemoptimizedforlargefileslikevirtual machinedisksandswapfiles.
VMkernel Resource Manager

Theresourcemanagerpartitionsthephysicalresourcesoftheunderlyingserver.It employsmechanismsincludingresourcereservationsandproportionalshare schedulingtoallocateCPU,memory,anddiskresourcestovirtualmachinesthatare poweredon.SeeChapter 9,AdvancedResourceManagement,onpage 123for informationaboutresourceallocation. Userscanspecifyshares,reservations,andlimitsforeachvirtualmachine.Theresource managertakesthatinformationintoaccountwhenitallocatesCPUandmemoryto eachvirtualmachine.SeeHowESXServerManagesResourcesonpage 35.
VMkernel Hardware Interface Layer

ThehardwareinterfacehideshardwaredifferencesfromESXServer(andvirtual machine)users.Itenableshardwarespecificservicedeliveryandincludesdevice drivers.
Virtual Machine Monitor

Thevirtualmachinemonitor(VMM)isresponsibleforvirtualizingx86hardware, includingprocessorsandmemory.Whenavirtualmachinestartsrunning,control transferstotheVMM,whichbeginsexecutinginstructionsfromthevirtualmachine. ThetransferofcontroltotheVMMinvolvessettingthesystemstatesothattheVMM runsdirectlyonthehardware.
VMware, Inc.
37
Service Console
TheserviceconsoleisalimiteddistributionofLinuxbasedonRedHatEnterprise Linux3,Update8(RHEL3U8).Theserviceconsoleprovidesanexecutionenvironment formonitoringandadministeringanESXServer3system.ESXServer3idoesnot provideaserviceconsole. NOTEInmostcases,administratorsuseaVIClientconnectedtoeitheranESXServer systemoraVirtualCenterServertomonitorandadministerESXServersystems.
How Administrators Affect CPU Management

YouhaveaccesstoinformationaboutcurrentCPUallocationthroughtheVIClientor usingtheVirtualInfrastructureSDK. SpecifyCPUallocationintheseways: UsetheattributesandspecialfeaturesavailablethroughtheVIClient.The VIClientgraphicaluserinterface(GUI)allowsyoutoconnecttoanESXServer hostoraVirtualCenterServer.SeeChapter 1,GettingStartedwithResource Management,onpage 13foranintroduction. Useadvancedsettingsundercertaincircumstances.SeeChapter 9,Advanced ResourceManagement,onpage 123. UsetheVirtualInfrastructureSDKforscriptedCPUallocation. Usehyperthreading,asdiscussedinHyperthreadingonpage 129. NOTECPUaffinityisnotusuallyrecommended.SeeUsingCPUAffinityto AssignVirtualMachinestoSpecificProcessorsonpage 126forinformationon CPUaffinityandpotentialproblemswithit. IfyoudonotcustomizeCPUallocation,theESXServerhostusesdefaultsthatwork wellinmostsituations.
38
VMware, Inc.
How Administrators Can Affect Memory Management

Youhaveaccesstoinformationaboutcurrentmemoryallocationsandotherstatus informationthroughtheVIClientorusingtheVirtualInfrastructureSDK. Specifymemoryallocationintheseways: UsetheattributesandspecialfeaturesavailablethroughtheVIClient.The VIClientGUIallowsyoutoconnecttoanESXServerhostoraVirtualCenter Server.SeeChapter 1,GettingStartedwithResourceManagement,onpage 13 foranintroduction. Useadvancedsettingsundercertaincircumstances.SeeChapter 9,Advanced ResourceManagement,onpage 123. UsetheVirtualInfrastructureSDKforscriptedmemoryallocation. Ifyoudonotcustomizememoryallocation,theESXServerhostusesdefaultsthatwork wellinmostsituations. ForserverswithNUMAarchitecture,seeChapter 10,UsingNUMASystemswithESX Server,onpage 151.
Understanding CPU and Memory Virtualization

Thissectiondiscussesvirtualizationandwhatitmeansfortheresourcesavailablefor thevirtualmachines.
CPU Virtualization Basics

Youcanconfigurevirtualmachineswithoneormorevirtualprocessors,eachwithits ownsetofregistersandcontrolstructures.Whenavirtualmachineisscheduled,its virtualprocessorsarescheduledtorunonphysicalprocessors.TheVMkernelResource ManagerschedulesthevirtualCPUsonphysicalCPUs,therebymanagingthevirtual machinesaccesstophysicalCPUresources.ESXServersupportsvirtualmachineswith uptofourvirtualprocessors.SeeMulticoreProcessorsonpage 128. NOTEWhenWindowsVistaistheguestoperatingsystem,onlytwovirtualCPUsare supportedpervirtualmachine.
VMware, Inc.
39
To view information about physical and logical processors 1 2 IntheVIClient,selectthehostandclicktheConfigurationtab. SelectProcessors.
Youcanviewtheinformationaboutthenumberandtypeofphysicalprocessors andthenumberoflogicalprocessors.Youcanalsodisableorenable hyperthreadingbyclickingProperties. NOTEInhyperthreadedsystems,eachhardwarethreadisalogicalprocessor.A dualcoreprocessorwithhyperthreadingenabledhastwocoresandfourlogical processors.
Memory Virtualization Basics

TheVMkernelmanagesallmachinememory.(Anexceptiontothisisthememorythat isallocatedtotheserviceconsoleinESXServer3.)TheVMkerneldedicatespartofthis managedmachinememoryforitsownuse.Therestisavailableforusebyvirtual machines.Virtualmachinesusemachinememoryfortwopurposes:eachvirtual machinerequiresitsownmemoryandtheVMMrequiressomememoryforitscode anddata. To view information on how a hosts memory is being used 1 2 IntheVIClient,selectthehost. ClicktheConfigurationtab.
40
VMware, Inc.
SelectMemory.
Youcanviewtheinformationaboutthetotalmemoryandmemoryavailableto virtualmachines.InESXServer3,youcanalsoviewmemoryassignedtothe serviceconsole.
Virtual Machine Memory

Eachvirtualmachineconsumesmemorybasedonitsconfiguredsize,plusadditional overheadmemoryforvirtualization. Configured Size. Theconfiguredsizeisaconstructmaintainedbythevirtualization layerforthevirtualmachine.Itistheamountofmemorythatispresentedtotheguest operatingsystem,butitisindependentoftheamountofphysicalRAMthatisallocated tothevirtualmachine,whichdependsontheresourcesettings(shares,reservation, limit)explainedbelow. Forexample,consideravirtualmachinewithaconfiguredsizeof1GB.Whentheguest operatingsystemboots,itbelievesthatitisrunningonadedicatedmachinewith1GB ofphysicalmemory.Theactualamountofphysicalhostmemoryallocatedtothe virtualmachinedependsonitsmemoryresourcesettingsandmemorycontentionon theESXServerhost.Insomecases,thevirtualmachinemightbeallocatedthefull1GB. Inothercases,itmightreceiveasmallerallocation.Regardlessoftheactualallocation, theguestoperatingsystemcontinuestobehaveasthoughitisrunningonadedicated machinewith1GBofphysicalmemory. Shares. Specifytherelativepriorityforavirtualmachineifmorethanthereservation isavailable.SeeSharesonpage 20.
VMware, Inc.
41
Reservation. Isaguaranteedlowerboundontheamountofphysicalmemorythatthe hostreservesforthevirtualmachine,evenwhenmemoryisovercommitted.Setthe reservationtoalevelthatensuresthevirtualmachinehassufficientmemorytorun efficiently,withoutexcessivepaging. Limit. Isanupperboundontheamountofphysicalmemorythatthehostwillallocate tothevirtualmachine.Thevirtualmachinesmemoryallocationisalsoimplicitly limitedbyitsconfiguredsize. Overheadmemoryincludesspacereservedforthevirtualmachineframebufferand variousvirtualizationdatastructures.SeeUnderstandingMemoryOverheadon page 136.
Memory Overcommitment
Foreachrunningvirtualmachine,thesystemreservesphysicalmemoryforthevirtual machinesreservation(ifany)andforitsvirtualizationoverhead.Becauseofthe memorymanagementtechniquestheESXServerhostuses,yourvirtualmachinescan usemorememorythanthephysicalmachine(thehost)hasavailable.Forexample,you canhaveahostwith2GBmemoryandrunfourvirtualmachineswith1GBmemory each.Inthatcase,thememoryisovercommitted. Overcommitmentmakessensebecause,typically,somevirtualmachinesarelightly loadedwhileothersaremoreheavilyloaded,andrelativeactivitylevelsvaryovertime. Toimprovememoryutilization,theESXServerhosttransfersmemoryfromidlevirtual machinestovirtualmachinesthatneedmorememory.UsetheReservationorShares parametertopreferentiallyallocatememorytoimportantvirtualmachines.This memoryremainsavailabletoothervirtualmachinesifitisnotinuse.
Memory Sharing
Manyworkloadspresentopportunitiesforsharingmemoryacrossvirtualmachines. Forexample,severalvirtualmachinesmightberunninginstancesofthesameguest operatingsystem,havethesameapplicationsorcomponentsloaded,orcontain commondata.ESXServersystemsuseaproprietarypagesharingtechniquetosecurely eliminateredundantcopiesofmemorypages. Withmemorysharing,aworkloadconsistingofmultiplevirtualmachinesoften consumeslessmemorythanitwouldwhenrunningonphysicalmachines.Asaresult, thesystemcanefficientlysupporthigherlevelsofovercommitment. Theamountofmemorysavedbymemorysharingdependsonworkload characteristics.Aworkloadofmanynearlyidenticalvirtualmachinesmightfreeup morethanthirtypercentofmemory,whileamorediverseworkloadmightresultin savingsoflessthanfivepercentofmemory.
42 VMware, Inc.
Understanding and Managing Resource Pools
ThischapterintroducesresourcepoolsandexplainshowVirtualInfrastructureallows youtoviewandmanipulatethem. Thischapterdiscussesthefollowingtopics: WhatAreResourcePools?onpage 44 ResourcePoolAdmissionControlonpage 47 CreatingResourcePoolsonpage 48 ViewingResourcePoolInformationonpage 50 ChangingResourcePoolAttributesonpage 54 MonitoringResourcePoolPerformanceonpage 54 AddingVirtualMachinestoResourcePoolsonpage 55 RemovingVirtualMachinesfromResourcePoolsonpage 56 ResourcePoolsandClustersonpage 56 Alltasksassumeyouhavepermissiontoperformthem.SeetheonlineHelpfor informationonpermissionsandhowtosetthem.
VMware, Inc.
43
What Are Resource Pools?

UseresourcepoolstohierarchicallypartitionavailableCPUandmemoryresources. EachstandalonehostandeachDRSclusterhasan(invisible)rootresourcepoolthat groupstheresourcesofthathostorcluster.Therootresourcepoolisnotdisplayed becausetheresourcesofthehost(orcluster)andtherootresourcepoolarealwaysthe same. NOTEVMwareDRShelpsyoubalanceresourcesacrossvirtualmachines.Itis discussedinUnderstandingVMwareDRSonpage 62. Ifyoudonotcreatechildresourcepools,onlytherootresourcepoolsexist. Userscancreatechildresourcepoolsoftherootresourcepoolorofanyusercreated childresourcepool.Eachchildresourcepoolownssomeoftheparentsresourcesand can,inturn,haveahierarchyofchildresourcepoolstorepresentsuccessivelysmaller unitsofcomputationalcapability. Aresourcepoolcancontainchildresourcepools,virtualmachines,orboth.Youcan createahierarchyofsharedresources.Theresourcepoolsatahigherlevelarecalled parentresourcepools.Resourcepoolsandvirtualmachinesthatareatthesamelevelare calledsiblings.Theclusteritselfrepresentstherootresourcepool. Figure 3-1. Parents, Children, and Siblings in Resource Pool Hierarchy
root resource pool siblings child resource pool parent resource pool siblings
InFigure 31,RPQAistheparentresourcepoolforRPQAUI.RPMarketingand RPQAaresiblings.ThethreevirtualmachinesimmediatelybelowRPMarketingare alsosiblings. Foreachresourcepool,specifyreservation,limit,shares,andwhetherthereservation shouldbeexpandable.Theresourcepoolresourcesarethenavailabletochildresource poolsandvirtualmachines.
44
VMware, Inc.
Chapter 3 Understanding and Managing Resource Pools
Why Use Resource Pools?

Resourcepoolsallowyoutodelegatecontroloverresourcesofahost(oracluster),but thebenefitsareespeciallyevidentwhenyouuseresourcepoolstocompartmentalize allresourcesinacluster.Createmultipleresourcepoolsasdirectchildrenofthehostor clusterandconfigurethem.Youcanthendelegatecontrolovertheresourcepoolsto otherindividualsororganizations. Usingresourcepoolscanresultinthefollowingbenefits: FlexiblehierarchicalorganizationAdd,remove,orreorganizeresourcepools orchangeresourceallocationsasneeded. Isolationbetweenpools,sharingwithinpoolsTopleveladministratorscan makeapoolofresourcesavailabletoadepartmentleveladministrator.Allocation changesthatareinternaltoonedepartmentalresourcepooldonotunfairlyaffect otherunrelatedresourcepools. AccesscontrolanddelegationWhenatopleveladministratormakesa resourcepoolavailabletoadepartmentleveladministrator,thatadministratorcan thenperformallvirtualmachinecreationandmanagementwithintheboundaries oftheresourcestowhichtheresourcepoolisentitledbythecurrentshares, reservation,andlimitsettings.Delegationisusuallydoneinconjunctionwith permissionssettings,whicharediscussedintheIntroductiontoVirtual Infrastructure. SeparationofresourcesfromhardwareIfyouareusingclustersenabledfor DRS,theresourcesofallhostsarealwaysassignedtothecluster.Thatmeans administratorscanperformresourcemanagementindependentlyoftheactual hoststhatcontributetheresources.Ifyoureplacethree2GBhostswithtwo3GB hosts,youdonotneedtomakechangestoyourresourceallocations. Thisseparationallowsadministratorstothinkmoreaboutaggregatecomputing capacityandlessaboutindividualhosts. ManagementofsetsofvirtualmachinesrunningamultitierserviceYoudo notneedtosetresourcesoneachvirtualmachine.Instead,youcancontrolthe aggregateallocationofresourcestothesetofvirtualmachinesbychanging settingsontheirenclosingresourcepool.
VMware, Inc.
45
Forexample,assumeahosthasanumberofvirtualmachines.Themarketing departmentusesthreeofthevirtualmachinesandtheQAdepartmentusestwovirtual machines.BecausetheQAdepartmentneedslargeramountsofCPUandmemory,the administratorcreatesoneresourcepoolforeachgroup.TheadministratorsetsCPU SharestoHighfortheQAdepartmentpoolandtoNormalfortheMarketing departmentpoolsothattheQAdepartmentuserscanrunautomatedtests.Thesecond resourcepoolwithfewerCPUandmemoryresourcesissufficientforthelighterload ofthemarketingstaff.WhenevertheQAdepartmentisnotfullyusingitsallocation,the marketingdepartmentcanusetheavailableresources. ThisscenarioisshowninFigure 32.Thenumbersshowtheeffectiveallocationstothe resourcepools. Figure 3-2. Allocating Resources to Resource Pools
ESX Server host 6GHz, 3GB
RP-QA
4GHz, 2GB
RPMarketing 2GHz, 1GB
VM-QA 1
VM-QA 2
VM-Marketing 1
VM-Marketing 2
VM-Marketing 3
Host Resource Pools and Cluster Resource Pools

YoucancreatechildresourcepoolsofstandaloneESXServerhostsorofDRSclusters. ForstandaloneESXServerhosts,youcreateandmanageresourcepoolsaschildren ofthehost.Eachhostsupportsitsownhierarchyofresourcepools. IfyouaddahosttoaclusterthatisnotenabledforDRS,thehostsresourcepool hierarchyisdiscarded,andnoresourcepoolhierarchycanbecreated. ForclustersenabledforDRS,theresourcesofallhostsareassignedtothecluster. WhenyouaddahostwithresourcepoolstoaDRScluster,youarepromptedto decideonresourcepoolplacement.Bydefault,theresourcepoolhierarchyis discardedandthehostisaddedatthesamelevelasthevirtualmachines.Youcan choosetograftthehostsresourcepoolsontotheclustersresourcepoolhierarchy andchooseanameforthetoplevelresourcepool.SeeResourcePoolsand Clustersonpage 56.
46 VMware, Inc.
Becauseallresourcesarecombined,younolongermanageresourcesfor individualhostsbutmanageallresourcesinthecontextofthecluster.Youassign virtualmachinestoresourcepoolswithpredefinedcharacteristics.Ifyoulater changecapacitybyadding,removing,orupgradinghosts,youmighthaveto changetheresourceallocationsyoumadefortheresourcepools. IftheVirtualCenterServerbecomesunavailable,makechangesusingaVIClient connectedtoanESXServerhost.However,theclustermightbecomeyellow (overcommitted)orred(invalid)whentheVirtualCenterServerbecomesavailable again.SeeValid,Yellow,andRedClustersonpage 81.Ifyourclusterisin automaticmode,VirtualCenterreappliesthelastknownclusterconfiguration(and potentiallyundoesyourchanges)whentheVirtualCenterServerbecomes availableagain.
Resource Pool Admission Control

WhenyoupoweronvirtualmachinesonanESXServerhost,thehostfirstperforms basicadmissioncontrol,asdiscussedinAdmissionControlonpage 22.Whenyou poweronavirtualmachineinsidearesourcepool,orattempttocreateachildresource pool,thesystemperformsadditionaladmissioncontroltoensuretheresourcepools restrictionsarenotviolated. Beforeyoupoweronavirtualmachineorcreatearesourcepool,checktheCPU UnreservedandMemoryUnreservedfieldsintheresourcepoolsResourceAllocation tabtodetermine(seeFigure 33)whethersufficientresourcesareavailable. Figure 3-3. Resource Pool Reservation Information
Unreserved
Reservation type
VMware, Inc.
47
HowunreservedCPUandmemoryarecomputedandwhetheractionsareperformed dependsonthereservationtype: Fixedreservationtype.Thesystemcheckswhethertheresourcepoolhassufficient unreservedresources.Ifitdoes,theactioncanbeperformed.Ifitdoesnot,a messageappearsandtheactioncannotbeperformed. Expandablereservationtype.Thesystemcheckswhethertheresourcepoolhas sufficientresourcestofulfilltherequirements. Iftherearesufficientresources,theactionisperformed. Iftherearenotsufficientresources,themanagingservercheckswhether resourcesareavailableinaparentresourcepool(directparentorancestor).If theyare,theactionisperformedandtheparentresourcepoolresourcesare reserved.Ifnoresourcesareavailable,amessageappearsandtheactionisnot performed.SeeUnderstandingExpandableReservationonpage 29. ThesystemdoesnotallowyoutoviolatepreconfiguredReservationorLimitsettings. Eachtimeyoureconfigurearesourcepoolorpoweronavirtualmachine,thesystem validatesallparameterssoallservicelevelguaranteescanstillbemet.
Creating Resource Pools

YoucancreateachildresourcepoolofanyESXServer3.xhost,resourcepool,orDRS cluster. NOTEIfahosthasbeenaddedtoacluster,youcannotcreatechildresourcepoolsof thathost.Youcancreatechildresourcepoolsoftheclusteriftheclusterisenabledfor DRS. Whenyoucreateachildresourcepool,youarepromptedforresourcepoolattribute information.Thesystemusesadmissioncontroltomakesureyoucannotallocate resourcesthatarenotavailable.Forexample,ifyouhavearesourcepoolwitha reservationof10GB,andyoucreatedachildresourcepoolwithareservationof6GB, youcannotcreateasecondchildresourcepoolwithareservationof6GBandTypeset toFixed. To create a resource pool 1 2 SelecttheintendedparentandchooseFile>New>NewResourcePool(orclick NewResourcePoolintheCommandspaneloftheSummarytab). IntheNewResourcePooldialogbox,providethefollowinginformationforyour resourcepool.
48
VMware, Inc.
Field Name CPU Resources Shares
Description Nameofthenewresourcepool.
NumberofCPUsharestheresourcepoolhaswithrespecttothe parentstotal.Siblingresourcepoolsshareresourcesaccording totheirrelativesharevaluesboundedbythereservationand limit.YoucanchooseLow,Normal,orHigh,orchooseCustom tospecifyanumberthatassignsasharevalue. GuaranteedCPUallocationforthisresourcepool. Indicateswhetherexpandablereservationsareconsidered duringadmissioncontrol.Ifyoupoweronavirtualmachinein thisresourcepool,andthereservationsofthevirtualmachines combinedarelargerthanthereservationoftheresourcepool, theresourcepoolcanuseresourcesfromitsparentorancestors ifthischeckboxisselected(thedefault). UpperlimitfortheamountofCPUthehostmakesavailableto thisresourcepool.DefaultisUnlimited.Tospecifyalimit, deselecttheUnlimitedcheckboxandtypeinthenumber.
Reservation Expandable Reservation
Limit
Memory Resources Shares Numberofmemorysharestheresourcepoolhaswithrespectto theparentstotal.Siblingresourcepoolsshareresources accordingtotheirrelativesharevaluesboundedbythe reservationandlimit.YoucanchooseLow,Normal,orHigh,or chooseCustomtospecifyanumberthatassignsasharevalue. Guaranteedmemoryallocationforthisresourcepool. Indicateswhetherexpandablereservationsareconsidered duringadmissioncontrol.Ifyoupoweronavirtualmachinein thisresourcepool,andthereservationsofthevirtualmachines combinedarelargerthanthereservationoftheresourcepool, theresourcepoolcanuseaparentsorancestorsresourcesifthis checkboxisselected(thedefault). Upperlimitforthisresourcepoolsmemoryallocation.Default isUnlimited.Tospecifyadifferentlimit,deselecttheUnlimited checkbox.
Reservation Expandable Reservation
Limit
Afteryouhavemadeallchoices,clickOK. VirtualCentercreatestheresourcepoolanddisplaysitintheinventorypanel. Ayellowtriangleisdisplayedifanyoftheselectedvaluesarenotlegalvalues becauseoflimitationsontotalavailableCPUandmemory.
VMware, Inc.
49
Understanding Expandable Reservations

Whenyoupoweronavirtualmachineorcreatearesourcepool,thesystemchecks whethertheCPUandmemoryreservationisavailableforthataction. IfExpandableReservationisnotselected,thesystemconsidersonlytheresources availableintheselectedresourcepool. IfExpandableReservationisselected(thedefault),thesystemconsiderstheresources availableintheselectedresourcepoolanditsdirectparentresourcepool.Iftheparent resourcepoolalsohastheExpandableReservationoptionselected,itcanborrow resourcesfromitsparentresourcepool.Borrowingresourcesoccursrecursivelyfrom theancestorsofthecurrentresourcepoolaslongastheExpandableReservation optionisselected.Leavingthisoptionselectedoffersmoreflexibility,but,atthesame timeprovideslessprotection.Achildresourcepoolownermightreservemore resourcesthanyouanticipate. NOTELeavethisoptionselectedonlyifyoutrusttheadministratorofthechild resourcepooltonotreservemoreresourcesthanappropriate. Expandable Reservations Example AssumeanadministratormanagespoolP,and definestwochildresourcepools,S1andS2,fortwodifferentusers(orgroups). Theadministratorknowsthatuserswillwanttopoweronvirtualmachineswith reservations,butdoesnotknowhowmucheachuserwillneedtoreserve.Makingthe reservationsforS1andS2expandableallowstheadministratortomoreflexiblyshare andinheritthecommonreservationforpoolP. Withoutexpandablereservations,theadministratorneedstoexplicitlyallocateS1and S2aspecificamount.Suchspecificallocationscanbeinflexible,especiallyindeep resourcepoolhierarchiesandcancomplicatesettingreservationsintheresourcepool hierarchy. Expandablereservationscausealossofstrictisolation;thatis,S1canstartusingallof Psreservation,sothatnomemoryorCPUisdirectlyavailabletoS2.
Viewing Resource Pool Information

WhenyouselectaresourcepoolintheVIClient,theSummarytabdisplays informationaboutthatresourcepool.Thefollowingsectionlistsinformationaboutthe ResourcePoolSummaryTabonpage 51andResourcePoolResourceAllocation Tabonpage 52. NOTEAllothertabsarediscussedindetailintheonlineHelp.
50
VMware, Inc.
Resource Pool Summary Tab

TheresourcepoolsSummarytabdisplayshighlevelstatisticalinformationaboutthe resourcepool.
Table 3-1. Resource Pool Summary Tab Sections

Section General Description TheGeneralpaneldisplaysstatisticalinformationfortheresourcepool. NumberofVirtualMachinesinthisresourcepool.Doesnotinclude thenumberofvirtualmachinesinchildresourcepools. NumberofRunningVirtualMachinesinthisresourcepool.Doesnot includethenumberofvirtualmachinesrunninginchildresourcepools. NumberofChildResourcePoolsDoesnotincludeallresourcepools inthehierarchybutonlydirectchildren. CPU DisplaystheCPUShares,Reservation,ReservationType,andLimitthat werespecifiedforthisresourcepool.AlsodisplaystheamountofCPU currentlyunreserved. Allowsyoutocallcommonlyusedcommands. NewVirtualMachineStartstheNewVirtualMachinewizardto createanewvirtualmachineinthisresourcepool. NewResourcePoolDisplaystheCreateResourcePooldialogbox, whichallowsyoutocreateachildresourcepooloftheselectedresource pool. EditSettingsAllowsyoutochangetheCPUandmemoryattributes fortheselectedresourcepool.
Commands
VMware, Inc.
51
Table 3-1. Resource Pool Summary Tab Sections (Continued)

Section Resources Memory Description DisplaysruntimeCPUUsageandMemoryUsageforthevirtualmachines withintheselectedresourcepool. DisplaystheShares,Reservation,ReservationType,andLimitthatwere specifiedforthisresourcepool.Alsodisplaystheamountofmemory currentlyunreserved.
Resource Pool Resource Allocation Tab

TheresourcepoolsResourceAllocationtabshowsdetailedinformationaboutthe resourcescurrentlyreservedandavailablefortheresourcepoolandliststheuserofthe resources,asdiscussedinTable 32andTable 33. Figure 3-4. Resource Pool Resource Allocation Tab
ThetopportionofFigure 34specifiestheinformationinTable 32abouttheresource poolitself. Table 3-2. Resource Allocation Tab Fields
Field CPUReservation/ MemoryReservation CPUReservationUsed/ MemoryReservationUsed Description AmountofCPUormemoryspecifiedinthereservationforthis resourcepool.Reservationcanbespecifiedduringresource poolcreation,orlaterbyeditingtheresourcepool. CPUormemoryreservationused.Reservationsareusedby runningvirtualmachinesorbychildresourcepoolswith reservations.
52
VMware, Inc.
Table 3-2. Resource Allocation Tab Fields (Continued)

Field CPUUnreserved/ MemoryUnreserved Description CPUormemorycurrentlyunreservedandavailabletobe reservedbyvirtualmachinesandresourcepools. Note:Lookatthisnumberwhentryingtodeterminewhether youcancreateachildresourcepoolofacertainsize,orwhether youcanpoweronavirtualmachinewithacertainreservation. CPUReservationType/ MemoryReservationType ExpandableorFixed.SeeUnderstandingExpandable Reservationsonpage 50.
InFigure 34,belowtheinformationspecifictotheresourcepoolisalistofthevirtual machinesandchildresourcepoolsofthisresourcepool.Thislistdoesnotcontain virtualmachinesassignedtochildresourcepoolsofthisresourcepool. ClicktheCPUorMemorytabtodisplaytheinformationdescribedinTable 33. Table 3-3. Resource Allocation CPU and Memory Fields
Field Name Reservation Limit Shares Description Nameoftheresourcepoolorvirtualmachine. Specifiedreservationforthisvirtualmachineorresourcepool.Defaultis0,that is,thesystemreservesnoresourcesforthisresourcepool. Specifiedlimitforthisvirtualmachineorresourcepool.DefaultisUnlimited, thatis,thesystemallocatesasmanyresourcestothisvirtualmachineasitcan. Specifiedsharesforthisvirtualmachineorresourcepool.OneofHigh, Normal,Lowifoneofthedefaultsettingshasbeenselected.Customifyou selectacustomsetting. Numberofsharesallocatedtothisvirtualmachineorresourcepool.This numberdependsonthesharessetting(High,Normal,Low,orCustom).See Sharesonpage 20. Sharesvalueforthisresourcepoolorvirtualmachinedividedbythetotal numberofsharesallocatedtoallchildrenoftheparentresourcepool.This valueisunrelatedtotheparentresourcepoolslocalsharesallocation. Reservationtype.EitherFixedorExpandable.SeeUnderstanding ExpandableReservationonpage 29.
Shares Value %Shares
Type
VMware, Inc.
53
Changing Resource Pool Attributes

To make changes to a resource pool 1 2 SelecttheresourcepoolintheVIClientinventorypanel. IntheSummarytabCommandpanel,chooseEditSettings.
IntheEditSettingsdialogbox,youcanchangeallattributesoftheselected resourcepool. ThechoicesarediscussedinCreatingResourcePoolsonpage 48.
Monitoring Resource Pool Performance

Monitoringaresourcepoolsperformanceisusefultounderstandtheeffectivenessof resourcepoolallocations. To monitor a resource pools performance 1 2 Selecttheresourcepoolintheinventorypanel. ClickthePerformancetab.
Youseeinformationaboutresourcepoolperformance.ClickChangeChartOptionsto customizetheperformancechart.SeetheonlineHelpforadiscussionofperformance chartsandhowtoconfigurethem.
54
VMware, Inc.
Adding Virtual Machines to Resource Pools

Whenyoucreateanewvirtualmachine,theVirtualMachinewizardallowsyoutoadd ittoaresourcepoolaspartofthecreationprocess.Youcanalsoaddanalreadyexisting virtualmachinetoaresourcepool.Thissectiondiscussesbothtasks. To create a virtual machine and add it to a resource pool 1 2 SelectahostandchooseFile>New>VirtualMachine(orpressCtrl+n). Supplytheinformationforthevirtualmachine,choosingaresourcepoolasthe locationwhenpromptedbythewizard. Thewizardplacesthevirtualmachineintotheresourcepoolyouselected. To add an existing virtual machine to a resource pool 1 Selectthevirtualmachinefromanylocationintheinventory. Thevirtualmachinecanbeassociatedwithastandalonehost,acluster,ora differentresourcepool. 2 Dragthevirtualmachine(ormachines)totheresourcepoolobjectyouwant. Whenyoumoveavirtualmachinetoanewresourcepool: Thevirtualmachinesreservationandlimitdonotchange. Ifthevirtualmachinessharesarehigh,medium,orlow,%Sharesadjuststo reflectthetotalnumberofsharesinuseinthenewresourcepool. Ifthevirtualmachinehascustomsharesassigned,thesharevalueis maintained. NOTEBecauseshareallocationsarerelativetoaresourcepool,youmight havetomanuallychangeavirtualmachinesshareswhenyoumoveitintoa resourcepoolsothatthevirtualmachinessharesareconsistentwiththe relativevaluesinthenewresourcepool.Awarningappearsifavirtual machinewouldreceiveaverylarge(orverysmall)percentageoftotalshares. TheinformationdisplayedintheResourceAllocationtababouttheresource poolsreservedandunreservedCPUandmemoryresourceschangestoreflect thereservationsassociatedwiththevirtualmachine(ifany). NOTEIfavirtualmachinehasbeenpoweredofforsuspended,itcanbe movedbutoverallavailableresources(suchasreservedandunreservedCPU andmemory)fortheresourcepoolarenotaffected.
VMware, Inc.
55
Ifavirtualmachineispoweredon,andthedestinationresourcepooldoesnothave enoughCPUormemorytoguaranteethevirtualmachinesreservation,themovefails becauseadmissioncontroldoesnotallowit.Anerrordialogboxexplainsthesituation. Theerrordialogboxcomparesavailableandrequestedresources,soyoucanconsider whetheranadjustmentmightresolvetheissue.SeeResourcePoolAdmission Controlonpage 47.
Removing Virtual Machines from Resource Pools

Youcanremoveavirtualmachinefromaresourcepoolinanumberofways, dependingonyourintentionforthemachine. Move the virtual machine to a different resource pool. SeeToaddanexisting virtualmachinetoaresourcepoolonpage 55.Youdonotneedtopoweroffavirtual machineifyouonlymoveit. Whenyouremoveavirtualmachinefromaresourcepool,thetotalnumberofshares associatedwiththeresourcepooldecreases,sothateachremainingsharerepresents moreresources.Forexample,assumeyouhaveapoolthatisentitledto6GHz, containingthreevirtualmachineswithsharessettoNormal.Assumingthevirtual machinesareCPUbound,eachgetsanequalallocationof2GHz.Ifoneofthevirtual machinesismovedtoadifferentresourcepool,thetworemainingvirtualmachines eachreceiveanequalallocationof3GHz. Remove the virtual machine from the inventory or delete it from the disk. Rightclickthevirtualmachine(orpressDelete). Youneedtopoweroffthevirtualmachinebeforeyoucancompletelyremoveit.Seethe VirtualInfrastructureUsersGuide.
Resource Pools and Clusters

Whenyouaddahostwithanexistingresourcepoolhierarchytoacluster,what happensdependsonthecluster.Youhavetwooptions: ClustersEnabledforDRSonpage 57 ClustersNotEnabledforDRSonpage 58
56
VMware, Inc.
Clusters Enabled for DRS

IfaclusterisenabledforDRS,andyoumoveoneormorehostsintothecluster,a wizardallowsyoutochoosewhathappenstothehostsresourcepools. Put this hosts virtual machines in the clusters root resources. Collapsesthe hostsresourcepoolhierarchyandmakesallvirtualmachinesdirectchildrenofthe cluster.ThisbehavioristhesameasthatshownforClustersNotEnabledforDRSon page 58. NOTEYoumighthavetomanuallyadjustthesharevaluesassociatedwith individualvirtualmachinesbecausethesharesintheoriginalhosthierarchyare relativetotheresourcepoolsonthehost. Create a new resource pool for the hosts virtual machines and resource pools. Createsaresourcepoolcorrespondingtothehostsrootresourcepool.Bydefault,the resourcepoolisnamedGraftedfrom<host_name>,butyoucanchooseadifferent name.Thetermgraftedwaschosenbecausethebranchesofthehoststreeareaddedto thebranchesoftheclusterstree. Figure 3-5. Resource Pool Hierarchy Grafted onto Cluster
cluster
Host1
cluster
Grafted from Host1 RP
X Y
X Y
IntheexampleinFigure 35,clusterClusterandhostHost1eachhaveahierarchyof resourcepools.Whenyouaddthehosttothecluster,thehostsinvisibletoplevel resourcepoolisgraftedontotheclustersresourcepoolhierarchyandisnamedgrafted fromHost1bydefault. NOTESharesremainallocatedastheywerebeforethehostmovedintothecluster. Percentagesareadjustedasappropriate.
VMware, Inc.
57
Theresourcepoolhierarchybecomescompletelyindependentofthehost.Ifyoulater removethehostfromthecluster,theclusterkeepstheresourcepoolhierarchyandthe hostlosestheresourcepoolhierarchy(thoughthevirtualmachinesareremovedalong withthehost).SeeRemovingHostsfromClustersonpage 104. NOTEThehostmustbeinmaintenancemodebeforeyoucanremoveitfromthe cluster.SeeHostMaintenanceandStandbyModesonpage 71.
Clusters Not Enabled for DRS

IftheclusterisenabledforHAonly(orneitherHAnorDRS),andyoumoveoneor morehostsintothecluster,theclustertakesownershipoftheresources.Thehostsand virtualmachinesbecomeassociatedwiththecluster.Theresourcepoolhierarchyis flattened. NOTEInanonDRSclusterthereisnoclusterwideresourcemanagementbasedon shares.Virtualmachinesharesremainrelativetoeachhost. InFigure 36,hostH1andhostH2eachhaveahierarchyofresourcepoolsandvirtual machines.WhenthetwohostsareaddedtoclusterC,theresourcepoolhierarchyis flattenedandallvirtualmachinesbecomedirectchildrenofthecluster. Figure 3-6. Flattened Resource Pool Hierarchy
cluster
Host1 RP
Host2 RP VM1
cluster
VM2
VM3
VM4
RP
RP
RP
VM1
VM2
VM3
VM4
58
VMware, Inc.
Understanding Clusters
ThischapterpresentsaconceptualintroductiontoclustersandtotheVMware DistributedResourceScheduler(DRS)andHighAvailability(HA)features. Thischapterdiscussesthefollowingtopics: IntroductiontoClustersonpage 59 UnderstandingVMwareDRSonpage 62 UnderstandingVMwareHAonpage 72 UsingHAandDRSTogetheronpage 80 Valid,Yellow,andRedClustersonpage 81 NOTEAlltasksdescribedassumeyouhavepermissiontoperformthem.Seetheonline Helpforinformationonpermissionsandhowtosetthem.
Introduction to Clusters
AclusterisacollectionofESXServerhostsandassociatedvirtualmachineswithshared resourcesandasharedmanagementinterface.Whenyouaddahosttoacluster,the hostsresourcesbecomepartoftheclustersresources.Whenyoucreateacluster,you canchoosetoenableitforDRS,HA,orboth. SeeClusterPrerequisitesonpage 89forinformationonthevirtualmachinesin clustersandonhowtoconfigurethem. NOTEYoucancreateaclusterwithoutaspeciallicense,butyoumusthavealicenseto enableaclusterforDRSorHA.
VMware, Inc.
59
VMware DRS
TheDRSfeatureimprovesresourceallocationacrossallhostsandresourcepools.DRS collectsresourceusageinformationforallhostsandvirtualmachinesintheclusterand generatesrecommendationsforvirtualmachineplacementandhostmachinepower state.Theserecommendationscanbeappliedautomatically.Dependingonthe configuredDRSautomationlevel,DRSdisplaysorappliesthefollowingtypesof recommendations: InitialplacementWhenyoufirstpoweronavirtualmachineinthecluster,DRS eitherplacesthevirtualmachineonanappropriatehostormakesa recommendation.SeeInitialPlacementonpage 62. MigrationAtruntime,DRStriestofixruleviolationsandimproveresource utilizationacrosstheclustereitherbyperformingmigrationsofvirtualmachines, orbyprovidingrecommendationsforvirtualmachinemigrations.SeeLoad BalancingandVirtualMachineMigrationonpage 66. PowermanagementWhentheDistributedPowerManagementfeatureis enabled,DRScomparesclusterandhostlevelcapacitytothedemandsofrunning theclustersvirtualmachines,includingrecenthistoricaldemand.Itmakes recommendationsforplacinghostsinstandbypowermodeifsufficientexcess capacityisfoundorpoweringonhostsifcapacityisneeded.Dependingonthe resultinghostpowerstaterecommendations,virtualmachinesmightneedtobe migratedtoandfromthehostsaswell.SeeDistributedPowerManagementon page 68.
VMware HA
AclusterenabledforHAmonitorsforhostfailure.Ifahostbecomesunavailable,all virtualmachinesthatwereonthehostarepromptlyrestartedondifferenthosts. WhenyouenableaclusterforHA,youcanspecifythenumberofhostfailuresallowed. Ifyousetthenumberofhostfailuresas1,HAmaintainsenoughcapacityacrossthe clustertotoleratethefailureofonehost,sothatallrunningvirtualmachinesonthat hostcanberestartedonremaininghosts.Bydefault,youcannotpoweronavirtual machineifdoingsoviolatesrequiredfailovercapacity(strictadmissioncontrol).See UnderstandingVMwareHAonpage 72.
60
VMware, Inc.
Chapter 4 Understanding Clusters
Figure 4-1. VMware HA

VM1 VM2 VM3 VM1 VM2 VM3 VM1 VM4 Host A Host A VM2 VM6 Host A VM3
VM4
VM5
VM6
VM4
VM5
VM6
Host B
Host B
VM7
VM8
VM9
VM7
VM8
VM9
VM7 VM5
VM8
VM9
Host C
Host C
Host C
All three hosts run
One host goes down
The affected virtual machines have been restarted on the remaining hosts
InFigure 41,threehostshavethreevirtualmachineseach,andthecorrespondingHA clusterisconfiguredforfailoverofonehost.WhenHostBbecomesunavailable,HA migratesthevirtualmachinesfromHostBtoHostAandHostC.
Clusters and VirtualCenter Failure

TheVirtualCenterServerplacesanagentoneachhost.IftheVirtualCenterServer becomesunavailable,HAandDRSfunctionalitychangesasfollows: HAHAclusterscontinuetoworkeveniftheVirtualCenterServerbecomes unavailable,andcanstillrestartvirtualmachinesonotherhostsincaseoffailover. However,theinformationaboutvirtualmachinespecificclusterproperties(such aspriorityandisolationresponse)isbasedonthestateoftheclusterbeforethe VirtualCenterServerwentdown. DRSThehostsinDRSclusterscontinuerunningusingavailableresources. However,therearenorecommendationsforresourceoptimization. IfyoumustmakechangestothehostsorvirtualmachinesusingaVIClientconnected toanESXServerhostwhiletheVirtualCenterServerisunavailable,thosechangesdo takeeffect.WhenVirtualCenterbecomesavailableagain,youmightfindthatclusters haveturnedredoryellowbecauseclusterrequirementsarenolongermet.
VMware, Inc. 61
Understanding VMware DRS

WhenyouenableaclusterforDRS,VirtualCentercontinuouslymonitorsthe distributionofCPUandmemoryresourcesforallhostsandvirtualmachinesinthe cluster.DRScomparesthesemetricstowhatresourceutilizationideallyshouldbe giventheattributesoftheresourcepoolsandvirtualmachinesintheclusterandthe currentdemandandmakesmigrationrecommendationsaccordingly.Also,whenthe DistributedPowerManagementfeatureisenabled,DRSmonitorstheclusterand hostlevelcapacityavailableforrunningvirtualmachinesandmakes recommendationsaboutpoweringhostsofforonbasedonwhethercapacityisfound tobeexcessiveorlacking. WhenyouaddahosttoaDRScluster,thathostsresourcesbecomeassociatedwiththe cluster.Thesystempromptsyouwhetheryouwanttoassociateanyexistingvirtual machinesandresourcepoolswiththeclustersrootresourcepoolorgrafttheresource poolhierarchy.SeeResourcePoolsandClustersonpage 56.VirtualCentercanthen performinitialplacementofvirtualmachines,virtualmachinemigrationforthesake ofloadbalancingorruleenforcement,anddistributedpowermanagement(ifenabled).
Initial Placement
Whenyouattempttopoweronasinglevirtualmachineoragroupofvirtualmachines inaDRSenabledcluster,VirtualCenterchecksthatthereareenoughresourcesinthe clustertosupportthevirtualmachine(s).Then,foreachvirtualmachine,VirtualCenter identifiesahostonwhichitcanrunanddoesoneofthefollowing: Automatically places it. Ifallplacementrelatedactions(virtualmachinesbeing poweredonormigrated,orhostsbeingpoweredon)areinautomaticmode,thesesteps aretakenautomaticallywithnorecommendationbeingshowntotheuser. Makes an initial placement recommendation. Iftheautomationlevelofanyofthe placementrelatedactionsareinmanualmode.Initialplacementrecommendations receivedbyusersinnonautomaticplacementscenariosdifferbasedonwhetheroneor morethanonevirtualmachineisbeingpoweredon. NOTENoinitialplacementrecommendationsaregivenforvirtualmachineson standalonehostsorinnonDRSclusters.Whenpoweredon,theyareplacedonthehost wheretheycurrentlyreside.
62
VMware, Inc.
Single Virtual Machine Power On

Whenasinglevirtualmachineisbeingpoweredon,youhavetwotypesofinitial placementrecommendations: Asinglevirtualmachineisbeingpoweredonandnoprerequisitestepsare needed. Theuserispresentedwithalistofmutuallyexclusiveinitialplacement recommendationsforthevirtualmachine(Figure 42).Youcanchooseonlyone. Figure 4-2. Single Virtual Machine, No Prerequisites
Asinglevirtualmachineisbeingpoweredon,butprerequisiteactionsare required. Theseactionsincludepoweringonahostinstandbymodeorthemigrationof othervirtualmachinesfromonehosttoanother.Inthiscase,onlyone recommendationisprovidedandithasmultiplelines,showingeachofthe prerequisiteactions.Theusercaneitheracceptthisentirerecommendationor cancelpoweringonthevirtualmachine(Figure 43). Figure 4-3. Single Virtual Machine, with Prerequisites
VMware, Inc.
63
Group Power On
Youcanattempttopoweronmultiplevirtualmachinesatthesametime(grouppower on).Itisnotrequiredthatthevirtualmachinesselectedforagrouppoweronattempt beinthesameDRScluster.Theycanbeselectedacrossclustersbutmustbewithinthe samedatacenter.ItisalsopossibletoincludevirtualmachineslocatedinnonDRS clustersoronstandalonehosts,butthesearepoweredonautomaticallyandnot includedinanyinitialplacementrecommendation. Theinitialplacementrecommendationsforgrouppoweronattemptsareprovidedon aperclusterbasis.Ifalloftheplacementrelatedactionsforagrouppoweronattempt areinautomaticmode,thevirtualmachinesarepoweredonwithnoinitialplacement recommendationgiven.Ifplacementrelatedactionsforanyofthevirtualmachinesare inmanualmode,thepoweringonofallofthevirtualmachines(includingthosethat areinautomaticmode)ismanualandisincludedinaninitialplacement recommendation. ForeachDRSclusterthatthevirtualmachinesbeingpoweredonbelongto,thereisa singlerecommendation,whichcontainsalloftheneededprerequisites(orno recommendation).Allsuchclusterspecificrecommendationsarepresentedtogether underthePowerOnRecommendationstab(Figure 44). Figure 4-4. Group Power On Recommendation
64
VMware, Inc.
Whenanonautomaticgrouppoweronattemptismade,andvirtualmachinesnot subjecttoaninitialplacementrecommendation(thatis,thoseonstandalonehostsorin nonDRSclusters)areincluded,VirtualCenterattemptstopowerthemon automatically.Ifthesepoweronsaresuccessful,theyarelistedundertheStarted PowerOnstab.Anyvirtualmachinesthatfailtopoweroninthiswayarelistedunder theFailedPowerOnstab.SeeFigure 45. Figure 4-5. Automatic Group Power On
GroupPowerOnExample:Theuserselectsthreevirtualmachinesinthesamedatacenter foragrouppoweronattempt.Thefirsttwovirtualmachines(VM1andVM2)areinthe sameDRScluster(Cluster1),whilethethirdvirtualmachine(VM3)isonastandalone host.VM1isinautomaticmodeandVM2isinmanualmode.Forthisscenario,theuser ispresentedwithaninitialplacementrecommendationforCluster1(underthePower OnRecommendationstab)whichconsistsofactionsforpoweringonVM1andVM2. AnattemptismadetopoweronVM3automaticallyand,ifsuccessful,itislistedunder theStartedPowerOnstab.Ifthisattemptfails,itislistedundertheFailedPowerOns tab.
VMware, Inc.
65
Load Balancing and Virtual Machine Migration

AclusterenabledforDRSmightbecomeunbalanced.Forexample,seeFigure 46.The threehostsontheleftsideofthisfigureareunbalanced.AssumethatHost1,Host 2, andHost3haveidenticalcapacity,andallvirtualmachineshavethesame configurationandload.However,becauseHost1hassixvirtualmachines,itsresources areoverusedwhileampleresourcesareavailableonHost2andHost3.DRSmigrates (orofferstomigrate)virtualmachinesfromHost1toHost2andHost3.Ontheright sideofthediagram,theproperlyloadbalancedconfigurationofthehoststhatresults isdisplayed. Figure 4-6. VMware DRS
VM1 VM4
VM2 VM5 Host 1
VM3 VM6
VM1
VM2
VM3
Host 1
VM7
VM7
VM4
VM5
Host 2
Host 2
VM8
VM9
VM8
VM9
VM6
Host 3
Host 3
66
VMware, Inc.
Whenaclusterbecomesunbalanced,DRSmakesrecommendationsormigratesvirtual machines,dependingonthedefaultautomationlevel: Iftheclusteroranyofthevirtualmachinesinvolvedaremanualorpartially automated,VirtualCenterdoesnottakeautomaticactionstobalanceresources. Instead,theSummarypageindicatesthatmigrationrecommendationsare availableandtheDRSRecommendationspagedisplaysrecommendationsfor changesthatmakethemostefficientuseofresourcesacrossthecluster. Iftheclusterandvirtualmachinesinvolvedareallfullyautomated,VirtualCenter migratesrunningvirtualmachinesbetweenhostsasneededtoensureefficientuse ofclusterresources. NOTEEveninanautomaticmigrationsetup,userscanexplicitlymigrate individualvirtualmachines,butVirtualCentermightmovethosevirtualmachines tootherhoststooptimizeclusterresources. Bydefault,automationlevelisspecifiedforthewholecluster.Youcanalsospecifya customautomationlevelforindividualvirtualmachines.
Migration Threshold
Themigrationthresholdallowsyoutospecifywhichrecommendationsareapplied whentheclusterisinfullyautomatedmode.SeeFigure 47.Youcanmovethesliderto useoneoffivelevels,rangingfromConservative,whichmakesthesmallestnumber ofmigrations,toAggressive,whichmakesthelargestnumberofmigrations.Thefive migrationlevelsapplyrecommendationsbasedontheirassignedstarratings. Figure 4-7. Migration Threshold Choices
VMware, Inc.
67
Starratingsareassignedtomigrationrecommendationsbasedontheamountof improvementintheclustersloadbalancethattheybringrangingfromfivestar recommendations,whicharemandatory,toonestarrecommendations,whichbring onlyaslightimprovement.Eachlevelyoumovetheslidertotherightallowsthe inclusiveapplicationofonemorelowerlevelofstarratings.TheConservativesetting appliesonlyfivestarrecommendations,thenextleveltotherightappliesfourstar recommendationsandhigher,andsoon,downtotheAggressivelevelwhichapplies onestarrecommendationsandhigher(thatis,itappliesallrecommendations.)
Migration Recommendations
Ifyoucreateaclusterwithadefaultmanualorpartiallyautomatedmode, VirtualCenterdisplaysmigrationrecommendationsontheDRSRecommendations page.Thesystemsuppliesasmanyrecommendationsasnecessarytoenforcerulesand balancetheresourcesofthecluster.Eachrecommendationincludesthevirtualmachine tobemoved,current(source)hostanddestinationhost,andareasonforthe recommendation.Thereasoncanbeoneofthefollowing: BalanceaverageCPUloads. Balanceaveragememoryloads. Satisfyresourcepoolreservations. Satisfyaffinity(orantiaffinity)rule.SeeUsingDRSAffinityRulesonpage 108. Hostisenteringmaintenancemode.SeeHostMaintenanceandStandbyModes onpage 71.
Distributed Power Management

Whenthisexperimentalfeatureisenabled,aDRSclustercanreduceitspower consumptionbymakingrecommendationsbasedonacomparisonofclusterlevel capacityversusdemand.Ifcapacityisdeemedtobeinadequate,DRSrecommends poweringonhostsandmigratingvirtualmachines(usingVMwareVMotion)to them.Conversely,whenexcesscapacityisfound,DRSrecommendsthatsomehostsbe placedinstandbymodeandanyvirtualmachinesrunningonthemareevacuatedto otherhosts.SeeStandbyModeonpage 72.Whetherthesehostpowerstateand migrationrecommendationsareexecutedautomaticallydependsupontheautomation levelselectedfortheDistributedPowerManagementfeature.
68
VMware, Inc.
BeforeDistributedPowerManagementcanbeenabledforaDRScluster,youmust ensurethattheESXServerhostshavetheappropriatehardwaresupportand configuration.Specifically,theNICsusedbytheVMkernelnetworkmusthave WakeonLAN(WOL)functionality,whichisusedtobringanESXServerhostupfrom apoweredoffstate.YoushouldtestwakecapabilityoperationoneachESXServer3.5 (orESXServer3iversion3.5)hostonwhichDistributedPowerManagementistobe deployed.Todothis,ensurethatthereisatleastoneotherESXServerhostpoweredon inthecluster(tosendthewakepackets),explicitlyputthehosttobetestedintostandby stateandonceithasentered,ensurethatexplicitlyrequestingthatthestandbyhost powerbackonissuccessful.Ifnot,VMwarerecommendsthatyoudonotconfigureit tobepowermanagedbyDistributedPowerManagement. CAUTIONBeforeimplementingDistributedPowerManagement,testthe WakeonLANcapabilityofyourhosts.IfWOLfunctionalityfails,thepower managementfeaturemaypoweroffhostsandnotbeabletopowerthembackonlater. ThedefaultpowermanagementautomationlevelforaDRSclusterisselectedfromthe PowerManagementtaboftheclustersSettingsdialogbox.SeeFigure 48.Thefeature isenabledwheneverthissettingisnotoff.Theoptionsavailableare: OffThefeatureisdisabledandnorecommendationswillbemade. ManualHostpoweroperationandrelatedvirtualmachinemigration recommendationsaremade,butnotautomaticallyexecuted. AutomaticHostpoweroperationsareautomaticallyexecutedifrelatedvirtual machinemigrationscanallbeexecutedautomatically. Inadditiontotheseclusterlevelsettings,youcanalsosetoverridesforindividualhosts sothattheirautomationleveldiffersfromthatofthecluster.Suchoverridesonlyapply ifthefeatureisenabled(notsettooff)forthecluster.
VMware, Inc.
69
Figure 4-8. Power Management Automation Level and Host Overrides
NOTEThepowermanagementautomationlevelisnotthesameastheDRSautomation level(forloadbalancing)describedearlier.Also,therecommendationsgeneratedby DistributedPowerManagementareassignedstarratingstoshowtheirrelative importance,buttheyarenotcontrolledbytheDRSmigrationthreshold.
DRS Clusters, Resource Pools, and ESX Server

ForclustersenabledforDRS,theresourcesofallhostsareassignedtothecluster. DRSinternallyusestheperhostresourcepoolhierarchiestoimplementthe clusterwideresourcepoolhierarchy.WhenyouviewtheclusterusingaVIClient connectedtoaVirtualCenterServer,youseetheresourcepoolhierarchyimplemented byDRS. WhenyouviewindividualhostsusingaVIClientconnectedtoanESXServerhost,the underlyinghierarchyofresourcepoolsispresented.BecauseDRSimplementsthemost balancedresourcepoolhierarchyitcan,donotmodifythehierarchyvisibleonthe individualESXServerhost.Ifyoudo,DRSwillundoyourchangesimmediately.
70
VMware, Inc.
Host Maintenance and Standby Modes

Maintenancemodeandstandbymodeforhostshavesimilarities.Inparticular,they prohibittherunningofvirtualmachines.However,thetwomodeshavedistinct purposes.Youmoveahostintomaintenancemode,whenyouneedtoserviceitfor example,byinstallingmorememoryorupgradingtheversionofESXServerrunning onitandthehostremainsinmaintenancemodeuntilyoumoveitout.Incontrast, DRSmoveshostsintoandoutofstandbymodeautomatically,tooptimizepower usage.
Maintenance Mode
Standalonehostsandhostswithinaclustersupportamaintenancemode,which restrictsthevirtualmachineoperationsonthehosttoallowyoutoshutdownrunning virtualmachinesinpreparationforhostshutdown.OnlyESXServer3.0andlater supportsmaintenancemodeforstandalonehosts. Ahostentersorleavesmaintenancemodeonlyastheresultofauserrequest.Ifthehost isinacluster,whenitentersmaintenancemodetheuserisgiventheoptiontoevacuate poweredoffvirtualmachines.Ifthisoptionisselected,eachpoweredoffvirtual machineismigratedtoanotherhost,unlessthereisnocompatiblehostavailableforthe virtualmachineinthecluster.Whileinmaintenancemode,thehostdoesnotallowyou todeployorpoweronavirtualmachine.Virtualmachinesthatarerunningonahost enteringmaintenancemodeneedtobeeithermigratedtoanotherhostorshutdown (eithermanuallyorautomaticallybyDRS). NOTEIfDRScanmakenomigrationrecommendationsforavirtualmachine,anevent isgenerated(checkthevirtualmachinesTasks&Eventstab).Thevirtualmachinemust bemigratedmanuallyorpoweredoffbeforethehostcanentermaintenancemode. Whennomorerunningvirtualmachinesareonthehost,thehostsiconchangesto includeundermaintenanceandthehostsSummarypanelindicatesthenewstate. Thedefaultautomationmodeofavirtualmachinedeterminesitsbehaviorwhenthe host(inaDRScluster)itisrunningonentersmaintenancemode: Anyfullyautomatedvirtualmachineismigratedautomatically. NOTEIfnoappropriatehostisavailable,DRSdisplaysinformationontheTasks &Eventstab. Forapartiallyautomatedormanualvirtualmachine,arecommendationfor furtheruseractionisgeneratedanddisplayed.
VMware, Inc.
71
Standby Mode
Whenahostmachineisplacedinstandbymode,itispoweredoff.Normally,hostsare placedinstandbymodebythedistributedpowermanagementfeature.Youcanalso placeahostinstandbymodemanually;however,DRSmightundo(orrecommend undoing)yourchangethenexttimeitruns.Toforceahosttoremainoff,placeitin maintenancemodeandpoweritoff.Youcanalsodisabledistributedpower management(orsetittomanualmode)onthehosttopreventthehostfrombeing automaticallypoweredon. Ifthedistributedpowermanagementfeaturedeterminesthatthehostneedstobe broughtoutofstandbymode(thatis,tobepoweredbackon),thisisdoneusing WakeonLANtechnology. Toaccommodatethis,youmustensurethatthefollowingstepsaretaken: TheNICthattheVMkernelnetworkingstackisattachedto(selectedasthe VMotionNIC)mustbeWOLcompatible.TodisplaytheWOLcompatibilitystatus foreachNIConahost,selectthehostintheinventorypaneloftheVIClient, choosetheConfigurationtab,andclickNetworkAdapters. TheVMotionnetworkmustbeonasingleIPsubnetpercluster. NOTEHoststhatdonothaveWOLcompatibleNICsareneverselectedforstandby mode.
Understanding VMware HA
TheHAclusterfeatureallowsthevirtualmachinesrunningonESXServersystemsto automaticallyrecoverfromhostfailures.Whenahostbecomesunavailable,all associatedvirtualmachinesareimmediatelyrestartedonotherhostsinthesystem. ThissectionfirstconsidersdifferencesbetweenVMwareHAclustersandtraditional clusteringsolutions,andpresentsHAclusterconcepts.
Traditional and HA Failover Solutions

VMwareHAandtraditionalclusteringsolutionssupportautomaticrecoveryfromhost failures.Theyarecomplementarybecausetheydifferintheseareas: Hardwareandsoftwarerequirements Timetorecovery Degreeofapplicationdependency.
72
VMware, Inc.
Traditional Clustering Solutions

AtraditionalclusteringsolutionsuchasMicrosoftClusterService(MSCS)orVeritas ClusteringServiceprovidesimmediaterecoverywithminimaldowntimefor applicationsincaseofhostorvirtualmachinefailure.Toachievethis,theIT infrastructuremustbesetupasfollows: Eachmachine(orvirtualmachine)musthaveamirrorvirtualmachine(potentially onadifferenthost). Themachine(orthevirtualmachineanditshost)aresetuptomirroreachother usingtheclusteringsoftware.Generally,theprimaryvirtualmachinesends heartbeatstothemirror.Incaseoffailure,themirrortakesoverseamlessly. Figure 49showsdifferentoptionsforthesetupofatraditionalclusterforvirtual machines. Figure 4-9. VMware Clustering Setup
virtual machine virtual machine virtual machine virtual machine
cluster in a box
cluster across boxes
Settingupandmaintainingsuchaclusteringsolutionisresourceintensive.Eachtime youaddanewvirtualmachine,youneedcorrespondingfailovervirtualmachinesand possiblyadditionalhosts.Youhavetosetup,connect,andconfigureallnewmachines andupdatetheclusteringapplicationsconfiguration. Thetraditionalsolutionguaranteesfastrecovery,butisresourceandlaborintensive. SeetheVMwaredocumentSetupforMicrosoftClusterServiceforadditionalinformation ondifferentclustertypesandhowtoconfigurethem.
VMware, Inc.
73
VMware HA Solution
InaVMwareHAsolution,asetofESXServerhostsiscombinedintoaclusterwitha sharedpoolofresources.VirtualCentermonitorsallhostsinthecluster.Ifoneofthe hostsfails,VirtualCenterimmediatelyrespondsbyrestartingeachassociatedvirtual machineonadifferenthost. UsingVMwareHAhasanumberofadvantages: MinimalsetupTheNewClusterWizardisusedforinitialsetup.Youcanadd hostsandnewvirtualmachinesusingtheVIClient.Allvirtualmachinesinthe clustergetfailoversupportwithoutadditionalconfiguration. ReducedhardwarecostandsetupInatraditionalclusteringsolution,duplicate hardwareandsoftwaremustbeconnectedandconfiguredproperly.Thevirtual machineactsasaportablecontainerfortheapplications,andcanbemoved around.Duplicateconfigurationsonmultiplemachinescanbeavoided.Whenyou useVMwareHA,youmusthavesufficientresourcestofailoverthenumberof hostsyouwanttoguarantee.However,theVirtualCenterServertakescareofthe resourcemanagementandclusterconfiguration. IncreasedapplicationavailabilityAnyapplicationrunninginsideavirtual machinehasaccesstoincreasedavailability.Becausethevirtualmachinecan recoverfromhardwarefailure,allapplicationsthataresetuptostartduringthe bootcyclehaveincreasedavailabilityatnoextracosteveniftheapplicationisnot itselfaclusteredapplication. PotentiallimitationsofusingHAclustersincludelossofruntimestateandalonger applicationdowntimethaninatraditionalclusteringenvironmentwithhotstandby.If thoselimitationsbecomeissues,considerusingthetwoapproachestogether.
VMware HA Features
AclusterenabledforHA: SupportseasytouseconfigurationusingtheVIClient. Providesfailoveronhardwarefailureforallrunningvirtualmachineswithinthe boundsoffailovercapacity(seeFailoverCapacityonpage 75). Workswithtraditionalapplicationlevelfailoverandenhancesit. IsfullyintegratedwithDRS.Ifahosthasfailedandvirtualmachineshavebeen restartedonotherhosts,DRScanprovidemigrationrecommendationsormigrate virtualmachinesforbalancedresourceallocation.Ifoneorbothofthesourceand destinationhostsofamigrationfail,HAcanhelprecoverfromthatfailure.
74
VMware, Inc.
Failover Capacity
WhenyouenableaclusterforHA,theNewClusterwizardpromptsyouforthe numberofhostsforwhichyouwantfailovercapacity.Thisnumberisshownasthe ConfiguredFailoverCapacityintheVIClient.HAusesthisnumbertodetermineif thereareenoughresourcestopoweronvirtualmachinesinthecluster. Youneedtospecifyonlythenumberofhostsforwhichyouwantfailovercapacity.HA computestheresourcesitrequirestofailovervirtualmachinesforthatmanyhosts, usingaconservativeestimate,anddisallowspoweringonvirtualmachinesiffailover capacitycannolongerbeguaranteed. NOTE(SEEUPDATE)Youcanallowtheclustertopoweronvirtualmachinesevenwhen theyviolateavailabilityconstraints.Ifyoudothat,theresultisaredcluster,which meansthatfailoverguaranteesmightnolongerbevalid.SeeValid,Yellow,andRed Clustersonpage 81. Afteryouhavecreatedacluster,youcanaddhoststoit.WhenyouaddahosttoanHA clusterthatisnotenabledforDRS,allresourcepoolsareimmediatelyremovedfrom thehost,andallvirtualmachinesbecomedirectlyassociatedwiththecluster. NOTEIftheclusterisalsoenabledforDRS,youcanchoosetokeeptheresourcepool hierarchy.SeeResourcePoolsandClustersonpage 56.
Planning for HA Clusters

WhenplanninganHAcluster,considerthefollowing: EachhosthassomememoryandCPUtopoweronvirtualmachines. EachvirtualmachinemustbeguaranteeditsCPUandmemoryreservation requirements. Ingeneral,usingauniformsetupisrecommended.HAplansforaworstcasefailure scenario.Whencomputingrequiredfailovercapacity,HAcalculatesthemaximum memoryandCPUreservationsneededforanycurrentlypoweredonvirtualmachine andcallsthisaslot.AslotistheamountofCPUandmemoryresourcesthatwillbe sufficientforanycurrentlypoweredonvirtualmachine(poweredofforsuspended virtualmachinesarenotconsideredwhencalculatingthecurrentfailoverlevel).
VMware, Inc.
75
Forexample,ifyouhaveavirtualmachinewith1GHzCPUreservationand1GB memoryreservationandanothervirtualmachinewith2GHzCPUreservationand 512MBmemoryreservation,theslotisdefinedas2GHzCPUreservationand1GB memoryreservation.HAdetermineshowmanyslotscanfitintoeachhostbasedon thehostsCPUandmemorycapacity.HAthendetermineshowmanyhostscouldfail withtheclusterstillhavingatleastasmanyslotsaspoweredonvirtualmachines.This numberisthecurrentfailoverlevel. Duringplanning,decideonthenumberofhostsforwhichyouwanttoguarantee failover.HAtriestoreserveresourcesforatleastasmanyhostfailuresbylimitingthe numberofvirtualmachinesthatarepoweredon,whichwillconsumetheseresources. IfyoulefttheAllowvirtualmachinetobestartedeveniftheyviolateavailability constraintsoptionunselected(strictadmissioncontrol),VMwareHAdoesnotallow youtopoweronvirtualmachinesiftheywouldcausethecurrentfailoverlevelto becomelessthantheconfiguredfailoverlevel.VMwareHAalsodoesnotallowthe followingoperationsiftheywouldcausethecurrentfailoverleveltoexceedthe configuredfailoverlevel: Revertingapoweredoffvirtualmachinetoapoweredonsnapshot. Migratingarunningvirtualmachineintothecluster. ReconfiguringarunningvirtualmachinetoincreaseitsCPUormemory reservation. IfyouselectedtheAllowvirtualmachinetobestartedeveniftheyviolateavailability constraintsoptionwhenyouenableHA,youcanpoweronmorevirtualmachinesthan HAwouldadvise.Becauseyouconfiguredthesystemtopermitthis,theclusterdoes notturnred.Thecurrent(available)failoverlevelcanalsofallbelowtheconfigured failoverlevelifthenumberofhoststhathavefailedexceedstheconfigurednumber.For example,ifyouhaveconfiguredtheclusterforaonehostfailure,youareatcapacity (thecurrentfailoverlevelisequaltotheconfiguredfailoverlevel),andtwohostsfail, theclusterturnsred. Aclusterthatisbeneaththeconfiguredfailoverlevelcanstillperformvirtualmachine failoverincaseofhostfailure,usingvirtualmachineprioritytodeterminewhich virtualmachinestopoweronfirst.SeeCustomizingHAforVirtualMachineson page 115. CAUTIONItisnotrecommendedthatyouworkwithredclusters.Ifyoudo,failoveris notguaranteedatthespecifiedlevel.
76
VMware, Inc.
VMware HA and Special Situations

VMwareHAunderstandshowtoworkwithspecialsituationstopreserveyourdata: Power off host. Ifyoupoweroffahost,HArestartsanyvirtualmachinesrunningon thathostonadifferenthost. Migrate virtual machine with VMotion. Ifyouaremigratingavirtualmachineto anotherhostusingVMotion,andthesourceordestinationhostbecomeunavailable, thevirtualmachinecouldbeleftinafailed(poweredoff)statedependingonthestage inthemigration.HAhandlesthisfailureandpowersonthevirtualmachineonan appropriatehost: Ifthesourcehostbecomesunavailable,HApowersonthevirtualmachineon thedestinationhost. Ifthedestinationhostbecomesunavailable,HApowersonthevirtual machineonthesourcehost. Ifbothhostsbecomeunavailable,HApowersonthevirtualmachineona thirdhostintheclusterifitexists. Current failover capacity does not match configured failover capacity. Acluster turnsredifcurrentfailovercapacityissmallerthanconfiguredfailovercapacity.This canhappenbecausemorehostsfailedthanyouconfiguredtheclustertotolerate.Ifyou turnedoffstrictadmissioncontrol,theclusterwillnotturnred,evenifyoupoweron morevirtualmachinesthancanbeaccommodated. Whenthereisinsufficientcapacity,HAfailsovervirtualmachineswithhigher prioritiesfirst,thenattemptstofailoverothervirtualmachines.Inthiscase,givehigh prioritiestovirtualmachinesthataremostcriticaltoyourenvironmentforrecovery. SeeCustomizingHAforVirtualMachinesonpage 115. Host network isolation. AhostinanHAclustermightloseitsconsolenetwork(or VMkernelnetwork,inESXServer3i)connectivity.Suchahostisisolatedfromother hostsinthecluster.Otherhostsintheclusterconsiderthathostfailedandattemptto failoveritsrunningvirtualmachines.Ifavirtualmachinecontinuestorunonthe isolatedhost,VMFSdisklockingpreventsitfrombeingpoweredonelsewhere.If virtualmachinessharethesamenetworkadapter,theywillnothaveaccesstothe network.Youmightwanttostartthevirtualmachineonanotherhost. Bydefault,virtualmachinesareshutdownontheisolatedhostincaseofahost networkisolationincidenttoallowthemtoberestartedonothernonisolatedhostsin thecluster.Youcanchangetheclustersdefaultbehaviortoleavethevirtualmachines poweredontheisolatedhostortopoweritoff.Youcanalsochangethatbehaviorfor eachvirtualmachine.SeeCustomizingHAforVirtualMachinesonpage 115.
VMware, Inc.
77
Primary and Secondary Hosts

SomeofthehostsinanHAclusteraredesignatedasprimaryhostsandtheymaintain themetadataandfailoverintelligence.Thefirstfivehostsintheclusterbecomeprimary hosts,allothersaresecondaryhosts.WhenyouaddahosttoanHAcluster,thathosthas tocommunicatewithanexistingprimaryhostinthesameclustertocompleteits configuration(unlessitsthefirsthostinthecluster,whichmakesittheprimaryhost). Whenaprimaryhostbecomesunavailableorisremoved,HApromotesanotherhost toprimarystatus.Primaryhostshelptoprovideredundancyandareusedtoinitiate failoveractions. Ifallthehostsintheclusterarenotresponding,andyouaddanewhosttothecluster, HAconfigurationfailsbecausethenewhostcannotcommunicatewithanyofthe primaryhosts.Inthissituation,youmustdisconnectallthehoststhatarenot respondingbeforeyoucanaddthenewhost.Thenewhostbecomesthefirstprimary host.Whentheotherhostsbecomeavailableagain,theirHAserviceisreconfigured andtheythenbecomeprimaryorsecondaryhostsdependingontheexistingnumber ofprimaryhosts.
HA Clusters and Maintenance Mode

Whenyouputahostinmaintenancemode,youarepreparingtoshutitdownordo maintenanceonit.Youcannotpoweronavirtualmachineonahostthatisin maintenancemode.Intheeventofahostfailure,HAdoesnotfailoveranyvirtual machinestoahostthatisinmaintenancemode.SuchahostisnotconsideredwhenHA computesthecurrentfailoverlevel. Whenahostexitsmaintenancemode,theHAserviceisreenabledonthathost,soit becomesavailableforfailoveragain. Whenahostisenteringmaintenancemode,ifitdoesnothaveanypoweredonvirtual machines,thistransitiontomaintenancemodecannotbecancelled.
HA Clusters and Disconnected Hosts

Whenahostbecomesdisconnected,itexistsintheVirtualCenterinventory,but VirtualCenterdoesnotgetanyupdatesfromthathost,doesnotmonitorit,andhasno knowledgeofthehealthofthathost.Becausethestatusofthehostisnotknown,and becauseVirtualCenterisnotcommunicatingwiththathost,HAcannotuseitasa guaranteedfailovertarget.HAdoesnotconsiderdisconnectedhostswhencomputing thecurrentfailoverlevel. Whenthehostbecomesreconnected,thehostbecomesavailableforfailoveragain.
78
VMware, Inc.
Thedifferencebetweenadisconnectedhostandahostthatisnotrespondingis describedinthelist. Adisconnectedhosthasbeenexplicitlydisconnectedbytheuser.Aspartof disconnectingahost,VirtualCenterdisablesHAonthathost.Thevirtualmachines onthathostarenotfailedoverandnotconsideredwhenVirtualCentercomputes thecurrentfailoverlevel. Ifahostisnotresponding,theVirtualCenterServernolongerreceivesheartbeats fromit.Thismighthappen,forexample,becauseofanetworkproblem,because thehostfailed,orbecausetheVirtualCenteragentfailed. VirtualCenterdoesnotincludesuchhostswhencomputingthecurrentfailover level,butassumesthatanyvirtualmachinesrunningonanonrespondinghostwill befailedoverifthehostfails.Thevirtualmachinesonahostthatisnotresponding affecttheadmissioncontrolcheck.
HA Clusters and Host Network Isolation

Hostfailuredetectionoccurs15secondsaftertheHAserviceonahosthasstopped sendingheartbeatstotheotherhostsinthecluster.(Thedefaultvalueofthisfailure detectionintervalcanbechanged.SeeSettingAdvancedHAOptionsonpage 121.) Ahoststopssendingheartbeatsifitfailsorifitisisolatedfromthenetwork.Atthat time,otherhostsintheclustertreatthishostasfailed,whilethishostdeclaresitselfas isolatedfromthenetworkafterithaslostnetworkconnectivityformorethan12 seconds. IftheisolatedhosthasSANaccess,itretainsthedisklockonthevirtualmachinefiles, andanyattemptstofailoverthevirtualmachinetoanotherhostfails.Thevirtual machinecontinuestorunontheisolatedhost.VMFSdisklockingprevents simultaneouswriteoperationstothevirtualmachinediskfilesandpotential corruption.Bydefault,theisolatedhostpowersoffitsvirtualmachines.Thesevirtual machinescanthensuccessfullyfailovertootherhostsinthecluster. Ifthenetworkconnectionisrestoredbefore12secondshaveelapsed,otherhostsinthe clusterdonottreatthisasahostfailure.Itistreatedastransient.Inaddition,thehost withthetransientnetworkconnectionproblemdoesnotdeclareitselfisolatedfromthe networkandcontinuesrunning. Ifthenetworkconnectionisnotrestoredfor15secondsorlonger,theotherhostsinthe clustertreatthehostasfailedandattempttofailoverthevirtualmachinesonthathost. Theisolatedhostpowersdownitsvirtualmachinessotheycanberestartedonother hoststhathavefunctioningnetworkconnectivity.
VMware, Inc.
79
Inthewindowbetween12and14seconds,theclusteringserviceontheisolatedhost declaresitselfasisolatedandstartspoweringoffvirtualmachineswithdefault isolationresponsesettings.Ifthenetworkconnectionisrestoredduringthattime,the virtualmachinethathadbeenpoweredoffisnotrestartedonotherhostsbecausethe HAservicesontheotherhostsdonotconsiderthishostasfailedyet. Asaresult,ifthenetworkconnectionisrestoredinthiswindowbetween12and14 secondsafterthehosthaslostconnectivity,thevirtualmachinesarepoweredoffbut notfailedover.
Using HA and DRS Together

WhenHAperformsfailoverandrestartsvirtualmachinesondifferenthosts,itsfirst priorityistheimmediateavailabilityofallvirtualmachines.Afterthevirtualmachines havebeenrestarted,thosehostsonwhichtheywerepoweredonmightbeheavily loaded,whileotherhostsarecomparativelylightlyloaded.HAusestheCPUand memoryreservationtodeterminefailover,whiletheactualusagemightbehigher. UsingHAandDRStogethercombinesautomaticfailoverwithloadbalancing.This combinationcanresultinafastrebalancingofvirtualmachinesafterHAhasmoved virtualmachinestodifferenthosts.Youcansetupaffinityandantiaffinityrulestostart twoormorevirtualmachinespreferentiallyonthesamehost(affinity)orondifferent hosts(antiaffinity).Forexample,youcanuseanantiaffinityruletoensurethattwo virtualmachinesrunningacriticalapplicationneverrunonthesamehost.SeeUsing DRSAffinityRulesonpage 108. NOTEInaclusterusingDRSandHAwithHAadmissioncontrolturnedon,virtual machinesmightnotbeevacuatedfromhostsenteringmaintenancemode.Thisis becauseoftheresourcesreservedtomaintainthefailoverlevel.Inthiscase,youmust manuallymigratethevirtualmachinesoffofthehostsusingVMotion.
80
VMware, Inc.
Valid, Yellow, and Red Clusters

TheVIClientindicateswhetheraclusterisvalid,overcommitted(yellow),orinvalid (red).ClusterscanbecomeovercommittedbecauseofaDRSviolation.Clusterscan becomeinvalidbecauseofaDRSviolationoranHAviolation.Amessagedisplayedin theSummarypageindicatestheissue.
Valid Cluster
Avalidclusterhasenoughresourcestomeetallreservationsandtosupportallrunning virtualmachines.Aclusterisvalidunlesssomethinghappensthatmakesit overcommittedorinvalid. ADRSclustercanbecomeovercommittedifahostfails. ADRSclustercanbecomeinvalidifVirtualCenterisunavailableandyoupower onvirtualmachinesusingaVIClientconnecteddirectlytoanESXServerhost. AnHAclusterbecomesinvalidifthecurrentfailovercapacityislowerthanthe configuredfailovercapacityorifalltheprimaryhostsintheclusterarenot responding.SeePrimaryandSecondaryHostsonpage 78. ADRSorHAclustercanbecomeinvalidiftheuserreducesthereservationona parentresourcepoolwhileavirtualmachineisintheprocessoffailingover. Beforeconsideringthefollowingexamples,notethedefinitionoftheseterms: Reservation(foraresourcepool)Afixed,guaranteedallocationfortheresource poolinputbytheuser. ReservationUsed(foraclusterorresourcepool)Thesumofthereservationor reservationused(whicheverislarger)foreachchild,addedrecursively. Unreserved(foraclusterorresourcepool)Anonnegativenumberthatalso differsaccordingtoresourcepooltype: Foraclusteritequalstotalcapacityreservationused. Fornonexpandableresourcepoolsitequalsreservationreservationused. Forexpandableresourcepoolsitisequalto(reservationreservationused)+ anyunreservedresourcesthatcanbeborrowedfromitsancestorresource pools.
VMware, Inc.
81
Example 1: Valid Cluster, All Resource Pools of Type Fixed

Figure 410showsavalidclusterandhowitsCPUresourcesarecomputed.Thecluster hasthefollowingcharacteristics: Aclusterwithtotalresourcesof12GHz. Threeresourcepools,eachoftypeFixed(ExpandableReservationisnotselected). Thetotalreservationofthethreeresourcepoolscombinedis11GHz(4+4+3GHz). ThetotalisshownintheReservationUsedfieldforthecluster. RP1wascreatedwithareservationof4GHz.Twovirtualmachines.(VM1and VM7)of2GHzeacharepoweredon(ReservationUsed:4GHz).Noresourcesare leftforpoweringonadditionalvirtualmachines.VM6isshownasnotpoweredon. Itconsumesnoneofthereservation. RP2wascreatedwithareservationof4GHz.Twovirtualmachinesof1GHzand 2GHzarepoweredon(ReservationUsed:3GHz).1GHzremainsunreserved. RP3wascreatedwithareservationof3GHz(Reservation).Onevirtualmachine with3GHzispoweredon.Noresourcesforpoweringonadditionalvirtual machinesareavailable. Figure 4-10. Valid Cluster (Fixed Resource Pools)
cluster Total Resources: 12G Reservation Used: 11G Unreserved: 1G
RP1 Reservation: 4G Reservation Used: 4G Unreserved: 0G
VM1, 2G VM7, 2G
VM6, 2G
VM2, 2G VM4, 1G
VM3, 3G VM8, 2G
VM5, 2G
82
VMware, Inc.
Example 2: Valid Cluster, Some Resource Pools of Type Expandable

Example2(Figure 411)usessimilarsettingstoExample1.However,RP1andRP3use reservationtypeExpandable.Avalidclustercanbeconfiguredasfollows: Aclusterwithtotalresourcesof16GHz. RP1andRP3areoftypeExpandable,RP2isoftypeFixed. Thetotalreservationusedofthethreeresourcepoolscombinedis16GHz(6GHz forRP1,5GHzforRP2,and5GHzforRP3).16GHzshowsupastheReservation Usedfortheclusterattoplevel. RP1wascreatedwithareservationof4GHz.Threevirtualmachinesof2GHzeach arepoweredon.Twoofthosevirtualmachines(forexample,VM1andVM7)can useRP1sreservations,thethirdvirtualmachine(VM6)canusereservationsfrom theclustersresourcepool.(IfthetypeofthisresourcepoolwereFixed,youcould notpowerontheadditionalvirtualmachine.) RP2wascreatedwithareservationof5GHz.Twovirtualmachinesof1GHzand 2GHzarepoweredon(ReservationUsed:3GHz).2GHzremainsunreserved. RP3wascreatedwithareservationof5GHz.Twovirtualmachinesof3GHzand 2GHzarepoweredon.EventhoughthisresourcepoolisoftypeExpandable,no additional2GHzvirtualmachinecanbepoweredonbecausetheparentsextra resourcesarealreadyusedbyRP1. Figure 4-11. Valid Cluster (Expandable Resource Pools)
cluster Total Resources: 16G Reservation Used: 16G Unreserved: 0G
RP1 (expandable) Reservation: 4G Reservation Used: 6G Unreserved: 0G
VM1, 2G VM7, 2G
VM6, 2G
VM2, 2G VM4, 1G
VM3, 3G VM8, 2G
VM5, 2G
VMware, Inc.
83
Yellow Cluster
Aclusterbecomesyellowwhenthetreeofresourcepoolsandvirtualmachinesis internallyconsistentbutthereisnotenoughcapacityintheclustertosupportall resourcesreservedbythechildresourcepools.Therewillalwaysbeenoughresources tosupportallrunningvirtualmachinesbecause,whenahostbecomesunavailable,all itsvirtualmachinesbecomeunavailable. Aclustertypicallyturnsyellowwhenclustercapacityissuddenlyreduced,for example,whenahostintheclusterbecomesunavailable.VMwarerecommendsthat youleaveadequateadditionalclusterresourcestoavoidyourclusterturningyellow. Considerthefollowingexample,asshowninFigure 412: Aclusterwithtotalresourcesof12GHzcomingfromthreehostsof4GHzeach. Threeresourcepoolsreservingatotalof12GHz. Thetotalreservationusedbythethreeresourcepoolscombinedis12GHz(4+5+3 GHz).ThatshowsupastheReservationUsedinthecluster. Oneofthe4GHzhostsbecomesunavailable,sototalresourcesreduceto8GHz. Atthesametime,VM4(1GHz)andVM3(3GHz),whichwererunningonthehost thatfailed,arenolongerrunning. Theclusterisnowrunningvirtualmachinesthatrequireatotalof6GHz.The clusterstillhas8GHzavailable,whichissufficienttomeetvirtualmachine requirements. Theresourcepoolreservationsof12GHzcannolongerbemet,sotheclusteris markedasyellow.
84
VMware, Inc.
Figure 4-12. Yellow Cluster
cluster Total Resources: 12G 8G Reservation Used: 12G Unreserved: 0G
VM1, 2G
VM6, 2G
VM2, 2G VM4, 1G
VM3, 3G VM7, 0G
VM5, 5G
Red Cluster
AclustercanbecomeredbecauseofaDRSviolationoranHAviolation.Thebehavior oftheclusterdependsonthetypeofviolation,asdiscussedinthissection.
Red DRS Cluster

AclusterenabledforDRSbecomesredwhenthetreeisnolongerinternallyconsistent, thatis,resourceconstraintsarenotobserved.Thetotalamountofresourcesinthe clusterdoesnotaffectwhethertheclusterisred.ItispossiblefortheclustertobeDRS redevenifenoughresourcesareattherootlevel,ifthereisaninconsistencyatachild level. YoucanresolvearedDRSclusterproblemeitherbypoweringoffoneormorevirtual machines,movingvirtualmachinestopartsofthetreethathavesufficientresources, oreditingtheresourcepoolsettingsintheredpart.Addingresourcestypicallyhelps onlywhenyouareintheyellowstate.
VMware, Inc.
85
Aclustercanalsoturnredifyoureconfigurearesourcepoolwhileavirtualmachineis intheprocessoffailingover.Avirtualmachinethatisintheprocessoffailingoveris disconnectedanddoesnotcounttowardthereservationusedbytheparentresource pool.Itispossiblethatyoureducethereservationoftheparentresourcepoolbeforethe failovercompletes.Afterthefailoveriscomplete,thevirtualmachineresourcesare againchargedtotheparentresourcepool.Ifthepoolsusagebecomeslargerthanthe newreservation,theclusterturnsred. ConsidertheexampleinFigure 413. Figure 4-13. Red Cluster
cluster Total Resources: 12G Reservation Used: 12G 15G Unreserved: 0G
RP2 Reservation: 2G Reservation Used: 2G 5G Unreserved: 0G
RP3 (expandable) Reservation: 6G Reservation Used: 2G Unreserved: 4G 0G
VM1, 1G
VM2, 3G
VM3, 1G
VM4, 1G VM7, 3G
VM5, 1G
VM6, 1G
AsisshownintheexampleinFigure 413,ifauserisabletostartavirtualmachine(in anunsupportedway)withareservationof3GHzunderresourcepool2,thecluster wouldbecomered.
Red HA Cluster
AclusterenabledforHAbecomesredwhenthenumberofvirtualmachinespowered onexceedsthefailoverrequirements,thatis,thecurrentfailovercapacityissmaller thanconfiguredfailovercapacity.Ifstrictadmissioncontrolisdisabled,clustersdonot becomered,regardlessofwhetherthehostscanguaranteefailover.
86
VMware, Inc.
Inadequatefailovercapacitycanhappen,forexample,ifyoupoweronsomanyvirtual machinesthattheclusternolongerhassufficientresourcestoguaranteefailoverforthe specifiednumberofhosts. ItcanalsohappenifHAissetupfortwohostfailureinafourhostclusterandonehost fails.Theremainingthreehostsmightnolongerbeabletosatisfyatwohostfailure. IfaclusterenabledforHAbecomesredorifcurrentfailovercapacityfallsbelowthe configuredfailovercapacity,itcannolongerguaranteefailoverforthespecified numberofhostsbutcontinuestoperformfailover.Incaseofhostfailure,HAfirstfails overthevirtualmachinesofonehostinorderofpriority,andthenthevirtualmachines ofthesecondhostinorderofpriority,andsoon.SeeCustomizingHAforVirtual Machinesonpage 115. TheSummarypagedisplaysalistofconfigurationissuesforredandyellowclusters. Thelistexplainswhatcausestheclustertobecomeovercommittedorinvalid. NOTEDRSbehaviorisnotaffectedifaclusterisredbecauseofanHAissue.
VMware, Inc.
87
88
VMware, Inc.
Creating a VMware Cluster
Thischapterdiscussesthefollowingtopics: ClusterPrerequisitesonpage 89 ClusterCreationOverviewonpage 92 CreatingaClusteronpage 93 ViewingClusterInformationonpage 96 NOTEAlltasksassumeyouhavepermissiontoperformthem.SeetheonlineHelpfor informationonpermissions.
Cluster Prerequisites
YoursystemmustmeetcertainprerequisitestouseVMwareclusterfeatures successfully. Ingeneral,DRSandHAworkbestifthevirtualmachinesmeetVMotion requirements,asdiscussedinthenextsection. IfyouwanttouseDRSforloadbalancing,thehostsinyourclustermustbepartof aVMotionnetwork.IfthehostsarenotintheVMotionnetwork,DRScanstill makeinitialplacementrecommendations.
VMware, Inc.
89
Clusters Enabled for HA

InclustersenabledforHA,allvirtualmachinesandtheirconfigurationfilesmust resideonsharedstorage(suchasaSAN),becauseyoumustbeabletopoweronthe virtualmachineonanyhostinthecluster.Hostsmustalsobeconfiguredtohaveaccess tothesamevirtualmachinenetworkandtootherresources. EachhostinanHAclustermustbeabletoresolvethehostnameandIPaddressofall otherhostsinthecluster.Toachievethis,youcaneithersetupDNSoneachhost (preferred)orfillinthe/etc/hostsentriesmanually(errorproneanddiscouraged). ToresolvenamesusingDNS,youmustverifythattheNISClientserviceonthefirewall oftheESXServerhostisenabled.(ThisisnotnecessaryifyouareusingESXServer3i.) To enable the NIS Client service 1 2 3 4 IntheVIClient,selectthehostandclicktheConfigurationtab. SelectSecurityProfile. IfyoudonotseeNISClientlistedundertheOutgoingConnectionsforthe Firewall,clickProperties. IntheFirewallPropertiesdialogbox,selectNISClientandclickOK.
NOTEAllhostsinanHAclustermusthaveDNSconfiguredsothattheshorthost name(withoutthedomainsuffix)ofanyhostintheclustercanberesolvedtothe appropriateIPaddressfromanyotherhostinthecluster.Otherwise,the Configuring HAtaskfails.IfyouaddthehostusingtheIPaddress,alsoenable reverseDNSlookup(theIPaddressshouldberesolvabletotheshorthostname). ForVMwareHAusagewithinESXServer3,redundantconsolenetworkingis recommended(thoughnotrequired).Similarly,withESXServer3i,redundant VMkernelnetworkingisrecommended.Ifredundancyisnotprovided,thereisasingle pointoffailureinyourfailoversetup.Whenahostsnetworkconnectionfails,the secondconnectioncansendheartbeatstootherhosts. Tosetupredundancy,youneedtwophysicalnetworkadaptersoneachhost.You connectthemtothecorrespondingserviceconsole(orVMkernelnetwork,in ESXServer3i),usingtwoserviceconsoleinterfaces(VMkernelnetworkinterfacesin ESXServer3i)orusingasingleinterfaceusingNICteaming. NOTEAfteryouhaveaddedaNICtoahostinyourHAcluster,youmustreconfigure HAonthathost.
90
VMware, Inc.
Chapter 5 Creating a VMware Cluster
VirtualCenter VMotion Requirements

TobeconfiguredforVMotion,eachhostintheclustermustmeetthefollowing requirements.
Shared Storage
Ensurethatthemanagedhostsusesharedstorage.Sharedstorageistypicallyona storageareanetwork(SAN).(SEEUPDATE)SeetheiSCSISANConfigurationGuideand theFibreChannelSANConfigurationGuideforadditionalinformationonSANandthe ESXServerConfigurationGuideforinformationonothersharedstorage.
Shared VMFS Volume

ConfigureallmanagedhoststousesharedVMFSvolumes. PlacethedisksofallvirtualmachinesonVMFSvolumesthatareaccessibleby sourceanddestinationhosts. SetaccessmodeforthesharedVMFStopublic. EnsuretheVMFSvolumeissufficientlylargetostoreallvirtualdisksforyour virtualmachines. EnsureallVMFSvolumesonsourceanddestinationhostsusevolumenames,and allvirtualmachinesusethosevolumenamesforspecifyingthevirtualdisks. NOTE(SEEUPDATE)VirtualmachineswapfilesalsoneedtobeonaVMFSaccessible tosourceanddestinationhosts(justlike.vmdkvirtualdiskfiles).Thisrequirementno longerappliesifallsourceanddestinationhostsareESXServer3.5orhigher.Inthat case,VMotionwithswapfilesonunsharedstorageissupported.Swapfilesareplaced onaVMFSbydefault,butadministratorsmightoverridethefilelocationusing advancedvirtualmachineconfigurationoptions.
Processor Compatibility
Ensurethatthesourceanddestinationhostshaveacompatiblesetofprocessors. VMotiontransferstherunningarchitecturalstateofavirtualmachinebetween underlyingVMwareESXServersystems.VMotioncompatibilitymeansthatthe processorsofthedestinationhostmustbeabletoresumeexecutionusingthe equivalentinstructionswheretheprocessorsofthesourcehostweresuspended. Processorclockspeedsandcachesizesmightvary,butprocessorsmustcomefromthe samevendorclass(IntelversusAMD)andsameprocessorfamilytobecompatiblefor migrationwithVMotion.
VMware, Inc.
91
ProcessorfamiliessuchasXeonMPandOpteronaredefinedbytheprocessorvendors. Youcandistinguishdifferentprocessorversionswithinthesamefamilybycomparing theprocessorsmodel,steppinglevel,andextendedfeatures. Inmostcases,differentprocessorversionswithinthesamefamilyaresimilar enoughtomaintaincompatibility. Insomecases,processorvendorshaveintroducedsignificantarchitectural changeswithinthesameprocessorfamily(suchas64bitextensionsandSSE3). VMwareidentifiestheseexceptionsifitcannotguaranteesuccessfulmigration withVMotion. NOTEVMwareisworkingtomaintainVMotioncompatibilityacrossthewidestrange ofprocessorsthroughpartnershipswithprocessorandhardwarevendors.Forcurrent information,contactyourVMwarerepresentative.
Other Requirements
OtherVMotionrequirementsyoumustobserve: ForESXServer3.x,thevirtualmachineconfigurationfileforESXServerhostsmust resideonaVMFS. VMotiondoesnotsupportrawdisksormigrationofapplicationsclusteredusing MicrosoftClusterService(MSCS). VMotionrequiresaprivateGigabitEthernetmigrationnetworkbetweenallofthe VMotionenabledmanagedhosts.WhenVMotionisenabledonamanagedhost, configureauniquenetworkidentityobjectforthemanagedhostandconnectitto theprivatemigrationnetwork.
Cluster Creation Overview

Whencreatingclusters,makesurethatyoursystemmeetsclusterprerequisites.(See ClusterPrerequisitesonpage 89.)StarttheNewClusterwizard. To start the cluster wizard 1 2 RightclickthedatacenterorfolderandchooseNewCluster. (Ctrl+listhekeyboardshortcut). Chooseclustersettingsaspromptedbythewizardandexplainedinthischapter.
Inthefirstpanel,youchoosewhethertocreateaclusterthatsupportsVMwareDRS, VMwareHA,orboth.Thatchoiceaffectsthepagesdisplayedsubsequently,and implicitlydeterminesthelistoftasksdisplayedintheleftpanelofthewizard.Ifyou selectDRSandHA,youarepromptedforconfigurationinformationfortheseoptions.

92 VMware, Inc.
Whenyoucreateacluster,itinitiallydoesnotincludeanyhostsorvirtualmachines: AddinghostsisdiscussedinAddingHoststoaDRSClusteronpage 102and AddingHoststoanHAClusteronpage 118. AddingvirtualmachinesisdiscussedinChapter 7,ClustersandVirtual Machines,onpage 111.
Creating a Cluster
ThissectiondiscusseseachofthepagesintheNewClusterwizard.
Choosing Cluster Features

ThefirstpanelintheNewClusterwizardallowsyoutospecifythefollowing information: NameSpecifiesthenameofthecluster.ThisnameappearsintheVIClient inventorypanel.Youmustspecifyanametocontinuewithclustercreation. EnableVMwareHAIfthisboxisselected,VirtualCenterrestartsrunning virtualmachinesonadifferenthostwhenthesourcehostfails.See UnderstandingVMwareHAonpage 72. EnableVMwareDRSIfthisboxisselected,DRSusesloaddistribution informationforinitialplacementandloadbalancingrecommendationsortoplace andmigratevirtualmachinesautomatically.SeeUnderstandingVMwareDRS onpage 62. Specifythenameandchooseoneorbothclusterfeatures.ClickNexttocontinue. NOTEYoucanchangetheselectedclusterfeatures.SeeChapter 6,ManagingVMware DRS,onpage 101andChapter 8,ManagingVMwareHA,onpage 117.
Selecting Automation Level

IfyouselectedtheEnableVMwareDRSoptioninthesecondpanelofthewizard,the VMwareDRSpanelallowsyoutoselectthedefaultlevelofautomation.See UnderstandingVMwareDRSonpage 62foradetaileddiscussionofthedifferent choices. NOTEYoucanchangethelevelofautomationforthewholeclusterorforindividual virtualmachines.SeeReconfiguringDRSonpage 107andCustomizingDRSfor VirtualMachinesonpage 114.
VMware, Inc.
93
Table 51summarizesthechoicesofferedbythewizard. Table 5-1. DRS Automation Levels

Initial Placement Manual Partially Automated Fully Automated Displayofrecommended host(s). Automaticplacement. Automaticplacement. Migration Migrationrecommendationisdisplayed. Migrationrecommendationisdisplayed. Migrationrecommendationsare executedautomatically.
NOTENeitherthedefaultclusterautomationlevelnoraspecificvirtualmachines automationmodeaffectrecommendationsandactionsarisingfromtheDistributed PowerManagementfeature.Thisisaccomplishedbychoosingapowermanagement automationlevel.SeeDistributedPowerManagementonpage 68.
Selecting HA Options
IfyouenabledHA,theNewClusterwizardallowsyoutosettheoptionslistedin Table 52.SeeWorkingwithVMwareHAonpage 121. Table 5-2. VMware HA Options
Option HostFailures RestartPriority Description Specifiesthenumberofhostfailuresforwhichyouwanttoguarantee failover. Determinestheorderinwhichvirtualmachinesarerestarteduponhost failure.Valuesare:Disabled,Low,Medium,High.Thedefaultis Medium.IfDisabledisselected,HAisdisabledforthevirtual machines. SeeRestartpriority.onpage 115.
94
VMware, Inc.
Table 5-2. VMware HA Options (Continued)

Option IsolationResponse Description DetermineswhathappenswhenahostinanHAclusterlosesitsconsole network(orVMkernelnetwork,inESXServer3i)connectionbut continuesrunning.Valuesare:Poweroff(thedefault)andLeave poweredon.SeeIsolationresponse.onpage 115. Offerstwochoices: Donotpoweronvirtualmachinesiftheyviolateavailability constraintsenforcesthefailovercapacitysetabove. Allowvirtualmachinestobepoweredoneveniftheyviolate availabilityconstraintsallowsyoutopoweronvirtualmachines eveniffailoverofthenumberofspecifiedhostscannolongerbe guaranteed.Ifthisoptionisnotselected(thedefault),thefollowing operationsarealsonotallowediftheywouldviolateavailability constraints: Revertingapoweredoffvirtualmachinetoapoweredon snapshot Migratingavirtualmachineintothecluster ReconfiguringavirtualmachinetoincreaseitsCPUormemory reservation
AdmissionControl
Selecting a Virtual Machine Swapfile Location

Thiswizardpageallowsyoutoselectalocationfortheswapfilesofyourvirtual machines.Youcaneitherstoreaswapfileinthesamedirectoryasthevirtualmachine itself,oradatastorespecifiedbythehost(hostlocalswap).SeeSwappingon page 142.
Finishing Cluster Creation

Afteryoucompleteallselectionsforyourcluster,thewizardpresentsaSummarypage thatliststheoptionsyouselected.ClickFinishtocompleteclustercreation,orclick Backtogobackandmakemodificationstotheclustersetup. Youcanviewtheclusterinformation(seeViewingClusterInformationonpage 96) oraddhostsandvirtualmachinestothecluster(seeAddingHoststoaDRSCluster onpage 102andAddingHoststoanHAClusteronpage 118). Youcanalsocustomizeclusteroptions,asdiscussedinChapter 6,ManagingVMware DRS,onpage 101andChapter 8,ManagingVMwareHA,onpage 117.
VMware, Inc.
95
Viewing Cluster Information

Thissectiondiscussestheinformationpagesdisplayedwhenyouselectaclusterinthe inventorypanel. NOTEForinformationaboutallotherpages,seetheonlineHelp.
Summary Page
TheclusterSummarypagedisplayssummaryinformationforthecluster.See Figure 51. Figure 5-1. Cluster Summary Tab
96
VMware, Inc.
Table 5-3. Cluster Summary Information

Panel General Description Includesinformationaboutthecluster: VMwareDRSEnabledorDisabled. VMwareHAEnabledorDisabled. TotalCPUResourcesTotalCPUresourcesavailableforthecluster.The sumofallresourcesavailablefromthehosts. TotalMemoryTotalmemoryforthecluster.Thesumofallresources availablefromthehosts. NumberofHostsNumberofhostsinthecluster.Thisnumbercan changeifyouaddorremovehosts. TotalProcessorsSumofallprocessorsofallhosts. NumberofVirtualMachinesTotalofallvirtualmachinesinthecluster oranyofitschildresourcepools.Includesvirtualmachinesthatarenot currentlypoweredon. TotalMigrationsTotalmigrationsperformedbyDRSorbytheusersince theclusterwascreated. Commands Allowsyoutocallcommonlyusedcommandsforacluster. NewVirtualMachineDisplaysaNewVirtualMachinewizard.The wizardpromptsyoutochooseoneofthehostsinthecluster. AddHostAddsahostnotcurrentlymanagedbythesameVirtualCenter Server.ToaddahostmanagedbythesameVirtualCenterServer,dragand dropthehostintheinventorypanel. NewResourcePoolCreatesachildresourcepoolofthecluster. EditSettingsDisplaystheclustersEditSettingsdialogbox. VMwareHA Displaystheadmissioncontrolsetting,currentfailovercapacity,and configuredfailovercapacityforclustersenabledforHA. Thesystemupdatesthecurrentfailovercapacitywheneverahosthasbeen addedtoorremovedfromtheclusterorwhenvirtualmachineshavebeen poweredonorpoweredoff. VMwareDRS Displaysthedefaultautomationlevel,migrationthreshold,and outstandingmigrationrecommendationsforthecluster. MigrationrecommendationsappearifyouselecttheDRS Recommendationstab.SeeMigrationRecommendationsonpage 68. Defaultautomationlevelandmigrationthresholdaresetduringcluster creation.SeeMigrationThresholdonpage 67. DRSResource Distribution Displaystworealtimehistograms,UtilizationPercentandPercentof EntitledResourcesDelivered.Thechartsillustratehowbalancedacluster is.SeeDRSResourceDistributionChartsonpage 98.
VMware, Inc.
97
DRS Resource Distribution Charts

ThetwoDRSResourceDistributionchartsallowyoutoevaluatethehealthofyour cluster.ThechartsupdateeachtimetheSummarypageappearsandupdate periodicallyasperformancelimitationspermit.
Top DRS Resource Distribution Chart

(SEEUPDATE)ThischartisahistogramthatshowsthenumberofhostsontheXaxisand theutilizationpercentageontheYaxis.Iftheclusterisunbalanced,youseemultiple bars,correspondingtodifferentutilizationlevels.Forexample,youmighthaveone hostat20percentCPUutilization,anotherat80percentCPUutilization,each representedbyabluebar.Inclustersthathaveanautomateddefaultautomationlevel, DRSmigratesvirtualmachinesfromtheheavilyloadedhosttothehostthatsat20 percentresourceutilization.Theresultisasinglebluebarinthe4050percentrangefor hostsofsimilarcapacity. Forabalancedcluster,thischartshowstwobars:oneforCPUutilizationandonefor memoryutilization.However,ifthehostsintheclusterarelightlyutilized,youmight havemultiplebarsforbothCPUandmemoryinabalancedcluster.
Bottom DRS Resource Distribution Chart

ThischartisahistogramthatshowsthenumberofhostsontheYaxisandthe percentageofentitledresourcesdeliveredforeachhostontheXaxis.Whilethetop chartreportsrawresourceutilizationvalues,thebottomchartalsoincorporates informationaboutresourcesettingsforvirtualmachinesandresourcepools. DRScomputesaresourceentitlementforeachvirtualmachine,basedonitsconfigured shares,reservation,andlimitsettings,aswellascurrentresourcepoolconfiguration andsettings.DRSthencomputesaresourceentitlementforeachhostbyaddingupthe resourceentitlementsforallvirtualmachinesrunningonthathost.Thepercentageof entitledresourcesdeliveredisequaltothehostscapacitydividedbyitsentitlement. Forabalancedcluster,ahostscapacityshouldbegreaterthanorequaltoits entitlement,sothechartshouldideallyhaveasinglebarforeachresourceinthe90100 percenthistogrambucket.Anunbalancedclusterhasmultiplebars.Barswithlow Xaxisvaluesindicatethatvirtualmachinesonthosehostsarenotgettingtheresources towhichtheyareentitled.
98
VMware, Inc.
DRS Recommendations Page

TheDRSRecommendationspagedisplaysthecurrentsetofrecommendations generatedforoptimizingresourceutilizationintheclusterthrougheithermigrations orpowermanagement.VirtualCenterupdatesthelistofrecommendations periodically,basedonthevaluessetforthecluster. Iftherearenocurrentrecommendations,theDRSRecommendationspagedisplaysNo DRSrecommendationsatthistime.Whentherearecurrentrecommendations,this sectionappearsasshowninFigure 52: Figure 5-2. DRS Recommendations
TheDRSRecommendationssectiondisplaysinformationabouteachiteminthe columnsdescribedinTable 54. Table 5-4. DRS Recommendations Information

Column Priority Description Priorityforthisrecommendation,asanumberofstars.Fivestars,the maximum,indicateamandatorymovebecauseofahostentering maintenancemodeoraffinityruleviolations.Otherratingsdenotehow muchtherecommendationwouldimprovetheclustersperformance; fromfourstars(significantimprovement)toonestar(slight).
VMware, Inc.
99
Table 5-4. DRS Recommendations Information (Continued)

Column Recommendation Description Whatappearsinthiscolumndependsonthetypeofrecommendation: Forvirtualmachinemigrations:thenameofthevirtualmachineto migrate,thesourcehost(onwhichthevirtualmachineiscurrently running),andthedestinationhost(towhichthevirtualmachineis migrated). Forhostpowerstatechanges:thenameofthehosttopoweronoroff. Reason Reasonfortherecommendation:whythevirtualmachineis recommendedformigrationorthehostisrecommendedforapower statetransition.Reasonscanberelatedtoanyofthefollowing: BalanceaverageCPUormemoryloads. Satisfyanaffinityorantiaffinityrule. Hostisenteringmaintenance. Decreasepowerconsumption. Poweroffaspecifichost. Increaseclustercapacity.
SeeApplyingDRSRecommendationsonpage 105forinformationaboutusingthis page. NOTETheDRSActionHistorysectionimmediatelybelowtheDRSRecommendations sectiondisplaystherecommendationsappliedforthisclusteroveraperiodoftime.
100
VMware, Inc.
Managing VMware DRS
ThischapterexplainshowtoaddhoststoaDRScluster,howtoremovethem,andhow tocustomizeDRS. Thischapterdiscussesthefollowingtopics: CustomizingDRSonpage 101 AddingHoststoaDRSClusteronpage 102 RemovingHostsfromClustersonpage 104 ApplyingDRSRecommendationsonpage 105 ReconfiguringDRSonpage 107 UsingDRSAffinityRulesonpage 108 NOTEAlltasksdescribedassumeyouarelicensedandyouhavepermissionto performthem.SeetheonlineHelpforinformationonpermissionsandhowtosetthem.
Customizing DRS
Afteryouhavecreatedacluster,youcanenableitforDRS,HA,orboth.Youcan proceedtoaddorremovehosts,andcustomizetheclusterinotherways. YoucancustomizeDRSasfollows: Specifythedefaultautomationlevelandmigrationthresholdduringcluster creation.SeeSelectingAutomationLevelonpage 93. Addhoststothecluster.SeeAddingHoststoaDRSClusteronpage 102
VMware, Inc.
101
Changethedefaultautomationlevelormigrationthresholdforexistingclusters, asdiscussedinReconfiguringDRSonpage 107. Setcustomautomationmodesforindividualvirtualmachinesinyourclusterto overridetheclusterwidesettings.Forexample,youcansettheclustersdefault automationleveltoautomaticbutthemodeofsomeindividualmachinesto manual.SeeCustomizingDRSforVirtualMachinesonpage 114. Groupvirtualmachinesbyusingaffinityrules.Affinityrulesspecifythatselected virtualmachineshouldalwaysbeplacedonthesamehost.Antiaffinityrules specifythatselectedvirtualmachinesshouldalwaysbeplacedondifferenthosts. SeeUsingDRSAffinityRulesonpage 108.
Adding Hosts to a DRS Cluster

Theprocedureforaddinghoststoaclusterisdifferentforhostscurrentlymanagedby thesameVirtualCenterServer(managedhost)thanforhostsnotcurrentlymanagedby thatserver. Afterthehosthasbeenadded,thevirtualmachinesdeployedtothehostbecomepart ofthecluster.DRSmightrecommendmigrationofsomevirtualmachinestootherhosts inthecluster.
Adding Managed Hosts to a Cluster

TheVirtualCenterinventorypaneldisplaysallclustersandallhostsmanagedbythat VirtualCenterServer.ForinformationonaddingahosttoaVirtualCenterServer,see theVirtualInfrastructureUsersGuide. To add a managed host to a cluster 1 2 3 Selectthehostfromeithertheinventoryorlistview. Dragthehosttothetargetclusterobject. Thewizardaskswhatyouwanttodowiththehostsvirtualmachinesandresource pools. IfyouchoosePutthishostsvirtualmachinesintheclustersrootresource pool,VirtualCenterremovesallexistingresourcepoolsofthehostandthe virtualmachinesinthehostshierarchyareallattachedtotheroot. NOTEBecauseshareallocationsarerelativetoaresourcepool,youmighthaveto manuallychangeavirtualmachinessharesafterperformingthepreceding operation,whichdestroystheresourcepoolhierarchy.
102
VMware, Inc.
Chapter 6 Managing VMware DRS
IfyouchooseCreateanewresourcepoolforthishostsvirtualmachinesand resourcepools,VirtualCentercreatesatoplevelresourcepoolthatbecomesa directchildoftheclusterandaddsallchildrenofthehosttothatnewresource pool.Youcansupplyanameforthatnewtoplevelresourcepool.Thedefault isGraftedfrom<host_name>. NOTEIfthehosthasnochildresourcepoolsorvirtualmachines,thehostsresources areaddedtotheclusterbutnoresourcepoolhierarchywithatoplevelresourcepool iscreated. Totakeadvantageofautomaticmigrationfeatures,youmustalsosetupthehosts VMotionnetwork. NOTEWhenyoulaterremovethehostfromthecluster,theresourcepoolhierarchy remainspartofthecluster.Thehostlosestheresourcepoolhierarchy.Thislossmakes sensebecauseoneofthegoalsofresourcepoolsistosupporthostindependent resourceallocation.Youcan,forexample,removetwohostsandreplacethemwitha singlehostwithsimilarcapabilitieswithoutadditionalreconfiguration.
Adding Unmanaged Hosts to a Cluster

YoucanaddahostthatisnotcurrentlymanagedbythesameVirtualCenterServeras thecluster(anditisnotvisibleintheVIClient). To add an unmanaged host to a cluster 1 2 3 4 SelecttheclustertowhichyouwanttoaddthehostandchooseAddHostfromthe rightclickmenu. Supplythehostname,username,andpassword,andclickNext. ViewthesummaryinformationandclickNext. Answerthepromptaboutvirtualmachineandresourcepoollocationdiscussedin AddingManagedHoststoaClusteronpage 102.
VMware, Inc.
103
Removing Hosts from Clusters

Toremoveahostfromacluster,youmustplacethehostinmaintenancemode.See HostMaintenanceandStandbyModesonpage 71forbackgroundinformation. To place a host in maintenance mode 1 SelectthehostandchooseEnterMaintenanceModefromtherightclickmenu. ThehostisinastateofEnteringMaintenanceModeuntilyoupowerdownall runningvirtualmachinesormigratethemtodifferenthosts.Youcannotpoweron virtualmachinesormigratevirtualmachinestoahostenteringmaintenance mode. Whentherearenopoweredonvirtualmachines,thehostisinmaintenancemode. 2 Whenthehostisinmaintenancemode,youcandragittoadifferentinventory location,eitherthetopleveldatacenteroraclusterotherthanthecurrentone. Whenyoumovethehost,itsresourcesareremovedfromthecluster.Ifyougrafted thehostsresourcepoolhierarchyontothecluster,thathierarchyremainswiththe cluster. 3 Afteryouhavemovedthehost,youcan: RemovethehostfromtheVirtualCenterServer(ChooseRemovefromthe rightclickmenu). RunthehostasastandalonehostunderVirtualCenter(ChooseExit MaintenanceModefromtherightclickmenu). Movethehostintoanothercluster.
Host Removal and Resource Pool Hierarchies

Whenyouremoveahostfromacluster,thehostendsupwithonlythe(invisible)root resourcepool,evenifyouusedaDRSclusteranddecidedtograftthehostresource poolwhenyouaddedthehosttothecluster.Inthatcase,thehierarchyremainswith thecluster. Youcancreateanew,hostspecificresourcepoolhierarchy.
104
VMware, Inc.
Host Removal and Virtual Machines

Becausethehostmustbeinmaintenancemodebeforeyoucanremoveit,allvirtual machinesmustbepoweredoff.Whenyouremovethehostfromthecluster,thevirtual machinesthatarecurrentlyassociatedwiththehostarealsoremovedfromthecluster. NOTEBecauseDRSmigratesvirtualmachinesfromonehosttoanother,youmightnot havethesamevirtualmachinesonthehostaswhenyouoriginallyaddedthehost.
Host Removal and Invalid Clusters

Ifyouremoveahostfromacluster,theresourcesavailablefortheclusterdecrease. IftheclusterisenabledforDRS,removingahostcanhavethefollowingresults: Iftheclusterstillhasenoughresourcestosatisfythereservationsofallvirtual machinesandresourcepoolsinthecluster,theclusteradjustsresourceallocation toreflectthereducedamountofresources. Iftheclusterdoesnothaveenoughresourcestosatisfythereservationsofall resourcepools,butthereareenoughresourcestosatisfythereservationsforall virtualmachines,analarmisissuedandtheclusterismarkedyellow.DRS continuestorun. IfaclusterenabledforHAlosessomanyresourcesthatitcannolongerfulfillits failoverrequirements,amessageappearsandtheclusterturnsred.Theclusterfails overvirtualmachinesincaseofhostfailure,butisnotguaranteedtohaveenough resourcesavailabletofailoverallvirtualmachines.
Applying DRS Recommendations

VirtualCenterdisplaysthemigrationandpowermanagementrecommendationsfora clusterontheDRSRecommendationspage.SeeDRSRecommendationsPageon page 99.Thispageisalsothelocationwhererecommendationsareapplied.See Figure 61.
VMware, Inc.
105
Figure 6-1. DRS Recommendations
Recommendation Grouping
Therecommendationspageisorganizedintoboxes.Eachboxcontains recommendationsthatsharesomedegreeofinterdependence,whilerecommendations thatappearindifferentboxesareconsideredindependentfromoneanother.Withina box,recommendationsthataredependentonotherrecommendationsareplacedbelow theirprerequisites.Theseinterdependenciesleadtothefollowingactionswhenapplying recommendations: Whenadependentrecommendationisselected(theApplycheckboxisselected), anyrecommendationsappearingaboveitinthesameboxthatareitsprerequisite arealsoselected.Thedependentrecommendationcannotbeappliedwithout them.Notallrecommendationsaboveareprerequisites. Whenaprerequisiterecommendationisdeselected,alloftherecommendations thatappearbelowitinthesameboxanddependonitarealsodeselected.Ifthe prerequisiteisnotapplied,theywillnotbeappliedeither.Notall recommendationsbelowaredependent. Anothertypeofinterdependenceamongstrecommendationsthatisdisplayedonthis pageiswhenactionsareatomicandcanonlybeappliedasasingleunit.Thesetypesof recommendationsmightbenecessarytosatisfyanaffinity(orantiaffinity)ruleand theyareindicatedbyalinkiconandasingleApplycheckbox.
106
VMware, Inc.
Using the DRS Recommendations Page

Bydefault,theApplycheckboxesofallDRSrecommendationsthatappearontheDRS Recommendationspageareselectedandunavailable(theycannotbedeselected).To deselectrecommendations,selecttheOverridesuggestedDRSactionscheckbox.This actionactivatestheApplycheckboxes.Afteryouhavedeterminedwhichofthe recommendationsyouwanttoapply,clicktheApplyRecommendationsbutton. TwootheractionsarepossiblefromtheDRSRecommendationspage: ClicktheGenerateRecommendationsbuttontorefreshtheentirepage.Youmight dothisifyoumadechangestotheconfigurationoftheclusterandwanttosee updatedrecommendationsforthenewconfigurationimmediately. Clickthethresholdlink,whichappearsabovetherecommendationstableasastar rating(forexample,2ormorestars),toopentheClusterSettingsdialogbox.You canadjustthedefaultclusterautomationlevelandthepowermanagement automationlevel,aswellastheotherclusterlevelsettings.
Reconfiguring DRS
YoucanturnoffDRSforacluster,oryoucanchangetheconfigurationoptions. To turn off DRS 1 2 3 Selectthecluster. ChooseEditSettingsfromtherightclickmenu. Intheleftpanel,selectGeneral,anddeselecttheVMwareDRScheckbox. YouarewarnedthatturningoffDRSdestroysallresourcepoolsinthecluster. 4 ClickOKtoturnoffDRSanddestroyallresourcepools.
CAUTIONTheresourcepoolsdonotbecomereestablishedwhenyouturnDRSback on.ChangingtheDRSautomationleveltomanual(insteadofturningitoff)prevents anyautomaticDRSactions,butpreservestheresourcepoolhierarchy. To reconfigure DRS 1 2 3 Selectthecluster. ChooseEditSettingsfromtherightclickmenu. IntheClusterSettingsdialogbox,selectVMwareDRS.
VMware, Inc.
107
Setthedefaultautomationlevel: Selectoneoftheradiobuttonstochangeautomationlevel.SeeSelecting AutomationLevelonpage 93. IfyouselectedFullyautomated,youcanmovetheMigrationThreshold slidertochangethemigrationthreshold.SeeMigrationThresholdon page 67.
NOTETheAdvancedOptionsdialogboxishelpfulwhenyouareworkingwith VMwarecustomersupportonresolvinganissue.Settingadvancedoptionsisnot otherwiserecommended. TosuspendDRSrecommendedVMotionmigrationswithoutchangingtheresource poolhierarchy,settheDRSautomationleveltomanualorpartiallyautomatic.When thisisdone,DRScontinuestorecommendVMotionmigrations,butVirtualCenterdoes notexecutethemwithoutuserapproval.Ifyouhavesettheautomationlevelof individualvirtualmachinestofullyautomatic,returnthemtotheclusterdefault.See CustomizingDRSforVirtualMachinesonpage 114forinstructions.
Using DRS Affinity Rules

AfteryouhavecreatedaDRScluster,youcanedititspropertiestocreaterulesthat specifyaffinity.DRSneverviolatesuserspecifiedrules,however,ifyouhavetworules thatareinconflict,youcannotenableboth.Forexample,ifonerulekeepstwovirtual machinestogetherandanotherrulekeepsthesametwovirtualmachinesapart,you cannotenablebothrules.Youcanusetheserulestodeterminethat: DRSshouldtrytokeepcertainvirtualmachinestogetheronthesamehost(for example,forperformancereasons). DRSshouldtrytomakesurethatcertainvirtualmachinesarenottogether(for example,forhighavailability).Youmightwanttoguaranteecertainvirtual machinesarealwaysondifferentphysicalhosts.Whentheresaproblemwithone host,youdonotlosebothvirtualmachines. NOTEDRSaffinityrulesarecompletelydifferentfromanindividualhostsCPU affinityrules.CPUaffinityrulesarediscussedinUsingCPUAffinitytoAssignVirtual MachinestoSpecificProcessorsonpage 126.
108
VMware, Inc.
To create a DRS rule 1 2 SelecttheclusterandchooseEditSettingsfromtherightclickmenu. IntheClusterSettingsdialogbox,chooseRules.
3 4
IntheVirtualMachineRuledialogbox,nametherulesoyoucanfindandeditit. Chooseoneoftheoptionsfromthepopupmenu: KeepVirtualMachinesTogether Onevirtualmachinecannotbepartofmorethanonesuchrule. SeparateVirtualMachines Thistypeofrulecannotcontainmorethantwovirtualmachines.
ClickAddtoaddvirtualmachines,andclickOKwhenyouaredone.
Afteryouaddtherule,youcaneditit,lookforconflictingrules,ordeleteit. To edit an existing rule 1 2 3 4 SelecttheclusterandchooseEditSettingsfromtherightclickmenu. Intheleftpanel,selectRules(underVMwareDRS). ClickDetailsforadditionalinformationontopicssuchasconflictingrules. Makethechangesinthedialogbox,andclickOKwhenyouredone.
VMware, Inc.
109
Understanding Rule Results

Whenyouaddoreditarule,andtheclusterisimmediatelyinviolationofthatrule,the systemcontinuestooperateandtriestocorrecttheviolation. ForDRSclustersthathaveadefaultautomationlevelofmanualorpartiallyautomated, migrationrecommendationsarebasedonbothrulefulfillmentandloadbalancing.You arenotrequiredtofulfilltherules,butthecorrespondingrecommendationsremain untiltherulesarefulfilled.
Disabling or Deleting Rules

Youcandisablearuleorremoveitcompletely. To disable a rule 1 2 3 SelecttheclusterandchooseEditSettingsfromtherightclickmenu. Intheleftpanel,selectRules(underVMwareDRS). DeselectthecheckboxtotheleftoftheruleandclickOK.
Youcanlaterenabletherulebyreselectingthecheckbox. To delete a rule 1 2 3 SelecttheclusterandchooseEditSettingsfromtherightclickmenu. Intheleftpanel,selectRules(underVMwareDRS). SelecttheruleyouwanttoremoveandclickRemove. Theruleisdeleted.
110
VMware, Inc.
Clusters and Virtual Machines
Thischapterexplainshowtoadd,remove,andcustomizevirtualmachines. Thischapterdiscussesthefollowingtopics: AddingVirtualMachinestoaClusteronpage 111 PoweringOnVirtualMachinesinaClusteronpage 112 RemovingVirtualMachinesfromaClusteronpage 113 CustomizingDRSforVirtualMachinesonpage 114 CustomizingHAforVirtualMachinesonpage 115 NOTEAlltasksassumeyouhavepermissiontoperformthem.SeetheonlineHelpfor informationonpermissionsandhowtosetthem.
Adding Virtual Machines to a Cluster

Youcanaddvirtualmachinestoaclusterwhentheclusteriscreated,bymigratingthe virtualmachinetothecluster,orbyaddingahostwithvirtualmachinestothecluster.
Adding a Virtual Machine During Creation

Whenyoucreateavirtualmachine,youcanaddittoaclusteraspartofthevirtual machinecreationprocess.WhentheNewVirtualMachinewizardpromptsyouforthe locationofthevirtualmachine,youcanchooseastandalonehostoraclusterandcan chooseanyresourcepoolinsidethehostorcluster. SeetheVirtualMachineManagementGuideformoreinformationondeployingvirtual machines.
VMware, Inc. 111
Migrating a Virtual Machine to a Cluster

Youcanmigrateanexistingvirtualmachinefromastandalonehosttoaclusterorfrom aclustertoanothercluster.Thevirtualmachinecanbepoweredonoroff.Tomovethe virtualmachineusingVirtualCenter,youhavetwochoices: Dragthevirtualmachineobjectontopoftheclusterobject. RightclickthevirtualmachinenameandchooseMigrate. ForDRSclusters,usersarepromptedforthefollowinginformation: Thelocation,whichcouldbetheclusteritselforaresourcepoolinsidethecluster. Ahostonwhichtopoweronandrunthevirtualmachineiftheclusterisinmanual mode.Iftheclusterhasadefaultautomationleveloffullyorpartiallyautomatic, DRSselectsthehost. NOTEYoucandragavirtualmachinedirectlytoaresourcepoolwithinacluster.In thiscase,theMigrationwizardisstartedbuttheresourcepoolselectionpagedoesnot appear.Migratingdirectlytoahostwithinaclusterisnotallowedbecausetheresource poolcontrolstheresources.
Adding a Host with Virtual Machines to a Cluster

Whenyouaddahosttoacluster,allvirtualmachinesonthathostareaddedtothe cluster.SeeAddingHoststoaDRSClusteronpage 102andAddingHoststoanHA Clusteronpage 118.
Powering On Virtual Machines in a Cluster

Whenyoupoweronvirtualmachinesonhoststhatarepartofacluster,theresulting VirtualCenterbehaviordependsonthetypeofcluster.
DRS Enabled
IfyoupoweronavirtualmachineorgroupofvirtualmachinesandDRSisenabled, VirtualCenterfirstperformsadmissioncontrol.Itcheckswhethertheclusterand resourcepoolhasenoughresourcesforthevirtualmachine(s).Iftheclusterdoesnot havesufficientresourcestopoweronasinglevirtualmachine,oranyofthevirtual machinesinagrouppoweronattempt,amessageappears.
112
VMware, Inc.
Chapter 7 Clusters and Virtual Machines
Ifsufficientresourcesareavailable,VirtualCenterproceedsasfollows: Iftheautomationlevelofanyoftheactionsthatwilltakeplace(virtualmachines beingpoweredonormigrated,orhostsbeingpoweredon)ismanual, VirtualCenterdisplaysaninitialplacementrecommendation.SeeInitial Placementonpage 62. Whenalloftheseactionsareautomatic,VirtualCenterplacesthevirtualmachine onthemostsuitablehostwithoutmakingarecommendation.
HA Enabled
IfyoupoweronavirtualmachineandHAisenabled,VirtualCenterperformsHA admissioncontrol.Itcheckswhetherenoughresourcesexisttoallowforthespecified numberofhostfailoversifyoupoweronthevirtualmachine. Ifenoughresourcesexist,thevirtualmachineispoweredon. Ifnotenoughresourcesexist,andifstrictadmissioncontrolisused(thedefault), amessageinformsyouthatthevirtualmachinecannotbepoweredon.Ifyouare notusingstrictadmissioncontrol,thevirtualmachineispoweredonwithout warnings.
Removing Virtual Machines from a Cluster

Youcanremovevirtualmachinesfromaclusterbymigratingthemoutoftheclusteror byremovingahostwithvirtualmachinesfromthecluster.
Migrating Virtual Machines out of a Cluster

Youcanmigrateavirtualmachinefromaclustertoastandalonehostorfromacluster toanotherclusterinoneoftwoways: Usethestandarddraganddropmethod. SelectMigratefromthevirtualmachinesrightclickmenuortheVirtualCenter menubar. IfthevirtualmachineisamemberofaDRSclusteraffinityrulesgroup(seeUsingDRS AffinityRulesonpage 108),VirtualCenterdisplaysawarningbeforeitallowsthe migrationtoproceed.Thewarningindicatesthatdependentvirtualmachinesarenot migratedautomatically.Youhavetoacknowledgethewarningbeforemigrationcan proceed.
VMware, Inc.
113
Removing a Host with Virtual Machines from a Cluster

Whenyouremoveahostwithvirtualmachinesfromacluster,allitsvirtualmachines areremovedaswell.Youcanremoveahostonlyifitisinmaintenancemodeor disconnected.SeeRemovingHostsfromClustersonpage 104. NOTEIfyouremoveahostfromanHAcluster,theclustercanbecomeredbecauseit nolongerhasenoughresourcesforfailover.IfyouremoveahostfromaDRScluster, theclustercanbecomeyellowbecauseitisovercommitted.SeeValid,Yellow,andRed Clustersonpage 81.
Customizing DRS for Virtual Machines

YoucancustomizetheautomationmodeforindividualvirtualmachinesinaDRS clustertooverridetheclustersdefaultautomationlevel.Forexampleyoucanselect ManualforspecificvirtualmachinesinaclusterwithfullautomationorPartially AutomatedforspecificvirtualmachinesinaclustersettoManual. IfavirtualmachineissettoDisabled,VirtualCenterdoesnotmigratethatvirtual machineorprovidemigrationrecommendationsforit. To set a custom automation mode for one or more virtual machines 1 2 SelecttheclusterandchooseEditSettingsfromtherightclickmenu. IntheClusterSettingsdialogbox,selectVirtualMachineOptionsintheleft column.
3 4
Selectanindividualvirtualmachine,orShiftselectorControlselectmultiple virtualmachines. Fromtherightclickmenu,chooseanautomationmodeandclickOK.
114
VMware, Inc.
Chapter 7 Clusters and Virtual Machines
Customizing HA for Virtual Machines

YoucancustomizeHAforrestartpriorityandisolationresponse: Restart priority. Determinestheorderinwhichvirtualmachinesarerestartedupon hostfailure.Restartpriorityisalwaysconsidered,butisimportantinthefollowing cases: Ifyousethostfailuretoacertainnumberofhosts(forexample,three)and morehostsfail(forexample,four). Ifyouturnedoffstrictadmissioncontrolandhavestartedmorevirtual machinesthanHAhasbeensetuptosupport. Isolation response. DetermineswhathappenswhenahostinanHAclusterlosesits consolenetwork(orVMkernelnetwork,inESXServer3i)connectionbutcontinues running.Theotherhostsintheclusternolongergetheartbeatsfromthishost,declare itdead,andtrytorestartitsvirtualmachines.Disklockingpreventstwoinstancesofa virtualmachinefromrunningontwodifferenthost.Bydefault,virtualmachinesare poweredoffontheisolatedhostincaseofahostisolationincident,sothattheycanbe restartedonadifferenthost.Youcanchangethatbehaviorforindividualvirtual machines. InthecaseofNASoriSCSIstorage,whenahostlosesconsolenetworking(orVMkernel networking,inESXServer3i),thevirtualmachinemightalsoloseaccesstoitsdisk.In thiscase,thedisklockcanbebrokenandthevirtualmachinemaysuccessfullybe poweredononadifferenthost.Thevirtualmachineontheisolatedhostcontinuesto run,butcannotaccessitsdisk(evenifitregainsnetworkconnectivity)becauseithas lostitsdisklock.ThatvirtualmachinemightbecreatingandconsumingnetworkI/O. VMwarerecommendsthatyoukeeptheIsolationResponseasPoweroff(thedefault) forvirtualmachineslocatedonNASoriSCSIstorage.
VMware, Inc.
115
To customize HA behavior for individual virtual machines 1 2 SelecttheclusterandchooseEditSettingsfromtherightclickmenu. ChooseVirtualMachineOptionsunderVMwareHA.
Foreachvirtualmachine,selectfromtheRestartPriorityorIsolationResponse menutocustomizeitssettings. RestartPriorityIndicatesrelativepriorityforrestartingthevirtualmachine incaseofhostfailure.Higherpriorityvirtualmachinesarestartedfirst. NOTEThispriorityappliesonlyonaperhostbasis.Ifmultiplehostsfail, VirtualCenterfirstmigratesallvirtualmachinesfromthefirsthostinorderof priority,andthenallvirtualmachinesfromthesecondhostinorderof priority,andsoon. IsolationResponseSpecifieswhattheESXServerhostthathaslost connectionwithitsclustershoulddowithrunningvirtualmachines.By default,eachvirtualmachineissettobeshutdownintheeventofahost isolationincident. YoucanchooseLeavepoweredontoindicatethatthevirtualmachineson isolatedhostsshouldcontinuetorunevenifthehostcannolonger communicatewithotherhostsinthecluster.Youmightdothis,forexample, ifthevirtualmachinenetworkisonadifferentnetworkthatisrobustand redundant,orifyouwanttokeepthevirtualmachinesrunning.
NOTEWhenyouaddahosttoacluster,allvirtualmachinesintheclusterdefaultto theclustersdefaultrestartpriority(Medium,ifunspecified)anddefaultisolation response(Poweroff,ifunspecified.)
116
VMware, Inc.
Managing VMware HA
ThischapterexplainshowtoaddhoststoanHAcluster,howtoremovethem,andhow tocustomizeHAclusters. Thischapterdiscussesthefollowingtopics: CustomizingHAonpage 117 AddingHoststoanHAClusteronpage 118 WorkingwithVMwareHAonpage 121 SettingAdvancedHAOptionsonpage 121 NOTEAlltasksdescribedassumeyouarelicensedandyouhavepermissionto performthem.SeetheonlineHelpforinformationonpermissions.
Customizing HA
Afteryoucreateacluster,youcanenableitforDRS,HA,orboth.Youcanthenaddor removehosts,andcustomizethecluster. YoucancustomizeHAasfollows: Duringclustercreation,acceptorchangetheclustersdefaultrestartpriorityand isolationresponse,choosethenumberofhostfailuresfortheclusterandindicate whetheryouwanttoenforcestrictadmissioncontrol.SeeSelectingHAOptions onpage 94. Addhosts,asdiscussedinAddingHoststoanHAClusteronpage 118.
VMware, Inc.
117
Changethenumberofhostfailuresortheadmissioncontrolforexistingclusters, asdiscussedinWorkingwithVMwareHAonpage 121. Setapriorityforindividualvirtualmachines.HAusesvirtualmachinepriorityto decidetheorderofrestartsothatvirtualmachineswithhigherpriorityfromthe samehostgetprecedenceincaseofinsufficientresources.SeeCustomizingHA forVirtualMachinesonpage 115. SetanIsolationResponseforindividualvirtualmachines.Bydefault,allvirtual machinesarepoweredoffifahostbecomesisolatedfromthenetwork.See CustomizingHAforVirtualMachinesonpage 115.
Adding Hosts to an HA Cluster

Theprocedureforaddinghoststoaclusterisdifferentforhostscurrentlymanagedby thesameVirtualCenterServer(managedhost)thanforhostsnotcurrentlymanagedby thatserver.Afterthehosthasbeenadded,thevirtualmachinesdeployedtothehost becomepartofthecluster.
Adding Managed Hosts to a Cluster

TheVirtualCenterinventorypaneldisplaysallclustersandallhostsmanagedbythat VirtualCenterServer.ForinformationonaddingahosttoaVirtualCenterServer,see theESXServerConfigurationGuide. To add a managed host to a cluster 1 2 Selectthehostfromeithertheinventoryorlistview. Dragthehosttothetargetclusterobject.
AddingthehosttotheclusterspawnsaConfiguring HAsystemtaskonthehost.After thistaskhascompleted,thehostisincludedintheHAservice.
Adding Unmanaged Hosts to a Cluster

YoucanaddahostthatisnotcurrentlymanagedbythesameVirtualCenterServeras thecluster(andisthereforenotvisible). To add an unmanaged host to a cluster 1 2 SelecttheclustertowhichyouwanttoaddthehostandchooseAddHostfromthe rightclickmenu. Supplythehostname,username,andpassword,andclickNext.
118
VMware, Inc.
Chapter 8 Managing VMware HA
Thehostisaddedtothecluster.Addingthehosttotheclusterspawnsasystemtask Configuring HAonthehost.Afterthistaskhascompleted,thehostisincludedinthe HAservice.
Results of Adding Hosts to a Cluster

WhenahostisaddedtoanHAcluster: Theresourcesforthathostareimmediatelyavailabletotheclusterforuseinthe clustersrootresourcepool. UnlesstheclusterisalsoenabledforDRS,allresourcepoolsarecollapsedintothe clusterstoplevel(invisible)resourcepool. NOTETheresourcepoolhierarchyislost.Itdoesnotbecomeavailablewhenyou laterremovethehostfromthecluster. Anycapacityonthehostbeyondwhatisrequiredorguaranteedforeachrunning virtualmachinebecomesavailableassparecapacityintheclusterpool.Usethis sparecapacityforstartingvirtualmachinesonotherhostsincaseofahostfailure. Ifyouaddahostwithseveralrunningvirtualmachines,andtheclusternolonger fulfillsitsfailoverrequirementsbecauseofthataddition,awarningappearsand theclusterismarkedred. Bydefault,allvirtualmachinesonthehostthatwasaddedaregiventhecluster defaultrestartpriority(Medium,ifunspecified)andtheclusterdefaultisolation response(Poweroff,ifunspecified).SeeSelectingHAOptionsonpage 94for informationontheseoptions. ThesystemalsomonitorsthestatusoftheHAserviceoneachhostanddisplays informationaboutconfigurationissuesontheSummarypage. Whenahostisremovedfromthecluster(ordisconnectedorputinmaintenance mode),theHAserviceisunconfigured.Youmightseeasystemspawned Unconfiguring HAsystemtaskonthehost,whichhastocomplete.
VMware, Inc.
119
Configuring and Unconfiguring HA on a Host

WhenyouaddahosttoanHAcluster,asystemtasksConfiguring HA isspawned. ThistaskhastocompletesuccessfullybeforethehostisreadyforHA.Thehoststateis yellowwhileitisbeingconfiguredorunconfiguredforHA,andtheSummarypage showstheoperationthatmightbepending. AhostisconfiguredforHAifyou: EnableHAforacluster ConnecttoahostinanHAcluster Exitmaintenancemodeonthehost AhostisunconfiguredforHAifyou: DisableHAonthecluster Disconnectthehost Entermaintenancemodeonthehost CAUTIONWhenyoudisconnectahostfromanHAcluster,youreducetheavailable resourcesforfailoveroperations.Ifaclustersfailovercapacityislessthanorequalto theconfiguredfailovercapacityandyoubegintheprocessofdisconnectingahost,you receiveaclusterfailoverwarning.Ifyoucompletethedisconnect,theclustermightbe unabletomaintainitsconfiguredfailoverlevel. AsystemtaskUnconfiguringHAmightgetspawned.Incaseofdisconnectorentering maintenancemode,theunconfigurationisdoneaspartoftherespectivetasks,andno separatesystemtaskisspawned.TheHAserviceisalsomonitoredoneachhost,and thehostsSummarypageindicatesanyerrors.Thehostismarkedred. Whenaconfigurationorunconfigurationtaskfails,youcangetadditionalinformation intherelatedeventsforthetask.Youmightalsoneedtocheckthelogsonthehost.If youfixtheerror,thehosthasaReconfigureHAtasktoreconfigureHAonahostwhere thehostfailed. NOTEWhenyouconfigureHA,aDNSserverisrequiredtoresolvehostnames. However,onceconfigured,HAcachesthenameresolutionanddoesnotrequireDNS lookuptoperformfailoveroperations.
120
VMware, Inc.
Chapter 8 Managing VMware HA
Working with VMware HA

ReconfiguringHAcanmeanturningitofforreconfiguringitsoptions. To turn off HA 1 2 3 Selectthecluster. ChooseEditSettingsfromtherightclickmenu. Intheleftpanel,selectGeneralanddeselecttheEnableVMwareHAcheckbox.
To reconfigure HA 1 2 3 4 Selectthecluster. ChooseEditSettingsfromtherightclickmenu. IntheClusterSettingsdialogbox,selectEnableVMwareHA. Makechangestothenumberofhostfailoversortheadmissioncontrolbehavior. SeeSelectingHAOptionsonpage 94.
Setting Advanced HA Options

ThissectionguidesyouthroughsettingadvancedattributesforHAandlistsafew attributesyoumightwanttoset.SincetheseattributesaffectthefunctioningofHA, changethemwithcaution. To set advanced options for HA 1 2 3 4 IntheclustersSettingsdialogbox,selectVMwareHA. ClicktheAdvancedOptionsbuttontoopentheAdvancedOptions(HA) dialog box. EntereachadvancedattributeyouwanttochangeinatextboxintheOption columnandthevalueitshouldbesetintheValue column. ClickOK.
VMware, Inc.
121
Table 8-1. Advanced HA Attributes

Attribute das.isolationaddress Description Setstheaddresstopingtodetermineifahostisisolated fromthenetwork.Ifthisoptionisnotspecified,the defaultgatewayoftheconsolenetworkisused.This defaultgatewayhastobesomereliableaddressthatis available,sothatthehostcandetermineifitisisolated fromthenetwork.Multipleisolationaddresses(upto10) canbespecifiedforthecluster: das.isolationaddressX,whereX=110. Bydefault,HAusesthedefaultgatewayoftheconsole networkasanisolationaddress.Thisattributespecifies whetherthatshouldbeused(true|false). Ifthisisset,HAtriestofailoverhoststothehost specifiedbythisoption.Thisattributeisusefultoutilize onehostasasparefailoverhost,butisnot recommended,becauseHAtriestoutilizeallavailable sparecapacityamongallhostsinthecluster. Ifthespecifiedhostdoesnothaveenoughsparecapacity, HAtriestofailoverthevirtualmachinetoanyotherhost intheclusterthathasenoughcapacity. das.failuredetectiontime Changesthedefaultfailuredetectiontime(witha defaultof15000milliseconds).Thisisthetimeperiod whenahosthasreceivednoheartbeatsfromanother host,thatitwaitsbeforedeclaringtheotherhostdead. ChangestheheartbeatintervalamongHAhosts.By default,thisoccurseverysecond. Specifiestheminimumamountofmemory(in megabytes)sufficientforanyvirtualmachineinthe clustertobeusable.Thisvalueisusedonlyifthe memoryreservationisnotspecifiedforthevirtual machineandisusedforHAadmissioncontroland calculatingthecurrentfailoverlevel.Ifnovalueis specified,thedefaultis256MB. SpecifiestheminimumamountofCPU(inmegahertz) sufficientforanyvirtualmachineintheclustertobe usable.ThisvalueisusedonlyiftheCPUreservationis notspecifiedforthevirtualmachineandisusedforHA admissioncontrolandcalculatingthecurrentfailover level.Ifnovalueisspecified,thedefaultis256MHz.
das.usedefaultisolationaddress
das.defaultfailoverhost
das.failuredetectioninverval das.vmMemoryMinMB
das.vmCpuMinMHz
122
VMware, Inc.
Advanced Resource Management
Thischapterdiscussessomeadvancedresourcemanagementtopics.Itincludes conceptualinformationandadiscussionoftheadvancedparametersyoucanset.In mostsituations,youdonotneedtousetheadvancedsettingsandusingtheadvanced settingsincorrectlymightbedetrimentaltoyoursystemsperformance.However, experiencedadministratorsmightfindtheseadvancedconfigurationoptionshelpful forfinetuningtheperformanceoftheESXServerenvironment. NOTELicensesforDRSandHAarenotrequiredforanyofthetopicsandtasks discussedinthischapter. Thischapterdiscussesthefollowingtopics: CPUVirtualizationonpage 124 UsingCPUAffinitytoAssignVirtualMachinestoSpecificProcessorson page 126 MulticoreProcessorsonpage 128 Hyperthreadingonpage 129 MemoryVirtualizationonpage 133 UnderstandingMemoryOverheadonpage 136 MemoryAllocationandIdleMemoryTaxonpage 138 HowESXServerHostsReclaimMemoryonpage 140 SharingMemoryAcrossVirtualMachinesonpage 144 AdvancedAttributesandWhatTheyDoonpage 145
VMware, Inc.
123
CPU Virtualization
TounderstandCPUrelatedissues,thedifferencebetweenemulationand virtualization. Withemulation,alloperationsareexecutedinsoftwarebyanemulator.Asoftware emulatorallowsprogramstorunonacomputersystemotherthantheoneforwhich theywereoriginallywritten.Theemulatordoesthisbyemulating,orreproducing,the originalcomputersbehaviorbyacceptingthesamedataorinputsandachievingthe sameresults.Emulationprovidesportabilityandisoftenusedtorunsoftwaredesigned foroneplatformacrossseveraldifferentplatforms. Withvirtualization,theunderlyingphysicalresourcesareusedwheneverpossibleand thevirtualizationlayerexecutesinstructionsonlyasneededtomakethevirtual machinesoperateasiftheywererunningdirectlyonaphysicalmachine.Virtualization emphasizesperformanceandrunsdirectlyontheprocessorwheneverpossible.
Software CPU Virtualization

WithsoftwareCPUvirtualization,theguestapplicationcoderunsdirectlyonthe processor,whiletheguestprivilegedcodeistranslatedandthetranslatedcode executesontheprocessor.Thetranslatedcodeisslightlylargerinsizethanthecode whichistranslated,andthisleadstoslowerguestexecution.Asaresult,guest programswhichhaveasmallprivilegedcodecomponentrunwithspeedsverycloseto native,whileprogramswithasignificantprivilegedcodecomponent,suchassystem calls,traps,orpagetableupdatescanrunslowerinthevirtualizedenvironment.
Hardware-Assisted CPU Virtualization

Certainprocessors(suchasIntelVTandAMDSVM)providehardwareassistancefor CPUvirtualization.Whenusingthisassistance,theguestisprovidedwithaseparate modeofexecutioncalledguestmode.Theguestcode,whetherapplicationcodeor privilegedcode,runsintheguestmode.Oncertainevents,theprocessorexitsoutof guestmodeandentersrootmode.Thehypervisorthenexecutesintherootmode, determinesthereasonfortheexit,takesanyrequiredactions,andrestartstheguestin guestmode. Whenyouusehardwareassistanceforvirtualization,theneedfortranslatingthecode iseliminated.Hencesystemcallsortrapintensiveworkloadsrunveryclosetonative speed.However,someworkloads,suchasthoseinvolvingupdatestopagetables,lead toalargenumberofexitsfromguestmodetorootmode.Dependingonthenumberof suchexitsandtotaltimespentinexits,thiscanslowdownexecutionsignificantly.
124
VMware, Inc.
Chapter 9 Advanced Resource Management
Virtualization and Processor-Specific Behavior

(SEEUPDATE)BecauseVMwaresoftwarevirtualizestheCPU,thevirtualmachineis awareofthespecificmodeloftheprocessoronwhichitisrunning.Someoperating systemsinstalldifferentkernelversionstunedforspecificprocessormodels,andthese kernelsareinstalledinvirtualmachinesaswell.Becauseofthedifferentkernel versions,itisnotpossibletomigratevirtualmachinesinstalledonasystemrunning oneprocessormodel(forexample,AMD)toasystemrunningonadifferentprocessor (forexample,Intel).
Performance Implications
CPUvirtualizationaddsvaryingamountsofoverheaddependingontheworkloadand thetypeofvirtualizationused. AnapplicationisCPUboundifmostoftheapplicationstimeisspentexecuting instructionsratherthanwaitingforexternaleventssuchasuserinteraction,device input,ordataretrieval.Forsuchapplications,theCPUvirtualizationoverheadrequires additionalinstructionstobeexecuted,whichtakesCPUprocessingtimethatcouldbe usedbytheapplicationitself.CPUvirtualizationoverheadusuallytranslatesintoa reductioninoverallperformance. ForapplicationsthatarenotCPUbound,CPUvirtualizationlikelytranslatesintoan increaseinCPUutilization.IfspareCPUcapacityisavailabletoabsorbtheoverhead, itcanstilldelivercomparableperformanceintermsofoverallthroughput. ESXServer3supportsuptofourvirtualprocessors(CPUs)foreachvirtualmachine. NOTEDeploysinglethreadedapplicationsonuniprocessorvirtualmachines(instead ofSMPvirtualmachines)forbestperformanceandresourceutilization. SinglethreadedapplicationscantakeadvantageonlyofasingleCPU.Deployingsuch applicationsindualprocessorvirtualmachinesdoesnotspeeduptheapplication. Instead,itcausesthesecondvirtualCPUtousephysicalresourcesthatcouldotherwise beusedbyothervirtualmachines.
VMware, Inc.
125
Using CPU Affinity to Assign Virtual Machines to Specific Processors

Affinitymeansthatyoucanrestricttheassignmentofvirtualmachinestoasubsetof theavailableprocessorsinmultiprocessorsystems.Youdosobyspecifyinganaffinity settingforeachvirtualmachine. CAUTIONUsingaffinitymightnotbeappropriate.SeePotentialIssueswithAffinity onpage 127. TheCPUaffinitysettingforavirtualmachineappliesnotonlytoallofthevirtualCPUs associatedwiththevirtualmachine,butalsotoallotherthreads(alsoknownas worlds)associatedwiththevirtualmachine.Suchvirtualmachinethreadsperform processingrequiredforemulatingmouse,keyboard,screen,CDROMand miscellaneouslegacydevices. Insomecases,suchasdisplayintensiveworkloads,significantcommunicationmight occurbetweenthevirtualCPUsandtheseothervirtualmachinethreads.Performance mightdegradeifthevirtualmachinesaffinitysettingpreventstheseadditionalthreads frombeingscheduledconcurrentlywiththevirtualmachinesvirtualCPUs(for example,auniprocessorvirtualmachinewithaffinitytoasingleCPU,oratwoway SMPvirtualmachinewithaffinitytoonlytwoCPUs). Forthebestperformance,whenmanualaffinitysettingsareused,VMware recommendsthatyouincludeatleastoneadditionalphysicalCPUintheaffinity settinginordertoallowatleastoneofthevirtualmachinesthreadstobescheduledat thesametimeasitsvirtualCPUs(forexample,auniprocessorvirtualmachinewith affinitytoatleasttwoCPUsoratwowaySMPvirtualmachinewithaffinitytoatleast threeCPUs). NOTECPUaffinityisdifferentfromDRSaffinity,discussedinCustomizingDRSfor VirtualMachinesonpage 114. To assign a virtual machine to a specific processor 1 2 IntheVIClientinventorypanel,selectavirtualmachineandchooseEditSettings. SelecttheResourcestabandchooseCPU.
126
VMware, Inc.
ClicktheRunonprocessor(s)button.
SelecttheprocessorsonwhichyouwantthevirtualmachinetorunandclickOK.
Potential Issues with Affinity Virtualmachineaffinityassignseachvirtualmachine toprocessorsinthespecifiedaffinityset.Beforeusingaffinity,considerthefollowing issues: Formultiprocessorsystems,ESXServersystemsperformautomaticload balancing.Avoidmanualspecificationofvirtualmachineaffinitytoimprovethe schedulersabilitytobalanceloadacrossprocessors. AffinitycaninterferewiththeESXServerhostsabilitytomeetthereservationand sharesspecifiedforavirtualmachine. BecauseCPUadmissioncontroldoesnotconsideraffinity,avirtualmachinewith manualaffinitysettingsmightnotalwaysreceiveitsfullreservation. Virtualmachinesthatdonothavemanualaffinitysettingsarenotadversely affectedbyvirtualmachineswithmanualaffinitysettings. Whenyoumoveavirtualmachinefromonehosttoanother,affinitymightno longerapplybecausethenewhostmighthaveadifferentnumberofprocessors. TheNUMAschedulermightnotbeabletomanageavirtualmachinethatsalready assignedtocertainprocessorsusingaffinity.SeeChapter 10,UsingNUMA SystemswithESXServer,onpage 151foradditionalinformationonusing NUMAwithESXServerhosts. AffinitycanaffectanESXServerhostsabilitytoschedulevirtualmachineson multicoreorhyperthreadedprocessorstotakefulladvantageofresourcesshared onsuchprocessors.
VMware, Inc.
127
Multicore Processors
IntelandAMDhaveeachdevelopedprocessorswhichcombinetwoormoreprocessor coresintoasingleintegratedcircuit(oftencalledapackageorsocket).VMwareusesthe termphysicalprocessororsockettodescribeasinglepackagewhichcanhaveoneormore processorcoreswithoneormorelogicalprocessorsinsideeachcore.Multicore processorsprovidemanyadvantagesforanESXServerhostperformingmultitasking ofvirtualmachines. Adualcoreprocessor,forexample,canprovidealmostdoubledperformance, comparedwithasinglecoreprocessor,byallowingtwovirtualmachinestobe executedatthesametime.Eachcorecanhaveitsownmemorycaches,orcanshare someofitscacheswithothercores,potentiallyreducingtherateofcachemissesand theneedtoaccessslowermainmemory.Asharedmemorybusthatconnectsaphysical processortomainmemorycanlimitperformanceofitslogicalprocessorsifthevirtual machinesrunningonthemarerunningmemoryintensiveworkloadswhichcompete forthesamememorybusresources. EachlogicalprocessorofeachprocessorcorecanbeusedindependentlybytheESX ServerCPUschedulertoexecutevirtualmachines,providingcapabilitiessimilarto traditionalsymmetricmultiprocessing(SMP)systems.Forexample,atwowayvirtual machinecanhaveitsvirtualprocessorsrunningonlogicalprocessorsthatbelongtothe samecore,oronlogicalprocessorsondifferentphysicalprocessors. Table 91providesalistofprocessorsandtheirattributes.
NOTEOnprocessorswithIntelHyperThreadingtechnology,eachcorecanhavetwo logicalprocessorswhichsharemostofthecoresresources,suchasmemorycaches andexecutionpipelines.Suchlogicalprocessorsareusuallycalledthreads. Table 9-1. Processors and Core Attributes

Processor IntelPentiumIII IntelPentium4 (HTdisabled) IntelPentium4 (HTenabled) IntelPentiumD940 IntelPentiumEE 840(HTenabled) IntelCore2Duo
128
Cores 1 1 1 2 2 2
Threads/Core 1 1 2 1 2 1
Logical Processors 1 1 2 2 4 2
VMware, Inc.
Table 9-1. Processors and Core Attributes (Continued)

Processor IntelCore2Quad AMDAthlon64 AMDAthlon64X2 AMDOpteron AMDOpteron DualCore Cores 4 1 2 1 2 Threads/Core 1 1 1 1 1 Logical Processors 4 1 2 1 2
AnESXServerCPUschedulerisawareoftheprocessortopologyandrelationships betweenprocessorcoresandthelogicalprocessorsonthem.Itusesthisknowledgeto schedulevirtualmachinesandoptimizeperformance.
Hyperthreading
IntelCorporationdevelopedhyperthreadingtechnologytoenhancetheperformanceof itsPentiumIVandXeonprocessorlines.Thetechnologyallowsasingleprocessorcore toexecutetwoindependentthreadssimultaneously.Whilethisfeaturedoesnot providetheperformanceofatruedualprocessorsystem,itcanimproveutilizationof onchipresources,leadingtogreaterthroughputforcertainimportantworkloadtypes. SeetheIntelWebsiteforanindepthdiscussionofhyperthreadingtechnology. Foradditionalinformation,seethewhitepaperHyperThreadingSupportinESX Server2,whichisavailableattheVMwareWebsite. Hyperthreadingtechnologyallowsasinglephysicalprocessorcoretobehaveliketwo logicalprocessors.Theprocessorcanruntwoindependentapplicationsatthesame time.Toavoidconfusionbetweenlogicalandphysicalprocessors,Intelreferstoa physicalprocessorasasocket,andthediscussioninthischapterusesthatterminology aswell. Whilehyperthreadingdoesnotdoubletheperformanceofasystem,itcanincrease performancebybetterutilizingidleresources.Anapplicationrunningononelogical processorofabusycorecanexpectslightlymorethanhalfofthethroughputthatit obtainswhilerunningaloneonanonhyperthreadedprocessor.However, hyperthreadingperformanceimprovementsarehighlyapplicationdependent,and someapplicationsmightseeperformancedegradationwithhyperthreadingbecause manyprocessorresources(suchasthecache)aresharedbetweenbothlogical processors.
VMware, Inc.
129
Enabling Hyperthreading
Bydefault,hyperthreadingisenabled.Ifitisdisabled,youcanenableit. AllIntelXeonMPprocessorsandallIntelXeonDPprocessorswith512KL2cache supporthyperthreading;however,noteveryIntelXeonsystemshipswithaBIOSthat supportshyperthreading.ConsultyoursystemdocumentationtoseeiftheBIOS includessupportforhyperthreading.VMwareESXServercannotenable hyperthreadingonasystemwithmorethan16physicalCPUs,becauseESXServerhas alogicallimitof32CPUs. To enable hyperthreading 1 2 Ensurethatyoursystemsupportshyperthreadingtechnology. EnablehyperthreadinginthesystemBIOS. SomemanufacturerslabelthisoptionLogicalProcessorwhileotherscallitEnable Hyperthreading. MakesurehyperthreadingforyourESXServerhostisturnedon. a b c IntheVIClient,selectthehostandclicktheConfigurationtab. SelectProcessorsandclickProperties. Inthedialogbox,youcanviewhyperthreadingstatusandturn hyperthreadingofforon(default).
Hyperthreading and ESX Server

AnESXServersystemenabledforhyperthreadingshouldbehavealmostexactlylikea standardsystem.LogicalprocessorsonthesamecorehaveadjacentCPUnumbers,so thatCPUs0and1areonthefirstcoretogether,CPUs2and3areonthesecondcore, andsoon.
130
VMware, Inc.
VMwareESXServersystemsmanageprocessortimeintelligentlytoguaranteethat loadisspreadsmoothlyacrossprocessorcoresinthesystem.Virtualmachinesare preferentiallyscheduledontwodifferentcoresratherthanontwologicalprocessorson thesamecore. Ifthereisnoworkforalogicalprocessor,itisputintoahaltedstate,whichfreesits executionresourcesandallowsthevirtualmachinerunningontheotherlogical processoronthesamecoretousethefullexecutionresourcesofthecore.TheVMware schedulerproperlyaccountsforthishalttime,sothatavirtualmachinerunningwith thefullresourcesofacoreischargedmorethanavirtualmachinerunningonahalf core.Thisapproachtoprocessormanagementensuresthattheserverdoesnotviolate anyofthestandardESXServerresourceallocationrules.
Advanced Server Configuration for Hyperthreading

YoucanspecifyhowthevirtualCPUsofavirtualmachinecansharephysicalcoreson ahyperthreadedsystem.TwovirtualCPUsshareacoreiftheyarebothrunningon logicalCPUsofthecoreatthesametime.Youcansetthisforindividualvirtual machines. To set hyperthreading sharing options for a virtual machine 1 2 3 IntheVIClientinventorypanel,rightclickthevirtualmachineandchooseEdit Settings. ClicktheResourcestab,andclickAdvancedCPU. ChoosefromtheModedropdownmenutospecifyhyperthreadingforthisvirtual machine.
VMware, Inc.
131
Youhavethefollowingchoices.
Option Any Description Thedefaultforallvirtualmachinesonahyperthreadedsystem.Thevirtual CPUsofavirtualmachinewiththissettingcanfreelysharecoreswithother virtualCPUsfromthisoranyothervirtualmachineatanytime. VirtualCPUsofavirtualmachineshouldnotsharecoreswitheachotherorwith virtualCPUsfromothervirtualmachines.Thatis,eachvirtualCPUfromthis virtualmachineshouldalwaysgetawholecoretoitself,withtheotherlogical CPUonthatcorebeingplacedintothehaltedstate. Thisoptionissimilartonone.VirtualCPUsfromthisvirtualmachinearenot allowedtosharecoreswithvirtualCPUsfromothervirtualmachines.Theycan sharecoreswiththeothervirtualCPUsfromthesamevirtualmachine. ThisoptionispermittedonlyforSMPvirtualmachines.Ifappliedtoa uniprocessorvirtualmachine,thesystemchangesthisoptiontonone.
None
Internal
TheseoptionshavenoeffectonfairnessorCPUtimeallocation.Regardlessofavirtual machineshyperthreadingsettings,itstillreceivesCPUtimeproportionaltoitsCPU shares,andconstrainedbyitsCPUreservationandCPUlimitvalues. Fortypicalworkloads,customhyperthreadingsettingsshouldnotbenecessary.The optionscanhelpincaseofunusualworkloadsthatinteractbadlywithhyperthreading. Forexample,anapplicationwithcachethrashingproblemsmightslowdownan applicationsharingitsphysicalcore.Youcanplacethevirtualmachinerunningthe applicationinthenoneorinternalhyperthreadingstatustoisolateitfromothervirtual machines. IfavirtualCPUhashyperthreadingconstraintsthatdonotallowittoshareacorewith anothervirtualCPU,thesystemmightdescheduleitwhenothervirtualCPUsare entitledtoconsumeprocessortime.Withoutthehyperthreadingconstraints,both virtualCPUscouldhavebeenscheduledonthesamecore. Theproblembecomesworseonsystemswithalimitednumberofcores(pervirtual machine).Insuchcases,theremightbenocoretowhichthevirtualmachinethatis descheduledcanbemigrated.Asaresult,itispossiblethatvirtualmachineswith hyperthreadingsettononeorinternalcanexperienceperformancedegradation, especiallyonsystemswithalimitednumberofcores.
132
VMware, Inc.
Quarantining
Incertain,rarecircumstances,anESXServersystemmightdetectthatanapplicationis interactingbadlywithhyperthreadingtechnology.Certaintypesofselfmodifying code,forexample,candisruptthenormalbehaviorofthePentiumIVtracecacheand leadtosubstantialslowdowns(upto90percent)foranapplicationsharingacorewith theproblematiccode.Inthosecases,theESXServerhostquarantinesthevirtualCPU runningthiscodeandplacesitsvirtualmachineinthenoneorinternalmode,as appropriate.Quarantiningisnecessaryonlyrarelyandistransparenttotheuser. SettheCpu.MachineClearThresholdadvancedsettingforthehostto0todisable quarantining.SeeSettingAdvancedHostAttributesonpage 145.
Hyperthreading and CPU Affinity

ConsideryoursituationbeforeyousetCPUaffinityonsystemsusinghyperthreading. Forexample,ifahighpriorityvirtualmachineisboundtoCPU0andanotherhigh priorityvirtualmachineisboundtoCPU1,thetwovirtualmachineshavetosharethe samephysicalcore.Inthiscase,itcanbeimpossibletomeettheresourcedemandsof thesevirtualmachines.Makesureanycustomaffinitysettingsmakesensefora hyperthreadedsystem.Inthisexample,bindingthevirtualmachinestoCPU0and CPU2,youshouldnotuseaffinitysettingsatall.SeeUsingCPUAffinitytoAssign VirtualMachinestoSpecificProcessorsonpage 126.
Memory Virtualization
Allmodernoperatingsystemsprovidesupportforvirtualmemory,allowingsoftware tousemorememorythanthemachinephysicallyhas.Thevirtualmemoryspaceis dividedintoblocks,typically4KB,calledpages.Thephysicalmemoryisalsodivided intoblocks,alsotypically4KB.Whenphysicalmemoryisfull,thedataforvirtualpages thatarenotpresentinphysicalmemoryarestoredondisk.
Software Memory Virtualization

ESXServervirtualizesguestphysicalmemorybyaddinganextralevelofaddress translation. TheVMMforeachvirtualmachinemaintainsamappingfromtheguestoperating systemsphysicalmemorypagestothephysicalmemorypagesontheunderlying machine.(VMwarereferstotheunderlyingphysicalpagesasmachinepagesand theguestoperatingsystemsphysicalpagesasphysicalpages.)
VMware, Inc.
133
Eachvirtualmachineseesacontiguous,zerobased,addressablephysicalmemory space.Theunderlyingmachinememoryontheserverusedbyeachvirtual machineisnotnecessarilycontiguous. TheVMMinterceptsvirtualmachineinstructionsthatmanipulateguestoperating systemmemorymanagementstructuressothattheactualmemorymanagement unit(MMU)ontheprocessorisnotupdateddirectlybythevirtualmachine. TheESXServerhostmaintainsthevirtualtomachinepagemappingsinashadow pagetablethatiskeptuptodatewiththephysicaltomachinemappings (maintainedbytheVMM,seeabove). Theshadowpagetablesareuseddirectlybytheprocessorspaginghardware. Thisapproachtoaddresstranslationallowsnormalmemoryaccessesinthevirtual machinetoexecutewithoutaddingaddresstranslationoverhead,aftertheshadow pagetablesaresetup.Becausethetranslationlookasidebuffer(TLB)ontheprocessor cachesdirectvirtualtomachinemappingsreadfromtheshadowpagetables,no additionaloverheadisaddedbytheVMMtoaccessthememory.
Hardware-Assisted Memory Virtualization

SomeCPUs,suchasAMDSVMV,providehardwaresupportformemory virtualizationbyusingtwolayersofpagetables.Thefirstlayerofpagetablesstores guestvirtualtophysicaltranslations,whilethesecondlayerofpagetablesstoresguest physicaltomachinetranslation.OnaTLBmisstoacertainguestvirtualaddress,the hardwarelooksatbothpagetablestotranslateguestvirtualaddresstohostphysical address. ThediagraminFigure 91illustratestheESXServerimplementationofmemory virtualization. Figure 9-1. ESX Server Memory Mapping
virtual machine 1 virtual machine 2
a b a b
c b b c
guest virtual memory guest physical memory
b b
machine memory
Theboxesrepresentpages,andthearrowsshowthedifferentmemorymappings.
134
VMware, Inc.
Thearrowsfromguestvirtualmemorytoguestphysicalmemoryshowthe mappingmaintainedbythepagetablesintheguestoperatingsystem.(The mappingfromvirtualmemorytolinearmemoryforx86architectureprocessorsis notshown.) Thearrowsfromguestphysicalmemorytomachinememoryshowthemapping maintainedbytheVMM. Thedashedarrowsshowthemappingfromguestvirtualmemorytomachine memoryintheshadowpagetablesalsomaintainedbytheVMM.Theunderlying processorrunningthevirtualmachineusestheshadowpagetablemappings. Becauseoftheextralevelofmemorymappingintroducedbyvirtualization,ESXServer canefficientlymanagememoryacrossallvirtualmachines.Someofthephysical memoryofavirtualmachinemightbemappedtosharedpagesortopagesthatare unmapped,orswappedout. AnESXServerhostperformsvirtualmemorymanagementwithouttheknowledgeof theguestoperatingsystemandwithoutinterferingwiththeguestoperatingsystems ownmemorymanagementsubsystem.
Performance Implications
Thissectiondiscussestheperformanceimplicationsofbothsoftwarebasedand hardwareassistedmemoryvirtualization.
For Software Memory Virtualization

Theuseoftwopagecoordinatedpagetableshastheseperformanceimplications: Nooverheadisincurredforregularguestmemoryaccesses. Additionaltimeisrequiredtomapmemorywithinavirtualmachine,whichmight mean: Thevirtualmachineoperatingsystemissettinguporupdatingvirtual addresstophysicaladdressmappings. Thevirtualmachineoperatingsystemisswitchingfromoneaddressspaceto another(contextswitch). LikeCPUvirtualization,memoryvirtualizationoverheaddependsonworkload.
VMware, Inc.
135
For Hardware-Assisted Memory Virtualization

Theoverheadforsoftwarememoryvirtualizationiseliminatedwhenyouuse hardwareassistance.Inparticular,hardwareassistanceeliminatestheoverhead requiredtokeepshadowpagetablesinsynchronizationwithguestpagetables. However,theTLBmisslatencywhenusinghardwareassistanceissignificantlyhigher. Asaresult,whetherornotaworkloadbenefitsbyusinghardwareassistanceprimarily dependsontheoverheadthememoryvirtualizationcauseswhenusingsoftware memoryvirtualization.Ifaworkloadinvolvesasmallamountofpagetableactivity (suchasprocesscreation,mappingthememory,orcontextswitches),thensoftware virtualizationdoesnotcausesignificantoverhead.Ontheotherhand,workloadswith alargeamountofpagetableactivityarelikelytobenefitfromhardwareassistance.
Understanding Memory Overhead

ESXServervirtualmachinescanincurtwokindsofmemoryoverhead: Theadditionaltimetoaccessmemorywithinavirtualmachine. TheextraspaceneededbytheESXServerhostforitsowncodeanddatastructures, beyondthememoryallocatedtoeachvirtualmachine. ESXServermemoryvirtualizationaddslittletimeoverheadtomemoryaccesses. Becausetheprocessorspaginghardwareusestheshadowpagetablesdirectly,most memoryaccessesinthevirtualmachinecanexecutewithoutaddresstranslation overhead. Forexample,ifapagefaultoccursinthevirtualmachine,controlswitchestotheVMM sothattheVMMcanupdateitsdatastructures. Thememoryspaceoverheadhastwocomponents: AfixedsystemwideoverheadfortheVMkerneland(forESXServer3only)the serviceconsole. Additionaloverheadforeachvirtualmachine. ForESXServer3,theserviceconsoletypicallyuses272MBandtheVMkernelusesa smalleramountofmemory.Theamountdependsonthenumberandsizeofthedevice driversthatarebeingused.SeeViewingHostResourceInformationonpage 14for informationonhowtodeterminetheavailablememoryforahost. Overheadmemoryincludesspacereservedforthevirtualmachineframebufferand variousvirtualizationdatastructures.Overheadmemorydependsonthenumberof virtualCPUs,theconfiguredmemoryfortheguestoperatingsystem,andonwhether youareusinga32bitor64bitguestoperatingsystem.Table 92liststheoverheadfor eachcase.
136 VMware, Inc.
Table 9-2. Overhead Memory on Virtual Machines

Virtual CPUs 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 4 4 4 4 4 4 4 4 4 Memory (MB) 256 512 1,024 2,048 4,096 8,192 16,384 32,768 65,536 256 512 1,024 2,048 4,096 8,192 16,384 32,768 65,536 256 512 1,024 2,048 4,096 8,192 16,384 32,768 65,536 Overhead for 32-Bit Virtual Machine (MB) 87.56 90.82 97.35 110.40 136.50 188.69 293.07 501.84 919.37 108.73 114.49 126.04 149.11 195.27 287.57 472.18 841.40 1,579.84 146.75 153.52 167.09 194.20 248.45 356.91 573.85 1,007.73 1,875.48 Overhead for 64-Bit Virtual Machine (MB) 107.54 110.81 117.35 130.42 156.57 208.85 313.42 522.56 940.84 146.41 152.20 163.79 186.96 233.30 325.98 511.34 882.06 1,623.50 219.82 226.64 240.30 267.61 322.22 431.44 649.88 1,086.75 1,960.52
VMware, Inc.
137
ESXServeralsoprovidesoptimizationssuchasmemorysharing(seeSharingMemory AcrossVirtualMachinesonpage 144)toreducetheamountofphysicalmemoryused ontheunderlyingserver.Theseoptimizationscansavemorememorythanistakenup bytheoverhead.
Memory Allocation and Idle Memory Tax

ThissectiondiscusseshowanESXServerhostallocatesmemoryandhowyoucanuse theMem.IdleTaxconfigurationparametertochangehowanESXServerhostreclaims idlememory.
How ESX Server Hosts Allocate Memory

AnESXServerhostallocatesthememoryspecifiedbytheLimitparametertoeach virtualmachineunlessmemoryisovercommitted.AnESXServerhostneverallocates morememorytoavirtualmachinethanitsspecifiedphysicalmemorysize.For example,a1GBvirtualmachinemighthavethedefaultlimit(unlimited)ora userspecifiedlimit(forexample2GB).Inbothcases,theESXServerhostnever allocatesmorethan1GB,thephysicalmemorysizethatwasspecifiedforit. Whenmemoryisovercommitted,eachvirtualmachineisallocatedanamountof memorysomewherebetweenwhatisspecifiedbyReservationandwhatisspecifiedby Limit(seeMemoryOvercommitmentonpage 42).Theamountofmemorygranted toavirtualmachineaboveitsreservationusuallyvarieswiththecurrentmemoryload. AnESXServerhostdeterminesallocationsforeachvirtualmachinebasedonthe numberofsharesallocatedtoitandanestimateofitsrecentworkingsetsize. SharesESXServerhostsuseamodifiedproportionalsharememoryallocation policy.Memorysharesentitleavirtualmachinetoafractionofavailablephysical memory.SeeSharesonpage 20. WorkingsetsizeESXServerhostsestimatetheworkingsetforavirtual machinebymonitoringmemoryactivityoversuccessiveperiodsofvirtual machineexecutiontime.Estimatesaresmoothedoverseveraltimeperiodsusing techniquesthatrespondrapidlytoincreasesinworkingsetsizeandmoreslowly todecreasesinworkingsetsize. Thisapproachensuresthatavirtualmachinefromwhichidlememoryhasbeen reclaimedcanrampupquicklytoitsfullsharebasedallocationwhenitstarts usingitsmemorymoreactively.
138
VMware, Inc.
Modifythedefaultmonitoringperiodof60secondsbyadjustingthe Mem.SamplePeriodadvancedsetting.Mem.SamplePeriodspecifiestheperiodic timeinterval,measuredinsecondsofvirtualmachineexecutiontime,overwhich memoryactivityismonitoredtoestimateworkingsetsizes.SeeSettingAdvanced HostAttributesonpage 145.
How Host Memory Is Used

YoucanusetheVIClienttoseehowhostmemoryisused. To view information about physical memory usage 1 2 IntheVIClient,selectahostandclicktheConfigurationtab. ClickMemory.
Thefollowinginformationappears,asdiscussedinTable 93.
Table 9-3. Host Memory Information

Field Total System Description Totalphysicalmemoryforthishost. MemoryusedbytheESXServersystem. ESXServer3.xusesatleast50MBofsystemmemoryfortheVMkernel, plusadditionalmemoryfordevicedrivers.Thismemoryisallocated whentheESXServerisloadedandisnotconfigurable. Theactualrequiredmemoryforthevirtualizationlayerdependsonthe numberandtypeofPCI(peripheralcomponentinterconnect)deviceson ahost.Somedriversneed40MB,whichalmostdoublesbasesystem memory. TheESXServerhostalsoattemptstokeepsomememoryfreeatalltimes tohandledynamicallocationrequestsefficiently.ESXServersetsthislevel atapproximatelysixpercentofthememoryavailableforrunningvirtual machines.
VMware, Inc.
139
Table 9-3. Host Memory Information (Continued)

Field VirtualMachines Description Memoryusedbyvirtualmachinesrunningontheselectedhost. Mostofthehostsmemoryisusedforrunningvirtualmachines.AnESX Serverhostmanagestheallocationofthismemorytovirtualmachines basedonadministrativeparametersandsystemload. ServiceConsole Memoryreservedfortheserviceconsole. ClickPropertiestochangehowmuchmemoryisavailablefortheservice console.ThisfieldappearsonlyinESXServer3.ESXServer3idoesnot provideaserviceconsole.
Memory Tax for Idle Virtual Machines

Ifavirtualmachineisnotactivelyusingitscurrentlyallocatedmemory,theESXServer chargesmoreforidlememorythanformemorythatisinuse.(ESXServerneveralters userspecifiedshareallocations,butmemorytaxhasasimilareffect.) Memorytaxhelpspreventvirtualmachinesfromhoardingidlememory.Thedefault taxrateis75percent,thatis,anidlepagecostsasmuchasfouractivepages. TheMem.IdleTaxadvancedsettingallowsyoutocontrolthepolicyforreclaimingidle memory.Usethisoption,togetherwiththeMem.SamplePeriodadvancedattribute,to controlhowthesystemreclaimsmemory.SeeSettingAdvancedHostAttributeson page 145. NOTEInmostcases,changestoMem.IdleTaxarenotnecessaryorevenappropriate.
How ESX Server Hosts Reclaim Memory

ThissectiongivesbackgroundinformationonhowESXServerhostsreclaimmemory fromvirtualmachines.Thehostsusetwotechniquesfordynamicallyexpandingor contractingtheamountofmemoryallocatedtovirtualmachines: ESXServersystemsuseamemoryballoondriver(vmmemctl),loadedintothe guestoperatingsystemrunninginavirtualmachine.SeeMemoryBalloon (vmmemctl)Driver. ESXServersystemspagefromavirtualmachinetoaserverswapfilewithoutany involvementbytheguestoperatingsystem.Eachvirtualmachinehasitsownswap file.SeeSwappingonpage 142.
140
VMware, Inc.
Memory Balloon (vmmemctl) Driver

Thevmmemctldrivercollaborateswiththeservertoreclaimpagesthatareconsidered leastvaluablebytheguestoperatingsystem.Thedriverusesaproprietaryballooning techniquethatprovidespredictableperformancewhichcloselymatchesthebehavior ofanativesystemundersimilarmemoryconstraints.Thistechniqueincreasesor decreasesmemorypressureontheguestoperatingsystem,causingtheguesttocallits ownnativememorymanagementalgorithms.Whenmemoryistight,theguest operatingsystemdetermineswhichpagestoreclaimand,ifnecessary,swapsthemto itsownvirtualdisk.SeeFigure 92. Figure 9-2. Memory Ballooning in the Guest Operating System
memory
2
swap space
memory
3
swap space
memory
NOTEYoumustconfiguretheguestoperatingsystemwithsufficientswapspace. Someguestoperatingsystemshaveadditionallimitations.SeeSwapSpaceandGuest OperatingSystemsonpage 142. Ifnecessary,youcanlimittheamountofmemoryvmmemctlreclaimsbysettingthe sched.mem.maxmemctlparameterforaspecificvirtualmachine.Thisoptionspecifies themaximumamountofmemorythatcanbereclaimedfromavirtualmachinein megabytes(MB).SeeSettingAdvancedVirtualMachineAttributesonpage 149.
VMware, Inc.
141
Swap Space and Guest Operating Systems

IfyouchoosetoovercommitmemorywithESXServer,youneedtobesureyourguest operatingsystemshavesufficientswapspace.Thisswapspacemustbegreaterthanor equaltothedifferencebetweenthevirtualmachinesconfiguredmemorysizeandits Reservation. CAUTIONIfmemoryisovercommitted,andtheguestoperatingsystemisconfigured withinsufficientswapspace,theguestoperatingsysteminthevirtualmachinecanfail. Topreventvirtualmachinefailure,increasethesizeoftheswapspaceinyourvirtual machines: WindowsguestoperatingsystemsWindowsoperatingsystemsrefertotheir swapspaceaspagingfiles.SomeWindowsoperatingsystemstrytoincreasethe sizeofpagingfiles,ifthereissufficientfreediskspace. RefertoyourMicrosoftWindowsdocumentationorsearchtheWindowshelpfiles forpagingfiles.Followtheinstructionsforchangingthesizeofthevirtual memorypagingfile. LinuxguestoperatingsystemLinuxoperatingsystemsrefertotheirswap spaceasswapfiles.Forinformationonincreasingswapfiles,refertothefollowing Linuxmanpages: mkswapSetsupaLinuxswaparea. swaponEnablesdevicesandfilesforpagingandswapping. Guestoperatingsystemswithalotofmemoryandsmallvirtualdisks(forexample,a virtualmachinewith8GBRAManda2GBvirtualdisk)aremoresusceptibletohaving insufficientswapspace.
Swapping
AswapfileiscreatedbytheESXServerhostwhenavirtualmachineispoweredon.If thisfilecannotbecreated,thevirtualmachinecannotpoweron.Bydefault,theswap fileiscreatedinthesamelocationasthevirtualmachinesconfigurationfile.Insteadof acceptingthisdefault,youcanalso: Usepervirtualmachineconfigurationoptionstochangethedatastoretoanother sharedstoragelocation.SeeSettingAdvancedVirtualMachineAttributeson page 149andTable 97.
142
VMware, Inc.
Usehostlocalswap,whichallowsyoutospecifyadatastorestoredlocallyonthe host.Thisallowsyoutoswapataperhostlevel,savingspaceontheSAN. However,itcanleadtoaslightdegradationinperformanceforVMotion. To enable host-local swap for a cluster 1 2 3 4 5 6 7 RightclicktheclusterintheVIClientinventorypanelandclickEditSettings. IntheleftpaneoftheclusterSettingsdialogboxthatappears,clickSwapfile Location. SelecttheStoretheswapfileinthedatastorespecifiedbythehostoptionand clickOK. SelectoneoftheclustershostsintheVIClientinventorypanelandclickthe Configurationtab. SelectVirtualMachineSwapfileLocation. ClicktheSwapfileDatastoretabandfromthelistprovidedselectthelocal datastoretouseandclickOK. RepeatStep 4throughStep 6foreachhostinthecluster.
To enable host-local swap for a standalone host 1 2 3 4 SelectthehostintheVIClientinventorypanelandclicktheConfigurationtab. SelectVirtualMachineSwapfileLocation. UndertheSwapfilelocationtaboftheVirtualMachineSwapfileLocationdialog boxthatappears,selecttheStoretheswapfileintheswapfiledatastoreoption. ClicktheSwapfileDatastoretab,fromthelistprovidedselectthelocaldatastore touseandclickOK.
ESXServerhostsuseswappingtoforciblyreclaimmemoryfromavirtualmachine whennovmmemctldriverisavailablebecausethevmmemctldriver: Wasneverinstalled Hasbeenexplicitlydisabled Isnotrunning(forexample,whiletheguestoperatingsystemisbooting) Istemporarilyunabletoreclaimmemoryquicklyenoughtosatisfycurrentsystem demands Isfunctioningproperly,butmaximumballoonsizehasbeenreached.
VMware, Inc.
143
Standarddemandpagingtechniquesswappagesbackinwhenthevirtualmachine needsthem. NOTEForoptimumperformance,ESXServerhostsusetheballooningapproach (implementedbythevmmemctldriver)wheneverpossible.Swappingisareliable mechanismoflastresortthatahostusesonlywhennecessarytoreclaimmemory.
Swap Space and Memory Overcommitment

SwapspacemustbereservedonanESXServerhostforanyunreservedvirtualmachine memory,whichisthedifferencebetweenthereservationandtheconfiguredmemory size.Thisswapreservationisrequiredtoensurethatthesystemisabletopreserve virtualmachinememoryunderanycircumstances.Inpractice,onlyasmallfractionof theswapspacemightbeused. Similarly,whilememoryreservationsareusedforadmissioncontrol,actualmemory allocationsvarydynamically,andunusedreservationsarenotwasted.
Swap Files and ESX Server Failure

IfanESXServersystemfails,andthatsystemhadrunningvirtualmachinesthatwere usingswapfiles,thoseswapfilescontinuetoexistandtakeupdiskspaceevenafterthe ESXServersystemrestarts. To delete swap files 1 2 Startthevirtualmachineagain. Stopthevirtualmachineexplicitly.
NOTETheseswapfilescanconsumemanygigabytesofdiskspacesoensurethatyou deletethemproperly.
Sharing Memory Across Virtual Machines

ManyESXServerworkloadspresentopportunitiesforsharingmemoryacrossvirtual machines.Forexample,severalvirtualmachinesmightberunninginstancesofthe sameguestoperatingsystem,havethesameapplicationsorcomponentsloaded,or containcommondata.Insuchcases,anESXServerhostusesaproprietarytransparent pagesharingtechniquetosecurelyeliminateredundantcopiesofmemorypages.With memorysharing,aworkloadrunninginvirtualmachinesoftenconsumeslessmemory thanitwouldwhenrunningonphysicalmachines.Asaresult,higherlevelsof overcommitmentcanbesupportedefficiently.
144
VMware, Inc.
UsetheMem.ShareScanTimeandMem.ShareScanGHzadvancedsettingstocontrol therateatwhichthesystemscansmemorytoidentifyopportunitiesforsharing memory. Youcanalsodisablesharingforindividualvirtualmachinesbysettingthe sched.mem.pshare.enableoptiontoFALSE(thisoptiondefaultstoTRUE).See SettingAdvancedVirtualMachineAttributesonpage 149. ESXServermemorysharingrunsasabackgroundactivitythatscansforsharing opportunitiesovertime.Theamountofmemorysavedvariesovertime.Forafairly constantworkload,theamountgenerallyincreasesslowlyuntilallsharing opportunitiesareexploited. Todeterminetheeffectivenessofmemorysharingforagivenworkload,tryrunningthe workload,anduseresxtoporesxtoptoobservetheactualsavings.Findthe informationinthePSHAREfieldoftheinteractivemodeintheMemorypage.SeeUsing theUtilitiesinInteractiveModeonpage 173.
Advanced Attributes and What They Do

Thissectionliststheadvancedattributesavailableforcustomizingmemory management. CAUTIONUsingtheseadvancedattributesisappropriateonlyunderspecial circumstances.Inmostcases,changingthebasicsettings(Reservation,Limit,Shares) orusingthedefaultsettingsresultsinanappropriateallocation.
Setting Advanced Host Attributes

(SEEUPDATE)Thissectionguidesyouthroughsettingadvancedattributesforahostand listsafewattributesyoumightwanttosetundercertaincircumstances. To set advanced attributes for a host 1 2 3 4 IntheVIClientinventorypanel,selectthevirtualmachineyouwanttocustomize. ChooseEditSettingsintheCommandspanelandselecttheOptionstab. SelectAdvanced,andclicktheConfigurationParametersbutton. ClickAdvancedSettings.
VMware, Inc.
145
IntheAdvancedSettingsdialogboxselecttheappropriateitem(forexample,CPU orMemory),andscrollintherightpaneltofindandchangetheattribute.
Table 95,Table 96,andTable 97listtheadvancedresourcemanagementattributes discussedinthisdocument. CAUTIONSettingtheseattributesisrecommendedonlyforadvanceduserswith experienceusingESXServerhosts.Inmostcases,thedefaultsettingsproducethe optimumresult. Table 9-4. Advanced CPU Attributes
Attribute CPU.MachineClearThreshold Description Ifyouareusingahostenabledforhyperthreadingandset thisattributeto0,quarantiningisdisabled.See Quarantiningonpage 133.
146
VMware, Inc.
Table 9-5. Advanced Memory Attributes

Attribute Mem.CtlMaxPercent Description Limitsthemaximumamountofmemorythatcanbe reclaimedfromanyvirtualmachineusing vmmemctl,basedonapercentageofitsconfigured memorysize.Specifying0disablesreclamationvia vmmemctlforallvirtualmachines. Specifiesthetime,inminutes,withinwhichanentire virtualmachineistobescannedforpagesharing opportunities.Defaultsto60minutes. Specifiesthemaximumamountofmemorypagesto scan(persecond)forpagesharingopportunitiesfor eachGHzofavailablehostCPUresource. Defaultsto4MB/secper1GHz Mem.IdleTax Specifiestheidlememorytaxrate,asapercentage. Thistaxeffectivelychargesvirtualmachinesmore foridlememorythanformemorytheyareactively using.Ataxrateof0percentdefinesanallocation policythatignoresworkingsetsandallocates memorystrictlybasedonshares.Ahightaxrate resultsinanallocationpolicythatallowsidle memorytobereallocatedawayfromvirtual machinesthatareunproductivelyhoardingit. Specifiestheperiodictimeinterval,measuredin secondsofthevirtualmachinesexecutiontime,over whichmemoryactivityismonitoredtoestimate workingsetsizes. Specifiestheperiodictimeinterval,inseconds,for automaticmemoryreallocations.Reallocationsare alsotriggeredbysignificantchangesintheamount offreememory. Setthisoptionto1toenablebackingofguestlarge pageswithhostlargepages.ReducesTLBmisses andimprovesperformanceinserverworkloadsthat useguestlargepages. Settheseoptionsto1toreducememory fragmentation.Ifhostmemoryisfragmented,the availabilityofhostlargepagesisreduced.These optionsimprovetheprobabilityofbackingguest largepageswithhostlargepages. 75 Default 65
Mem.ShareScanTime
60
Mem.ShareScanGHz
Mem.SamplePeriod
60
Mem.BalancePeriod
15
Mem.AllocGuestLargePage
Mem.AllocUsePSharePool and Mem.AllocUseGuestPool
VMware, Inc.
147
Table 9-6. Advanced NUMA Attributes

Attribute Numa.RebalanceEnable Description Setthisoptionto0todisableallNUMA rebalancingandinitialplacementofvirtual machines,effectivelydisablingtheNUMA schedulingsystem. Ifthisoptionissetto0,thesystemdoesnot automaticallymigratepagesbetweennodesto improvememorylocality.Pagemigrationrates setmanuallyarestillineffect. Ifthisoptionissetto0,thesystemdoesnot automaticallysetmemoryaffinityforvirtual machineswithCPUaffinityset. TheNUMArebalancercomputestheCPU imbalancebetweennodes,takingintoaccount thedifferencebetweeneachvirtualmachines CPUtimeentitlementanditsactual consumption.Thisoptioncontrolsthe minimumloadimbalancebetweennodes neededtotriggeravirtualmachinemigration, inpercent. Controlsthefrequencyofrebalanceperiods, specifiedinmilliseconds.Morefrequent rebalancingcanincreaseCPUoverheads, particularlyonmachineswithalargenumber ofrunningvirtualmachines.Morefrequent rebalancingcanalsoimprovefairness. Specifiestheminimumnumberoftotal processorcoresonthehostrequiredtoenable theNUMArebalancer. Specifiestheminimumnumberofprocessor corespernoderequiredtoenabletheNUMA rebalancer. ThisoptionandNuma.RebalanceCoresTotal areusefulwhenyouwanttodisableNUMA rebalancingonsmallNUMAconfigurations (forexample,twowayOpteronhosts),where thesmallnumberoftotalorpernode processorscancompromisescheduling fairnesswhenNUMArebalancingisenabled. Default 1
Numa.PageMigEnable
Numa.AutoMemAffinity
Numa.MigImbalanceThreshold
10
Numa.RebalancePeriod
2000
Numa.RebalanceCoresTotal
Numa.RebalanceCoresNode
SeeChapter 10,UsingNUMASystemswithESXServer,onpage 151foradditional information.
148
VMware, Inc.
Setting Advanced Virtual Machine Attributes

Thissectiontakesyouthroughsettingadvancedattributesforavirtualmachine,and liststheattributesyoumightwanttoset. To set advanced attributes for a virtual machine 1 2 3 SelectthevirtualmachineintheVIClientinventorypanel,andchooseEdit Settingsfromtherightclickmenu. ClickOptionsandclickAdvanced>General. ClicktheConfigurationParametersbutton.
Inthedialogboxthatappears,clickAddRowtoenteranewparameterandits value.
VMware, Inc.
149
Setthefollowingadvancedattributesforvirtualmachines. Table 9-7. Advanced Virtual Machine Attributes

Attribute sched.mem.maxmemctl Description Maximumamountofmemorythatcanbereclaimedfromthe selectedvirtualmachinebyballooning,inmegabytes(MB).Ifthe ESXServerhostneedstoreclaimadditionalmemory,itisforced toswap.Swappingislessdesirablethanballooning. Enablesmemorysharingforaselectedvirtualmachine. ThisbooleanvaluedefaultstoTrue.IfyousetittoFalsefora virtualmachine,memorysharingisturnedoff. sched.swap.persist Specifieswhetherthevirtualmachinesswapfilesshouldpersist orbedeletedwhenthevirtualmachineispoweredoff.By default,thesystemcreatestheswapfileforavirtualmachine whenthevirtualmachineispoweredon,anddeletestheswap filewhenthevirtualmachineispoweredoff. VMFSdirectorywherethevirtualmachinesswapfileislocated. Defaultstothevirtualmachinesworkingdirectory,thatis,the VMFSdirectorythatcontainsitsconfigurationfile. Filenameforthevirtualmachinesswapfile.Bydefault,the systemgeneratesauniquenamewhenitcreatestheswapfile.
sched.mem.pshare.enable
sched.swap.dir
sched.swap.file
CAUTIONIfyoumodifythesched.swap.dirattributeforavirtualmachineinaDRS cluster,ensurethateveryhostintheclustercanaccesstheswapfilelocationyouspecify oryoumustdisableDRSforthatvirtualmachine.
150
VMware, Inc.
10
Using NUMA Systems with ESX Server
10
ESXServersupportsmemoryaccessoptimizationforIntelandAMDOpteron processorsinserverarchitecturesthatsupportNUMA(nonuniformmemoryaccess). ThischaptergivesbackgroundinformationonNUMAtechnologiesanddescribes optimizationsavailablewithESXServer. Thischapterdiscussesthefollowingtopics: IntroductiontoNUMAonpage 152 ESXServerNUMASchedulingonpage 153 VMwareNUMAOptimizationAlgorithmsonpage 154 ManualNUMAControlsonpage 156 IBMEnterpriseXArchitectureOverviewonpage 157 AMDOpteronBasedSystemsOverviewonpage 158 ObtainingNUMAConfigurationInformationandStatisticsonpage 159 CPUAffinityforAssociatingVirtualMachineswithaSingleNUMANodeon page 159 MemoryAffinityforAssociatingMemoryAllocationswithaNUMANodeon page 160
VMware, Inc.
151
Introduction to NUMA
NUMAsystemsareadvancedserverplatformswithmorethanonesystembus.They canharnesslargenumbersofprocessorsinasinglesystemimagewithsuperiorprice toperformanceratios.ThesystemsthatofferaNUMAplatformtosupport industrystandardoperatingsystemsincludethosebasedoneitherAMDCPUsorthe IBMEnterpriseXArchitecture.
What Is NUMA?
Forthepastdecade,processorclockspeedhasincreaseddramatically.A multigigahertzCPU,however,needstobesuppliedwithalargeamountofmemory bandwidthtouseitsprocessingpowereffectively.EvenasingleCPUrunninga memoryintensiveworkload,suchasascientificcomputingapplication,canbe constrainedbymemorybandwidth. Thisproblemisamplifiedonsymmetricmultiprocessing(SMP)systems,wheremany processorsmustcompeteforbandwidthonthesamesystembus.Somehighend systemsoftentrytosolvethisproblembybuildingahighspeeddatabus.However, suchasolutionisexpensiveandlimitedinscalability. NUMAisanalternativeapproachthatlinksseveralsmall,costeffectivenodesviaa highperformanceconnection.Eachnodecontainsprocessorsandmemory,muchlike asmallSMPsystem.However,anadvancedmemorycontrollerallowsanodetouse memoryonallothernodes,creatingasinglesystemimage.Whenaprocessoraccesses memorythatdoesnotliewithinitsownnode(remotememory),thedatamustbe transferredovertheNUMAconnection,whichisslowerthanaccessinglocalmemory. Memoryaccesstimesarenotuniformanddependonthelocationofthememoryand thenodefromwhichitisaccessed,asthetechnologysnameimplies.
NUMA Challenges for Operating Systems

BecauseaNUMAarchitectureprovidesasinglesystemimage,itcanoftenrunan operatingsystemwithnospecialoptimizations.Forexample,Windows2000isfully supportedontheIBMx440,althoughitisnotdesignedforusewithNUMA. TherearemanydisadvantagestousingsuchanoperatingsystemonaNUMA platform.Thehighlatencyofremotememoryaccessescanleavetheprocessors underutilized,constantlywaitingfordatatobetransferredtothelocalnode,andthe NUMAconnectioncanbecomeabottleneckforapplicationswithhighmemory bandwidthdemands.
152
VMware, Inc.
Chapter 10 Using NUMA Systems with ESX Server
Furthermore,performanceonsuchasystemcanbehighlyvariable.Itvaries,for example,ifanapplicationhasmemorylocatedlocallyononebenchmarkingrun,buta subsequentrunhappenstoplaceallofthatmemoryonaremotenode.This phenomenoncanmakecapacityplanningdifficult.Finally,processorclocksmightnot besynchronizedbetweenmultiplenodes,soapplicationsthatreadtheclockdirectly mightbehaveincorrectly. SomehighendUNIXsystemsprovidesupportforNUMAoptimizationsintheir compilersandprogramminglibraries.Thissupportrequiressoftwaredevelopersto tuneandrecompiletheirprogramsforoptimalperformance.Optimizationsforone systemarenotguaranteedtoworkwellonthenextgenerationofthesamesystem. Othersystemshaveallowedanadministratortoexplicitlydecideonthenodeonwhich anapplicationshouldrun.Whilethismightbeacceptableforcertainapplicationsthat demand100percentoftheirmemorytobelocal,itcreatesanadministrativeburden andcanleadtoimbalancebetweennodeswhenworkloadschange. Ideally,thesystemsoftwareprovidestransparentNUMAsupport,sothatapplications canbenefitimmediatelywithoutmodifications.Thesystemshouldmaximizetheuse oflocalmemoryandscheduleprogramsintelligentlywithoutrequiringconstant administratorintervention.Finally,itmustrespondwelltochangingconditions withoutcompromisingfairnessorperformance.
ESX Server NUMA Scheduling

ESXServerusesasophisticatedNUMAschedulertodynamicallybalanceprocessor loadandmemorylocalityorprocessorloadbalance,asfollows: 1 EachvirtualmachinemanagedbytheNUMAschedulerisassignedahome nodeoneofthesystemsNUMAnodescontainingprocessorsandlocalmemory, asindicatedbytheSystemResourceAllocationTable(SRAT). Whenmemoryisallocatedtoavirtualmachine,theESXServerhostpreferentially allocatesitfromthehomenode. TheNUMAschedulercandynamicallychangeavirtualmachineshomenodeto respondtochangesinsystemload.Theschedulermightmigrateavirtualmachine toanewhomenodetoreduceprocessorloadimbalance.Becausethismightcause moreofitsmemorytoberemote,theschedulermightmigratethevirtual machinesmemorydynamicallytoitsnewhomenodetoimprovememorylocality. TheNUMAschedulermightalsoswapvirtualmachinesbetweennodeswhenthis improvesoverallmemorylocality.
2 3
VMware, Inc.
153
SomevirtualmachinesarenotmanagedbytheESXServerNUMAscheduler.For example,ifyoumanuallysettheprocessoraffinityforavirtualmachine,theNUMA schedulermightnotbeabletomanagethisvirtualmachine.Virtualmachinesthathave morevirtualprocessorsthanthenumberofphysicalprocessorcoresavailableona singlehardwarenodecannotbemanagedautomatically.Virtualmachinesthatarenot managedbytheNUMAschedulerstillruncorrectly.However,theydontbenefitfrom ESXServersNUMAoptimizations. TheNUMAschedulingandmemoryplacementpoliciesinVMwareESXServercan manageallvirtualmachinestransparently,sothatadministratorsdonotneedto addressthecomplexityofbalancingvirtualmachinesbetweennodesexplicitly. Theoptimizationsworkseamlesslyregardlessofthetypeofguestoperatingsystem. ESXServerprovidesNUMAsupporteventovirtualmachinesthatdonotsupport NUMAhardware,suchasWindowsNT4.0.Asaresult,youcantakeadvantageofnew hardwareevenwithlegacyoperatingsystems.
VMware NUMA Optimization Algorithms

ThissectiondescribesthealgorithmsusedbyVMwareESXServertomaximize applicationperformancewhilestillmaintainingresourceguarantees.
Home Nodes and Initial Placement

Whenavirtualmachineispoweredon,ESXServerassignsitahomenode.Avirtual machinerunsonlyonprocessorswithinitshomenode,anditsnewlyallocated memorycomesfromthehomenodeaswell.Unlessavirtualmachineshomenode changes,itusesonlylocalmemory,avoidingtheperformancepenaltiesassociatedwith remotememoryaccessestootherNUMAnodes. Newvirtualmachinesareinitiallyassignedtohomenodesinaroundrobinfashion, withthefirstvirtualmachinegoingtothefirstnode,thesecondvirtualmachinetothe secondnode,andsoforth.Thispolicyensuresthatmemoryisevenlyusedthroughout allnodesofthesystem. Severaloperatingsystems,suchasWindowsServer2003,providethislevelofNUMA support,whichisknownasinitialplacement.Itmightbesufficientforsystemsthatrun onlyasingleworkload,suchasabenchmarkingconfiguration,whichdoesnotchange overthecourseofthesystemsuptime.However,initialplacementisnotsophisticated enoughtoguaranteegoodperformanceandfairnessforadatacenterclasssystemthat isexpectedtosupportchangingworkloads.
154
VMware, Inc.
Tounderstandtheweaknessesofaninitialplacementonlysystem,considerthe followingexample:anadministratorstartsfourvirtualmachinesandthesystemplaces twoofthemonthefirstnode.Thesecondtwovirtualmachinesareplacedonthe secondnode.Ifbothvirtualmachinesonthesecondnodearestopped,orifthey becomeidle,thesystembecomescompletelyimbalanced,withtheentireloadplaced onthefirstnode.Evenifthesystemallowsoneoftheremainingvirtualmachinesto runremotelyonthesecondnode,itsuffersaseriousperformancepenaltybecauseall itsmemoryremainsonitsoriginalnode.
Dynamic Load Balancing and Page Migration

ESXServercombinesthetraditionalinitialplacementapproachwithadynamic rebalancingalgorithm.Periodically(everytwosecondsbydefault),thesystem examinestheloadsofthevariousnodesanddeterminesifitshouldrebalancetheload bymovingavirtualmachinefromonenodetoanother.Thiscalculationtakesinto accounttheresourcesettingsforvirtualmachinesandresourcepoolstoimprove performancewithoutviolatingfairnessorresourceentitlements. Therebalancerselectsanappropriatevirtualmachineandchangesitshomenodetothe leastloadednode.Whenitcan,therebalancermovesavirtualmachinethatalreadyhas somememorylocatedonthedestinationnode.Fromthatpointon(unlessitismoved again),thevirtualmachineallocatesmemoryonitsnewhomenodeanditrunsonlyon processorswithinthenewhomenode. Rebalancingisaneffectivesolutiontomaintainfairnessandensurethatallnodesare fullyused.Therebalancermightneedtomoveavirtualmachinetoanodeonwhichit hasallocatedlittleornomemory.Inthiscase,thevirtualmachineincursaperformance penaltyassociatedwithalargenumberofremotememoryaccesses.ESXServercan eliminatethispenaltybytransparentlymigratingmemoryfromthevirtualmachines originalnodetoitsnewhomenode: 1 2 Thesystemselectsapage(4KBofcontiguousmemory)ontheoriginalnodeand copiesitsdatatoapageinthedestinationnode. Thesystemusesthevirtualmachinemonitorlayerandtheprocessorsmemory managementhardwaretoseamlesslyremapthevirtualmachinesviewofmemory, sothatitusesthepageonthedestinationnodeforallfurtherreferences, eliminatingthepenaltyofremotememoryaccess.
VMware, Inc.
155
Whenavirtualmachinemovestoanewnode,theESXServerhostimmediatelybegins tomigrateitsmemoryinthisfashion.Itmanagestheratetoavoidovertaxingthe system,particularlywhenthevirtualmachinehaslittleremotememoryremainingor whenthedestinationnodehaslittlefreememoryavailable.Thememorymigration algorithmalsoensuresthattheESXServerhostdoesnotmovememoryneedlesslyifa virtualmachineismovedtoanewnodeforonlyashortperiod. Wheninitialplacement,dynamicrebalancing,andintelligentmemorymigrationwork inconjunction,theyensuregoodmemoryperformanceonNUMAsystems,eveninthe presenceofchangingworkloads.Whenamajorworkloadchangeoccurs,forinstance whennewvirtualmachinesarestarted,thesystemtakestimetoreadjust,migrating virtualmachinesandmemorytonewlocations.Afterashortperiod,typicallyseconds orminutes,thesystemcompletesitsreadjustmentsandreachesasteadystate.
Transparent Page Sharing Optimized for NUMA

ManyESXServerworkloadspresentopportunitiesforsharingmemoryacrossvirtual machines.Forexample,severalvirtualmachinesmightberunninginstancesofthe sameguestoperatingsystem,havethesameapplicationsorcomponentsloaded,or containcommondata.Insuchcases,ESXServersystemsuseaproprietarytransparent pagesharingtechniquetosecurelyeliminateredundantcopiesofmemorypages.With memorysharing,aworkloadrunninginvirtualmachinesoftenconsumeslessmemory thanitwouldwhenrunningonphysicalmachines.Asaresult,higherlevelsof overcommitmentcanbesupportedefficiently. TransparentpagesharingforESXServersystemshasalsobeenoptimizedforuseon NUMAsystems.OnNUMAsystems,pagesaresharedpernode,soeachNUMAnode hasitsownlocalcopyofheavilysharedpages.Whenvirtualmachinesuseshared pages,theydontneedtoaccessremotememory.
Manual NUMA Controls

Ifyouhaveapplicationsthatusealotofmemoryorhaveasmallnumberofvirtual machines,youmightwanttooptimizeperformancebyspecifyingvirtualmachineCPU andmemoryplacementexplicitly.Thisisusefulifavirtualmachinerunsa memoryintensiveworkload,suchasaninmemorydatabaseorascientificcomputing applicationwithalargedataset.YoumightalsowanttooptimizeNUMAplacements manuallyifthesystemworkloadisknowntobesimpleandunchanging.Forexample, aneightprocessorsystemrunningeightvirtualmachineswithsimilarworkloadsis easytooptimizeexplicitly. NOTEInmostsituations,anESXServerhostsautomaticNUMAoptimizationsresult ingoodperformance.
156 VMware, Inc.
ESXServerprovidestwosetsofcontrolsforNUMAplacement,sothatadministrators cancontrolmemoryandprocessorplacementofavirtualmachine. TheVIClientallowsyoutospecify: CPUAffinityAvirtualmachineshoulduseonlytheprocessorsonagivennode. SeeCPUAffinityforAssociatingVirtualMachineswithaSingleNUMANode onpage 159. MemoryAffinityTheservershouldallocatememoryonlyonthespecifiednode. SeeMemoryAffinityforAssociatingMemoryAllocationswithaNUMANode onpage 160. Ifbothoptionsaresetbeforeavirtualmachinestarts,thevirtualmachinerunsonlyon theselectednodeandallofitsmemoryisallocatedlocally. Anadministratorcanalsomanuallymoveavirtualmachinetoanothernodeafterthe virtualmachinehasstartedrunning.Inthiscase,thepagemigrationrateofthevirtual machineshouldalsobesetmanually,sothatmemoryfromthevirtualmachines previousnodecanbemovedtoitsnewnode. ManualNUMAplacementmightinterferewiththeESXServerresourcemanagement algorithms,whichattempttogiveeachvirtualmachineafairshareofthesystems processorresources.Forexample,iftenvirtualmachineswithprocessorintensive workloadsaremanuallyplacedononenode,andonlytwovirtualmachinesare manuallyplacedonanothernode,itisimpossibleforthesystemtogivealltwelve virtualmachinesequalsharesofthesystemsresources.Youmustconsidertheseissues whenmakingmanualNUMAplacementdecisions.
IBM Enterprise X-Architecture Overview

TheIBMEnterpriseXArchitecturesupportsserverswithuptofournodes(alsocalled CECsorSMPExpansionComplexesinIBMterminology).Eachnodemaycontainup tofourIntelXeonMPprocessorsforatotalof16CPUs.ThenextgenerationIBM eServerx445usesanenhancedversionoftheEnterpriseXArchitecture,andscalesto eightnodeswithuptofourXeonMPprocessorsforatotalof32CPUs.The thirdgenerationIBMeServerx460providessimilarscalabilitybutalsosupports64bit XeonMPprocessors.ThehighscalabilityofallthesesystemsstemsfromtheEnterprise XArchitecturesNUMAdesignthatissharedwithIBMhighendPOWER4based pSeriesservers.SeetheIBMRedbookIBMeServerxSeries440PlanningandInstallation GuideforamoredetaileddescriptionoftheEnterpriseXArchitecture.
VMware, Inc.
157
AMD Opteron-Based Systems Overview

AMDOpteronbasedsystems,suchastheHPProLiantDL585Server,alsoprovide NUMAsupport.TheBIOSsettingfornodeinterleavingdetermineswhetherthesystem behavesmorelikeaNUMAsystemormorelikeaUniformMemoryArchitecture (UMA)system.SeetheHPProLiantDL585ServerTechnologytechnologybrief.See alsotheHPROMBasedSetupUtilityUserGuideattheHPWebsite. Bydefault,nodeinterleavingisdisabled,soeachprocessorhasitsownmemory.The BIOSbuildsaSystemResourceAllocationTable(SRAT),sotheESXServerhostdetects thesystemasNUMAandappliesNUMAoptimizations.Ifyouenablenode interleaving(alsoknownasinterleavedmemory),theBIOSdoesnotbuildanSRAT,so theESXServerhostdoesnotdetectthesystemasNUMA. CurrentlyshippingOpteronprocessorshaveuptofourcorespersocket.Whennode memoryisenabled,thememoryontheOpteronprocessorsisdividedsuchthateach sockethassomelocalmemory,butmemoryforothersocketsisremote.Thesinglecore OpteronsystemshaveasingleprocessorperNUMAnodeandthedualcoreOpteron systemshavetwoprocessorsforeachNUMAnode. SMPvirtualmachines(havingtwovirtualprocessors)cannotresidewithinaNUMA nodethathasasinglecore,suchasthesinglecoreOpteronprocessors.Thisalsomeans theycannotbemanagedbytheESXServerNUMAscheduler.Virtualmachinesthatare notmanagedbytheNUMAschedulerstillruncorrectly.However,thosevirtual machinesdontbenefitfromtheESXServerNUMAoptimizations.Uniprocessor virtualmachines(withasinglevirtualprocessor)canresidewithinasingleNUMA nodeandaremanagedbytheESXServerNUMAscheduler. NOTEForsmallOpteronsystems,NUMArebalancingisnowdisabledbydefaultto ensureschedulingfairness.UsetheNuma.RebalanceCoresTotaland Numa.RebalanceCoresNodeoptionstochangethisbehavior.SeeSettingAdvanced VirtualMachineAttributesonpage 149.
158
VMware, Inc.
Obtaining NUMA Configuration Information and Statistics

NUMAconfigurationinformationandstatisticscanbeviewedintheMemorypanelof theresxtop(oresxtop)utility.SeeMemoryPanelonpage 179.
CPU Affinity for Associating Virtual Machines with a Single NUMA Node
Youmightbeabletoimprovetheperformanceoftheapplicationsonavirtualmachine byassociatingittotheCPUnumbersonasingleNUMAnode(manualCPUaffinity). CAUTIONThereareanumberofpotentialissuesifyouuseCPUaffinity.SeePotential IssueswithAffinityonpage 127. To set CPU affinity for a single NUMA node 1 2 3 UsingaVIClient,rightclickavirtualmachineandchooseEditSettings. IntheVirtualMachinePropertiesdialogbox,selecttheResourcestabandchoose AdvancedCPU. IntheSchedulingAffinitypanel,setCPUaffinityfordifferentNUMAnodes. NOTEYoumustmanuallyselecttheboxesforallprocessorsintheNUMAnode. CPUaffinityisspecifiedonaperprocessor,notonapernode,basis.
VMware, Inc.
159
Memory Affinity for Associating Memory Allocations with a NUMA Node

Youcanspecifythatallfuturememoryallocationsonavirtualmachineusepages associatedwithasingleNUMAnode(alsoknownasmanualmemoryaffinity).When thevirtualmachineuseslocalmemory,theperformanceimprovesonthatvirtual machine. NOTESpecifynodestobeusedforfuturememoryallocationsonlyifyouhavealso specifiedCPUaffinity.Ifyoumakemanualchangesonlytothememoryaffinity settings,automaticNUMArebalancingdoesnotworkproperly. To associate memory allocations with a NUMA node 1 2 3 UsingaVIClient,rightclickavirtualmachineandchooseEditSettings. IntheVirtualMachinePropertiesdialogbox,selecttheResourcestab,andchoose Memory. IntheNUMAMemoryAffinitypanel,setmemoryaffinity.
Example: Binding a Virtual Machine to a Single NUMA Node Thefollowing exampleillustratesmanuallybindingfourCPUstoasingleNUMAnodeforavirtual machineonaneightwayserver.Youwantthisvirtualmachinetorunonlyonnode1. TheCPUsforexample,4,5,6,and7arethephysicalCPUnumbers. To bind a two-way virtual machine to use the last four physical CPUs of an eight-processor machine 1 2 3 4 IntheVIClientinventorypanel,selectthevirtualmachineandchooseEdit Settings. SelectOptionsandclickAdvanced. ClicktheConfigurationParametersbutton. IntheVIClient,turnonCPUaffinityforprocessors4,5,and6.
160
VMware, Inc.
To set the virtual machines memory to specify that all of the virtual machines memory should be allocated on node 1 1 2 3 4 IntheVIClientinventorypanel,selectthevirtualmachineandchooseEdit Settings. SelectOptionsandclickAdvanced. ClicktheConfigurationParametersbutton. IntheVIClient,setmemoryaffinityfortheNUMAnodeto1.
CompletingthesetwotasksensuresthatthevirtualmachinerunsonlyonNUMA node 1and,whenpossible,allocatesmemoryfromthesamenode.
VMware, Inc.
161
162
VMware, Inc.
11
Best Practices
11
ThischapterdiscussessomebestpracticesforusersofESXServerandVirtualCenter. Thischapterdiscussesthefollowingtopics: ResourceManagementBestPracticesonpage 163 CreatingandDeployingVirtualMachinesonpage 164 VMwareHABestPracticesonpage 166
Resource Management Best Practices

Thefollowingguidelinescanhelpyouachieveoptimalperformanceforyourvirtual machines: Ifyouexpectfrequentchangestothetotalavailableresources,useSharesto allocateresourcesfairlyacrossvirtualmachines.IfyouuseShares,andyou upgradethehost,forexample,eachvirtualmachinestaysatthesamepriority (keepsthesamenumberofshares)eventhougheachsharerepresentsalarger amountofmemoryorCPU. UseReservationtospecifytheminimumacceptableamountofCPUormemory, nottheamountyouwanttohaveavailable.Thehostassignsadditionalresources asavailablebasedonthenumberofsharesandthelimitforyourvirtualmachine. Theamountofconcreteresourcesrepresentedbyareservationdoesnotchange whenyouchangetheenvironment,suchasbyaddingorremovingvirtual machines. DonotsetReservationtoohigh.Areservationthatistoohighcanlimitthenumber ofvirtualmachinesinaresourcepool.
VMware, Inc.
163
Whenspecifyingthereservationsforvirtualmachines,donotcommitall resources.Asyoumoveclosertofullyreservingallcapacityinthesystem,it becomesincreasinglydifficulttomakechangestoreservationsandtotheresource poolhierarchywithoutviolatingadmissioncontrol.InaDRSenabledcluster, reservationsthatfullycommitthecapacityoftheclusterorofindividualhostsin theclustercanpreventDRSfrommigratingvirtualmachinesbetweenhosts. Useresourcepoolsfordelegatedresourcemanagement.Tofullyisolatearesource pool,maketheresourcepooltypeFixedanduseReservationandLimit. Groupvirtualmachinesforamultitierserviceinaresourcepool.Resourcepools allowtheESXServerhosttoassignresourcesfortheserviceasawhole.
Creating and Deploying Virtual Machines

Thissectiongivesbestpracticesinformationforplanningandcreatingvirtual machines.
Planning
Beforeyoudeployavirtualmachine,youneedto: Planyourloadmix. Understandgoalsandexpectations. Understandtherequirements,andwhatitmeanstobesuccessful. Avoidmixingvirtualmachinesthathavecompetingresourcerequirements. Testbeforeyoudeployifyouhavespecificperformanceexpectations,. Virtualizationallowsanumberofvirtualmachinestosharethehostsresources.Itdoes notcreatenewresources.Virtualizationcanresultinoverheads.
Creating Virtual Machines

Whenyoucreatevirtualmachines,besuretosizethemaccordingtoyouractualneeds, justlikephysicalmachines.Overconfiguredvirtualmachineswasteshareable resources. Tooptimizeperformance,disableunusedvirtualdevicessuchasCOMports,LPT ports,floppydrives,CDROMs,USBadapters,andsoon.Thosedevicesare periodicallypolledbytheguestoperatingsystemeveniftheyarenotinuse.This unproductivepollingwastesshareableresources.
164
VMware, Inc.
Chapter 11 Best Practices
InstallVMwareTools,whichhelpsyouachievehigherperformance,canresultinmore efficientCPUutilization,andincludesdisk,network,andmemoryreclamationdrivers.
Deploying the Guest Operating System

Tuneandsizethevirtualmachineoperatingsystemjustasyoutunetheoperating systemofaphysicalmachinewithregistry,swapspace,andsoon.Disableunnecessary programsandservicessuchasscreensavers.Unnecessaryprogramsandserviceswaste shareableresources. Keeptheguestoperatingsystemuptodatewiththelatestpatches.Ifyouareusing MicrosoftWindowsastheguestoperatingsystem,checkforanyknownoperating systemissuesinMicrosoftknowledgebasearticles. NOTEYoumustconfiguretheguestoperatingsystemwithsufficientswapspace. Someguestoperatingsystemshaveadditionallimitations.SeeSwapSpaceandGuest OperatingSystemsonpage 142.
Deploying Guest Applications

Tuneandsizeapplicationsonyourvirtualmachinesinthesamewayyoutuneandsize applicationsonyourphysicalmachine. DonotrunsinglethreadedapplicationsinanSMPvirtualmachine.Singlethreaded workloadscannottakeadvantageofadditionalvirtualCPUs,andunusedvirtualCPUs wasteshareableresources.However,aworkloadconsistingofseveralsinglethreaded applicationsrunningconcurrentlymightbeabletotakeadvantageofadditional virtual CPUs.
Configuring VMkernel Memory

VMkernelreclaimsmemorybyballooningandswapping.SeeChapter 9,Advanced ResourceManagement,onpage 123.Tousememoryresourcesoptimally,avoidhigh reclamationactivitybycorrectlysizingvirtualmachinesandbyavoidinghighmemory overcommitment.SeeMemoryOvercommitmentonpage 42. VMkernelimplementsaNUMAscheduler,whichsupportsIBMandAMDNUMA architectures.TheschedulerlocatesvirtualmachinememoryandvirtualCPUsonthe sameNUMAnode.Thispreventspossibleperformancedegradationbecauseofremote memoryaccesses.Thehosthardwareshouldbeconfiguredsothatphysicalhost memoryisevenlybalancedacrossNUMAnodes.SeeChapter 10,UsingNUMA SystemswithESXServer,onpage 151.
VMware, Inc.
165
VMware HA Best Practices

UsethefollowingVMwareHAbestpracticesthatareapplicableforyourESXServer implementationandnetworkingarchitecture.
Networking Best Practices

TheconfigurationofESXServerhostnetworkingandnameresolution,aswellasthe networkinginfrastructureexternaltoESXServerhosts(switches,routers,and firewalls)iscriticaltooptimizingVMwareHAsetup.Whenyouconfigurethese components,usethefollowingbestpracticestoimproveVMwareHAperformance. IfyourswitchessupportthePortFast(oranequivalent)setting,enableitonthe physicalnetworkswitchesthatconnectservers.Thishelpsavoidspanningtree isolationevents.Formoreinformationonthisoption,seethedocumentation providedbyyournetworkingswitchvendor. Makesurethatthefollowingfirewallportsareopenforcommunicationbythe serviceconsoleforallESXServer3hosts: Incomingport:TCP/UDP80428045 Outgoingport:TCP/UDP20502250 Forbetterheartbeatreliability,configureendtoenddualnetworkpathsbetween serversforserviceconsolenetworking.Configureshorternetworkpathsbetween theserversinacluster.Routeswithtoomanyhopscancausenetworkingpacket delaysforheartbeats. DisableVMwareHA(usingVirtualCenter,cleartheEnableVMwareHA checkboxintheSettingsdialogboxforthecluster)whenyouperformany networkingmaintenancethatmightdisableallheartbeatpathsbetweenhosts. UseDNSfornameresolutionratherthantheerrorpronemethodofmanually editingthelocal/etc/hostsfileonESXServerhosts.Ifyoudoedit/etc/hosts,you mustincludebothlongandshortnames. UseconsistentportnamesonVLANsforpublicnetworks.Portnamesareusedto reconfigureaccesstothenetworkbyvirtualmachines.Ifyouuseinconsistent namesbetweentheoriginalserverandthefailoverserver,virtualmachinesare disconnectedfromtheirnetworksafterfailover. UsevalidvirtualmachinenetworklabelsonallserversinaVMwareHAcluster. Virtualmachinesusetheselabelstoreestablishnetworkconnectivityuponrestart.
166
VMware, Inc.
Setting Up Networking Redundancy

NetworkingredundancybetweenclusternodesisimportantforVMwareHA reliability.RedundantserviceconsolenetworkingonESXServer3(orVMkernel networkingonESXServer3i)allowsthereliabledetectionoffailuresandprevents isolationconditionsfromoccurring,sinceheartbeatscanbesentovermultiple networks. YoucanimplementnetworkredundancyattheNIClevelorattheservice console/VMkernelportlevel.Inmostimplementations,NICteamingprovides sufficientredundancy,butyoucanuseoraddserviceconsole/portredundancyif additionalredundancyisrequired.
NIC Teaming
AsshowninFigure 111,usingateamoftwoNICsconnectedtoseparatephysical switchesimprovesthereliabilityofaserviceconsole(or,inESXServer3i,VMkernel) network.BecauseserversconnectedthroughtwoNICs(andthroughseparateswitches) havetwoindependentpathsforsendingandreceivingheartbeats,theclusterismore resilient. ToconfigureaNICteamfortheserviceconsole,configurethevNICsinvSwitch configurationforActive/standbyconfiguration.Therecommendedparametersettings forthevNICsare: RollingFailover=Yes DefaultLoadbalancing=routebasedonoriginatingportID NOTEAfteryouhaveaddedaNICtoahostinyourVMwareHAcluster,youmust reconfigureHAonthathost.
VMware, Inc.
167
Figure 11-1. Service Console Redundancy Using NIC Teaming
NIC Teaming Scenario Thefollowingscenarioillustratestheuseofasingleservice consolenetworkwithNICteamingfornetworkredundancy: Youassumesomeriskwhenyouconfigurehostsintheclusterwithonlyone serviceconsolenetwork(subnet10.20.XX.XX).UsetwoteamedNICstoprotect againstNICfailure. Thedefaulttimeoutisincreasedto60seconds(das.failuredetectiontime= 60000).
Secondary Service Console Network

AsanalternativetoNICteamingforprovidingredundancyforheartbeats,youcan createasecondaryserviceconsole(orVMkernelportforESXServer3i),whichisthen attachedtoaseparatevirtualswitch.Theprimaryserviceconsoleisstillusedfor networkandmanagementpurposes.Whenthesecondaryserviceconsolenetworkis created,VMwareHAsendsheartbeatsoverboththeprimaryandsecondaryservice consoles.Ifonepathfails,VMwareHAcanstillsendandreceiveheartbeatsoverthe otherpath. Bydefault,thegatewayIPaddressspecifiedineachESXServerhostsserviceconsole networkconfigurationisusedastheisolationaddress.Eachserviceconsolenetwork musthaveoneisolationaddressitcanreach.Whenyousetupserviceconsole redundancy,youmustspecifyanadditionalisolationresponseaddress (das.isolationaddress2)forthesecondaryserviceconsolenetwork.Thisisolation addressshouldhaveasfewnetworkhopsaspossible.Whenyouspecifyasecondary isolationaddress,VMwarerecommendsthatyouincreasethe das.failuredetectiontimesettingto20000millisecondsorgreater.SeeSetting AdvancedHAOptionsonpage 121.
168
VMware, Inc.
Youcanfurtheroptimizeyournetwork(ifyouhavealreadyconfiguredaVMotion network)byaddingasecondaryserviceconsolenetworktotheVMotionvswitch. As showninFigure 112,avirtualswitchcanbesharedbetweenVMotionnetworksanda secondaryserviceconsolenetwork. Figure 11-2. Network Redundancy with a Secondary Service Console
Redundant Service Console Network Scenario Thefollowingscenarioillustrates theuseofaredundantserviceconsolenetwork: Configureeachhostintheclusterwithtwoserviceconsolenetworksbyleveraging anexistingVMotionnetwork(subnets10.20.YY.YYand192.168.ZZ.ZZ). Usethedefaultgatewayforthefirstnetworkandspecify das.isolationaddress2=192.168.1.103astheadditionalisolationaddressfor thesecondnetwork. Increasethedefaulttimeoutto20seconds(das.failuredetectiontime=20000).
VMware, Inc.
169
Other VMwareHA Cluster Considerations

OtherconsiderationsforoptimizingtheperformanceofyourVMwareHAcluster include: Uselargergroupsofhomogenousserverstoallowhigherlevelsofutilization acrossanVMwareHAenabledcluster(onaverage). Morenodesperclustercantoleratemultiplehostfailureswhilestill guaranteeingfailovercapacities. Admissioncontrolheuristicsareconservativelyweighted,sothatlarge serverswithmanyvirtualmachinescanfailovertosmallerservers. Todefinethesizingestimatesusedforadmissioncontrol,setreasonable reservationsfortheminimumresourcesneeded. Admissioncontrolwillexceedfailovercapacitieswhenreservationsarenot set;otherwiseVMwareHAwillusethelargestreservationspecifiedasthe slotsize(seePlanningforHAClustersonpage 75.) Ataminimum,setreservationsforafewvirtualmachinesconsidered average. PerformyourowncapacityplanningbychoosingAllowvirtualmachinestobe poweredoneveniftheyviolateavailabilityconstraints.Admissioncontrolmay betooconservativewhenhostandvirtualmachinesizesvarywidely.VMwareHA stilltriestorestartasmanyvirtualmachinesasitcan.
170
VMware, Inc.
Performance Monitoring Utilities: resxtop and esxtop
Theresxtopandesxtopcommandlineutilitiesprovideadetailedlookathow ESXServerusesresourcesinrealtime.Youcanstarteitherutilityinoneofthreemodes: interactive(default),batch,orreplay.Thisappendixexplainshowtouseresxtopand esxtopineachofthesemodesandgivereferencestoavailablecommandsanddisplay statistics.Unlessspecifiedotherwise,thecommandsandstatisticsfortheutilitiesare thesame.Thefollowingtopicsarediscussed: DecidingtoUseresxtoporesxtoponpage 171 UsingtheUtilitiesinInteractiveModeonpage 173 UsingtheUtilitiesinBatchModeonpage 194 UsingtheUtilitiesinReplayModeonpage 195
Deciding to Use resxtop or esxtop

Thefundamentaldifferencebetweenresxtopandesxtopisthatyoucanuseresxtop remotely(orlocally),whereasesxtopcanbestartedonlythroughtheserviceconsole ofalocalESXServerhost.
Using the resxtop Utility

TheresxtoputilityisaRemoteCommandLineInterface(RemoteCLI)commandand beforeyoucanuseanyRemoteCLIcommands,youmustdownload,install,and configuretheRemoteCLIvirtualappliance.SeetheESXServer3iversion3.5 ConfigurationGuide. Afterthevirtualappliancehasbeensetup,startresxtopfromthecommandlineofa remoteLinuxclient,withthesamecommandlineoptionsasesxtop.Sothattheclient
VMware, Inc. 171
youareusingcanconnecttoandbeauthenticatedbytheremoteserver,usethe followingadditionaloptions: [server]Nameoftheremoteserverhosttoconnectto(required). [portnumber]Portnumbertoconnecttoontheremoteserver.Thedefaultportis 443,unlessthishasbeenchangedontheserver,thisoptionisnotneeded. [username]Usernametobeauthenticatedwhenconnectingtotheremotehost.You willbepromptedbytheremoteserverforapassword,aswell. NOTEresxtopdoesnotusealltheoptionssharedbyotherRemoteCLIcommands. YoucanalsouseresxtoponalocalESXServerhost.Todoso,omittheserveroption onthecommandlineandthecommandwilldefaulttolocalhost.
Using the esxtop Utility

TheesxtoputilityrunsonlyontheESXServerhostsserviceconsole. To start esxtop 1 2 Makesureyouhaverootuserprivileges. Typethecommand,usingtheoptionsyouwant:
vm-support_dir_path]
esxtop [-] [h] [v] [b] [s] [a] [c filename] [R [d delay] [n iter]
Theesxtoputilityreadsitsdefaultconfigurationfrom.esxtop310rc.This configurationfileconsistsofsevenlines. Thefirstsixlinescontainlowercaseanduppercaseletterstospecifywhichfieldsappear inwhichorderontheCPU,memory,storageadapter,storagedevice,virtualmachine storage,andnetworkpanel.TheletterscorrespondtothelettersintheFieldsorOrder panelsfortherespectiveesxtoppanel. Theseventhlinecontainsinformationontheotheroptions.Mostimportant,ifyou savedaconfigurationinsecuremode,youdonotgetaninsecureesxtopwithout removingthesfromtheseventhlineofyour.esxtop310rcfile.Anumberspecifies thedelaytimebetweenupdates.Asininteractivemode,typingc,m,d,u,v,orn determinesthepanelwithwhichesxtopstarts. NOTEEditingthisfileisnotrecommended.Instead,selectthefieldsandtheorderin arunningesxtopprocess,makechanges,andsavethisfileusingtheWinteractive command.
172
VMware, Inc.
Using the Utilities in Interactive Mode

Bydefault,resxtopandesxtoprunininteractivemode.Interactivemodedisplays statisticsindifferentpanels. Ahelpmenuisavailableforeachpanel.
Interactive Mode Command-Line Options

ThecommandlineoptionslistedinTable A1areavailableininteractivemode. Table A-1. Interactive Mode Command-Line Options
Option h v s d Description Printshelpforresxtop(oresxtop)commandlineoptions. Printsresxtop(oresxtop)versionnumber. Callsresxtop(oresxtop)insecuremode.Insecuremode,the-d command,whichspecifiesdelaybetweenupdates,isdisabled. Specifiesthedelaybetweenupdates.Thedefaultisfiveseconds.The minimumistwoseconds.Changethiswiththeinteractivecommand s.Ifyouspecifyadelayoflessthantwoseconds,thedelayissettotwo seconds. Numberofiterations.Updatesthedisplayntimesandexits. Thenameoftheremoteserverhosttoconnectto(requiredfor resxtoponly). Theportnumbertoconnecttoontheremoteserver.Thedefaultport is443,andunlessthishasbeenchangedontheserver,thisoptionis notneeded.(resxtoponly) Theusernametobeauthenticatedwhenconnectingtotheremote host.Youarepromptedbytheremoteserverforapassword,aswell (resxtoponly). Showallstatistics.Thisoptionoverridesconfigurationfilesetupsand showsallstatistics.Theconfigurationfilecanbethedefault ~/.esxtop310rcconfigurationfileorauserdefinedconfiguration file. Loadauserdefinedconfigurationfile.Ifthecoptionisnotused,the defaultconfigurationfilenameis~/.esxtop310rc.Createyourown configurationfile,specifyingadifferentfilename,usingtheW singlekeyinteractivecommand.SeeInteractiveModeSingleKey Commandsonpage 174forinformationaboutW.
n server portnumber
username
c <filename>
VMware, Inc.
173
Common Statistics Description

Severalstatisticsappearonthedifferentpanelswhileresxtop(oresxtop)isrunning ininteractivemode.Thefollowingstatisticsarecommonacrossallfourpanels. TheUptimeline,foundatthetopofeachofthefourresxtop(oresxtop)panels, displaysthecurrenttime,timesincelastreboot,numberofcurrentlyrunningworlds andloadaverages.AworldisanESXServerVMkernelschedulableentity,similartoa processorthreadinotheroperatingsystems. Belowthattheloadaveragesoverthepastone,five,andfifteenminutesappear.Load averagestakeintoaccountbothrunningandreadytorunworlds.Aloadaverageof 1.00meansthatallthephysicalCPUsarefullyutilized.Aloadaverageof2.00means thattheESXServersystemmightneedtwiceasmanyphysicalCPUsasarecurrently available.Similarly,aloadaverageof0.50meansthatthephysicalCPUsonthe ESXServersystemarehalfutilized.
Interactive Mode Single-Key Commands

Whenrunningininteractivemode,resxtop(oresxtop)recognizesseveralsinglekey commands.CommandslistedinTable A2arerecognizedinallfourpanels.The commandtospecifythedelaybetweenupdatesisdisabledifthesoptionhasbeen givenonthecommandline(seeInteractiveModeCommandLineOptionson page 173).Allsortinginteractivecommandssortindescendingorder. Table A-2. Interactive Mode Single-Key Commands
Key hor? space ^L forF oorO # Description Displaysahelpmenuforthecurrentpanel,givingabriefsummaryofcommands, andthestatusofsecuremode. Immediatelyupdatesthecurrentpanel. Erasesandredrawsthecurrentpanel. Displaysapanelforaddingorremovingstatisticscolumns(fields)toorfromthe currentpanel. Displaysapanelforchangingtheorderofstatisticscolumnsonthecurrentpanel. Promptsyouforthenumberofstatisticsrowstodisplay.Anyvaluegreaterthan0 overridesautomaticdeterminationofthenumberofrowstoshow,whichisbased onwindowsizemeasurement.Ifyouchangethisnumberinoneresxtop(or esxtop)panel,thechangeaffectsallfourpanels. Promptsyouforthedelaybetweenupdates,inseconds.Fractionalvaluesare recognizeddowntomicroseconds.Thedefaultvalueisfiveseconds.Theminimum valueistwoseconds.Thiscommandisnotavailableinsecuremode.
174
VMware, Inc.
Table A-2. Interactive Mode Single-Key Commands (Continued)

Key W Description Writethecurrentsetuptoanesxtop(orresxtop)configurationfile.Thisisthe recommendedwaytowriteaconfigurationfile.Thedefaultfilenameistheone specifiedbycoption,or~/.esxtop310rcifthe-coptionisnotused.Youcanalso specifyadifferentfilenameonthepromptgeneratedbythisWcommand. Quitsinteractivemode. SwitchestotheCPUresourceutilizationpanel. Switchestothememoryresourceutilizationpanel. Switchestothestorage(disk)adapterresourceutilizationpanel. Switchtostorage(disk)deviceresourceutilizationscreen.SeeStorageDevice Panelonpage 187. Switchtostorage(disk)virtualmachineresourceutilizationscreen.SeeVirtual MachineStoragePanelonpage 190 Switchestothenetworkresourceutilizationpanel.
q c m d u v n
Statistics Columns and Order Pages

Ifyoupressf,F,o,orO,thesystemdisplaysapagethatspecifiesthefieldorderonthe toplineandshortdescriptionsofthefieldcontents.Iftheletterinthefieldstring correspondingtoafieldisuppercase,thefieldisdisplayed.Anasteriskinfrontofthe fielddescriptionindicateswhetherafieldisdisplayed. Theorderofthefieldscorrespondstotheorderofthelettersinthestring. FromtheFieldSelectpanel,youcan: Togglethedisplayofafieldbypressingthecorrespondingletter Moveafieldtotheleftbypressingthecorrespondinguppercaseletter. Moveafieldtotherightbypressingthecorrespondinglowercaseletter. Figure A1showsafieldorderchange.
VMware, Inc.
175
Figure A-1. Field Order Change
CPU Panel
TheCPUpaneldisplaysserverwidestatisticsaswellasstatisticsforindividualworld, resourcepool,andvirtualmachineCPUutilization.Resourcepools,runningvirtual machines,orotherworldsareattimesreferredtoasgroups.Forworldsbelongingtoa virtualmachine,statisticsfortherunningvirtualmachinearedisplayed.Allother worldsarelogicallyaggregatedintotheresourcepoolsthatcontainthem. Figure A-2. CPU Panel
Youcanchangethedisplayusingsinglekeycommands.Statisticsandsinglekey commandsarediscussedinTable A3andTable A4.
176
VMware, Inc.
Table A-3. CPU Panel Statistics

Line PCPU(%) LCPU(%) Description PercentageofCPUutilizationperphysicalCPUandtotalaveragephysical CPUutilization. PercentageofCPUutilizationperlogicalCPU.Thepercentagesforthe logicalCPUsbelongingtoapackageaddupto100percent.Thislineappears onlyifhyperthreadingispresentandenabled.SeeHyperthreadingand ESXServeronpage 130. PercentagesoftotalCPUtimeasreportedbytheESXServerserviceconsole. usPercentageusertime. syPercentagesystemtime. idPercentageidletime. waPercentagewaittime. cs/secContextswitchespersecondrecordedbytheserviceconsole. ID GID NAME NWLD ResourcepoolIDorvirtualmachineIDoftherunningworldsresourcepool orvirtualmachine,orworldIDofrunningworld. ResourcepoolIDoftherunningworldsresourcepoolorvirtualmachine. Nameofrunningworldsresourcepoolorvirtualmachine,ornameof runningworld. Numberofmembersinrunningworldsresourcepoolorvirtualmachine.If aGroupisexpandedusingtheinteractivecommande(seeinteractive commands),thenNWLDforalltheresultingworldsis1(someresource poolsliketheconsoleresourcepoolhaveonlyonemember). SetofCPUstatisticsmadeupofthefollowingpercentages.Foraworld,the percentagesareapercentageofonephysicalCPU. PercentagephysicalCPUusedbytheresourcepool,virtualmachine,or world. PercentageoftimespentintheESXServerVMkernelonbehalfofthe resourcepool,virtualmachine,orworldtoprocessinterruptsandto performothersystemactivities.Thistimeispartofthetimeusedtocalculate %USED,above. Percentageoftimetheresourcepool,virtualmachine,orworldspentinthe blockedorbusywaitstate.Thispercentageincludesthepercentageoftime theresourcepool,virtualmachine,orworldwasidle. (SEEUPDATE)Percentageoftimetheresourcepool,virtualmachine,orworld wasidle.Subtractthispercentagefrom%WAIT,abovetoseethepercentage oftimetheresourcepool,virtualmachine,orworldwaswaitingforsome event. Percentageoftimetheresourcepool,virtualmachine,orworldwasreadyto run.
CCPU(%)
%STATE TIMES %USED %SYS
%WAIT
%IDLE
%RDY
VMware, Inc.
177
Table A-3. CPU Panel Statistics (Continued)

Line %MLMTD Description PercentageoftimetheESXServerVMkerneldeliberatelydidnotrunthe resourcepool,virtualmachine,orworldbecausedoingsowouldviolatethe resourcepool,virtualmachine,orworldslimitsetting.Eventhoughthe resourcepool,virtualmachine,orworldisreadytorunwhenitisprevented fromrunninginthisway,the%MLMTDtimeisnotincludedin%RDY time. SetofCPUstatisticsmadeupofpersecondeventrates.Thesestatisticsare forVMwareinternaluseonly. SetofCPUstatisticsmadeupofthefollowingCPUallocationconfiguration parameters. Resourcepool,virtualmachine,orworldattributeReservation.See CreatingandCustomizingResourcePoolsonpage 25. Resourcepool,virtualmachine,orworldattributeLimit.Avalueof1 meansunlimited.SeeCreatingandCustomizingResourcePoolson page 25. Resourcepool,virtualmachine,orworldattributeShares.SeeCreating andCustomizingResourcePoolsonpage 25. SetofCPUstatisticsmadeupofthefollowingCPUconfiguration parametersandstatistics.Thesestatisticsareapplicableonlytoworldsand nottovirtualmachinesorresourcepools. Bitmaskshowingthecurrentschedulingaffinityfortheworld.SeeUsing CPUAffinitytoAssignVirtualMachinestoSpecificProcessorson page 126. Currenthyperthreadingconfiguration.SeeAdvancedServer ConfigurationforHyperthreadingonpage 131. Thephysicalorlogicalprocessoronwhichtheworldwasrunningwhen resxtop(oresxtop)obtainedthisinformation. Indicateswhethertheworldiscurrentlyquarantinedornot.Nmeansno andYmeansyes.SeeQuarantiningonpage 133. Timerrateforthisworld. Percentageofsystemtimespentduringschedulingofaresourcepool, virtualmachine,orworldonbehalfofadifferentresourcepool,virtual machine,orworldwhiletheresourcepool,virtualmachine,orworldwas scheduled.Thistimeisnotincludedin%SYS.Forexample,ifvirtual machineAiscurrentlybeingscheduledandanetworkpacketforvirtual machineBisprocessedbytheESXServerVMkernel,thetimespentappears as%OVRLPforvirtualmachineAand%SYSforvirtualmachineB.
EVENT COUNTS/s CPUALLOC AMIN AMAX
ASHRS SUMMARY STATS AFFINITYBIT MASK HTSHARING CPU HTQ TIMER/s %OVRLP
178
VMware, Inc.
Table A-3. CPU Panel Statistics (Continued)

Line %RUN Description Percentageoftotaltimescheduled.Thistimedoesnotaccountfor hyperthreadingandsystemtime.Onahyperthreadingenabledserver,the %RUNcanbetwiceaslargeas%USED. Percentageoftimearesourcepoolspendsinaready,codeschedulestate. (Note:youmightseethisstatisticdisplayed,butitisintendedforVMware useonly).
%CSTP
Table A-4. CPU Panel Single-Key Commands

Command e Description ToggleswhetherCPUstatisticsaredisplayedexpandedorunexpanded. TheexpandeddisplayincludesCPUresourceutilizationstatisticsbroken downbyindividualworldsbelongingtoaresourcepoolorvirtualmachine. Allpercentagesfortheindividualworldsarepercentageofasinglephysical CPU. Considertheseexample: Ifthe%Usedbyaresourcepoolis30%onatwowayserver,theresource poolisutilizing30percentoftwophysicalCPUs. Ifthe%Usedbyaworldbelongingtoaresourcepoolis30percentona twowayserver,thatworldisutilizing30%ofonephysicalCPU. U R N V Sortresourcepools,virtualmachines,andworldsbytheresourcepoolsor virtualmachines%Usedcolumn. Sortresourcepools,virtualmachines,andworldsbytheresourcepoolsor virtualmachines%RDYcolumn. Sortresourcepools,virtualmachines,andworldsbytheGIDcolumn.Thisis thedefaultsortorder. Displayvirtualmachineinstancesonly.
Memory Panel
TheMemorypaneldisplaysserverwideandgroupmemoryutilizationstatistics.Ason theCPUpanel,groupscorrespondtoresourcepools,runningvirtualmachines,or otherworldsthatareconsumingmemory.Fordistinctionsbetweenmachinememory andphysicalmemoryseeMemoryVirtualizationonpage 133. Thefirstline,foundatthetopoftheMemorypanel(seeFigure A3)displaysthe currenttime,timesincelastreboot,numberofcurrentlyrunningworlds,andmemory overcommitmentaverages.Thememoryovercommitmentaveragesoverthepastone, five,andfifteenminutesappear.Memoryovercommitmentof1.00meansamemory overcommitof100percent.SeeMemoryOvercommitmentonpage 42.
VMware, Inc. 179
Figure A-3. Memory Panel
Table A-5. Memory Panel Statistics

Field PMEM(MB) Description Displaysthemachinememorystatisticsfortheserver.Allnumbersarein megabytes. totalTotalamountofmachinememoryintheserver. cosAmountofmachinememoryallocatedtotheESXServerservice console(ESXServer3only). vmkAmountofmachinememorybeingusedbytheESXServer VMkernel. otherAmountofmachinememorybeingusedbyeverythingother thantheESXserviceconsole(ESXServer3only)andESXServer VMkernel. freeAmountofmachinememorythatisfree. VMKMEM (MB) DisplaysthemachinememorystatisticsfortheESXServerVMkernel.All numbersareinmegabytes. managedTotalamountofmachinememorymanagedbytheESX ServerVMkernel. minfreeMinimumamountofmachinememorythattheESXServer VMkernelaimstokeepfree. rsvdTotalamountofmachinememorycurrentlyreservedbyresource pools. ursvdTotalamountofmachinememorycurrentlyunreserved. stateCurrentmachinememoryavailabilitystate.Possiblevaluesare high,soft,hardandlow.Highmeansthatthemachinememoryisnot underanypressureandlowmeansthatitis. COSMEM (MB) DisplaysthememorystatisticsasreportedbytheESXServerserviceconsole (ESXServer3only).Allnumbersareinmegabytes. freeAmountofidlememory. swap_tTotalswapconfigured. swap_fAmountofswapfree. r/sisRateatwhichmemoryisswappedinfromdisk. w/sRateatwhichmemoryisswappedtodisk.
180
VMware, Inc.
Table A-5. Memory Panel Statistics (Continued)

Field NUMA(MB) Description DisplaystheESXServerNUMAstatistics.Thislineappearsonlyifthe ESXServerhostisrunningonaNUMAserver.Allnumbersareinmegabytes. ForeachNUMAnodeintheserver,twostatisticsaredisplayed: ThetotalamountofmachinememoryintheNUMAnodethatis managedbytheESXServer. Theamountofmachinememoryinthenodethatiscurrentlyfree(in parentheses). PSHARE (MB) DisplaystheESXServerpagesharingstatistics.Allnumbersarein megabytes. sharedAmountofphysicalmemorythatisbeingshared. commonAmountofmachinememorythatiscommonacrossworlds. savingAmountofmachinememorythatissavedbecauseofpage sharing. SWAP(MB) DisplaystheESXServerswapusagestatistics.Allnumbersareinmegabytes. currCurrentswapusage targetWheretheESXServersystemexpectstheswapusagetobe. r/sRateatwhichmemoryisswappedinbytheESXServersystem fromdisk. w/sRateatwhichmemoryisswappedtodiskbytheESXServer system. SeeSwappingonpage 142forbackgroundinformation. MEMCTL (MB) Displaysthememoryballoonstatistics.Allnumbersareinmegabytes. currTotalamountofphysicalmemoryreclaimedusingthevmmemctl module. targetTotalamountofphysicalmemorytheESXServerhostattempts toreclaimusingthevmmemctlmodule. maxMaximumamountofphysicalmemorytheESXServerhostcan reclaimusingthevmmemctlmodule. SeeMemoryBalloon(vmmemctl)Driveronpage 141. AMIN AMAX ASHRS NHN Memoryreservationforthisresourcepoolorvirtualmachine.See Reservationonpage 21. Memorylimitforthisresourcepoolorvirtualmachine.Avalueof1means Unlimited.SeeLimitonpage 22. Memorysharesforthisresourcepoolorvirtualmachine.SeeShareson page 20. Currenthomenodefortheresourcepoolorvirtualmachine.Thisstatisticis applicableonlyonNUMAsystems.Ifthevirtualmachinehasnohomenode, adash()isdisplayed.
VMware, Inc.
181

Field NRMEM (MB) N%L MEMSZ(MB) SZTGT(MB) TCHD(MB) %ACTV %ACTVS %ACTVF %ACTVN Description Currentamountofremotememoryallocatedtothevirtualmachineor resourcepool.ThisstatisticisapplicableonlyonNUMAsystems.See VMwareNUMAOptimizationAlgorithmsonpage 154. Currentpercentageofmemoryallocatedtothevirtualmachineorresource poolthatislocal. Amountofphysicalmemoryallocatedtoaresourcepoolorvirtualmachine. AmountofmachinememorytheESXServerVMkernelwantstoallocatetoa resourcepoolorvirtualmachine. Workingsetestimatefortheresourcepoolorvirtualmachine.SeeMemory AllocationandIdleMemoryTaxonpage 138. Percentageofguestphysicalmemorythatisbeingreferencedbytheguest. Thisisaninstantaneousvalue. Percentageofguestphysicalmemorythatisbeingreferencedbytheguest. Thisisaslowmovingaverage. Percentageofguestphysicalmemorythatisbeingreferencedbytheguest. Thisisafastmovingaverage. Percentageofguestphysicalmemorythatisbeingreferencedbytheguest. Thisisanestimation.(Youmightseethisstatisticdisplayed,butitisintended forVMwareuseonly.) Memoryballoondriverisinstalledornot.Nmeansno,Ymeansyes. Amountofphysicalmemoryreclaimedfromtheresourcepoolbywayof ballooning. AmountofphysicalmemorytheESXServersystemcanreclaimfromthe resourcepoolorvirtualmachinebywayofballooning. MaximumamountofphysicalmemorytheESXServersystemcanreclaim fromtheresourcepoolorvirtualmachinebywayofballooning.This maximumdependsontheguestoperatingsystemtype. Currentswapusagebythisresourcepoolorvirtualmachine. TargetwheretheESXServerhostexpectstheswapusagebytheresource poolorvirtualmachinetobe. RateatwhichtheESXServerhostswapsinmemoryfromdiskforthe resourcepoolorvirtualmachine. RateatwhichtheESXServerhostswapsresourcepoolorvirtualmachine memorytodisk. Amountofdatareadfromcheckpointfile.
MCTL? MCTLSZ (MB) MCTLTGT (MB) MCTLMAX (MB) SWCUR(MB) SWTGT(MB) SWR/s(MB) SWW/s(MB) CPTRD(MB)
182
VMware, Inc.

Field CPTTGT (MB) ZERO(MB) SHRD(MB) SHRDSVD (MB) OVHD(MB) OVHDMAX (MB) OVHDUW (MB) GST_NDx (MB) OVD_NDx (MB) Description Sizeofcheckpointfile. Resourcepoolorvirtualmachinephysicalpagesthatarezeroed. Resourcepoolorvirtualmachinephysicalpagesthatareshared. Machinepagesthataresavedbecauseofresourcepoolorvirtualmachine sharedpages. Currentspaceoverheadforresourcepool.SeeUnderstandingMemory Overheadonpage 136. Maximumspaceoverheadthatmightbeincurredbyresourcepoolorvirtual machine.SeeUnderstandingMemoryOverheadonpage 136. Currentspaceoverheadforauserworld.(Youmightseethisstatistic displayed,butitisintendedforVMwareuseonly.) GuestmemoryallocatedforaresourcepoolonNUMAnodex.Thisstatistic isapplicableonNUMAsystemsonly. VMMoverheadmemoryallocatedforaresourcepoolonNUMAnodex.This statisticisapplicableonNUMAsystemsonly.
Table A-6. Memory Panel Interactive Commands

Command M B N V Description SortresourcepoolsorvirtualmachinesbyGroupMappedcolumn. SortresourcepoolsorvirtualmachinesbyGroupMemctlcolumn. SortresourcepoolsorvirtualmachinesbyGIDcolumn.Thisisthedefaultsort order. Displayvirtualmachineinstancesonly.
VMware, Inc.
183
Storage Panels
Threestoragepanelsdisplayserverwidestorageutilizationstatistics. Thissectiondescribesthethreestoragepanels: StorageAdapterPanelonpage 184 StorageDevicePanelonpage 187 VirtualMachineStoragePanelonpage 190
Storage Adapter Panel

TheStorageAdapterpaneldisplaystheinformationshowninFigure A4.Statisticsare aggregatedperstorageadapterbydefault.Statisticscanalsobeviewedperstorage channel,target,LUN,orworldusingaLUN. Figure A-4. Storage Adapter Panel
Table A-7. Storage Adapter Panel Statistics
Column ADAPTR CID TID
Description Nameofthestorageadapter. StorageadapterchannelID.ThisIDisvisibleonlyifthecorresponding adapterisexpanded.Seetheinteractivecommandebelow. StorageadapterchanneltargetID.ThisIDisvisibleonlyifthecorresponding adapterandchannelareexpanded.Seetheinteractivecommandseanda below. StorageadapterchanneltargetLUNID.ThisIDisvisibleonlyifthe correspondingadapter,channelandtargetareexpanded.Seetheinteractive commandse,a,andtbelow. StorageadapterchanneltargetLUNworldID.ThisIDisvisibleonlyifthe correspondingadapter,channel,targetandLUNareexpanded.Seeinteractive commandse,a,t,and lbelow. Numberofchannels. Numberoftargets. NumberofLUNs. Numberofworlds. Numberofshares.
LID
WID
NCHNS NTGTS NLUNS NVMS SHARES
184
VMware, Inc.
Table A-7. Storage Adapter Panel Statistics (Continued)

Column BLKSZ AQLEN LQLEN WQLEN Description Blocksizeinbytes.ThisstatisticisapplicableonlytoLUNs. Storageadapterqueuedepth.MaximumnumberofESXServerVMkernel activecommandsthattheadapterdriverisconfiguredtosupport. LUNqueuedepth.MaximumnumberofESXServerVMkernelactive commandsthattheLUNisallowedtohave. Worldqueuedepth.MaximumnumberofESXServerVMkernelactive commandsthattheworldisallowedtohave.ThisisaperLUNmaximumfor theworld. Percentageofqueuedepth(adapter,LUNorworld)usedbyESXServer VMkernelactivecommands. RatioofESXServerVMkernelactivecommandsplusESXServerVMkernel queuedcommandstoqueuedepth(adapter,LUNorworld). NumberofcommandsintheESXServerVMkernelthatarecurrentlyactive. NumberofcommandsintheESXServerVMkernelthatarecurrentlyqueued. Numberofcommandsissuedpersecond. Numberofreadcommandsissuedpersecond. Numberofwritecommandsissuedpersecond. Megabytesreadpersecond. Megabyteswrittenpersecond. Averagedevicelatencypercommand,inmilliseconds. AverageESXServerVMkernellatencypercommand,inmilliseconds. Averagevirtualmachineoperatingsystemlatencypercommand,in milliseconds. Averagedevicereadlatencyperreadoperation,inmilliseconds. AverageESXServerVMkernelreadlatencyperreadoperation,in milliseconds. Averageguestoperatingsystemreadlatencyperreadoperation,in milliseconds. Averagedevicewritelatencyperwriteoperation,inmilliseconds. AverageESXServerVMkernelwritelatencyperwriteoperation,in milliseconds. Averageguestoperatingsystemwritelatencyperwriteoperation,in milliseconds.
%USD LOAD ACTV QUED CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/cmd GAVG/cmd DAVG/rd KAVG/rd GAVG/rd DAVG/wr KAVG/wr GAVG/wr
VMware, Inc.
185
Table A-7. Storage Adapter Panel Statistics (Continued)

Column QAVG/cmd QAVG/rd QAVG/wr ABRTS/s RESETS/s PAECMD/s PAECP/s SPLTCMD/s SPLTCP/s Description Averagequeuelatencypercommand,inmilliseconds. Averagequeuelatencyperreadoperation,inmilliseconds. Averagequeuelatencyperwriteoperation,inmilliseconds. Numberofcommandsabortedpersecond. Numberofcommandsresetpersecond. ThenumberofPAE(PhysicalAddressExtension)commandspersecond. ThenumberofPAEcopiespersecond. Thenumberofsplitcommandspersecond. Thenumberofsplitcopiespersecond.
Table A-8. Storage Adapter Panel Interactive Commands

Command e Description Toggleswhetherstorageadapterstatisticsaredisplayedexpandedor unexpanded.Allowsviewingstorageresourceutilizationstatisticsbroken downbyindividualchannelsbelongingtoanexpandedstorageadapter.You arepromptedfortheadaptername. Toggleswhetherstorageadapterstatisticsaredisplayedexpandedor unexpanded.Allowsviewingstorageresourceutilizationstatisticsbroken downbyworldsbelongingtoanexpandedstorageadapter.Doesnotrollupto adapterstatistics.Youarepromptedfortheadaptername. Toggleswhetherstorageadapterstatisticsaredisplayedexpandedor unexpanded.Allowsviewingstorageresourceutilizationstatisticsbroken downbypathsbelongingtoanexpandedstorageadapter.Doesnotrollupto adapterstatistics.Youarepromptedfortheadaptername. Toggleswhetherstoragechannelstatisticsaredisplayedexpandedor unexpanded.Allowsviewingstorageresourceutilizationstatisticsbroken downbyindividualtargetsbelongingtoanexpandedstoragechannel.Youare promptedfortheadapternameandthechannelID.Thechanneladapterneeds tobeexpandedbeforethechannelitselfcanbeexpanded. Toggleswhetherstoragetargetstatisticsaredisplayedinexpandedor unexpandedmode.Allowsviewingstorageresourceutilizationstatistics brokendownbyindividualpathsbelongingtoanexpandedstoragetarget. Youarepromptedfortheadaptername,thechannelID,andthetargetID.The targetchannelandadapterneedtobeexpandedbeforethetargetitselfcanbe expanded.
186
VMware, Inc.
Table A-8. Storage Adapter Panel Interactive Commands (Continued)

Command l Description Toggleswhetherpathisdisplayedinexpandedorunexpandedmode.Allows viewingstorageresourceutilizationstatisticsbrokendownbyindividual worldsutilizinganexpandedstoragepath.Youarepromptedfortheadapter name,thechannelID,thetargetID,andtheLUNID.Thepathtarget,channel, andadaptermustbeexpandedbeforethepathitselfcanbeexpanded. SortsbyReadscolumn. SortsbyWritescolumn. SortsbyMBreadcolumn. SortsbyMBwrittencolumn. SortsfirstbyADAPTRcolumn,thenbyCIDcolumnwithineachADAPTR, thenbyTIDcolumnwithineachCID,thenbyLIDcolumnwithineachTID, andfinallybyWIDcolumnwithineachLID.Thisisthedefaultsortorder.
r w R T N
Storage Device Panel

Thestoragedevicepaneldisplaysserverwidestorageutilizationstatistics.Bydefault, theinformationisgroupedperstoragedevice.Youcanalsogroupthestatisticsper path,perworld,orperpartition. Figure A-5. Storage Device Panel
Table A-9. Storage Device Panel Statistics

Column DEVICE PATH WORLD Description Nameofthestoragedevice. Pathname.Thisnameisvisibleonlyifthecorrespondingdeviceisexpanded topaths.Seetheinteractivecommandpbelow. WorldID.ThisIDisvisibleonlyifthecorrespondingdeviceisexpandedto worlds.Seetheinteractivecommandebelow.Theworldstatisticsareper worldperdevice. PartitionID.ThisIDisvisibleonlyifthecorrespondingdeviceisexpanded topartitions.Seeinteractivecommandtbelow.
PARTITION
VMware, Inc.
187
Table A-9. Storage Device Panel Statistics (Continued)

Column NPH NWD NPN SHARES BLKSZ NUMBLKS DQLEN WQLEN Description Numberofpaths. Numberofworlds. Numberofpartitions. Numberofshares.Thisstatisticisapplicableonlytoworlds. Blocksizeinbytes. Numberofblocksofthedevice. Storagedevicequeuedepth.ThisisthemaximumnumberofESXServer VMkernelactivecommandsthatthedeviceisconfiguredtosupport. Worldqueuedepth.ThisisthemaximumnumberofESXServerVMkernel activecommandsthattheworldisallowedtohave.Thisisaperdevice maximumfortheworld.Itisvalidonlyifthecorrespondingdeviceis expandedtoworlds. NumberofcommandsintheESXServerVMkernelthatarecurrentlyactive. Thisstatisticisapplicableonlytoworldsanddevices. NumberofcommandsintheESXServerVMkernelthatarecurrently queued.Thisstatisticisapplicableonlytoworldsanddevices. PercentageofthequeuedepthusedbyESXServerVMkernelactive commands.Thisstatisticisapplicableonlytoworldsanddevices. RatioofESXServerVMkernelactivecommandsplusESXServerVMkernel queuedcommandstoqueuedepth.Thisstatisticisapplicableonlytoworlds anddevices. Numberofcommandsissuedpersecond. Numberofreadcommandsissuedpersecond. Numberofwritecommandsissuedpersecond. Megabytesreadpersecond. Megabyteswrittenpersecond. Averagedevicelatencypercommandinmilliseconds. AverageESXServerVMkernellatencypercommandinmilliseconds. Averageguestoperatingsystemlatencypercommandinmilliseconds. Averagequeuelatencypercommandinmilliseconds. Averagedevicereadlatencyperreadoperationinmilliseconds. AverageESXServerVMkernelreadlatencyperreadoperationin milliseconds.
ACTV QUED %USD LOAD
CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/cmd GAVG/cmd QAVG/cmd DAVG/rd KAVG/rd
188
VMware, Inc.
Table A-9. Storage Device Panel Statistics (Continued)

Column GAVG/rd QAVG/rd DAVG/wr KAVG/wr GAVG/wr QAVG/wr ABRTS/s RESETS/s PAECMD/s PAECP/s SPLTCMD/s SPLTCP/s Description Averageguestoperatingsystemreadlatencyperreadoperationin milliseconds. Averagequeuereadlatencyperreadoperationinmilliseconds. Averagedevicewritelatencyperwriteoperationinmilliseconds. AverageESXServerVMkernelwritelatencyperwriteoperationin milliseconds. Averageguestoperatingsystemwritelatencyperwriteoperationin milliseconds. Averagequeuewritelatencyperwriteoperationinmilliseconds. Numberofcommandsabortedpersecond. Numberofcommandsresetpersecond. NumberofPAEcommandspersecond.Thisstatisticisapplicableonlyto paths. NumberofPAEcopiespersecond.Thisstatisticisapplicableonlytopaths. Numberofsplitcommandspersecond.Thisstatisticisapplicableonlyto paths. Numberofsplitcopiespersecond.Thisstatisticisapplicableonlytopaths.
Table A-10. Storage Device Panel Interactive Commands

Command e Description Expandorrollupstorageworldstatistics.Thiscommandallowsyoutoview storageresourceutilizationstatisticsseparatedbyindividualworlds belongingtoanexpandedstoragedevice.Youarepromptedforthedevice name.Thestatisticsareperworldperdevice. Expandorrollupstoragepathstatistics.Thiscommandallowsyoutoview storageresourceutilizationstatisticsseparatedbyindividualpathsbelonging toanexpandedstoragedevice.Youarepromptedforthedevicename. Expandorrollupstoragepartitionstatistics.Thiscommandallowsyouto viewstorageresourceutilizationstatisticsseparatedbyindividualpartitions belongingtoanexpandedstoragedevice.Youarepromptedforthedevice name. SortbyREADS/scolumn. SortbyWRITES/scolumn. SortbyMBREAD/scolumn.
r w R
VMware, Inc.
189
Table A-10. Storage Device Panel Interactive Commands (Continued)

Command T N Description SortbyMBWRTNcolumn. SortfirstbyDEVICEcolumn,thenbyPATH,WORLD,andPARTITION column.Thisisthedefaultsortorder.
Virtual Machine Storage Panel

Thispaneldisplaysvirtualmachinecentricstoragestatistics.Bydefault,statisticsare aggregatedonaperresourcepoolbasisbydefault.Onevirtualmachinehasone correspondingresourcepool,sothepanelreallydisplaysstatisticsona pervirtualmachinebasis.Youcanalsoviewstatisticsonaperworld,ora perworldperdevicebasis. Figure A-6. Virtual Machine Storage Panel
Table A-11. Virtual Machine Storage Panel Statistics

Column ID GID NAME Device NWD NDV SHARES BLKSZ NUMBLKS Description ResourcepoolIDoftherunningworldsresourcepoolortheworldIDof therunningworld. ResourcepoolIDofrunningworldsresourcepool. Nameofrunningworldsresourcepoolornameoftherunningworld. Storagedevicename.Thisnameisvisibleonlyifcorrespondingworldis expandedtodevices.Seetheinteractivecommandibelow. Numberofworlds. Thenumberofdevices.Thisnumberisvalidonlyifthecorresponding resourcepoolisexpandedtoworlds Numberofshares.Thisstatisticisonlyapplicabletoworlds.Itisvalidonly ifthecorrespondingresourcepoolisexpandedtoworlds Blocksizeinbytes.Itisvalidonlyifthecorrespondingworldisexpanded todevices. Numberofblocksofthedevice.Itisvalidonlyifthecorrespondingworld isexpandedtodevices.
190
VMware, Inc.
Table A-11. Virtual Machine Storage Panel Statistics (Continued)

Column DQLEN Description Storagedevicequeuedepth.ThisisthemaximumnumberofESXServer VMkernelactivecommandsthatthedeviceisconfiguredtosupport.The displayednumberisvalidonlyifthecorrespondingworldisexpandedto devices. Worldqueuedepth.ThiscolumndisplaysthemaximumnumberofESX ServerVMkernelactivecommandsthattheworldisallowedtohave.The numberisvalidonlyifthecorrespondingworldisexpandedtodevices. Thisisaperdevicemaximumfortheworld. NumberofcommandsintheESXServerVMkernelthatarecurrently active.Thisnumberisapplicableonlytoworldsanddevices. NumberofcommandsintheESXServerVMkernelthatarecurrently queued.Thisnumberisapplicableonlytoworldsanddevices. PercentageofqueuedepthusedbyESXServerVMkernelactive commands.Thisnumberisapplicableonlytoworldsanddevices. RatioofESXServerVMkernelactivecommandsplusESXServerVMkernel queuedcommandstoqueuedepth.Thisnumberisapplicableonlyto worldsanddevices. Numberofcommandsissuedpersecond. Numberofreadcommandsissuedpersecond. Numberofwritecommandsissuedpersecond. Megabytesreadpersecond. Megabyteswrittenpersecond. Averagedevicelatencypercommandinmilliseconds. AverageESXServerVMkernellatencypercommandinmilliseconds. Averageguestoperatingsystemlatencypercommandinmilliseconds. Averagequeuelatencypercommandinmilliseconds. Averagedevicereadlatencyperreadoperationinmilliseconds. AverageESXServerVMkernelreadlatencyperreadoperationin milliseconds. Averageguestoperatingsystemreadlatencyperreadoperationin milliseconds. Averagequeuereadlatencyperreadoperationinmilliseconds. Averagedevicewritelatencyperwriteoperationinmilliseconds. AverageESXServerVMkernelwritelatencyperwriteoperationin milliseconds.
191
WQLEN
ACTV QUED %USD LOAD
CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/cmd GAVG/cmd QAVG/cmd DAVG/rd KAVG/rd GAVG/rd QAVG/rd DAVG/wr KAVG/wr
VMware, Inc.
Table A-11. Virtual Machine Storage Panel Statistics (Continued)

Column GAVG/wr QAVG/wr ABRTS/s RESETS/s Description Averageguestoperatingsystemwritelatencyperwriteoperationin milliseconds. Averagequeuewritelatencyperwriteoperationinmilliseconds. Numberofcommandsabortedpersecondinmilliseconds. Numberofcommandsresetpersecondinmilliseconds.
Table A-12. Virtual Machine Storage Panel Interactive Commands

Command e Description Expandorrollupstorageworldstatistics.Allowsyoutoviewstorageresource utilizationstatisticsseparatedbyindividualworldsbelongingtoagroup.You arepromptedtoenterthegroupID.Thestatisticsareperworld. Expandorrollupstoragedevice,thatisLUN,statistics.Allowsyoutoview storageresourceutilizationstatisticsseparatedbyindividualdevices belongingtoanexpandedworld.YouarepromptedtoentertheworldID. Displayvirtualmachineinstancesonly. SortbyREADS/scolumn. SortbyWRITES/scolumn. SortbyMBREAD/scolumn. SortbyMBWRTN/scolumn. Sortfirstbyvirtualmachinecolumn,andthenbyWORLDcolumn.Thisisthe defaultsortorder.
V r w R T N
192
VMware, Inc.
Network Panel
ThepanelshowninFigure A7displaysserverwidenetworkutilizationstatistics. Statisticsarearrangedbyportforeachvirtualnetworkdeviceconfigured.Forphysical networkadapterstatistics,seetherowcorrespondingtotheporttowhichthephysical networkadapterisconnected.Forstatisticsonavirtualnetworkadapterconfiguredin aparticularvirtualmachine,seetherowcorrespondingtotheporttowhichthevirtual networkadapterisconnected. Figure A-7. Network Panel
Table A-13. Network Panel Statistics

Column PORT UPLINK UP SPEED FDUPLX USED DTYP DNAME PKTTX/s PKTRX/s MbTX/s MbRX/s %DRPTX %DRPRX Description VirtualnetworkdeviceportID. Ymeansthecorrespondingportisanuplink.Nmeansitisnot. Ymeansthecorrespondinglinkisup.Nmeansitisnot. LinkspeedinMegaBitspersecond. Ymeansthecorrespondinglinkisoperatingatfullduplex.Nmeansitisnot. Virtualnetworkdeviceportuser. Virtualnetworkdevicetype.HmeansHUBandSmeansswitch. Virtualnetworkdevicename. Numberofpacketstransmittedpersecond. Numberofpacketsreceivedpersecond. MegaBitstransmittedpersecond. MegaBitsreceivedpersecond. Percentageoftransmitpacketsdropped. Percentageofreceivepacketsdropped.
VMware, Inc.
193
Table A-14. Network Panel Interactive Commands

Command T R t r N Description SortsbyMbTxcolumn. SortsbyMbRxcolumn. SortsbyPacketsTxcolumn. SortsbyPacketsRxcolumn. SortsbyPORTIDcolumn.Thisisthedefaultsortorder.
Using the Utilities in Batch Mode

Batchmodeallowsyoutocollectandsaveresourceutilizationstatisticsinafile.Torun inbatchmode,youmustfirstprepareforbatchmode. To prepare for running resxtop or esxtop in batch mode 1 2 3 Runresxtop(oresxtop)ininteractivemode. Ineachofthepanels,selectthecolumnsyouwant. Savethisconfigurationtoafile(bydefault~/.esxtop310rc)usingtheW interactivecommand.
To run resxtop or esxtop in batch mode 1 Startresxtop(oresxtop)toredirecttheoutputtoafile.Forexample:

esxtop -b > my_file.csv
Thefilenamemusthavea.csvextension.Theutilitydoesnotenforcethis,butthe postprocessingtoolsrequireit. 2 ProcessstatisticscollectedinbatchmodeusingtoolssuchasMicrosoftExceland Perfmon.
Inbatchmode,resxtop(oresxtop)doesnotacceptinteractivecommands.Inbatch mode,theutilityrunsuntilitproducesthenumberofiterationsrequested(see commandlineoptionn,below,formoredetails),oruntilyoukilltheprocessby pressingCtrl+c. ThecommandlineoptionsinTable A15areavailableinbatchmode.
194
VMware, Inc.
Table A-15. Command-Line Options in Batch Mode

Option a Description Showallstatistics.Thisoptionoverridesconfigurationfilesetupsand showsallstatistics.Theconfigurationfilecanbethedefault ~/.esxtop310rcconfigurationfileorauserdefinedconfigurationfile. Runsresxtop(oresxtop)inbatchmode. Loadauserdefinedconfigurationfile.Ifthe-coptionisnotused,the defaultconfigurationfilenameis~/.esxtop310rc.Createyourown configurationfile,specifyingadifferentfilename,usingtheWsinglekey interactivecommand.SeeInteractiveModeSingleKeyCommandson page 174forinformationaboutW. Specifiesthedelaybetweenstatisticssnapshots.Thedefaultisfive seconds.Theminimumistwoseconds.Ifadelayoflessthantwoseconds isspecified,thedelayissettotwoseconds. Numberofiterations.resxtop(oresxtop)collectsandsavesstatistics thisnumberoftimes,andthenexits. Thenameoftheremoteserverhosttoconnectto(required,resxtop only). Theportnumbertoconnecttoontheremoteserver.Thedefaultportis 443,andunlessthishasbeenchangedontheserver,thisoptionisnot needed.(resxtop only) Theusernametobeauthenticatedwhenconnectingtotheremotehost. Youarepromptedbytheremoteserverforapassword,aswell(resxtop only).
b c <filename>
n server portnumber
username
Using the Utilities in Replay Mode

Inreplaymode,resxtop(oresxtop)replaysresourceutilizationstatisticscollected usingvm-support.Seethevm-supportmanpage. Toruninreplaymode,youmustfirstprepareforreplaymode. To prepare for running resxtop or esxtop in replay mode 1 Runvm-supportinsnapshotmodeontheESXServerserviceconsole(ESXServer 3only). Usethefollowingcommand:
vm-support -S -d duration -i interval
Unzipanduntartheresultingtarfilesothatresxtop(oresxtop)canuseitin replaymode.
VMware, Inc.
195
To run resxtop or esxtop in replay mode Enterthefollowingatthecommandlineprompt:

resxtop -R <vm-support_dir_path>
AdditionalcommandlineoptionsarelistedinTable A16. YoudonothavetorunreplaymodeontheESXServerserviceconsole. Replaymodecanberuntoproduceoutputinthesamestyleasbatchmode(seethe commandlineoptionb,below). Inreplaymode,resxtop(oresxtop)acceptsthesamesetofinteractivecommandsas ininteractivemodeandrunsuntiltherearenomoresnapshotscollectedby vm-supporttobereadoruntiltherequestednumberofiterationsarecompleted(see thecommandlineoptionnformoredetails). Table A16liststhecommandlineoptionsavailableforresxtop(oresxtop)replay mode. Table A-16. Command-Line Options in Replay Mode
Option R a Description Pathtothevmsupportcollectedsnapshotsdirectory. Showallstatistics.Thisoptionoverridesconfigurationfilesetupsand showsallstatistics.Theconfigurationfilecanbethedefault ~/.esxtop310rcconfigurationfileorauserdefinedconfiguration file. Runsresxtop(oresxtop)inBatchmode. Loadauserdefinedconfigurationfile.Ifthe-coptionisnotused,the defaultconfigurationfilenameis~/.esxtop310rc.Createyourown configurationfileandspecifyadifferentfilenameusingtheW singlekeyinteractivecommand.SeeInteractiveModeSingleKey Commandsonpage 174forinformationaboutW. Specifiesthedelaybetweenpanelupdates.Thedefaultisfiveseconds. Theminimumistwoseconds.Ifadelayoflessthantwosecondsis specified,thedelayissettotwoseconds. Numberofiterations.resxtop(oresxtop)updatesthedisplaythis numberoftimesandthenexits.
b c <filename>
196
VMware, Inc.
Index
A
admission control 22, 23 CPU 127 DRS 112 HA 95, 113 resource pools 47 strict 60, 76, 86, 115, 117 with expandable resource pools 29 advanced attributes 145 CPU 146 HA 121 hosts 145 memory 147 NUMA 148 virtual machines 149 affinity CPU 38, 126 CPU, and hyperthreading 133 defined 80 memory, NUMA nodes 160 potential issues 127 rules, using 108 algorithms, NUMA 154 AMD Opteron-based systems 158 anti-affinity 80 applications CPU-bound 125 deploying 165 single-threaded 125 architecture, ESX Server 36
automation level 93 and DRS recommendations 60 distributed power management 68, 94 automation modes, virtual machines 114 available memory 15
B
ballooning, memory 141 best practices 163
C
cluster creation overview 92 cluster features, choosing 93 cluster resource pools 46 clusters adding hosts 31, 112 adding managed hosts 102, 118 adding unmanaged hosts 103, 118 adding virtual machines 111, 112 creating 30, 31, 89, 93, 95 customizing 30, 31 distributed power management 23, 30, 60, 68 DRS 57 DRS, adding hosts 102 HA 90 introduction 59 invalid 105 powering on virtual machines 112 prerequisites 89 processor compatibility 91 removing hosts 114 removing virtual machines 113
197
VMware, Inc.
resource pools 56 shared storage 91 shared VMFS volume 91 summary page 96 virtual machines 111 VirtualCenter failure 61 CPU admission control 127 advanced attributes 146 managing allocation 38 overcommitment 19 virtual machines 19 CPU affinity 38, 126 hyperthreading 133 NUMA 159 NUMA nodes 159 potential issues 127 CPU panel esxtop 176 resxtop 176 CPU Reservation 16 CPU Unreserved 16 CPU virtualization 39, 124 CPU.MachineClearThreshold 133, 146 CPU-bound applications 125 custom automation mode 114
D
delegation of control through resource pools 45 device drivers 37 disabled virtual machine 114 disk resources 35 distributed power management 30, 60 and admission control 23 automation level 68, 94 enabling 68
DNS 90, 120 short name 90 DRS action history 100 adding managed hosts 102 adding unmanaged hosts 103 admission control 112 affinity rules 108 automation level 93 clusters, adding hosts 102 custom automation mode 114 customizing virtual machines 114 disabled virtual machine 114 fully automatic 94 host removal and virtual machines 105 initial placement 60, 62 introduction 62, 101 load balancing 60 maintenance mode 71 manual 94 migration 60 migration recommendations 68 overview 60 partially automatic 94 recommendation dependencies 106 recommendations 105 recommendations page 99 recommendations, and automation level 60 reconfiguring 107 red clusters 85 rules 110 turning off 107 using together with HA 80 virtual machine migration 66 VMotion network 89 DRS clusters 70 DRS recommendations page 99 DRS Resource Distribution histograms 98
VMware, Inc.
198
Index
DRS rules 109 dual-processor virtual machine 19 dynamic load balancing, NUMA 155
E
emulation 124 entering Maintenance Mode 104 ESX Server architecture 36 memory allocation 138 memory reclamation 140 resource management 35 esxtop batch mode 194 command-line options 172 common statistics description 174 CPU panel 176 CPU panel single-key commands 179 CPU panel statistics 177 interactive mode 173 interactive mode command-line options 173 interactive mode single-key commands 174 invoking 172 memory panel 179 network panel 193 network panel statistics 193 order pages 175 performance monitoring 172 replay mode 195 statistics column 175 storage adapter panel interactive commands 186 storage adapter panel statistics 184 storage device panel interactive commands 189 storage device panel statistics 187 storage panels 184
examples expandable reservations 50 memory overhead 136 NUMA 160 red cluster 85 reservation 22 resource pools 26 shares 21 valid cluster using resource pools of type expandable 83 valid cluster, all resource pools of type fixed 82 yellow cluster 84 expandable reservations 29, 50 example 50
F
failover capacity 75 fully automatic DRS 94
G
grafted, resource pool 103
H
HA 117, 121 adding managed hosts 118 adding unmanaged hosts 118 admission control 95, 113 advanced attributes 121 and host power off 77 and traditional cluster solutions 72 best practices 166 customizing virtual machines 115 DNS connectivity 90 failover capacity 75 host network isolation 77 introduction 72, 117 iSCSI storage 115 migration with VMotion 77
VMware, Inc.
199
NAS storage 115 networking redundancy 167 NIC teaming 167 options 94 red clusters 86 redundant network paths 90 shared storage 90 turning off 121 using together with DRS 80 HA clusters adding hosts 118 maintenance mode 78 planning 75 high, shares 20, 21 histograms,DRS Resource Distribution 98 home nodes, NUMA 154 host network isolation 77 host resource pools 46 hosts adding to cluster 112 adding to DRS clusters 102, 103 adding to HA clusters 118 entering maintenance mode 104 losing resource pool hierarchy 103 memory use 139 removing and invalid clusters 105 removing and resource pool hierarchies 104 removing from clusters 114 resource information 14 hyperthreading 129, 130, 131 CPU affinity 133 CPU.MachineClearThreshold 133 disabling 40 disabling quarantining 146 performance implications 129 quarantining 133
hyperthreading modes any 132 internal 132 none 132
I
idle memory tax 138, 140 initial placement 60, 62 NUMA 154 invalid clusters, host removal 105 iSCSI storage HA 115 isolation response 115 default 119 isolation through resource pools 45
L
limit attribute 22 hyperthreading 132 pros and cons 22 resource pools 25 load balancing 60 DRS 30 migration recommendations 110 virtual machines, migration 66 logical processors 40, 128 low, shares 20, 21
M
maintenance mode 71, 104 entering 104 HA clusters 78 manual DRS 94 Mem.AllocGuestLargePage 147 Mem.AllocUseGuestPool 147 Mem.AllocUsePSharePool 147 Mem.BalancePeriod 147 Mem.CtlMaxPercent 147
200
VMware, Inc.
Index
Mem.IdleTax 140, 147 Mem.SamplePeriod 139, 147 Mem.ShareScanGHz 145, 147 Mem.ShareScanTime 145, 147 memory advanced attributes 147 available 15 managing allocation 39 overhead 42 reclaiming unused 140 service console 15, 40 sharing across virtual machines 144 virtual machines 18 virtualization basics 40 VMkernel memory 165 memory affinity, NUMA 160 memory balloon driver 141 memory idle tax 138, 140 Mem.IdleTax 147 memory overcommitment 42, 144 memory overhead 136 examples 136 Memory Reservation 17 memory sharing 42 Memory Unreserved 17 memory virtualization 39 migration recommendations 68 migration threshold 67 migration with VMotion, failure, and HA 77 multicore processors 128
NUMA advanced attributes 148 AMD Opteron-based systems 158 CPU affinity 159 CPU assignment 160 dynamic load balancing 155 example 160 home nodes and initial placement 154 introduction 152 manual controls 156 memory affinity 160 optimization algorithms 154 page migration 155 transparent page sharing 156 using with ESX server 151 NUMA scheduling 153 Numa.AutoMemAffinity 148 Numa.MigImbalanceThreshold 148 Numa.PageMigEnable 148 Numa.RebalanceCoresNode 148 Numa.RebalanceCoresTotal 148 Numa.RebalanceEnable 148 Numa.RebalancePeriod 148
O
Opteron 158 overcommitment 42, 144 overcommitted cluster 84 overhead 136 examples 136 overhead memory 42
N
NAS storage, HA 115 network resources 35 NIC teaming 167 normal, shares 20, 21
P
page migration, NUMA 155 partially automatic DRS 94 performance 36 CPU-bound applications 125 monitoring 54 performance monitoring, esxtop 172
201
VMware, Inc.
performance monitoring, resxtop 172 physical and logical processors 40 physical memory usage 139 physical processors 40 processors logical 40 multicore 128 physical 40 processor-specific behavior 125
Q
quarantining, hyperthreading 133
R
red clusters 85 red DRS cluster 85 red HA cluster 86 redundant network paths for HA 90 Remote CLI 171 reservation 16 attribute 21 example 22 hyperthreading 132 resource pools 25 reservation type 25 resource management best practices 163 concepts 33 resource pools 46 adding virtual machines 55 admission control 47 attributes, changing 54 clusters 56 creating 25, 26, 48 customizing 25, 26 delegation of control 45 DRS clusters 70 example 26 grafted 103
202
hierarchies, host removal 104 information 50 introduction 43, 44 isolation 45 performance 54 removing virtual machines 56 reservation type 25 resource allocation tab 52 root resource pool 44 siblings 44 summary tab 51 resources, reserving 20 restart priority 115 default 119 resxtop batch mode 194 common statistics description 174 CPU panel 176 CPU panel single-key commands 179 CPU panel statistics 177 interactive mode 173 interactive mode command-line options 173 interactive mode single-key commands 174 memory panel 179 network panel 193 network panel statistics 193 options 172 order pages 175 performance monitoring 172 replay mode 195 statistics column 175 storage adapter panel interactive commands 186 storage adapter panel statistics 184 storage device panel interactive commands 189
VMware, Inc.
Index
storage device panel statistics 187 storage panels 184 root resource pool 44 rules 109 deleting 110 disabling 110 DRS 110 editing 109 results 110
swap space 142, 144 Linux systems 142 Windows systems 142 swapping 142
T
threads 128 threshold, migration 67 traditional clustering solutions 73
S
SAN and HA 90 sched.mem.maxmemctl 141, 150 sched.mem.pshare.enable 150 sched.swap.dir 150 sched.swap.file 150 sched.swap.persist 150 secondary service console 168 server configuration for hyperthreading 131 service console memory use 15, 40 redundancy 168 shares 20 example 21 high 20 low 20 normal 20 ratio 20 resource pools 25 sharing memory 42 siblings 44 single-processor virtual machine 19 single-threaded applications 125 SMP virtual machines 125 standby mode 60, 63, 68, 72 stars, migration threshold 67 statistics, esxtop 174 statistics, resxtop 174
V
valid clusters 81 example 83 Virtual Infrastructure SDK 38 virtual machine attributes changing 23 shares, reservation, and limit 20 Virtual Machine File System (VMFS) 37, 77, 91 virtual machine migration 66 virtual machines adding during cluster creation 111 adding to cluster 111, 112 adding to resource pools 55 advanced attributes 149 assigning to a specific processor 126 automation modes 114 changing resource allocation 23 configuration file 92 CPU 19 creating (best practice) 164 customizing for DRS 114 customizing for HA 115 deploying (best practice) 164 deploying applications 165 deploying operating system 165 disabled (DRS) 114 dual-processor 19 host removal 105
VMware, Inc.
203
memory 18, 41 memory overhead 136 monitor 133 number of virtual processors 125 removing from cluster 113 removing from resource pools 56 resource allocation 18 single-processor 19 virtual memory 133 virtual memory in virtual machines 133 virtual processors per virtual machine 125 VMFS (Virtual Machine File System) 37, 77, 91 VMkernel 37 hardware interface layer 37 memory 165 resource manager 37
VMkernel port redundancy 168 VMM 37, 133 vmmemctl 141 Mem.CtlMaxPercent 147 sched.mem.maxmemctl 150 VMotion requirements 91
W
wake on LAN (WOL) 72 working set size 138
Y
yellow cluster 84
204
VMware, Inc.
Updates for the Resource Management Guide
LastUpdated:June12,2009 ThisdocumentprovidesupdatestotheESXServer3.5,ESXServer3iversion3.5, VirtualCenter2.5versionoftheResourceManagementGuide.Updateddescriptions, procedures,andgraphicsareorganizedbypagenumbersothatyoucaneasilylocate theareasoftheguidethathavechanges.Ifthechangespansmultiplesequentialpages, thisdocumentprovidesthestartingpagenumberonly. ThefollowingisalistofupdatestotheResourceManagementGuide: UpdatefortheFailoverCapacitySectiononPage 75 UpdatefortheSharedStorageSectiononPage 91 UpdatefortheSharedVMFSVolumeSectiononPage 91 UpdatefortheTopDRSResourceDistributionChartSectiononPage 98 UpdatefortheVirtualizationandProcessorSpecificBehaviorSectiononPage 125 UpdatefortheSettingAdvancedHostAttributesSectiononPage 145 UpdatefortheCPUPanelSectiononPage 177
Update for the Failover Capacity Section on Page 75

ThefirstnotethatappearsintheFailoverCapacitysectionYoucanallowthecluster topoweronvirtualmachinesevenwhentheyviolateavailabilityconstraints.Ifyoudo that,theresultisaredcluster,whichmeansthatfailoverguaranteesmightnolonger bevalidisincorrectandshouldbedisregarded.
Update for the Shared Storage Section on Page 91

IntheSharedStoragesection,informationonNASstoragesupportforsharedstorage isadded.Theexistingsentenceismodifiedasfollows:Sharedstorageistypicallyona storageareanetwork(SAN)butcanalsobeimplementedusingNASsharedstorage.
VMware, Inc.
Update1
Update for the Shared VMFS Volume Section on Page 91

InthenotethatappearsintheSharedVMFSVolumesection,thesentenceThis requirementnolongerappliesifallsourceanddestinationhostsareESXServer3.5or highershouldbequalifiedtosay...ESXServer3.5orhigherandusinghostlocalswap.
Update for the Top DRS Resource Distribution Chart Section on Page 98
ThedescriptionoftheTopDRSResourceDistributionChartclaims:Thischartisa histogramthatshowsthenumberofhostsontheXaxisandtheutilizationpercentage ontheYaxis. Thisisincorrectbecausetheaxesaretheoppositeofwhatisdescribed.Thesentence shouldread:ThischartisahistogramthatshowsthenumberofhostsontheYaxis andtheutilizationpercentageontheXaxis.
Update for the Virtualization and Processor-Specific Behavior Section on Page 125
TheVirtualizationandProcessorSpecificBehaviorsectionstates:Becauseofthe differentkernelversions,itisnotpossibletomigratevirtualmachinesinstalledona systemrunningoneprocessormodel(forexample,AMD)toasystemrunningona differentprocessor(forexample,Intel).Thissentenceshouldbequalifiedtosay: Becauseofthedifferentkernelversions,itisnotpossibletouseVMotiontomigrate virtualmachinesinstalledonasystemrunningoneprocessormodel(forexample, AMD)toasystemrunningonadifferentprocessor(forexample,Intel).
Update for the Setting Advanced Host Attributes Section on Page 145
TheprocedureTosetadvancedattributesforahostisincorrectandshouldbereplaced bythefollowingprocedure: To set advanced attributes for a host 1 2 3 4 IntheVIClientinventorypanel,selectthehosttocustomize. ClicktheConfigurationtab. IntheSoftwaremenu,clickAdvancedSettings. IntheAdvancedSettingsdialogbox,selecttheappropriateitem(forexample,CPU orMemory)andscrollintherightpaneltofindandchangetheattribute.
VMware, Inc.
Update2
Update for the CPU Panel Section on Page 177

TheentryinTable A3for%IDLEshouldalsosay:Thedifference,%WAIT%IDLE, oftheVCPUworldscanbeusedtoestimateguestI/Owaittime.TofindtheVCPU worlds,usethesinglekeycommandetoexpandavirtualmachineandsearchforthe worldNAMEstartingwithvcpu.TheVCPUworldsmightwaitforotherevents besidesI/Oevents,sothismeasurementisonlyanestimate.
VMware, Inc.
Update3
Update4
VMware, Inc.

Vi3!35!25 Resource MGMT

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Vi3!35!25 Resource MGMT

Uploaded by

Copyright:

Available Formats

Resource Management Guide

ESX Server 3.5, ESX Server 3i version 3.5 VirtualCenter 2.5

Resource Management Guide

Resource Management Guide Revision: 20090612 Item: VI-ENG-Q407-439

VMware, Inc. 3401 Hillview Ave. Palo Alto, CA 94304 www.vmware.com

Resource Management Guide

Valid,Yellow,andRedClusters 81 ValidCluster 81 YellowCluster 84 RedCluster 85

Resource Management Guide

Resource Management Guide

About This Book

Resource Management Guide

VMware Infrastructure Documentation

Abbreviations Used in Figures

About This Book

Technical Support and Education Resources

Online and Telephone Support

VMware Professional Services

Resource Management Guide

Getting Started with Resource Management

Resource Management Guide

Viewing Host Resource Information

To view information about a host 1 2 StartaVMwareInfrastructureClient(VIClient)andconnecttoaVirtualCenter Server. Intheinventorypanelontheleft,selectthehost. WiththeSummarytabselected,thepanelsdisplaythefollowinginformation aboutthehost.

Chapter 1 Getting Started with Resource Management

Fordetailedinformationaboutavailablememory,clicktheConfigurationtab,and selectMemory. Thepanelliststotalresources,howmuchisusedbyvirtualmachines,andhow muchisusedbytheserviceconsole(ESXServer3only).

Theamountofphysicalmemorythevirtualmachinescanuseisalwayslessthan whatisinthephysicalhostbecausethevirtualizationlayertakesupsome resources.Forexample,ahostwithadual3.2GHzCPUand2GBofmemorymight

Resource Management Guide

make6GHzofCPUpowerand1.5GBofmemoryavailableforusebyvirtual machines. 4 Fordetailedinformationabouthowthetwovirtualmachinesusethehosts resources,clicktheResourceAllocationtab.

YouseetheCPUReservationandMemoryReservation,howmuchofthe reservationisused,andhowmuchisavailable. NOTEIntheResourceAllocationtabshown,novirtualmachinesarerunning,so noCPUormemoryisused.Yourevisitthistabafterpoweringonavirtual machine. Thefieldsdisplaythefollowinginformation.

Chapter 1 Getting Started with Resource Management

Description Totalmemoryresourcesavailableforthishost. Ifavirtualmachinehasamemoryreservationbuthasnot yetaccesseditsfullreservation,theunusedmemorycanbe reallocatedtoothervirtualmachines.

Resource Management Guide

Table 1-1. Virtual Machine Attributes

Understanding Virtual Machine Resource Allocation

Chapter 1 Getting Started with Resource Management

Figure 1-2. Virtual Machine Memory Configuration

Thesystemalsopromptsforthenumberofvirtualprocessors(CPUs)iftheoperating systemyouhavechosensupportsmorethanone. Figure 1-3. Virtual CPU Configuration

Resource Management Guide

Reserving Host Resources

Virtual Machine Attributes: Shares, Reservation, and Limit

Chapter 1 Getting Started with Resource Management

Resource Management Guide

Chapter 1 Getting Started with Resource Management

Changing Virtual Machine Attributes

Resource Management Guide

Chapter 1 Getting Started with Resource Management

Becausethevirtualmachineshavenotbeenpoweredon,theReservationUsed fieldshavenotchanged. 10 PoweronVMMarketingandseehowtheCPUReservationUsedandCPU Unreservedfieldschange.

Creating and Customizing Resource Pools

Resource Management Guide

RPMarketing 2GHz, 1GB

Chapter 1 Getting Started with Resource Management

Createasecondresourcepool,RPMarketing: a b c LeaveSharesatNormalforCPUandmemory. SpecifyaReservationforCPUandmemory. ClickOKtoexit.

SelectthehostintheinventorypanelandclicktheResourceAllocationtab. Theresourcepoolshavebeenaddedtothedisplay.Inthetoppanel,the Reservationforthesecondresourcepoolhasbeensubtractedfromtheunreserved resources.Inthesecondpanel,resourcepoolinformation,includingtheresource pooltype,isnowavailable.

Resource Management Guide

Table 12isasummaryofthevaluesthatyoucanspecifyforaresourcepool. Table 1-2. Resource Pool Attributes

Chapter 1 Getting Started with Resource Management

Understanding Expandable Reservation

Figure 1-5. Admission Control with Expandable Resource Pools, Example 1

VM-M1, 1GHz 2GHz RP-KID

Resource Management Guide

Figure 1-6. Admission Control with Expandable Resource Pools, Example 2

VM-M1, 1GHz 2GHz RP-KID

Creating and Customizing Clusters

Chapter 1 Getting Started with Resource Management