Information Guide

VMware Distributed Power Management Concepts and Use

VMware ESX 3.5 and VMware vCenter Server 2.5
Consolidationofphysicalserversintovirtualmachinesthatsharehostphysicalresourcescanresultin significantreductionsinthecostsassociatedwithhardwaremaintenanceandpowerconsumption.VMware DistributedPowerManagement(VMwareDPM)providesadditionalpowersavingsbeyondthisinitialbenefit bydynamicallyconsolidatingworkloadsevenfurtherduringperiodsoflowresourceutilization.Virtual machinesaremigratedontofewerhostsandtheunneededESXhostsarepoweredoff.Whenworkload demandsincrease,ESXhostsarepoweredbackonandvirtualmachinesareredistributedtothem.VMware DPMisanoptionalfeatureofVMwareDistributedResourceScheduler(DRS). ThisinformationguideprovidesatechnicaloverviewofVMwareDPMoperationinESX3.5andvCenter Server2.5.ItisintendedforVMwarepartners,resellers,andcustomerswhowantdetailedinformationon VMwareDPMfunctionalityinthatrelease. Theguidecoversthefollowingtopics: VMwareInfrastructureandClusterServicesonpage 1 VMwareDPMUsageonpage 2 VMwareDPMOperationonpage 4 VMwareDPMAdvancedOptionsonpage 8 VMwareDPMandDatacenterMonitoringToolsonpage 10 VMwareDPMUsageScenarioonpage 11 Resourcesonpage 12

VMware Infrastructure and Cluster Services

OneofthekeymanagementconstructsinVMwareInfrastructure3,whichcomprisesESX3.5andvCenter Server2.5,isthecluster.GroupingmultipleESXhostsintoaclusterenablesyoutomanagethemasasingle computeresource.TheclusterservicesthatbringaboutthisbenefitincludeVMwareHighAvailability(HA), DRS,andVMwareDPM. VMwareHAhandleshostandvirtualmachinefailuresinaclusterofESXhosts.Itrespectsyoursettingsfor thedesiredpoliciesandtheassociatedresourcestobesetasideforusebyvirtualmachinesintheeventofa failure.HAimplementsmechanismsfordetectingproblemsandrestartingvirtualmachines.Thewhitepaper VMwareHighAvailability:Concepts,Implementation,andBestPractices(seeResourcesonpage 12fora link)presentsinformationonHAoperation.HAfailoverresourceconstraintsarerespectedbyDRSand VMwareDPM.

Copyright 2009 VMware, Inc. All rights reserved.

VMware Distributed Power Management Concepts and Use

DRSmanagestheallocationofresourcestoasetofvirtualmachinesrunningonaclusterofESXhostswiththe goaloffairandeffectiveuseofresources.DRSmakesvirtualmachineplacementandmigration recommendationsthatservetoenforceresourcebasedservicelevelagreements,honorsystemand userspecifiedconstraints,andmaintainloadbalancingacrosstheclusterevenasworkloadschange.Thebest practicespaperResourceManagementwithVMwareDRS(seeResourcesonpage 12foralink)provides materialonDRSusageandbestpractices. VMwareDPMsavespowerbydynamicallyrightsizingclustercapacityaccordingtoworkloaddemands. VMwareDPMrecommendstheevacuationandpoweringoffofESXhostswhenbothCPUandmemory resourcesarelightlyutilized.VMwareDPMrecommendspoweringESXhostsbackonwheneitherCPUor memoryresourceutilizationincreasesappropriatelyoradditionalhostresourcesareneededtomeetHAor userspecifiedconstraints.VMwareDPMexecutesDRSinawhatifmodetoensureitshostpower recommendationsareconsistentwiththeclusterconstraintsandobjectivesbeingmanagedbyDRS.VMware DPMwasintroducedasanexperimentalfeatureintheVMwareInfrastructurereleasethatincludesvCenter Server2.5andESX3.5.SeeResourcesonpage 12foralinktotheVMwareexperimentalsupportstatement.

VMware DPM Usage

ThecomprehensiveResourceManagementGuide(seeResourcesonpage 12foralink)istheprimaryuser guideforDRSandVMwareDPM.Thissectioncoversasubsetofthecontentsoftheguidetoprovidecontext forlatersectionsthatdescribedetailsoftheVMwareDPMalgorithm.

Hosts Entering and Exiting Standby

HostspoweredoffbyVMwareDPMaremarkedbyvCenterServerasbeinginstandbymode,indicatingthat theyareavailabletobepoweredonwheneverneeded.VMwareDPMoperatesonESX3.5hoststhatcanbe awakenedfromapoweredoff(ACPIS5)stateviawakeonLAN(WoL)packets.Thesepacketsaresentover theVMotionnetworkinginterfacebyanotherESX3.5hostinthecluster,soVMwareDPMkeepsatleastone suchhostpoweredonatalltimes.BeforeenablingVMwareDPMonanESXhost,itisimportanttomanually testtheexitstandbyprocedureforthathosttoensurethatitcanbepoweredonsuccessfullyviaWoL.This canbedoneusingtheVIClient.ForwakeonLANcompatibilityrequirements,seetheVMwareknowledge basearticleWakeOnLANCompatibilityasaPrerequisiteforDistributedPowerManagement(see Resourcesonpage 12foralink). ThereasonVMwareDPMevacuateshostsandpowersthemdowntotheACPIS5stateisthathoststypically use60percentormoreoftheirpeakpowerwhentotallyidle,sothepowersavingspossiblewiththisapproach aresubstantial.VacatingahostandplacingitinalightersleepstatethanS5,suchastheACPIS3 suspendtoRAMstate,canconsumeanorderofmagnitudemorepowerthanACPIS5becauseoftheneed tokeepthehostsRAMpoweredon.FormoreinformationontheACPIglobalpowerstates,seeAppendix: ACPIGlobalPowerStatesonpage 12orAdvancedConfigurationandPowerInterfaceSpecification(see Resourcesonpage 12foralink). AfterVMwareDPMhasdeterminedhowmanyhostsneedtoremainpoweredontohandletheloadandto satisfyallrelevantconstraints,andDRShasdistributedvirtualmachinesacrossthehostsinkeepingwith resourceallocationconstraintsandobjectives,eachindividualpoweredonhostisfreetohandlepower managementofitshardwaretorunthepresentedloadefficiently,withoutanyneedforVMwareDPM involvement.ForCPUpowermanagement,ESX3.5placesidleCPUsinC1haltstate,andfutureversionsof ESXwillexploitadditionalhostlevelpowersavingmechanisms.Hostlevelpowermanagementissynergistic withVMwareDPM.ItcanprovideadditionalpowersavingsbeyondVMwareDPM,thoughitcannotsaveas muchpowerasVMwareDPMdoesbypoweringhostscompletelydown.

Enabling and Disabling VMware DPM

VMwareDPMleveragesDRSwhenitautomaticallymigratesvirtualmachinesawayfromESXhoststhatare tobepoweredoff.ThusVMwareDPMrequiresthatyouenableDRSonclusterswhereitruns.ToenableDRS, rightclicktheclusterinvCenterServerandselectEditSettings.ChecktheboxlabeledEnableVMwareDRS intheClusterSettingsdialogbox.

Copyright 2009 VMware, Inc. All rights reserved.

VMware Distributed Power Management Concepts and Use

YoucanthenenableVMwareDPMontheDRScluster.InthesameClusterSettingsdialogbox,selectPower ManagementintheleftpaneasshowninFigure 1.VMwareDPMisdisabled(settoOff)bydefault.Enable VMwareDPMbyselectingeitherManualorAutomatic.Inmanualmode,executionofVMwareDPM recommendationsrequiresconfirmationbytheuser.Inautomaticmode,VMwareDPMrecommendationsare executedwithoutuserconfirmation.BydefaultthisVMwareDPMautomationsettingappliestoallhostsin theclusterasindicatedbytheUseClusterDefaultsettingforeachhost,butyoucanoverridethesettingona perhostbasis.Forexample,youshouldsetanyhostsintheclusterthatcannotbepoweredonviaWoLto Disabled.YoushouldalsosettoDisabledanyotherhoststhatyouneverwantVMwareDPMtopoweroff. OtherpossibleoverridesettingsareAlwaysManualandAlwaysAutomatic.Theseperhostoverridesettings aremeaningfulonlywhenVMwareDPMisenabledfortheclusterasawhole. Figure 1. Enabling VMware DPM

VMware DPM Recommendation Ratings

VMwareDPMpowerrecommendationsareassignedstarratingsfromonestartofivestars,withfivestars indicatingthehighestrecommendationpossible.Theseratingssignifytheexpectedimportanceofparticular recommendationsgiventhecurrentutilizationofESXhostsacrosstheclusterandgiventheconstraints,ifany, onpoweredoncapacity. Hostpoweroffrecommendationsareratedasonetofourstars.Ahigherstarratingforapoweroff recommendationsignifiesalargeramountofunderutilized,poweredoncapacityinthecluster.Hencethe recommendationwiththehigherstarratingpresentsamoreattractiveopportunityforpowersavings. Hostpoweronrecommendationsareratedasthreetofivestars.Poweronrecommendationsgeneratedto meetHAoroptionaluserspecifiedpoweredoncapacityrequirementsreceivefivestarratings.Poweron recommendationsproducedtoaddresshighhostutilizationareratedasthreeorfourstars,withthehigher starratingindicatingthatoverallhostutilizationisclosertosaturation. VMwareDPMrecommendationratingsarecomparedtotheconfiguredVMwareDPMrecommendation threshold,andthosebelowthethresholdarediscarded.Inmanualmode,theVMwareDPMpower recommendationsaredisplayedintheVIClient,allowingtheVMwareInfrastructureadministratortochoose whethertoexecutethemornot.Inautomaticmode,therecommendationsareexecutedautomatically.The defaultVMwareDPMrecommendationthresholdisonestar,meaningthatallVMwareDPM recommendationsmeetthethreshold.InvCenterServer2.5Update2andlater,youcanspecifytheVMware DPMrecommendationthresholdasonetofivestarsusingtheadvancedoptionPowerStarThreshold.ESX hostpoweronrecommendationsproducedbyDRSasprerequisitesforDRSmigrationsalsoreceivestar ratings.TheDRSstarratingsaredonotcorrespondtoVMwareDPMstarratingsatall,andtheyaregoverned bythesettingfortheDRSmigrationthreshold.
Copyright 2009 VMware, Inc. All rights reserved. 3

VMware Distributed Power Management Concepts and Use

VMware DPM Operation

ThegoalofVMwareDPMistokeeptheutilizationofESXhostsintheclusterwithinatargetrange,subjectto theconstraintsspecifiedbytheVMwareDPMoperatingparametersandthoseassociatedwithHAandDRS. VMwareDPMevaluatesrecommendinghostpoweronoperationswhentherearehostswhoseutilizationis abovethisrangeandhostpoweroffoperationswhentherearehostswhoseutilizationisbelowit.Although thisapproachmightseemrelativelystraightforward,therearekeychallengesthatVMwareDPMmust overcometobeaneffectivepowersavingsolution.Theseincludethefollowing: Accuratelyassessworkloadresourcedemands.Overestimatingcanleadtolessthanidealpowersavings. UnderestimatingcanresultinpoorperformanceandviolationsofDRSresourcelevelSLAs. Avoidpoweringserversonandofffrequently,evenifrunningworkloadsarehighlyvariable.Powering serversonandofftoooftenimpairsperformancebecauseitrequiressuperfluousVMotionoperations. Reactrapidlytosuddenincreaseinworkloaddemandssothatperformanceisnotsacrificedwhensaving power. Selecttheappropriatehoststopoweronoroff.Poweringoffalargerhostwithnumerousvirtualmachines mightviolatethetargetutilizationrangeononeormoresmallerhosts. Redistributevirtualmachineintelligentlyafterhostsarepoweredonandoffbyseamlesslyleveraging DRS. ThefollowingsubsectionsdescribeindetailthekeycomponentsoftheVMwareDPMdecisionmaking algorithmthataddressthesechallenges: EvaluatingUtilizationonpage 4describesthemethodVMwareDPMusestoevaluatehostutilization. HostPoweroffRecommendationsonpage 5andHostPoweroffCostBenefitAnalysisonpage 6 explaintheprocedureVMwareDPMusestodeterminewhenthereisexcesshostcapacity. HostPoweronRecommendationsonpage 7coverstheprocessforensuringthathostcapacityis poweredonwhenneeded. HostSortforPoweronorPoweroffEvaluationonpage 7describestheorderinwhichVMwareDPM evaluateshoststodeterminewhethertheyshouldbepoweredonorpoweredoff. VMwareDPMisrunaspartoftheperiodicDRSinvocation(everyfiveminutesbydefault),immediatelyafter thecoreDRSclusteranalysisandrebalancingstepiscomplete.DRSitselfmayrecommendhostpoweron operations,iftheadditionalcapacityisneededasaprerequisiteformigrationrecommendationstohonorHA orDRSconstraints,tohandleuserrequestsinvolvinghostevacuation(suchasmaintenancemode),ortoplace newlypoweredonvirtualmachines.

Evaluating Utilization
VMwareDPMevaluatestheCPUandmemoryresourceutilizationofeachESXhostandaimstokeepthe hostsresourceutilizationwithinatargetutilizationrange.VMwareDPMmaytakeappropriateactionwhen thehostsutilizationfallsoutsidethetargetrange.Thetargetutilizationrangeisdefinedas:
Target resource utilization range = DemandCapacityRatioTarget DemandCapacityRatioToleranceHost

Bydefault,theutilizationrangeis45percentto81percent(thatis,63percent18percent). EachESXhostsresourceutilizationiscalculatedasdemand/capacityforeachresource(CPUandmemory). Inthiscalculation,demandisthetotalamountoftheresourceneededbythevirtualmachinescurrently runningontheESXhostandcapacityisthetotalamountoftheresourcecurrentlyavailableontheESXhost. Avirtualmachinesdemandincludesbothitsactualusageandanestimateofitsunsatisfieddemand,to accountforcasesinwhichthedemandvalueisconstrainedbytheESXhostsavailableresources.IfanESX hostfacesheavycontentionforitsresources,itsdemandcanexceed100percent.VMwareDPMcomputes actualmemoryusageusingastatisticalsamplingestimateofthevirtualmachinesworkingsetsize.Italso computestheestimateofunsatisfieddemandformemoryusingaheuristictechnique.

Copyright 2009 VMware, Inc. All rights reserved.

VMware Distributed Power Management Concepts and Use

VMwareDPMcalculatesanESXhostsresourcedemandastheaggregatedemandoverallthevirtual machinesrunningonthathost.Itcalculatesavirtualmachinesdemandasitsaveragedemandoverahistorical periodoftimeplustwostandarddeviations(cappedatthevirtualmachinesmaximumdemandobserved overthatperiod).Usingavirtualmachinesaveragedemandoveraperiodoftime,ratherthansimplyits currentdemand,isintendedtoensurethatthedemandusedinthecalculationisnotanomalous.This approachalsosmoothsoutanyintermediatedemandspikesthatmightleadtopoweringhostsonandofftoo frequently.ThedefaultperiodoftimeVMwareDPMevaluateswhenitcalculatesaveragedemandthatmay leadtohostpoweronrecommendationsisthepast300seconds(fiveminutes).Whenitcalculatesaverage demandforhostpoweroffrecommendations,thedefaultperiodoftimeVMwareDPMevaluatesisthepast 2400seconds(40minutes).Thedefaulttimeperiodforevaluatinghostpoweronrecommendationsisshorter becauserapidreactionstopoweronhostsareconsideredmoreimportantthanrapidreactionstopoweroff hosts.Inotherwords,providingthenecessaryresourcesforworkloaddemandshasahigherprioritythan savingpower. IfanyhostsCPUormemoryresourceutilizationduringtheperiodevaluatedforhostpoweron recommendationsisabovethetargetutilizationrange,VMwareDPMevaluatespoweringhostson.Ifany hostsCPUandanyhostsmemoryresourceutilizationovertheperiodevaluatedforhostpoweroff recommendationsisbelowthetargetutilizationrangeandtherearenorecommendationstopowerhostson, VMwareDPMevaluatespoweringhostsoff.

Host Power-off Recommendations

IfthehostresourceutilizationevaluationdescribedinEvaluatingUtilizationonpage 4leadsVMwareDPM toevaluaterecommendingpoweringoffahosttoaddresslowutilization,VMwareDPMiteratesthroughthe poweredonhoststoselectoneormorehoststhatcanbepoweredofftobringthehostsutilizationlevelsback up,ideallyintothetargetutilizationrange.Theorderofiteratingthroughthepoweredonhostsisdescribed inHostSortforPoweronorPoweroffEvaluationonpage 7. Foreachpoweredonhost,VMwareDPMevaluatesitasacandidatehostandinvokesDRSonatheoretical scenarioinwhichthiscandidatehostispoweredoffinthecluster.Toquantifytheimpactthatpoweringoffa candidatehostwouldhaveinincreasingthenumberofhighlyutilizedhostsintheclusterorincreasinghost utilizationlevels,VMwareDPMcomputesalowscorevalueforeachresource(CPUandmemory).Wecall thesecpuLowScoreandmemLowScore.Eachvalueiscomputedasthesumoftheweighteddistancesbelow thetargetutilizationforonlythosehostsbelowthattarget.
{RESOURCE}LowScore = Sum across hosts below target utilization for RESOURCE of (targetUtilization - hostUtilization)

Inthiscalculation,RESOURCEisCPUormemory.Thesameformulaisusedforevaluatingbothresources. VMwareDPMcomparesthelowscorevaluesfortheclusterwithoutthecandidatehostpoweredofftothelow scorevaluesifthecandidatehostispoweredoff.IfeitherthecpuLowScorevalueorthememLowScorevalue isimprovedfortheclusterwiththecandidatehostpoweredoffandifthevaluesofcpuHighScoreand memHighScore(seeHostPoweronRecommendationsonpage 7)fortheresultingclusterarenotworsethan thatwiththehostkeptpoweredon,VMwareDPMgeneratesarecommendationtopoweroffthehost.This includesrecommendationstofirstmigratevirtualmachinesoffthathost.VMwareDPMcontinuestoiterate throughthepoweredoncandidatehostsforpoweroffevaluationaslongastheclustercontainsanyhosts belowthetargetutilizationrangeforCPUandmemoryresources. VMwareDPMalsoevaluatesthreeadditionalfactorsthataffectplacingahostinstandbymode. VMwareDPMdoesnotrecommendanyhostpoweroffoperations(henceVMwareDPMiseffectively disabled)iftheDRSmigrationrecommendationthresholdissettofivestars.Withthissetting,DRS generatesVMotionrecommendationsonlytoaddressconstraintviolationsandnottorebalancevirtual machinesacrosshostsinthecluster,meaningthatwhenVMwareDPMrunsDRSinwhatifmodeto evaluatetheimpactofpoweringonastandbyhost,DRSdoesnotproduceanynonmandatory recommendationstomovevirtualmachinestothosehosts. VMwareDPMrejectspoweringdownahostifit,byenteringstandbymode,wouldtakethepoweredon capacityoftheclusterbelowthespecifiedminimum,MinPoweredOnCpuCapacityand MinPoweredOnMemCapacity(seePoweredonCapacityOptionsonpage 9).

Copyright 2009 VMware, Inc. All rights reserved.

VMware Distributed Power Management Concepts and Use

VMwareDPMdoesnotpowerdownahostiftheconservativelyprojectedbenefitofplacingthathostinto standbymodedoesnotexceedbyaspecifiedmultiplierthepotentialriskadjustedcostofdoingso,as describedinHostPoweroffCostBenefitAnalysisonpage 6.

Host Power-off Cost-Benefit Analysis

WhenVMwareDPMpowersoffanESXhost,theoperationhasanumberofpotentialassociatedcostssuchas thefollowing: Costofmigratinganyrunningvirtualmachinesoffoftheassociatedhost,suchasthecostofCPUand memoryrequiredforVMotion Lossofthehostsresourcesduringthepowerdownandsubsequentpoweronoperations Powerconsumedduringthepowerdownperiod Lossofperformanceifthehostsresourcesbecomeneededtomeetdemandwhilethehostispoweredoff Powerconsumedduringthepowerupperiod Costsofmigratingvirtualmachinesbackontothehost. Foreachhostevaluatedforapoweroffrecommendation,VMwareDPMcomparesthesecosts,takinginto accountanestimateoftheirassociatedrisks,withaconservativeprojectionofthepowersavingsbenefitthat canbeobtainedbypoweringoffthehost.ThisanalysisstepiscalledVMwareDPMpoweroffcostbenefit. VMwareDPMcomparesthehostpoweroffbenefittothehostpoweroffcost,bothmeasuredasCPU resourcesandmemoryresources.VMwareDPMcostbenefitanalysisacceptsapotentialhostpoweroff recommendationonlyifthebenefitisgreaterthanorequaltothecostmultipliedbyPowerPerformanceRatio (defaultof40)forbothresources.(ThetermresourcereferstobothCPUandmemory.)Thepoweroffbenefit andpoweroffcostiscomputedasfollows. Thepoweroffbenefityouachievebypoweringoffthecandidatehostistheresourcesavingsduringthe estimatedtimewhennoadditionalhostsarerequiredforpoweron.Wecallthistimeestimate StableOffTime.Inotherwords,StableOffTimeistheestimateoftimewhilethecandidatehostisoffbefore theoverallutilizationoftheclusterisexpectedtosignificantlyincrease.Thepoweroffbenefitisactually computedastheaggregateofeachresourceovertheStableOffTime.StableOffTimeiscomputedas:
StableOffTime = ClusterStableTime - (HostEvacuationTime + HostPowerOffTime).

Inotherwords,StableOffTimeistheestimatedtimethatthevirtualmachinesutilizationwillremainstable (ClusterStableTime),minusthetimeneededtopoweroffthecandidatehost(HostEvacuationTime + HostPowerOffTime).Thetimetopoweroffahostincludesthetimetoevacuateitsvirtualmachinesaswell asconductanorderlyshutdown.ClusterStableTimeistheVMwareDPMcalculationofwhenthecluster willrequireoneormorehoststobepoweredontosatisfyincreasingresourceutilization.StableOffTimecan beequaltoorlessthanzerowhenClusterStableTimeisespeciallylow.Inthatcase,VMwareDPMdoesnot evaluatethecandidatehostforapoweroffrecommendationbecausenobenefitwouldberealized. Thepoweroffcostiscalculatedasthesummationofthefollowingestimatedresourcecosts: Migrationofvirtualmachinesoffofthehostbeforepoweringitoff UnsatisfiedresourcedemandduringhostpoweronattheendofClusterStableTime Migrationofvirtualmachinesbackontohostsafterthehostsarepoweredon Toestimatethelasttwopointsabove,VMwareDPMcomputesthenumberofhoststhatneedtobepowered onattheendofClusterStableTimeusingaconservativeprojectionthatthedemandofeachvirtualmachine willrisetoahighlevel.Specifically,thisprojectionisthemeanofeachvirtualmachinesactualutilizationover theprevious3600seconds(60minutes)plusthreestandarddeviations.

Copyright 2009 VMware, Inc. All rights reserved.

VMware Distributed Power Management Concepts and Use

Host Power-on Recommendations

IftheevaluationofhostresourceutilizationdescribedinEvaluatingUtilizationonpage 4leadsVMware DPMtoevaluaterecommendinghostpoweronoperationstoaddresshighutilization,VMwareDPMiterates throughthestandbyhoststoselectoneormorehoststhatcanbepoweredontobringthehostsutilization levelsbackdown,ideallyintothetargetutilizationrange.Theorderofiteratingthroughthestandbyhostsis describedinHostSortforPoweronorPoweroffEvaluationonpage 7. Foreachstandbyhost,VMwareDPMevaluatesitasacandidatehostandinvokesDRSonatheoretical scenarioinwhichthiscandidatehostispoweredoninthecluster.VMwareDPMcomputesahighscorevalue foreachresource,calledcpuHighScoreandmemHighScore,inthesamefashionasitcomputedthelowscore values(seeHostPoweroffRecommendationsonpage 5).Itcomputeseachvalueasthesumoftheweighted distanceabovethetargetutilizationforonlythosehostsabovethattarget.IfeitherthecpuHighScorevalue orthememHighScorevalueisstablyimprovedfortheclusterwiththecandidatehostpoweredon,VMware DPMgeneratesapoweronrecommendationforthatcandidatehost. Incomparingthehighscorevalues,ifthememoryresourceisovercommittedonhostsinthecluster,VMware DPMgivesreductioninmemoryutilizationhigherimportancethanitgivesimpactonCPUresources. VMwareDPMcontinuestoiteratethroughthecandidatehostsforpoweronevaluationaslongasthereare anyhostsintheclusterexceedingthetargetutilizationrangeforeitherCPUormemoryresources.For efficiencypurposes,VMwareDPMskipsoveranycandidatehostthatitfindsequivalenttoanothercandidate hostthatwasrejectedinthesameroundforpoweronevaluation.Thecurrentcandidatehostisequivalentto apriorcandidatewhentheyarebothVMotioncompatiblewitheachotherandwhenthecurrenthosthasthe sameorfewerCPUandmemoryresourcesasthepriorhost. VMwareDPMthenrecommendspoweringonanyadditionalhostsneededtoreachaminimumamountof poweredonCPUormemoryresources,whichisthemaximumof: AnyvaluesspecifiedbyVMwareHA Valuessetbytheuser Valuesdefinedbydefault ThedefaultminimumpoweredonCPUandmemoryresources(seePoweredonCapacityOptionson page 9)are1MHzand1MB,respectivelythatis,atleastonehostintheclusteriskeptpoweredon.Hosts poweredonsolelytoreachaspecifiedminimumamountofCPUormemoryresourcesarenotneededto accommodatethevirtualmachinescurrentlyrunningintheclusterandmaybeidle.

Host Sort for Power-on or Power-off Evaluation

Typically,morethanoneESXhostisevaluatedforVMwareDPMpoweronorpoweroffoperations.This meansthathostsmustbesortedintoaparticularorderforevaluation. ForbothVMwareDPMpoweronandpoweroffoperations,ESXhostsinVMwareDPMautomaticmodeare evaluatedbeforehostsinVMwareDPMmanualmode.HostsatthesameVMwareDPMautomationlevelare favoredinorderofCPUandmemorycapacity,withthemorecriticalresourcesortedbeforetheother. Largercapacityhostsarefavoredforpoweronoperationsandsmallercapacityhostsarefavoredfor poweroffoperations.Hostsatthesameautomationlevelandcapacityareevaluatedforpoweringoffinorder oflowervirtualmachineevacuationcost.Tiesaredecidedbyrandomizingtheorderforhostpoweroff operationstospreadtheselectionevenlyacrosshosts.Inthefuture,VMwareDPMmayevaluateotherfactors suchashostpowerefficiencyindetermininghostorderingforpoweronorpoweroffevaluation. TheorderinwhichhostsareevaluatedbyVMwareDPMdoesnotdeterminetheactualorderinwhichhosts areselectedforpoweronorpoweroffoperations.Thisissimplyanorderingofthecandidatehostsso VMwareDPMandDRScanevaluatethemasforpoweringonandoff.Acandidatehostmayberejectedfora numberofreasons,basedonDRSoperatingconstraintsandobjectives. Someexamplesituationslimitingselectionofahosttobepoweredoffincludeconstraintsthatmakeit impossibletoevacuateallvirtualmachinesfromacandidatehostorcasesinwhichthevirtualmachinestobe evacuatedcanbemovedonlytohoststhatthenbecometooheavilyutilized.

Copyright 2009 VMware, Inc. All rights reserved.

VMware Distributed Power Management Concepts and Use

Someexamplesituationslimitingselectionofahosttobepoweredonincludeconstraintsthatwouldprevent virtualmachinesfrommovingtoahostifitwerepoweredonorsituationsinwhichmovingthevirtual machinestoacandidatehostisnotexpectedtoreduceloadonthehighlyutilizedhostsinthecluster. Inaddition,VMwareDPMdoesnotstrictlyadheretoitshostsortorderifdoingsowouldleadtochoosinga hostwithcapacityfargreaterthanneeded,ifasmallercapacityhostthatcanadequatelyhandlethedemand isalsoavailable.

VMware DPM Advanced Options

ThedefaultsettingsofVMwareDPMadvancedoptionsareintendedtosupportperformantand powerefficientuseofclusterresources.YoucanchangevarioussettingsrelevanttoVMwareDPMoperation usingtheDRSadvancedoptionsinterface.Toaccesstheadvancedoptionsinterfacerightclicktheclusterin vCenterServerandselectEditSettings.SelectVMwareDRSintheleftpane,andthenclickAdvanced Optionsatthelowerrightcorner,asshowninFigure 2. Figure 2. Gaining Access to the Advanced Options

TheAdvancedOptionsdialogboxappears.Itallowsyoutoenteroptionnamesandoptionvalues.VMware providesexperimentalsupportforVMwareDPMaswellasfortheseadvancedoptions.VMwaredoesnot guaranteethattheseadvancedoptionswillbesupportedinfuturereleasesofDRSandVMwareDPM.

Utilization Options
TheutilizationoptionsshowninTable 1settheCPUandmemoryresourceutilizationtargetsforeachESX hostoveraperiodoftime. Table 1. Utilization Options and Settings
Option Name DemandCapacityRatioTarget Description:UtilizationtargetforeachESXhost. DemandCapacityRatioToleranceHost Description:TolerancearoundtheutilizationtargetforeachESXhost. Usedtocalculatetheutilizationrange. 18% Note:Defaultutilization rangeis6318. 10to40 Default Value 63% Value Range 40to90

Copyright 2009 VMware, Inc. All rights reserved.

VMware Distributed Power Management Concepts and Use

Table 1. Utilization Options and Settings

Option Name VmDemandHistorySecsHostOn Description:PeriodofdemandhistoryevaluatedforESXhostpoweron recommendations. VmDemandHistorySecsHostOff Description:PeriodofdemandhistoryevaluatedforESXhostpoweroff recommendations. 2400seconds 0to3600 Default Value 300seconds Value Range 0to3600

Power-off Cost-Benefit Options

ThepoweroffcostbenefitoptionsshowninTable 2enableandsettheparametersusedtoanalyzewhento poweroffESXhosts. Table 2. Power-off Cost-Benefit Options and Settings
Option Name EnablePowerPerformance Description:Setto1toenablehostpoweroffcostbenefitanalysis.Setto0todisable it. PowerPerformanceRatio Description:Multiplierbywhichbenefitmustmeetorexceedperformanceimpact. Usedbypoweroffcostbenefitanalysis. PowerPerformanceHistorySecs Description:Periodofdemandhistoryevaluatedforpoweroffcostbenefit recommendations. 3600seconds 0to3600 40 0to500 Default Value 1 Value Range 0or1

Powered-on Capacity Options

YoudonotneedtospecifyaminimumamountofpoweredoncapacityforcorrectDRSclusteroperation.DRS andVMwareDPMrecommendthatappropriatehostsbepoweredonwhenneededandkeephostspowered ontorespectHAfailoversettings.YoucanusetheadvancedoptionsinTable 3tospecifythataparticular minimumamountofCPUormemorycapacity,orboth,bekeptpoweredon,evenwhenthatcapacityisnot deemednecessarybyDRSandVMwareDPM.Thehostcapacitykeptpoweredontosatisfytheseoptionsis notnecessarilycompatiblewiththefutureneedsofsomearbitraryvirtualmachine(forexample,itmaynot matchtherequiredCPUcharacteristics),sotheseoptionsaremostusefulinclustersofsimilarhoststhatare compatiblewiththemajorityofvirtualmachines. Table 3. Powered-on Capacity Options and Settings
Option Name MinPoweredOnCpuCapacity Description:MinimumamountofpoweredonCPUcapacitymaintainedbyVMware DPM. MinPoweredOnMemCapacity Description:Minimumamountofpoweredonmemorycapacitymaintainedby VMwareDPM. PowerStarThreshold Description:StarratingthresholdforVMwareDPMpowerrecommendations. SupportedinvCenterServer2.5Update2andlater. 1star 1to5 1MB 1tomaxint Default Value 1MHz Value Range 1tomaxint

Copyright 2009 VMware, Inc. All rights reserved.

VMware Distributed Power Management Concepts and Use

VMware DPM and Datacenter Monitoring Tools

Manydatacentersusemonitoringtoolstotrackthehealthofdatacenterentitiessuchasservers,switches, storage,andcoolingunits.Somecommonmonitoringtoolsare: BMCPerformanceManager(formerlyPATROL) HPSoftware(formerlyHPOpenView) CAUnicenter MicrosoftSystemCenterOperationsManager IBMTivoli Thesemonitoringtoolsaretypicallysettoprovideanalarmifaserverfails.Thesealarmsarecriticalin informingdatacenteradministratorsofpotentialproblemsthattheymayneedtoaddress. ServershutdownscontrolledbyVMwareDPMareinitiatedtoreducepowerconsumptionandmightfalsely appeartomonitoringtoolsasserverfailures.AlthoughVMwareDPMserverpoweroffoperationsare plannedevents,whereasserverfailuresareunplannedevents,somemonitoringtoolsmightnotbeableto distinguishbetweenthem.YouprobablydonotwantalarmstobetriggeredbyVMwareDPMactions. ToavoidunnecessaryalarmstriggeredbyVMwareDPMactions,youcanimplementthefollowing monitoringlogic: WhenanESXhostispoweredoffintostandbymode(ACPIS5softoff),suppressallalarmsrelatedtothe hostbeingdownorunavailable.Alternatively,markthesealarmsasnotrequiringanyactionoras informationalonly. YoucanimplementthislogicusingVMwareInfrastructureAPI2.5.Specifically,youcanobtain informationonVMwareDPMactionsbysubscribingtovCenterServereventsrelatedtoVMwareDPM. Table 4describestherelevanteventsingreaterdetail. TriggeranalarmwhenVMwareDPMattemptstopoweronahostandfails.Youshouldrespondby poweringthehostonusingadifferentmethodandthendisablingVMwareDPMonthathostuntilyou haveresolvedtheissue. vCenterServereventsupportforhostpoweronfailureswillbedeliveredinafuturerelease.Atthattime, youcanimplementthispartofthemonitoringlogic. Table 4. vCenter Server Events Related to VMware DPM
Data Object Name DrsEnteredStandbyModeEvent Data Object Description ThiseventrecordsthatVMwareDPMhassuccessfullybroughtthehostintostandby mode.Ahostinthismodehasnorunningvirtualmachinesandnoprovisioning operationsareoccurring. ThiseventrecordsthatVMwareDPMhassuccessfullybroughtthehostoutof standbymode.

DrsExitedStandbyModeEvent

Formoreinformationonusingthesedataobjects,seetheVMwareAPIsandSDKsDocumentationpageonthe VMwareWebsite.SeeResourcesonpage 12foralink.

Copyright 2009 VMware, Inc. All rights reserved.

10

VMware Distributed Power Management Concepts and Use

VMware DPM Usage Scenario

VMwareDPMenablesyoutopoweronandshutdownhostsautomatically.Theproperconfigurationand usageofVMwareDPMdependonyourpowerconsumptionandservicelevelgoals. ThebasicwaytouseVMwareDPMistopoweronandshutdownESXhostsbasedonutilizationpatterns duringatypicalworkdayorworkweek.Forexample,servicessuchasemail,fax,intranet,anddatabase queriesareusedmoreduringtypicalbusinesshoursfrom9a.m.to5p.m.Atothertimes,utilizationlevelsfor theseservicescandipconsiderably,leavingmostofthehostsunderutilized.Theirmainworkduringtheseoff hoursmightbeperformingoffhoursbackup,archiving,andservicingoverseasrequests.Inthiscase, consolidatingvirtualmachinesontofewhostsandshuttingdownunneededhostsduringoffhoursreduces powerconsumption. Toachievethisusagescenario,considertakingoneofthefollowinggeneralapproaches: SetVMwareDPMtoautomaticmodeandlettheVMwareDPMalgorithmdictatewhenhostsare poweredonandshutdown.Keeptheadvancedoptionssetattheirdefaultvalues. TuneVMwareDPMtobemoreconservativeormoreaggressivebychangingthevaluesof PowerStarThresholdorDemandCapacityRatioTarget.IfVMwareDPMperformshostpoweroffand poweronoperationstooaggressively,increasePowerStarThresholdforexample,fromthedefault valueofonestartofourstars.VMwareDPMwouldthenrecommendonlythemostimportanthost poweroffandpoweronoperations. Tosavemorepowerbyincreasinghostutilization(thatis,consolidatingmorevirtualmachinesontofew hosts),increasethevalueofDemandCapacityRatioTargetfromthedefaultof63percentto,forexample, 70percent.FromtheguidelinesinEvaluatingUtilizationonpage 4,thetargetresourceutilizationrange becomes52percentto88percent.Usingthemodifiedsetting,whenutilizationexceeds88percent,more hostsareneeded.Whenutilizationdipsbelow52percent,VMwareDPMcanconsolidateworkloadsonto fewerhostsandallowunneededhoststobeshutdown. UseVMwareDPMtoforcethepoweringonofallhostsbeforebusinesshoursandthenselectivelyshut downhostsafterthepeakworkloadperiod.Thisisamoreproactiveapproachthatwouldavoidany performanceimpactofwaitingforVMwareDPMtopoweronhostsinresponsetosuddenspikesin workloaddemand. Totakethisapproach,useVMwareInfrastructureAPI2.5toscheduleataskthatsetsthe MinPoweredOnCpuCapacityadvancedoptiontothefullclusterCPUcapacityinadvanceofbusiness hours.ThiscausesVMwareDPMtopowerhostsonbeforetheinitialspikeatthestartofbusiness. ScheduleanothertasktoresetMinPoweredOnCpuCapacityshortlyaftertheinitialpeakperiod,settingit backtoitspreviousvaluetorestorenormalVMwareDPMbehaviorfortheremainderofbusinesshours. Or,moreconservatively,restorenormalVMwareDPMbehaviorattheendofbusinesshourstoallowhost tobeshutdownonlyduringoffhours. TheVMwareDistributedPowerMgmt(DPM)video(seeResourcesonpage 12foralink)demonstrates theeffectsofpoweringoffhostsduringlowutilizationhoursandpoweringthembackonduringbusiness hours.Thedemonstrationshowsvirtualmachinesrunningatfullperformancewhilethedatacentersaves55 percentinoverallpowerusingdefaultVMwareDPMadvancedoptionsettings.Youcanautomatehost poweronandpoweroffoperationswithVMwareDPMinautomaticmodeoronascheduledbasis.

Copyright 2009 VMware, Inc. All rights reserved.

11

VMware Distributed Power Management Concepts and Use

Appendix: ACPI Global Power States

Table 5showstheglobalpowerstatesdefinedintheACPIspecification.SeeResourcesonpage 12foralink tothefullspecification. Table 5. ACPI Global Power States
Global System State G0(S0)Working G1(S1S4)Sleeping Power Consumption Large Smaller Software Runs Yes No OS Restart Required No No Comments Normalrunningconditions. S1:CPUcachesflushedandexecution halted.Powerismaintained. S2:CPUoff. S3:SuspendtoRAM. S4:Hibernation. G2(S5)Softoff Verynear0 No Yes SimilartoG3,butsomecomponents remainpoweredontoenable wakeonLAN. Safetopullpluganddisassemble.

G3MechanicalOff

Realtimeclock batteryonly

No

Yes

Resources
AdvancedConfigurationandPowerInterfaceSpecification http://www.acpi.info/DOWNLOADS/ACPIspec30b.pdf ResourceManagementGuide http://www.vmware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_resource_mgmt.pdf ResourceManagementwithVMwareDRS http://www.vmware.com/resources/techresources/401 VMwareAPIsandSDKsDocumentation http://www.vmware.com/support/pubs/sdk_pubs.html VMwareDistributedPowerMgmt(DPM)video http://www.youtube.com/watch?v=7CbRS0GGuNc VMwareexperimentalsupportstatement http://www.vmware.com/support/policies/experimental.html VMwareHighAvailability:Concepts,Implementation,andBestPractices http://www.vmware.com/resources/techresources/402 VMwareInfrastructure(VI)APIReferenceDocumentation http://www.vmware.com/support/developer/vcsdk/visdk25pubs/ReferenceGuide/ WakeOnLANCompatibilityasaPrerequisiteforDistributedPowerManagement http://kb.vmware.com/kb/1003373

If you have comments about this documentation, submit your feedback to: docfeedback@vmware.com VMware, Inc. 3401 Hillview Ave., Palo Alto, CA 94304 www.vmware.com Copyright 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware, the VMware boxes logo and design, Virtual SMP, and VMotion are registered trademarks or trademarks of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies. Revision: 20090120 Item: IN-073-PRD-01-01

12