EMC NetWorker Design

Best Practices with Data Domain

GlassHouse Whitepaper

Introduction
Written by:
Ken Ciolek and Natalie
Meek
GlassHouse
Technologies, Inc.

IntodaysITindustrywhereinformationiscrucialtocompanybusinessgrowth,
protectingthedataagainstlosshasbecomeverycritical.Evenwiththedevelopmentof
morecosteffectivetechnologies,completelyandeffectivelyprotectingwhatisinevitably
multiplecopiesofthesameinformationisamajorchallengeformanybusinesses.
Standardpracticesandtechnologiesarebeingpushedtotheirlimitsinmanybackup
environments,mosttoanunsuccessfulorinefficientoutcome.Thischallengeiscommon
formanyEMCNetWorkercustomers,andfindingtherightnewtechnologytobridge
thegapbetweencurrentstateanddesiredstateisoftenalong,difficultprocess.
DataDomainprovidesanalternativenearlinestoragesolutionforNetWorker
customerswhoarefacedwithneverendingdatagrowthandunabatedstorage
expansionassociatedwithballooningbackupandarchivedata.WhileNetWorkerisone
ofthemostscalabledataprotectionsolutionsavailabletothemarket,datagrowthand
dataretentionrequirementsdrivenearcontinualexpansiontoNetWorkerpools.
ThescopeofthiswhitepaperfocusesonhowtheDataDomaindeduplicationstorage
solutionintegrateswithstandardNetWorkerarchitecturalandoperational
environmentsinordertoovercomethegrowinggapbetweenwhatisactuallybeing
accomplishedandwhatneedstobeaccomplished.
.

Copyright2007GlassHouseTechnologies,Inc.Allrights
reserved

TheNetWorkerArchitectureandTerminology
TheEMCNetWorkerclient/serverenvironmentprovidestheabilitytoprotectyour
enterpriseagainstthelossofvaluabledata.Inanetworkenvironment,wherethe
amountofdatagrowsrapidlyasserversareaddedtothenetwork,theneedtoprotect
databecomescrucial.TheEMCNetWorkerproductgivesyouthepowerandflexibility
tomeetsuchachallenge.

Figure1:ClassicNetWorkerArchitectureDesign

Copyright2007GlassHouseTechnologies,Inc.Allrights
reserved

EachNetWorkerserverinstanceissupportedbymultipleselfmanagedrelationaldatabases(resource,
clientfileindexandmedia)andvariouslogs.Historically,customersbackedupdatadirectlytotapefor
nightlybackups.Alternatively,morecustomershavemovedtousingfileandadvancedfiletypedevices
tobackupdatatodiskforfasterwritespeeds.Thisdataisthenmigrateddailytophysicaltapeasa
replicaoftheoriginaldata(commonlyreferredtoasclonedata)forpurposesofdisasterrecoveryand
mediarecovery.Commonterminologyusedinthiswhitepaperisprovidedinthefollowingtable.

Term

Definition

Client

Acomputerwhosedatacanbebackeduporrecoveredbya
NetWorkerServer.

StorageNode

AClientwiththeability(viaSANorSCSIconnection)toback
updatatoalibraryoranotherstoragelocation.StorageNodes
canalsoreceivedatafromotherNetWorkerclients.

SANStorageNode

ANetWorkerClientwiththeability(viaSANorSCSI
connection)tobackupitsowndatatoalibraryoranother
storagelocation(SANStoragenodescannotbackupother
clients).

NetWorkerServer

ThecomputerrunningtheNetWorkerserversoftwarethat
containstheNetWorkerdatabases(clientfileindex,mediaand
resource)andprovidesbackupandrecoveryoperationstothe
clientsonthesamenetwork.Serverscan(andusuallydo)
functionasstoragenodes.

DataZone

Allcomputersadministeredforbackupandrecoverybya
NetWorkerserver.Therecanonlybe1NetWorkerserverina
datazone.

AdvancedFileTypeDevice

NetWorkertermfordiskbackuplocationswiththeabilityto
readandwritesimultaneously.Theadvancedfiletype
(adv_file)isusuallycreatedforlargedisklocationswiththe
abilitytodynamicallyextended(usingavolumemanager)if
thediskrunsoutofspacewhileinuseasanadvancedfiletype
device(usuallyduringbackup).

SaveSet

Agroupoffilesorlocationsbackedupontoadevice

Group

Oneormoreclientsconfiguredtobackupdataataspecific
timeoftheday.

Schedule

Thelevelofbackup(full,incremental,differential,skip)aclient
isdesignatedtorunduringabackup

BrowsePolicy

ThepolicytodeterminehowlongdatawillresideintheClient
FileIndex

Term

Definition

RetentionPolicy

Thepolicytodeterminehowlongdatacanberecovered.

ClientResource

Theclientresourceliststhesavesetstobebackedupona
client.Italsoincludestheschedule,directivebrowsepolicy,
andretentionpolicyfortheclient.Aclientcanhavemultiple
clientresourceshowever;theSavesetscannotcontainthesame
informationtobelocatedwithinthesamegroup.

Pools

AnattributeinNetWorkerthatallowsyoutoclassifydata.
Poolsareseparatedbythefollowinginformation(in
hierarchicalorder):Groups,clients,savesets,level.

Cloning

Theactofcopyingbackedupdata.Clonesfunctionidentically
astheoriginalbackupvolume.Cloningcanbecompleted
anywherefromonesavesetstoanentirevolume(ormultiple
volumes).

Staging

Movingdatafromonestoragetypetoanother.Stagingalso
laterremovesthedatafromitsoriginallocation.

FullBackup

Afullbackupisallfileswithinasaveset,regardlessof
whethertheyhavebeenmodified.

IncrementalBackup

Anincrementalbackuponlybacksupanyfilesthathave
changedsincethepreviousbackup.

DifferentialBackup

Differentialbackupisbasedonanumericalsystem.
NetWorkerhas9levelsofbackup.Levels1through9backsup
filesthathavechangedsincethelastlowernumberedbackup
level.Differentialbackupstypicallyareconfiguredtobackup
alldatathathaschangedsincethelastfullbackup.

ClientFileIndexDatabase

Thedatabasethetrackseveryfileorobjectthatisbackedup.
ThereisoneClientFileIndexperphysicalclient.

MediaDatabase

Thedatabasethattrackstheallsavesetandvolumelifecycle
information.ThereisoneMediadatabaseperdatazone

ResourceDatabase

Thedatabasethattracksallresourceinformation(suchas
clients,schedules,groups,etc).Thereisoneresourcedata
baseperdatazone
Table1:NetWorkerTerminology

TypicalNetWorkerChallenges
ThetypicalNetWorkerenvironmentsupportsahandfultothousandsofclients.NetWorkerscalesby
addingadditionalNetWorkerstoragenode/SANStoragenodes,andassociateddiskandtaperesources.
ThemostcommonchallengesinNetWorkerenvironmentsinclude:

Completingbackups,stagingandcloningwithlimitedtimeandphysicalresources

Contendingwithextremelylargeclientbackups(severalmillionfilesperclient)

ScalingNetWorkerdatabases,logs,mediamanagementandpoolstokeepupwithsheer
demand

Eliminatingredundantdatabackuplocations(multiplefull/incrementalcopiesofdatabases,
aggressivebackupretentionpolicies,etc.)

Eliminatingperformancebottlenecks(NetWorkerservertype,networking,clientissues,etc.)

Lackofcapacityplanningandreportingdisciplines

TechnologyOverview
DataDomainreducesunnecessaryNetWorkerdatastorageviainlinedatadeduplicationandtraditional
compression.Datadeduplicationisperformedonincomingdatastreamsandallowsonlythenew
segmentsofdatatobeidentifiedandstoredasuniqueinstanceswithintheDataDomainfilesystem.The
followingtablelistskeyterminologyforDataDomain.

Term

Definition

DataDomainSystem

AstandaloneDataDomainstorageappliance,gateway,ora
singlecontrollerinaDDXarray.

ProtectedDataSize

Thesumtotalofallfilesizesinthesetofprimarydatabeing
backedup.

LogicalStorageSize

ThetotalsizeofallbackupimagesinallpoolsonaData
Domainsystem.ThistotalsizeincludesallpoolsinNetWorker
mappedtoaDataDomainsysteminstance,whichcaninclude
primarydiskpools,andclonestoragepools.

DiskPoolDumpSize

Thesizeofanindividualbackupimagewrittentoapool(for
example,onenightsworthofbackupdata).

AddressableCapacity

TheamountofphysicalspaceavailableonaDataDomain
systemtostorededuplicatedandcompressedbackupimages.

PhysicallyConsumedStorage

TheamountofaddressablecapacityonaDataDomainsystem
currentlystoringbackupdataandassociatedmetadata.

CumulativeCompressionFactor

Theratioofthelogicalstoragesizetothephysicallystored

Term

Definition
space.

PeriodicCompressionFactor

Theratioofoneormorediskpooldumpstothephysically
consumedstorageforthosedumps.Notethattheperiodic
compressionfactoroveranyintervalbeyondthefirstfewdays
willtypicallyexceedthecumulativecompressionfactorbya
largemarginbecausethefirstversionofafilewrittentoaData
Domainsystemwillcompresslessthansubsequentversions.
Considerforexampletwoselectivebackupsof100GBof
protecteddataovertwonights:typicalperiodiccompression
factorsmightbe2:1thefirstnightand50:1thesecondnight,
butthecumulativecompressionfactorwouldonlybe~4:1(200
GB/50+2GB)ratherthanthe25:1orsoonemightexpect.
Notefurtherthatwhilethecumulativecompressionfactoris
whatdeterminescostperGB,itistheperiodiccompression
factorthatmostaffectsreplicationbandwidthrequirements.

Deduplication

Replacingredundant4KBto16KBsegmentsinincomingdata
streamswithverysmallreferencestoidenticalsegments
alreadystoredondisk.Alsoknownasglobalcompression.

LocalCompression

Standardlosslesscompressionalgorithms.TheavailableLocal
CompressionalgorithmsavailableonaDataDomainsystem
includeLZ(LempelZiv),gzandgzfast.

Cleaning

Aperiodicprocesstofindanyunreferencedsegmentsona
DataDomainsystemandmakethatspaceavailablefor
subsequentbackups.BecauseDataDomainsystemsnever
overwritedata,filedeletesbyaNetWorkerserverdonot
immediatelymakespaceavailableforsubsequentbackups
cleaningmustrunfirst.Thiscleaningprocessisnotuniqueto
DataDomainsystems.CleaningmaybeperformedonaData
Domainsystematthesametimeasbackup/restoreI/O,but
becausecleaningisafairlyresourceintensiveprocessitisbest
toscheduleitfornonpeakhours.Thedefaultschedulefor
cleaningisTuesdaymorningat6:00a.m.butmaybe
rescheduledforanyconvenienttimesduringtheweekor
manuallyviascriptorcommandline..
Table2:DataDomainTerminology

Note:DataDomainspatentedapproachtodeduplicationiscalledGlobalCompressionTMinData
Domainproductliterature,butforpurposesofthiswhitepaperwillbereferredtoasdeduplication.
DataDomaindatadeduplicationmethodsaremoregranularandvariablethanfixedsegmentsizedata
deduplicationcompetitors.DataDomainsegmentlengthisvariable,rangingfrom416KB.Thisisa
6

significantdifferentiatorfromcompetitiveproductswhichperformdeduplicationatthefilelevelorata
blocklevel,resultinginmoreefficientdeduplicationcapabilities.
Sincetherateofprimarydatachange(newlyintroducedunique4Kto16Ksegments)staysaboutthe
samefromnighttonightatmostsites,theamountofphysicallyconsumedstorageforaNetWorker
backupisroughlythesameasthephysicallyconsumedstorageforanincrementalNetWorkerbackup.
Theratioofprotectedstoragesizetoincrementallyconsumedphysicalstorageeachnightstaysaboutthe
same,buttheperiodiccompressionfactorofanincrementalbackupismuchlowerthantheperiodic
compressionfactorofaselectivebackup(becausetheformerismuchsmallerinsize).Asaresult,itis
veryinexpensivetoincludemanyversionsoffilesinastoragepoolonaDataDomainsystem.The
relativesizeofprotecteddataandincrementalbackupdata,beforeandafterdeduplicationand
compressionisillustratedinthefollowingfigure.

Figure2:BackupDataDeduplicationandCompression
Bothdeduplicationandstandarddatacompression(alsoreferredasLocalCompressioninproduct
literature)areexecutedvialosslesscompressionmethods(i.e.nodataintegrityimpact).LempelZiv(LZ)
compressionisstandard,howeverGZFastorGZarealternativesavailabletoeachDataDomainsystem
instanceforstandarddatacompression.Asalways,backupdatashouldnotbecompressedpriorto
attemptingadditionalcompressionatthedevicelevel.

DataDomainArchitectureandModels
AbaseDataDomainsystemsupportsacertaincapacityofaddressablestorage(postRAID,postspares).
Basedonbackuppolicy,thiswillenable10x30xmorelogicalcapacity.Forexample,asystemthatoffers
10TBofaddressablecapacitywouldoffer100TBto300TBoflogicalcapacity..
EachDataDomainsysteminstancesupports200MB/secaveragethroughput.Thisbasemetricapplies
bothtoreadandwriteoperations,asthearchitectureisoptimizedforsequentialI/O.

SystemName

PhysicalCapacity

LogicalDataStorage(TB)

MaximumI/O
Performance(GB/Hour)

12,800

(TB)

DDX(with16
arrays)

504

8,80020,000

DDX(with8
arrays)

252

4,40010,000

DD580

31.5

5501,250

800

DD565

23.5

400980

630

DD530

4.5

55140

360

DD510

2.25

2565

290

DD120

0.75

718

150

6,400

Table3:2007DataDomainSystems,AddressableandLogicalCapacity
Note:LogicalDataStorageValuesabovereflectdeduplicationandcompressioneffectsonbackupdata.
Theactualvaluesarehighlydependentonrateofchangeandbackuppolicies.
ThesolutionscalesmodularlybyincrementallyaddingeithercapacitytoanexistingDataDomainsystem
instanceinthecaseoftheDD580ortheDDX,oraddinganewDataDomainsysteminstancetothe
NetWorkerproductionenvironment.MultipleDataDomainsysteminstancescanberackedand
managedthroughanenterpriseconsole;howeverlogicalmanagementofeachDataDomainsystem
instanceisstillrequired.ThefollowingfigureillustratesDataDomainsystemarchitecturescalability.

Figure3:DataDomainSystemArchitectureScalability
FileSystemandVTLIntegration
DataDomainsystemssupporttwointegrationmethodswithNetWorker,eithervianetworkfilesystem
mountsorasastandaloneVirtualTapeLibrary(VTL).DataDomainsystemscanruninamixedmode
capacity,providingbothinterfacemethodsconcurrentlytooneormanyNetWorkerserverinstances.
ThisflexibilityaffordsagreatnumberofintegrationscenariosforNetWorker.Thefollowingfigure
illustratesbothintegrationscenarioswithNetWorkerServersandStorageNodes.

Figure4:NetWorkerDataDomainSystemIntegration
Fornetworkfilesystemaccess,NetWorkeraddressestheDataDomainsystemviaanativeNFSmountor
CIFS.NetWorkeraddressestheusablespaceexactlyasitwouldastandardfilesystemmountpoint
(NTFS,JFS,UFS,etc.).

TheVTLinterfaceemulatesaSTKL180tapelibrary,andrequiresafiberchannelconnectionalongwith
theappropriateNetWorkerdevicedriver.NDMPbackupsaresupportedwithaDDRattacheddirectlyto
theNAShostviaafibrechannelconnection.MultipleinstancesofVTLcanbecreatedperDataDomain
systeminstance.Upto64LTOtapedrives,10,000slots,and100,000virtualcartridgescanbecreatedper
DataDomainsysteminstance.AsastandaloneVTL,existingphysicaltaperesourcescanbeleveragedby
nativeNetWorkercapabilities.

Replication
AsynchronousdatareplicationissupportedbetweenDataDomainsysteminstances.Oncetheinitial
mirrorreplicaisestablished,onlychangestoindex/metadataandnewdatasegmentsarereplicatedtothe
10

targetsite.Asaresult,WANbandwidthrequirementsarereducedbyupto99%andtheamountoftime
toreplicatedatatoanoffsitelocationisreducedsignificantly
ReplicationisconfiguredinCollectionorDirectorymode.CollectionmodeallowssingleDataDomain
systeminstances,bothNFSandVTL,tobeconfiguredinasourcetargetrelationship,withoneway
replicationonly.Directoryreplicationsupportsmanytooneconfigurationswhichareestablishedatthe
directory/mountlevel.DirectoryreplicationsupportsbidirectionalreplicationbetweenDataDomain
systeminstances,whichisidealforvariousDRarchitectures,includinghubspokearchitecturesfor
remoteoffices.VTLinstancesemulateNFSDirectoryreplicationattheVTLpoollevelwhereoptionsare
settoindicatethesourceisaVTLpool.

Figure5:CollectionandDirectoryReplicationModes
HowDataDomainBestFitswithNetWorker
ADataDomainsystemprovidesanalternativefordiskandtapevolumepoolsinNetWorker.TheData
Domainfilesystemisoptimizedforsequentialreadandwriteoperations.Thisprovidesforagreatfit
withexistingNetWorkerdiskbasedorVTLabilities.ADataDomainsystemisbestconfiguredasan
advancedfiletypeinNetWorker.

11

NetWorkerdatabasesshouldcontinuetobeprovisionedontraditionaldiskdevices.ADataDomain
systemshouldnotbeusedforstoringactiveNetWorkerdatabases,logs,andconfigurationfiles.Instead,
theseNetWorkerelementscanbebackeduptoaDataDomainsystemforoperationalrecoveryand
replicatedtoaremotesitefordisasterrecovery.
Note:SomeNetWorkerenvironmentssupportextremelyhighperformancebackupsforhighvolume
clients.Typically,specializeddesignsareimplementedtosupportbackupsof14TB/hour.TheData
Domainsystemarchitecturecanbeconfiguredtosupporthighperformanceworkloads(viamultiple
savestreams)witheachDataDomainsysteminstancesupporting200MB/secaggregateworkloadsper
controlleroncurrentlyshippingDataDomainsystems.BecauseDataDomainsproductarchitectureis
CPUcentric,thisnumbertypicallychanges(upward)withnewproductreleasesinagivenpriceband.
ThetopendoftheDataDomaincontrollerline,withdualsocketIntelCPUcomponents,hasgonefrom
40MB/sec.(DD200inyear2004)tomorethan200MB/sec(DD580/ginyear2007),afactoroffiveincrease
overthreeyears.PleasecheckcurrentDataDomainliteratureforcurrentplatformnamesand
throughput.

Planning/SizingConsiderations
BackupPoliciesandDataRateofChange
NetWorkerpoliciesareuniquetoeachcustomerenvironment,buttypicallyfollowacommon
methodology.Mostsitesuseamixtureofincrementalbackupswithfullbackupsrunonaregular
schedule(weeklyormonthly).Incrementalbackupsaremoretypicalbecausetheyarefasterandtakeup
lessspaceonthebackupdevice.Thisleavesthefullbackupstakingupmorediskandtapespace.Every
fullbackupwillwriteredundantdatathatexistsinpreviousfullbackups,resultinginalargeamountof
thebudgetbeinglosttoconsumeddiskcapacity,moreandunnecessarytapestorage,offsitecharges,and
driveresourcecontention.
Theimpactofthisredundantfullbackupdatabecomesmuchlesssignificantwhendeduplicationis
introduced.DataDomainsystemsfacilitateSyntheticFullbackups,thegoalofwhichistocreateafull
backupimagefromexistingbackupdata.ThisprocessallowstheNetWorkerbackupenvironmentto
benefitfromanincrementalforevermethodologywithoutofficiallyadoptingsuchascheme.Intheend,
though,thechangerateofthedataisthefinalarbiterfortheamountofbackupdatastored.
FromaNetWorkerperspective,adatabasebackupmayappearnetneweachtimeitisbackedup,but
fromaDataDomainsystemperspective,theactualdatachangesmayresultinminimalnewphysical
storageconsumption.Databases,email,andunstructureddata(fileserverdata)willbenefitthemost
fromdatadeduplicationinmostproductionenvironments.Datagrowthissuesarecompoundedbynon
workingcopiesofdatausedforreportingortesting,allofwhicharetypicallybackedupdailyby
NetWorker.Thenetresultisaneverendingdemandforphysicalstorageresources.DataDomain
counterstheeffectsofuncontrolledbackupdatagrowth.
Deduplicationbenefitsarerealizedovertimeandeventuallyplateauoncethebackupversioningpolicy
andtheincrementalbackuptrafficisfullyrealized.Sincedatachangeratesvarybydatatypeand
productionenvironment,acombinationofbackuppolicies,datachangerate,anddatastructureimpacts
DataDomainsystemsizingestimates.

12

Sizing
Sizingstoragecapacitiesforadatadeduplicationsolutiontakesintoconsiderationactualdatachange
rates,whicharenotvisiblefromaNetWorkerperspectivesinceNetWorkerviewsdatachangeatthe
file/objectlevel.
TheratioofcurrentbackupdatatoDataDomainsystemstorage(afterdatadeduplication+compression)
varies,butNetWorkercustomerscanexpectonaverage20xdatareductionfortypicalbackup
environments.Again,thisratioiswhollydependentontherateofdatachange,backupmethodsbeing
used,andbackuppolicystandards.
Note:ForcustomersusingVTLmode,DataDomainsystemsdonotrequirespacepreallocationfor
virtualtapevolumes.AsvirtualtapevolumesaremountedandfilledintheDataDomainsystem,
physicalspaceisnothedgedforscratchvirtualtapevolumes.OtherVTLtechnologiespreallocate
physicaltapevolumes,regardlessofwhetherornottheyareempty,full,oractive.
AninitialsizingmetricfortheDataDomainsystemistoestimatetheprimaryvolumeofdataonall
backupclients,ora1:1ratioofprimarydata(nottobeconfusedwithbackupdata)totheDataDomain
systemaddressablecapacity.ThenrunbackupsagainsttheDataDomainsystem.Dependingon
retentionpolicies,compressionrates,changerates,etc,threetosixmonthsofbackupscanberetainedon
disk.Asanexample,anenvironmentwith5TBofdatarequiringbackupswouldneedaDataDomain
systemwith5TBofstorage.Inthiscase,threetosixmonthsofbackupscouldbestoredonasingle5TB
DataDomainsystem.
Fornewcustomers,DataDomainrecommendsongoingcapacityplanning,throughadisciplineofsizing,
provisioning,utilization,andongoingmeasurement.Thisidealcapacityplanningmethodincludes
ongoingmeasurement,anddemandforecastingonceasubsetofproductionbackupdataissenttoa
NetWorkerserverusingaDataDomainsystem.Generally,thebenefitsofdatadeduplicationarerealized
overtime.Onlythen,dodatavolumesstabilizewithinDataDomainsysteminstances.
Note:AlthoughmostNetWorkerenvironmentslimittheeffectivenumberofbackupsavesetsdueto
typicalNetWorkersetup(forexample1yearretentionpolicy,weeklyfullbackupswhichequatesto52
backupsperyear),theimpactofadditionalbackupsavesetstoDataDomainsystemsisoftennegligible.
Whilethisbenefitallowsmoreflexibilityforaggressivebackuppolicies,NetWorkerdatabasesizeshould
remainakeyconsiderationinNetWorkerpolicyplanning.

13

Integrationplanning
ADataDomainsystemintegratesintoaNetWorkerenvironmentastheprimarystoragedestinationfor
directedbackups.Ineitherconfiguration,NASorVTL,theDataDomainsystemcantakeadvantageof
severalalternateagentsandmoreefficientmethodologies.Bothinstancestakeadvantageofthespeedof
diskandeasilyintegratewithapreviouslyconfiguredNetWorkerenvironmentasaVTLorasdisk,an
advancedfiletypedevicewithaNASconfiguration.

Figure6:NetWorkerSiteIntegration

14

Oneormoredirectoriescanbespecifiedasthelocationoffilesforanadvancedfiletypedevice.Asa
result,multipleDataDomainsysteminstancescanbededicatedtoasingleNetWorkerserver;however
thebenefitsofdatadeduplicationdonotspanmultipleDataDomainsysteminstancesatthistime.Pools
containinglikedatacanbemappedtoindividualDataDomainsysteminstances,ortoindividualmount
pointswithinaDataDomainsystem.Forexample,someNetWorkeradministratorscreateagroup
associatedtoadvancedfiletypedevice(andpool)forallfileservers,whereasignificantvolumeof
primaryunstructureddataisredundant.Mappingthisbackupgroupandassociatedstoragepool(s)toa
DataDomainsystemexpandsthebenefitsofdatadeduplicationaboveandbeyondredundantbackup
copies.Thefollowingfigureillustratesthisconcept.

Figure7:MappingDataTypestoStoragePoolsandDataDomainSystemInstances

15

Therearesignificantadministrativebenefitstomappingclientdatatypestopoolsmountedtospecific
directorieswithinaDataDomainsysteminstance.Thesebenefitsinclude:

Moregranularviewofcompressionstatistics,whichareavailableonadirectorybydirectory
basis

Abilitytoreplicatespecificdirectories

SimplifiedadministrationassociatedwithscalingtoadditionalDataDomainsysteminstances

IfaDataDomainsysteminstancereachescapacity,individualdirectoriescanbemigratedtoanewData
Domainsysteminstanceusingreplication,withminimalreconfigurationanddowntime.Forexample,if
allOracle,Exchange,andfileserverbackupsaremappedtoindividualDataDomainsystemdirectories,a
specificbackupdatatypecanbemigratedtoanewDataDomainsystemtoreducecapacityinthe
originalDataDomainsysteminstance,allowingforadditionalcapacityandgrowth.
LargeNetWorkerenvironmentscontendwithaneverendingchallengeofperformanceoptimizationfor
systemresources.ToavoidperformancebottleneckingandtoobtainoptimalDataDomainsystem
performance,multipleDataDomainsystemscanbedeployedtosupportparallelworkload/performance
demands.Thesamearchitecturalconceptappliestotraditionalfilesystemandphysicaldisklayout
planningforperformanceworkloads.

16

Figure8:LargeSiteNetWorkerIntegration

OperationalConsiderations
Whenyourinfrastructureisconfiguredoptimally,administeringaNetWorkerDataZoneisrelatively
painless.Whenanyaspectofthearchitecture,networking,firmwareversion,servercapacity,etc,is
misaligneditwillshowindailysuccessrates.ADataDomainsystemseamlesslyinstallsintoany
architectureandwillenableNetWorkeradministratorstomoreefficientlytakeadvantageofthe
resourcesalreadyinplace.
UtilizingtheadvantagesoftheadvancedfiletypedevicethataDataDomainsystempresentsallowsa
NetWorkeradministratortoefficientlymanageatimelinefordailyoperations,andsuccessfullymeetthe
timeline.Thesedailyoperationsconsistofthetypicalroutines,outlinedbythefollowingdiagram.

17

Figure9:NetWorkerDailyOperations
ADataDomainsysteminstancecangreatlyreduceboththebackupwindowandcloningprocess.With
NetWorkerstagingorcloningyoucaneasilyoffloadtotape.However,iftheDataDomainsystem
architectureisleveragedforreplicationtooffsitestorage,anystagingorcloningprocessescanbe
completelyeliminated.Eitherimplementationstrategysignificantlyreducestheamountoftimerequired
tocompletedailyoperationsasillustratedinthefollowinggraphic.

18

Figure10:NetWorkerDailyOperationswithDataDomain
CustomerswillneedtoconsidercapacityplanningandspacemanagementwhenimplementingData
DomainsystemswithNetWorker.However,unlikemanagingdiskarraysusedforbackupand
especiallytape,managementofDataDomaindeduplicationsystemsissimple,makingitmucheasierto
achieveefficientutilizationandensuresufficientavailablespace.
TheDataDomainsystemrequiresanoperationalprocesstoperformcleaningoftheDataDomainfile
system,whicheffectivelyreclaimsspaceandoptimizestheDataDomainfilesystemforperformance.The
defaultscheduleforcleaningistorunweekly,howeverdependingondatachangeratesandavailable
DataDomainsystemspacethisprocesscanbescheduledmorefrequently(23timesperweek).
Note:TheDataDomainsystemcleaningprocesscanberesourceintensive.Athrottlecommandis
availabletoassigntherelativeprioritytocleaningprocessestonormalbackupandrestoreI/O.We
recommendtheDataDomainsystemcleaningprocessesrunperiodicallyaspartofstandardNetWorker
operations,scheduledwhenbackup/restoreactivityisatminimumlevels.SeetheDDOSAdminGuide
formorediscussion.

19

RecoveryConsiderations
Withatapeonlybackupinfrastructure,restoreactivitiesarehamperedbyallofthelimitationsthatcome
withtryingtoreaddatastoredonamechanicaldevice.WiththeDataDomainsystemandinlinenative
deduplication,NetWorkersitescanleverageanincrementalforeverapproachforproductionbackups
withouthavingtochangetheircurrentmethodologies.Ifoffsitecloningisconfiguredtogototape,an
environmentwillstillsufferthedelayandchargesfortaperecallswhichputpressureonanyapplication
dataRTO(RecoveryTimeObjectives).IftheadministratortakesadvantageofDataDomainsabilityto
replicatetoanotherDataDomainsystematanalternatelocation,amuchshorterRTOisachievable.Data
DomaininlinedepulicationalsoprovidesbetterRPO(RecoveryPointObjectives)ascomparedtotypical
implementationsofpostprocesseddeduplication.
ThecombinationoftheNetWorkerAdvancedFileTypeDevicesandtheDataDomainsystemsisthevery
purposeofbackupstoenablefaster,morereliablerecoveries.Byrecoveringdataatthespeedofdisk,
NetWorkerAdvancedFileTypeDevicescandelivervastimprovementsinrecoveryperformanceand
reductionindowntimecosts.
DirectonsiterecoveryconditionsimprovefirstandforemostduetoaDataDomainsystembasedondisk
hardware.Diskismuchmorereliablethantraditionaltapemechanics,andoffersmuchfastersearchand
readtimeswithoutanydelaysduetorobotmounting/positioningfunctions.DataDomaintakesthisone
stepfurtherwithitsDataInvulnerabilityArchitecture,asystemdesignthatextendstheresiliencyofthe
systemwellbeyondthattypicallyfoundinthisclassofstorage.BackupsdirectlytoaDataDomain
systemalsoeliminatethecomplexityofmultiplexingandtheinconvenienceofincrementalbackupsthat
spanacrossmultiplephysicaltapevolumes.Consequently,multipletapemountswhichincreaserestore
operationtimesinlargerestoreoperationswillnotberequired.

20

IntegrationBasics
NetWorkerServerTuning
NetWorkerServertuningisrecommendedfornewDataDomainsystemimplementationsusing
NFS/CIFSandIPprotocol.
Note:SomeNetWorkerServerconfigurationchangesrequireaNetWorkerserverrefreshtotakeeffect.
ThefollowingtableoutlinesstandardservertuningparametersforoptimalDataDomainsystem
performancewithNetWorker,usingNFS/CIFSandIPprotocol.

ConfigurationType

Comments

AIXNetwork
Configuration

ifconfigen0tcp_recvspace65536tcp_sendspace65536tcp_nodelay1
tcp_nodelayack
sb_max

HPUXNetwork
Configuration

EnterthefollowingtwocommandsthenremounttheDataDomain
systemNFSsharetoenablethevalues:
nddset/dev/tcptcp_recv_hiwater_def262144
nddset/dev/tcptcp_xmit_hiwater_def262144

SOLARISSystemSettings

Createafile/etc/rc3.d/S90ddr.Enterthefollowingtwolinesinthefile:
nddset/dev/tcptcp_recv_hiwat131072
nddset/dev/tcptcp_xmit_hiwat131072
Inthefile/etc/system,addthefollowinglines:
setnfs:nfs3_max_threads=16
setnfs:nfs3_async_clusters=4
setnfs:nfs3_nra=16
setrpcmod:clnt_max_conns=1
setfastscan=131072
sethandspreadpages=131072
setmaxpgio=65536

LINUXServerSettings

echo40962621441048576>/proc/sys/net/ipv4/tcp_rmem

echo40962621441048576>/proc/sys/net/ipv4/tcp_wmem

echo262144>/proc/sys/net/core/rmem_max
echo262144>/proc/sys/net/core/wmem_max

21

ConfigurationType

Comments

echo262144>/proc/sys/net/core/rmem_default
echo262144>/proc/sys/net/core/wmem_default
WINDOWSNetwork
Configuration

Note:DonotmodifytheWindowsregistryparameterAFDifthe
NetWorkerserveroranyassociatedNetWorkerclientsaresupported
byWindowsNT4.0

1.OpenREGEDT32andnavigateto:
HKEY_LOCAL_MACHINE\SYSTEM\CURRENTCONTROLSET\SER
VICES\AFD\PARAMETERS
2.AddanewDWORDvaluetotheDefaultSendWindowkeyandset
thevalueto65536(decimal).
3.AddanewDWORDvaluetotheDefaultReceiveWindowkeyandset
thevalueto65536(decimal).
4.WithinREGEDT32,navigatetothefollowinglocation:
HKEY_LOCAL_MACHINE\SYSTEM\CURRENTCONTROLSET\SER
VICES\TCPIP\PARAMETERS
5.AddanewDWORDvaluetotheGlobalMaxTcpWindowSizekeyand
setthevalueto65536(decimal).
6.AddanewDWORDvaluetotheTcpWindowSizekeyandsetthe
valueto65536(decimal).
7.AddanewDWORDvaluetotheTcp1323Optskeyandsetthevalue
to3.
8.RestarttheWindowsserver.

Table14:NetWorkerServerConfigurationGuidelinesforNFS/CIFSandIPProtocol

22

NetworkFileSystemIntegration
NFSmountsrequireIPprotocolandadedicatedGigabitVLANordirectGigabit,connectionfor
NetWorkerserverintegration.
DataDomainrecommendsthefollowinggeneralNFSconfigurationsettingsformountingaDataDomain
systemtoaNetWorkerserver.
NetWorkerServer
Platform

NFSConfiguration

AIX

mountvnfso
proto=tcp,vers=3,intr,hard,combehind,rsize=32768,wsize=32768,llockn
dd200/backup/mountpoint

Solaris

mountFnfsohard,intr,vers=3,proto=tcp,rsize=32768,wsize=32768restorer
name:/backup/mountpoint

HPUX

mountFnfsorsize=32768,wsize=32768,hardrestorername:/backup/mount
point

LINUX

mounttnfsointr,hard,rsize=32768,wsize=32768,proto=tcp,vers=3
ddr:/backup/dd/<mountpoint>

Windows

CIFSaccesstoDataDomainsystemisrecommendedforWindows
NetWorkerServers
Table6:NFSMountConfigurationGuidelines

Note:DataDomainonlysupportsTCPprotocolwithNFS,andrecommendshardmountstoensure
availabilityafterNetWorkerserveroutages.

AdvancedFileTypeDesignConsiderations
NetWorkerhassupportedbackupstodiskdevicefromasearlyasversionNetWorker5.xwiththefile
typedeviceoption.NetWorker7.0includedtheadvancedfiletypedevice(whichallowssimultaneous
reads/writestothedevice).Botharerelativelysimpletoconfigureandcanbeusedwithstagingand/or
cloning.
Theadvancedfiletypedeviceoperationsincludereclaimingdiskspacefromexpiredandaborted
savesetstooptimizetheuseofdiskstorage.Theadvancedfiletypedeviceisoptimizedforverylarge
diskdevicesandisnevermarkedfullaswouldbeencounteredintapeorfiletypedeviceswhenthe
volumereachesafullcapacity.Whentheadvancedfiletypedevicerunsoutofspace,thecurrentbackup
isputonholdandanotificationisgenerated(thiscanbeconfiguredviaemail,consoleorlog
notifications),indicatingthatitiswaitingformorespace.NetWorkerwillthenbegintodeleteexpired
savesetsandreclaimspaceuntilenoughdataisavailableforbackuptocontinue.Whenenoughspaceis
available,thebackupwillcontinue.

23

Note:FreeingupspacewithintheAdvancedFileTypedevicesdoesincreasethebackupwindow
thereforeNetWorkerAdministratorsshouldalwaysmonitorthesizeofthedeviceifitisclosetofulland
meetingthebackupwindowisalargeconcern.
WithinNetWorker,thecreationofanadvancedfiletypedeviceappearsastwoseperatedevices.When
theadvancedfiletypedeviceislabeled,theNetWorkerServersoftwarecreatesasecondarydevicewith
readonlyaccessibility.TheNetWorkersoftwarethencreatesandmountsavolumewitha.ROsuffixin
thisdevice.
Note:Thesecondarydeviceisreadonly.Thisenablesconcurrentoperations(reading)fromthe
secondarydevice.NetWorkerAdministratorsshouldnotmakeanychangestothe_AF_readonlydevice
resource.Changesmadetotheprimaryvolumeorsaveset,areautomaticallypropagatedtothe
secondaryvolumeorsaveset.
NDMPIntegration
NDMPprotocolisstandardwithNetworkAttachedStorage(NAS)devicesindustrywide.Following
standardintegrationtechniquesusedwithNetWorkerandphysicaltapelibraries,theDataDomain
systemprovidesthesamefunctionalityforNDMPintegration.

Figure11:NDMPIntegration1withNetWorkerandDataDomainSystem
24

Figure12:NDMPIntegration2withNetWorkerandDataDomainSystem

25

ArchivingIntegration
LongtermdataretentioninNetWorkerisaccomplishedeitherviatraditionalbackuporarchive
functionalitywithDiskExtenderorEmailExtender.AnExtenderbasedlongtermbackupsolution
automaticallymanagestheduplicationofdataacrosstiersofarchivestorage,eachwithasetretention.
TheDiskandEmailExtendersolutionsprovideforafullyfunctionalarchivingsolution.
ForNetWorkersitesdesiringlongtermretentionaDataDomainsystemcanbeleveragedasanuppertier
forarchivedata.ThefollowingdiagramillustratesNetWorkerdatamovementtoaDataDomainsystem
longtermtierviaDiskExtender,whichisconfiguredtoperiodicallymigratedatatoanarchivetapepool
forextendedretention.

Figure13:ArchiveIntegrationwithNetWorkerandDataDomainSystem

NetWorkerDatabase(Bootstrap)Backups
NetWorkerbootstrapbackupsarecriticaltotherecoveryofaNetWorkerserver.ANetWorkerbootstrap
backupcanbedoneafterregularbackupsbyplacingtheNetWorkerserverinaproductiongroup.The
bootstrapiscomprisedofthreecomponentsthatresideontheNetWorkerserver:themediadatabase,the
resourcedatabase,andtheNetWorkerserversclientfileindex.Bootstrapbackupsareusuallywrittento
alocaltapedeviceanditisagoodideatohavetheNetWorkerbootstrapbackupscompleteatleasttwice
aday.Onebackupshouldbesentoffsitefordisasterrecoverypurposesandtheothershouldbekept
onsiteforoperationalrecovery.Thesecondbootstraptapecanbecreatedusingcustomizedcloning
scriptsautomaticallyrundaily(usingasavegroupOonthegrouptheserverisassignedto).Inaddition,
abootstrapreportcanbecreatedwithvitalinformationtorebuildyourNetWorkerserver.
WhenaDataDomainsystemisaddedtotheNetWorkerdatazone,NetWorkerbootstrapbackupscanbe
writtendirectlytotheDataDomainsystem.WhenNetWorkerbootstrapbackupsaresenttotheData
Domainsystem,storagespaceisnotwasted.IfasecondDataDomainsystemisintroducedatan

26

alternatesite,theNetWorkerbootstrapcanbereplicatedusingDataDomainReplicatorSoftware.This
setupwilleliminatealloftheinefficientlyusedbootstrapbackuptapes.Itwillalsoreducetheamountof
NetWorkerservertimerequiredtowritetwobootstrapbackupsdaily,byleveragingDataDomain
replicationfortheoffsitedatabasebackupcopy.

Figure14:BootstrapExample

NetWorkerDisasterRecovery
SeveralarchitecturalscenariosemploytheuseofaDataDomainsystemasacentralmechanismfor
replicatingNetWorkerdatatoanalternatesiteforpurposesofdisasterrecovery.Aswithanyother
methodofperformingNetWorkerdisasterrecoveryoperations,theNetWorkerdatabase,and
configurationfilesmustbeavailableforrecoveryoperations.
TheMMRECOVprocedureisusedtorecoveraNetWorkerservertoitsoriginalstatepriortothedisaster.
ThegeneralsequenceofeventstorecoveraNetWorkerserverincludesthefollowingsteps.

Note:TostarttherecoverprocessyouwillneedanewserverrunningthesameOSandpatchlevelasthe
productionNetWorkerserveranditwillneedtobeconnectedtotheDataDomainsystemattheDRsite.
1.InstalltheNetWorkerserversoftware.

27

2.ConfiguretheNetWorkerservertoseetheDataDomainfilesystems,theymustbeconfigured
identicaltothewaytheywereinproduction.
3.Ensurethatthediskwhereyouarerestoringthebootstrapfilescontainsthedirectorywherethe
catalogpreviouslyresided.Recreateanysymboliclinkstocataloglocationsthatmayhaveexisted
4.UseMMRECOVtorestorethedatabase
5.RestarttheNetWorkerservicesonserver.
6.Inventoryrequiredmedia(onlyifphysicaltapeismanagedbyNetWorker)
7.RestoretheIndexesoftheclients(ifneeded)
8.Resumebackupandrestoreoperationsinalternatebackupenvironment
BeforerunninganydisasterrecoveryplanitisbesttorefertotherelevantNetWorkerdocumentationand
testbeforeimplementationinproductionenvironments.ADataDomainsystemeliminatestheneedtogo
totapeattheprimarysite,andaltogetherwhenusinganalternatesiteinstancewithreplication.For
largercataloginstancesthiswilldecreasetheRTOsignificantly.

28

ArchitecturalScenarios
LocalDataDomainPrimaryStoragewithManualOffsiteTape
Inthescenarioillustratedinthefollowingfigure,aDataDomainsysteminstanceisusedtoreplace
traditionalNetWorkerDiskBackupoption..

Figure15:LocalDataDomainSystemPrimaryBackupStoragewithManualTapeVaulting
Atapelibraryisusedtocreatephysicaltapesforbootstrapbackupsandthetapesaremanually
transportedoffsiteviacourier.DuetotheuseofaDataDomainsystemastheprimarystoragepoolfor
backups,themigrationstephasbeenremovedfromthedailybatchprocessingtasks,shorteningthedaily
operationalworkload.

29

LocalDataDomainPrimaryBackupStoragewithNetWorkercloning
Inthescenarioillustratedinthefollowingfigure,aDataDomainsysteminstanceisusedtoreplace
traditionalNetWorkerdiskandtapecloning.AtapelibraryisaccessibleviaextendedSANatthe
alternatesite,andisusedfordailytapecloning.

Figure16:LocalDataDomainSystemPrimaryBackupStoragewithNetWorkerCloning
Again,tapecloningandstaginghasbeenremovedfromthedailytasks,shorteningthedailyoperational
workload.Thisscenariobenefitssiteswithexistingtapecloningorstagingandoffsitetapeinfrastructure
inplace.

30

LocalDataDomainPrimaryBackupStoragewithDataDomainSystemReplicationtoAlternateSite
Inthescenarioillustratedinthefollowingfigure,physicaltapeiseliminatedbyusingaDataDomain
systemattheprimarydatacentertoholdtheprimarystoragepools.AnotherDataDomainsystemis
placedatthealternatedatacenteranddataisreplicatedbetweendatacentersusingDataDomain
ReplicatorSoftware.TheuseoftheDataDomainreplicationdramaticallyreducedthenetwork
bandwidthneededforelectronicvaultingofdata.

Figure17:LocalDataDomainSystemPrimaryBackupStoragewithReplicationtoOffsiteData
DomainSystem
ByutilizingtheDataDomainsystemsatboththeprimaryandalternatedatacenter,thedailyoperational
tasksforbackupstoragepoolsarereduced.

31

MixedProduction/DRwithDataDomainReplication
Inthescenarioillustratedinthefollowingfigure,multipleNetWorkerserversclonedatatotheData
Domainsystemsattheoppositesite.

Figure18:MixedNetWorkerProduction/DRwithDataDomainReplication
Eachdatacentersupportsamixtureofproductionanddisasterrecoverycapacity,andeachNetWorker
serverclonesdatatothealternatesiteusingDataDomainreplication.

32

CentralDisasterRecoveryDataCenterConfiguration
Inthescenarioillustratedinthefollowingfigure,acentralizedDRsitesupportsmultipleproductiondata
centers,byreplicatingNetWorkerbackupdatatotheDRsiteviaDataDomainreplication.

Figure19:CentralDisasterRecoveryDataCenterConfiguration

33

RemoteOfficeLocalandDisasterRecoveryConfiguration
Inthescenarioillustratedinthefollowingfigure,acentralizedDRsitesupportsmultipleremotesites,
whichreplicateNetWorkerbackupdatatoasingleDataDomainsysteminstancetheDRsiteusingData
Domainreplication.

Figure20:RemoteOfficeLocalandDisasterRecoveryConfiguration

Conclusion
DataDomaininlinededuplicationstorageoffersavarietyofnewstoragearchitecturestrategiesfor
NetWorkercustomers.TheNetWorkerapplicationfunctionsasiswiththeDataDomainsystem
interfacemethods,minusthecomplexitiesandmanagementoverheadassociatedwithphysicaltape
media.Fortheenterprise,disasterrecoveryreplicationofNetWorkerdatahastraditionallyrequired
significantarchitecturalmeasuresandmanagementoverhead.DataDomainreplicationprovidesaviable
offsitedisasterrecoveryandlongertermretentionalternative.

34

WeseeaparticularlycompellingcaseforDataDomaintorevolutionizethewaysmall,medium,large
andremoteNetWorkersitesmanagephysicalstorageresourcesandreplicationfordisasterrecovery.For
extremelylargeNetWorkerinstances,appropriateplanning,sizing,andintegrationstrategiesmakethe
DataDomainsystemaviablesolutionfortheenterprise.
AboutGlassHouseTechnologies,Inc.
GlassHouseTechnologiesistheindustrysleadingindependentconsultingfirmwithprovenexperience
transformingITinfrastructure.GlassHousesproprietarymethodologyalignsbusinessprocessesand
informationtechnologysystems,transformingourclientsexistinginfrastructureintoscalable,compliant,
costefficientandtightlyorganizedenvironments.GlassHouseconsultantsarchitect,implementand
operateITenvironmentstodrivehighperformanceandagility.GlassHouseclientsinclude,Allianz,
MorganStanley,AetnaandWellsFargo.

35