LinuxClusterManager

OperationsManual

Revision2.9011
February9,2008

Contents
Introduction.............................................................................................................................................................3
About.................................................................................................................................................................3
Features.............................................................................................................................................................3
SoftwarePackages............................................................................................................................................4
Installation...............................................................................................................................................................5
TestedPlatforms................................................................................................................................................5
Prerequisites......................................................................................................................................................5
CompiledLCMServer:.................................................................................................................................5
SourceLCMServer:.....................................................................................................................................5
CompiledLCMClient:...................................................................................................................................6
SourceLCMClient:.......................................................................................................................................6
PlatformNotes...................................................................................................................................................6
Debian..........................................................................................................................................................6
PackageInstallation...........................................................................................................................................6
LCMApplications....................................................................................................................................................7
LCM...................................................................................................................................................................7
LCMclient...........................................................................................................................................................7
LCMexec............................................................................................................................................................7
Examples......................................................................................................................................................8
Deploy................................................................................................................................................................8
Get_Host/WOL................................................................................................................................................8
ClusterManagement...............................................................................................................................................9
ManagingNodes................................................................................................................................................9
MainStatus......................................................................................................................................................10
Monitoring........................................................................................................................................................11
SearchingProcesses.......................................................................................................................................13
ExecuteCommand...........................................................................................................................................13
InventoryManagement....................................................................................................................................13
SystemImaging....................................................................................................................................................15
ImageServerSetup.........................................................................................................................................15
ImageCreation................................................................................................................................................16
BlockLevelImages.....................................................................................................................................16
FileLevelImages........................................................................................................................................17
ImageInformationandLocations.....................................................................................................................18
CustomizingImages........................................................................................................................................18
ImageDeployment...........................................................................................................................................19
DeploymentOperations..............................................................................................................................20
CreatingACustomBootKernel.......................................................................................................................20
LCMConfiguration................................................................................................................................................22
LCM.conf.........................................................................................................................................................22
LogInformation................................................................................................................................................23
Acknowledgments.................................................................................................................................................24
MichaelEngland..............................................................................................................................................24
AppendixA...........................................................................................................................................................25
LCMImageCapture.........................................................................................................................................25
LCMImageDeploy..........................................................................................................................................26

Introduction

About
LinuxClusterManager(LCM)isamanagementapplicationintendedforsetup,
administration,andmonitoringofLinuxbased,highperformancecomputeclusters.
Whiledevelopmenthasbeencenteredaroundclusters,LCM'sfunctionsareusefulfora
numberofdeploymentandmonitoringprojects.

Features

Monitoring

Realtimestatusandrecordingforallnodesinacluster

CPU,memory,network,andI/Operformance

Searchforrunningprocessesacrossthecluster
RemoteExecution

Allowsforanyscriptorcommandtoberunacrossthecluster

Requiresnoclientsoftwareutilizingexistingconnectionmethodssuchas
ssh,telnet,orrsh.
Imaging

RemoteWakeonLANandPrebootExecutionEnvironment(PXE)bootof
clientsforcollectionordeploymentofimages

Filebasedimagesallowchangestotargetattributessuchasfilesystem
sizes,destinationdevices,orfilesystemtypes

Blockbasedimagesallowfornonstandardconfigurationsoralternate
operatingsystemssuchasunsupportedLinuxdistributions,Solarisx86,or
Windows
HardwareInventoryReports

CPUspeed,type,andcorecount

L1andL2cachesize

Memorycapacity,speed,andchiplayout

Expansionslotlayoutandusage

Status/Monitor
refreshinterval
RemoteConnection

Function
Selector

ClusterInventory

NodeActivity

ConnectionStatus

Software Packages
Therearetwoprimaryformatsfordistribution,eitherasasourcepackageorcompiled
binary.Thecompiledbinariesareofteneasiertorunastheyreducedependency
requirements,however,sourcepackagesproducemoredebugginginformationand
providegreatercustomization.Bothareequalinfunctionality.
Withineachofthesegroupsthereiseitheraserverorclientpackage.Theserver
containsthegraphicalinterface,coderequiredforsoftwaredeployments,andaclient
agent.Itismeanttobeinstalledonamanagementsystemeitheraworkstationor
managementnodeinthecluster.Theclientsoftwareissimplyaclientagentmeantto
beinstalledonmanagednodes.
CompletepackagesarebuiltaseitheranRPM,Debianpackage,orSolarispackage
formatproducingsevenpossibleinstallations:

lcm<version><build>.i586.rpm
lcmsource<version><build>.i586.rpm
lcmsource_<version>_<build>.i386.deb
lcmclient<version><build>.i586.rpm
lcmclientsource<version><build>.i586.rpm
lcmclientsource_<version>_<build>.i386.deb
lcmclient<version>sol10x86.pkg

Debianpackagesareonlydistributedusingsourcefiles,ifbinariesaredesired,RPM
packagescanbeinstalledusingalien.Solarisisonlyavailableasaclientusingsource
codeonly.Itistestedonlyonx86basedsystemsbutshouldfunctiononSPARCaswell.

Installation

Tested Platforms
LCMGraphicalInterface(LCMServer)hasbeentestedwiththefollowingoperating
systemsandversions:

FedoraCore8

openSUSE10.3

Debian4.0r2etch
ItshouldalsoworkwellwithSUSELinuxEnterpriseServerandRedHatEnterprisebutis
notactivelytested.
LCMClienthasbeentestedonallserverplatformsplus:

Solaris10update3(x86/x64)

Solaris10update4(x86/x64)

Prerequisites
CompiledLCMServer:
BasicOperations:

Bashshell

Xserver

libstdc++libcwhichisgenerallypartofeitherthecompat2004orcompat2006
rpm
Note:IfyouarerunningRedHatorFedorayouwillhavetomanually
link/usr/lib/libstdc++3libc6.222.10.0to/usr/lib/libstdc++libc6.11.so.2
ln -s /usr/lib/libstdc++-3-libc6.2-2-2.10.0 /usr/lib/libstdc++-libc6.1-1.so.2

ImagingFunctions:

NFSserver
DHCPserver(version3orhigher)
TFTPserver

AdvancedInventoryFunctions:

/usr/sbin/hwinfo(fromhwinforpmpackage)
SourceLCMServer:
BasicOperations:

Bashshell

Xserver

TCL8.4

TK8.4

iwidgets4.0.1

iTCL3.3

expect5.43

tcllib1.953

ImagingFunctions:

Aspercompiledpackage
AdvancedInventoryFunctions:

Aspercompiledpackage
CompiledLCMClient:

libstdc++libcwhichisgenerallypartofeitherthecompat2004orcompat2006
rpm

Note:IfyouarerunningRedHatorFedorayouwillhavetomanually
link/usr/lib/libstdc++3libc6.222.10.0to/usr/lib/libstdc++libc6.11.so.2
ln -s /usr/lib/libstdc++-3-libc6.2-2-2.10.0 /usr/lib/libstdc++-libc6.1-1.so.2

SourceLCMClient:

TCL8.4

Platform Notes
Debian
Debian'spackagesareslightlydifferentthanthatofRedHatorSUSE,thefollowingisa
listofadditionalorspecificpackagestested:

itk3

nfskernelserver

dhcp3server

tftpdhpa

Package Installation
Thedecisiontoruneitherfromsourceoracompiledbinaryisamatterofconvenience.
Theyarebothfunctionallyequivalent,onesimplyhasfewerdependencies.
Toinstalleither,typerpm - -i <package name> or dpkg - -install <package name> as
root.IfyoudonotwishtouseallofthefeaturesinLCMyoucanforceadependency
overridewiththe- -nodepsoptionwhenusinganRPM.Regardlessofthetypeof
package(sourceorcompiledbinary),LCMisinstalledinfourdirectories:

/etcconfigurationandstartupscripts

/tftpbootsoftwarerequiredforPXEbootandimaging

/usr/local/lcmlcmprograms

/var/log/lcmLCMspecificlogs

LCM Applications

LCM
LCMisthemaingraphicalapplicationrunningfromamanagementnode.Itwillcall
otherapplicationsasneededandgenerateanyrequiredconfigurationsfilesatruntime.
SeveralTCPportsareopenedforfunctionsaslistedbelow:

60,003asacontrolportduringblockorfilelevelimagecollection

60,004asacontrolportduringblockorfilelevelimagedeployments

60,005usedtopushimagestoclientsduringthedeploymentprocess

60,006usedtoreceiveimagedatafromclientsduringeitherafileorblock
basedimagecollection
AmorecompleteflowchartofimagingfunctionscanbefoundinAppendixA.

LCMclient
TheclientportionofLCMthatrunsoneverymanagednodeinthecluster.LCMclient
allowsthemanagementstationtogatherinformationsuchasCPUornetworkstatistics,
searchrunningprocesses,hardwareinventory,andprovidesthemanstocreatefile
basedimages.ThreeTCPportsarerequiredforthesefunctions:

60,000formonitoring,inventory,andgeneralcontrolfunctions

60,002tosearchthelocalsystemforrunningprocesses

60,003tocreatefilebasedimages

LCMexec
LCMexecisusedtoruncommandsonremotenodesusingauserspecifiedshell.Itis
calledfromLCMbutcanalsoberunfromthecommandline.Asanexternalshellis
used,suchassshortelnet,noclientagentisneeded.Thisisparticularlyusefulto
installorconfigureapplicationstoarunningsystem.Forexampleinstallingthelcmclient
RPMtoanalreadyinstalledcluster.Inordertoaccessnodes,allclientsmustbe
populatedin/etc/cluster.confasexternalhostnameresolutionisnotused.
Tip:BesuretocommitchangestonewlymanagednodesintheManageNodespage
asLCMexecrequiresanupdated/etc/cluster.conffiletooperateproperly
Usage:lcmexec[nnode_list][ccommand][uuser][ppassword][llog_file][hhelp][
sshell]

nnode_list

Nodescanbelistedaseitherasinglenode,commaseparatedlist,range
using'',orthekeywordall.Forexample:

nnode1

nnode1,node3,node10

nnode1node10

nall

ccommand

Wherecommandisanyexecutableavailableonanodeeitherreferencedas
anabsolutepathorinthepathenvironmentvariableofthespecifieduser.
Anydatarequiredtocompletethecommandmustbeaccessiblebyeach

node.Enclosecommandswithspacesinsinglequotes,forexample:

c'rpmimyapp.rpm'

c'lsl/work|grepfile_group|wcl'

u/p

Anoptionalentrytospecifyausernameandcorrespondingpassword.If
yourremoteprotocoldoesn'trequireapassword(asispossiblewithssh)
youcanomitoneorbothoftheseoptions.

Thecommandusageaslistedabove

Anoptionalentrytorecordtheoutputfromeachnodeinalogfilestoredon
themachineLCMexecisbeingrunfrom.Ifthisoptionisomitted,theoutput
willbedisplayedtostandardout.

Anoptionalparametertospecifytheshelltobeusedintheconnection.This
valuecanbeanyremotecapableshellinstalledonyoursystemsuchas
telnet,ssh,orrsh.Ifthisoptionisomitted,thevaluespecified
in/etc/cluster.confwillbeusedallowingdifferentshellstobeusedby
differenthosts

Specifyatimeoutintervaltowaitforthecommandtocomplete

Examples
lcmexecncluster_node1cluster_node5c'rpmi/nfs_share/lcmclient2.904i586.rpm'
urootproot_passwordsssh
Thiswillinstalllcmclientonclusternodes1through5withrootcredentialsoverthessh
protocolanddisplaytheresultstothescreen.Notethepasswordfieldsarenot
obfuscatedatthecommandlinebutarenotretainedbeyondyournormalshellhistory.If
thisisexecutedthroughtheLCMGUI,passwordsareobfuscatedwithnohistory
maintained.
lcmexecncluster_node6c'echonew_cluster_node192.168.0.4>>/etc/hosts'uroot
proot_passwordssshl/root/edit_hosts.txt
Thiscommandwilladdanewentrytothe/etc/hostsfileandlogtheresultstoafile
callededit_hosts.txtinroot'shomedirectory.

Deploy
Usedonlyinimagemanagement,thisistheclientportionexecutedwhenanodeis
bootedforeitherblockimagecreationorfile/blockleveldeployment.Thisapplicationis
installedasasourcefiletobeinterpretedbytheincludedoperatingenvironmentlocated
in/tftpboot/bootimage

Get_Host / WOL
ThesearebothhelperapplicationstoLCMandarenotmeanttobeexecuteddirectly.
Get_Hostisusedfornameresolutionwhennodesareaddedtoacluster,WOLissues
wakeonLANpacketsforimagingnodes.

Cluster Management

Managing Nodes
Topopulateacluster,selectManageNodesfromthemainmenuandrightclickonthe
Hostslabelatthetopofthewindow.Fromthedropdownmenu,selectAddHost.A
NewHostlabelwillappear,highlightthislabelandtypeinanameforthefirstclient
system.Ifthehostnameenteredcanberesolved,additionalinformationwill
automaticallybepopulatedinthesubsequentcategories.Nameresolutionishandledby
asmallCprogramcalledget_hostbyrunninggethostbyname()whichinturnresolves
namesasdefinedthroughnsswitch.conf.Ifanodecannotberesolved,adefaultIPof
123.123.123.123isshown,highlightthisaddressandchangetotheactualordesired
value.
Ifthenewlyaddednodewillbeimagedbeforeitfullyjoinsthecluster(byrunning
lcmclient)youwillalsohavetoenteraMACaddressbyclickingrightoftheIPaddress.
RemembertoentertheaddressasXX:XX:XX:XX:XX:XX,e.g.00:11:11:1F:D3:58.If
lcmclientwillbeusedonanexistingnode,additionalinformationincludingtheMAC
addresswillbeautomaticallypopulated.

RightClicktoAddNodes
orCommitChanges
Highlighttochange
Rightclicktoremove
Autodetectedfrom
lcmclient

Draganddroptoadd
tostoragegroups

OthernodeoptionsincludeHostConnectionwhichdefaultstosshandStorageGroups.
AstoragegroupcanbecreatedbyrightclickingtheStorageGroupslabelandselecting
AddStorageGroup.ANewGroupisaddedwhich,likethehostname,canbe
highlightedandchangedtoamoredesirablelabel.Onceasystemconnectsthrough
lcmclient,individualDiskDriveswillappearwhichcanbedraggedintoastoragegroup.
Bothindividualdiskdrivesanduserdefinedgroupsaremadeavailableformonitoring.

Thisisparticularlyusefulwhendrivesarecombinedintoalogicalgroupandrequire
monitoringasasinglegroup.
Note:Detecteddevicesfromlcmclientsuchasacompletenetworkconfigurationor
availablediskdrivescannotbemanuallymodified.
Hosticonswillalsochangecolourdependingonthestatusofagivensystemasfollows:

Blueafullyconnectedsystemrunninglcmclient
Yellowasystemthatisrespondingtoapingbutnotrunninglcmclient
Grayasystemthatisunavailable

Hint:Nodescanbefullyopenedorclosedbyholdingthe'shift'keywhenopeningor
closingthehandlenexttothehostname.
Hint:Statusinformationisupdatedevery15seconds
Hint:Informationisupdated(suchasIPaddress)whenafieldisreleasedfromfocusso
clickonwhitespacewhenyouhavefinishedupdatinganentry
Whenchangesaremadetotheclusterincludinganyautomaticallyaddedinformation
fromlcmclientremembertosavechangesbyrightclickingtheHostslabelandselecting
CommitChanges.ThisallowsforLCMtoretaininformationifnodesaretakenoffline.
Allhostinformationisstoredin/etc/cluster.confasastandardtextfile.Whileisisnot
advisabletoalterbyhand,theformatisasfollows:
<hostname>{connect<method}{network{<eth#><ip><mac><primarydesignator>}}
{disk_group{<groupname><disk_1><disk_2><etc>}}{disk_device{<dev><capacity
inKB>}}
Example:
node1{connectssh}{network{eth0192.168.0.400:11:11:1F:D7:341}{eth1192.168.1.4
00:0C:29:02:F3:5F0}}{disk_group{MyStorageGroupsda}}{disk_device{sda
488386584}{sdb78150744}}
Tip:Toquicklybuildoraddmultiplenodessimplyenterthehostnameasasinglelinein
/etc/cluster.conf.WhenLCMstartsitwillresolveanIPaddressandifthenodeis
runningwillconnecttodetermineavailableinformation.Simplyselect'ManageNodes'
and'CommitChanges'.

Main Status
Thereareseveralcomponentstothispage(seescreenshotinChapter1Introduction):

AninventoryofclustermembersintheformofahostnameandIPaddress

CPUstatusinformation

Connectionstatus

Remoteshellconnectivity

RefreshintervalforCPUstatusupdates
HoveringoverahostnameorIPwillshowavailablehardwareinformationincludingboth
logicalandphysicalCPUcounts,speedandtypeofCPU,andtotalmemoryinstalled.
Clickingonanodenamewillbringupdetailedadetailedinventoryscreenforthatnode
alsoreachablethroughtheInventorymenuitem.

CPUstatusisupdatedinaccordancewiththerefreshintervalinthetoprightwitha
defaultof2seconds.ThegraphshowsthetotalperformanceofallCPUsonthisnode
asanaggregatedpercent.Forexample,anodewithfourCPUswithonebusywillshow
ausageof25%.Hoveringoverthegraphwillshowaroundednumericalpercentage.
Connectionstatusindicatorsarenextonthedisplayandcanhaveoneofthefollowing
states:
Greenindicatesthenodeisafullyfunctioningclustermemberwitha
connectiontoanLCMclientdaemon
YellowindicatesthenodesisrespondingtoapingrequestbutanLCMclient
daemonisnotrunning
Redindicatesthenodeisnotresponding
Aremoteshellconnectionbuttonisavailableforeachnodeprovidingonetouchaccess
toeachsystemusinganyprotocoldesired.Aprotocolcanbedefinedthroughhost
managementasdocumentedabove.

Monitoring
RealtimemonitoringofaclustercanbeperformedthroughtheMonitorbuttononthe
mainmenu.Alistofnodeswillbedisplayedwithcheckboxesforeachoftheavailable
monitortargets.Thislistwillincludeuserdefinedstoragegroupswhicharemonitoredas
asingleentity.Forexample,ifastoragegroupcontainsthreediskdevices,allI/O
characteristicsareaddedtogethertoproduceasingleresult.Valuesarenotbroken
down,however,individualdisksbelongingtoagroupcanalsobemonitoredatthesame.
Massselectioncheckboxesareavailableforconvenienceatthetopofeachmonitor
targettoselectallitemswithinthecluster.Forexample,allCPUsforallnodescanbe
selectedwithoneclick,individualnodescanthenbeuncheckedifdesired.
HoveringoveranEthernetdevicewillshowtheIPaddressandsubnetmaskforthat
device.Hoveringoveradiskgroupwilldisplayanymemberdiskdevices.
Whenselectingtargetstomonitoralogfilecanbespecifiedtorecordincomingdatafor
longtermretentionorfutureanalysis.Dataiskeptinacsvfilesuitableforimportinginto
anyspreadsheetapplication.

MassSelectbuttons

StorageGroups

DiskDevices

Selecttoenablelogging

Whenyouhavecompletedyourselections,clickSubmittobringuptheappropriate
graphs.AnyCPUinformationisdisplayedrelativeto100%regardlessofthequantityof
CPUspresentinthesystem.Forexample,ifoneCPUisbusyinatwoCPUsystem,the
graphwillshow50%.Thegraphiscolourcodedtorepresentusageaseitheruser,
system,orniceprocessesasindicatedinthelegend.
Networktrafficisshownaseitheraredbarforincoming(RX)trafficorabluebarfor
outgoing(TX)traffic.ThescaleisdynamicandwillswitchbetweeneitherKiloBytes/sec
orMegaBytes/secasshowninthelowerlefthandcornerofeachgraph.When
recordedtoafile,networktrafficisalwaysrecordedinKiloBytes/sec.Asnetwork
adapterscanhavedifferentconnectionrates,thespeedisshownaftertheEthernet
adapternameinthegraphtitle.Connectionspeedsarecapturedfromdmesgoutputon
LCMclientstartupwithvalidrangesat10Mbit,100Mbit,1000Mbit,or10000Mbit/sec.For
interfacesthatdonotreporttheirconnectionrate,adefaultcanbeassignedasspecified
in/etc/lcm.conf.HoveringoveranEthernettitlewillshowtheassignedIPaddressand
subnetmask.
Memoryinformationisshownasamulticolouredbarwithconsumeddiskcache,disk
buffers,application,andswapmemory.Hoveringoverthememorytitlewilldisplaythe
totalamountofmemoryinstalledinanygivennode.
I/Obasedtargetsshowatwocolouredbargraphrepresentingdiskreadsandwrites.As
withEthernettraffic,thescaleisdynamicswitchingbetweenKiloBytes/secandMega
Bytes/secasrequired.InformationloggedtoafileisalwaysrecordedinKiloBytes/sec.
Hoveringoveranybarwillshowthebreakdownofresourcesused.CPUwillshow
percentuser,system,andnice.Ethernetwillshowtransmittedandreceivedtrafficas
KiloBytespersecond.Memorywilldisplayuseddiskcache,diskbuffers,application,
andswapmemory.AnyI/OtargetwillshowdiskreadsandwritesinKiloBytesper
second.

Arefreshpulldownisalsoavailabletodynamicallychangethepollinginterval.Thisis
thesamepulldownaspresentedonthemainstatuspagewiththevaluesselected
linkedtogether(i.e.changinginanylocationwillimpacttheother).
Ifrecordingisselected,informationwillbeloggedtoafileinthebackgroundforaslong
asthegraphsarebeingdisplayed.Thatis,ifyounavigateawayfromthemonitor
display,loggingstopsandthefileisclosedoff.

Searching Processes
Tosearchrunningprocessesinthecluster,selecttheProcessSearchbuttonfromthe
mainmenu.Thiscommandeffectivelyrunsa'pself'acrosstheclustersoyoucan
searchforanythingthisoutputwouldincludesuchas:

Username

Applicationname

Terminalnumber

StateCode(D=uninterruptiblesleep,R=runnable,T=tracesorstopped,Z=a
defunctprocess)

Execute Command
TheExecuteCmdbuttonfromthemainmenuallowsforagraphicalinterfacetothe
LCMexecapplication.Anyrequireddatatocompletethecommand(scriptorbinary)
mustbeaccessibletoallnodesthroughexternalmeanssuchasacommonNFSmount
orbelocallyavailablebinaries.Ifyourconnectionprotocoldoesnotrequireausername
orpassword(suchascertificatemanagedssh)thesetwofieldscanbeignored.For
moreinformationseetheLCMexecsectionabove.

Inventory Management
Therearetwomethodsofenteringtheinventoryinformationscreen,eitherbyclicking

theInventorybuttononthemainmenuorclickingthenameofthenodeyouare
interestedinonthemainscreen.Selectingthenodefromthemainscreenbringsup
thatspecificnode.Theinventorybuttondisplaysablankpageawaitingyournode
selectionfromthedropdownboxontherighthandside.
InventoryinformationiscollectedbyLCMclientfrommultiplesourcesoneachnode.The
firstmethodisdonefromtherunningkernelthroughthe/procfilesystem.Thisgathers
basicinformationsuchasCPUandOSversion.Theother,moredetailed,methodis
donethroughthehwinfobinarywhichisincludedwithSUSEdistributionsandavailable
forothers.ThismethodgathersfurtherdetailsaboutCPUcache,memorylayout,and
PCIconfigurations.Otherdistributionsrunninghwinfoispossiblebuthavenotbeen
testedatthistime.
MultiplenodescanbeexaminedbyexportingtheinventorylisttoaCSVfileandviewing
inanexternalapplicationsuchasaspreadsheet.Todoso,simplyfillintheexportfile
locationandselecteitherAllNodesorThisNodetoexportthedata.

System Imaging

Image Server Setup

AllimagingfunctionsarefoundbyselectingtheImageNodesbuttonfromthemain
menu.Anewpageisdisplayedwithfourtabsacrossthetop.ImageServerisalways
thedefaultpagedisplayedandpresentstheoptionstosetupservicesrequiredfor
imaginganodeoverthenetwork.Theseservicesareusedforallimagedeployments
andcollectionofblockbaseimagesdescribedlaterinthischapter.
Requiredservices:

NFSserverusedtomounttherootpartitionofclientnodes./tftpboot/boot
imageisthedefaultlocationandmustbeexportedviaNFS.LCMwillmake
requiredchangesto/etc/exportsforyou.

DHCPservergiveseachnodeaspecificIPaddresskeyedtotheMACaddress
eitherdetectedfromapreviouslyrunningimageorenteredaspartofthecluster
setup.ThisallowsLCMtoappropriatelyaltertheclientsnetworkinginformation
givingeachauniquepersonalityonthenetwork.

TFTPserverdeliverstheinitialPXEimagetostartthenetworkbootprocess.

Tabselection

Ethernetdrivermenu

Servicestatus

Start/Stopservices

NFSserviceswillbemodifiedforLCMuseandischeckedwitheveryvisittothispage.
Theentrymadeisasfollows:

/tftpboot/bootimage*(rw,no_root_squash,insecure)
Afterverifyingthisentry,simplyensureNFSisrunningbyclickingthestarticon.
ToverifyTFTPservicescheckyourconfigurationfileandensuretheserverarguments
aresetto/tftpboot(LCMdoesnotverifythis).Formostdistributionsthisisthedefaultas
storedin/etc/xinetd.d/tftpdbutforDebianchangesmayhavetobemade.Thefile

locationfortftpdhpaunderDebianis/etc/default/tftphpa.Again,afterverifyingthe
entry,simplyensureTFTPisrunningbyclickingonthestarticon.
TheDHCPDservicehasafewoptionsandrequirementstofunctionproperly.Thereare
twoconfigurationoptionsthatmustbesetmanually.Oneisthelocationandfilename
LCMwillusetostoreinformationasdefinedin/etc/lcm.conf.Thedefaultlocation
is/etc/dhcpd.confforRPMdistributionsand/etc/dhcp3/dhcpd.conffordebpackages.
AlisteninginterfacemustalsobedefinedaspartofaDHCPDserverconfiguration.For
RedHat/SUSEthisfileislocatedin/etc/sysconfig/dhcpd.Thefieldrequiredis
DHCPD_INTERFACE.ForDebianthisfileis/etc/default/dhcp3serverandthefieldis
INTERFACES.Foralldistributionsthedefaultisablankvalueandmustbechangedto
theinterface(s)facingyourclusternodes,e.g.eth0.
Oncetheseitemshavebeenconfiguredaclientinterfacecanbeselectedfromthedrop
downmenu.ThisisthespecificdriverrequiredforyourEthernetcard.Ifthereisa
runningnodeyoucandeterminethepropermodulewiththeoutputoflsmod.Youcan
thenselecttheappropriatedriverfromthemenuandclicktheRegenerateDHCPbutton.
EveryclickoftheRegeneratebuttonoverwritestheexistingdhcpd.conffilesoensure
youhavenecessarybackupsavailableifanylegacyinformationisrequired.
DHCPDcheckverifiesthatallsystemsconfiguredwithaMACaddresshavebeen
properlyenteredinthedhcpd.conffile.ItshouldchangefromINVALIDtoVALIDandis
simplymeantasareminderthattheclusterconfigurationhaschangedsincethelast
dhcpd.conffilewasgenerated.ItdoesnotverifythatyouhaveassignedthehostanIP
addresscompatiblewiththeimagingserverorselectedtheproperEthernetdriver.

Image Creation
Therearetwotypesofimagesthatcanbecreated,filebasedorblockbased.Ingeneral
afilebasedimageismoregranularallowingforchangestotheredeployedsystemsuch
asfilesystemsizesortypes(ext3orreiserforexample).Itisalsoquickeroccupying
lessspaceontheserverwithlessdatapushedbacktotheclientsandcanbecreated
fromarunningsystem.ConverselyablockbasedimagemustbecreatedfromaPXE
bootedsystemandmustbedeployedtoasystemwithequalorgreaterdiskcapacitybut
cansupportanyoperatingsystemwithanybootconfiguration(includingSolarisand
Windows).
TostarttheimagingprocessselectCreateImagefromthetabsundertheImageNodes
menuitem.Selectthesystemtocreateanimagefrominthedropdownbox,namethe
image,andcreateanyoptionalcommentsdesired.Detailedinformationforeachimage
typearelistedbelow.Inaddition,aflowchartdiagramisincludedintheappendix
outliningLCMoperationswhencreatinganimage.

Block Level Images

WhenselectingablocklevelimageyoumustalsotellLCMwhichblockdevicesto
gather.ThisisdoneintheBlockDevicefield.Multipledevicescanbespecifiedby
usingacommaseparatedlistwithoutanywhitespace.e.g./dev/sda,/dev/sdb,/dev/sdc.
LCMrequiresanodetobePXEbootedinordertobeforeablocklevelimagecollection.
ThefirstthingitwilldoissendawakeonLANpackettopoweronthenodeandhaveit
bootareducedLinuxOSoverthenetwork.Ifyoursystemdoesnotsupportwakeon
LANordoesnotissueaPXEbootuponreceivingaWOLpacketyouwillhaveto
manuallyboottheserverandinstructtheBIOStobootoverEthernet.Thiscanoftenbe

doneaspartofaninitialbootselectionasaonetimebootratherthanhavingtochange
BIOSentriesmultipletimes.
Imagecollectionwillstartautomaticallyoncethenodecomesonline.Theimageitselfis
acompressedTARimageoftheentiredrive,blockforblock.Thismeansalloftheboot
sectorinformation,partitioninformation,data,andemptyspaceiscopied.Asageneral
ruleofthumb,theimagesizewillbeaboutofthetotalharddrivespace.Forexample,
thetheoriginaldriveis40GB,theimagewouldbeapproximately10Gbinsize.Where
thisisn'tthecaseisgenerallyifthedriveislargelyfilledorifdatahadatonetimeresided
onthedriveandhassincebeendeleted.Ifyouwishtoattempttocorrectthisyoucan
zerofiletheremainingdiskspacetocreatehighlycompressibleblankdata.For
example,withtheclientsystemrunningnormallytypeddif=/dev/zeroof=blank_file
bs=1024kcount=<x>wherexisthefreespaceinMegaBytes.Oncethecommandhas
completedthefilecanbedeleted(rmblank_file)freeingthespacebacktotheoperating
systembutcreatingalargeareaofhighlycompressiblezeros.
Ablocklevelimageissuitableforanyoperatingsystemwithanyfilesystem,however,
whenredeployinganoperatingsystemotherthanLinux,LCMwillproduceerrorswhen
tryingtochangethehostnameandIPaddress.Theseerrorsareexpectedandcanbe
ignored.

File Level Images

Tocreateafilelevelimage,arunningLCMclientdaemonneedstobeestablishedonthe
clientsystemwiththeOSrunningnormally.Ideallyanyadditionalapplicationswouldbe
shutdowntopreventmodificationoffilesduringtheimagingprocessbutthismaynotbe
requireddependingonthecircumstances.LCMclientwillgatheralllocallymountedfile
systemsandpartitionscapturingpartitionnames,sizes,filesystemtypes,andall
residingdata.Itdoesnotcapturebootsectorinformation,thisisreappliedaspartofthe
deploymentprocessfromscratch.Therequirementsforasuccessfulfilelevelimage
are:

RedHat,Fedora,SUSE,orDebian(allrecentversionsshouldwork)
CurrentlyusingtheGRUBbootloader
NomorethanfourprimarypartitionsonanIDEdrive,eightonaSCSIdisk(no
extendedpartitions)
Nologicalvolumemanagementused

Oncetheimageisstartedthetableatthebottomofthescreenwillbepopulatedwith
informationcollectedfromtheclient.Thestoragecapacitiesforafilebasedimageare
generallylessthanthatrequiredforablockbasedimage.Theruleofthumbisagain
butonlyoftheuseddata,notthedrive.Forexample,ifyouhavethesame40GBdrive
asabovebutwithonly2GBofdata,theimagewouldbeapproximately500MBinsize.
Obviouslyquickertocreateanddeploybutwillnotworkforallenvironments.

Image Information and Locations

Allimagesarestoredin/tftpboot/imageswiththeimagenameprovidedpluseitherthe
physicaldevicenameforablockimageorthemountpointforafileimage.Image
informationsuchasname,datecreate,comments,andimageparticularsarestorage
in/etc/image.confasaflattextfile.Exampleimageentriesarelistedbelow:
ImageFile
baseimage
SunJul1804:38:25PMPDT2004
node3192.168.0.3
1
//dev/hdc17271461439ext3yes\
{/tftpboot/images/baseimageslash.tar.gz}
1
swap/dev/hdc2893021728swapno
Abaseimageincludingapplications...
//

#Designatingafilebasedimage
#Imagename
#Dateimagewastaken
#SourcenodeandIP
#Numberofdatapartitions
#Filerecord(s)
#Numberofswappartitions
#Swaprecord(s)
#Comment
#Endofrecord

Ablocklevelimageforthesamesystemwouldappearasfollows:
ImageBlock
baseimage
SunJun2006:25:47PMPDT2004
node3192.168.0.3
1
/dev/hdc{/tftpboot/images/basefilehdc.tar.gz}
Abaseimageincludingapplication...
//

#Designatingablockbasedimage
#Imagename
#Dateimagewastaken
#SourcenodeandIP
#Numberofblockdevices
#Blockdeviceandimagefilename
#Comment
#Endofrecord

Customizing Images
ByselectingtheImageOptionstabyoucanalteryourcollectedimages.Beforeany
changesaremadeitisrecommendedthatyoubackup/etc/image.confasanyincorrect
optionscanrenderimagesunavailableforredeployment.Filelevelimagesprovidethe
mostflexibilityallowingyoutochangetheimagename,partitionsize,andfilesystem
type.Blocklevelimagesallowonlythetargetdevicetobealtered.
Tip:Whenapartitionsizeorblockdeviceischanged,besurethenewtargetisequalto
orlargerthantheoriginal.Anyadditionalcapacitywillbeleftasunpartitionedspacefor
futureuse

Tip:Changingabootdeviceorpartitionmaypreventaclientnodefrombootingproperly
asentriesinitslocalgrub.confaregenerallykeyedtoaspecificdiskpartition
Tip:Deletinganimagewillremovetheappropriateentryin/etc/image.confaswellas
thefile(s)specifiedundertheimagenamecolumnfrom/tftpboot/images

Image Deployment
TodeployanimageselecttheDeployImagestabfromthebootnodeswindow.Select
theimagedesiredandnodestobeimagedbyeitherclickingtheSelectAllcheckboxor
eachsystemindividually.Systemstobeimagedmustbenetworkedbootwhichcan
eitherbedonemanuallyorifsupported,byselectingtheIssueWOL(WakeOnLAN)
checkbox.OncetheExecutebuttonhasbeenpressed,allnodesselectedwilldisplay
'waitingforconnection...'.Toabortanimagebeforetheclienthasstartedremovethe
checkboxandpresstheExecutebuttonagain.
Anynodethatconnectswithoutbeingselectedforimagingwilldropoutofthe
deploymentprocessandpowerdownwithoutalteringanylocaldata.Thisistoprevent
anyaccidentalreimagingofnodes.
Thecurrentstatusforeachnodeinthedeploymentprocessisshowninthestatus
columnwithahistoricallogavailableforeachnodeavailableforviewingwiththeview
logbutton.Logsforeachnodearekeptin/var/log/lcmonthemasterLCMsystem.
Additionallocallogsaremadebyeachnodekeptunder/tftpboot/bootimageonthe
masternode.Thesearemeantfortroubleshootingpurposesonlyandcannormallybe
ignored.

Therearemanyaspectsofimagingconfigurablethrough/etc/lcm.conf.Ifthisfileis
altered,LCMmustberestartedforthechangestotakeeffect,seetheLCM
Configurationsectionformoreinformation.

Deployment Operations
Adetailedflowdiagramisincludedforreferenceintheappendixbutasabasicoverview
thefollowingstepsoccur.
BlockImages

Theimageiscopiedtothedestinationdeviceasspecified

Eachnodeisrebootedinorderfortherunningkerneltoreadthepartition
information

AnattemptismadetomounteachpartitiontochangethehostnameandIP
address

Thenodeispowereddown
FileImages

Thedestinationdevicesarepartitionedasspecified

Afilesystemiscreatedforeachpartition

Swappartitionsarecreated

Dataiscopied

Theentirefilesystemisremountedasitwouldlookinarunningsystem
under/mnt/dest

ThehostnameandIPfilesarealteredtogiveauniquenetworkidentity

GRUBisinstalled

anewinitialramdiskiscreated

Creating A Custom Boot Kernel

Abootkernelisusedduringtheimagingprocesswhenanodeisnetworkbooted.This
isthekernelthatwillgovernthecapabilitiesoftheminiLinuxOSbootedinitsabilitiesto
accessdevicesandproducearunningclient.Theincludedkernelsshouldbeadequate

formostdeployments,however,theremaybecircumstanceswhereyouwouldwantto
createyourownaseithersomethingdoesn'tworkproperlyonyoursystemsoryouwant
tofurthercustomizesomeaspectofthedeployment.Youwillneedseveralcomponents:

Kernelsourcetree

mknbirpmpackage(includedinsomedistributions)

Etherbootpackage(optionalforadditionalorupdatedPXEnetworkdrivers)
Tostarttheprocess,configurealloptionsyouneedintothenewkernel.Donotuse
modulesunlessyoureallyknowwhatyouaredoingasthereareseveraladditionalsteps
notdocumentedhere.Besuretoincludeallhardwarerequiredsuchasavideodevice,
keyboard,mouse,SCSIorIDEcontroller,networkdevices,andfilesystems.Two
optionsthatarealsorequiredare:

RootfilesystemonNFS

IP:kernellevelautoconfiguration,IP:DHCPsupport
ThesewillpermitthenewkerneltobootfromthenetworkandobtainitsIPaddressfrom
theDHCPserver.
Oncethekerneliscompiledyoumustrunmknbiagainstittomakethekernelnetwork
bootableusingPXE.Thecommandusedintheincludedkernelisasfollows:
mkelfformat=linuxip=dhcpappend=noexec=offbzImage2.6.17x64>
vmlinuz2.6.17x64nbi
or
mkelflinuxip=dhcpappend=noexec=offbzImage2.6.17x64>vmlinuz2.6.17x64nbi
TheappendoptionisusedfrompastexperiencewhereincompatibilitieswithsomeAMD
systemshavecausedthekerneltohangonboot.Youcanthenplacethiskernel
in/tftpboot/eboot/,editlcm.conf,restartLCM,andbesuretoregenerateyourDHCPfile
regardlessofwhetherLCMindicatesitisvalidornot.
IfyoualsowishtocompilenewPXEEthernetdrivers,downloadarecentversionof
EtherbootandcompilethePXEbootstraploaderformatROMimages.Todothis
changedirectoryto<etherboot_dir>/srcandtypemakeallzpxes.Thiswillplace
compiledbinariesforeachdriverin<etherboot_dir>/src/bin.Ifthereisaspecificdriver
youarelookingfor,youcanalsotypemakebin/<driver_name>.zpxe.Forexample,
makebin/pcnet32.zpxe.
ThePXEdriversincludedwithLCMalsohaveaDCONFIG_PCI_DIRECTcompiledinto
themtoovercomeaBIOS32errorforsomeAMDbasedsystems.Todothis,editand
addtheoptiontotheConfigfileunder<etherboot_dir>/src,thenbuildtheimages.The
entryshouldlooksomethingsimilartothis:
CFLAGS+=

DCONFIG_PCIDCONFIG_ISADCONFIG_PCI_DIRECT

AnyPXEdriversbuiltshouldbeplacedin/tftpboot/eboot/pxe/.Oncethisisdone,LCM
willdisplaythemasadropdownoptionundertheImageServertabasdescribedabove.

LCM Configuration

LCM.conf
Thelcm.conffileislocatedin/etcdirectoryandisusedtocontrolmanyaspectsofthe
imagingprocess,someontheserverside,someontheclient.Theavailableoptionsare
listedbelowbutthefileisalsocommentedforimmediatereference:

hostnamethisspecifiesthelocationofthefileLCMwillmodifywhenimaging
toensuretheclientnodehasauniquename

Thedefaultvalueisautowhichwillattempttodeterminethehosttype
beingimagedusingthefollowingmap

RedHat/Fedora/etc/sysconfig/network

Debian/etc/hostname

SUSE/etc/HOSTNAME
ipfilespecifiestheIPconfigurationfileontheoriginalclientbeingimaged
whichwillbealteredduringdeployment.

Thedefaultvalueisautowhichwillattempttodeterminetheappropriate
fileusingthefollowingmap

RedHat/Fedora/etc/sysconfig/networkscripts/ifcfg$adapter

Debian/etc/network/interfaces

SUSE/etc/sysconfig/network/ifcfg$adapter

ForSUSEsystemsthatuseifcfgethid<mac>asafilenamean*isused
asawildcardtomatchaspecificmanufacturer'scard.Forexample:

ifcfgeth*willmatchthefirstadapterLCMfinds

ifcfgethid00:11:11*willmatchaspecificvendorscard(eth1for
example)

rememberthesearethefilesontheoriginalsystem
adapterallowsyoutoselecttheadapternumberforthenodetheimageis
beingdeployedto.Typicallythisisthefistadapterfound(i.e.eth0)butyour
hardwaresetupmaydictatealteringthisvaluetoadifferentcard

Thedefaultvalueiseth0
client_subnetdefinesthesubnetvalueforthenewclientbeingdeployed

defaultvaluedefaultmeaningLCMwillfollowstandardclassfulInternet
Networking(e.g.255.0.0.0or255.255.0.0or255.255.255.0).Ifyouhave
adifferentsubnetforyourclusterenterithere.
kernelanentrytospecifythekernelinwhichnodeswillbootfromaspartof
thedeploymentorblockimagecollectionprocess.Thekernelmustexist
in/tftpboot/bootimage/ebootandmustbeanetworkbootableimagecreated
withmknbi(seeCreatingaCustomBootKernel).LCMincludestwokernelsa
32bitand64bitversion.
excludeallowsyoutospecifyanylocalmountpointsthatyouwishtoexclude
whencreatinganimage.Thisisusefulifyourhosthasextramountpointsthat
arenotrequiredornotavailableonothernodes.
completed_powerindicateswhetherLCMshouldpowerdownnodeswhen
imagingcompletesorleavethemon(usefulfordebuggingorother
administrationtasks)

defaultoff
pre_grub_cmd/post_grub_cmduserspecifiedcommandsthatcanprovide
anyadditionalconfigurationforacustomizedimagingprocess.GRUBis
installedattheveryendoftheimagingprocessbutmayrequiresometuning

beforeitisabletoinstallcorrectly,orperhapsthereissomecustomizationyou
wishtoincludebeforeoraftergrubhasfinished.
dhcpd_locationusedtospecifywhereyouwantLCMtocreateanydhcpd
serverinformation

default/etc/dhcpd.conf(RedHat/Fedora/SUSE)

default/etc/dhcp3/dhcpd.conf(Debian)
dhcpd_interfacethenameoftheinterfaceontheLCMserverusedtoservice
dhcprequests.Thisisusedtoaddrequiredentriestodhcpd.conf.

defaulteth0
default_rateTheonlynonimagingbasedoption,thisisusedtoprovidea
defaultEthernetconnectionratefornodesthatarenotabletodetectavalue

default1000
Optionalfilesystemdefinitions.BydefaultLCMknowshowtocreatethefile
systemlistedbelow,however,ifyouhaveanothertypeyouwishtouse,enterthe
commandtocreateit.Ifyourequireadditionaluserlevelexecutables,copy
themto/tftpboot/bootimage/sbinonthemasterLCMserver.Required
keywordsareLABELforapartitionlabelandPARTITIONforthepartition
number.Thevalueswillbefilledinautomaticallyduringthedeployment
process.

ext3=/sbin/mke2fsqjLLABELPARTITION

reiserfs=/sbin/mkreiserfsqlLABELPARTITION

xfs=/sbin/mkfs.xfsqLLABELPARTITION

jfs=/sbin/mkfs.jfsqLLABELPARTITION

Log Information
AlllogfilesforLCMarekeptin/var/log/lcm/forbothclientnodesandontheLCM
managementsystemasbothLCMclientandtheLCMGUIkeeplogsinthelcm.logfile.
Inaddition,deploymentlogsarekeptonthemanagementnodeforeachmachine
imagesas<nodename>.log.Thisfileliststhecommandsandresultsforalloperations
LCMperformedduringtheclientimagingprocess.Thisfilecanalsobeviewedby
clickingViewLogintheDeployImagewindow.

Acknowledgments
MichaelEngland
AllLCMrelatedprogramming,manuals,helpfiles,andwebsite
GregPederson
BannerGraphic
EtherbootProject
PXEbootprocess
Scyld
WakeOnLANapplication

Appendix A
LCM Image Capture

LCM Image Deploy