6/1/2016

CommandsGuide

Apache>Hadoop>Core>docs>stable1

Search
LastPublished:08/04/201309:13:21

Searchthesitewithgoogle

PDF

CommandsGuide
Overview
GenericOptions
UserCommands
archive
distcp
fs
fsck
fetchdt
jar
job
pipes
queue
version
CLASSNAME
classpath
AdministrationCommands
balancer
daemonlog
datanode
dfsadmin
mradmin
jobtracker
namenode
secondarynamenode
tasktracker

Overview
Allhadoopcommandsareinvokedbythebin/hadoopscript.Runningthehadoopscriptwithout
anyargumentsprintsthedescriptionforallcommands.

Usage:hadoop[configconfdir][COMMAND][GENERIC_OPTIONS]
[COMMAND_OPTIONS]
http://hadoop.apache.org/docs/stable1/commands_manual.html

1/8

6/1/2016

CommandsGuide

Hadoophasanoptionparsingframeworkthatemploysparsinggenericoptionsaswellas
runningclasses.
COMMAND_OPTION

Description
configconfdir OverwritesthedefaultConfigurationdirectory.Defaultis
${HADOOP_HOME}/conf.
GENERIC_OPTIONS Thecommonsetofoptionssupportedbymultiplecommands.
COMMAND
Variouscommandswiththeiroptionsaredescribedinthefollowing
COMMAND_OPTIONS sections.ThecommandshavebeengroupedintoUserCommandsand
AdministrationCommands.

GenericOptions
Thefollowingoptionsaresupportedbydfsadmin,fs,fsck,jobandfetchdt.Applicationsshould
implementTooltosupportGenericOptions.
GENERIC_OPTION

Description

conf<configurationfile> Specifyanapplicationconfigurationfile.
D<property=value>
Usevalueforgivenproperty.
fs<local|namenode:port> Specifyanamenode.
jt
Specifyajobtracker.Appliesonlytojob.
<local|jobtracker:port>
files<commaseparated
Specifycommaseparatedfilestobecopiedtothemap
reducecluster.Appliesonlytojob.
listoffiles>
libjars<commaseperated Specifycommaseparatedjarfilestoincludeinthe
classpath.Appliesonlytojob.
listofjars>
archives<commaseparated Specifycommaseparatedarchivestobeunarchivedon
thecomputemachines.Appliesonlytojob.
listofarchives>

UserCommands
Commandsusefulforusersofahadoopcluster.

archive
Createsahadooparchive.MoreinformationcanbefoundatHadoopArchives.

Usage:hadooparchivearchiveNameNAME<src>*<dest>
COMMAND_OPTION

Description
archiveNameNAME Nameofthearchivetobecreated.
src
Filesystempathnameswhichworkasusualwithregularexpressions.
dest
Destinationdirectorywhichwouldcontainthearchive.

distcp
Copyfileordirectoriesrecursively.MoreinformationcanbefoundatHadoopDistCpGuide.

Usage:hadoopdistcp<srcurl><desturl>
COMMAND_OPTION

Description
SourceUrl

http://hadoop.apache.org/docs/stable1/commands_manual.html

2/8

6/1/2016

CommandsGuide

srcurl
desturl

DestinationUrl

fs
Usage:hadoopfs[GENERIC_OPTIONS][COMMAND_OPTIONS]
Runsagenericfilesystemuserclient.
ThevariousCOMMAND_OPTIONScanbefoundatFileSystemShellGuide.

fsck
RunsaHDFSfilesystemcheckingutility.SeeFsckformoreinfo.

Usage:hadoopfsck[GENERIC_OPTIONS]<path>[move|delete|
openforwrite][files[blocks[locations|racks]]]
COMMAND_OPTION

<path>
move
delete
openforwrite
files
blocks
locations
racks

Description
Startcheckingfromthispath.
Movecorruptedfilesto/lost+found
Deletecorruptedfiles.
Printoutfilesopenedforwrite.
Printoutfilesbeingchecked.
Printoutblockreport.
Printoutlocationsforeveryblock.
Printoutnetworktopologyfordatanodelocations.

fetchdt
GetsDelegationTokenfromaNameNode.Seefetchdtformoreinfo.

Usage:hadoopfetchdt[GENERIC_OPTIONS][webservice
<namenode_http_addr>]<path>
COMMAND_OPTION

<fileName>
webservice<https_address>

Description
Filenametostorethetokeninto.
usehttpprotocolinsteadofRPC

jar
Runsajarfile.UserscanbundletheirMapReducecodeinajarfileandexecuteitusingthis
command.

Usage:hadoopjar<jar>[mainClass]args...
Thestreamingjobsarerunviathiscommand.ExamplescanbereferredfromStreaming
examples
Wordcountexampleisalsorunusingjarcommand.ItcanbereferredfromWordcountexample

job
http://hadoop.apache.org/docs/stable1/commands_manual.html

3/8

6/1/2016

CommandsGuide

CommandtointeractwithMapReduceJobs.

Usage:hadoopjob[GENERIC_OPTIONS][submit<jobfile>]|[
status<jobid>]|[counter<jobid><groupname><counter
name>]|[kill<jobid>]|[events<jobid><fromevent#><#
ofevents>]|[history[all]<jobOutputDir>]|[list[all]]|
[killtask<taskid>]|[failtask<taskid>]|[setpriority
<jobid><priority>]
COMMAND_OPTION

submit<job
file>
status<jobid>
counter<job
id><groupname>
<countername>
kill<jobid>
events<jobid>
<fromevent#>
<#ofevents>
history[all]
<jobOutputDir>
list[all]
killtask
<taskid>
failtask
<taskid>
setpriority
<jobid>
<priority>

Description
Submitsthejob.
Printsthemapandreducecompletionpercentageandalljobcounters.
Printsthecountervalue.

Killsthejob.
Printstheevents'detailsreceivedbyjobtrackerforthegivenrange.

history<jobOutputDir>printsjobdetails,failedandkilledtipdetails.
Moredetailsaboutthejobsuchassuccessfultasksandtaskattempts
madeforeachtaskcanbeviewedbyspecifyingthe[all]option.
listalldisplaysalljobs.listdisplaysonlyjobswhichareyetto
complete.
Killsthetask.KilledtasksareNOTcountedagainstfailedattempts.
Failsthetask.Failedtasksarecountedagainstfailedattempts.
Changesthepriorityofthejob.Allowedpriorityvaluesare
VERY_HIGH,HIGH,NORMAL,LOW,VERY_LOW

pipes
Runsapipesjob.

Usage:hadooppipes[conf<path>][jobconf<key=value>,
<key=value>,...][input<path>][output<path>][jar<jar
file>][inputformat<class>][map<class>][partitioner
<class>][reduce<class>][writer<class>][program
<executable>][reduces<num>]
COMMAND_OPTION

conf<path>
jobconf<key=value>,<key=value>,...
input<path>
output<path>
jar<jarfile>
inputformat<class>
map<class>
partitioner<class>
http://hadoop.apache.org/docs/stable1/commands_manual.html

Description
Configurationforjob
Add/overrideconfigurationforjob
Inputdirectory
Outputdirectory
Jarfilename
InputFormatclass
JavaMapclass
JavaPartitioner
4/8

6/1/2016

CommandsGuide

reduce<class>
writer<class>
program<executable>
reduces<num>

JavaReduceclass
JavaRecordWriter
ExecutableURI
Numberofreduces

queue
commandtointeractandviewJobQueueinformation

Usage:hadoopqueue[list]|[info<jobqueuename>[
showJobs]]|[showacls]
COMMAND_OPTION

list
info<job
queuename>[
showJobs]
showacls

Description
GetslistofJobQueuesconfiguredinthesystem.Alongwithscheduling
informationassociatedwiththejobqueues.
Displaysthejobqueueinformationandassociatedscheduling
informationofparticularjobqueue.IfshowJobsoptionsispresenta
listofjobssubmittedtotheparticularjobqueueisdisplayed.
Displaysthequeuenameandassociatedqueueoperationsallowedfor
thecurrentuser.Thelistconsistsofonlythosequeuestowhichtheuser
hasaccess.

version
Printstheversion.

Usage:hadoopversion

CLASSNAME
hadoopscriptcanbeusedtoinvokeanyclass.

Usage:hadoopCLASSNAME
RunstheclassnamedCLASSNAME.

classpath
PrintstheclasspathneededtogettheHadoopjarandtherequiredlibraries.

Usage:hadoopclasspath

AdministrationCommands
Commandsusefulforadministratorsofahadoopcluster.

balancer
Runsaclusterbalancingutility.AnadministratorcansimplypressCtrlCtostoptherebalancing
process.SeeRebalancerformoredetails.

Usage:hadoopbalancer[threshold<threshold>]
http://hadoop.apache.org/docs/stable1/commands_manual.html

5/8

6/1/2016

CommandsGuide

COMMAND_OPTION

threshold
<threshold>

Description
Percentageofdiskcapacity.Thisoverwritesthedefault
threshold.

daemonlog
Get/Settheloglevelforeachdaemon.

Usage:hadoopdaemonloggetlevel<host:port><name>
Usage:hadoopdaemonlogsetlevel<host:port><name><level>
COMMAND_OPTION

Description

getlevel
Printstheloglevelofthedaemonrunningat<host:port>.This
<host:port><name> commandinternallyconnectstohttp://<host:port>/logLevel?log=
<name>
setlevel
Setstheloglevelofthedaemonrunningat<host:port>.This
<host:port><name> commandinternallyconnectstohttp://<host:port>/logLevel?log=
<name>
<level>

datanode
RunsaHDFSdatanode.

Usage:hadoopdatanode[rollback]
COMMAND_OPTION

rollback

Description
Rollsbackthedatanodetothepreviousversion.Thisshouldbeused
afterstoppingthedatanodeanddistributingtheoldhadoopversion.

dfsadmin
RunsaHDFSdfsadminclient.

Usage:hadoopdfsadmin[GENERIC_OPTIONS][report][safemode
enter|leave|get|wait][refreshNodes][finalizeUpgrade][
upgradeProgressstatus|details|force][metasavefilename][
setQuota<quota><dirname>...<dirname>][clrQuota<dirname>...
<dirname>][help[cmd]]
COMMAND_OPTION

Description
Reportsbasicfilesysteminformationandstatistics.
Safemodemaintenancecommand.SafemodeisaNamenodestatein
whichit
1.doesnotacceptchangestothenamespace(readonly)
2.doesnotreplicateordeleteblocks.
SafemodeisenteredautomaticallyatNamenodestartup,andleaves
safemodeautomaticallywhentheconfiguredminimumpercentageof
blockssatisfiestheminimumreplicationcondition.Safemodecanalso
beenteredmanually,butthenitcanonlybeturnedoffmanuallyaswell.
refreshNodes
RereadthehostsandexcludefilestoupdatethesetofDatanodesthat
areallowedtoconnecttotheNamenodeandthosethatshouldbe
decommissionedorrecommissioned.
finalizeUpgrade FinalizeupgradeofHDFS.Datanodesdeletetheirpreviousversion
workingdirectories,followedbyNamenodedoingthesame.This

report
safemodeenter
|leave|get|
wait

http://hadoop.apache.org/docs/stable1/commands_manual.html

6/8

6/1/2016

upgradeProgress
status|details
|force
metasave
filename

setQuota
<quota>
<dirname>...
<dirname>

clrQuota
<dirname>...
<dirname>
help[cmd]

CommandsGuide

completestheupgradeprocess.
Requestcurrentdistributedupgradestatus,adetailedstatusorforce
theupgradetoproceed.
SaveNamenode'sprimarydatastructuresto<filename>inthedirectory
specifiedbyhadoop.log.dirproperty.<filename>willcontainoneline
foreachofthefollowing
1.DatanodesheartbeatingwithNamenode
2.Blockswaitingtobereplicated
3.Blockscurrrentlybeingreplicated
4.Blockswaitingtobedeleted
Setthequota<quota>foreachdirectory<dirname>.Thedirectory
quotaisalongintegerthatputsahardlimitonthenumberofnamesin
thedirectorytree.
Besteffortforthedirectory,withfaultsreportedif
1.Nisnotapositiveinteger,or
2.userisnotanadministrator,or
3.thedirectorydoesnotexistorisafile,or
4.thedirectorywouldimmediatelyexceedthenewquota.
Clearthequotaforeachdirectory<dirname>.
Besteffortforthedirectory.withfaultreportedif
1.thedirectorydoesnotexistorisafile,or
2.userisnotanadministrator.
Itdoesnotfaultifthedirectoryhasnoquota.
Displayshelpforthegivencommandorallcommandsifnoneis
specified.

mradmin
RunsMRadminclient

Usage:hadoopmradmin[GENERIC_OPTIONS][refreshQueueAcls]
COMMAND_OPTION

Description

Refreshthequeueaclsusedbyhadoop,tocheckaccessduring
refreshQueueAcls submissionsandadministrationofthejobbytheuser.Theproperties
presentinmapredqueueacls.xmlisreloadedbythequeue
manager.

jobtracker
RunstheMapReducejobTrackernode.

Usage:hadoopjobtracker[dumpConfiguration]
COMMAND_OPTION

Description

DumpstheconfigurationusedbytheJobTrackeralongwithqueue
dumpConfiguration configurationinJSONformatintoStandardoutputusedbythe
jobtrackerandexits.

namenode
Runsthenamenode.Moreinfoabouttheupgrade,rollbackandfinalizeisatUpgradeRollback

Usage:hadoopnamenode[format[force][nonInteractive]]|[
upgrade]|[rollback]|[finalize]|[importCheckpoint]

http://hadoop.apache.org/docs/stable1/commands_manual.html

7/8

6/1/2016

CommandsGuide

upgrade]|[rollback]|[finalize]|[importCheckpoint]
COMMAND_OPTION

Description
format[force] Formatsthenamenode.Itstartsthenamenode,formatsitandthen
shutsitdown.Userwillbepromptedforinputifthenamedirectories
[
nonInteractive] existonthelocalfilesystem.
nonInteractive:Userwillnotbepromptedforinputifthename
directoriesexistinthelocalfilesystemandtheformatwillfail.
force:formatsthenamenodeandtheuserwillNOTbepromptedto
confirmformattingofnamedirectoriesinthelocalfilesystem.If
nonInteractiveoptionisspecifieditwillbeignored.
upgrade
Namenodeshouldbestartedwithupgradeoptionafterthedistribution
ofnewhadoopversion.
rollback
Rollsbackthenamenodetothepreviousversion.Thisshouldbeused
afterstoppingtheclusteranddistributingtheoldhadoopversion.
finalize
Finalizewillremovethepreviousstateofthefilessystem.Recent
upgradewillbecomepermanent.Rollbackoptionwillnotbeavailable
anymore.Afterfinalizationitshutsthenamenodedown.

Loadsimagefromacheckpointdirectoryandsaveitintothecurrent
importCheckpoint one.Checkpointdirisreadfrompropertyfs.checkpoint.dir

secondarynamenode
RunstheHDFSsecondarynamenode.SeeSecondaryNamenodeformoreinfo.

Usage:hadoopsecondarynamenode[checkpoint[force]]|[
geteditsize]
COMMAND_OPTION

checkpoint
[force]
geteditsize

Description
CheckpointstheSecondarynamenodeifEditLogsize>=
fs.checkpoint.size.Ifforceisused,checkpointirrespectiveofEditLog
size.
PrintstheEditLogsize.

tasktracker
RunsaMapReducetaskTrackernode.

Usage:hadooptasktracker

LastPublished:08/04/201309:13:21
Copyright2008TheApacheSoftwareFoundation.

http://hadoop.apache.org/docs/stable1/commands_manual.html

8/8