IntroductiontoUNIX CourseOutline

Theselecturenotesandexercisesaredesignedtosupportanintensiveintroductory courseonUNIXortoactasareferencetouserswhoarenewtoUNIX.Thiscoursewas designedparticularlyforusewiththeLinuxoperatingsystembutmuchofitappliesto otherUNIXsystemsaswell.Thesenotesarealsoavailableonlineat http://www.doc.ic.ac.uk/~wjk/UnixIntro.

LectureOne Whatisanoperatingsystem? AbriefhistoryofUNIX ArchitectureoftheLinuxoperatingsystem Logginginto(andoutof)UNIXsystems Changingyourpassword GeneralformatofUNIXcommands ExerciseSheetOne LectureTwo TheUNIXfilesystem TypicalUNIXdirectorystructure Directoryandfilehandlingcommands Makinghardandsoft(symbolic)links Specifyingmultiplefilenames

Quotes ExerciseSheetTwo

LectureThree Fileanddirectorypermissions Inspectingfilecontent Findingfiles Findingtextinfiles Sortingfiles Filecompressionandbackup Handlingremovablemedia ExerciseSheetThree LectureFour Processes Pipes Redirectinginputandoutput Controllingprocessesassociatedwiththecurrentshell Controllingotherprocesses ExerciseSheetFour LectureFive Connectingtoremotemachines Networkroutingutilities Remotefiletransfer OtherInternetrelatedutilities UserInformationandCommunication Printercontrol

Emailutilities Advancedtextfileprocessing Targetdirectedcompilation VersioncontrolwithCVS C/C++compilationutilities Manualpages ExerciseSheetFive

LectureSix Introductiontovi Basictextinputandnavigationinvi Movingandcopyingtextinvi Searchingforandreplacingtextinvi Otherusefulvicommands Quickreferenceforvi Introductiontoemacs Basictextinputandnavigationinemacs Movingandcopyingtextinemacs Searchingforandreplacingtextinemacs Otherusefulemacscommands OtherUNIXeditors ExerciseSheetSix LectureSeven Thesuperuserroot Shutdownandsystemstartup Addingusers

Controllingusergroups ReconfiguringandrecompilingtheLinuxkernel Cronjobs Keepingessentialsystemprocessesalive ExerciseSheetSeven

LectureEight Shellsandshellscripts Shellvariablesandtheenvironment Simpleshellscripting Moreadvancedshellscripting Startupshellscripts ExerciseSheetEight

1.1Objectives
Thislecturecovers:

Theconceptofanoperatingsystem. Theinternalarchitectureofanoperatingsystem. TheevolutionoftheUNIXoperatingsystemintotwobroadschools (BSDandSYSV)andthedevelopmentofLinux,apopularopensource operatingsystem. ThearchitectureoftheLinuxoperatingsysteminmoredetail. Howtologinto(andoutof)UNIXandchangeyourpassword. ThegeneralformatofUNIXcommands.

1.2WhatisanOperatingSystem?
Anoperatingsystem(OS)isaresourcemanager.Ittakestheformofasetof softwareroutinesthatallowusersandapplicationprogramstoaccesssystem resources(e.g.theCPU,memory,disks,modems,printersnetworkcardsetc.) inasafe,efficientandabstractway. Forexample,anOSensuressafeaccesstoaprinterbyallowingonlyone applicationprogramtosenddatadirectlytotheprinteratanyonetime.AnOS encouragesefficientuseoftheCPUbysuspendingprogramsthatarewaiting forI/Ooperationstocompletetomakewayforprogramsthatcanusethe CPUmoreproductively.AnOSalsoprovidesconvenientabstractions(such asfilesratherthandisklocations)whichisolateapplicationprogrammersand usersfromthedetailsoftheunderlyinghardware.

Fig.1.1:Generaloperatingsystemarchitecture

Fig.1.1presentsthearchitectureofatypicaloperatingsystemandshowshow anOSsucceedsinpresentingusersandapplicationprogramswithauniform interfacewithoutregardtothedetailsoftheunderlyinghardware.Wesee that:

Theoperatingsystemkernelisindirectcontroloftheunderlying hardware.Thekernelprovideslowleveldevice,memoryandprocessor managementfunctions(e.g.dealingwithinterruptsfromhardware devices,sharingtheprocessoramongmultipleprograms,allocating memoryforprogramsetc.) Basichardwareindependentkernelservicesareexposedtohigherlevel programsthroughalibraryofsystemcalls(e.g.servicestocreateafile, beginexecutionofaprogram,oropenalogicalnetworkconnectionto anothercomputer). Applicationprograms(e.g.wordprocessors,spreadsheets)andsystem

utilityprograms(simplebutusefulapplicationprogramsthatcome withtheoperatingsystem,e.g.programswhichfindtextinsideagroup offiles)makeuseofsystemcalls.Applicationsandsystemutilitiesare launchedusingashell(atextualcommandlineinterface)ora graphicaluserinterfacethatprovidesdirectuserinteraction. Operatingsystems(anddifferentflavoursofthesameoperatingsystem)can bedistinguishedfromoneanotherbythesystemcalls,systemutilitiesand userinterfacetheyprovide,aswellasbytheresourceschedulingpolicies implementedbythekernel.

1.3ABriefHistoryofUNIX
UNIXhasbeenapopularOSformorethantwodecadesbecauseofitsmulti user,multitaskingenvironment,stability,portabilityandpowerful networkingcapabilities.Whatfollowshereisasimplifiedhistoryofhow UNIXhasdeveloped(togetanideaforhowcomplicatedthingsreallyare,see thewebsitehttp://www.levenez.com/unix/).

Fig.1.2:SimplifiedUNIXFamilyTree

Inthelate1960s,researchersfromGeneralElectric,MITandBellLabs launchedajointprojecttodevelopanambitiousmultiuser,multitaskingOS formainframecomputersknownasMULTICS(MultiplexedInformationand ComputingSystem).MULTICSfailed(forsomeMULTICSenthusiasts "failed"isperhapstoostrongawordtousehere),butitdidinspireKen Thompson,whowasaresearcheratBellLabs,tohaveagoatwritinga simpleroperatingsystemhimself.HewroteasimplerversionofMULTICS onaPDP7inassemblerandcalledhisattemptUNICS(Uniplexed InformationandComputingSystem).BecausememoryandCPUpowerwere

atapremiuminthosedays,UNICS(eventuallyshortenedtoUNIX)used shortcommandstominimizethespaceneededtostorethemandthetime neededtodecodethemhencethetraditionofshortUNIXcommandsweuse today,e.g.ls, cp, rm, mvetc. KenThompsonthenteamedupwithDennisRitchie,theauthorofthefirstC compilerin1973.TheyrewrotetheUNIXkernelinCthiswasabigstep forwardsintermsofthesystem'sportabilityandreleasedtheFifthEditionof UNIXtouniversitiesin1974.TheSeventhEdition,releasedin1978,marked asplitinUNIXdevelopmentintotwomainbranches:SYSV(System5)and BSD(BerkeleySoftwareDistribution).BSDarosefromtheUniversityof CaliforniaatBerkeleywhereKenThompsonspentasabbaticalyear.Its developmentwascontinuedbystudentsatBerkeleyandotherresearch institutions.SYSVwasdevelopedbyAT&Tandothercommercial companies.UNIXflavoursbasedonSYSVhavetraditionallybeenmore conservative,butbettersupportedthanBSDbasedflavours. ThelatestincarnationsofSYSV(SVR4orSystem5Release4)andBSD Unixareactuallyverysimilar.Someminordifferencesaretobefoundinfile systemstructure,systemutilitynamesandoptionsandsystemcalllibrariesas showninFig1.3.

Feature kernel name boot init mounted FS default shell FS block size print subsystem echo command (no new line) ps command multiple wait syscalls memory access syscalls

Typical SYSV /unix /etc/rc.d directories /etc/mnttab sh, ksh 512 bytes->2K lp, lpstat, cancel echo "\c" ps -fae poll memset, memcpy

Typical BSD /vmunix /etc/rc.* files /etc/mtab csh, tcsh 4K->8K lpr, lpq, lprm echo -n ps -aux select bzero, bcopy

Fig.1.3:DifferencesbetweenSYSVandBSD LinuxisafreeopensourceUNIXOSforPCsthatwasoriginallydevelopedin 1991byLinusTorvalds,aFinnishundergraduatestudent.Linuxisneither pureSYSVorpureBSD.Instead,incorporatessomefeaturesfromeach(e.g. SYSVstylestartupfilesbutBSDstylefilesystemlayout)andaimsto conformwithasetofIEEEstandardscalledPOSIX(PortableOperating SystemInterface).Tomaximisecodeportability,ittypicallysupportsSYSV, BSDandPOSIXsystemcalls(e.g.poll, select, memset, memcpy, bzeroandbcopyareallsupported). TheopensourcenatureofLinuxmeansthatthesourcecodefortheLinux kernelisfreelyavailablesothatanyonecanaddfeaturesandcorrect deficiencies.Thisapproachhasbeenverysuccessfulandwhatstartedasone person'sprojecthasnowturnedintoacollaborationofhundredsofvolunteer developersfromaroundtheglobe.Theopensourceapproachhasnotjust successfullybeenappliedtokernelcode,butalsotoapplicationprogramsfor Linux(seee.g.http://www.freshmeat.net). AsLinuxhasbecomemorepopular,severaldifferentdevelopmentstreamsor distributionshaveemerged,e.g.Redhat,Slackware,Mandrake,Debian,and Caldera.Adistributioncomprisesaprepackagedkernel,systemutilities,GUI interfacesandapplicationprograms. Redhatisthemostpopulardistributionbecauseithasbeenportedtoalarge numberofhardwareplatforms(includingIntel,Alpha,andSPARC),itiseasy touseandinstallanditcomeswithacomprehensivesetofutilitiesand applicationsincludingtheXWindowsgraphicssystem,GNOMEandKDE GUIenvironments,andtheStarOfficesuite(anopensourceMSOfficeclone forLinux).

1.4ArchitectureoftheLinuxOperatingSystem
LinuxhasallofthecomponentsofatypicalOS(atthispointyoumightliketo referbacktoFig1.1):

Kernel TheLinuxkernelincludesdevicedriversupportforalargenumberof PChardwaredevices(graphicscards,networkcards,harddisksetc.), advancedprocessorandmemorymanagementfeatures,andsupportfor manydifferenttypesoffilesystems(includingDOSfloppiesandthe ISO9660standardforCDROMs).Intermsoftheservicesthatit providestoapplicationprogramsandsystemutilities,thekernel implementsmostBSDandSYSVsystemcalls,aswellasthesystem callsdescribedinthePOSIX.1specification. Thekernel(inrawbinaryformthatisloadeddirectlyintomemoryat systemstartuptime)istypicallyfoundinthefile/boot/vmlinuz,while thesourcefilescanusuallybefoundin/usr/src/linux.Thelatestversion oftheLinuxkernelsourcescanbedownloadedfrom http://www.kernel.org.

ShellsandGUIs Linuxsupportstwoformsofcommandinput:throughtextualcommand lineshellssimilartothosefoundonmostUNIXsystems(e.g.shthe Bourneshell,bashtheBourneagainshellandcshtheCshell)and throughgraphicalinterfaces(GUIs)suchastheKDEandGNOME windowmanagers.Ifyouareconnectingremotelytoaserveryour accesswilltypicallybethroughacommandlineshell.

SystemUtilities Virtuallyeverysystemutilitythatyouwouldexpecttofindonstandard implementationsofUNIX(includingeverysystemutilitydescribedin thePOSIX.2specification)hasbeenportedtoLinux.Thisincludes commandssuchasls, cp, grep, awk, sed, bc, wc, more,andsoon.Thesesystemutilitiesaredesignedtobepowerful toolsthatdoasingletaskextremelywell(e.g.grepfindstextinside fileswhilewccountsthenumberofwords,linesandbytesinsideafile). Userscanoftensolveproblemsbyinterconnectingthesetoolsinsteadof writingalargemonolithicapplicationprogram.

LikeotherUNIXflavours,Linux'ssystemutilitiesalsoincludeserver programscalleddaemonswhichprovideremotenetworkand administrationservices(e.g.telnetdandsshdprovideremotelogin facilities,lpdprovidesprintingservices,httpdserveswebpages, crondrunsregularsystemadministrationtasksautomatically).A daemon(probablyderivedfromtheLatinwordwhichreferstoa beneficientspiritwhowatchesoversomeone,orperhapsshortfor"Disk AndExecutionMONitor")isusuallyspawnedautomaticallyatsystem startupandspendsmostofitstimelyingdormant(lurking?)waitingfor someeventtooccur.

Applicationprograms Linuxdistributionstypicallycomewithseveralusefulapplication programsasstandard.Examplesincludetheemacseditor,xv(an imageviewer),gcc(aCcompiler),g++(aC++compiler),xfig(a drawingpackage),latex(apowerfultypesettinglanguage)and soffice(StarOffice,whichisanMSOfficestyleclonethatcanread andwriteWord,ExcelandPowerPointfiles). RedhatLinuxalsocomeswithrpm,theRedhatPackageManager whichmakesiteasytoinstallanduninstallapplicationprograms.

1.5Logginginto(andoutof)UNIXSystems
Textbased(TTY)terminals: WhenyouconnecttoaUNIXcomputerremotely(usingtelnet)orwhenyou loginlocallyusingatextonlyterminal,youwillseetheprompt: login: Atthisprompt,typeinyourusenameandpresstheenter/return/ key. RememberthatUNIXiscasesensitive(i.e.Will,WILLandwillareall differentlogins).Youshouldthenbepromptedforyourpassword: login: will

password: Typeyourpasswordinatthepromptandpresstheenter/return/ key.Note thatyourpasswordwillnotbedisplayedonthescreenasyoutypeitin. Ifyoumistypeyourusernameorpasswordyouwillgetanappropriate messagefromthecomputerandyouwillbepresentedwiththelogin: promptagain.Otherwiseyoushouldbepresentedwithashellpromptwhich lookssomethinglikethis: $ TologoutofatextbasedUNIXshell,type"exit"attheshellprompt(orif thatdoesn'tworktry"logout";ifthatdoesn'tworkpressctrld). Graphicalterminals: Ifyou'reloggingintoaUNIXcomputerlocally,orifyouareusingaremote loginfacilitythatsupportsgraphics,youmightinsteadbepresentedwitha graphicalpromptwithloginandpasswordfields.Enteryourusernameand passwordinthesamewayasabove(N.B.youmayneedtopresstheTABkey tomovebetweenfields). Onceyouareloggedin,youshouldbepresentedwithagraphicalwindow managerthatlookssimilartotheMicrosoftWindowsinterface.Tobringupa windowcontainingashellpromptlookformenusoriconswhichmentionthe words"shell","xterm","console"or"terminalemulator". Tologoutofagraphicalwindowmanager,lookformenuoptionssimilarto "Logout"or"Exit".

1.6Changingyourpassword
Oneofthethingsyoushoulddowhenyouloginforthefirsttimeistochange yourpassword. TheUNIXcommandtochangeyourpasswordispasswd:

$ passwd

Thesystemwillpromptyouforyouroldpassword,thenforyournew password.Toeliminateanypossibletypingerrorsyouhavemadeinyournew password,itwillaskyoutoreconfirmyournewpassword. Rememberthefollowingpointswhenchoosingyourpassword:

Avoidcharacterswhichmightnotappearonallkeyboards,e.g.''. Theweakestlinkinmostcomputersecurityisuserpasswordssokeepyour passwordasecret,don'twriteitdownanddon'ttellittoanyoneelse.Also avoiddictionarywordsorwordsrelatedtoyourpersonaldetails(e.g.your boyfriendorgirlfriend'snameoryourlogin). Makeitatleast7or8characterslongandtrytouseamixofletters, numbersandpunctuation.

1.7GeneralformatofUNIXcommands
AUNIXcommandlineconsistsofthenameofaUNIXcommand(actually the"command"isthenameofabuiltinshellcommand,asystemutilityoran applicationprogram)followedbyits"arguments"(optionsandthetarget filenamesand/orexpressions).ThegeneralsyntaxforaUNIXcommandis $ command -options targets

Herecommandcanbethoughofasaverb,options asanadverband targetsasthedirectobjectsoftheverb.Inthecasethattheuserwishesto specifyseveraloptions,theseneednotalwaysbelistedseparately(theoptions cansometimesbelistedaltogetherafterasingledash).

IntroductiontoUNIX: ExerciseSheetOne

1. LogonaLinuxmachineorconnecttoonefromaWindowsmachine

(e.g.clickontheExceediconandthenuseputtytoconnecttotheserver kiwi.Enteryourlogin(username)andpasswordatrelevantprompts. 2. EnterthesecommandsattheUNIXprompt,andtrytointerpretthe output.Askquestionsanddon'tbeafraidtoexperiment(asanormal useryoucannotdomuchharm): echo hello world passwd date hostname arch uname -a dmesg | more (youmayneedtopressqtoquit) uptime who am i who id last finger w top (youmayneedtopressqtoquit) echo $SHELL echo {con,pre}{sent,fer}{s,ed} man "automatic door" man ls (youmayneedtopressqtoquit) man who (youmayneedtopressqtoquit) who can tell me why i got divorced lost

clear cal 2000 cal 9 1752 (doyounoticeanythingunusual?) bc -l (typequit orpressCtrldtoquit) echo 5+4 | bc -l yes please (youmayneedtopressCtrlctoquit) time sleep 5 history

IntroductiontoUNIX: LectureTwo

2.1Objectives
Thislecturecovers:

TheUNIXfilesystemanddirectorystructure. Fileanddirectoryhandlingcommands. Howtomakesymbolicandhardlinks. Howwildcardfilenameexpansionworks. Whatargumentquotingisandwhenitshouldbeused.

2.2TheUNIXFilesystem
TheUNIXoperatingsystemisbuiltaroundtheconceptofafilesystemwhich isusedtostorealloftheinformationthatconstitutesthelongtermstateofthe system.Thisstateincludestheoperatingsystemkernelitself,theexecutable filesforthecommandssupportedbytheoperatingsystem,configuration information,temporaryworkfiles,userdata,andvariousspecialfilesthatare usedtogivecontrolledaccesstosystemhardwareandoperatingsystem functions. EveryitemstoredinaUNIXfilesystembelongstooneoffourtypes:
1. Ordinaryfiles

Ordinaryfilescancontaintext,data,orprograminformation.Files cannotcontainotherfilesordirectories.Unlikeotheroperatingsystems, UNIXfilenamesarenotbrokenintoanamepartandanextensionpart

(althoughextensionsarestillfrequentlyusedasameanstoclassify files).Insteadtheycancontainanykeyboardcharacterexceptfor'/'and beupto256characterslong(notehoweverthatcharacterssuchas*,?,# and&havespecialmeaninginmostshellsandshouldnotthereforebe usedinfilenames).Puttingspacesinfilenamesalsomakesthem difficulttomanipulateratherusetheunderscore'_'. 2. Directories Directoriesarecontainersorfoldersthatholdfiles,andother directories. 3. Devices Toprovideapplicationswitheasyaccesstohardwaredevices,UNIX allowsthemtobeusedinmuchthesamewayasordinaryfiles.There aretwotypesofdevicesinUNIXblockorienteddeviceswhich transferdatainblocks(e.g.harddisks)andcharacterorienteddevices thattransferdataonabytebybytebasis(e.g.modemsanddumb terminals). 4. Links Alinkisapointertoanotherfile.Therearetwotypesoflinksahard linktoafileisindistinguishablefromthefileitself.Asoftlink(or symboliclink)providesanindirectpointerorshortcuttoafile.Asoft linkisimplementedasadirectoryfileentrycontainingapathname.

2.3TypicalUNIXDirectoryStructure
TheUNIXfilesystemislaidoutasahierarchicaltreestructurewhichis anchoredataspecialtopleveldirectoryknownastheroot(designatedbya slash'/').Becauseofthetreestructure,adirectorycanhavemanychild directories,butonlyoneparentdirectory.Fig.2.1illustratesthislayout.

Fig.2.1:PartofatypicalUNIXfilesystemtree

Tospecifyalocationinthedirectoryhierarchy,wemustspecifyapath throughthetree.Thepathtoalocationcanbedefinedbyanabsolutepath fromtheroot/,orasarelativepathfromthecurrentworkingdirectory.To specifyapath,eachdirectoryalongtheroutefromthesourcetothe destinationmustbeincludedinthepath,witheachdirectoryinthesequence beingseparatedbyaslash.Tohelpwiththespecificationofrelativepaths, UNIXprovidestheshorthand"."forthecurrentdirectoryand".."forthe parentdirectory.Forexample,theabsolutepathtothedirectory"play"is /home/will/play,whiletherelativepathtothisdirectoryfrom"zeb"is ../will/play. Fig.2.2showssometypicaldirectoriesyouwillfindonUNIXsystemsand brieflydescribestheircontents.Notethatthesealthoughthesesubdirectories appearaspartofaseamlesslogicalfilesystem,theydonotneedbepresenton thesameharddiskdevice;somemayevenbelocatedonaremotemachine andaccessedacrossanetwork.

Directory / /bin /usr/bin /sbin /lib /usr/lib /tmp /home or /homes /etc /dev /proc

Typical Contents The "root" directory Essential low-level system utilities Higher-level system utilities and application programs Superuser system utilities (for performing system administration tasks) Program libraries (collections of system calls that can be included in programs by a compiler) for low-level system utilities Program libraries for higher-level user programs Temporary file storage space (can be used by any user) User home directories containing personal file space for each user. Each directory is named after the login of the user. UNIX system configuration and information files Hardware devices A pseudo-filesystem which is used as an interface to the kernel. Includes a subdirectory for each active program (or process).

Fig.2.2:TypicalUNIXdirectories WhenyoulogintoUNIX,yourcurrentworkingdirectoryisyouruserhome directory.Youcanrefertoyourhomedirectoryatanytimeas"~"andthe homedirectoryofotherusersas"~<login>".So~will/playisanother wayforuserjanetospecifyanabsolutepathtothedirectory /homes/will/play.Userwillmayrefertothedirectoryas~/play.

2.4DirectoryandFileHandlingCommands
Thissectiondescribessomeofthemoreimportantdirectoryandfilehandling commands.

pwd(print[current]workingdirectory)

pwddisplaysthefullabsolutepathtotheyourcurrentlocationinthe filesystem.So $ pwd /usr/bin impliesthat/usr/binisthecurrentworkingdirectory.

ls(listdirectory) ls liststhecontentsofadirectory.Ifnotargetdirectoryisgiven,then thecontentsofthecurrentworkingdirectoryaredisplayed.So,ifthe currentworkingdirectoryis/, $ ls bin boot

dev etc

home lib

mnt proc

share sbin

usr tmp

var vol

Actually,lsdoesn'tshowyoualltheentriesinadirectoryfilesand directoriesthatbeginwithadot(.)arehidden(thisincludesthe directories'.'and'..'whicharealwayspresent).Thereasonforthisis thatfilesthatbeginwitha.usuallycontainimportantconfiguration informationandshouldnotbechangedundernormalcircumstances.If youwanttoseeallfiles,lssupportsthe-aoption: $ ls -a

Eventhislistingisnotthathelpfultherearenohintstopropertiessuch asthesize,typeandownershipoffiles,justtheirnames.Toseemore detailedinformation,usethe-loption(longlisting),whichcanbe combinedwiththe-aoptionasfollows: $ ls -a -l (or,equivalently,) $ ls -al

Eachlineoftheoutputlookslikethis:

where: typeisasinglecharacterwhichiseither'd'(directory),'' (ordinaryfile),'l'(symboliclink),'b'(blockorienteddevice)or'c' (characterorienteddevice). permissionsisasetofcharactersdescribingaccessrights.There are9permissioncharacters,describing3accesstypesgivento3 usercategories.Thethreeaccesstypesareread('r'),write('w') andexecute('x'),andthethreeuserscategoriesaretheuserwho ownsthefile,usersinthegroupthatthefilebelongstoandother users(thegeneralpublic).An'r','w'or'x'charactermeansthe correspondingpermissionispresent;a''meansitisabsent. linksreferstothenumberoffilesystemlinkspointingtothe file/directory(seethediscussiononhard/softlinksinthenext section). ownerisusuallytheuserwhocreatedthefileordirectory. groupdenotesacollectionofuserswhoareallowedtoaccessthe fileaccordingtothegroupaccessrightsspecifiedinthe permissionsfield. sizeisthelengthofafile,orthenumberofbytesusedbythe operatingsystemtostorethelistoffilesinadirectory. dateisthedatewhenthefileordirectorywaslastmodified (writtento).The-uoptiondisplaythetimewhenthefilewaslast accessed(read). nameisthenameofthefileordirectory. lssupportsmoreoptions.Tofindoutwhattheyare,type:

$ man ls

man istheonlineUNIXusermanual,andyoucanuseittogethelp withcommandsandfindoutaboutwhatoptionsaresupported.Ithas quiteatersestylewhichisoftennotthathelpful,sosomeuserspreferto theusethe(nonstandard)infoutilityifitisinstalled: $ info ls

cd(change[currentworking]directory) $ cdpath changesyourcurrentworkingdirectorytopath(whichcanbean absoluteorarelativepath).Oneofthemostcommonrelativepathsto useis'..'(i.e.theparentdirectoryofthecurrentdirectory). Usedwithoutanytargetdirectory $ cd

resetsyourcurrentworkingdirectorytoyourhomedirectory(usefulif yougetlost).Ifyouchangeintoadirectoryandyousubsequentlywant toreturntoyouroriginaldirectory,use $ cd

mkdir(makedirectory) $ mkdir directory createsasubdirectorycalleddirectoryinthecurrentworkingdirectory. Youcanonlycreatesubdirectoriesinadirectoryifyouhavewrite permissiononthatdirectory.

rmdir (removedirectory)

$ rmdir directory removesthesubdirectorydirectoryfromthecurrentworkingdirectory. Youcanonlyremovesubdirectoriesiftheyarecompletelyempty(i.e. ofallentriesbesidesthe'.'and'..'directories).

cp(copy) cpisusedtomakecopiesoffilesorentiredirectories.Tocopyfiles, use: $ cp sourcefile(s)destination wheresourcefile(s)anddestinationspecifythesourceanddestination ofthecopyrespectively.Thebehaviourofcpdependsonwhetherthe destinationisafileoradirectory.Ifthedestinationisafile,onlyone sourcefileisallowedandcpmakesanewfilecalleddestinationthat hasthesamecontentsasthesourcefile.Ifthedestinationisadirectory, manysourcefilescanbespecified,eachofwhichwillbecopiedinto thedestinationdirectory.Section2.6willdiscussefficientspecification ofsourcefilesusingwildcardcharacters. Tocopyentiredirectories(includingtheircontents),usearecursive copy:

$ cp -rd sourcedirectoriesdestinationdirectory

mv(move/rename) mvisusedtorenamefiles/directoriesand/ormovethemfromone directoryintoanother.Exactlyonesourceandonedestinationmustbe specified: $ mv sourcedestination Ifdestinationisanexistingdirectory,thenewnameforsource(whether

itbeafileoradirectory)willbedestination/source.Ifsourceand destinationarebothfiles,sourceisrenameddestination.N.B.:if destinationisanexistingfileitwillbedestroyedandoverwrittenby source(youcanusethe-ioptionifyouwouldliketobeaskedfor confirmationbeforeafileisoverwritteninthisway).

rm (remove/delete) $ rm targetfile(s) removesthespecifiedfiles.Unlikeotheroperatingsystems,itisalmost impossibletorecoveradeletedfileunlessyouhaveabackup(thereis norecyclebin!)sousethiscommandwithcare.Ifyouwouldliketobe askedbeforefilesaredeleted,usethe-i option: $ rm -i myfile rm: remove 'myfile'? rmcanalsobeusedtodeletedirectories(alongwithalloftheir contents,includinganysubdirectoriestheycontain).Todothis,usethe -roption.Toavoidrmfromaskinganyquestionsorgivingerrors(e.g. ifthefiledoesn'texist)youusedthe-f (force)option.Extremecare needstobetakenwhenusingthisoptionconsiderwhatwouldhappen ifasystemadministratorwastryingtodeleteuser will'shome directoryandaccidentallytyped: $ rm -rf / home/will

(insteadof rm -rf /home/will). cat(catenate/type) $ cat targetfile(s) displaysthecontentsoftargetfile(s)onthescreen,oneaftertheother. Youcanalsouseittocreatefilesfromkeyboardinputasfollows(>is

theoutputredirectionoperator,whichwillbediscussedinthenext chapter): $ cat > hello.txt hello world! [ctrld] $ ls hello.txt hello.txt $ cat hello.txt hello world! $

moreandless(catenatewithpause) $ more targetfile(s) displaysthecontentsoftargetfile(s)onthescreen,pausingattheend ofeachscreenfulandaskingtheusertopressakey(usefulforlong files).Italsoincorporatesasearchingfacility(press'/'andthentypea phrasethatyouwanttolookfor). Youcanalsousemoretobreakuptheoutputofcommandsthat producemorethanonescreenfulofoutputasfollows(| isthepipe operator,whichwillbediscussedinthenextchapter): $ ls -l | more

lessisjustlikemore,exceptthathasafewextrafeatures(suchas allowinguserstoscrollbackwardsandforwardsthroughthedisplayed file).lessnotastandardutility,howeverandmaynotbepresenton allUNIXsystems.

2.5MakingHardandSoft(Symbolic)Links
Direct(hard)andindirect(softorsymbolic)linksfromonefileordirectoryto anothercanbecreatedusingthelncommand.

$ ln filenamelinkname createsanotherdirectoryentryforfilenamecalledlinkname(i.e.linknameisa hardlink).Bothdirectoryentriesappearidentical(andbothnowhavealink countof2).Ifeitherfilenameorlinknameismodified,thechangewillbe reflectedintheotherfile(sincetheyareinfactjusttwodifferentdirectory entriespointingtothesamefile). $ ln -s filenamelinkname createsashortcutcalledlinkname(i.e.linknameisasoftlink).Theshortcut appearsasanentrywithaspecialtype('l'): $ ln -s hello.txt bye.txt $ ls -l bye.txt lrwxrwxrwx 1 will finance 13 bye.txt -> hello.txt $ Thelinkcountofthesourcefileremainsunaffected.Noticethatthe permissionbitsonasymboliclinkarenotused(alwaysappearingas rwxrwxrwx).Insteadthepermissionsonthelinkaredeterminedbythe permissionsonthetarget(hello.txtinthiscase). Notethatyoucancreateasymboliclinktoafilethatdoesn'texist,butnota hardlink.Anotherdifferencebetweenthetwoisthatyoucancreatesymbolic linksacrossdifferentphysicaldiskdevicesorpartitions,buthardlinksare restrictedtothesamediskpartition.Finally,mostcurrentUNIX implementationsdonotallowhardlinkstopointtodirectories.

2.6Specifyingmultiplefilenames
Multiplefilenamescanbespecifiedusingspecialpatternmatchingcharacters. Therulesare:

'?'matchesanysinglecharacterinthatpositioninthefilename. '*'matcheszeroormorecharactersinthefilename.A'*'onitsown

willmatchallfiles.'*.*'matchesallfileswithcontaininga'.'. Charactersenclosedinsquarebrackets('['and']')willmatchany filenamethathasoneofthosecharactersinthatposition. Alistofcommaseparatedstringsenclosedincurlybraces("{"and"}") willbeexpandedasaCartesianproductwiththesurrounding characters.

Forexample:
1. ??? matchesallthreecharacterfilenames. 2. ?ell?matchesanyfivecharacterfilenameswith'ell'inthemiddle. 3. he*matchesanyfilenamebeginningwith'he'. 4. [m-z]*[a-l]matchesanyfilenamethatbeginswithaletterfrom'm'

to'z'andendsinaletterfrom'a'to'l'. 5. {/usr,}{/bin,/lib}/fileexpandsto/usr/bin/file /usr/lib/file /bin/file and /lib/file. NotethattheUNIXshellperformstheseexpansions(includinganyfilename matching)onacommand'sargumentsbeforethecommandisexecuted.

2.7Quotes
Aswehaveseencertainspecialcharacters(e.g.'*','-','{'etc.)areinterpreted inaspecialwaybytheshell.Inordertopassargumentsthatusethese characterstocommandsdirectly(i.e.withoutfilenameexpansionetc.),we needtousespecialquotingcharacters.Therearethreelevelsofquotingthat youcantry:
1. Tryinserta'\'infrontofthespecialcharacter. 2. Usedoublequotes(")aroundargumentstopreventmostexpansions. 3. Usesingleforwardquotes(')aroundargumentstopreventall

expansions.

ThereisafourthtypeofquotinginUNIX.Singlebackwardquotes(`)are usedtopasstheoutputofsomecommandasaninputargumenttoanother.For example:

$ hostname rose $ echo this machine is called `hostname` this machine is called rose

IntroductiontoUNIX: ExerciseSheetTwo
1. Trythefollowingcommandsequence: cd pwd ls -al cd .

pwd (wheredidthatgetyou?) cd .. pwd ls -al cd .. pwd ls -al cd .. pwd (whathappensnow) cd /etc ls -al | more cat passwd cd - pwd 2. Continuetoexplorethefilesystemtreeusingcd,ls,pwdandcat. Lookin/bin,/usr/bin,/sbin,/tmpand/boot.Whatdoyou see? 3. Explore/dev.Canyouidentifywhatdevicesareavailable?Whichare characterorientedandwhichareblockoriented?Canyouidentifyyour tty(terminal)device(typingwho am i mighthelp);whoistheowner ofyourtty(usels -l)? 4. Explore/proc.Displaythecontentsofthefilesinterrupts, devices,cpuinfo,meminfoanduptimeusingcat.Canyousee

whywesay/procisapseudofilesystemwhichallowsaccessto kerneldatastructures? 5. Changetothehomedirectoryofanotheruserdirectly,usingcd ~username. 6. Changebackintoyourhomedirectory. 7. Makesubdirectoriescalledworkandplay. 8. Deletethesubdirectorycalledwork. 9. Copythefile/etc/passwdintoyourhomedirectory. 10.Moveitintothesubdirectoryplay. 11.Changeintosubdirectoryplayandcreateasymboliclinkcalled terminalthatpointstoyourttydevice.Whathappensifyoutryto makeahardlinktothettydevice? 12.Whatisthedifferencebetweenlistingthecontentsofdirectoryplay withls -landls -L? 13.Createafilecalledhello.txtthatcontainsthewords"hello world".Canyouuse"cp"using"terminal"asthesourcefileto achievethesameeffect? 14.Copyhello.txttoterminal.Whathappens? 15.Imagineyouwereworkingonasystemandsomeoneaccidentally deletedthelscommand(/bin/ls).Howcouldyougetalistofthe filesinthecurrentdirectory?Tryit. 16.Howwouldyoucreateandthendeleteafilecalled"$SHELL"?Tryit. 17.Howwouldyoucreateandthendeleteafilethatbeginswiththesymbol #?Tryit. 18.Howwouldyoucreateandthendeleteafilethatbeginswiththesymbol -?Tryit. 19.Whatistheoutputofthecommand: echo {con,pre}{sent,fer}{s,ed}?Now,fromyourhome directory,copy/etc/passwdand/etc/groupintoyourhome directoryinonecommandgiventhatyoucanonlytype/etconce. 20.Stillinyourhomedirectory,copytheentiredirectoryplaytoa directorycalledwork,preservingthesymboliclink. 21.Deletetheworkdirectoryanditscontentswithonecommand.Accept nocomplaintsorqueries. 22.Changeintoadirectorythatdoesnotbelongtoyouandtrytodeleteall

thefiles(avoid/procor/dev,justincase!) 23.Experimentwiththeoptionsonthelscommand.Whatdothed,i,R andFoptionsdo?

IntroductiontoUNIX: LectureThree

3.1Objectives
Thislecturecovers:

Fileanddirectorypermissionsinmoredetailandhowthesecanbe changed. Waystoexaminethecontentsoffiles. Howtofindfileswhenyoudon'tknowhowtheirexactlocation. Waysofsearchingfilesfortextpatterns. Howtosortfiles. Toolsforcompressingfilesandmakingbackups. Accessingfloppydisksandotherremovablemedia.

3.2FileandDirectoryPermissions
Permission File read User can look at the contents of the file User can modify the contents of the file User can use the filename as a UNIX Directory User can list the files in the directory User can create new files and remove existing files in the directory User can change into the directory, but cannot list the files unless (s)he has read permission. User can

write execute

command

read files if (s)he has read permission on them.

Fig3.1:Interpretationofpermissionsforfilesanddirectories Aswehaveseeninthepreviouschapter,everyfileordirectoryonaUNIX systemhasthreetypesofpermissions,describingwhatoperationscanbe performedonitbyvariouscategoriesofusers.Thepermissionsareread(r), write(w)andexecute(x),andthethreecategoriesofusersareuser/owner(u), group(g)andothers(o).Becausefilesanddirectoriesaredifferententities, theinterpretationofthepermissionsassignedtoeachdiffersslightly,as showninFig3.1. Fileanddirectorypermissionscanonlybemodifiedbytheirowners,orbythe superuser(root),byusingthechmodsystemutility.

chmod (change[fileordirectory]mode) $ chmod optionsfiles chmodacceptsoptionsintwoforms.Firstly,permissionsmaybe specifiedasasequenceof3octaldigits(octalislikedecimalexceptthat thedigitrangeis0to7insteadof0to9).Eachoctaldigitrepresentsthe accesspermissionsfortheuser/owner,groupandothersrespectively. Themappingsofpermissionsontotheircorrespondingoctaldigitsisas follows:
x w wx r rx rw rwx 0 1 2 3 4 5 6 7

Forexamplethecommand:

$ chmod 600 private.txt setsthepermissionsonprivate.txttorw-------(i.e.onlythe ownercanreadandwritetothefile). Permissionsmaybespecifiedsymbolically,usingthesymbolsu(user), g(group),o(other),a(all),r(read),w(write),x(execute),+(add permission),-(takeawaypermission)and=(assignpermission).For example,thecommand: $ chmod ug=rw,o-rw,a-x *.txt setsthepermissionsonallfilesendingin*.txttorw-rw----(i.e. theownerandusersinthefile'sgroupcanreadandwritetothefile, whilethegeneralpublicdonothaveanysortofaccess). chmodalsosupportsa-Roptionwhichcanbeusedtorecursively modifyfilepermissions,e.g. $ chmod -R go+r play willgrantgroupandotherreadrightstothedirectoryplayandallof thefilesanddirectorieswithinplay.

chgrp(changegroup) $ chgrp groupfiles canbeusedtochangethegroupthatafileordirectorybelongsto.It alsosupportsa-Roption.

3.3InspectingFileContent
Besidescat thereareseveralotherusefulutilitiesforinvestigatingthe contentsoffiles:

filefilename(s)

file analyzesafile'scontentsforyouandreportsahighlevel descriptionofwhattypeoffileitappearstobe: $ file myprog.c letter.txt webpage.html myprog.c: C program text letter.txt: English text webpage.html: HTML document text

filecanidentifyawiderangeoffilesbutsometimesgetsunderstandably confused(e.g.whentryingtoautomaticallydetectthedifference betweenC++andJavacode).

head, tailfilename headandtaildisplaythefirstandlastfewlinesinafile respectively.Youcanspecifythenumberoflinesasanoption,e.g. $ tail -20 messages.txt $ head -5 messages.txt tail includesauseful-foptionthatcanbeusedtocontinuously monitorthelastfewlinesofa(possiblychanging)file.Thiscanbeused tomonitorlogfiles,forexample: $ tail -f /var/log/messages

continuouslyoutputsthelatestadditionstothesystemlogfile. objdumpoptionsbinaryfile objdumpcanbeusedtodisassemblebinaryfilesthatisitcanshow themachinelanguageinstructionswhichmakeupcompiledapplication programsandsystemutilities.

odoptionsfilename(octaldump)

odcanbeusedtodisplaysthecontentsofabinaryortextfileina varietyofformats,e.g. $ cat hello.txt hello world $ od -c hello.txt 0000000 h e l l o w o r l d \n 0000014 $ od -x hello.txt 0000000 6865 6c6c 6f20 776f 726c 640a 0000014 Therearealsoseveralotherusefulcontentinspectorsthatarenonstandard(in termsofavailabilityonUNIXsystems)butareneverthelessinwidespread use.TheyaresummarisedinFig.3.2.
File type Portable Document Format Postscript Document DVI Document JPEG Image GIF Image MPEG movie WAV sound file HTML document Typical extension .pdf .ps .dvi .jpg .gif .mpg .wav .html Content viewer acroread ghostview xdvi xv xv mpeg_play realplayer netscape

Fig3.2:Otherfiletypesandappropriatecontentviewers.

3.4FindingFiles
Thereareatleastthreewaystofindfileswhenyoudon'tknowtheirexact location:

find Ifyouhavearoughideaofthedirectorytreethefilemightbein(or evenifyoudon'tandyou'repreparedtowaitawhile)youcanuse

find: $ find directory -name targetfile -print

findwilllookforafilecalledtargetfileinanypartofthedirectory treerootedatdirectory.targetfilecanincludewildcardcharacters.For example: $ find /home -name "*.txt" -print 2>/dev/null willsearchalluserdirectoriesforanyfileendingin".txt"andoutput anymatchingfiles(withafullabsoluteorrelativepath).Herethe quotes(")arenecessarytoavoidfilenameexpansion,whilethe 2>/dev/null suppresseserrormessages(arisingfromerrorssuch asnotbeingabletoreadthecontentsofdirectoriesforwhichtheuser doesnothavetherightpermissions). findcaninfactdoalotmorethanjustfindfilesbyname.Itcanfind filesbytype(e.g. -type fforfiles,-type dfordirectories),by permissions(e.g.-perm o=rforallfilesanddirectoriesthatcanbe readbyothers),bysize(-size)etc.Youcanalsoexecutecommands onthefilesyoufind.Forexample,
$ find . -name "*.txt" -exec wc -l '{}' ';'

countsthenumberoflinesineverytextfileinandbelowthecurrent directory.The'{}' isreplacedbythenameofeachfilefoundandthe ';'endsthe-execclause. Formoreinformationaboutfindanditsabilities,useman find and/orinfo find.

which(sometimesalsocalledwhence)command Ifyoucanexecuteanapplicationprogramorsystemutilitybytypingits nameattheshellprompt,youcanusewhichtofindoutwhereitis storedondisk.Forexample:

$ which ls /bin/ls

locate string findcantakealongtimetoexecuteifyouaresearchingalarge filespace(e.g.searchingfrom/downwards).Thelocate command providesamuchfasterwayoflocatingallfileswhosenamesmatcha particularsearchstring.Forexample: $ locate ".txt"

willfindallfilenamesinthefilesystemthatcontain".txt"anywhere intheirfullpaths. Onedisadvantageoflocateisitstoresallfilenamesonthesystemin anindexthatisusuallyupdatedonlyonceaday.Thismeanslocate willnotfindfilesthathavebeencreatedveryrecently.Itmayalso reportfilenamesasbeingpresenteventhoughthefilehasjustbeen deleted.Unlikefind,locate cannottrackdownfilesonthebasisof theirpermissions,sizeandsoon.

3.5FindingTextinFiles

grep(GeneralRegularExpressionPrint) $ grep optionspatternfiles

grep searchesthenamedfiles(orstandardinputifnofilesarenamed) forlinesthatmatchagivenpattern.Thedefaultbehaviourofgrepisto printoutthematchinglines.Forexample: $ grep hello *.txt

searchesalltextfilesinthecurrentdirectoryforlinescontaining "hello".Someofthemoreusefuloptionsthatgrepprovidesare: -c(printacountofthenumberoflinesthatmatch),-i(ignorecase),

-v(printoutthelinesthatdon'tmatchthepattern)and-n(printoutthe linenumberbeforeprintingthematchingline).So $ grep -vi hello *.txt

searchesalltextfilesinthecurrentdirectoryforlinesthatdonot containanyformofthewordhello(e.g.Hello,HELLO,orhELlO). Ifyouwanttosearchallfilesinanentiredirectorytreeforaparticular pattern,youcancombinegrepwithfind usingbackwardsingle quotestopasstheoutputfromfindintogrep.So $ grep hello `find . -name "*.txt" -print` willsearchalltextfilesinthedirectorytreerootedatthecurrent directoryforlinescontainingtheword"hello". Thepatternsthatgrepusesareactuallyaspecialtypeofpattern knownasregularexpressions.Justlikearithemeticexpressions, regularexpressionsaremadeupofbasicsubexpressionscombinedby operators. Themostfundamentalexpressionisaregularexpressionthatmatchesa singlecharacter.Mostcharacters,includingalllettersanddigits,are regularexpressionsthatmatchthemselves.Anyothercharacterwith specialmeaningmaybequotedbyprecedingitwithabackslash(\).A listofcharactersenclosedby'['and']'matchesanysinglecharacterin thatlist;ifthefirstcharacterofthelististhecaret`^',thenitmatches anycharacternotinthelist.Arangeofcharacterscanbespecified usingadash(-)betweenthefirstandlastitemsinthelist.So[0-9] matchesanydigitand[^a-z]matchesanycharacterthatisnotadigit. Thecaret`^'andthedollarsign`$'arespecialcharactersthat matchthebeginningandendofalinerespectively.Thedot'.'matches anycharacter.So $ grep ^..[l-z]$ hello.txt

matchesanylineinhello.txtthatcontainsathreecharacter sequencethatendswithalowercaseletterfromltoz. egrep(extendedgrep)isavariantofgrepthatsupportsmore sophisticatedregularexpressions.Heretworegularexpressionsmaybe joinedbytheoperator`|';theresultingregularexpressionmatchesany stringmatchingeithersubexpression.Brackets'('and')'maybeused forgroupingregularexpressions.Inaddition,aregularexpressionmay befollowedbyoneofseveralrepetitionoperators: `?'meanstheprecedingitemisoptional(matchedatmostonce). `*' meanstheprecedingitemwillbematchedzeroormoretimes. `+'meanstheprecedingitemwillbematchedoneormoretimes. `{N}' meanstheprecedingitemismatchedexactlyNtimes. `{N,}' meanstheprecedingitemismatchedNormoretimes. `{N,M}'meanstheprecedingitemismatchedatleastNtimes,butnot morethanMtimes. Forexample,ifegrepwasgiventheregularexpression '(^[0-9]{1,5}[a-zA-Z ]+$)|none' itwouldmatchanylinethateither: beginswithanumberuptofivedigitslong,followedbya sequenceofoneormorelettersorspaces,or containsthewordnone Youcanreadmoreaboutregularexpressionsonthegrepandegrep manualpages.

NotethatUNIXsystemsalsousuallysupportanothergrepvariant calledfgrep(fixedgrep)whichsimplylooksforafixedstringinsidea file(butthisfacilityislargelyredundant).

3.6Sortingfiles
TherearetwofacilitiesthatareusefulforsortingfilesinUNIX:

sortfilenames sortsortslinescontainedinagroupoffilesalphabetically(orifthe -noptionisspecified)numerically.Thesortedoutputisdisplayedon thescreen,andmaybestoredinanotherfilebyredirectingtheoutput. So $ sort input1.txt input2.txt > output.txt outputsthesortedconcentenationoffilesinput1.txtand input2.txt tothefileoutput.txt.

uniq filename uniq removesduplicateadjacentlinesfromafile.Thisfacilityismost usefulwhencombinedwithsort: $ sort input.txt | uniq > output.txt

3.7FileCompressionandBackup
UNIXsystemsusuallysupportanumberofutilitiesforbackingupand compressingfiles.Themostusefulare:

tar(tapearchiver) tarbacksupentiredirectoriesandfilesontoatapedeviceor(more commonly)intoasinglediskfileknownasanarchive.Anarchiveisa filethatcontainsotherfilesplusinformationaboutthem,suchastheir filename,owner,timestamps,andaccesspermissions.tardoesnot performanycompressionbydefault. Tocreateadiskfiletararchive,use $ tar -cvf archivenamefilenames wherearchivenamewillusuallyhavea.tarextension.Herethec optionmeanscreate,vmeansverbose(outputfilenamesastheyare

archived),andfmeansfile.Tolistthecontentsofatararchive,use $ tar -tvf archivename Torestorefilesfromatararchive,use $ tar -xvf archivename

cpio cpio isanotherfacilityforcreatingandreadingarchives.Unliketar, cpiodoesn'tautomaticallyarchivethecontentsofdirectories,soit's commontocombinecpiowithfindwhencreatinganarchive: $ find . -print -depth | cpio -ov -Htar > archivename Thiswilltakeallthefilesinthecurrentdirectoryandthe directoriesbelowandplacetheminanarchivecalledarchivename.The -depthoptioncontrolstheorderinwhichthefilenamesareproduced andisrecommendedtopreventproblemswithdirectorypermissions whendoingarestore.The-ooptioncreatesthearchive,the-voption printsthenamesofthefilesarchivedastheyareaddedandthe-H optionspecifiesanarchiveformattype(inthiscaseitcreatesatar archive).Anothercommonarchivetypeiscrc,aportableformatwith achecksumforerrorcontrol. Tolistthecontentsofacpioarchive,use $ cpio -tv < archivename Torestorefiles,use: $ cpio -idv < archivename Herethe-doptionwillcreatedirectoriesasnecessary.Toforcecpio toextractfilesontopoffilesofthesamenamethatalreadyexist(and

havethesameorlatermodificationtime),usethe-uoption.

compress,gzip compressandgzipareutilitiesforcompressinganddecompressing individualfiles(whichmaybeormaynotbearchivefiles).To compressfiles,use: $ compress filename or $ gzip filename

Ineachcase,filenamewillbedeletedandreplacedbyacompressedfile calledfilename.Zorfilename.gz.Toreversethecompressionprocess, use: or $ compress -d filename $ gzip -d filename

3.8HandlingRemovableMedia(e.g.floppydisks)
UNIXsupportstoolsforaccessingremovablemediasuchasCDROMsand floppydisks.

mount, umount Themountcommandservestoattachthefilesystemfoundonsome devicetothefilesystemtree.Conversely,theumountcommandwill detachitagain(itisveryimportanttoremembertodothiswhen removingthefloppyorCDROM).Thefile/etc/fstabcontainsa listofdevicesandthepointsatwhichtheywillbeattachedtothemain filesystem: $ cat /etc/fstab /dev/fd0 /mnt/floppy 0 0 /dev/hdc /mnt/cdrom auto rw,user,noauto

iso9660 ro,user,noauto

0 0 Inthiscase,themountpointforthefloppydriveis/mnt/floppyand themountpointfortheCDROMis/mnt/cdrom.Toaccessafloppy wecanuse: $ mount /mnt/floppy $ cd /mnt/floppy $ ls (etc...)

Toforceallchangeddatatobewrittenbacktothefloppyandtodetach thefloppydiskfromthefilesystem,weuse: $ umount /mnt/floppy

mtools Iftheyareinstalled,the(nonstandard)mtoolsutilitiesprovidea convenientwayofaccessingDOSformattedfloppieswithouthavingto mountandunmountfilesystems.YoucanuseDOStypecommandslike "mdir a:","mcopy a:*.* .","mformat a:",etc.(seethe mtools manualpagesformoredetails).

IntroductiontoUNIX: ExerciseSheetThree
2.

1. Describethreedifferentwaysofsettingthepermissionsonafileor

3.

4. 5.

6.

7.

8. 9.

directorytor--r--r--.Createafileandseeifthisworks. Teamupwithapartner.Copy/bin/shtoyourhomedirectory.Type "chmod +s sh".Checkthepermissionsonshinthedirectory listing.Nowaskyourpartnertochangeintoyourhomedirectoryand runtheprogram./sh.Askthemtoruntheidcommand.What's happened?Yourpartnercantypeexittoreturntotheirshell. Whatwouldhappenifthesystemadministratorcreatedashfileinthis way?Whyisitsometimesnecessaryforasystemadministratortouse thisfeatureusingprogramsotherthansh? Deleteshfromyourhomedirectory(oratleasttodoachmod -s sh). Modifythepermissionsonyourhomedirectorytomakeitcompletely private.Checkthatyourpartnercan'taccessyourdirectory.Nowput thepermissionsbacktohowtheywere. Typeumask 000andthencreateafilecalledworld.txt containingthewords"hello world".Lookatthepermissionsonthe file.What'shappened?Nowtypeumask 022 andcreateafilecalled world2.txt.Whenmightthisfeaturebeuseful? Createafilecalled"hello.txt"inyourhomedirectoryusingthe commandcat -u > hello.txt.Askyourpartnertochangeinto yourhomedirectoryandruntail -f hello.txt.Nowtype severallinesintohello.txt.Whatappearsonyourpartner'sscreen? Usefindtodisplaythenamesofallfilesinthe/homesubdirectory tree.Canyoudothiswithoutdisplayingerrorsforfilesyoucan'tread? Usefindtodisplaythenamesofallfilesinthesystemthatarebigger

than1MB. 10.Usefindandfiletodisplayallfilesinthe/homesubdirectorytree, aswellasaguessatwhatsortofafiletheyare.Dothisintwodifferent ways. 11.Usegreptoisolatethelinein/etc/passwdthatcontainsyourlogin details. 12.Usefindandgrepandsorttodisplayasortedlistofallfilesinthe /homesubdirectorytreethatcontainthewordhello somewhere insidethem. 13.Uselocatetofindallfilenamesthatcontainthewordemacs.Can youcombinethiswithgreptoavoiddisplayingallfilenames containingthewordlib? 14.Createafilecontainingsomelinesthatyouthinkwouldmatchthe regularexpression:(^[0-9]{1,5}[a-zA-z ]+$)|noneand somelinesthatyouthinkwouldnotmatch.Useegreptoseeifyour intuitioniscorrect. 15.Archivethecontentsofyourhomedirectory(includingany subdirectories)usingtarandcpio.Compressthetararchivewith compress,andthecpioarchivewithgzip.Nowextracttheir contents. 16.OnLinuxsystems,thefile/dev/urandomisaconstantlygenerated randomstreamofcharacters.Canyouusethisfilewithodtoprintouta randomdecimalnumber? 17.Typemount(withnoparameters)andtrytointerprettheoutput.

IntroductiontoUNIX: LectureFour

4.1Objectives
Thislecturecovers:

Theconceptofaprocess. Passingoutputfromoneprocessasinputtoanotherusingpipes. Redirectingprocessinputandoutput. Controllingprocessesassociatedwiththecurrentshell. Controllingotherprocesses.

4.2Processes
Aprocessisaprograminexecution.Everytimeyouinvokeasystemutility oranapplicationprogramfromashell,oneormore"child"processesare createdbytheshellinresponsetoyourcommand.AllUNIXprocessesare identifiedbyauniqueprocessidentifierorPID.Animportantprocessthatis alwayspresentistheinitprocess.Thisisthefirstprocesstobecreated whenaUNIXsystemstartsupandusuallyhasaPIDof1.Allotherprocesses aresaidtobe"descendants"ofinit.

4.3Pipes
Thepipe('|')operatorisusedtocreateconcurrentlyexecutingprocessesthat passdatadirectlytooneanother.Itisusefulforcombiningsystemutilitiesto performmorecomplexfunctions.Forexample:

$ cat hello.txt | sort | uniq

createsthreeprocesses(correspondingtocat,sort anduniq)which executeconcurrently.Astheyexecute,theoutputofthewhoprocessispassed ontothesort processwhichisinturnpassedontotheuniqprocess. uniqdisplaysitsoutputonthescreen(asortedlistofuserswithduplicate linesremoved).Similarly: $ cat hello.txt | grep "dog" | grep -v "cat"

findsalllinesinhello.txtthatcontainthestring"dog"butdonotcontain thestring"cat".

4.4Redirectinginputandoutput
Theoutputfromprogramsisusuallywrittentothescreen,whiletheirinput usuallycomesfromthekeyboard(ifnofileargumentsaregiven).Intechnical terms,wesaythatprocessesusuallywritetostandardoutput(thescreen) andtaketheirinputfromstandardinput(thekeyboard).Thereisinfact anotheroutputchannelcalledstandarderror,whereprocesseswritetheir errormessages;bydefaulterrormessagesarealsosenttothescreen. Toredirectstandardoutputtoafileinsteadofthescreen,weusethe> operator: $ echo hello hello $ echo hello > output $ cat output hello

Inthiscase,thecontentsofthefileoutputwillbedestroyedifthefile alreadyexists.Ifinsteadwewanttoappendtheoutputoftheechocommand tothefile,wecanusethe>>operator: $ echo bye >> output $ cat output hello

bye Tocapturestandarderror,prefixthe>operatorwitha2(inUNIXthefile numbers0,1and2areassignedtostandardinput,standardoutputand standarderrorrespectively),e.g.: $ cat nonexistent 2>errors $ cat errors cat: nonexistent: No such file or directory $ Youcanredirectstandarderrorandstandardoutputtotwodifferentfiles: $ find . -print 1>errors 2>files ortothesamefile: $ find . -print 1>output 2>output or $ find . -print >& output

Standardinputcanalsoberedirectedusingthe<operator,sothatinputisread fromafileinsteadofthekeyboard: $ cat < output hello bye

Youcancombineinputredirectionwithoutputredirection,butbecarefulnot tousethesamefilenameinbothplaces.Forexample: $ cat < output > output

willdestroythecontentsofthefileoutput.Thisisbecausethefirstthingthe shelldoeswhenitseesthe>operatoristocreateanemptyfilereadyforthe output. Onelastpointtonoteisthatwecanpassstandardoutputtosystemutilities thatrequirefilenamesas"-":

$ cat package.tar.gz | gzip -d | tar tvf - Heretheoutputofthegzip -dcommandisusedastheinputfiletothetar command.

4.5Controllingprocessesassociatedwiththecurrent shell
Mostshellsprovidesophisticatedjobcontrolfacilitiesthatletyoucontrol manyrunningjobs(i.e.processes)atthesametime.Thisisusefulif,for example,youareeditingatextfileandwantotinterruptyoureditingtodo somethingelse.Withjobcontrol,youcansuspendtheeditor,gobacktothe shellprompt,andstartworkonsomethingelse.Whenyouarefinished,you canswitchbacktotheeditorandcontinueasifyouhadn'tleft. Jobscaneitherbeintheforegroundorthebackground.Therecanbeonly onejobintheforegroundatanytime.Theforegroundjobhascontrolofthe shellwithwhichyouinteractitreceivesinputfromthekeyboardandsends outputtothescreen.Jobsinthebackgrounddonotreceiveinputfromthe terminal,generallyrunningalongquietlywithouttheneedforinteraction(and drawingittoyourattentioniftheydo). Theforegroundjobmaybesuspended,i.e.temporarilystopped,bypressing theCtrlZkey.Asuspendedjobcanbemadetocontinuerunninginthe foregroundorbackgroundasneededbytyping"fg"or"bg"respectively. Notethatsuspendingajobisverydifferentfrominterruptingajob(by pressingtheinterruptkey,usuallyCtrlC);interruptedjobsarekilledoff permanentlyandcannotberesumed. Backgroundjobscanalsoberundirectlyfromthecommandline,by appendinga'&'charactertothecommandline.Forexample: $ find / -print 1>output 2>errors & [1] 27501 $

Herethe[1]returnedbytheshellrepresentsthejobnumberofthe

backgroundprocess,andthe27501isthePIDoftheprocess.Toseealistof allthejobsassociatedwiththecurrentshell,typejobs: $ jobs [1]+ Running $ find / -print 1>output 2>errors &

Notethatifyouhavemorethanonejobyoucanrefertothejobas%nwhere n isthejobnumber.Soforexamplefg %3 resumesjobnumber3inthe foreground. TofindouttheprocessID'softheunderlyingprocessesassociatedwiththe shellanditsjobs,useps (processshow): $ ps PID 17717 27501 27502 TTY pts/10 pts/10 pts/10

TIME 00:00:00 00:00:01 00:00:00

CMD bash find ps

SoherethePIDoftheshell(bash)is17717,thePIDoffindis27501and thePIDofpsis27502. Toterminateaprocessorjobabrubtly,usethekillcommand.killallows jobstoreferredtointwowaysbytheirPIDorbytheirjobnumber. So $ kill %1 or $ kill 27501 wouldterminatethefindprocess.Actuallykillonlysendstheprocessa signalrequestingitshutdownandexitgracefully(theSIGTERMsignal),so thismaynotalwayswork.Toforceaprocesstoterminateabruptly(andwith ahigherprobabilityofsucess),usea-9option(theSIGKILLsignal):

$ kill -9 27501

killcanbeusedtosendmanyothertypesofsignalstorunningprocesses. Forexamplea-19option(SIGSTOP)willsuspendarunningprocess.Tosee

alistofsuchsignals,runkill -l.

4.6Controllingotherprocesses
Youcanalsousepstoshowallprocessesrunningonthemachine(notjust theprocessesinyourcurrentshell): $ ps -fae (orps -auxonBSDmachines)

ps -aeHdisplaysafullprocesshierarchy(includingtheinitprocess).

ManyUNIXversionshaveasystemutilitycalledtopthatprovidesan interactivewaytomonitorsystemactivity.Detailedstatisticsaboutcurrently runningprocessesaredisplayedandconstantlyrefreshed.Processesare displayedinorderofCPUutilization.Usefulkeysintopare: s - setupdatefrequencyk - killprocess(byPID) u - displayprocessesofoneuserq - quit Onsomesystems,theutilitywisanoninteractivesubstitutefortop. OneotherusefulprocesscontrolutilitythatcanbefoundonmostUNIX systemsisthepkillcommand.Youcanusepkilltokillprocessesby nameinsteadofPIDorjobnumber.Soanotherwaytokilloffourbackground findprocess(alongwithanyanotherfindprocesseswearerunning)would be: $ pkill find [1]+ Terminated find / -print 1>output 2>errors $ Notethat,forobvioussecurityreasons,youcanonlykillprocessesthatbelong toyou(unlessyouarethesuperuser).

IntroductiontoUNIX: ExerciseSheetFour

1. Archivethecontentsofyourhomedirectoryusingtar.Compressthe

tarfilewithgzip.Nowuncompressandunarchivethe.tar.gzfile usingcat,tarandgzipononecommandline. 2. Usefindtocompilealistofalldirectoriesinthesystem,redirecting theoutputsothatthelistofdirectoriesendsupinafilecalled directories.txtandthelistoferrormessagesendsupinafile callederrors.txt. 3. Trythecommandsleep 5.Whatdoesthiscommanddo? 4. Runthecommandinthebackgroundusing&. 5. Runsleep 15intheforeground,suspenditwithCtrlzandthenput itintothebackgroundwithbg.Typejobs.Typeps.Bringthejob backintotheforegroundwithfg. 6. Runsleep 15inthebackgroundusing&,andthenusekillto terminatetheprocessbyitsjobnumber.Repeat,exceptthistimekillthe processbyspecifyingitsPID. 7. Runsleep 15inthebackgroundusing&,andthenusekillto suspendtheprocess.Usebgtocontinuerunningtheprocess. 8. Startupanumberof sleep 60processesinthebackground,and terminatethemallatthesametimeusingthepkillcommand. 9. Useps,wandtoptoshowallprocessesthatareexecuting. 10.Useps -aeHtodisplaytheprocesshierarchy.Lookfortheinit process.Seeifyoucanidentifyimportantsystemdaemons.Canyou alsoidentifyyourshellanditssubprocesses? 11.Combineps -faewithgreptoshowallprocessesthatyouare executing,withtheexceptionoftheps -faeandgrepcommands. 12.Startasleep 300processrunninginthebackground.Logoffthe

server,andlogbackinagain.Listalltheprocessesthatyouarerunning. Whathappenedtoyoursleepprocess?Nowrepeat,exceptthistime startbyrunningnohup sleep 300. 13.Multiplejobscanbeissuedfromthesamecommandlineusingthe operators;,&&and||.Trycombiningthecommandscat nonexistentandecho hellousingeachoftheseoperators. Reversetheorderofthecommandsandtryagain.Whataretherules aboutwhenthecommandswillbeexecuted? 14.Whatdoesthexargscommanddo?Canyoucombineitwithfind andgreptofindyetanotherwayofsearchingallfilesinthe/home subdirectorytreeforthewordhello? 15.Whatdoesthecutcommanddo?Canyouuseittogetherwithwto producealistofloginnamesandCPUtimescorrespondingtoeach activeprocess?Canyounow(allonthesamecommandline)usesort andheadortailtofindtheuserwhoseprocessisusingthemost CPU?

IntroductiontoUNIX: LectureFive
5.1Objectives

ThislectureintroducesotherusefulUNIXsystemutilitiesandcovers:

Connectingtoremotemachines. Networkingroutingutilities. Remotefiletransfer. OtherInternetrelatedutilities. Facilitiesforuserinformationandcommunication. Printercontrol. Emailutilities. Advancedtextfileprocessingwithsedandawk. Targetdirectedcompilationwithmake. VersioncontrolwithCVS. C++compilationfacilities. Manualpages.

5.2ConnectingtoRemoteMachines

telnetmachinename telnetprovidesaninsecuremechanismforloggingintoremote machines.Itisinsecurebecausealldata(includingyourusernameand password)ispassedinunencryptedformatoverthenetwork.Forthis reason,telnetloginaccessisdisabledonmostsystemsandwhere possibleitshouldbeavoidedinfavourofsecurealternativessuchas ssh.

telnetisstillausefulutility,however,because,byspecifying differentportnumbers,telnet canbeusedtoconnecttoother servicesofferedbyremotemachinesbesidesremotelogin(e.g.web pages,email,etc.)andrevealthemechanismsbehindhowthose servicesareoffered.Forexample, $ telnet www.doc.ic.ac.uk 80 Trying 146.169.1.10... Connected to seagull.doc.ic.ac.uk (146.169.1.10). Escape character is '^]'. GET / HTTP/1.0 HTTP/1.1 200 OK Date: Sun, 10 Dec 2000 21:06:34 GMT Server: Apache/1.3.14 (Unix) Last-Modified: Tue, 28 Nov 2000 16:09:20 GMT ETag: "23dcfd-3806-3a23d8b0" Accept-Ranges: bytes Content-Length: 14342 Connection: close Content-Type: text/html <HTML> <HEAD> <TITLE>Department of Computing, Imperial College, London: Home Page</TITLE> </HEAD> (etc) Herewww.doc.ic.ac.ukisthenameoftheremotemachine(inthis casethewebserverfortheDepartmentofComputingatImperial CollegeinLondon).Likemostwebservers,itofferswebpageservices onport80throughthedaemonhttpd(toseewhatotherservicesare potentiallyavailableonamachine,havealookatthefile /etc/services;andtoseewhatservicesareactuallyactive,see /etc/inetd.conf).ByenteringavalidHTTPGETcommand (HTTPistheprotocolusedtoservewebpages)weobtainthetoplevel

homepageinHTMLformat.Thisisexactlythesameprocessthatis usedbyawebbrowsertoaccesswebpages.

rlogin, rsh rlogin and rsh areinsecurefacilitiesforloggingintoremote machinesandforexecutingcommandsonremotemachines respectively.Alongwithtelnet,theyhavebeensupersededbyssh.

ssh machinename(secureshell) ssh isasecurealternativeforremoteloginandalsoforexecuting commandsinaremotemachine.Itisintendedtoreplacerloginand rsh,andprovidesecureencryptedcommunicationsbetweentwo untrustedhostsoveraninsecurenetwork.X11connections(i.e. graphics)canalsobeforwardedoverthesecurechannel(another advantageovertelnet,rloginandrsh).sshisnotastandard systemutility,althoughitisadefactostandard.Itcanbeobtainedfrom http://www.ssh.org.Agoodintroductionpagegivingmorebackground andshowingyouhowtosetupsshis http://www.tac.nyc.ny.us/~kim/ssh/. sshclientsarealsoavailableforWindowsmachines(e.g.thereisa goodsshclientcalledputty).

5.3Networkroutingutilities

pingmachinename Thepingutilityisusefulforcheckingroundtripresponsetime betweenmachines.e.g. $ ping www.doc.ic.ac.uk

measuresthereponsetimedelaybetweenthecurrentmachineandthe webserverattheDepartmentofComputingatImperialCollege.ping

isalsousefultocheckwhetheramachineisstill"alive"insomesense.

traceroute machinename traceroute showsthefullpathtakentoreacharemotemachine, includingthedelaytoeachmachinealongtheroute.Thisisparticularly usefulintrackingdownthelocationofnetworkproblems.

5.4RemoteFileTransfer

ftp machinename(filetransferprotocol) ftpisaninsecurewayoftransferringfilesbetweencomputers.When youconnecttoamachineviaftp,youwillbeaskedforyourusername andpassword.Ifyouhaveanaccountonthemachine,youcanuseit,or youcancanoftenusetheuser"ftp"or"anonymous".Oncelogged inviaFTP,youcanlistfiles(dir),receivefiles(get and mget)and sendfiles(putandmput).(UnusuallyforUNIX)help willshow youalistofavailablecommands.Particularlyusefularebinary (transferfilespreservingall8bits)andprompt n(donotconfirm eachfileonmultiplefiletransfers).Typequittoleaveftpandreturn totheshellprompt.

scpsourcefilesdestination(securecopy) scpisasecurewayoftransferringfilesbetweencomputers.Itworks justliketheUNIXcpcommandexceptthattheargumentscanspecifya userandmachineaswellasfiles.Forexample: $ scp will@rose.doc.ic.ac.uk:~/hello.txt . will(subjecttocorrectauthentication)copythefilehello.txtfrom theuseraccountwill ontheremotemachine rose.doc.ic.ac.ukintothecurrentdirectory(.)onthelocal machine.

5.5OtherInternetrelatedutilities

netscape netscape isafullyfledgedgraphicalwebbrowser(likeInternet Explorer).

lynx lynxprovidesawaytobrowsethewebonatextonlyterminal.

wgetURL wget providesawaytoretrievefilesfromtheweb(usingtheHTTP protocol).wgetisnoninteractive,whichmeansitcanruninthe background,whiletheuserisnotloggedin(unlikemostweb browsers).ThecontentretrievedbywgetisstoredasrawHTMLtext (whichcanbeviewedlaterusingawebbrowser). Notethatnetscape,lynxandwgetarenotstandardUNIXsystem utilities,butarefrequentlyinstalledapplicationpackages.

5.6UserInformationandCommunication

finger,who fingerandwhoshowthelistofusersloggedintoamachine,the terminaltheyareusing,andthedatetheyloggedinon. $ who will $

pts/2

Dec

5 19:41

write,talk writeisusedbyusersonthesamemachinewhowanttotalktoeach

other.Youshouldspecifytheuserand(optionally)theterminaltheyare on: $ write will pts/2 hello will

Linesareonlytransmittedwhenyoupress .Toreturntotheshell prompt,pressctrld(theUNIXendoffilemarker). talkisamoresophisticatedinteractivechatclientthatcanbeused betweenremotemachines: $ talk will@rose.doc.ic.ac.uk

Unfortunatelybecauseofincreasinglytightsecurityrestrictions,itis increasinglyunlikelythattalkwillwork(thisisbecauseitrequiresa specialdaemoncalledtalkdtoberunningontheremotecomputer). Sometimesanapplicationcalledytalkwillsucceediftalkfails.

5.7PrinterControl

lpr -Pprintqueuefilename lpr addsadocumenttoaprintqueue,sothatthedocumentisprinted whentheprinterisavailable.Lookat/etc/printcaptofindoutwhat printersareavailable.

lpq -Pprintqueue lpq checksthestatusofthespecifiedprintqueue.Eachjobwillhavean associatedjobnumber.

lprm -Pprintqueuejobnumber lprmremovesthegivenjobfromthespecifiedprintqueue.

Notethatlpr,lpqandlprmareBSDstyleprintmanagementutilities.Ifyou

areusingastrictSYSVUNIX,youmayneedtousetheSYSVequivalentslp, lpstatandcancel.

5.8EmailUtilities

mail mailisthestandardUNIXutilityforsendingandreceivingemail. $ mail Mail version 8.1 6/6/93. "/var/spool/mail/will": 2 1 jack@sprat.com Mon "Beanstalks" 2 bill@whitehouse.gov Mon Monica" & Type ? for help. messages 2 new Dec 11 10:37 Dec 11 11:00 "Re:

Someofthemoreimportantcommands(type?forafulllist)aregiven belowinFig.5.1.Hereamessagelistiseitherasinglemessage specifiedbyanumber(e.g.1)orarange(e.g.1-2).Thespecial messagelist*matchesallmessages.

? q x t messagelist +/d messagelist u messagelist m address r messagelist

help quit,savingchangestomailbox quit,restoringmailboxtoitsoriginalstate displaysmessages shownext/previousmessage deletesmessages undeletemessages sendanewemail replytosenderandotherreceipients

R messagelist

replyonlytosender Fig.5.1:Commonmailcommands

Youcanalsousemail tosendemaildirectlyfromthecommandline. Forexample: $ mail -s "Hi" wjk@doc.ic.ac.uk < message.txt $ emailsthecontentsofthe(ASCII)file message.txt tothe recipient wjk@doc.ic.ac.uk withthesubject "Hi".

mutt, elm, pine mutt, elm and pinearemorefriendly(butnonstandard)email interfacesthatyouwillprobablyprefertouseinsteadofmail.Allhave goodinbuilthelpfacilities.

sendmail, exim EmailisactuallysentusinganEmailTransferAgent,whichusesa protocolcalledSMTP(SimpleMailTransferProtocol).Thetwomost popularEmailTransferAgentsaresendmail andexim.Youcan seehowtheseagentsworkbyusingtelnettoconnecttoport25of anymailserver,forexample: $ telnet mail.doc.ic.ac.uk 25 Trying 146.169.1.47... Connected to diver.doc.ic.ac.uk (146.169.1.47). Escape character is '^]'. 220 diver.doc.ic.ac.uk ESMTP Exim 3.16 #7 HELP 214-Commands supported: 214HELO EHLO MAIL RCPT DATA AUTH 214 NOOP QUIT RSET HELP MAIL FROM: alien@xfiles.com 250 <alien@xfiles.com> is syntactically correct

RCPT TO: wjk@doc.ic.ac.uk 250 <wjk@doc.ic.ac.uk> verified DATA 354 Enter message, ending with "." on a line Hi This is a message from an alien

. 250 OK id=145UqB-0002t6-00 QUIT 221 diver.doc.ic.ac.uk closing connection Connection closed by foreign host. $ Thissendsanemailtowjk@doc.ic.ac.uk,apparentlyfrom alien@xfiles.com.Emailadvertisers(akaspammers)oftenuse thistechniquetoattempttoconfuserecipientsastothetruesourceof messages.Fortunatelyeximandsendmailincludeextensiveheader informationwhentheyforwardemail,includingtheIPaddressofthe computerfromwherethemessagewassent.

5.9AdvancedTextFileProcessing

sed (streameditor) sedallowsyoutoperformbasictexttransformationsonaninput stream(i.e.afileorinputfromapipeline).Forexample,youcandelete linescontainingparticularstringoftext,oryoucansubstituteone patternforanotherwhereveritoccursinafile.Althoughsed isa miniprogramminglanguageallonitsownandcanexecuteentire scripts,itsfulllanguageisobscureandprobablybestforgotten(being basedontheoldandesotericUNIXlineeditored).sedisprobablyat itsmostusefulwhenuseddirectlyfromthecommandlinewithsimple parameters: $ sed "s/pattern1/pattern2/" inputfile > outputfile

(substitutespattern2forpattern1onceperline) $ sed "s/pattern1/pattern2/g" inputfile > outputfile (substitutespattern2forpattern1foreverypattern1perline) $ sed "/pattern1/d" inputfile > outputfile (deletesalllinescontainingpattern1)

$ sed "y/string1/string2/" inputfile > outputfile (substitutescharactersinstring2forthoseinstring1)

awk (Aho,WeinbergerandKernigan) awkisusefulformanipulatingfilesthatcontaincolumnsofdataona linebylinebasis.Likesed,youcaneitherpassawkstatements directlyonthecommandline,oryoucanwriteascriptfileandletawk readthecommandsfromthescript. Saywehaveafileofcricketscorescalledcricket.datcontaining columnsforplayernumber,name,runsandthewayinwhichtheywere dismissed: 1 2 3 4 5 atherton hussain stewart thorpe gough 0 20 47 33 6 bowled caught stumped lbw run-out

Toprintoutonlythefirstandthirdcolumnswecansay: $ awk '{ print $1 " " $3 }' cricket.dat atherton 0 hussain 20 stewart 47 thorpe 33 gough 6 $

Here$nstandsforthenthfieldorcolumnofeachlineinthedatafile. $0 canbeusedtodenotethewholeline. Wecandomuchmorewithawk.Forexample,wecanwriteascript cricket.awktocalculatetheteam'sbattingaverageandtocheckif MikeAthertongotanotherduck: $ cat > cricket.awk BEGIN { players = 0; runs = 0 } { players++; runs +=$3 } /atherton/ { if (runs==0) print "atherton duck!" } END { print "the batting average is " runs/players } (ctrld) $awk -f cricket.awk cricket.dat atherton duck! the batting average is 21.2 $ TheBEGINclauseisexecutedonceatthestartofthescript,themain clauseonceforeveryline,the/atherton/ clauseonlyiftheword atherton occursinthelineandtheENDclauseonceattheendof thescript. awkcandoalotmore.Seethemanualpagesfordetails(typeman awk).

5.10TargetDirectedCompilation

make makeisautilitywhichcandetermineautomaticallywhichpiecesofa largeprogramneedtoberecompiled,andissuethecommandsto recompilethem.Tousemake,youneedtocreateafilecalled Makefileormakefilethatdescribestherelationshipsamongfiles inyourprogram,andthestatesthecommandsforupdatingeachfile.

Hereisanexampleofasimplemakefile: scores.out: cricket.awk cricket.dat [TAB]awk -f cricket.awk cricket.dat > scores.out Here[TAB]indicatestheTABkey.Theinterpretationofthismakefile isasfollows: scores.outisthetargetofthecompilation scores.outdependsoncricket.awkandcricket.dat ifeithercricket.awkorcricket.dathavebeenmodified sincescores.out waslastmodifiedorifscores.outdoes notexist,updatescores.outbyexecutingthecommand: awk -f cricket.awk cricket.dat > scores.out makeisinvokedsimplybytyping

$ make awk -f cricket.awk cricket.dat > scores.out $ Sincescores.outdidnotexist,makeexecutedthecommandsto createit.Ifwenowinvokemakeagain,nothinghappens: $ make make: `scores.out' is up to date. $ Butifwemodifycricket.datandthenrunmakeagain, scores.outwillbeupdated: $ touch cricket.dat (touch simulatesfilemodification) $ make awk -f cricket.awk cricket.dat > scores.out $ makeismostlyusedwhencompilinglargeC,C++orJavaprograms, butcan(aswehaveseen)beusedtoautomaticallyandintelligently produceatargetfileofanykind.

5.11VersioncontrolwithCVS

cvs(ConcurrentVersioningSystem) cvsisasourcecodecontrolsystemoftenusedonlargeprogramming projectstocontroltheconcurrenteditingofsourcefilesbymultiple authors.Itkeepsoldversionsoffilesandmaintainsalogofwhen,and whychangesoccurred,andwhomadethem. cvskeepsasinglecopyofthemastersources.Thiscopyiscalledthe source``repository'';itcontainsalltheinformationtopermitextracting previoussoftwarereleasesatanytimebasedoneitherasymbolic revisiontag,oradateinthepast. cvshasalargenumberofcommands(typeinfo cvsforafullcvs tutorial,includinghowtosetuparepositoryfromscratchorfrom existingcode).Themostusefulcommandsare:

cvs checkout modules Thisgivesyouaprivatecopyofsourcecodethatyoucanwork onwithwithoutinterferingwithothers. cvs update Thisupdatesthecodeyouhavecheckedout,toreflectany changesthathavesubsequentlybeenmadebyotherdevelopers. cvs addfiles Youcanusethistoaddnewfilesintoarepositorythatyouhave checkedout.Doesnotactuallyaffecttherepositoryuntila"cvs commit"isperformed. cvs remove files Removesfilesfromacheckedoutrepository.Doesn'taffectthe repositoryuntila"cvs commit"isperformed.

cvs commitfiles Thiscommandpublishesyourchangestootherdevelopersby updatingthesourcecodeinthecentralrepository.

5.12C/C++compilationutilities

cc, gcc, CC, g++ UNIXinstallationsusuallycomewithaCand/orC++compiler.TheC compilerisusuallycalledccorgcc,andtheC++compilerisusually calledCCorg++.MostlargeCorC++programswillcomewitha makefileandwillsupporttheconfigureutility,sothatcompiling andinstallingapackageisoftenassimpleas: $ ./configure $ make $ make install However,thereisnothingtopreventyoufromwritingandcompilinga simpleCprogramyourself: $ cat > hello.c #include <stdio.h> int main() { printf("hello world!\n"); return 0; } (ctrld) $ cc hello.c -o hello $ ./hello hello world! $

HeretheCcompiler(cc)takesasinputtheCsourcefilehello.cand producesasoutputanexecutableprogramcalledhello.Theprogram hellomaythenbeexecuted(the./ tellstheshelltolookinthe currentdirectorytofindthehelloprogram).

5.13ManualPages

man MoreinformationisavailableonmostUNIXcommandsisavailablevia theonlinemanualpages,whichareaccessiblethroughtheman command.Theonlinedocumentationisinfactdividedintosections. Traditionally,theyare 1 User-level commands 2 System calls 3 Library functions 4 Devices and device drivers 5 File formats 6 Games 7 Various miscellaneous stuff - macro packages etc. 8 System maintenance and operation commands Sometimesmangivesyouamanualpagefromthewrongsection.For example,sayyouwerewritingaprogramandyouneededtousethe rmdirsystemcall.man rmdirgivesyouthemanualpageforthe userlevelcommandrmdir.ToforcemantolookinSection2ofthe manualinstead,typeman 2 rmdir(orman -s2 rmdironsome systems). mancanalsofindmanualpageswhichmentionaparticulartopic.For example,man -k postscript shouldproducealistofutilities thatcanproduceandmanipulatepostscriptfiles.

info info isaninteractive,somewhatmorefriendlyandhelpfulalternative toman.Itmaynotbeinstalledonallsystems,however.

IntroductiontoUNIX: ExerciseSheetFive

1. Usetelnettorequestawebpagefromthewebserver

www.doc.ic.ac.ukbyconnectingtoport80,asshowninthenotes. 2. Usepingtofindtheroundtripdelaytowww.altavista.com. 3. Usetraceroutetoseethenetworkroutetakento www.altavista.com(whichisintheUSA).Canyoutellwhich citiesyournetworktrafficpassesthrough? 4. UseftptoconnecttotheFTPsitesunsite.doc.ic.ac.uk. Obtainthelatestversionofthepackageunits(intheformofa .tar.gzfile)fromthedirectorypackages/gnu/units. Decompressandunarchivethe.tar.gzfile.Typeconfigureand thenmake.Runtheexecutableprogramthatisproducedas"./units -f units.dat".Whatdoestheprogramdo?Ifyouwerethesystem administrator,whatwouldyouhavetodotoinstallthepackagefor everyonetouse? 5. Usewgettogetacopyofthewebpage http://www.doc.ic.ac.uk/index.html.Havealookatthe contentsofthefile.CanyouusesedtostripouttheHTMLtags(text enclosedin<and>)toleaveyouwithjustplaintext? 6. Usefingerorwhotogetalistofusersonthemachine. 7. Usewritetosendthemamessage.Tostoppeoplefromsendingyou messages,type"mesg n".Toreenablemessages,type"mesg y". 8. Tryusetalktosendamessagetosomeone(N.B.thismaynotwork). 9. Listallyourprocesses,usingsedtosubstitute"me"foryourusername. 10.Usewho,awk,sortanduniqtoprintoutasortedlistofthelogins ofactiveusers. 11.Useawkon/etc/passwdtoproducealistofusersandtheirlogin

shells. 12.Writeanawkscriptthatprintsoutalllinesinafileexceptforthefirst two. 13.Modifytheawkscriptinthenotessothatitdoesn'tincreasethenumber ofplayersusedtocalculatetheaverageifthemannerofdismissalis "notout". 14.Createafilecalledhello.ccontainingthesimple"helloworld" programinthenotes.Createanappropriatemakefileforcompiling it.Runmake. 15.Useman -ktofindasuitableutilityforviewingpostscriptfiles.

IntroductiontoUNIX: LectureSix
6.1Objectives

ThislectureintroducesthetwomostpopularUNIXeditors:vi andemacs. Foreacheditor,itcovers:

Basictextinputandnavigation. Movingandcopyingtext. Searchingforandreplacingtext. Otherusefulcommands. Aquickreferencechart.

6.2.1Introductiontovi
vi(pronounced"veeeye",shortforvisual,orperhapsvile)isadisplay orientedtexteditorbasedonanunderlyinglineeditorcalledex.Although beginnersusuallyfindvi somewhatawkwardtouse,itisusefultolearn becauseitisuniversallyavailable(beingsuppliedwithallUNIXsystems).It alsousesstandardalphanumerickeysforcommands,soitcanbeusedon almostanyterminalorworkstationwithouthavingtoworryaboutunusual keyboardmappings.Systemadministratorslikeuserstousevibecauseit usesveryfewsystemresources. Tostartvi,enter: $ vi filename

wherefilenameisthenameofthefileyouwanttoedit.Ifthefiledoesn'texist, viwillcreateitforyou.

6.2.2BasicTextInputandNavigationinvi
Themainfeaturethatmakesviuniqueasaneditorisitsmodebased operation.vihastwomodes:commandmodeandinputmode.Incommand mode,charactersyoutypeperformactions(e.g.movingthecursor,cuttingor copyingtext,etc.)Ininputmode,charactersyoutypeareinsertedoroverwrite existingtext. Whenyoubeginvi,itisincommandmode.Toputviintoinputmode,press i (insert).Youcanthentypetextwhichisinsertedatthecurrentcursor location;youcancorrectmistakeswiththebackspacekeyasyoutype.Toget backintocommandmode,pressESC(theescapekey).Anotherwayof insertingtext,especiallyusefulwhenyouareattheendofalineistopressa (append). Incommandmode,youareabletomovethecursoraroundyourdocument.h, j,kandl movethecursorleft,down,upandrightrespectively(ifyouare luckythearrowkeysmayalsowork).Otherusefulkeysare^and$which moveyoutothebeginningandendofalinerespectively.wskipstothe beginningofthenextwordandbskipsbacktothebeginningoftheprevious word.Togorighttothetopofthedocument,press1andthenG.Togothe bottomofthedocument,pressG.Toskipforwardapage,press^F,andtogo backapage,press^B.Togotoaparticularlinenumber,typethelinenumber andpressG,e.g.55Gtakesyoutoline55. Todeletetext,movethecursoroverthefirstcharacterofthegroupyouwant todeleteandmakesureyouareincommandmode.Pressxtodeletethe currentcharacter,dwtodeletethenextword,d4wtodeletethenext4words, ddtodeletethenextline,4dd todeletethenext4lines,d$todeletetothe endofthelineorevendGtodeletetotheendofthedocument.Ifyou accidentallydeletetoomuch,pressinguwillundothelastchange. Occasionallyyouwillwanttojointwolinestogether.PressJtodothis (tryingtopressbackspaceonthebeginningofthesecondlinedoesnothave theintuitiveeffect!)

6.2.3MovingandCopyingTextinvi
viusesbufferstostoretextthatisdeleted.Thereareninenumberedbuffers (19)aswellastheundobuffer.Usuallybuffer1containsthemostrecent deletion,buffer2thenextrecent,etc. Tocutandpasteinvi,deletethetext(usinge.g.5ddtodelete5lines).Then movetothelinewhereyouwantthetexttoappearandpressp.Ifyoudelete somethingelsebeforeyoupaste,youcanstillretrievethedeletetextby pastingthecontentsofthedeletebuffers.Youcandothisbytyping"1p, "2p,etc. Tocopyandpaste,"yank"thetext(usinge.g.5yytocopy5lines).Then movetothelinewhereyouwantthetexttoappearandpressp.

6.2.4SearchingforandReplacingTextinvi
Incommandmode,youcansearchfortextbyspecifyingregularexpressions. Tosearchforward,type/andthenaregularexpressionandpress .To searchbackwards,beginwitha?insteadofa/.Tofindthenexttextthat matchesyourregularexpressionpressn. Tosearchandreplacealloccurencesofpattern1withpattern2,type :%s/pattern1/pattern2/g .Tobeaskedtoconfirmeachreplacement,add actothissubstitutioncommand.Insteadofthe%youcanalsogivearangeof lines(e.g.1,17)overwhichyouwantthesubstitutiontoapply.

6.2.5OtherUsefulviCommands
Programmersmightlikethe:set number commandwhichdisplays linenumbers(:set nonumberturnsthemoff). Tosaveafile,type:w .Tosaveandquit,type:wq forceaquitwithoutsavingtype:q! . Tostarteditinganotherfile,type:efilename . orpressZZ.To

Toexecuteshellcommandsfromwithinvi,andthenreturntoviafterwards, type:!shellcommand .Youcanusetheletter% asasubstituteforthe nameofthefilethatyouareediting(so:!echo %printsthenameofthe currentfile). .repeatsthelastcommand.

6.2.6Quickreferenceforvi
Insertingandtypingtext: i insert text (and enter input mode) $a append text (to end of line) ESC re-enter command mode J join lines Cursormovement: h left j down k up l right ^ beginning of line $ end of line 1 G top of document G end of document <n> G go to line <n> ^F page forward ^B page backward w word forwards b word backwards Deletingandmovingtext: Backspace delete character before cursor (only works in insert mode) x delete character under cursor dw delete word dd delete line (restore with p or P) <n> dd delete n lines d$ delete to end of line

dG yy <n> yy

delete to end of file yank/copy line (restore with p or P) yank/copy <n> lines

Searchandreplace: %s/<search string>/<replace string>/g Miscellaneous: u undo :w save :wq save ZZ save :q! quit

file file and quit file and quit without saving

6.3.1Introductiontoemacs
emacs isapopulareditorforUNIX,WindowsandMacintoshsystems. Unlikevi,itisnotastandardUNIXsystemutility,butisavailablefromthe FreeSoftwareFoundation. An emacszealotwilltellyouhowemacsprovidesadvancedfacilitiesthat gobeyondsimpleinsertionanddeletionoftext:youcanviewtwoaremore filesatthesametime,compileanddebugprogramsinalmostany programminglanguage,typesetdocuments,runshellcommands,readmanual pages,emailandnewsandevenbrowsethewebfrominsideemacs.emacs isalsoveryflexibleyoucanredefinekeystrokesandcommandseasily,and (forthemoreambitious)youcanevenwriteLispprogramstoaddnew commandsanddisplaymodestoemacs,sinceemacshasitsownLisp interpreter.InfactmostoftheeditingcommandsofEmacsarewritteninLisp already;thefewexceptionsarewritteninCforefficiency.However,usersdo notneedtoknowhowtoprogramLisptouseemacs(whileitistruethat onlyaprogrammercanwriteasubstantialextensiontoemacs,itiseasyfor anyonetouseitafterwards). Criticsofemacspointoutthatitusesarelativelylargeamountofsystem resourcescomparedtoviandthatithasquiteacomplicatedcommand structure(jokingthatemacsstandsforEscapeMetaAltControlShift).

Inpracticemostuserstendtousebotheditorsvitoquicklyeditshortscripts andprogramsandemacsformorecomplexjobsthatrequirereferenceto morethanonefilesimultaneously. OnUNIXsystems,emacscanruningraphicalmodeundertheXWindows system,inwhichcasesomehelpfulmenusandmousebuttoncommand mappingsareprovided.However,mostofitsfacilitiesarealsoavailableona textterminal. Tostartemacs,enter: $ emacs filename

wherefilenameisthenameofthefileyouwanttoedit.Ifthefiledoesn'texist, emacswillcreateitforyou.

6.3.2BasicTextInputandNavigationinemacs
Textinputandnavigationinemacsismostlyamatterofusingthearrow keystopositionthecursorandtypingsometext.Issuingmorecomplex emacscommandsusuallyinvolvespressingtheCtrlkey(sometimesalso labelledControlorCtl)ortheMetakey(sometimesalsolabelledAlt).emacs commandsareusuallydescribedinthisway:

C-<chr> meansholdtheCtrlkeywhiletypingthecharacter<chr>. Thus,C-fwouldbe:holdtheCtrlkeyandtypef. M-<chr>meansholdtheMetaorAltkeydownwhiletyping<chr>. IfthereisnoMetaorAltkey,insteadpressandreleasetheESCkeyand thentype<chr>.

Oneinitiallyannoyingfeatureofemacsisthatitshelpfacilityhasbeen installedonC-h(Ctrlh).Unfortunatelythisisalsothecodeforthebackspace keyonmostsystems.Youcan,however,easilymakethekeyworkas expectedbycreatinga.emacsfile(afilealwaysreadonstartup)inyour homedirectorycontainingthefollowingline: (global-set-key "\C-h" 'backward-delete-char-

untabify) Hereisa.emacsfilethatcontainsthislineaswellasseveralotheruseful facilities(seeSection6.3.6). ToaccessthehelpsystemyoucanstilltypeM-x help or(fora comprehensivetutorial)M-x help-with-tutorial . UsefulnavigationcommandsareC-a(beginningofline),C-e(endofline), C-v(forwardpage),M-v(backwardspage),M-<(beginningofdocument) andM->(endofdocument).C-d willdeletethecharacterunderthecursor, whileC-kwilldeletetotheendoftheline.TextdeletedwithC-kisplaced inabufferwhichcanbe"yanked"backlaterwithC-y.

6.3.3MovingandCopyingTextinemacs
Theeasiestwaytocutandpastetextistogotothestartofthetextandpress C-kuntilyouhavedeletedthetextyouwant.Thenmovetothespotwhere youwanttopastethetextandpressC-ytorestorethetext.Ifyoumakea mistake,C-uwillundothechange(emacssupportsseverallevelsofundo). AnotherwaytodeleteachunkoftextistogothestartofthetextandpressCSPC(SPCisthespacebar;thissetsamark).Thengototheendofthetextyou wishtodeleteandpressC-w.RestorethetextintherightspotwithC-y. Tocopyandpaste,deletethetargettextasabove,andthenuseC-ytwice (oncetorestoretheoriginaltext,andoncetocreatethecopy).

6.3.4SearchingforandReplacingTextinemacs
Tosearchforwardsandbackwardsincrementally,useC-sandC-r respectively.PressingC-sorC-ragainwillrepeattheoperation.Whenyou havefoundthetextyouwant,press ,orpressC-gtocanceltheoperation andreturnyourcursortothepositionwherethesearchstarted. Toreplaceastring,typeM-x replace-string (youmaywantto modifyyour.emacsfilesothatthiscommandisonC-x r).M-%performs aquerysearchandreplace.

6.3.5OtherUsefulemacsCommands
PressingC-gwillcancelanyemacscommandinprogress. Tosaveafile,pressC-x C-s.Tostarteditinganewfile,pressC-x C-f. Tobringuptwowindows(or"buffers"inemacsspeak),pressC-x 2(C-x 1getsyoubackto1).C-x oswitchesbetweenbuffersonthesamescreen. C-x bletsyouswitchbetweenallactivebuffers,whetheryoucanseethem ornot.C-x C-kdeletesabufferthatyouarefinishedwith. M-x shell bringsupaUNIXshellinsideabuffer(youmayliketoput thisonC-x C-u).M-x goto-line skipstoaparticularline(youmay liketoputthisonC-x g).M-x compile willattempttocompilea program(usingthemakeutilityorsomeothercommandthatyoucan specify).Ifyouaredoingalotofprogrammingyouwillprobablywanttoput thisonakeylikeC-x c. M-qwillreformataparagraph. C-x C-c quitsemacs.

6.3.6Quickreferenceforemacs
Cursormovement: C-a beginning of line C-e end of line C-< top of document C-> end of document M-x goto-line Go to line C-v page forward M-v page backward Deletingandmovingtext: Backspace delete character before cursor (subject to modification of .emacs) C-d delete character under cursor

M-d C-k C-SPC C-w

delete word delete to end of line(s) (restore with C-y) set mark delete from mark to cursor (restore with C-y)

Searchandreplace: C-s incremental search forward C-r incremental search backward C-% query replace M-x replace-string <searchstr> <replstr> Miscellaneous: C-x u C-x C-s C-x C-f C-x 2 C-x 1 C-x o C-x b M-q C-x C-c

undo save file find file 2 windows 1 window switch between windows switch buffers reformat paragraph quit

Usefulcustomisablekeys(configuredusingtheprovided .emacsfile): C-x g goto line C-x c compile program (useful with Cx 2) C-x C-u open Unix shell (useful with Cx 2) C-x a repeat command C-x m manual entry

6.4OtherUNIXeditors
TherearemanyothereditorsforUNIXsystems.Twopopularalternativesto viandemacsarenedit andpico.

IntroductiontoUNIX: ExerciseSheetSix

1. Copythefilemole.txtintoyourhomedirectory(pressshiftandthe

leftmousebuttontodownloadthefileusingNetscape). 2. Edityourcopyofthedocumentusingvi. 3. Gototheendofthedocumentandtypeinthefollowingparagraph:

Joined the library. Got Care of the Skin, Origin of the Species, and a book by a woman my mother is always going on about. It is called Pride and Prejudice, by a woman called Jane Austen. I could tell the librarian was impressed. Perhaps she is an intellectual like me. She didn't look at my spot, so perhaps it is getting smaller. 4. Correctthethreespellingerrorsinthefirstthreelinesofthefirst paragraph(oneerrorperline)andremovetheextra"Geography"in the3rdlineofthefirstparagraph. 5. Addthewords"About time!"totheendofthesecondparagraph. 6. Deletethesentence"Time flies like an arrow but fruit flies like a banana"andreformtheparagraph. 7. Replacealloccurrencesof"is"with"was". 8. Swapthetwoparagraphs. 9. Savethefileandquit. 10.Repeattheexercisewithemacs.Whichdidyoufindeasier? 11.CanyouwriteasimpleCprogram(saythehelloworldprogram)and makefile,compileandrunitallfrominsideemacs? 12.Ifyou'dlikeanindepthvitutorialtryrunning"vimtutor".Foran indepthemacstutorial,typeM-x help-with-tutorialfrom insideemacs.

IntroductiontoUNIX: LectureSeven
7.1Objectives

Thislecturecoversbasicsystemadministrationconceptsandtasks,namely:

Thesuperuserroot. Shutdownandsystemstartup. Addingusers. Controllingusergroups. ReconfiguringandrecompilingtheLinuxkernel. Cronjobs. Keepingessentialprocessesalive.

Notethatyouwillnotbegivenadministratoraccessonthelabmachines. However,youmightliketotrysomebasicadministrationtasksonyourhome PC.

7.2TheSuperuserroot
Thesuperuserisaprivilegeduserwhohasunrestrictedaccesstoall commandsandfilesonasystemregardlessoftheirpermissions.The superuser'sloginisusuallyroot.Accesstotherootaccountisrestrictedbya password(therootpassword).Becausetherootaccounthashugepotentialfor destruction,therootpasswordshouldbechosencarefully,onlygiventothose whoneedit,andchangedregularly. Onewaytobecomeroot istologinasusualusingtheusernamerootand therootpassword(usuallysecuritymeasuresareinplacesothatthisisonly

possibleifyouareusinga"secure"consoleandnotconnectingovera network).Usingrootasyourdefaultlogininthiswayisnotrecommended, however,becausenormalsafeguardsthatapplytootheruseraccountsdonot applytoroot.Consequentlyusingrootformundanetasksoftenresultsina memorylapseormisplacedkeystrokeshavingcatastrophiceffects(e.g. forgettingforamomentwhichdirectoryyouareinandaccidentallydeleting anotheruser'sfiles,oraccidentallytyping"rm -rf * .txt"insteadof "rm -rf *.txt"). Abetterwaytobecomerootistousethesuutility.su(switchuser)letsyou becomeanotheruser(atleastasfarasthecomputerisconcerned).Ifyoudon't specifythenameoftheuseryouwishtobecome,thesystemwillassumeyou wanttobecomeroot.Usingsudoesnotusuallychangeyourcurrent directory,unlessyouspecifya"-"optionwhichwillrunthetargetuser's startupscriptsandchangeintotheirhomedirectory(providedyoucansupply therightpasswordofcourse).So: $ su - Password: xxxxxxxx #

Notethattherootaccountoftendisplaysadifferentprompt(usuallya#).To returntoyouroldself,simplytype"exit"attheshellprompt. Youshouldavoidleavingarootwindowopenwhileyouarenotatyour machine.Considerthisparagraphfromahumorous1986ComputerLanguage articlebyAlanFilipski: "Theprudentadministratorshouldbeawareofcommontechniquesusedto breachUNIXsecurity.Themostwidelyknownandpractisedattackonthe securityoftheUNIXbrandoperatingsystemiselegantinitssimplicity.The perpetratorsimplyhangsaroundthesystemconsoleuntiltheoperatorleaves togetadrinkorgotothebathroom.Theintruderlungesfortheconsoleand typesrm -rf /beforeanyonecanpryhisorherhandsoffthekeyboard. Amateureffortsarecharacterisedbytypinginthingssuchaslsorpwd.A skilledUNIXbrandoperatingsystemsecurityexpertwouldlaughatsuch

attempts."

7.3ShutdownandSystemStartup

Shutdown: shutdown, halt,reboot (in /sbin) /sbin/shutdown allowsaUNIXsystemtoshutdowngracefully andsecurely.Allloggedinusersarenotifiedthatthesystemis goingdown,andnewloginsareblocked.Itispossibletoshutthe systemdownimmediatelyorafteraspecifieddelayandtospecifywhat shouldhappenafterthesystemhasbeenshutdown: # /sbin/shutdown -r now (shutdownnowandreboot) # /sbin/shutdown -h +5 (shutdownin5minutes&halt) # /sbin/shutdown -k 17:00 (fakeashutdownat5pm) halt and reboot areequivalentto shutdown -h and shutdown -r respectively. Ifyouhavetoshutasystemdownextremelyurgentlyorforsome reasoncannotuseshutdown,itisatleastagoodideatofirstrunthe command: # sync

whichforcesthestateofthefilesystemtobebroughtuptodate. Systemstartup: Atsystemstartup,theoperatingsystemperformsvariouslowlevel tasks,suchasinitialisingthememorysystem,loadingupdevicedrivers tocommunicatewithhardwaredevices,mountingfilesystemsand creatingtheinitprocess(theparentofallprocesses).init's primaryresponsibilityistostartupthesystemservicesasspecifiedin /etc/inittab.Typicallytheseservicesincludegettys(i.e.virtual terminalswhereuserscanlogin),andthescriptsinthedirectory

/etc/rc.d/init.dwhichusuallyspawnhighleveldaemonssuch ashttpd(thewebserver).OnmostUNIXsystemsyoucantype dmesgtoseesystemstartupmessages,orlookin /var/log/messages. Ifamountedfilesystemisnot"clean"(e.g.themachinewasturnedoff withoutshuttingdownproperly),asystemutilityfsck is automaticallyruntorepairit.Automaticrunningcanonlyfixcertain errors,however,andyoumayhavetorunitmanually: # fsck filesys

wherefilesysisthenameofadevice(e.g. /dev/hda1)oramount point(like/)."Lost"filesrecoveredduringthisprocessendupinthe lost+founddirectory.Somemoremodernfilesystemscalled "journaling"filesystemsdon'trequirefsck,sincetheykeepextensive logsoffilesystemeventsandareabletorecoverinasimilarwaytoa transactionaldatabase.

7.4AddingUsers

useradd(in/usr/sbin): useraddisautilityforaddingnewuserstoaUNIXsystem.Itadds newuserinformationtothe/etc/passwdfileandcreatesanew homedirectoryfortheuser.Whenyouaddanewuser,youshouldalso settheirpassword(usingthe-poptiononuseradd,orusingthe passwdutility): # useradd bob # passwd bob

7.5ControllingUserGroups

groupadd(in/usr/sbin): groupaddcreatesanewusergroupandaddsthenew

informationto/etc/group: # groupadd groupname

usermod(in/usr/sbin):

Everyuserbelongstoaprimarygroupandpossiblyalsotoa setofsupplementarygroups.Tomodifythegroup permissionsofanexistinguser,use # usermod -g initialgroupusername -G othergroups whereothergroupsisalistofsupplementarygroupnames separatedbycommas(withnointerveningwhitespace). groups Youcanfindoutwhichgroupsauserbelongstobytyping: # groups username

7.6ReconfiguringandRecompilingtheLinuxKernel
Linuxhasamodular,customisablekernelwithseveralswitchableoptions (e.g.supportformultipleprocessorsanddevicedriversforvarioushardware devices).ItmayhappenthatsomenewhardwareisaddedtoaLinuxmachine whichrequiresyoutorecompilethekernelsothatitincludesdevicedriver support(andpossiblynewsystemcalls)forthenewhardware.Todothis,you willneedtorebuildtheLinuxkernelfromscratchasfollows:

Lookin/usr/src/linuxforthekernelsourcecode.Ifitisn'tthere (orifthereisjustamessagesayingthatonlykernelbinarieshavebeen installed),getholdofacopyofthelatestkernelsourcecodefrom http://www.kernel.organduntaritinto/usr/src/linux. Changedirectoryto/usr/src/linux.

Toconfigurethekerneltypeeither # make config (simpletextmodeconfiguration),or # make menuconfig (menudriventextconfiguration),or # make xconfig (graphicalconfigurationforX) Youwillbeaskedtoselectwhichmodules(devicedrivers, multiprocessorsupportetc.)youwishtoinclude.Foreachmodule,you canchosetoincludeitinthekernelcode(y),incorporateitasan optionalmodulethatwillbeloadedifneeded(m)ortoexcludeitfrom thekernelcode(n).Tofindoutwhichoptionalmoduleshaveactually beenloadedyoucanrunlsmodwhenthesystemreboots. Nowtype: # make dep (tobuildsourcecodedependencies) # make clean (todeleteallstaleobjectfiles) # make bzImage (tobuildthenewkernel) # make modules (tobuildthenewoptionalmodules) # make modules_install (toinstallthemodules) Thefile/usr/src/linux/arch/i386/boot/bzImagenow containsyournewkernelimage.Itremainsonlytoinstallit. Changedirectoryto/usr/src/linux/arch/i386/boot.Inthe samedirectoryshouldbeascriptcalledinstall.shwhichwillcopy yourkernelimageinto/boot/vmlinuz: # install.sh version bzImage /boot/System.map /boot whereversionisthekernelversionnumber(ofform2.2.xx). Finally,youmayneedtoupdatethe/etc/lilo.conffilesothat lilo(theLinuxbootloader)includesanentryforyournewkernel. Thenrun

# lilo

toupdatethechanges.Whenyourebootyourmachine,youshouldbe abletoselectyournewkernelimagefromthelilobootloader.

7.7CronJobs
crondisadaemonthatexecutescommandsthatneedtoberunregularly accordingtosomeschedule.Thescheduleandcorrespondingcommandsare storedinthefile/etc/crontab. Eachentryinthe/etc/crontabfileentrycontainssixfieldsseparatedby spacesortabsinthefollowingform:
minutehourday_of_monthmonthweekdaycommand

Thesefieldsacceptthefollowingvalues: minute hour day_of_month month weekday command 0 through 59 0 through 23 1 through 31 1 through 12 0 (Sun) through 6 (Sat) a shell command

Youmustspecifyavalueforeachfield.Exceptforthecommandfield,these fieldscancontainthefollowing:

Anumberinthespecifiedrange,e.g.torunacommandinMay,specify 5inthemonthfield. Twonumbersseparatedbyadashtoindicateaninclusiverange,e.g.to runacronjobonTuesdaythroughFriday,place25intheweekday field. Alistofnumbersseparatedbycommas,e.g.torunacommandonthe firstandlastdayofJanuary,youwouldspecify1,31inthe day_of_monthfield. *(asterisk),meaningallallowedvalues,e.g.torunajobeveryhour,

specifyanasteriskinthehourfield. Youcanalsospecifysomeexecutionenvironmentoptionsatthetopofthe /etc/crontabfile: SHELL=/bin/bash PATH=/sbin:/bin:/usr/sbin:/usr/bin MAILTO=root Torunthecalendarcommandat6:30am.everyMon,Wed,andFri,asuitable /etc/crontabentrywouldbe: 30 6 * * 1,3,5 /usr/bin/calendar Theoutputofthecommandwillbemailedtotheuserspecifiedinthe MAILTOenvironmentoption. Youdon'tneedtorestartthecrondaemoncrondafterchanging /etc/crontabitautomaticallydetectschanges.

7.8KeepingEssentialProcessesAlive
Itisimportantthatdaemonsrelatedtomissioncriticalservicesare immediatelyrespawnediftheyfailforsomereason.Youcandothisby addingyourownentriestothe/etc/inittabfile.Forexample: rs:2345:respawn:/home/sms/server/RingToneServer Herersisa2charactercodeidentifyingtheservice,and2345arethe runlevels(tofindaboutrunlevels,typeman runlevel)forwhichthe processshouldbecreated.The init processwillcreatethe RingToneServerprocessatsystemstartup,andrespawnitshoulditdie foranyreason.

IntroductiontoUNIX: ExerciseSheetSeven

N.B.PleaseperformthesetasksonyourhomePCandtakeextracarewhen usingtherootaccount.Sinceyourcommandswillbecarriedoutwithoutthe safeguardsthatapplytoordinaryusers,youmaydoseriousdamagetothe system.Ifindoubt,pleaseask(preferablybeforepressing !)

1. Usesu -tobecomeroot. 2. Addanewuserofyourchoosingtothesystem.Settheirpassword. 3.

4. 5. 6. 7.

Checkthattheycanlogin. Addanewusergroupofyourchoosingtothesystem.Placeyourself (i.e.yourlogin,notroot),andtheuserthatyouhaveaddedinthe group.Checkthatyouarebothinthenewgroup. Removetheuserthatyouadded. Makeyourselfacronjobthatsendsyouamessage(usingwrite) every10minutes.Removethiswhenyoustarttofinditirritating. Discoverhowtorestartthewebserverhttpd(oranyothersystem daemon). tinyhttpd.plisaminiwebserverwritteninPerl.Switchbackto beinganormaluser,andmakeacopyoftinyhttpd.pl inyour homedirectory.Changetheportnumberonwhichitoperatestoaport ofyourchoice(currently8080,butuseadifferentonethatnooneelse willbeusing).Createapublic_htmldirectoryinyourhome directoryandcreateahomepagefileinsideitcalled"index.html". Nowruntinyhttpd.plinthebackgroundandusetelnet (telnet machine port,wheremachineisthehostnameandport istheportnumber,andthenissuetheHTTPcommandGET / HTTP/1.0 )orawebbrowser(callup

http://machine:port)totestthatyourhomepagefileisinfact beingserved. 8. Switchbacktobeingroot.Addalineto/etc/inittabwhichwill ensurethatyourminiwebserverwillrespawnitselfifitdies.Killyour webserver(usingkill)andseeiftherespawningworks(initre examinesthe/etc/inittabfilewheneveroneofitsdescendant processesdiessoyoushouldnothavelongtowaitforthistotake effect).Itmaybehelpfultomonitorthesystemmessagesfilewith tail -f /var/log/messages.

IntroductiontoUNIX: LectureEight
8.1Objectives
Thischaptercovers:

Shellsandshellscripts. Shellsvariablesandtheenvironment. Simpleshellscripting Advancedshellscripting. Startupshellscripts.

8.2ShellsandShellScripts
Ashellisaprogramwhichreadsandexecutescommandsfortheuser.Shells alsousuallyprovidefeaturessuchjobcontrol,inputandoutputredirectionand acommandlanguageforwritingshellscripts.Ashellscriptissimplyan ordinarytextfilecontainingaseriesofcommandsinashellcommand language(justlikea"batchfile"underMSDOS). TherearemanydifferentshellsavailableonUNIXsystems(e.g.sh,bash, csh,ksh,tcshetc.),andtheyeachsupportadifferentcommandlanguage. HerewewilldiscussthecommandlanguagefortheBourneshellshsinceitis availableonalmostallUNIXsystems(andisalsosupportedunderbashand ksh).

8.3ShellVariablesandtheEnvironment
Ashellletsyoudefinevariables(likemostprogramminglanguages).A variableisapieceofdatathatisgivenaname.Onceyouhaveassigneda

valuetoavariable,youaccessitsvaluebyprependinga$tothename: $ bob='hello world' $ echo $bob hello world $

Variablescreatedwithinashellarelocaltothatshell,soonlythatshellcan accessthem.Thesetcommandwillshowyoualistofallvariablescurrently definedinashell.Ifyouwishavariabletobeaccessibletocommandsoutside theshell,youcanexportitintotheenvironment: $ export bob

(undercshyouusedsetenv).Theenvironmentisthesetofvariablesthat aremadeavailabletocommands(includingshells)whentheyareexecuted. UNIXcommandsandprogramscanreadthevaluesofenvironmentvariables, andadjusttheirbehaviouraccordingly.Forexample,theenvironmentvariable PAGERisusedbythemancommand(andothers)toseewhatcommand shouldbeusedtodisplaymultiplepages.Ifyousay: $ export PAGER=cat

andthentrythemancommand(sayman pwd),thepagewillgoflyingpast withoutstopping.Ifyounowsay: $ export PAGER=more

normalserviceshouldberesumed(sincenowmorewillbeusedtodisplay thepagesoneatatime).Anotherenvironmentvariablethatiscommonlyused istheEDITORvariablewhichspecifiesthedefaulteditortouse(soyoucan setthistovioremacsorwhicheverothereditoryouprefer).Tofindout whichenvironmentvariablesareusedbyaparticularcommand,consultthe manpagesforthatcommand. AnotherinterestingenvironmentvariableisPS1,themainshellpromptstring whichyoucanusetocreateyourowncustomprompt.Forexample:

$ export PS1="(\h) \w> " (lumberjack) ~>

Theshelloftenincorporatesefficientmechanismsforspecifyingcommon partsoftheshellprompt(e.g.inbashyoucanuse\hforthecurrenthost,\w forthecurrentworkingdirectory,\dforthedate,\tforthetime,\uforthe currentuserandsoonseethebashmanpage). AnotherimportantenvironmentvariableisPATH.PATHisalistofdirectories thattheshellusestolocateexecutablefilesforcommands.SoifthePATHis setto: /bin:/usr/bin:/usr/local/bin:. andyoutypedls,theshellwouldlookfor/bin/ls,/usr/bin/lsetc. NotethatthePATHcontains'.',i.e.thecurrentworkingdirectory.Thisallows youtocreateashellscriptorprogramandrunitasacommandfromyour currentdirectorywithouthavingtoexplicitlysay"./filename". NotethatPATHhasnothingtowithfilenamesthatarespecifiedasarguments tocommands(e.g.cat myfile.txt wouldonlylookfor ./myfile.txt,notfor/bin/myfile.txt, /usr/bin/myfile.txt etc.)

8.4SimpleShellScripting
Considerthefollowingsimpleshellscript,whichhasbeencreated(usingan editor)inatextfilecalledsimple: #!/bin/sh # this is a comment echo "The number of arguments is $#" echo "The arguments are $*" echo "The first is $1" echo "My process number is $$" echo "Enter a number from the keyboard: " read number echo "The number you entered was $number"

Theshellscriptbeginswiththeline"#!/bin/sh".Usually"#"denotesthe startofacomment,but#!isaspecialcombinationthattellsUNIXtousethe Bourneshell(sh)tointerpretthisscript.The#!mustbethefirsttwo charactersofthescript.Theargumentspassedtothescriptcanbeaccessed through$1,$2,$3etc.$*standsforallthearguments,and$#forthe numberofarguments.Theprocessnumberoftheshellexecutingthescriptis givenby$$.thereadnumberstatementassignskeyboardinputtothe variablenumber. Toexecutethisscript,wefirsthavetomakethefilesimpleexecutable: $ ls -l simple -rw-r--r-1 will finance 175 simple $ chmod +x simple $ ls -l simple -rwxr-xr-x 1 will finance 175 simple $ ./simple hello world The number of arguments is 2 The arguments are hello world The first is hello My process number is 2669 Enter a number from the keyboard: 5 The number you entered was 5 $ Dec 13

Dec 13

Wecanuseinputandoutputredirectioninthenormalwaywithscripts,so: $ echo 5 | simple hello world

wouldproducesimilaroutputbutwouldnotpausetoreadanumberfromthe keyboard.

8.5MoreAdvancedShellScripting

if-then-elsestatements

Shellscriptsareabletoperformsimpleconditionalbranches: if [ test ] then commandsiftestistrue else commandsiftestisfalse fi Thetestconditionmayinvolvefilecharacteristicsorsimplestringor numericalcomparisons.The[usedhereisactuallythenameofa command(/bin/[)whichperformstheevaluationofthetest condition.Thereforetheremustbespacesbeforeandafteritaswellas beforetheclosingbracket.Somecommontestconditionsare: -s file trueiffileexistsandisnotempty -f file trueiffileisanordinaryfile -d file trueiffileisadirectory -r file trueiffileisreadable -w file trueiffileiswritable -x file trueiffileisexecutable $X -eq $Y trueifXequalsY $X -ne $Y trueifXnotequaltoY $X -lt $Y trueifXlessthan$Y $X -gt $Y trueifXgreaterthan$Y $X -le $Y

trueifXlessthanorequaltoY $X -ge $Y trueifXgreaterthanorequaltoY "$A" = "$B" trueifstringAequalsstringB "$A" != "$B" trueifstringAnotequaltostringB $X ! -gt $Y trueifstringXisnotgreaterthanY $E -a $F trueifexpressions E and F arebothtrue $E -o $F trueif eitherexpression E orexpression F istrue

forloops Sometimeswewanttoloopthroughalistoffiles,executingsome commandsoneachfile.Wecandothisbyusingaforloop: for variable in list do statements (referringto $variable) done Thefollowingscriptsortseachtextfilesinthecurrentdirectory: #!/bin/sh for f in *.txt do echo sorting file $f cat $f | sort > $f.sorted echo sorted file has been output to $f.sorted done

whileloops

Anotherformofloopisthewhileloop: while [ test ] do statements(tobeexecutedwhiletestistrue) done Thefollowingscriptwaitsuntilanonemptyfileinput.txthasbeen created: #!/bin/sh while [ ! -s input.txt ] do echo waiting... sleep 5 done echo input.txt is ready Youcanabortashellscriptatanypointusingtheexitstatement,so thefollowingscriptisequivalent: #!/bin/sh while true do if [ -s input.txt ] echo input.txt is ready exit fi echo waiting... sleep 5 done

casestatements casestatementsareaconvenientwaytoperformmultiwaybranches whereoneinputpatternmustbecomparedtoseveralalternatives: case variable in

pattern1) statement(executedifvariablematchespattern1) ;; pattern2) statement ;; etc. esac Thefollowingscriptusesacasestatementtohaveaguessatthetype ofnondirectorynonexecutablefilespassedasargumentsonthebasis oftheirextensions(notehowthe"or"operator|canbeusedtodenote multiplepatterns,how"*"hasbeenusedasacatchall,andtheeffectof theforwardsinglequotes`): #!/bin/sh for f in $* do if [ -f $f -a ! -x $f ] then case $f in core) echo "$f: a core dump file" ;; *.c) echo "$f: a C program" ;; *.cpp|*.cc|*.cxx) echo "$f: a C++ program" ;; *.txt) echo "$f: a text file" ;; *.pl) echo "$f: a PERL script" ;; *.html|*.htm) echo "$f: a web document"

;; *) echo "$f: appears to be "`file -b $f` ;; esac fi done

capturingcommandoutput AnyUNIXcommandorprogramcanbeexecutedfromashellscript justasifyouwouldonthelinecommandline.Youcanalsocapturethe outputofacommandandassignittoavariablebyusingtheforward singlequotes` `: #!\bin\sh lines=`wc -l $1` echo "the file $1 has $lines lines" Thisscriptoutputsthenumberoflinesinthefilepassedasthefirst parameter.

arithmeticoperations TheBourneshelldoesn'thaveanybuiltinabilitytoevaluatesimple mathematicalexpressions.FortunatelytheUNIXexpr commandis availabletodothis.Itisfrequentlyusedinshellscriptswithforward singlequotestoupdatethevalueofavariable.Forexample: lines = `expr $lines + 1` adds1tothevariablelines.exprsupportstheoperators+,-,*,/, %(remainder),<,<=,=,!=,>=,>,|(or)and&(and).

8.6StartupShellScripts
Whenyoufirstlogintoashell,yourshellrunsasystemwidestartupscript

(usuallycalled/etc/profileundersh,bashandkshand /etc/.login undercsh).Itthenlooksinyourhomedirectoryandruns yourpersonalstartupscript(.profileundersh, bashandkshand .cshrcundercshandtcsh).Yourpersonalstartupscriptistherefore usuallyagoodplacetosetupenvironmentvariablessuchasPATH,EDITOR etc.Forexamplewithbash,toaddthedirectory ~/bintoyourPATH,you canincludetheline: export PATH=$PATH:~/bin inyour.profile.Ifyousubsequentlymodifyyour.profileandyou wishtoimportthechangesintoyourcurrentshell,type: $ source .profile or $ . ./profile

Thesource commandisbuiltintotheshell.Itensuresthatchangestothe environmentmadein.profileaffectthecurrentshell,andnottheshell thatwouldotherwisebecreatedtoexecutethe.profilescript. Withcsh,toaddthedirectory ~/bintoyourPATH,youcanincludethe line: set path = ( $PATH $HOME/bin ) inyour.cshrc.

IntroductiontoUNIX: ExerciseSheetEight

1. UseyourfavouriteUNIXeditortocreatethesimpleshellscriptgiven

2.

3. 4. 5.

inSection8.4ofthenotes.Runit,andseehowthecontentsofthescript relatestotheoutput. Extendthescriptsothatitgeneratesarandomsecretnumberbetween1 and100(c.f.ExerciseSheet3,Question16)andthenkeepsaskingthe usertoguessthesecretnumberuntiltheyguesscorrectly.Thescript shouldgivetheuserhintssuchas"I'msorryyourguessistoolow"or "I'msorryyourguessistoohigh". Writeashellscriptwhichrenamesall.txtfilesas.textfiles.The commandbasenamemighthelpyouhere. Writeashellscriptcalledpidofwhichtakesanameasparameterand returnsthePID(s)ofprocesseswiththatname. Shellscriptscanalsoincludefunctions.Functionsaredeclaredas: function funcname() { statements }

andinvokedasfuncnameparam1param2...Theparameterspassedto thefunctionareaccessibleinsidethefunctionthroughthevariables$1, $2,etc.Nowaddausage()functiontoyourpidofscriptwhich printsusageinstructions.Callusage()ifthewrongnumberof parametersispassedtothescript. 6. Modifyyour.bash_profilescriptsothatyourPATHincludesthe currentdirectory(.)andsothatyourenvironmentvariableEDITORis settoemacsorvi (orwhateverelseyouprefertouse).Runthe

modifiedscriptusingsource .bash_profile andcheckthatthe changesyoumadehavebeenappliedtothecurrentshell(typeset).