HowYAFFSWorks

CharlesManning 20072010 TableofContents

1Purpose...................................................................................................................................................2 2WhatIsYAFFS?....................................................................................................................................2 3Designandcodingstrategies..................................................................................................................2 4Terminology:YAFFSvsYAFFS1vsYAFFS2....................................................................................3 5Objects...................................................................................................................................................3 6TheYAFFSNANDModel....................................................................................................................4 7HowYAFFS1StoresFiles.....................................................................................................................4 8Garbagecollection.................................................................................................................................7 9YAFFS1Serialnumbers........................................................................................................................8 10YAFFS2NANDmodel........................................................................................................................9 10.1.YAFFS2Extendedtags...............................................................................................................11 11BadblockhandlingNANDerrorhandling........................................................................................11 12RAMstructures..................................................................................................................................12 12.1.yaffs_Object................................................................................................................................13 12.2.ObjectIdlookup..........................................................................................................................13 12.3.Directorystructure.......................................................................................................................14 12.4.Hardlinks....................................................................................................................................15 12.5.Symboliclinkandspecialobject.................................................................................................16 12.6.Fileobject....................................................................................................................................16 12.6.1Tnodetree...........................................................................................................................16 13Howvariousmechanismswork.........................................................................................................17 13.1.BlockandChunkManagement...................................................................................................17 13.1.1BlockStates........................................................................................................................17 13.1.2BlockandChunkAllocation..............................................................................................19 13.1.3Awordaboutwearleveling................................................................................................20 13.1.4yaffs_Tnodeandyaffs_ObjectManagement......................................................................20 13.2.InternalCache..............................................................................................................................21 13.3.Scanning......................................................................................................................................22 13.3.1YAFFS1Scanning..............................................................................................................22 13.3.2YAFFS2Scanning..............................................................................................................22 13.4.Checkpoint...................................................................................................................................23 13.5.ExtendedTagsandPackedTags.................................................................................................24 13.6.Inbandtags..................................................................................................................................24 13.7.SoftDeletion................................................................................................................................25

Page1/25

1 Purpose
Thepurposeofthisdocumentistogiveareasonableexplanationofmostofthecoremechanismsthatmakeyaffswork.As ofthetimeofwritingthis,theyaffs2codebaseincludesapproximately19,000linesofcodemakingaverydetailed discussionimpractical. Thisdocumentshouldserveasafirstportofcallforunderstandingyaffs.Furtherdetailedquestionsanddiscussionsare welcomedontheyaffsdiscussionlist. Thisdocumentisupdatedperiodically.Yaffsmechanismdesignchangesfromtimetotimeandthisdocumentwillnot alwaysreflectthemostuptodatecode.

2 WhatIsYAFFS?
YAFFSstandsforYetAnotherFlashFileSystem,atermcoinedbyCharlesManningin2001whensuggestingthatthe alreadyclutteredflashfilesystemspacecoulddowithyetanotherofferingthistimeaflashfilesystemdesignedfromthe grounduptoworkwithNANDflash. ThefirstreleaseofYAFFSwasdevelopedduringtheendof2001andthebeginningof2002.Bymid2002,YAFFSwas beingtrialledinvariousproducts.InMay2002,YAFFSwasannouncedandafewmoreprojectsstartedplayingwithit.In June,portingtootherOSsstarted.InSeptember2002,YAFFSwasannouncedonlinuxdevices.comwhichstartedafar wideruptakeofYAFFS. YAFFS1,theoriginalversionofYAFFS,supported512bytepageNANDdeviceswithaSmartMedialikememorylayout. YAFFS2isafollowuptoYAFFS1extendingYAFFS1tosupportlargeranddifferentdeviceswithdifferentconstraints. YAFFSishighlyportableandhasbeenusedinmanydifferentproductsandapplicationsasvariedassewingmachines, pointofsale,phonesandaerospace.YAFFShasbeenusedwithmultipledifferentoperatingsystems.Nativesupportis availableforLinux,WindowsCEandeCOSwhileyaffsDirectInterfaceprovidesalayerthatcanbeusedinother applicationssuchasRTOSs.YAFFShasbeenusedwithmultipledifferentCPUsandcompilers. YAFFShasalsobeenusedasaNORfilesystemandevenasaRAMfilesystem.

3 Designandcodingstrategies
YAFFSisdesignedtoworkinmultipleenvironmentswhichdrivestheneedforportability.Portabilityalsoimprovesthe debuggingandtestingsinceitiseasiertodevelopanddebugcodeinanapplicationenvironmentthanwithinanOSkernel. ThisgenerallyallowsYAFFStoprogressfasterwithlessprogrammingresources.Primarystrategiesthatimprove portabilityinclude:

NoOSspecificfeaturesusedinmaincodebody. Nocompilerspecificfeaturesusedinthemaincodebody. AbstracttypesandfunctionsusedtoallowUnicodeorASCIIoperation.

Page2/25

Simplicityisanotherkeygoal.Complexityislimitedtothosesituationsthatprovidesignificantimprovementsto performanceorutility.Simplicityimprovesrobustnessandtheeaseofintegrationanddevelopment.Primarystrategiesthat improvesimplicityare:

Singlethreadedmodel.YAFFSislockedonaperpartitionbasisatahighlevel.Thisissimplerthantracking lowerlevellocking.YaffsDirectInterfaceusesasinglelockforallpartitions. Logstructuremakesforasimplergarbagecollectorandallocationmethodology. Abstractionsthatbuildlayeredcodewhichiseasiertounderstandanddebug.

4 Terminology:YAFFSvsYAFFS1vsYAFFS2
BeforeweembarkonourquesttounderstandwhatYAFFSis,aterminologyexplanationisinorder. TherearetwoversionsofYAFFScode:theoriginalyaffscodebaseandthecurrentyaffs2codebase.Theyaffs2codebase supportsfunctionalityprovidedbytheyaffscodebaseaswellasbeingextendedwithnewfunctionalitytoprovideextra modesofoperation,mostimportantlythoserequiredtoworkwithlargerandmoremodernNANDparts.Theyaffs2code basesupportstheyaffs1modeofoperationthroughabackwardcompatibilitymode. Unlessexplicitlymentioned,thistextreferstotheyaffs2codebase. ThistextusesthreetermsYAFFS,YAFFS1andYAFFS2dependingonvariousmodesofoperation.
YAFFS1isasimplermodeofoperationthatusesdeletionmarkerstotrackstate. YAFFS2isamorecomplexmodeofoperatingthatwasdevelopedtosupportlargerflashtypesthatcannotuseddeletion

markers.

YAFFSmeansoperationscommontoboth.

YAFFS1originallyonlyworkedwith512bytepagedevicesbuthasbeenextendedtosupportlargerflashpagessuchas IntelM18flashwith1kpages. YAFFS2wasoriginallydesignedfor1korlargerpagesizesbutcanbeusedwithsmallerpagesbyusinginbandtags. SincetheYAFFS1modeofoperationissimplertounderstandthantheYAFFS2modeofoperation,YAFFS1willbe describedfirstandreadersinterestedinYAFFS2operationshouldfirstfullyacquaintthemselveswiththeYAFFS1modeof operation.Itissuggestedthatthisdocumentbereadsequentiallyratherthanbrowsingit.

5 Objects
InYAFFSspeak,anObjectisanythingthatisstoredinthefilesystem.Theseare:

Regulardatafiles Directories Hardlinks Symboliclinks

Page3/25

Specialobjects(pipes,devicesetc).

AllobjectsareidentifiedbyauniqueintegerobjectId. InthestandardPOSIXmodel,asusedbyLinux,thereareinodesanddirectoryentries(dentries). Aninodemaybearegularfile,directory,orspecialfile. Directoryentriesprovidethenamingmechanismthatisusedtolocateinodes. UnderPOSIX,eachinodemayhavezero,oneormanydentries:

Onedentryperinodeiscommonlyunderstood. Multipledentriesperinodeallowthesameinodetobeaccessedviamultiplenames.Thisisachievedthroughhard links. Zerodentriesperinodeislessoftenunderstood.Thishappenswhenaninodeisunlinkedwhilethereisstilla handleopentotheinode.Theunlinkingdeletesthedentrybuttheinodestillexists.Assoonasthehandleisclosed theinodeisdeletedtoo.

6 TheYAFFSNANDModel
ThememoryinNANDflashisarrangedinpages.Apageistheunitofallocationandprogramming.InYAFFS,theunitof allocationisthechunk.TypicallyachunkwillbethesameasaNANDpage,butthereisflexibilitytousechunkswhich maptomultiplepages(eg.AsystemmayhavetwoNANDchipsinparallelrequiring2x512=1024bytechunks).This distinctiongivesalotofflexibilityinhowthesystemcanbeconfigured.Inthistextthetermpageandchunkare synonymousunlessstatedotherwise. Many,typically32to128butasmanyasafewhundred,chunksformablock.Ablockistheunitoferasure. NANDflashmaybeshippedwithbadblocksandfurtherblocksmaygobadduringtheoperationofthedevice.Thus, YAFFSisawareofbadblocksandneedstobeabletodetectandmarkbadblocks. NANDflashalsotypicallyrequirestheuseofsomesortoferrordetectionandcorrectioncode(ECC).YAFFScaneither useexistingECClogicorprovideitsown.

7 HowYAFFS1StoresFiles
YAFFS1hasamodifiedlogstructure,whileYAFFS2(seebelow)hasatruelogstructure.Atruelogstructuredfilesystem onlyeverwritessequentially.YAFFS1usesdeletionmarkers,whichbreaksthesequentialwriterule.YAFFS2doesnotuse deletionmarkersandisthusatruelogstructuredfilesystem. Insteadofwritingdatainlocationsspecifictothefiles,thefilesystemdataiswrittenintheformofasequentiallog,The entriesinthelogareallonechunkinsizeandcanholdoneoftwotypesofchunk:

Datachunk:Achunkholdingregulardatafilecontents. ObjectHeader:Adescriptorforanobject(directory,regulardatafile,hardlink,softlink,specialdescriptor,...). Thisholdsdetailssuchastheidentifierfortheparentdirectory,objectname,etc.

Page4/25

Eachchunkhastagsassociatedwithit.Thetagscomprisethefollowingimportantfields(othersbeingignoredfornow):

ObjectId:Identifieswhichobjectthechunkbelongsto. ChunkId:Identifieswhereinthefilethischunkbelongs.AchunkIdofzerosignifiesthatthischunkcontainsan objectHeader.ChunkId==1signifiesthefirstchunkinthefile(ie.Atfileoffset0),chunkId==2isthenextchunk andsoon. DeletionMarker:(YAFFS1only)Showsthatthischunkisnolongerinuse. ByteCount:Numberofbytesofdataifthisisadatachunk. SerialNumber:(YAFFS1only)SerialnumberusedtodifferentiatechunkswiththesameobjectIdandchunkId.

Nowlet'sbringthatalltogetherinanexplanation.ThisisasimplifiedexplanationandskipsoverissuessuchasSoft Deletion(mentionedlaterinthedocument).Forexplanationpurposes,weshallconsiderusageoffictitiousNANDtypethat has4chunksperblock,startingwithanempty(erased)flash. Firstwe'llcreateafile.

Block 1

Chunk ObjectId 0 500

ChunkId 0

Deletion Live

Comment ObjectHeaderforthisfile(length0)

Nextwewriteafewchunksofdatatothefile.

Block 1 1 1 1

Chunk ObjectId 0 1 2 3 500 500 500 500

ChunkId 0 1 2 3

Deletion Live Live Live Live

Comment ObjectHeaderforthisfile(length0) Firstchunkofdata Secondchunkofdata Thirdchunkofdata

Nextweclosethefile.Thiswritersanewobjectheaderforthefile.Noticehowthepreviousobjectheaderisdeleted.

Block 1 1 1 1

Chunk ObjectId 0 1 2 3 500 500 500 500

ChunkId 0 1 2 3

Deletion Deleted Live Live Live

Comment Obsoletedobjectheader(length0) Firstchunkofdata Secondchunkofdata Thirdchunkofdata

Page5/25

500

Live

Newobjectheader(lengthn)

Let'snowopenthefileforread/write,overwritepartofthefirstchunkinthefileandclosethefile.Thereplaceddataand objectheaderchunksbecomedeleted.

Block 1 1 1 1 2 2 2

Chunk ObjectId 0 1 2 3 0 1 2 500 500 500 500 500 500 500

ChunkId 0 1 2 3 0 1 0

Deletion Deleted Deleted Live Live Deleted Live Live

Comment Obsoleteobjectheader. Obsoletefirstchunkofdata Secondchunkofdata Thirdchunkofdata Obsoleteobjectheader Newfirstchunkoffile Newobjectheader

Nowlet'sresizethefiletozerobyopeningthefilewithO_TRUNCandclosingthefile.Thiswritesanewobjectheader withlength0andthecontentsareprunedmakingthedatachunksdeleted.

Block 1 1 1 1 2 2 2 2

Chunk ObjectId 0 1 2 3 0 1 2 3 500 500 500 500 500 500 500 500

ChunkId 0 1 2 3 0 1 0 0

Deletion Deleted Deleted Deleted Deleted Deleted Deleted Delted Live

Comment Obsoleteobjectheader. Obsoletefirstchunkofdata Secondchunkofdata Thirdchunkofdata Obsoleteobjectheader Deletedfirstchunkoffile Obsoletedobjectheader Newobjectheader(length0).

Notehowallthepagesinblock1arenowmarkedasdeleted.Thismeansthatblock1nolongercontainsusefulinformation sowecannoweraseblock1andreusethespace. Wewillnowrenamethefile.Todothiswewriteanewobjectheaderforthefile

Page6/25

Block 1 1 1 1 2 2 2 2 3

Chunk ObjectId 0 1 2 3 0 1 2 3 0 500 500 500 500 500

ChunkId

Deleted

Comment Erased Erased Erased Erased

0 1 2 0 0

Deleted Deleted Deleted Deleted Live

Obsoleteobjectheader Deletedfirstchunkoffile Obsoletedobjectheader Obsoleteobjectheader Newobjectheadershowingnewname

Block2nowcontainsonlydeletedchunksandcanbeerasedandreused. Notehowthetagstellus:
whichchunksbelongtowhichobject, thepositionofthechunkswithinafile, whichchunksarecurrent

WiththatinformationwecanrecreatethestateofthefilesregardlessoftheplacementofthechunksinNAND.Thismeans thatshouldthesystemfail(crash,powerfail,...)atanypointduringthesequencesabove,wearestillabletorecoverthefile systemstateuptothatpoint. Notethattherearenofileallocationtablesorsimilarstructuresstoredintheflash.Thisreducestheamountofwritingand erasing(improvingwriteperformance)whilealsoimprovingrobustness.Corruptionoffileallocationtablesandsimilar structuresisacommonfailuremechanismforembeddedfilesystems.Thelackofsuchstructuresmakesyaffsmorerobust. Or,toputitanotherway,youcan'tcorruptwhatyoudon'tstore.

8 Garbagecollection
Aswehaveshownsofar,whenablockismadeuponlyofdeletedchunks,thatblockcanbeerasedandreused.However, considerwhatwillhappenifwehavemanyblockswhicheachhaveafewcurrentchunksinthem.Clearlywecannoterase anyofthecurrentblocksorwewilldestroydata.Whatweneedtodoiscopytheusefuldatachunksoffablock,deleting theoriginalsandallowingtheblocktobeerasedandreused.Thisprocessisreferredtoasgarbagecollection. TheYAFFSgarbagecollectionprocessworksasfollows: 1. 2. Findablockworthcollecting(accordingthetheheuristicsbelow).Ifnonefound,thenexit. Iteratethroughthechunksintheblock.Ifthechunkisinusethencreateanewcopyanddeletetheoriginal.Fixup theRAMdatastructurestoreflectthechange.

Page7/25

Onceallchunksinthisblockhavebeendeleted,theblockcanbeerasedreadyforreuse. Notethatalthoughstep2deletestheoriginalchunkwhenitgetscopiedthereisnotmuchpointinactuallyprogrammingthe deletionmarkerinflashsincewe'reabouttoerasetheblockanyway.Thusyaffsdoesnotprogramthedeletionmarkerin thesecases.Ifpowerislostpartwaythroughtheoperationwecanstilltellthedifferencebetweentheoriginalandthecopy bylookingattheserialnumber(notthatitreallymattersanywaysincethetwohavethesamecontentssoeithercanbe used). Theheuristicstodeterminewhetherablockisworthcollectingareasfollows: 1. 2. Iftherearemanyerasedblocksavailable,thenYAFFSdoesnotneedtoworkhard,butwillinsteadonlyattemptto performgarbagecollectionofblockswithveryfewchunksinuse.Thisistermedpassivegarbagecollection. If,however,thereareveryfewerasedblocks,YAFFSwillworkmuchhardertorecovermorespaceandwill garbagecollectblockswithmorechunksinuse.Thisistermedaggressivegarbagecollection.

OK,thepassive/aggressivenamesareabitsilly,buttheywerecontrivedaroundmidnight...givemeabreak! Ifgarbagecollectionisaggressive,thewholeblockiscollectedinasinglegarbagecollectioncycle.Ifthecollectionis passivethenthenumberofcopiesisreducedthusspreadingtheeffortovermanygarbagecollectioncycles.Thisisdoneto reducegarbagecollectionloadandimproveresponsiveness. Therationalebehindtheaboveheuristicsistodelaygarbagecollectionwhenpossibletoreducetheamountofcollection thatneedstobeperformed,thusincreasingaveragesystemperformance.Yetthereisaconflictinggoaloftryingtospread thegarbagecollectionsothatitdoesnotallhappenatthesamecausingfluctuationsinfilesystemthroughput.These conflictinggoalsmakegarbagetuningquitechallenging. Allflashfilesystemsneedsomesortofgarbagecollectortoreclaimspace.Thegarbagecollectorisoftenadetermining factorintheperformanceoftheflashfilesystem.Manyfilesystemshavetodoasignificantamountofeffortinasingle collectionsweep.YAFFSonlydealswithoneblockatatimewhichconstrainstheamountofeffortinagarbagecollection cycle,thusreducingthestalltime. TheYAFFSgarbagecollectionalgorithmisunderreviewwiththegoalofreducingstalltimeandincreasingthroughput.

9 YAFFS1Serialnumbers
InYAFFS1,eachchunkismarkedwitha2bitserialnumberthatisincrementedeverytimeachunkwiththesametags valueisreplaced(eg.Whenachunkisoverwrittenorwhencopiedbygarbagecollection). Nowlet'ssupposefirstthatwedidnothaveaserialnumber.

Page8/25

Let'sconsiderarenamewhereanobjectheaderchunkmustbereplacedwithanewonewiththenewname.YAFFScould dothefollowing:
Deletetheoldchunk. Writethenewchunk.

Ifthesystemfailsafterthedelete,thesystemcouldendupwithnochunksmatchingtherelevanttagsvalues.Thiscould causefilestobelost(endupinlost+found).Thus,wehavetowritethenewchunkbeforedeletingtheoldchunk. Butnowconsiderwhatmighthappenifwedidthis:

Writenewchunk. Deleteoldchunk.

Ifthesystemfailedafterthewritewenowhavetwochunksthatmatchthetags.Howcanwetellwhichoneiscurrent? Thecurrentchunkisdeterminedbyinspectingtheserialnumber.Sincetheserialnumberisonlyincrementedbyone beforetheoldchunkisdeleted,a2bitserialnumberissufficient.Validpairsare: Old 00 01 10 11 New 01 10 11 00

Thus,byinspectingtheserialnumberattachedtotheoldandnewtagsthecurrentchunkcanbedeterminedandtheold chunkcanbedeleted.

10 YAFFS2NANDmodel
YAFFS2isanextensionofYAFFSdesignedtofulfillanewsetofobjectivestoworkwithnewerNANDtypesincluding:

Zerooverwrites.YAFFS1onlyeveroverwritesasinglebyteinthespareareaofthechunkstosetthedeletion marker.MoremodernNANDdevicesarelesstolerantofoverwritesandYAFFS2performsnooverwritesatall. Sequentialchunkwritingwithinablock.MoremodernNANDreliabilityspecificationstendtoassumesequential writing.SinceYAFFS2doesnotwritedeletionmarkers,thewritingofchunkswithinablockisstrictlysequential.

SinceYAFFS2performslesswrites,thereispotentialforYAFFS2tohaveimprovedwriteperformance. YAFFS2cannotuseadeletionmarkerbecausethatwouldviolatethezerooverwritesmandateandalternatemechanisms

Page9/25

areprovidedtodeterminewhichchunksarecurrentandwhicharedeleted.Thesemechanismsare:

SequenceNumber:Aseachblockisallocated,thefilesystem'ssequencenumberisincrementedandeachchunk intheblockismarkedwiththatsequencenumber.Thesequencenumberthusprovidesawayoforganisingthelog inchronologicalorder. ShrinkHeaderMarker:Theshrinkheadermarkerisusedtomarkobjectheadersthatarewrittentoshrinkthesize ofadatafile.Thisisexplainedmoreintheaccompanyingtext.

Note:TheYAFFS2sequencenumberisnotthesameastheYAFFS1serialnumber! Note:Westillrefertochunkdeletioninyaffs2.ThechunksaremarkedasdeletedintheRAMdatastructuresusedby garbagecollectionetc,butthereisnodeletionmarkerwrittentoflash. ThesequencenumberallowsthechronologicalordertobedeterminedwhichallowsYAFFStodeterminethesequenceof eventsandrestorethefilesystemstate.Thisisachievedbyscanningbackwardschronologically(ie.Fromthehighest sequencenumberbackwardstothelowestsequencenumber).Thus:

Sincewe'rescanningbackwards,themostrecentlywrittenandthuscurrentchunkmatchingan objectId:chunkIdpairwillbeencounteredfirstandallsubsequentmatchingchunksmustbeobsoleteandtreatedas deleted. Thefilelengthinobjectheadersareusedtotrimbackresizedfiles.Ifanobjectheaderisencountered,then subsequentdatachunksthatliebeyondthatfilelengthareclearlyobsoleteandtreatedasdeleted.Notethatboth currentandobsoleteobjectheadersizesmustbeusedtoobtainanaccuratereconstruction.

Explainingthepurposeoftheshrinkheadermarkerisabitconvoluted.Thepurposeofashrinkheadermarkeristoshow thatthisobjectheaderindicatesthatafilehasshrunkeninsize,andtopreventthoseobjectheadersfrombeingdeletedby garbagecollection.Tounderstandthis,itiseasiesttoconsiderwhatwouldhappeniftheshrinkheadermarkersdidnot exist. Forsimplificationwe'llassumenogarbagecollectionhappensduringthispartoftheexercise.Itissafetomakethat assumptionsinceitdoesnotalterthereasoning. Considerafilethathasbeenthroughthefollowing: h=open(foo,O_CREAT|O_RDWR,S_IREAD|S_IWRITE);/*createfile/ write(h,data,51024*1024);/*write5MBofdata/ truncate(h,1024*1024);/*truncateto1MB*/ lseek(h,2*1024*1024,SEEK_SET);/*setthefileaccesspositionto2MB*/ write(h,data,1024*1024);/*write1MBofdata*/ close(h); Thefilewillnowbe3MBinlength,buttherewillbeaholebetween1MBand2MB.AccordingtoPOSIX(andtoaid securityetc)thatholeshouldalwaysreadbackaszeros. YAFFS2wouldrepresentthatinNANDwiththefollowingsequenceofchunks: 1. 2. Objectheaderforfilecreation(filelength=0) 5MBworthofdatachunks(0to5MB)

Page10/25

3. 4. 5.

Objectheaderfortruncation(filelength=1MB) 1MBworthofdatachunks(2MBto3MB) Objectheaderforclose(filelength=3MB)

Atthisstage,onlythethefollowingdatachunksarecurrent:

Thefirst1MBofdatacreatedinstep2above. The1MBofdatacreatedinstep4above. Theobjectheadercreatedinstep5above.

ButYAFFSmustrememberthetruncationthathappenedinstep3,otherwiseYAFFSwillforgetthatthereisahole, whichwouldcausetheolddatatobevisibleagain.Thus,ifyaffscreatesaholeitfirstwritesashrinkheadertoindicatethe startoftheholeandaregularobjectheadertomarktheendofthehole. Theshrinkheadermodifiesthebehaviorofthegarbagecollectortoensurethattheseshrinkheadersarenoteraseduntilitis safetodoso.Thisunfortunatelyhasanegativeimpactongarbagecollectioninsystemswhichmakeextensiveuseoffiles withholes. Shrinkheadersarealsousedtoindicatethatafilehasbeendeletedandthattherecordofthefiledeletionisnotlost.

10.1. YAFFS2Extendedtags
YAFFS2extendsthetagsinobjectheaderswithextrafieldstoimprovemountscanningperformance.Theoperationof extendedtagsisdescribedlaterinthisdocument.

11 BadblockhandlingNANDerrorhandling
NANDflashisdesignedtobeverylowcostandveryhighdensity.Inordertogethighyields,andthuskeepcostsdown, NANDflashistypicallyshippedwithafewbadblocksandsomefurtherblocksareexpectedtogobadduringtheuseofthe device.Thus,anyflashfilesystemwithoutaneffectivebadblockhandlingpolicyisunsuitableforNANDflash. YAFFS1usestheSmartMediastyleofbadblockmarking.Thischecksthesixthbyte(byte5)ofthesparearea.Inagood blockthisshouldbe0xFF.Afactorymarkedbadblockshouldbe0x00.IfYAFFS1determinesthatablockhasgonebadit ismarkedbadwith0x59('Y').UsingadistinctivemarkerallowsYAFFSmarkedbadblockstobedifferentiatedfrom factorymarkedbadblocks. YAFFS2modeisdesignedtosupportawiderrangeofdevicesandsparearealayouts.Thus,YAFFS2doesnotdecide whichbytestomarkbutinsteadcallsdriverfunctionstodeterminewhethertheblockisbad,ortomarkitbad. So,whendoesYAFFSmarkablockbad?YAFFSwillmarkablockbadifareadorwriteoperationfailsorifthreeECC errorsaredetected.Ablockthatismarkedbadisretiredfromuse,thusimprovingtherobustnessofthefilesystem.This policyissuitableforSLCflash.MLCpoliciesareunderdevelopment. Further,NANDflashcellscansometimesbedisturbed(corrupted)byNANDactivityandchargeloss.Theseerrorsare correctedbyerrorcorrectioncodes(ECC)whichmaybeimplementedinhardware,softwaredrivers,orwithinYAFFS

Page11/25

itself.Again,anyflashfilesystemlackingeffectiveECChandlingisunsuitableforNANDflash. YAFFS1modecanuseeitherthebuiltinECCoruseECCprovidedbythedriverorhardware. SinceYAFFS2modeisdesignedforawiderrangeofdevices,itdoesnotprovideECCinternallybutrequiresthatthedriver providetheECC. TheECCcodesuppliedwithYAFFS(yaffs_ecc.c)isthefastestCcodeimplementationofaSmartMediacompatibleECC algorithmthatweareawareof.Thiscodewillcorrectanysinglebiterrorwithina256bytedatablockanddetect2errors per256bytedatablock.ThisissufficienterrorcorrectiontoprovideveryhighreliabilityonSLCtypeNANDflash. Foramorecompletedescriptionoffailurehandling,pleaserefertotheYAFFSErrormitigationdocument.

12 RAMstructures
WhileitistheoreticallypossibletousealogstructuredfilesystemwithveryfewRAMstructures,thiswouldprovide reduceperformance.Thus,significantRAMdatastructuresarerequiredtostoreenoughinformationtoprovidesufficiently highperformance. Beforewejumpin,itisagoodideatofirstfigureoutwhatpurposetheRAMstructuresserve.Themainstructuresare definedinyaffs_guts.handtheirmainpurposes,are:

Device/partition:Thisisnamedyaffs_Device.Thisisrequiredtoholdinformationrelatingtoayaffspartitionor mountpoint.Usingthese,ratherthanglobals,allowsYAFFStosimultaneouslysupportmultiplepartitionsand partitionsofdifferenttypes.Infact,almostotherdatastructuresmentionedherearepartof,oraccessedvia,the structure. NANDBlockInformation:Thisisnamedyaffs_BlockInfoandholdsthecurrentstateoftheNANDblocks.Each yaffs_Devicehasanarrayofthese. NANDChunkInformation:Thisisabitfieldattachedtotheyaffs_Devicethatholdsthecurrentinusestateofeach chunkinthesystem.Thereisonebitperchunkinthepartition. Object:Thisisnamedyaffs_Objectandholdsthestateofanobject(seesection3).Thereisoneoftheseforeach objectinthefilesystem,whereanobjectisoneofaregularfile,directory,hardlink,symboliclinkorspeciallink. Therearedifferentvariantsofyaffs_Objectstoreflectthedifferentdatarequiredforthesedifferentobjecttypes. EachobjectisuniquelyidentifiedbyanobjectId. Filestructure:Foreachfileobject,YAFFSholdsatreewhichprovidesthemechanismtofindthedatachunksina file.Thetreeconsistsofnodescalledyaffs_Tnodes(treenodes). Directorystructure:Thedirectorystructureallowsobjectstobefoundbyname.Thedirectorystructureisbuilt fromadoublylinkedlistthatisrootedinthedirectoryandbindstogetherthesiblingobjectswithinadirectory. Objectnumberhashtable:TheobjectnumberhashtableprovidesamechanismtofindanobjectfromitsobjectId. TheobjectIdishashedtoselectahashbucket.Eachhashbuckethasadoublylinkedlistofobjectsthatbelongin thishashbucket,aswellasacountoftheobjectsinthebucket. Cache:YAFFSprovidesaread/writecachethatsignificantlyimprovesperformanceforshortoperations.Thesize ofthecachecanbesetatruntime.

Page12/25

Theusageofthesestructuresisdescribedinmoredetailbelow. YAFFSmakessignificantuseofdoublylinkedlists.Thesedatastructuresareveryusefulbecausetheyproviderapid insertion,removalandtraversal.Theydohowevercomeattheexpenseofrequiringtwopointersratherthanjustone.

12.1. yaffs_Object
Eachobject(file,directory,...)inthesystemisrepresentedbyayaffs_Object.Theyaffs_Object'smainfunctionistostore mostoftheobjectmetadataandtypespecificinformation.Thismeansthatalmostalltheinformationaboutanobjectcanbe accessedimmediatelywithoutreadingflash. Themetadataincludes,amongstothers:
objectId:Thenumberusedtoidentifytheobject. parent:Apointertotheparentdirectory.Notallobjectshaveaparent.Therootdirectorandspecialdirectoriesfor

unlinkedanddeletedfilesdonothaveparentssothisisNULL.

shortname:Ifthenameisshortenoughtofitinasmallfixedsizedarray(default16charaacters)thenitisstoredhere

otherwiseitmustbefetchedfromflasheachtimeitisrequrested.
type:Typeofobject. permissions,time,overshipandotherattributes

Dependingontheobjecttype,yaffs_Objectalsostores:
atnodetree,fileextentsetcfordatafiles adirectorychainfordirectories anequivalentobjectforhardlinks astringforsymboliclinks

12.2. ObjectIdlookup
ThepurposeoftheobjectIdlookupmechanismistorapidlyaccessanobjectbyitsdeviceandobjectId.Thisisrequired duringgarbagecollection,scanningandsimilaroperations. Eachyaffs_Deviceusesahashtableforthispurpose.Thehashtablehas256buckets(tunable),eachholdingadoubly linkedlistoftheyaffs_Objectsinthatbucket. Thehashtable'shashfunctionisjustmaskingtheleastsignificantbitsoftheobjectId.Thisisaverycheapfunctionthat providesreasonablespread. YAFFSmakesanattempttoassignobjectIdsinawaythatkeepsthenumberofobjectsineachhashbucketquitelow,thus preventinganyonehashentrygettingtoolong.

Page13/25

12.3. Directorystructure
Thepurposeofthedirectorystructureistorapidlyaccessanobjectbyname.Thisisrequiredtoperformfileoperations suchasopening,renamingetc. TheYAFFSdirectorystructureisformedbyatreeofyaffs_ObjectsoftypeYAFFS_OBJECT_TYPE_DIRECTORY.These objectshaveadoublylinkednodeofchildren. Eachyaffs_Objectalsohasadoublylinkedlistnodecalledsiblingwhichchaintogetherthesiblingsinadirectory. EachYAFFSpartitionhasarootdirectory. EachYAFFSpartitionalsohassomespecialpurposefakedirectorieswhicharenotstoredinNANDbutareinstead createdonmounting:

lost+found:Thisdirectoryisusedasaplacetostoreanylostfilepartsthatcannotbeplacedinthenormaldirectory tree. Unlinkedanddeleted:Thesearepseudodirectoriesthatarenotinthedirectorytree.Placingobjectsinthese directoriesgivesthemastatusofbeingunlinkedordeleted.

Eachobjecthasanamethatisusedtolookuptheobjectwithinthedirectory,Thus,eachnamewithinadirectorymustbe unique.Thisruledoesnotapplytoobjectsintheunlinkedanddeletedpseudodirectoriessinceweneverusenamestolook upunlinked/deletedfilesandtheremaybemanyfilesofthesamenamethathavebeendeleted. Nameresolutionisspedupbytwomechanisms:

Shortnamesarestoreddirectlyintheyaffs_Objectsoithattheydon'thavetobeloadedfromflash. Eachobjecthasanamesum.Thisisasimplehashingofthenamewhichspeedsupthematchingprocessbecauseitis

cheapertocomparetehashthancomparethewholenamestring.

Sibling

Children

Sibling

Children

d root Sibling Children e

Sibling

data

Sibling

data

Sibling

Children

lost+ Sibling found

Children

Page14/25

Theabovedirectorystructurerepresentsthefollowingdirectorytree / /a /b /lost+found /a/c /a/d /a/e Rootdirectory Directorycontainingc,d,e Emptydirectory Emptydirectory Directory Datafile Datafile

IfaYAFFSpartitionhasbeensomehowcorruptedthenitispossibleforanobjecttobecreatedduringscanningbutwithno objectheader(whichcontainstheobjectname)beingfound.Theobjectwouldthennothaveaname.Inthesecases,YAFFS createsapseudonymfromtheobjectnumberoftheformobjnnnnsothattheobjectcanbeplacedinthedirectorystructure.

12.4. Hardlinks
InPOSIXfilesystems,theobjects(inodesinLinuxterminology,vnodeinBSDspeak)andtheobjectnames(dentryin Linuxterminology)areindependentinthatanobjectcanhavezero,oneormorenames. Havingfileswithzerolinksiscausedbyconstructssuchas:
h=open(foo,O_CREAT|O_RDWR,S_IREAD|S_IWRITE);/*createfilefoo*/ unlink(foo); /*unlinkitsoitcan'tbefoundinthedirectory*/ /*westillhaveahandletothefilesocanstillaccessitviathehandle*/ read(h,buffer,100); write(h,buffer,100); close(h);/*assoonasthehandleisclosedthefileisdeleted*/

YAFFSdoesnotstoreseparateobjectandnamerecordsbecausethatwouldwastespaceandincurextrareading/writing. Instead,YAFFSisoptimisedforthemosttypicalcaseofonenameperobject.Thatis,YAFFSstoresonenameper yaffs_Objectandusescheatstoachieveotherbehavior:

Objectsthathavezeronames(ieunlinkedobjects)arehandledbyrenamingtheobjectintoafakedirectorythatis usedtodesignateunlinkedobjects. Multiplenamesobjectarehandledbydesignatingoneoftheobjectsthemainobjectandhavingmultiplehardlink pseudoobjectsthatrefertoit. Theinodenumber(vnidinBSDspeak)isaperfilesystemuniquenumberthatidentifiestheobject.TheYAFFS objectidofthemainobjectispresentedtotheuser.Thatis,iftheinodenumberisqueriedfromahardlinkobject thentheobjectidoftheequivalentobjectisreturned.

Thelinkagebetweenhardlinkpseudoobjectsandtheirequivalentobjectisthroughtwofields:

EachobjecthasahardLinksdoublylinkedlistnodethatlinksthelistofhardlinkstothatobject. Eachhardlinkhasapointertotheequivalentobject.

Theusageofthisstructureisstraightforwardexceptforoneinterestingcasewhenhardlinksarebeingdeleted.Consider thefollowingsequenceofevents: 1 #toucha Createfilea

Page15/25

2 3

#lnab #rma

Createahardlinkbwithequivalentobjecta. Removea.Theobjectmuststillexistunderthenameb

Instep3,YAFFScannotjustdeletetheequivalentobjectsincethatwouldleaveahanginghardlinkwithnoequivalent object. Wealsocannotjustchangethetypeofoneofthehardlinkobjectsbecausethatwouldcausetheobject'sobjectidtochange. Thiswouldbeunacceptablesinceitwouldbreakanymechanismthatusetheobjectid(ofwhichtherearemanywithinand outsideofYAFFS). Thus,ifanobjectisdeletedandithashardlinkstoitthenwedothefollowing:

Selectoneoftheobject'shardlinks. Renametheobjecttothenameinthehardlink. Deletethehardlinkpseudoobject.

12.5. Symboliclinkandspecialobject
TheseobjecttypesjuststorevaluesthatareneededtoprovideafullPOSIXfilesystem. AsymboliclinkobjectsavesasymboliclinkstringthatisusedtoprovidethePOSIXsymbolicstringmechanism. Aspecialobjectisusedtostoredevicenumbers,pipesandothersimilarspecialobjects.

12.6. Fileobject
Theyaffs_Objectshaveatypeandavariantportionwhichisaunionofdifferentdatarequiredtoholdthestateofdifferent objecttypes. ThefileobjectshavetypeYAFFS_OBJECT_TYPE_FILE,andanassociatedfilevariant.Thisstoresthefollowingmain values:

fileSize:filesize topLevel:depthoftheTnodetree top:pointertotopofTnodetree

Themeaningofthefilesizeisquiteobvious.Thisisthecurrentlengthofthefile.Notethatafilemighthaveholesinit causedbyseekingforwardpasttheextentsofthefile. ThetopandtopLevelworktogethertocontroltheTnodetrees.

12.6.1 Tnodetree
EachfilehasaTnodetreetoprovidethemappingfromfilepositiontoactualNANDchunkaddress.topisapointertothe topoftheTnodetree

Page16/25

yaffs_Object

Level2

Level1

Level0

NAND

Note,only4entriesperTnodeareshowntosimplifythediagram.

TheTnodetreeismadeupofTnodes(treenodes)arrangedinlevels,eachofwhichholdseither:

Atlevelsgreaterthan0,aTnodehas8pointerstootherTnodesinthenextleveldown. Atlevel0,aTnodehas16NANDchunkIdswhichidentifythechunk'slocationinRAM.

Thesenumbersarepowersof2whichmakesforsimplelookupsbyjustapplyingbitmasks.Thecodethattraversesthistree isinyaffs_FindLevel0Tnode(). Asthefilesizegrows,moreTnodesareaddedandasthelevelsfill,morelevelsareadded.Ifthefileistruncatedorresized downwards,thenthetreeispruned.Ifachunkinafileisreplaced(eg.dataisoverwrittenorcopiedbygarbagecollection) thentheTnodetreemustbeupdatedtoreflectthechange. TnodestypicallymakeupthebulkoftheYAFFS'RAMusage.

13 Howvariousmechanismswork
HavinglookedattheYAFFSdatastructures,wenowconsiderhowsomeofthemechanismsinYAFFSoperates.Thisis mosteasilydonebywalkingthroughsomeoperationsthatYAFFSperforms.

13.1. BlockandChunkManagement 13.1.1 BlockStates

YAFFStracksthestateofeachblockandchunkinthepartition.Thisinformationisfirstbuiltupduringscanning(or recoveredfromacheckpoint).Ablockmaybeinoneofthefollowingstates(seedefinitionofyaffs_BlockState): UNKNOWN Theblockstateisnotyetknown.

Page17/25

NEEDS_SCANNING Duringprescanningithasbeendeterminedthatthisblockhassomethingonitandneeds scanning. SCANNING EMPTY ALLOCATING FULL DIRTY Thisblockiscurrentlybeingscanned. Thisblockhasnothinginit(iserased)andmaybeusedforallocation.Theblockcangettothis statebybeingerased. Thisistheblockcurrentlybeingusedforallocationbythechunkallocator. Allthechunksinthisblockhavebeenallocatedandatleastonechunkcontainsusefulinformation thathasnotyetbeendeleted. Allthechunksinthisblockhavebeenallocatedandallhavebeendeleted.Theblockmighthave beenintheFULLorCOLLECTINGstatebeforethis.Thisblockcannowbeerasedandreturned toEMPTYstate. Thisblockhascheckpointdatainit.Whenthecheckpointisinvalidatedthisblockwillbeerased andreturnedtotheEMPTYstate. Thisblockisbeinggarbagecollected.Assoonasalllivedatahasbeeninspectedthisblock becomesDIRTY. Thisblockhasbeenretiredorwasmarkedbad.Thisisaterminalstate.

CHECKPOINT COLLECTING DEAD

Page18/25

UNKNOWN
Regularruntimestates Prescanning Scanning

DEAD NEEDS SCANNING EMPTY

SCANNING

FULL

ALLOCATING

CHECKPOINT
Checkpointing

DIRTY COLLECTING
Garbagecollection

Regularruntimetransitions Errortransitions Minortransitions

Theregularruntimestatesare;EMPTY,ALLOCATING,FULL,COLLECTINGandDIRTY. Thecheckpointstateisusedforcheckpointblocks. TheDEADstateistheterminalstateforblocksthatwereretiredbecausetheywereeithermarkedbadoranerrorwas detected.

13.1.2 BlockandChunkAllocation
Anavailablechunkmustbeallocatedbeforeitcanbewritten.Thechunkallocationmechanismisprovidedby yaffs_AllocateChunk()andisverysimple.Eachpartitionhasacurrentblockthatitisallocatingfrom.Thisistermedthe allocationblock.Chunksareallocatedsequentiallyfromtheallocationblock.Asthechunksareallocated,blockandchunk managementinformationisupdated,includingtheblock'spagesInUsecountandthechunkusebitmap. Whentheblockisfullyallocated,anotheremptyblockisselectedtobecometheallocationblock.Theblocksareselected bysearchingupwardsfromthepreviousallocationblock.

Page19/25

Previoustoyaffs_guts.cV1.99,yaffsscanningwouldidentifytheallocationblockbeingusedwhenthefilesystemwaslast shutdownandwouldcontinueallocationfromthatpoint.Thishadthebenefitofcontinuingtousethesameblockand slightlyreducegarbagecreation.However,doingthiscouldpotentiallycauseproblemsduetoattemptingtowriteablock thathadbeeninthemiddleofawritewhenpowerwaslost.Thus,fromV1.99onwards,yaffsnowalwaysstartsallocation onanewblockafteramount.Thisincreasestheamountofgarbage(typicallybyaninsignificantamount)butimproves robustness.

13.1.3 Awordaboutwearleveling
Sinceflashmemoryblockscanonlyendurealimitednumberofwritesanderases(termedtheflashendurance),aflashfile systemneedstoensurethatthesamefewblocksarenotoverwrittencontinuouslycausingthoseblockstowearoutwhilethe otherblocksarehardlyusedatall..Insteaditisimportantthatthefilesystemspreadsoutthewear. ThisisextremelyimportantinafilesystemsuchasFATwhichusesalimitedsetoflogicalblockstostorethefile allocationtableentrieswhichgetmodifiedfrequently.IfFATisusedwithastaticlogicaltophysicalblockmappingthen thefileallocationtableblockswillwearoutfarbeforeotherblockscausingprematuredevicefailure.Thus,forFAT,itis veryimportantthatthelogicaltophysicalmappinggetsstirredupabittocausethefileallocationtablestobewrittento differentphysicalblocks,thuslevellingoutthewear. WearlevellingisfarlessofaconcerninalogstructuredfilesystemsuchasYAFFSsincewealwayswritetotheendofthe logandthusdon'tkeeponoverwritingthesameblocks. Therearebasicallytwowaystoachievewearleveling:
useaspecificallywrittensetoffunctionsthatactivelyperformwearleveling. allowadegreeofwearlevelingtohappenasasideeffectofotheractivity.

YAFFStakesthesecondapproach. Firstly,beinglogstructured,yaffsinherentlyspreadsoutthewearbyperformingallwritesinsequence.Thatmeansthatwe don'tsingleoutafewblocksforaspecificpurposeandwritethemmorethanothers. Secondly,blocksareallocatedseriallyfromtheerasedblocksinthepartition.Thustheprocessoferasing(afterdeletionof garbagecollection)andthenallocatingblocksdoestendtospreadtheblockuseandprovidesalevelofwearlevelling. Thus,eventhoughthereisnocodetospecificallyperformwearlevelingitdoeshappenasasideeffectofotheractivity. ThisstrategyperformswellandhassurvivedmanyacceleratedlifetimetestsinbothsimulationsandonrealNANDbased products.

13.1.4 yaffs_Tnodeandyaffs_ObjectManagement
yaffs_Tnodesandyaffs_Objectsaredynamicallycreatedandreleasedastheyareneeded. Theallocationandmanagementcouldbedonebyjustusingthestandardmemoryallocator(malloc,kmallocetc),butthis couldcauseperformanceproblems,particularlyinembeddedsystemswhichtendtohaverathersimplisticmemory allocators. Theallocationanddeletionofmanysmallstructurescancausememoryfragmentationcausingoutofmemoryand slowdownproblems. Topreventtheseproblems,YAFFSmanagestheseobjectsinternally.YAFFScreatesmanystructures(100orso)atatime

Page20/25

andthenallocatesthemsinglyfromafreelist.Whenstructuresarereleasedtheyareplacedonthefreelistforreuse.

13.2. InternalCache
ThisdescribesthestandardYAFFSinternalcacheimplementation.Alternativecacheshavebeen devisedforotherpurposesandatsomefuturepointYAFFSmightbeextendedtosupportpluggable cachepolicies. ThemainpurposeoftheinternalcacheistoreduceNANDaccessforbadlybehavedapplications whichperformmanysmallreadsandwrites.Believeitornot,butthereareapplicationsouttherethat readorwritefilesafewbytesatatime:
while(...) write(h,buf,1); Thissortofbehaviourwouldbeterribleforperformanceandflashmemorylifeifeachreadorwriteoperationresultedina writetoflash.YAFFSprovidesaninternalcachetomitigateagainstthissortofbehaviour. TheYAFFSinternalcacheiscalledaShortOpCache(shortoperationcache)andonlycomesintoplayforoperations whicharenotchunkaligned.ThecachingmechanismisverysimpleandwasprimarilywrittenforlesssophisticatedOSs thatlacktheirownfilecachinglayer.ThecacheisstillbeneficialinOSslikeLinux,whichprovideacachinglayer,because itcachesboththereadandwritepath.Thus,whileshortreadswillbeservicedbytheLinuxpagecache,theshortOpCache doesaddressnonchunkalignedwrites. Pagealignedreads/writesbypassthecacheunlessthatchunkisalreadyinthecache. Thecacheisstoredinanarrayofcacheentries.Thenumberofcacheentriesissetintheyaffs_Deviceconfigurationby dev>nShortOpCachesandthecacheisconfiguredduringinitialisation.Acachesizeofzerodisablesthecache.Thecache managementalgorithmsaresimplisticandprobablydon'tscalewelltoverylargenumbers.Thismeansthatitisprobably besttostickwitharound5to20cacheentries. Eachcacheentryholds:

objectid,chunkid,nbytes(thetaginfoofthechunkbeingcached) theactualdatabytes cacheslotstatus(lastusedcounter,dirty,...)

Findinganunusedcacheentryisasimpleoperation.Wejustiteratethroughthecacheentriessearchingforonethatisnot busy.Ifallcacheentriesarebusythenacachepushoutisperformedandthenweretrythefind. Wheneveracacheentryisaccessed(readorwritten)thelastusecounterisissettoanincrementingnumber.Thiscounter valueisusedtodrivetheleastrecentlyused(LRU)algorithm. Cacheentrieswhichhavebeenmodifiedandnotyetwrittenbacktoflasharemarkedwithadirtyflag.Thisistellsus whetherweneedtowriteoutthedataduringaflushorpushout. Apushouthappenswheneverthecacheisfullandweneedtopushanentryoutofthecachetomakespacetoloadanew entry.TodothisweusetheLRUalgorithmtoselecttheLRUcacheentry.Wethenwriteoutalldirtyentriesforthatfile,in ascendingorder,andmarktheseentriesasfree. Wheneverafileisflushed(fileflushorclose),thefile'sdirtyentriesarewrittenout.Atcertaintimes(eg.beforean unmountoratsync())theentirecacheisflushed.

Page21/25

Thiscachingpolicyisfarfromoptimalbutgivesausefulperformanceimprovementwithverysimplealgorithms(good bangforbucks). Apartfromitsnormalcachingfunction,theshortOpCachealsodoesmuchoftheheavyliftingforinbandtagssupportsince alltheinbandtagsaccessesarenotchunkaligned,theymustallgothroughtheshortOpCache.

13.3. Scanning
Scanningistheprocesswherebyfilesystemstateisrebuiltfromscratch.ThishappensonmountingaYAFFSpartitionifno checkpointdataisavailableorifthemountingwastoldtoignorethecheckpointdata. Scanninginvolvesreadingthetagsfromalltheactivechunksinthesystemsocantakequiteawhile,butprobablylesstime thanyou'dexpect. ThescanningprocessisdifferentforYAFFS1vsYAFFS2becauseofthedifferencesmentionedbefore.

13.3.1 YAFFS1Scanning
YAFFS1scanningisratherstraightforwardbecauseYAFFS1usesdeletionmarkers. Allwereallyneedtodoisreadthetagsoffallthechunksinthepartitionanddetermineifthechunkisactive.Orderisnot importantandblockscouldbereadinanyorder.Ifthedeletionmarkerhasnotbeensetthenitisactiveandweneedtoadd thechunktothefilesystemstateasfollows: Ifthechunkisadatachunk(chunkId>0)thenthechunkmustbeaddedtotherelevantfile'stnodetree.Iftheobjectdoes notexist,thenwecreateitinlost+found,althoughithasnonameandotherfiledetailsyet. Ifthechunkisanobjectheader(chunkId==0)thentheobjectmustbecreated(oritmightalreadyexistinlost+foundand weneedtofixupitsdetailsandslotitintothecorrectpartofthedirectorytree. IfatanytimeduringscanningwefindthatwearereadingasecondchunkwiththesameobjectId:chunkIdasapreviously encounteredchunkthenthe2bitserialnumberisusedtoresolvethedispute. Oncethescanningiscompletewestillneedtofixupafewthings: Hardlinksmustbeconnectedupproperly(itismoreefficienttodeferthisuntilafterthescanninghascompleted). Also,objectswhichweremarkedasunlinkedmustbedeletedsincetherearenolongerhandlesopentotheseobjects.

13.3.2 YAFFS2Scanning
YAFFS2hasnodeletionmarkerswhichmakesthescanningsignificantlymorecomplex. Thefirstthingthatneedstobedoneistoprescantheblockstodeterminethesequencenumberfortheblocks.Thelistof blocksandsequencenumbersisthensortedtomakeachronologicallyorderedlist. Wenowscanbackwardsthroughtheseblocks(ieinreversechronologicalorder).Thus,thefirstoccurrenceofany objectId:chunkIdisthemostcurrentoneanditiseasytodeterminewhichischunksareoutofdate. Wereadthetagsforeachchunkineachblock. Ifitisadatachunk(ie.chunkId>0)then:

IfachunkwiththesameobjectId:chunkIdhasalreadybeenfoundthenthisisnotthecurrentchunk. Deletethisone. Elseiftheobjectismarkedasdeleted,orcorrupted,thendeleteit. Elseiftheobjectdoesnotyetexistthenthischunkwaswrittensincethelastobjectheaderwriteforthis objectsothischunkmustbethelastwrittenchunkinthefileandmusthavehappenedjustbeforean

Page22/25

uncleanshutdownwiththefilestillopen.WecancreatethefileandusethechunkIdandnbytestosetthe fileextents.

Elseiftheobjectdoesexistandthechunkliesbeyondtheobject'sfilescansize(extents)thenitisa truncatedchunkandcanbedeleted. Elseitisplacedintothefile'stnodetree. Iftheobjectisinthedeleteddirectorythenmarktheobjectasdeletedandsettheobject'sshrinksizeto zero. Elseifanobjectheaderhasnotyetbeenfoundforthisobject,thenthisisthemostcurrentobjectheader andholdsthemostcurrentnameetc.Ifnodatachunkshaveyetbeenfoundforthischunkthenthesizein theobjectheaderisalsothefileextents.Thefilesizeisusedtosetthefilescansizesinceatthetimeit waswrittenthefiledidnotextendpastthatpointandwecansafelyignorefurtherdatachunksforthatfile thatextendpastthatpoint. Elsejustusethefilesizeasthescansizeifitissmallerthanthecurrentfilescansize. Elsethisanobsoleteobjectheader.Itisdeleted.

Ifitisanobjectheader(ie.chunkId==0)then:

Abitmoreexplanationisprobablyneededtodescribethefilescansize.Thewholepurposeofthisistodeterminedata chunksthatshouldbeignoredduetothefilebeingtruncated.

13.4. Checkpoint
Mountscanningtakesquitealotoftimeandslowsmounting,thoughperhapsbylessthanyou'dthink.Checkpointingisa mechanismtospeedthemountingbytakingasnapshotoftheYAFFSruntimestateatunmountorsync()andthen reconstitutingtheruntimestateonremounting.Thespeedupisdramatic. Theactualcheckpointmechanismisquitesimple.Astreamofdataiswrittentoasetofblockswhicharemarkedas holdingcheckpointdataandtheimportantruntimestateiswrittentothestream. Notallstateneedstobesaved,onlythestateneededtoreconstitutethestructureoftheruntimedata.Forexample,file metadataneednotbesavedsinceitisveryeasytoloadthatthroughlazyloading. Thecheckpointstoresdatainthefollowingorder:
startmarker,includingcheckpointformatid. yaffs_Deviceinformation blockinformation chunkflags objects,includingfilestructure endmarker checksum

Thecheckpointvalidityisensuredbythefollowingmechanisms:
thedataisstoredusingwhateverECCmechanismsareinplace. thestartmarkerincludesacheckpointversionsothatanobsoletecheckpointwillnotbereadifthecheckpointcode

changes.
dataiswrittentothestreamasstructuredrecords,eachwithatypeandsize.

Page23/25

theendmarkermustbereadwhenexpected. twochecksumsaremaintainedacrossthewholecheckpointdataset.

Ifanychecksfail,thecheckpointisignoredandthestateisrescanned. Theastutereadermightnoticethatblockstatechangesaswe'rewritingcheckpointblocks,sohowdowemanagetore constitutetheblockstatescorrectly?Thisisachievedbythefollowing:

Aswewritethecheckpoint,thecheckpointblockallocationselectsEMPTYblocks.Wedon'tchangethestateaswe're

wrtitngthecheckpointsotheirstateiswrittentothecheckpointasEMPTYorCHECKPOINT,itdoesnotreallymatter which.
Afterreadingthecheckpoint,weknowwhichblockshavebeenusedtostorecheckpointdata.Weupdatethoseblocksto

reflecttheCHECKPOINTstate.

Acheckpointisinvalidatedbyanymodifications(writes,erases,...)thusanymodificationpathalsocheckstoseeifthereis acheckpointanderasesit. Theregularyaffsblockallocatorandgarbagecollectionlogicalsoneedstobeawareoftheapproximatesizeofthe checkpointsothatthesecanensureenoughsparespaceisreservedtostoreacheckpoint.Thereisacalculationthatupdates thevaluewheneverthenumberofobjectsetcchanges.

13.5. ExtendedTagsandPackedTags
Extendedtagsisanabstractionofawaytostoremoreinformationinthetags,allowingforfasterscantimes.PackedTagsis aphysicalimplementationofthestorageofextendedtags. Whenwe'rescanningwereallyonlyneedtobuildupthestructureofthefilesystem.Inotherwords,rebuildthedirectory relationships,tnodetreesandsuch.Muchofthedetailsuchasfilenames,permissionsetccanbelazyloaded. Thisrequiredstoringextrainformation,suchastheparentdirectoryandobjecttype,inthetagsofobjectheaders.Buthow dowedothiswithoutenlargingthetagssize? ObjectheadersalwayshaveachunkIdofzeroandnbyteshasnomeaningbecauseitisnotadatachunk.Byusingasingle bittoidentifyobjectheaderswesuddenlyopenupawholelotofspacethatwecanstufffullofusefuldata.Bybeingclever astowhatdataweuse,wecanpackintheimportantdatainmostcircumstances. Sometimes,forexampleverylargefiles,thepackingwillnotworkbecausethereareinsufficientbitstoholdthefieldsowe justwriteregulartagsinstead.That'sOKsincetherecanonlybefewsuchfilesinthesystemsotheperformancereduction onlyimpactsveryfewfilesandthesystemlevelimpactisunnoticable.Asimple,cheap,strategythatgivesspeedup99%of thetimeisaveryhandyone. Otherdetailsarelazyloadedondemand.Thiswillhappenwhentheobjectislookedup(eg.byafileopenorsearchingfor thefileinthedirectory).

13.6. Inbandtags
Thusfarwehaveconsideredchunkswheretherearesufficientunallocatedbytesinthespare(oroutofband)areatostore thetags.Thismeansthatthedatapartofthepageisapowerof2insize.Powersof2areveryfasttodealwithwhichis whymostoperatingsystemsusevariouspowersof2intheirinternals. Butwhathappensifwedon'thaveenoughspaceinthespareareatostorethetags?Inbandtagstotherescue!Withinband tagswestorethetagsintheflashdataareaalongsidethedata.Thisgivesusfarmoreflexibilityinthetypesofstoragethat yaffscansupport,butbreaksthepowerof2alignment.Therearethreemajorpenaltieswithinbandtags:

Duetothelossofchunkalignment,alltransactionsmustpassthroughtheshortOpCache.Thiscostsanextra memcpy(). Offsettochunk,freespaceandothercalculationsarenolongerjustshiftandmaskoperationsbutinsteadrequire

Page24/25

multiplicationsanddivisionswhichcanrelativelyexpensive.

Readingtagsduringscanningetccaninvolvereadingthewholepage,makingthisslower.Theimpactisminimised byusingcheckpointing.

Thus,itisbetter,performancewise,toonlyuseinbandtagswhenregulartagswillnotwork.

13.7. SoftDeletion
SoftdeletionisamechanismthatwasaddedtospeedupfiledeletioninYAFFS1mode. LookingbacktothedescriptionofhowfilesarestoredinYAFFS1,youcanseethatdeletingafileinvolveswritingthe deletionmarkerofeverydatachunkinthefile.Thiscantakequitesometimeforlargefiles. Thesoftdeletionmechanismwasaddedtoavoidwritingallthosedeletionmarkers.Insteadofwritingthedeletionmarkers, softdeletioninsteadtracksthenumberofchunksineachblockthatbelongtodeletedfilesandleavesthecleanuptothe garbagecollector. Whenthegarbagecollectorlooksatablockitcanseewhatcurrentchunks(iechunksnotyetdeleted)belongtodeleted filesanddoesnotcopythose,thuseffectivelydeletingthechunk.Onceallthechunksassociatedwithadeletedfilehave cleanedupbythegarbagecollectorthefileheaderobjectcanalsobedeleted. YAFFS2modedoesnotwritedeletionmarkersandthusdoesnotusesoftdeletion.

Page25/25