FaceDetection EE368FinalProject Spring2003

Group5 CherylLam Yu-KuanLin DavidMahashin

Introduction Thegoaloftheprojectistoidentifyfacesinagroupphotograph.Fromacolorimage, weuseanumberofimageprocessingtechniquestodetectthelocationsoffaceswithin theimage. Therearetwomainstepsinourimplementation:maskcreationandfacecorrelation. Maskcreationinvolvesdetectingcandidatefaceregionswithintheoriginalimage.This preliminarystepselectsallpossiblefaceregioncandidates,andrejectsobviousnonfaceregions.Thesecandidatefaceregionsarethenanalyzedandcorrelatedwithan "average"facetemplate,derivedfromatrainingimage.Theendresultisasetof(x,y) imagecoordinates,identifyingallfacelocationswithintheoriginalimage. MaskCreation-ImplementationDetails

Themaskcreationportionofourimplementationreliesheavilyoncolorsegmentation andmorphologicalimageprocessing.Theresultingmask,whichidentifiespossible faceregionswithintheoriginalimage,isthencorrelatedagainsttheaveragetemplate face(derivedpriortoimageanalysisfromasingletrainingimage).Anumberof statisticsanddecisioncriteriaarethenusedtointerpretthecorrelationoutputforeach regionanddeterminethenumberandlocation(s)offaceswithineachregion.These regionsarethenmappedbacktotheoriginalimageinordertogivethelocationsof faceswithintheoriginalimage. YIQColorSegmentation Thefirststepincreatingthemaskistherejectionofnon-facialcomponents,which includeclothingandbackgroundobjects.Wedecidedtosearchforskin-color componentsintheimageusingtheYIQcolorspace.Thisapproachismotivatedbythe factthatluminanceisanotoriouslyfickledeterminantofskintones.TheYIQspace describesanimageintermsofitsluminance(Y),hue(I),andsaturation(Q).Usingthe trainingimagesastestcases,allthefaceswithintheseimagesareextractedandtheir

luminanceandhuedistributionsareobserved.Becausethefacepixelsoccupyasmall portionoftheIandQspaces,weareabletofilteroutnon-facialcomponentswith relativelysuccess.Severaldrawbacksincludetheinclusionofhandsandarmsinthe mask,aswellasotherelementswithcolortonessimilartohumanskin.

14 x10
5

HistogramofQComponentsofTraining .jpg
7

12

10

0 -150

-100

-50

50

100

150

Fig.1,a-d.HistogramsinYIQSpace SmallObjectsRejection Thesecondstepaimstofilteroutobjectsthataredeemedtoosmalltobefaces. Thethresholdisdetermineddynamicallybyobservingthedistributionofallthe non-zeroregionsinthemask.Thethresholdissetsuchthattheprobabilityof rejectingthesmallestfacesisverysmall.Aftersmallobjectrejection,weare abletovirtuallyeliminatenon-facialcomponents.Theproblemthatremainsis thejoiningofmultiplefacesintoasingularregion.Ideallywewouldlikeeach regiontocontainasingleface,thusrelegatingthefacecorrelatortodecide

whethertheregionrepresentsonefaceoranon-face.

JoinedFacesSeparation

Fig.2.Smallobjectssuccessfullyrejected

Asignificantchallengeincreatingthemaskistheseparationofjoinedfaces.In agroupphotograph,peoplewhostandclosetooneanotherwillhavetheirfaces mergedinthepreliminarymask.Thefollowingisanexampleofthreefaces joinedtogether:

Fig.3.Joinedfaceswithinasingleregion Weconsideredmanyapproachestosolvethisproblem,eachwithvaryingpoints ofsuccessandfailure.Asimpleapproachwouldbetorepeatedlyapplyan erosionoperationtotheentiremask.Thisissuccessfulonlyifthefacesoverlap onlyslightly.

Inamoreinvolvedattempt,weattempttoclassifyjoined-faceregionsby calculatingandcomparingthesizeofablobtothemedianblobsize.Regionsof sizegreaterthanonestandarddeviationofthemedianarecandidatesfor separation.Toseparatetheblobs,weemployseveralmorphologicaloperations. Simpleerosionofthemaskworkswellforfacesthatoverlapoverasmallregion. However,becausethetrainingimagescontainaregionofupto8overlapping faces,wehavetodeviseamoreinvolvedstrategy.Theprocessisdividedinto twogeneralphases.Inthefirstphase,thecandidateregionundergoes aggressiveerosiontobreakuptheregionintosmallerpieces.Atthispoint,we evaluatethesizeofeachpiece;ifitisgreaterthanacertainthreshold,thepiece isremovedandisgiventothetemplatematcherasapossibleface.Itis temptingtorejectthesmallerpiecesasnon-faces.However,experimentsshow thaterosionoftencausesthesmallerfacetobreakinhalf.Tocompensate,we dilatethesmallerpieceswiththeintentionofrecoveringfacesthatare unintentionallybrokendown.Inmostcases,thisprocessofselectivedilationis effectiveinreconstructingfaces.Thereare,however,falsealarms,whenregions suchastheneckarefalselyreconstructed.Weultimatelydecidethatgivenour edgecorrelator(describedinthefollowing)thatthisaggressiveseparationofthe blobsisunnecessary.Infact,repeatedapplicationoferosion,dilation,and thickeninggiveunnaturallyjaggededgesthatarenotbeneficialtoouredge correlator. FaceCorrelator-ImplementationDetails Oncethecolorsegmentationandmorphologicaloperationshaveoccurred,we areleftwithshapelessblob,containingboththeimageofinterestaswellas clutterandnoise.Onecommonmethodtodistinguishdesiredimage(i.e.,faces inourcase),andclutter,isthroughcorrelatingtheunknownimageswitha template,andthenmatchingtheimagebasedonthecorrelationlevel.Methods forgeneratingatemplateincludeaveragingsamplefacesfromtrainingsetsor

constructingatemplatefromeigenfacialdeconstructions.Templatematchingisa simple,yetrobust Inourproject,wefirstapproachtowardconstructingthefacetemplateisfirstby constructinganaveragefacefromallthetrainingsets,usinggroundtruthmasks forextraction.Allfacesareequallyweightedasfollows:

I H= N
N

x
i =1

Wherexirepresentseachfacialimage,scaledtothemedianwidthand

heightofthetrainingimagesetsusingbicubicinterpolation.Wechosethe medianwidthandheighttomaximizethecorrelationwithimagesofdifferent sizes.Ourfirsttemplatelookslikethis:

Fig.4.Firsttemplateface Wethenusedcalculatedthe2-Dcross-correlationtodeterminethegoodnessof match.Wewereexpectingtoseegoodseparationbetweenfacesandclutter. Insteadwediscoveredthattheaveragetemplatedoesnotperformverywellas imageidentifier.

Fig.5.a)2-faceregion.b)Cross-correlationwithfirsttemplateface

Asonecansee,thisaveragefacedoesnotgeneratehighlevelsofseparation (thusonecannotidentifythetwopicturesfromonesinglepeak),whichwouldbe necessaryfordistinguishingfaces.Worse,thisfacetemplatewasproneto identifynotonlyfaces,butskin-colorregionsingeneral,whetheritbebodyparts, khakis,orthenaturalenvironment.Thisdoesnotserveourdetectionpurposes well.Onepossibleexplanationforthisfactcouldbethathumanskincolorall happentohavesimilarhue,saturation,andintensity.Furthermore,theaveraging operationblendsandreducestheuniqueanddistinguishingfeaturesofall images,thustheseparationbetweenfacesisquitepoor. Wethenresortedtoadifferentmethodoffacetemplate.Realizingthatmuchof whatdistinguishesafacelieswithinitsfeature,wedecidedtoconvertall grayscaleimagestoblackandwhite,andthenuseedgeextractiontoextractthe importantfeatures,thencombinethemtoformatemplateface.Thefollowing facewasconstructedfromasamplingof25facestakenfromthetrainingset.

Fig.6.Edgeextractiontemplateface Weextractedtheedgesfirstbyconvertingtheimagesintobinaryimages,then usingtheCannyfiltertoextractedges.Furthermore,wenormalizethisimageby subtractingitsmeanfromitself,thusremovingtheelementscommontomost pixels.Theremainingcomponentshighlightthedifferencesinfeatures,aswellas reduceanupwardbiasinthecorrelationoutput.Anotheradvantageofthis approachisthatsinceanedgetemplateisasparsematrixwithmostlyzeros,it

haslowcorrelationexceptwhenitisexactlyalignedwiththetemplate.The followingfigureillustratesthelargeseparationbetweennon-correlatedand correlatedoutput,andthelargespikehelpstodistinguishthelocationsofdesired faces.

Fig.7.Edgeextractiontemplatecorrelation Anotheradvantageofusingedgesfordetectionisthattheyformclearly identiableregions,whichcanthenbecountedandusedtohelpdistinguish unknownimagesextractedfromthesegmentationmask.Forexample,the followingtwoimagesrepresenttheedgerepresentationofamansfaceand hand.Ifweweretoattempttoprocessthecolorimages,becausetheirsizeand colorcontentaresimilar,theircorrelationoutputwouldprobablynotbetoo different,thusintroducingtheriskoffalsepositivesormissinghits.However,if weweretoprocessthemintheedgedomain,thenwecanutilizethenumberof enclosedregionsforidentification.

Fig.8a&b.Edgesandenclosedregions

Usingbwlabel,wecanlabelandcounttheregions,anditshowsthattheface has43connectedregions,whilethehandonlyhas14.Sincefacesareingeneral morecomplexthanhands,clothing,orotherclutter,byusingtheregionstatistics oftheseedgesimages,itcanfurthersupplementouralgorithm. DetectionAlgorithm Threemaincriteriawereusedinourdetectionalgorithm.Weused: TheprecisealgorithmisdescribedinourMatlabimplementation,butwebasically usedthecorrelationlevelonthefirstpass,eliminatingobjectsthatareclearly unlikelytobefaces.Thenweusedheuristicssuchasdimensionsandregion countingtohelpdeterminewhetherzero,one,ormultiplefacesresideinagiven region.Ifanimagedoesnotqualifyasaface,wemoveontothenextobject.Ifa singlefaceexists,wereturnthelocationofthehighestcorrelationpeakasthe facesestimatedlocation.If,however,wesuspectedanimageformedbyoneor morepartiallyorwhollyjoinedfaces,wehavetothenuseamultiplefacial detectionroutine. DetectingMultipleFaceswithinaSingleFaceRegion Forthecaseofmultiplefaceswithinasingleregion,wecannotrelysolelyonthe singlecorrelationpeakdetectionusedforsinglefaces.Onceanestimateofthe expectednumberoffaceswithinaregionhasbeenmade,weattempttofindthat numberofpeaks(andcorrespondingfacelocations)inthecorrelationoutputfor thegivenregion. CorrelationUsingtheedgetemplateasameasureofhowcloselyit matchesouraveragefacialfeatures DimensionsUsingwidth,heightandarea,whethertheimageislarge enoughtocontainaface,observingaspectratios,etc. RegioncountingUsingtheregionnumberstatisticstohelpdifferentiate facesfromclutter.

Webeginbyfindingthelargestpeak(largestcorrelationoutputvalue)withinthe region.Wecountthisasthefirstfacelocation.Wethenexcludethesubregion of"average"faceradiusaroundthefirstpeaklocation,andcontinuesearching forthenextlargestpeakoutsideofthissub-region.Wecontinuethisprocessof subregionexclusionandpeakdetectionuntilthedesirednumberofpeak locationshavebeenfound.

Fig.9.Imageregionwith3faces

Fig.10a-c.a)Beginbydetectinglargestpeak(denotedbybluesquare) andexcludecirclearoundbluepeakfromfutherpeaksearch.b)Findnext largestpeakoutsideofcircle,andexcludecirclearoundgreenpeak.c) Continuepeaksearchandexclusiontofindthirdpeak

The"average"faceradiusisdetermineddynamicallyforeachregion,basedon thesizeoftheregionandnumberofexpectedfaces.Foranygivenregion,the averagefaceradiusisthemeanofthex-andy-dimensionsoftheregion,divided bythenumberoffaceswithintheregion.Wehavealsoincludeda"fudgefactor" (whichcanbemanuallyset)whichisaddedtotheaveragefaceradiusinorderto improvetheperformance.Thenominalvalueofthisfudgefactorwasdetermined bytrialanderrorwithdifferentimagefaceregions,derivedfromthetrainingset. Conclusion Onthewhole,ouralgorithmperformedwithmoderatetohighsuccesswhenused todetectfacesinthetrainingimages.Insomecases,westillmissobvious faceswithintheimage,whileinotherswedetectfalsepositives.Sincethe detectionalgorithmishighlydependantonfacesizeandedges,ithasparticular difficultywithpartiallyobstructedfacesandfaceswhichareveryclosetoeach other. Thereisstillroomforalgorithmrefinement,particularlyintheseparationof closely-spacedfacesanddetectingmultiplefaceswithinasinglecandidate region.Thegeneralimplementation,however,ofcolorsegmentationfollowedby edgeimagetemplatematching,appearstogivereasonablesuccessincorrect facedetection.