AruneshMittal

HannahPiper

IndependentComponentAnalysisforBlindSourceSignalSeparation

Introduction

Independentcomponentanalysis(ICA)isanimportantcomputationaltechniqueusedtodecompose
nongaussian,statisticallyindependentsignalsintoadditivesubcomponents.ICAdecomposesa
multidimensionalsignalintocomponentsthataremaximallyindependentintermsoftheirnegativeentropy
andmaximallyindependentstatistically.Inourpaper,wediscussthemathematicsbehindthecomputation
ofIndependentComponentsandinadditionwewillpresentarealworldexampletoillustrateits
effectiveness.WedemonstratehowICAcanbeusedtoblindlyseparatetwosignals,withproperties
describedabove,intoindependentcomponents.Wewilldemonstratehowdifferentvoicesinanaudiosignal
canbeseparatedoutusingICA.

BlindSourceSeparation

Fig.1SchematicoftheCocktailPartyProblem

IndependentComponentAnalysis(ICA)isoftenexplainedincoursesusingthecanonicalBlindSource
SeparationproblemknownasthetheCocktailPartyProblem[Fig.1].TheCocktailPartyProblemtermwas
coinedbyColinCherryin1953todescribetheproblemofseparatingoneperson'svoiceamidstmanyother
peoplehavingaconversationatacocktailparty.OnlyinthelasttwodecadeshastheICAtechniqueevolved
enoughtomaketheCocktailPartyProblematractableproblem.TheCocktailPartyProblem(CPP)isan
instancefromalargerdomainofproblemsaimingatseparatingmixedsourcestheseproblemsaretermed
blindsourceseparation(BSS)problems.CPP,becauseoftheassumptionsimpliedintheproblem,isa
problemthatarguablybelongstoasubsetofBSSproblemsthatcanbesolvedusingICA.Manyproblems
fromtheBSSdomaincannotbesolved,however,thisisanactiveareaofresearchwithinthecomputer
science,appliedmathandstatisticscommunities.

CPPisagreatexamplebecauseofthephysicalunderpinningsofsoundi.e.soundaddslinearly.The
recordedamplitudeofasoundwavebyamicrophoneistheadditivelinearsumofthepressurewaves
emanatingfromthesources.


ProblemIntuition

TobuildintuitiontowardsICA,westartbylookingatsomedatarecordedbytwomicrophones.Lets1bea
signalgeneratedbythemusicands2isgeneratedbyaperson'svoice.Thesignalsaresimultaneously
recordedbytwomicrophones.Thetwomicrophonesrecordaunique,linearsummationofthetwosounds.
Thelinearweightsforeachmicrophonearea1,b1anda2,b2,respectively.Theweightsreflectthe
proximityofthesignalsourcetotherespectivemicrophones.Ourgoalistorecovertheoriginalsourcess1
ands2.

WecanplotthedataandvisualizeittogetanintuitivefeelforwhatanIndependentComponent(IC)is.We
plotthesimultaneouslyrecordedamplitudesfrommicrophones1and2onthexandyaxis,respectively.If
therewasonlyonesignalsources(music),allsampleswouldlieonthevectora=(a1,a2),wherea1,a2
signifytheproximityofthemusictomicrophones1and2,respectively.Similarly,themiddleplotdepictsall
datapointsarisingiftheonlysignalbeingrecordedwasthevoice.Ifboththesignalswerepresent,given
thatthesignalisadditive,wewouldexpecttoseetheplotintherightpanelthelinearsumofboth
sources.

Fig.2VisualizationofIndependentComponentsandResultingAdditiveSignal

Thevectorsaandbaretheindependentcomponentsofthedata.Independentcomponentanalysisallows
ustoextracttheindependentcomponentsfromthedatagivenonlythesumofthecomponents.Inthe
followingsectionswewilldiscussthetechniquebywhichwecandecomposealinearlyadditivesignalinto
itsICs.

Nowthatwehavelookedattheprocessofgettingtothemixedsignalfromtheindependentcomponents,
wecannowdiscusstheextractionoftheICsgivenonlythesumoftheICs.Recallthatsoundisidealforthe
CPPbecausethesignalrecordedbythemicrophoneislinearlysummed.

Inthefollowingfigure(Fig.3),welookattherawdataaswellastheindependentcomponentsoftheraw
data.NotethatthedirectionoflargestvarianceisnotalwaysthesameasthatoftheICthisisan
importantdistinctiontonotefromthetechniquePCA.

MathematicalMethods

Fig.3ICsofselectedrawdata

Therawdataxisassumedtobealinearmixtureoftheunderlyingsources.ThisholdstruefortheCPP(as
thesignal)isalinearmixture.

xisoftheform:

Aisknownasthemixturematrix.Itisanunknown,invertible,squarematrixthatlinearlymixesthe
componentsofthesources.
Inthesimpleexamplediscussedabove,withtwosources(musicandvoice)thetheentriesoftheAmatrix
aretheweightsgiventothesignals(proximitytothesource).Thiscanbewrittenas:

NotethatinanICAproblemwecannotsimplyfilltheentriesoftheAmatrixusingtheweightsbecausethe
weightsareunknowninfact,thegoalofICAistofindthemixingmatrixA.Giventherelationshipbetween
A,sandx,giventherawdatax,andifweknowA,wecanseparatethesources.Thiscanbe
mathematicallyinterpretedasthefollowing:

Notetheuseofshat.ThisisbecausewecanonlyapproximatethesourcesignalasWandsareunknown
noclosedformsolutionexists.Theproblemisoftheformab=constant,where,aandbareunknown.This
anunderconstrainedproblembecausethenumberofunknownsexceedsthenumberofobservations.But,
notallhopeislost,itturnsoutthatthestatisticsoftheobserveddataxinconjunctionwithinformation
theorycanbeusedtofindasolutionforthisproblem.

InsteadoftryingtosolveforbothAands.WefirstdetermineA.Wedecomposeourmixturematrixusing
singularvaluedecomposition(SVD)

Geometrically,SVDdecomposesthematrixintointothreelinearoperations[Theanimationonthewikipage
[3]isdoesanexcellentjobconveyingthegeometricalinterpretationofthedecompositioncomponentsV,
andU]

ArotationV
Astretchalongtheaxes
AsecondrotationU.

WeestimateAandWfromtheSVD.Weknowthattheinverseofarotationmatrixisitstransposeandthat
becauseAisinvertible,theinverseofexists.WecanwriteWas:

WesolveforWbyexaminingthecovarianceofthedataxinordertocalculateUand,andusethenotion
ofstatisticalindependenceofstosolveforV.

ComputingUand:

Covarianceofamatrixistheexpectedvalueoftheouterproductofindividualdatapoints<xx>.Recall,for
ICAoftheCPPwemadetheassumptionthatthesourcesarestatisticallyindependent.Thisimpliesthat
covarianceofthesourcesiswhitenedi.e.thesourcesignalsareuncorrelatedandallsourceshavevariance
1.Thisalsoimpliesthatthecovarianceistheidentitymatrix.

Given,

Frombasiclinearalgebraicproperties,thecovarianceofthedatacanbeexpressedintermsofthe
underlyingsourcesas:

Recall,thatweassumedthatifthesiswhitenedthisimplies:

Also,itisknownthatthefollowingpropertyholdsfororthogonalmatrices:

Thisgivesus:

Thistakeonthefamiliarofacanonicaleigenvectoreigenvaluedecomposition:

ThecovarianceofthedatainexpressedintermsofadiagonalmatrixDbetweentwoorthogonalmatricesE.
DisadiagonalmatrixofassociatedeigenvaluesandEisanorthogonalmatrix.Diagonalizationofa
symmetricmatrixintoeigenvectorsisauniquesolutioni.e.nootherbasiscandiagonalizeasymmetric
matrix.

Derivedearlier:

Bysuperpositiongivesup:

DaretheeigenvaluesandEtheeigenvectorsofthecovarianceofthedatax.Allweneedisthefirstrotation
matrixV.

ComputingV:

Wecanwhitenourobserveddatax.Thisgivesus:

Thereforwecanwriteourapproximationofsas:

WeexploitthestatisticsofindependencetoidentifyV.Thisstepisthereasonoursourcesignalsneedto
bestatisticallyindependent.Thecovariancematrixisameasureofthelineardependencebetweenallpairs
ofentries.Itcapturesthesecondordercorrelationsinthedata.Tworandomvariablesaandbare
independentif:

ThesubclassofBSSPwearetryingtosolveallowsustoassumethatallsourcesarestatistically
independentthisgivesus:

WeneedtofindarotationmatrixVsuchthatshatisstatisticallyindependent.Itcanbeeasilyshownthat
thewhiteningtransformofthedataremovesallsecondordercorrelations,theunknownmatrixVmustthen
minimizehigherorderdependenciesbetweenthesources.Thisisequivalenttoincreasingthe
independence.HencetocomputeVweuseadifferentmetrictomeasuredependencebetweenpairof
signalsources.Thismetricisderivedfromthefieldofinformationtheory.

Themetric,mutualinformation,measuresthedepartureoftwovariablesfromstatisticalindependence.
Multiinformation,ageneralizationofmutualinformation,measuresthestatisticaldependencebetween
multiplevariablesandisgivenby:

I(y)isacontinuousnonnegativefunctionthatthatonlygives0ifallvariablesarestatisticallyindependent.
ThemultiinformationisafunctionoftheentropyHofthedata.Whichisgivenby:

Themultiinformationisthedifferencebetweenthesumofentropiesofthemarginaldistributionsandthe
entropyofthejointdistribution.Whichis:


Theproblemthenturnsintoanoptimizationproblemoftheform:

Thisminimizationcanbeperformedusingmanypopularoptimizationalgorithmssuchasgradientdescent
orsimulatedannealing.

Conclusion

AlthoughmanyaspectsofICA,PCA,andBlindSourceSeperationwerediscussedthroughoutthispaper,
thereisstillamultitudeofresearchlefttocompletelyunderstandthesetopics.TheCocktailPartyProblem
isjustoneofthemanyexamplesthatshowstheimportanceofBSS.It'samazingtothinkthatasimple
mathematicalmethod,suchastheuseofeigenvectorsandmatrices,canbeutilizedtosolvesuchcomplex
processesofsignalanalysis.

References

[1]http://www.mit.edu/~gari/teaching/6.555/LECTURE_NOTES/ch15_bss.pdf
[2]http://www.cis.hut.fi/projects/ica/cocktail/cocktail_en.cgi
[3]http://en.wikipedia.org/wiki/Singular_value_decomposition
[4]Shlens,Jonathon."ATutorialonIndependentComponentAnalysis."arXivpreprintarXiv:1404.2986
(2014).