SOUND MODELING ANDMORPHING

PersonalresearchwebpageforKellyFitz,Ph.D.

Soundmorphingimpliesagradualtransformationfromonesoundintoanothersound,orthetransferoffeaturesofonesoundonto anothersound.Morphingcanbeusedtogeneratehybridtimbres,soundshavingcharacteristicsoftwodifferentmusicalinstruments(for example,theattackofanacousticbasswiththesustainandreleaseofabassoon,ortheonsetofacelloandthedecayandbeatingofa piano)andsoundsthatevolvefromonetoanother(onespeakerevolvingintoanother,oracatevolvingintoatrombone).Morphing hasbeenusedtosynthesizethevoicesofaliensandanimalsandavarietyofsoundeffectsinfilmsandpopularmusic.

MIXINGISN'TMORPHING
Manysoundfileeditors(suchasAudacity,showninFigure1below)provideafacilityforgraduallyfadingoutonesoundwhile graduallyfadinginanothersound.Couldthiscrossfadingmechanismbeusedformorphingonesoundintoanother?

Figure1.ScreenshotshowingacrossfadeoftwosoundsinAudacity ,availableat audacity.sourceforge.net .

Listentothisexampleofacrossfadebetweenafluteandatrumpet,andnoticewhathappensintheregionofthecrossfade.
Playexample.

Bothsourcesareclearlyaudibleduringthecrossfade.Mixingisnotthesameasmorphing.Inamix,allthesourcesoundsare(ideally) individuallyperceivableinthefinalsound.Thisiswhyanensemblerecordingcanbemadebymixingthesignalsfromseveral microphones. Inasoundmorph,onlyasinglesoundshouldbeaudiblethroughoutthetransformation.Inthisexample,thefluteandtrumpetdifferedin pitch,sotherewaslittlechancethattheywouldfuseintoasinglesoundinthemix.Listentothisexampleofacrossfadebetweenaflute andatrumpethavingthesamepitch.

Playexample.

Thissoundsslightlybetter,butbothsourcesarestillaudibleduringthecrossfade.Manysoundscanbedecomposedintotheir componentsinewaves.Thefrequencyandamplitudeofthesinewaves(calledharmonics,inthecaseofpitchedsounds)varyoverthe durationofthesound.Theoriginalsoundcanbereconstructedbyaddingtogetherallofthecomponentsinewaves. Evensoundshavingthesamepitchareindividuallyperceivableinamix,unlesstheindividualcomponentsinewaves(harmonics)are tunedtoexactlythesamefrequenciesinthetwosources.Listentothisexample,inwhichthefrequenciesoftheharmonicsoftheflute andtrumpethavebeensyntheticallyaligned.

Playexample.

Now,onlyasingleinstrumenttoneisaudiblethroughout.Perfectlytunedharmonicssuchasthesecanbegeneratedsynthetically(using acomputeroramusicsynthesizer,forexample),buttheydonotoccurnaturally.

A DDITIVESOUND MODELS
Inordertoartificiallyaligntheharmonicsofonerecordedsoundtothoseofanother,itisfirstnecessarytodeterminethefrequenciesof thoseharmonics.Fortunately,avarietyoftechniquesareavailablefordeterminingthetimevaryingfrequenciesandamplitudesofthe sinewavecomponentsofasound,andmodernmethodsworkwellformanysounds.Inparticular,weuseamethodcalledtime frequencyreassignmenttoobtainsharperestimatesofthetimeandfrequencybehaviorofthosecomponentsthanisavailableusing classicalanalysistechniques. AsoundanalyzedinthiswayisplottedinFigure2,below.Timeincreasesfromlefttoright,andfrequencyincreasesfrombottomto top.Darkareasshowregionsinwhichenergyisconcentrated,andtheharmoniccomponentsofthesoundareclearlyvisibleasdark, horizontalstripes.Inapitchedsound,asinthisexample,theharmoniccomponentsarespacedapproximatelyevenlyinfrequency.The lightergreysmudgesbetweenthedarkstripesshowregionsofnoisinessandnonharmonicenergyinthesound.

sinewavecomponentsofasound,andmodernmethodsworkwellformanysounds.Inparticular,weuseamethodcalledtime frequencyreassignmenttoobtainsharperestimatesofthetimeandfrequencybehaviorofthosecomponentsthanisavailableusing classicalanalysistechniques. AsoundanalyzedinthiswayisplottedinFigure2,below.Timeincreasesfromlefttoright,andfrequencyincreasesfrombottomto top.Darkareasshowregionsinwhichenergyisconcentrated,andtheharmoniccomponentsofthesoundareclearlyvisibleasdark, horizontalstripes.Inapitchedsound,asinthisexample,theharmoniccomponentsarespacedapproximatelyevenlyinfrequency.The lightergreysmudgesbetweenthedarkstripesshowregionsofnoisinessandnonharmonicenergyinthesound.

Figure2. Timefrequencyreassignedanalysisofacatmeowing. Time(measuredinseconds)increasesfromlefttoright, andfrequency(measuredinHertz)increasesfrom bottomtotop. Theharmoniccomponentsrepresentingmostofthe energyinthissoundarevisibleasdarkhorizontalstripes,spacedapproximatelyevenlyinfrequency.Thelightergrey smudgesbetweenthedarkstripesshowregionsofnoisinessandnonharmonicenergyinthesound.

Fromsuchdata,itispossibletoconstructamodelofasoundasacollectionofsinewaves.Asoundrepresentedinthiswayisplotted inFigure3,below.Theindividualcomponentsinthemodelaredepictedaswigglylines.AsinFigure2,above,timeincreasesfromleft toright,andfrequencyincreasesfrombottomtotop.Loudercomponentsareplottedindarkergrey,quieteronesinlightergrey.

Figure3. Additivesoundmodeldataforacatmeowing,plottedusing SpearbyMichaelKlingbeil, (availableat www.klingbeil.com/spear ).Time(measuredinseconds)increasesfromlefttoright,and frequency(measuredin Hertz)increasesfrombottomtotop.Theindividualcomponentsintherepresentationaredepictedaswigglylines.Louder componentsareplottedindarkergrey,quieteronesinlightergrey.

Itisasimplemattertoreconstructasoundfromsucharepresentationbysynthesizingtheindividualsinewavesandaddingtogetherthe results(forthisreason,theseareoftencalledadditivesoundmodels).Listentoatrumpetsoundbeingconstructedonesinewaveata time.

Playexample.

Listentothecatmeow,depictedinthefiguresabove,reconstructedfromadditivemodeldata.
Playexample.

Additivemodelsaffordintimatecontrolovertheevolutionofasynthesizedsound.Forexample,thesoundcanbestretchedor compressedintimebychangingtherateatwhichthesinewaveparameterschange.Listentotheeffectoftimestretchingonthecat meow(*) .

Playexample.

Listentotheeffectoftimestretchingonthesoundofaflexitone(apercussioninstrumentsometimesusedtomakeeeriesoundeffects). Theunmodifiedflexitonereconstructionisplayedfirst,followedbyastretchedversion.
Playexample.

Playexample.

Listentotheeffectoftimestretchingonthesoundofaflexitone(apercussioninstrumentsometimesusedtomakeeeriesoundeffects). Theunmodifiedflexitonereconstructionisplayedfirst,followedbyastretchedversion.
Playexample.

Thesoundcanbemodifiedinvariouswaysbymodifyingtheparametersofthesinewavemodel.Thepitchcanbealteredby systematicallymodifyingthefrequenciesofthesinewaves.Theindividualharmonicsofapairofsoundscanbeperfectlyaligned,asin thetrumpetandflutemixingexampleabove.Theadditivemodeldatacanalsobemorpheddirectly.

M ORPHINGWITHADDITIVESOUND MODELS
Soundmorphingcanbeperformedbyblendingtheparameters(amplitudesandfrequencies)ofthesinewavecomponentsinapairof additivemodels.Thesinewavefrequenciesdonotneedtobeperfectlyalignedinordertoperformthemorph.Rather,therawadditive modelcomponentscanbemorphedtoproduceanewmorphedmodelfromwhichthemorphedsoundissynthesized.Aloudsine waveinonesoundmorphedwithaquietsinewaveinanotheryieldsasinewaveofmediumamplitude.Ahighfrequencysinewavein onesoundmorphedwithalowfrequencysinewaveinanotheryieldsasinewaveofintermediatefrequency.Thisparameterblending operationisdepictedinFigure4,below,forasinglemorphedsinewave.

Figure4.Plotsshowingmorphingofamplitude(topplot)andfrequency(middleplot)parametersforonecomponentina morph.Thered'x'showstheparametersfortheonesourcesound,theblue'+'showstheparametersfortheothersource. Themorphedparametersareshowninpurplecircles.

Inthisfigure,thered'x'showstheparametersforthesoundatthestartofthemorph.Theblue'+'showstheparametersforthesound attheendofthemorph.Themorphedparametersareshowninpurplecircles.Themorphmakesagradualtransitionfromthe parametersofthefirstsoundtothoseofthesecond. Listentothisexampleofaviolatonemorphedwithaflutetone.Thepitchesofthetwotonesareslightlydifferent,soitiseasytohear thetransformationfromthelowerviolatimbretothehigherflutetimbreasthemorphprogresses.

Playexample.

Listentothisexampleofapianotonemorphedwithatemplebell.Thetemplebellisnotpitched,soitisnotpossibletoalignthe harmonics(asinthetrumpetfluteexample)withoutdramaticallyalteringthesoundofthebell.Nonetheless,bycarefullypairingthe componentsinthetwoadditivemodels,itispossibletoachieveapleasingmorphbetweenthetwosounds.

Playexample.

Listentothisexampleofamalesingermorphedwithatrombone,playedusingatechniquecalled"fluttertoungue".Thesingerandthe trombonearepresentedindividually,followedbyasequenceofmorphs.
Playexample.

R EHABILITATINGTHEWORLD'SWORSTSOUNDS
TrevorCox,aprofessorofacousticengineeringatSalfordUniversity,conductedayearlongwebbasedstudyofpeople'sperception ofhorriblesounds,andtodeterminewhatmakescertainnoisessoobjectionable.YoucanreadaboutthestudyanditsresultsonThe Guardianwebsite,orinhispaper"Theeffectofvisualstimulionthehorriblenessofawfulsounds,"inAppliedAcoustics,availablefrom ScienceDirect,orvisitwww.sound101.org. WehavecollectedafewofthesoundsusedinProfessorCox'sstudy,andattemptedto"rehabilitate"thembymorphingthemwith other,lesshorriblesounds,or,atleast,moreentertainingones.Someexamplesareincludedhere. Hereisanexampleofacryingbabymorphedintoapiano.
Playexample.

Hereisanexampleofatrumpetmorphedinacryingbaby(notsurewhyyou'dwanttodothat).

other,lesshorriblesounds,or,atleast,moreentertainingones.Someexamplesareincludedhere. Hereisanexampleofacryingbabymorphedintoapiano.
Playexample.

Hereisanexampleofatrumpetmorphedinacryingbaby(notsurewhyyou'dwanttodothat).
Playexample.

Hereareexamplesofseveralmorphsperformedusingthesoundofameowingcat (*) .
Playexample.

Notsurprisingly,wefoundthatmorphingtwoawfulsounds(thecryingbabyandthemeowingcat,inthiscase)doesnotimproveeither ofthem.
Playexample.

Inthislastexample,ametalicscrapingsoundistransformedintoaflexitone(apercussioninstrumentsometimesusedtomakeeerie soundeffects).Twoslightlydifferentmorphsarepresented.
Playexample.

HOTTOPICSINMUSICAL ACOUSTICS
SomeofthismaterialwaspresentedontheHotTopicssessionatthe153rdMeetingoftheAcousticalSocietyofAmericainSaltLake City,UtahinJune,2007.Youcanlistentosomeofthemediacoverageofthateventatthesesites:
l l l

"SoundMorphing"ScienceUpdate,AmericanAssociationfortheAdvancementofScience. "ManWorkstoImprovetheWorstSoundsintheWorld"KSLTelevision&Radio,SaltLakeCityUT "HardrockaufderVenusAkustikersimulierenKlngeauffremdenWelten"Deutschlandradio(Germannationalradio)

FORMORE INFORMATION
AllsoundmorphsandsynthesespresentedherewerecreatedusingtheopensourceLorissoftwareforsoundanalysis,synthesis,and manipulation.VisittheLoriswebsitetoreadmoreaboutsoundmorphingusingtheLoris,software,includingexamplecodeandsound morphingdemonstrations. ThisresearchisconductedbyKellyFitzincollaborationwith LippoldHakenfromtheDepartmentofElectricalandComputer EngineeringattheUniversityofIllinoisinUrbanaChampaign.
GotoKelly'shomepage.

(*)

Noanimalswereharmedintheconstructionofthiswebsite.

ThispageisvalidHTML4.01Strict.
Lastupdate:28August2007