Scribd Logo
Upload

Welcome to Scribd!

  • Data Preparation: Missing Values (Excel)

    Uploaded by

    NumXL Pro
    307 views5 pages
    In this issue, we start with the sampling assumptions of the time series: equal spacing and completeness. Then we consider a time series with missing values and discuss how to represent them in Excel, with the aid of NumXL processing. Finally, we look at unequally spaced time series, how they come into existence, how they are related to the missing values scenario, and what to do with them. For the example spreadsheet and a tutorial video, please visit us at: http://bitly.com/HwJZVT

    Copyright

    © Attribution Non-Commercial (BY-NC)

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document
    In this issue, we start with the sampling assumptions of the time series: equal spacing and completeness. Then we consider a time series with missing values and discuss how to represent them in Excel, with the aid of NumXL processing. Finally, we look at unequally spaced time series, how they come into existence, how they are related to the missing values scenario, and what to do with them. For the example spreadsheet and a tutorial video, please visit us at: http://bitly.com/HwJZVT

    Copyright:

    Attribution Non-Commercial (BY-NC)
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    307 views5 pages

    Data Preparation: Missing Values (Excel)

    Uploaded by

    NumXL Pro
    In this issue, we start with the sampling assumptions of the time series: equal spacing and completeness. Then we consider a time series with missing values and discuss how to represent them in Excel, with the aid of NumXL processing. Finally, we look at unequally spaced time series, how they come into existence, how they are related to the missing values scenario, and what to do with them. For the example spreadsheet and a tutorial video, please visit us at: http://bitly.com/HwJZVT

    Copyright:

    Attribution Non-Commercial (BY-NC)
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    of 5

    #N/A

    Missing Values

    Thisissueisthefirstinaseriesofarticlesthatexplorethedatapreparationaspectoftimeseries analysis.Datapreparationisoftenoverlookedbyanalysts,butwebelieveitisavitalphasethatwieldsa vastinfluenceontheoverallanalysisandmodelingprocess.Thevastmajorityoftimeseriesand econometrictheoriesassumeinputtimeseriestobestationaryandhomogenous,withequallyspaced observationsandvaluesthatarepresentandreal.Inpractice,weoftenhandlesampleswithmissing values,unequallyspacedobservationspossibleoutliers,mean/variancedependency,restrictedvalues rangesandotherphenomena.Theaimofthisseriesofarticlesistoaddresseachoftheseproblemsand introducepracticalmethodstoovercomethem. Inthisissue,westartwiththesamplingassumptionsofthetimeseries:equalspacingand completeness.Thenweconsideratimeserieswithmissingvaluesanddiscusshowtorepresentthemin Excel,withtheaidofNumXLprocessing.Finally,welookatunequallyspacedtimeseries,howthey comeintoexistence,howtheyarerelatedtothemissingvaluesscenario,andwhattodowiththem.

    Sampling
    Thecommon(perfect)situationforatimeseriessampleisonethathasequallyspacedobservationsand presentvaluesforallpoints.Thisariseseitherbecauseobservationsaremadedeliberatelyateven intervals(continuousprocess),orbecausetheprocessonlygeneratesoutputsatsuchintervalintime (discreteprocess). Furthermore,thetimeunitforasamplingperiod(i.e.step)betweentwoconsecutiveobservationscan beeitherabsolute(e.g.daily,weekly,monthly,orannual),orbasedonaholidaycalendar(i.e.adjusted forweekendandholidays).Forexample,adailyfinancialtimeseriesofIBMstockclosingpricesisbased ontheNYSEholidayscalendar,soeachobservationistakenonaNYSEtradingday(open/close). Withrespecttotimeseriesmodelingandforecasting,itisnotimportantwhetherweuseabsolutetime orifweadjustforweekendsandholidays.Whatisimportantishowweinterprettheoutofsample dates,astheytooarebasedonthesamesamplingmethod. Next,letsexaminesomecaseswheretheinputtimeseriesisnotsoperfect.

    Tips&HintsMissingValues

    SpiderFinancialCorp,2012

    Issue 1: Missing values


    Insomesituations,oneormoreobservationdatesyieldinvalidormissing values.Thesevaluesaredesignatedasnotavalues,orNaNforshort.In Excel,NaNisidentifiedbythespecial#N/Arepresentation,andfewbuilt infunctionscanbeusedtodetect(e.g.NA(),ISNA(.),IFERROR(.),etc.)or ignorethem(e.g.MIN(.),MAX(.)),andotherfunctionsarenotsupportive. Intimeseriesanalysis,weoftenencountermissingvaluesphenomena, eitherintheoriginalrawtimeseriesorasaresultofatimeseriesoperator (e.g.lag,differencing,etc.). Q:Whatcanwedowithatimeserieswithmissingvalues? NumXLhastwosimplerules: 1. Themissingvaluesatthebeginningortheendofthetimeseries aresimplyignored. NumXLwilltruncatetheinputtimeseriestostartfromthe1stnonmissingvalueandendwith thelastnonmissingvalue. 2. Theintermediatemissingvaluesareconsideredseriousflawsintheinputtimeseries,and NumXLcantprocessthem. Theserulesbegthequestion:howdowehandlemissingintermediatevalues?

    Manytechniqueshavebeenproposedtohandletimeserieswithmissingdata,butwecansummarize theseproposalswithtwoprinciples:ignoreandinterpolate.

    Tips&HintsMissingValues

    SpiderFinancialCorp,2012

    IGNORE Theignoresolutionsimplydropsthemissingvaluefromthetimeseries.YoucanusetheNumXLRMNA (.)functionforthispurpose.However,youshouldapproachthissolutioncautiouslyasitaltersthe samplingofthetimeseriesitself. INTERPOLATE Theinterpolateapproachreplacesthemissingvalueswithinterpolatedvalues.Thereareseveral interpolationalgorithms:linear,polynomial,smoothing,spline,filtering,etc. Interpolationdoesnotchangethefrequencyofthesampling,butitmayaffecttheperceiveddynamics oftheunderlyingprocessifitisusedforseveralpointsinthetimeseries. NumXLcomeswithaninterpolationfunctionINTERPOLATEwhichsupportsfour(4)different interpolationalgorithms: Forward & Backward Flat Interpolation

    Linear & Cubic Spline Interpolation

    NOTE:TheInterpolatefunctiondiscardsallpointswithmissingvalues,sowecanusethefunction directlyontherawdatasetwithoutanyintermediatepreparation.

    Tips&HintsMissingValues

    SpiderFinancialCorp,2012

    Issue 2: Unevenlyspaced time series


    Unevenlyspacedtimeseriesarecommoninmanyreallifeapplicationswhenmeasurementsare constrainedbypracticalconditions.Theirregularityofobservationscanhaveseveralfundamental reasons.First,anyeventdrivencollectionprocess(inwhichobservationarecollectedwhensomeevent occurs)isinherentlyirregular.Second,insuchapplicationsassensornetworksoranydistributed monitoringinfrastructure,datacollectionisdistributedandcollectionagentscanteasilysynchronize withoneanother.Inaddition,thesamplingintervalsandpoliciesmaybedifferent.Finally, measurementscannotbemaderegularlyormayhavetobeinterruptedduetosomeevents(either foreseenornot).

    Note:Unliketheequalspacedtimeseriescase,intermediateobservationswithmissingvaluescanbe safelydroppedfromtheoriginalserieswithoutanylossofinformation,and,obviously,theresultant seriesisunevenlyspacedaswell. Manytechniqueshavebeenproposedtohandletimeserieswithmissingdata,whichinthelimitcanbe viewedasirregularlysampled. Indataanalysispractice,irregularityisarecognizeddatacharacteristic,andpractitionersdealwithit heuristically.

    Solution 1: Convert to equallyspaced time series


    1. IGNORE: IGNOREtheirregularityinthetimesandtreatthedataasifitwereregular. 2. RESAMPLE:RESAMPLEusingalowersamplingrate.Thereductionsimplifiestheproblemto onethathasalreadybeenthoroughlyanalyzedandforwhichmanyapproachesareavailable. Note:Forapricetimeseries,downsamplingrequirestakingthelastobservationinthenew sampleperiod.Forthisstrategyslogreturn,theresampledreturnisthecumulativereturnsof allperiodsintheoriginalsampleperiods. Tips&HintsMissingValues 4 SpiderFinancialCorp,2012

    3. INTERPOLATE:Interpolatetheintermediatemissingvaluesandconverttheseriestoonewith equallyspacedsamplingtimes.Whilethisisareasonableheuristicfordealingwithmissing values,theinterpolationprocesstypicallyresultsinasignicantbias(e.g.smoothingofthedata) thatchangesthedynamicsoftheprocess,thusthesemodelscannotbeappliedifthedatais trulyunequallyspaced. 4. Kernel Smoothing 5. Brownian Bridging:Anumberofauthorshavesuggestedusingcontinuoustimediusion processestofindmissingvalues.Inprinciple,tointerpolateamissingvalue,weassumea Brownianmotionbetweenthevaluesimmediatelypriortoandafterthenonmissing observations. Note:Asofthedateofthisissue,NumXLdoesnotsupporttheBrownianbridginginterpolationmethod.

    Solution II Use unequallyspaced time series Models


    Thesemodelsareslightlymorecomplexthantheirequallyspacedcounterpartmodels,andmanycanbe viewedasanextensionoftheequallyspacedtimeseriesmodels. Supposing Y (t ) isatimeserieswithirregularsampling,wecandecomposeitinto: Where

    Y (t ) a (t ) X (t )

    a (t ) isaslowlychangingdeterministicfunction(trendcomponent) X (t ) isarandomnoisecomponent

    Ingeneral,onecanonlyobserve Y (t ) ,ourfirstgoalistoestimatethedeterministiccomponentand forprocess X (t ) . Note:Asofthedateofthisissue,NumXLdoesnotsupportunevenlyspacedtimeseriesmodels.

    extracttherandomnoise X (t ) Y (t ) a (t ) ;oursecondgoalistofindasatisfactoryprobabilisticmodel

    Tips&HintsMissingValues

    SpiderFinancialCorp,2012

    You might also like