Professional Documents
Culture Documents
BayesianStatistics
Intheclassical,orfrequentistapproachtoStatisticsweconsideraparameterafixedalthough
unknownquantity.ArandomsampleX1,..,Xnisdrawnfromapopulationindexedbyand,
basedontheobservedvaluesinthesample,knowledgeaboutthetruevalueofisobtained.In
theBayesianapproachisconsideredaquantitywhosevariationcanbedescribedbya
probabilitydistribution(calledapriordistribution),whichisasubjectivedistribution
describingtheexperimentersbeliefandwhichisformulatedbeforethedataisseen.Asampleis
thentakenfromapopulationindexedbyandthepriordistributionisupdatedwiththisnew
information.Theupdateddistributioniscalledtheposteriordistribution.Thisupdatingisdone
usingBayes'formula,hencethenameBayesianStatistics.
Let'sdenotethepriordistributionby()andthesamplingdistributionbyf(x|),thenthejoint
pdf(pmf)ofXandisgivenby
f(x,)=f(x|)()
themarginalofthedistributionofXis
m(x)=f(x|)()d
andfinallytheposteriordistributionistheconditionaldistributionofgiventhesamplexand
isgivenby
(|x)=f(x|)()/m(x)
Wecanwritethisalsointermsofthethelikelihoodfunction:
(|x1,..,xn)=L(|x1,..,xn)()/m(x)
Example:Youwanttoseewhetheritisreallytruethatcoinscomeupheadsandtailswith
probability1/2.Youtakeacoinfromyourpocketandflipit10times.Itcomesupheads3
times.Asafrequentistwewouldnowusethesamplemeanasanestimateofthetrue
probabilityofheads,pandfind =0.3.
ABayesiananalysiswouldproceedasfollows:letX1,..,XnbeiidBer(p).ThenY=X1+..+Xn
isBin(n,p).Nowweneedaprioronp.Ofcoursepisaprobability,soithasvalueson[0,1].
Onedistributionon[0,1]weknowistheBeta,sowewilluseaBeta(,)asourprior.
Remember,thisisaperfectlysubjectivechoice,andanybodycanusetheirown.Thejoint
distributiononYandpisgivenby
http://academic.uprm.edu/wrolke/esma6661/bayes.htm 1/6
6/12/2015 BayesianStatistics
whichisknownasthebetabinomialdistribution.
Notethatthat(Y,p)isarandomvectorwhereonecomponentiscontinuous(p)andtheotheris
discrete(Y).Soherewearecombiningapdfwithapmf.Itturnsoutthatthisisok.
Theposteriordestributionofpgivenyisthen
Ofcoursewestillneedto"extract"someinformationabouttheparameterpfromtheposterior
distribution.Oncethesamplingdistributionandthepriorarechosen,theposteriordistribution
isfixed(eventhoughitmaynotbeeasyorevenpossibletofinditanalytically)buthowwe
proceednowiscomp[letelyopenandthereareingeneralmanychoices.Ifwewanttoestimate
panaturalestimatoristhemeanoftheposteriordistribution,givenhereby
B=(y+)/(++n)
Thiscanbewrittenas
andweseethattheposteriormeanisalinearcombinationofthepriormeanandthesample
http://academic.uprm.edu/wrolke/esma6661/bayes.htm 2/6
6/12/2015 BayesianStatistics
mean.
Howaboutourproblemwiththe3headsinthe10flips?Well,wehavetocompletelyspecify
thepriordistribution,thatiswehavetochooseand.Thechoicedependsagainonourbelief.
Forexample,ifwefeelstronglythatthiscoinisjustlikeanyothercoinandthereforereally
shouldbeafaircoinweshouldchoosethemsothatthepriorputsalmostallitsweightat
around1/2.Forexamplewith==100wegetE[p]=0.5andV[p]=0.0016.Then B=
(3+100)/(100+100+10)=0.4905isourestimatefortheprobabilityofheads.Clearlyforsucha
strongpriortheactualsamplealmostdoesnotmatter,Forexamplefory=0wewouldhave
found B=0.476andfory=10itwouldbe B=0.524.
Maybewehaveneverevenheardtheword"coin"andhavenoideawhatonelookslike,let
alonewhatprobabilityof"heads"mightbe.Thenwecouldchoose==1,thatistheuniform
distribution,asourprior.Reallythiswouldindicateourcompletelackofknowledgeregarding
p.(Thisiscalledanuninformativeprior).Nowwefind B=(3+1)/(1+1+10)=0.3,whichis
justthesamplemeanagain.
inbayescoinwithwhich==1westudytheeffectofthesamplesizeontheestimateofp.
inbayescoinwithwhich==2westudytheeffectofalpha=betaontheestimateofp.Alarger
alphameansapriormoreconcentratedaround1/2.
ExamplesayX~Bin(n,p),pknown.Again,aBayesiananalysisbeginswithaprioronn.Now
n=1,2,..andsoapriorisanysequencea1,a2,..withai0andai=1
Ifwewanttofindanestimatefornwecanuseforexamplethemode,thatisthenwhichhas
thelargestposteriorprobability.
Herearesomespecificexamples:sayweobservex=217andweknowp=0.37.Also
a)weknowonlythatn750,sowechooseai=1/750if1i750,0otherwise,
http://academic.uprm.edu/wrolke/esma6661/bayes.htm 3/6
6/12/2015 BayesianStatistics
bayes.bin.n(217,0.37,rep(1/750,750))
b)weknownismostlikely500withastandarddeviationof50,
bayes.bin.n(217,0.37,dnorm(1:750,500,50))
c)weknowthatn750andthatnisamultipleof50,
bayes.bin.n(217,0.37,ifelse(c(1:750)%%50,0,1))
d)weknowthiswasoneofthefourexperimentswedid,withn=510,525,550or575
a=rep(0,750)
a[c(510,525,550,575)]=1
bayes.bin.n(217,0.37,a)
TheBigQuestion:BayesianorFrequentist?
ShouldyoubeaBayesian?
BayesianStatisticshasalotofgoodfeatures.Tobeginwith,itanswerstherightquestion,
P(Hypothesis|Data).Thereareothersaswell:
DecisionTheory
Thereisabranchofmathematicsconcerendwithdecisionmaking.Itisconceptuallyavery
usefulandimportantone:
Shouldyoubuyanewcar,orkeeptheoldoneforanotheryear?
Shouldyouinvestyourmoneyintothestockmarketorbuyfixedinterestbonds?
Shouldthegovermentlowerthetaxesorinsteadusethetaxesfordirectinvestments?
Indecisiontheoryonestartsoutbychoosingalossfunction,thatisafunctionthatassignsa
value(maybeintermsofmoney)toeverypossibleactionandeverypossibleoutcome.
ExampleYouareofferedthefollowinggame:youcaneithertake$10(let'scallthisactiona),
oryoucanflipacoin(actionb).Ifthecoincomesupheadsyouwin$50,ifitcomesupheads
youloose$10.Sotherearetwopossibleactions:takethe$10orflipthecoin,andthree
possibleoutcomes,youwin$10,$50orloose$10.Weneedavalueforeachcombination.One
obviousansweristhisone:
L(a)=10,L(b,"heads")=50,L(b,"tails")=10
Butthereareotherpossibilities.Sayyouareinabar.Youalreadyhadfoodanddrinksandyour
tabis$27.Nowyounoticethatyouonlyhave$8inyourpocket(andnocreditcardetc.)Now
ifyouwinorloose$10itdoesn'tmatther,eitherwayyoucan'tpayyourbill,anditwillbevery
embarrassingwhenitcomestopaying.Butifyouwin$50,youarefine.Nowyourloss
http://academic.uprm.edu/wrolke/esma6661/bayes.htm 4/6
6/12/2015 BayesianStatistics
functionmightbe:
L(a)=0,L(b,"heads")=1000,L(b,"tails")=0
Thenextpieceindecisiontheoryisthedecisionfunction.Theideaisthis:let'scarryoutan
experiment,anddependingontheoutcomeoftheexperimentwechoseanaction.
Shouldyouinvestyourmoneyintothestockmarketorbuyfixedinterestbonds?
Let'sdothis:wewaituntiltommorrow.IftheDowJonesgoesup,weinvestinthestock
market,otherwiseinbonds.
Indecisiontheoryadecisionruleiscalledinadmissibleifthereisanotherrulethatisbetterno
matterwhattheoutcomeoftheexperiment.Obviouslyitmakesnosensetopickan
inadmissiblerule.
Sowhat'stheconnectiontoBayesianStatistics?FirstthereareBayesiandecisionrules,which
combinepriorknowledgewiththeoutcomeoftheexperiment.
basedonthemovementoftheDowJonesinthelastyear,Ihaveacertainprobabilitythatit
willgoupoverthenextyear.
Nowthereisafamoustheorem(thecompleteclasstheorem)thatsaysthatalladmissiblerules
areBayesiandecisionrulesforsomeprior.
Optimality
Obviouslywhenwedosomethingitwouldbenicetodoitinanoptimal(best)way.Itturnsout
thatinBayesianstatisticsitisoftenpossibletoshowthatacertainmethodisbest,betterorat
leastasgoodasanyother.
WhytobeaFrequentist
Becauseyouhatepriors
orbettertosayyoudon'tlikethesubjectivityintroducedbypriors.InBayesianstatisticsitis
entirelypossiblethattwoScientistswhohavethesamedataavailableandusethesamemethod
foranalysiscometodifferentconclusions,becausetheyhavedifferentpriors.
BecauseFrequentistmethodswork
FormostofthehistoryofStatistics,thatisfromabout1900toabout1960,therewas
(essentially)onlyFrequentistStatistics.Inthistime(andsince)manymethodshavebeen
developedthatworkedverywellinpractise.ManyofthoseturnouttobealsoBayesian
methodswhentherightpriorisused,butnotall!
http://academic.uprm.edu/wrolke/esma6661/bayes.htm 5/6
6/12/2015 BayesianStatistics
Exampleoneofthemostusefulmodernmethods,calledtheBootstrap,isapurelyFrequentist
methodwithnoBayesiantheory.(ActuallythereissomethingcalledtheBayesianbootstrap,
butitisnotthesameastheclassicalbootstrap)
ExampleAstandardtechnicinregressionistostudytheresiduals.This,though,violatesthe
likelihoodprincipleandisthereforenotalloweduntertheBayesianparadigm.Actually,most
Bayesiansstudytheresidualsanyway.
Simplicity
Evenfortheeasiestproblems("estimatethemeanGPAofstudentsattheColegio")aBayesian
analysisalwaysseemstobecomplicated(chooseapriorandalossfunction,calculatethe
posterior,extracttheestimatefromtheposterior,trytodoallofthisoptimally)Frequentist
solutionsareoftenquickandeasy.
So?BeBoth!
http://academic.uprm.edu/wrolke/esma6661/bayes.htm 6/6