Bayesian Statistics Lecture 1 Motivation

Bayesian Statistics
Lecture 1
Motivation
Alvaro Mauricio Montenegro Daz

August, 2011

Alvaro Mauricio Montenegro Daz

Bayesian Statistics Lecture 1 Motivation

Bayesian Statistics Lecture 1 Motivation

Table of contents

Brief History

Bayesian-frequentist controversy

Motivating Examples.

Alvaro Mauricio Montenegro Daz

Bayesian Statistics Lecture 1 Motivation

Bayesian Statistics Lecture 1 Motivation

Brief History

Table of contents

1 Brief History
2 Bayesian-frequentist controversy
3 Some basic Bayesian models

Alvaro Mauricio Montenegro Daz

Bayesian Statistics Lecture 1 Motivation

Bayesian Statistics Lecture 1 Motivation

Brief History

Classical Statistics
Classical Age
1 Probability is old. Several hundreds of years ago, most mathematicians

supported by rich noblemen advise them on how to maximize their

winnings in games of chance.
2 Statistics is young. Linear regression first appeared in the work of

Francis Galton in the late 1800s. Karl Pearson added correlation and
goodness-of-fit measures around the turn of the last century.
3 Statistics blossomed. Between 1920 and 1930 R.A. Fisher developed the

notion of likelihood of general estimation. Jerry Neyman and Egon

Pearson developed the basis for classical hypothesis testing.
4 The war was an engine. A flurry of research activity was energized by

the World War II.

Alvaro Mauricio Montenegro Daz

Bayesian Statistics Lecture 1 Motivation

Bayesian Statistics Lecture 1 Motivation

Brief History

Bayesian Statistics
Bayesian Age
1 Bayesian methods are much older. The original paper dating in 1763

by the Rev. Thomas Bayes, a minister and amateur mathematician.

2 Laplace and Gauss and others were interest ed in the 19th century.
3 Bayesian approach was ignored or actively opposed in the early of 20th

century.
4 Several non-statisticians as Jeffreys (a physicist) and Bolye (an

econometrician), continued to lobby on behalf of the Bayesian ideas,

(which they referred as inverse probability.
5 Bayesian statistics is here. Beginning around 1950 statisticians as

Savage, Finetti, Lindley, and many others began advocating Bayesian

methods as remedies for certain deficiencies in the classical approach.

Alvaro Mauricio Montenegro Daz

Bayesian Statistics Lecture 1 Motivation

Bayesian Statistics Lecture 1 Motivation

Bayesian-frequentist controversy

Table of contents

1 Brief History
2 Bayesian-frequentist controversy
3 Some basic Bayesian models

Alvaro Mauricio Montenegro Daz

Bayesian Statistics Lecture 1 Motivation

Bayesian Statistics Lecture 1 Motivation

Bayesian-frequentist controversy

Example 1. Classical point of view

Example 1. Frequentist interval estimation
iid

Suppose Xi N(, 2 ), i = 1, , n. We desire a 95% interval estimate for

the population mean .
Provided n sufficiently large (say bigger than 30), a classical approach would
use the confidence interval.

(x) = x 1.96s/ n
Interpretation. Before any data are collected the probability that the interval
contains the true value is 0.95, for any value of and 2 .
After collecting the data and computing (x), the interval contains the true
or it does not; its coverage probability is not 0.95, but either 0 or 1.
It is not correct to say that the true has a 95% chance of falling in (x).
Alvaro Mauricio Montenegro Daz

Bayesian Statistics Lecture 1 Motivation

Bayesian Statistics Lecture 1 Motivation

Bayesian-frequentist controversy

Example1. Bayesian point of view

Bayesian interval estimation
Bayesian confidence intervals are called credibility sets. They are free of the
awkward frequentist interpretation.
In the Bayesian, approach, the data are first observed. Then, conditional on the
observed data the posterior distribution p(|x) is obtained.
A 95% credibility interval can be obtained by taking the 2.5 and 97.5
percentiles of the posterior distribution.
Interpretation. The actual value of has a 95% chance of falling in
confidence interval. However, there is a price to pay. It is necessary to specify a
(possible vague) prior distribution for .

Alvaro Mauricio Montenegro Daz

Bayesian Statistics Lecture 1 Motivation

Bayesian Statistics Lecture 1 Motivation

Bayesian-frequentist controversy

Example 2. Test of hypothesis

Frequentist Test of hypothesis

Suggested by Lindley and Phillips(1976).
Suppose in 12 independent toses of a coin, we observe 9 heads and 3 tails.
We wish to tests the hypothesis H0 : = 1/2 versus the alternative Ha > 1/2,
where is the true probability of heads.

Alvaro Mauricio Montenegro Daz

Bayesian Statistics Lecture 1 Motivation

Bayesian Statistics Lecture 1 Motivation

Bayesian-frequentist controversy

Example 2. Test of hypothesis

Possible models
Given only this information, two choices for the sampling distribution emerge:
1 Binomial: The number n = 12 tosses was fixed beforehand, and the

random quantity X was the number of heads observed in the n toses.

Then the likelihood function is given by
!
!
n x
12 9
1x
=
L1 () =
(1 )
(1 )3
x
9
2 Negative binomial: Data collection involve flipping the coin until the third

tail appeared. Here X is the number of heads required to complete the

experiment, so that X NegBin(r = 3, ), with likelihood
!
!
r +x 1 x
11 9
1x
L2 () =
(1 )
=
(1 )3
x
9

Alvaro Mauricio Montenegro Daz

Bayesian Statistics Lecture 1 Motivation

Bayesian Statistics Lecture 1 Motivation

Bayesian-frequentist controversy

Example 2. Test of hypothesis

Computations
The p-value to the rejection regionReject H0 if X c are computed as: Using
the binomial likelihood we have
!
12
X
j
1 = P= 1 (X 9) =
j (1 )12j = 0.075,
2
j

12
j=9
Using the negative binomial likelihood we have
!

X
2+j j
(1 )3 = 0.0325.
2 = P= 1 (X 9) =
2
j
j=9
Uppss!!
Using the usual type I error level = 0.05, the two model assumptions lead to
two different decisions. The problem is that there is not sufficient information
in the problem setting to help us in the selection of the correct model.
Alvaro Mauricio Montenegro Daz

Bayesian Statistics Lecture 1 Motivation

Bayesian Statistics Lecture 1 Motivation

Some basic Bayesian models

Table of contents

1 Brief History
2 Bayesian-frequentist controversy
3 Some basic Bayesian models

Alvaro Mauricio Montenegro Daz

Bayesian Statistics Lecture 1 Motivation

Bayesian Statistics Lecture 1 Motivation

Some basic Bayesian models

Bayes theorem
The most basic Bayesian model has three stages:
1 Likelihood specification Y f (y |).
2 Prior specification p(), where Y or can be vectors. p() can be

completed specified (the simplest case) or not.

3 Computation of the posterior distribution

The posterior distribution of is given by

(|y ) =
where

f (y |)p()
,
m(y )

Z
m(y ) =

f (y |)p()d

is the marginal distribution of Y . This is the Bayes Theorem.

Alvaro Mauricio Montenegro Daz

Bayesian Statistics Lecture 1 Motivation

Bayesian Statistics Lecture 1 Motivation

Some basic Bayesian models

Distributions in the Bayesian perspective.

Alvaro Mauricio Montenegro Daz

Bayesian Statistics Lecture 1 Motivation

Bayesian Statistics Lecture 1 Motivation

Some basic Bayesian models

A Gaussian/Gaussian (normal/normal) model

Y N(, 2 ),

2 known.

In this case both the prior and the likelihood (the observational model) are
Gaussian distributions, namely,
N(, 2 )
Y | N(, 2 ),
thus, the marginal distribution of Y is N(, 2 + 2 ), and the posterior
distribution of is Gaussian with mean and variance given by
E [|Y ] = B + (1 B)Y
Var [|Y ] = (1 B) 2 ,
where
B = 2 /( 2 + 2 ).
Alvaro Mauricio Montenegro Daz

Bayesian Statistics Lecture 1 Motivation

Bayesian Statistics Lecture 1 Motivation

Some basic Bayesian models

A beta/binomial model
Y Bin(n, )
This is a model which could be used in the problem of test of hypothesis
presented above. After observe Y , the number of success in n independent
trials, the sampling distribution (observational model) is given by
!
n y
P(Y = y |) = f (y |) =
(1 )1y
y
To obtain a closed form for the marginal distribution, we use the Beta(a, b)
prior distribution. That is
P() =

1
a1 (1 )b1 ,
B(a, b)

where B(a, b) is the beta function.

For convenience we use the reparametrization (, M) where = a/(a + b), is
the prior mean and M = a + b, a measure of the prior precision.
Alvaro Mauricio Montenegro Daz

Bayesian Statistics Lecture 1 Motivation

Bayesian Statistics Lecture 1 Motivation

Some basic Bayesian models

A beta/binomial model II
Y Bin(n, )
The prior variance given by (1 )/(M + 1) is a decreasing function of of M.
The marginal distribution of Y is called beta-binomial, and can be shown that
 
Y
E
=
n
 


(1 )
Y
n1
Var
=
1+
.
n
n
M +1
The posterior distribution of is again a Beta distribution with
 
M
n
Y
+
M +n
M +n n
h
i
)/(M

Var [|Y ] = (1
+ n) .

= E [|Y ] =

Alvaro Mauricio Montenegro Daz

Bayesian Statistics Lecture 1 Motivation

Bayesian Statistics Lecture 1 Motivation

Some basic Bayesian models

Poisson distribution

Yi Po(),

i = 1...,n

It usual to assume a gamma distribution for the prior of , say G (, ). For

this example, we assume the parameterization
p(x; , ) =

x 1 e x
,
()

so, E [X ] = / and Var [X ] = / 2 . The posterior distribution is

"
p(|y)

n
Y

#
P

e yi 1 e = e (n+)

yi +1

i=1

Alvaro Mauricio Montenegro Daz

Bayesian Statistics Lecture 1 Motivation

Bayesian Statistics Lecture 1 Motivation

Some basic Bayesian models

Example 1. Bayesian estimate of

Yi Po(),

i = 1...,n

Then,
p(|y) = G

n
X

!
yi + , n +

i=1

Hence, the EAP estimate of is

Pn
 Pn

 
n

i=1 Yi +
i=1 yi
b=

+
.
=
n+
n+
n
n+

Alvaro Mauricio Montenegro Daz

Bayesian Statistics Lecture 1 Motivation