1

BETA-BINOMIAL BINOMIAL JOINT DISTRIBUTION OF THE SAMPLE

(LIKELIHOOD), BETA PRIOR
D. Uniform Prior, Binomial Likelihood:
1. Suppose we take a random sample from a Bernoulli distribution with
parameter p. Our joint distribution of the sample is (Likelihood function) is:

1 1 2 2 n n
n
X 1-X X 1-X X 1-X
n i
i=1
n
y n-y
i
i=1
f (x | p) = f(x | p)=p (1 p) p (1 p) ...p (1 p)
=p (1 - p) and y = X

[

2. Now, suppose our prior distribution of p is simply Uniform on 0 to 1; that is:
1 0 < p < 1
(p)=
0 otherwise

3. Hence the joint distribution of the sample and p is
1 2 3 n n 1 2 3 n
h(x , x , x , ... , x , p) = f (x , x , x , ... , x | p)(p)
Or simply:
h(x,p) = f
n
(x | ) () =
n
y n-y
i
i=1
p (1 - p) and y = X

4. The marginal distribution of the sample is:
n 1 2 3 n 1 2 3 n
y n-y
(x , x , x , ... , x )= h(x , x , x , ... , x , p) =
( 1) ( 1)
p (1 - p)
( 2)
p
p
g dp
y n y
dp
n
I + I +
=
I +
}
}

This result is from the form of the Beta distribution is:
1 1
1 2
( )
(1 ) 0 1
( ) ( ) ( | , ) 0, 0
0 otherwise
x x x
f x
o |
o |
o | o o o |

I +
< <

I I = > >

2
Where =y+1 and =n-y+1.
5. So that the posterior distribution is:
n 1 2 3 n
1 2 3 n
n 1 2 3 n
f (x , x , x , ... , x | )()
(| x , x , x , ... , x )=
g (x , x , x , ... , x )
=

y n-y
( 2)
p (1 - p)
( 1) ( 1)
n
y n y
I +
I + I +

This is a Beta distribution with parameters:
1 1
1 1 and 1 1
n n
i i
i i
y X n y n X o |
= =
= + = + = + = +


6. The expected value and variance of the Beta distribution is:
2
( ) and VAR(X) =
( ) ( 1)
E X
o o|
o | o | o |
=
+ + + +

7. Hence, the Bayesian Estimator for the Mean and Variance is:
1
1

2
n
i
i
X
p
n
=
+
=
+

and
1 1
2
1 1
( )
( 2) ( 3)
n n
i i
i i
X n X
VAR p
n n
= =
| || |
+ +
| |
\ .\ .
=
+ +


8. And the MLE for the Mean and Variance is:
1

n
i
i
mle
X
p
n
=
=

and
(1 )
( )
mle
p p
VAR p
n

=
Note that, as the sample size increases:

bayes mle
p p
This is also true of the variances. To see this, divide the numerator and denominator by
n
2
; that is:
3
1 1
2
1 1
1 1 1
( )
4 3 ( 2) ( 3)
4 1
n n
i i mle mle
i i
X n X p p
n n
VAR p
n n
n
n n
= =
| || |
| || |
+ + + +
| | | |
\ .\ . \ .\ .
= =
+ + | || |
+ + +
| |
\ .\ .


So that, as the sample size increases:
( ) ( )
bayes mle
VAR p VAR p

D. Conjugate Priors (Part 1) Binomial Joint Distribution of the Sample
(Likelihood function) and Beta Prior Distribution Bayesian Computation With R
example of Beta-Binomial

1 1 2 2 n n
n
X 1-X X 1-X X 1-X
n i
i=1
n
y n-y
i
i=1
f (x | p) = f(x | p)=p (1 p) p (1 p) ...p (1 p)
=p (1 - p) and y = X

[

-1 -1
(+)
(p)= p (1 - p) , 0 < p < 1, , > 0
()()

Recall that the joint distribution of the sample and p is equal to the product of the joint
distribution of the sample (likelihood function) and the prior distribution of p:
h(x
1
, x
2
, , x
n
, p ) = f
n
(x | p) (p) =
y n-y -1 -1 y+-1 n-y+-1
(+) (+)
p (1 - p) p (1 - p) p (1 - p)
()() ()()
=
To get the marginal distribution of the sample we need to integrate out p.
1
y+-1 n-y+-1
0
(+) ( ) ( )
( ) *
()() ( )
(n++) (+) ( ) ( )
p (1 - p)
(y+)(n-y+) ()() ( )
n
y n y
g x
n
y n y
dp
n
o |
o |
o |
o |
I + I +
=
I + +
I + I +
=
I + +
}

And the Posterior distribution is:
4
y+-1 n-y+-1
y+-1 n-y+-1
(+)
p (1 - p)
( , ) ()()
( | )
(+) ( ) ( )
( )
()() ( )
(n++)
p (1 - p)
(y+)(n-y+)
n
n
h x p
p x
y n y
g x
n

o |
o |
= = =
I + I +
I + +


This a Beta distribution with * = y + and * = n y + , so the posterior is:
*-1 *-1
(*+*)
(p | x)= p (1 - p)
(*)(*)

The mean of the posterior is:

* y +
E(X) p
* + * + + n
= = =
If we take a second sample and use the posterior as our new prior then
*-1 *-1
2 1 1
(*+*)
(p)= (p | x )= p (1 - p)
(*)(*)

and the joint distribution of the sample is (likelihood function) for the second sample
is:
2 2 2
2
y n -y
n 2
f (x | p) = p (1 - p)
where the subscript gives the sample number. The posterior is the Beta distribution
-1 -1
2 2
( + )
(p | x )= p (1 - p)
()()

where
2 2 1
2 2 2 2 1 1 1 2 1 2
= y + * = y + y + and
= n - y + * = n - y + n - y + = n + n - y - y +

and
5
1 2
1 2
y + y +
E(X) p
+ + n + n +
= = =

As the total sample size gets large this converges to the MLE estimator:

m
k
k=1
m
k
k=1
y
E(X) = = y = p
n