Exercise

Show the corrected sample variance is an unbiased estimator of

population variance.
Pn
2 (Xi − X̄ )2
S = i=1
n−1

Mezzetti Exercise Estimation

Exercise

Pn Pn
2 i=1 (Xi− x̄)2 (Xi − µ + µ − X̄ )2
S = = i=1 =
n−1 n−1
Pn 2
i=1 ((Xi − µ) − (X̄ − µ))
=
n−1
Pn 2 2


i=1 (Xi − µ) + (X̄ − µ) − 2(Xi − µ)(X̄ − µ)
= =
n−1
Pn 2 2
P

i 2(Xi − µ)(X̄ − µ)
P
i=1 (Xi − µ) i (X̄ − µ)
= + −
n−1 n−1 n−1

Mezzetti Exercise Estimation

Exercise

E ((Xi − µ)2 ) = σ 2
σ2
E (X̄ − µ)2 =
! n !
X X
(Xi − µ)(X̄ − µ) = (X̄ − µ) (Xi − µ) = n(X̄ − µ)2
i i
nσ 2 σ2 nσ 2
E (S 2 ) = + −2
n−1 n−1 n(n − 1)
nσ 2 σ 2
E (S 2 ) = −
n−1 n−1
E (S 2 ) = σ 2

Mezzetti Exercise Estimation

Uniform Distribution

Example: Let X1 , . . . , Xn be iid r.v. distributed as continuous

uniform distribution on [0, θ]. The probability distribution function
of Xi for each i is:
 −1
θ , 0≤x ≤θ
f (x|θ) =
0, otherwise

Consider T = maxi Xi an estimator of θ, is T an unbiased

estimator for θ?

Mezzetti Exercise Estimation

Uniform Distribution

Example: Consider T = maxi Xi an estimator of θ, is T an

unbiased estimator for θ?
We need to compute E (T ), thus we need distribution for T .
Let F be the cumulative distribution function for T ,
FT (y ) = P(T ≤ y ) = P(maxi Xi ≤ y )
 y n
P(maxi Xi ≤ y ) = P(X1 ≤ y , . . . , Xn ≤ y ) = P(X ≤ y )n =
θ
∂FT (y )  y n−1 1
fT (t) = =n
∂y θ θ
Z Z  
y n n
E (T ) = yfT (y )dy = n dy = θ
θ n+1
−1
Bias(T ) = E (T ) − θ = θ
n+1

Mezzetti Exercise Estimation

Uniform Distribution
Example: Consider T = maxi Xi an estimator of θ, is T an
unbiased estimator for θ?
We need to compute E (T ), thus we need distribution for T .
Let F be the cumulative distribution function for T ,
FT (y ) = P(T ≤ y ) = P(maxi Xi ≤ y )
 y n
P(maxi Xi ≤ y ) = P(X1 ≤ y , . . . , Xn ≤ y ) = P(X ≤ y )n =
θ
∂FT (y )  y n−1 1

fT (t) = =n
∂y θ θ
Z Z
y (n+1) n 2
E (T 2 ) = y 2 fT (y )dy = n n dy = θ
θ n+2
 2
2 2 n 2 n
Var (T ) = E (T ) − (E (T )) = θ − θ
n+2 n+1
2
−1 2 2 θ2
    
n 2
MSE (T ) = θ + θ =
(n + 2)(n + 1)2 n+1 (n + 2)(n + 1)
Mezzetti Exercise Estimation
Uniform Distribution: θ̂MOM

Example:
θ
x2
Z Z
x θ
E (X ) = xf (x|θ)dx = dx =|θ0 =
0 θ 2θ 2
θ
X̄ =
2
θ̂MOM = 2X̄

Mezzetti Exercise Estimation

Uniform Distribution: θ̂MOM

Example:

 
E θ̂MOM = θ
 
θ2 θ2
Var θ̂MOM = 4Var X̄ = 4 =
12n 3n

   
MSE θ̂MLE ≤ Var θ̂MOM
θ2
 
1 2
θ ≤
(n + 1)(n + 2) 3n
 
1 1
≤
(n + 1)(n + 2) 3n

Mezzetti Exercise Estimation

Exercise

Let (X1 , . . . , Xn ) be independent identically distributed random

variables with p.d.f.

f (x) = θ2 x exp(−θx) x >0

Is T (X1 , . . . , Xn ) = 1/X1 an unbiased estimator of θ?

Mezzetti Exercise Estimation

Exercise

Let (X1 , . . . , Xn ) be independent identically distributed random

variables with E (X ) = µ, Var (X ) = σ 2 , Are the following
estimators unbiased estimator for σ 2 ?
(X1 − X2 )2
T1 (X1 , . . . , Xn ) =
2
(X1 + X2 )2
T2 (X1 , . . . , Xn ) = − X1 X2
2

Mezzetti Exercise Estimation

Exercise 4

Let T1 and T2 be two independent and unbiased estimators of the

parameter θ, with Var (T1 ) = σ12 and Var (T2 ) = σ22 . Find the
UMVUE for θ among all linear combinations of T1 and T2 . What
is its variance?

Mezzetti Exercise Estimation

Solution Exercise 4

T = a1 T1 + a2 T2
E (T ) = a1 E (T1 ) + a2 E (T2 )
E (T ) = (a1 + a2 )θ

To be unbiased: a1 + a2 = 1, a2 = 1 − a1

Var (T ) = a2 Var (T1 ) + (1 − a)2 Var (T2 )

Var (T ) = a2 σ12 + (1 − a)2 σ22

Mezzetti Exercise Estimation

Solution Exercise 4

dVar (T )
= 2aσ12 − 2(1 − a)σ22
da
2aσ12 + 2(1 − a)σ22 = 0
σ22
a? =
σ12 + σ22

The UMVUE estimator for θ among all linear combinations of T1

and T2 is
σ2 σ2
T = 2 2 2 T1 + 2 1 2 T2
σ1 + σ2 σ1 + σ2

Mezzetti Exercise Estimation

Exercise

Let (X1 , . . . , Xn ) be a random sample of i.i.d. random variables

with expected value µ and variance σ 2 . Consider the following
estimator of µ:

Tn (a) = a × Xn + (1 − a) × X̄n−1
where Xn is the n − th observed random variable and X̄n−1 is the
sample mean based on n − 1 observations.
1 Find value of a such that Tn (a) is an unbiased estimator for µ
2 Find value of a? such that Tn (a?) is the most efficient
estimator for µ within the class Tn (a)?
3 Define concept of efficiency

Mezzetti Exercise Estimation

Pareto

Let (X1 , . . . , Xn ) be a random sample of i.i.d. random variables

distributed as a Pareto distribution with unknown parameters α
and x0 known

f (x; α, x0 ) = α x0α x −(α+1) for x ≥ x0

.
The log-likelihood function is
n
X
l(α, x0 ) = nlog α + nαlog (x0 ) − (α + 1) log xi
i=1

Mezzetti Exercise Estimation

Pareto

Thus
n
δl(α, x0 ) n X
= + nlog (x0 ) − log xi
δα α
i=1
δl(α,x0 )
Solving for δα = 0, the mle of α is given by
n
α̂ = Pn
i=1 log xi − nlog (x0 )

Mezzetti Exercise Estimation

Pareto: sufficiency

Observe that the joint pdf of X = (X1 , . . . , Xn )

n α
Y α xm
f (x; α, xm ) =
x α+1
i=1 i
αn x nα
= Qn mα+1
i=1 xi
= g (t, α)h(x)

where t = ni=1 xi g (t, α) = cαQ

n x nα t −(α+1) and h(x) = 1. By the
Q
m
factorization theorem, T (X ) = ni=1 Xi is sufficient for α.

Mezzetti Exercise Estimation

Pareto: Fisher Information

Thus

δ 2 l(α, x0 ) n
2
=− 2
δα α
n
In (θ) = 2
α

Mezzetti Exercise Estimation

EXERCISE

Let (X1 , . . . , Xn ) be a random sample of i.i.d. random variables

distributed as follows:
θ2θ
f (x; θ) = x >2
x θ+1
P
1 Show that i log (Xi ) is a sufficient statistics for θ
2 Find θ̂MLE maximum likelihood estimator (MLE) for θ and
discuss properties of this estimator.
3 Find θ̂MOM method of moment estimator (MOM) for θ.

Mezzetti Exercise Estimation

EXERCISE: solution

Y θ2θ
f (x1 , x2 , . . . , xn ; θ) =
i
xiθ+1
!θ+1
Y 1
f (x1 , x2 , . . . , xn ; θ) = θn 2nθ
xi
i
Q 1
Sufficient
P statistics i Xi , or any trasformation, as for example,
i log (X i)

Mezzetti Exercise Estimation

EXERCISE: solution

!θ+1
Y 1
n nθ
f (x1 , x2 , . . . , xn ; θ) = θ 2
xi
i
!θ+1
Y 1
L(θ|x1 , x2 , . . . , xn ) = θn 2nθ
xi
i
!
X
l(θ|x1 , x2 , . . . , xn ) = nlog (θ) + nθlog (2) − (θ + 1) log (xi )
i
!
∂l(θ) n X
= + nlog 2 − log (xi )
∂θ θ
i
2
∂ 2 l(θ) n
= −
∂θ θ2

Mezzetti Exercise Estimation

EXERCISE: solution

∂l(θ)
= 0
∂θ
n n
θ̂MLE = P = P xi



i log (xi ) − nlog (2) i log 2

Mezzetti Exercise Estimation

EXERCISE: solution

∞
θ2θ
Z Z
xf (x; θ)dx = x dx
2 x θ+1
Z Z ∞ θ
θ2
xf (x; θ)dx = dx
2 xθ
(−θ+1)
θx
E (X ) = |∞
2 θ2 dx
−θ + 1
θ
E (X ) = 2
θ−1
ˆ
θMOM
x̄ = 2
θMOM ˆ −1
x̄
θ̂MOM =
x̄ − 2

Mezzetti Exercise Estimation

Poisson distribution

Example:
Let X be distributed as a Possion;
λx exp(−λ)
f (x; λ) =
x!
Compare the following estimator for exp (−λ):
Pn
I (Xi = 0)
T2 = i=1


T1 = exp −X̄
n

Mezzetti Exercise Estimation

Maximum Likelihood Estimator for the Geometric
distribution

Considering for n observations from a Geometric distribution:

p(x|π) = π(1 − π)x

Find maximum likelihood estimator for E (X ) = 1−π

π
P
dlogL(π|x) n xi
= − i
dπ π 1−π
The second derivative:
d 2 logL(π|x)
P
n i xi
2
=− 2 −
dπ π (1 − π)2

Mezzetti Exercise Estimation

Maximum Likelihood Estimator for the Geometric
distribution

P
n xi
− i = 0
π 1−π
n
π̂ = P
n + i xi

Mezzetti Exercise Estimation

Example: Exam January 2016

Let (X1 , . . . , Xn ) be a random sample of i.i.d. random variables

distributed as a Pareto distribution with parameters α and xm both
un known
α −(α+1)
f (x; α, xm ) = α xm x for x ≥ xm

.
Calculate the Fisher information matrix for the parameter vector
θ = (xm , α). How do you interpret the off-diagonal terms?

Mezzetti Exercise Estimation

Example: Exam January 2016
The log-likelihood function is
n
X
l(α, xm ) = nlog α + nαlog (xm ) − (α + 1) log xi
i=1
n
∂l(α, xm ) n X
= + nlog (xm ) − log xi
∂α α
i=1
∂l(α, xm ) nα
=
∂xm xm
2
∂ l(α, xm ) n
= − 2
∂α2 α
∂ 2 l(α, xm ) nα
2
= − 2
∂xm xm
∂ 2 l(α, xm ) n
=
∂α∂xm xm

Mezzetti Exercise Estimation

Exercise
We consider two continuous independent random variables U and
W normally distributed with N(0, σ 2 ). The variable X defined by
p
X = U2 + V 2
has a Rayleigh distribution with a parameter σ 2
x2
 
x
f (x; θ) = 2 exp − 2 , x ≥0
σ 2σ
Let (X1 , . . . , Xn ) be a random sample of i.i.d. random variables
distributed as X
1 Apply the method of the moments to find the estimator

σ̂MOM of the parameter σ.

2 Find σ ˆ2 MLE maximum likelihood estimator (MLE) for σ 2 and
discuss properties of this 1estimator.
3 Compute the score function and the Fisher information.

4 Specify asymptotic distribution of θ̂

MLE .
Mezzetti Exercise Estimation
Exercise

∞ ∞
x2 x2
Z Z  
xf (x; θ)dx = exp − 2 dx
0 0 σ2 2σ
Z ∞ 2
x2
 
1 x
= exp − 2 dx
σ σ 2σ
√ 0 Z ∞
x2 x2
 
2π 1
= √ exp − 2 dx
σ 2 −∞ 2πσ 2σ

Y ∼ N(0, σ 2 )
∞
y2 y2
Z  
√ exp − 2 dy = E (Y 2 ) = Var (Y ) + E (Y )2 = σ 2
−∞ 2πσ 2σ

Mezzetti Exercise Estimation

Exercise

Z ∞
√ √
2π 1 2 σ π
xf (x; θ)dx = ×σ = √
0 σ 2 2

r
π
E (X ) = σ
2
r
2
σ̂MOM = x̄
π

Mezzetti Exercise Estimation

Exercise

2
Find σ̂MLE maximum likelihood estimator (MLE) for σ 2 and
discuss properties of this 1estimator.
Q  P 2
2 i xi x
L(σ |x) = exp − i 2i
σ 2n 2σ
P 2
2
X
2 x
logL(σ |x) = log (xi ) − n × log σ − i 2i
2σ
i
2
∂logL(σ 2 |x)
P
n i xi
= − +
∂σ 2 σ2 2σ 4

Mezzetti Exercise Estimation

Exercise

∂logL(σ 2 |x) xi2

P
2 i
=0 if σ =
∂σ 2 2n
P 2
∂ 2 logL(σ 2 |x) n x
= + − i6 i
∂ 2σ2 σ4 σ
∂ logL(σ 2 |x)
2 n n
|σˆ2 = + 4 −2 4 <0
∂ 2σ2 Pσ̂ 2 σ̂
2 i xi
σ̂MLE =
2n

Mezzetti Exercise Estimation

Exercise

Compute the score function and the Fisher information.

P 2
∂logL(σ 2 |x) n x
Score(σ) = = − 2 + i 4i
∂σ 2 σ P 2σ
2
∂ logL(σ 2 |x)
2 n i xi
= −
∂ 2σ2 σ4 σ6

∞ ∞
x3 x2
Z Z  
2
x f (x; θ)dx = exp − 2 dx
0 0 σ2 2σ

Mezzetti Exercise Estimation

Exercise
Integration by parts
Z ∞ Z
0
f ×g = f ×g − f0×g
0
Z ∞  Z ∞ 
x2 x2
   
2 x ∞ 2
x 2 exp − 2 dx = |0 x −exp − 2 − 2x −ex
0 σ 2σ 2σ 0
Z ∞
x2
 
x
= 2σ 2 × exp − dx
0 σ2 2σ 2
E (X 2 ) = 2σ 2

P 2
∂ 2 logL(σ 2 |x)
  
n x
I (θ) = −E = E − i6 i
∂ 2σ2 σ4 σ
n 2nσ 2
I (θ) = − 4 − 6
σ σ
n
I (θ) =
σ4
Mezzetti Exercise Estimation
Exercise

Specify asymptotic distribution of θ̂MLE .

σ4
 
2 2
σ̂MLE ∼ N σ ,
n

Mezzetti Exercise Estimation