Economics 310 Handout # VII

The Normal Distribution and Properties of Estimators

Miscellaneous Notes

Multivariate Normal.

Ordinary Least Squares

Properties of Estimators
I. Unbiased: Let We define II Efficiency: Let be an estimator of . Then if is a scalar. be an estimator of . Then is an unbiased estimator if E( )= )

(Note: this definition applies equally well whether

if

is a vector. and )<MSE( )-MSE( be two estimators of ) if is a scalar is a vector. . Then is a more efficient estimator

Definition: Let than if MSE( MSE( Note: If

) is non-negative definite and if

are two unbiased estimators of

these definitions can be stated in

terms of variances. Var( )<Var( ) if is a scalar. VAR( )-VAR( ) is nonnegative definite and if is a vector. In the vector case this can be seen to imply that

Definition: Cramer Rao Lower Bound. Let be any unbiased estimator of the information matrix of is defined as follows:

. Then

is called the Cramer Rao Lower Bound (CRLB) for any unbiased estimator of . This means that for any unbiased estimator nonnegative definite. III. Asymptotic Properties of Estimators. Definition: A sequence of random variables constant c if is said to converge in probability to a . (We use the notation plim ( )=c) , VAR( )-CRLB is

Definition: An estimator Theorem: Let plim( plim[g(

is said to be a consistent estimator of

if plim(

)=

) = c. Let g(

) be a continuous function. Then

)] = g(c).

Theorem: A sufficient condition for an estimator to be consistent is its bias and variance each approach a limit of 0 as n approaches infinity. Definition: Let Z = { } be a matrix of random variables each of which has a probability )}

limit. Then plim Z = {plim (

Theorem: Let A and B be two matrices such that plim A and plim B and the product AB exist. Then plim AB = (plim A)(plim B). Theorem: plim A-1 = [plim A]-1 Convergence in distribution: Let {xn} n = 1, 2, ... be a sequence of random variables. Let {Fn} n = 1,2,.... be the sequence of cumulative distribution functions (CDF) of the random variables {xn} n = 1,2.... This simply means that P(xn < a) = Fn(a), n = 1,2.... The sequence of random variables xn is said to converge in distribution to a random variable x with cumulative distribution function, F, if at all points where F is continuous. Alternatively, we can say that xn converges in distribution to a random variable x if for every a at which F is continuous. A familiar example of convergence in distribution is given by the central limit theorem which states that for any underlying population with finite mean and variance the distribution of

converges to a standard normal distribution. If xn converges in distribution to a random variable x with CDF, F(x), we say that F(x) is the limiting distribution of xn. Asymptotic distributions: Most of the estimators which we are interested in this course have degenerate limiting distributions, which is to say that in the limit the distribution collapses around a point. This is not a very useful property if we want to compare the asymptotic behaviors of two or more estimators. For example we might have two estimators both of which are consistent. Both distributions would collapse around the true

value of the parameter which we are estimating. But it is possible that the distribution of first estimator collapses more rapidly around the true value than the distribution of the second estimator so that for any large sample size n the variance of the first estimator is smaller than the variance of the second estimator (assume that both estimators are unbiased.) Then we would say that, in some sense, the first estimator is more efficient than the second estimator at least for large sample sizes. But the limiting distribution is not very useful for this purpose because in the limit the variances of both estimators are equal to zero. In order to make the kinds of comparisons which we are considering we need a different concept, the concept of an asymptotic distribution. An asymptotic distribution is a distribution which is used to approximate the true distribution of an estimator for large but finite sample sizes. The most common way of deriving an asymptotic distribution of an estimator is to find a transformation of that estimator which has a non-degenerate limiting distribution. For example, suppose that we want to find the asymptotic distribution of the sample mean. The sample mean has a degenerate limiting distribution. However we know that the function zn has a standard normal distibution where

Then using the relationship

we can derive the following asymptotic

distribution for the sample mean: . IV.

Maximum Liklihood Estimators Suppose that the probability density function of a random variable x, , satisfies some regularity conditions. ( The regularity conditions are conditions involving such things as the differentiability of the log of the density functions. All the density functions which we will encounter in this course will satisfy these discussions. For a full discussion of these conditions see Theil, Principles of Econometrics,1971) If the regularity conditions are met then the maximum likelihood estimator of is:

I. consistent II. asymptotically normal III. asymptotically efficient IV. achieves the Cramer Rao Lower Bound. ( This last condition means that the asymptotic variance covariance matrix of the estimator is equal to the CRLB.) Maximum likelihood estimators have another desirable property, the property of invariance. This property simply says that if is the maximum likelihood estimator of , and is a continuous function then MLE[g( )] = g[MLE( )]

Example: Let variance

be a random sample from a normal population with mean

and

. Then the log of likelihood function of the sample is