Ptsp material

by V.Ramchandrareddy date:19-10-2011

Central Limit Theorem

Consider independent random variables .The mean and variance and of each of the random variables are assumed to be known. Suppose . Form a random variable

The mean and variance of

are given by

and Thus we can determine the mean and the variance of .

Can we guess about the probability distribution of ? The central limit theorem (CLT) provides an answer to this question.

The CLT states that under very general conditions distribution to The conditions are: 1. The random variables as .

converges in

are independent and identically distributed.

2. The random variables are independent with same mean and variance, but not identically distributed. 3. The random variables are independent with different means and same variance and not identically distributed. 4. The random variables are independent with different means and each variance being neither too small nor too large. We shall consider the first condition only. In this case, the central-limit theorem can be stated as follows:

Ptsp material
by V.Ramchandrareddy date:19-10-2011

Suppose

is a sequence of independent and identically distributed and variance and Then, the with mean 0

random variables each with mean sequence { and variance . That is,

} converges in distribution to a Gaussian random variable

The central limit theorem is really a property of convolution. Consider the sum of two statistically independent random variables, say, . Then the pdf is the convolution of . This can also be shown with the help of the characteristic functions as follows:

where * is the convolution operation.

We can illustrate the CLT by convolving two uniform distributions repeatedly. In Figure 1, the convolution of two uniform distributions gives a triangular distribution. Further convolution gives a parabolic distribution and so on.

Ptsp material
by V.Ramchandrareddy date:19-10-2011

Proof of the Central Limit Theorem :

where

is the average of terms involving

and higher powers of

. and therefore,

Note also that each term in

involves a ratio of a higher moment and a power of

We give a less rigorous proof of the theorem with the help of the characteristic function. Further we consider each of to have zero mean. Thus,

Clearly, The characteristic function of is given by

We will show that as the characteristic function characteristic function of a Gaussian random variable.

is of the form of the

Ptsp material
by V.Ramchandrareddy date:19-10-2011

Expanding

in power series

Assume all the moments of

to be finite. Then

Substituting which is the characteristic function of a Gaussian random variable with 0 mean and variance .

NOTE:

1. Under the conditions of the CLT, the sample mean distribution to

converges in

In other words, if samples are taken from any

distribution with mean and variance , as the sample size increases, the distribution function of the sample mean approaches to the distribution function of a Gaussian random variable. 2. The CLT states that the distribution function converges to a Gaussian is a Gaussian

distribution function. The theorem does not say that the pdf

pdf in the limit. For example, suppose each has a Bernoulli distribution. Then the pdf of Y consists of impulses and can never approach the Gaussian pdf. 3. The Cauchy distribution does not meet the conditions for the central limit theorem to hold. As we have noted earlier, this distribution does not have a finite mean or a finite variance. Suppose a random variable has the Cauchy distribution

The characteristic function of

is given by

Ptsp material
by V.Ramchandrareddy date:19-10-2011

The sample mean

will have the characteristic function

Thus the sum of large number of Cauchy random variables will not follow a Gaussian distribution. 4.The central-limit theorem is one of the most widely used results of probability. If a random variable is result of several independent causes, then the random variable can be considered to be Gaussian. For example,

1. the thermal noise in a resistor is result of the independent motion of billions of electrons and is modelled as Gaussian. 2. the observation error/ measurement error of any process is modeled as a Gaussian.

5.The CLT can be used to simulate a Gaussian distribution given a routine to simulate a particular random variable.

Normal approximation of the Binomial distribution One of the applications of the CLT is in approximation of the Binomial coefficients. We have already stated about this approximation.Suppose, is a sequence of Bernoulli(p) random variables with ....

Then Thus, or,

is a Binomial distribution with

and

Ptsp material
by V.Ramchandrareddy date:19-10-2011

( assume the integrand interval = 1 ) This is normal approximation to the Binomial coefficients and is known as the DeMoirre-Laplace approximation.