Time Series Analysis Essential Concepts

Essential concepts for time series analysis

This chapter reiterates important concepts that you should be aware of from the rst year. Section 0.9 is a small and obvious bit from Statistics 2.

0.1

Random variables

Let X be a random variable (r.v.) taking values in R. The distribution function F of X is given by F (x) = P(X x) (1) where P(A) is the probability of event A occurring. If X is a continuous r.v. its distribution function F may be written as
x

F (x) =

f (u)du

(2)

for some x R and integrable function f : R [0, ). The function f is called the density function of X.

0.2

Properties of density functions

A density function f always integrates to 1. In mathematical notation, f (x)dx = 1.

(3)

0.3

Expectation

Dene the expectation or mean of X as EX =

xf (x)dx.

(4)

Note that the expectation can be equal to and in this case we say that the moment does not exist (the mean is the rst moment). Expectation is linear. That is, if a, b R are constants and X, Y are r.v.s then E(aX + bY ) = aEX + bEY.

(5)

c University of Bristol 2011. This material is copyright of the University unless explicitly stated otherwise. It is provided exclusively for educational purposes at the University and is to be downloaded or copied for your private study only. The material from previous years is not necessarily a reliable indicator of what will be covered or examined this year.

0.4

Variance
var(X) = E{(X EX)2 } (6)

The variance of a r.v. X is dened to be

and describes the variation of X about its mean. The following relation is always true: if a R is constant then var(aX) = a2 var(X). If the collection {Xi }n of r.v.s is independent then i=1
n n

(7)

var
i=1

Xi

=
i=1

var(Xi ).

(8)

0.5

Covariance and correlation

cov(X, Y ) = E{(X EX)(Y EY )} (9)

The covariance between two r.v.s X and Y is given by

and describes the extent of the linear relationship between X and Y . The correlation between X and Y is a normalized measure of the linear association: cov(X, Y ) corr(X, Y ) = 1 . 1 var(X) 2 var(Y ) 2 Always |corr(X, Y )| 1. (11)

(10)

0.6

Multivariate distribution
FX1 ,...,Xn (x1 , . . . , xn ) = P(X1 x1 , . . . , Xn xn ). (12)

If X1 , . . . , Xn are r.v.s then their joint distribution function is If the r.v.s are all mutually independent then
n

FX (x) =
i=1

FXi (xi ).

(13)

0.7

Normal distribution

A r.v. X is said to have the normal distribution with mean and variance 2 (or standard deviation of ) or in symbols X N(, 2 ) if . (14) 2 2 The density and distribution functions of a standard normal distribution N(0, 1) are denoted (x) and (x) respectively. If X N(1 , 1 ) and Y N(2 , 2 ) and they are independent then
2 2 X + Y N(1 + 2 , 1 + 2 ).

fX (x) =

exp

(x )2 2 2

(15)

0.8

Estimation

Suppose X1 , . . . , Xn are an independently distributed random sample with mean and variance 2 (not necessarily normal). We can estimate (population mean) by 1 X= n
n

Xi ,
i=1

(16)

the sample mean. Note the following properties of the sample mean: 1 E[X] = n
n

EXi =
i=1

1 n = . n

(17)

That is the sample mean, which is unbiased for the population mean (i.e. on average the sample mean hits the target). Further
n

var(X) = n2
i=1

var(Xi ) = n2 n 2 =

2 . n

(18)

That is the accuracy of the sample mean, which improves as more and more data are collected (n ; i.e. the spread around the target gets smaller and smaller). We can estimate 2 (population variance) with s2 n1 Also, it can be shown that E[s2 ] = 2 . n1 The variance of s2 can be worked out. n1 Note that the quantities X and s2 n1 are random. They depend on the data sample, X1 , . . . , Xn that you collected. If you take another random sample then the values of X and s2 change. n1 (20) 1 = n1
n

(Xi X)2 .
i=1

(19)

0.9

Bias

Suppose that X1 , . . . , Xn is an independent and identically distributed (i.i.d.) random sample from a distribution dependent on a parameter (e.g., mean, variance). Suppose that is an estimator is dened by for . Then the bias of bias() = E( ) = E{(X)} . (21)

The notation (X) is to suggest the dependence of the estimator on the data sample X collected. The variance of the estimator is given by var() = E{( E)2 }. (22)

Sometimes it is important to take bias and variance into account jointly. This can be done using the expected mean squared error of which takes an overall view of how well the estimator does in estimating : MSE() = E{( )2 } = E{( E + E )2 } = E{( E)2 } + 2E{( E)(E )} + E{(E )2 } = var() + 2E{ E}{E } + E{bias()2 } = var() + bias()2 . The above calculation was done in Statistics 2 but it is the only small addition to the prerequisites.

0.10

Sample covariance

If X1 , . . . , Xn and Y1 , . . . , Yn are both i.i.d. samples from (possibly) dierent distributions then their sample covariance, c, is given by c= 1 n
n

(Xi X)(Yi Y )
i=1

(23)

and is an estimator of cov(X, Y ) (where X has the same distribution as any of X1 , . . . , Xn , they all have the same distribution as they are i.i.d.). The estimator c is not unbiased. The sample correlation is given by r= where sX = c sX sY

(24)

s2 for the Xs and similarly for the Ys. Also |r| 1 for any sample of observations. n1