SIMPLE REGRESSION MODEL

Y
Y b1 b 2 X

b1

X1

X2

X3

X4

Suppose that a variable Y is a linear function of another variable X, with unknown parameters b1 and b2 that we wish to estimate.
Christopher Dougherty 19992006 1

SIMPLE REGRESSION MODEL

Y
Y b1 b 2 X

b1

X1

X2

X3

X4

Suppose that we have a sample of 4 observations with X values as shown.

Christopher Dougherty 19992006

SIMPLE REGRESSION MODEL

Y
Y b1 b 2 X

b1

Q1

Q2

Q3

Q4

X1

X2

X3

X4

If the relationship were an exact one, the observations would lie on a straight line and we would have no trouble obtaining accurate estimates of b1 and b2.
Christopher Dougherty 19992006 3

SIMPLE REGRESSION MODEL

P4
Y b1 b 2 X

P1

b1

Q1

Q2 P2

Q3 P3

Q4

X1

X2

X3

X4

In practice, most economic relationships are not exact and the actual values of Y are different from those corresponding to the straight line.
Christopher Dougherty 19992006 4

SIMPLE REGRESSION MODEL

P4
Y b1 b 2 X

P1

b1

Q1

Q2 P2

Q3 P3

Q4

X1

X2

X3

X4

To allow for such divergences, we will write the model as Y = b1 + b2X + u, where u is a disturbance term.
Christopher Dougherty 19992006 5

SIMPLE REGRESSION MODEL

P4
Y b1 b 2 X

u1 P1

b1
b1 b 2 X 1

Q1

Q2 P2

Q3 P3

Q4

X1

X2

X3

X4

Each value of Y thus has a nonrandom component, b1 + b2X, and a random component, u. The first observation has been decomposed into these two components.
Christopher Dougherty 19992006 6

SIMPLE REGRESSION MODEL

P4

P1 P2 P3

X1

X2

X3

X4

In practice we can see only the P points.

Christopher Dougherty 19992006

SIMPLE REGRESSION MODEL

P4
b1 b2 X Y

P1 P2 P3

b1

X1

X2

X3

X4

Obviously, we can use the P points to draw a line which is an approximation to the line ^ Y = b1 + b2X. If we write this line Y = b1 + b2X, b1 is an estimate of b1 and b2 is an estimate of b2.
Christopher Dougherty 19992006 8

SIMPLE REGRESSION MODEL Y (actual value) Y

(fitted value) Y
R3 P1 R1 R2 P3

P4
b1 b2 X Y

R4

b1

P2

X1

X2

X3

X4

The line is called the fitted model and the values of Y predicted by it are called the fitted values of Y. They are given by the heights of the R points.
Christopher Dougherty 19992006 9

SIMPLE REGRESSION MODEL Y (actual value) Y

(fitted value) Y
e (residual) Y Y

P4
e4 R3 R4
b1 b2 X Y

e1 b1

P1 R1 e2

R2

e3 P3

P2

X1

X2

X3

X4

The discrepancies between the actual and fitted values of Y are known as the residuals.

Christopher Dougherty 19992006

10

SIMPLE REGRESSION MODEL Y (actual value) Y

(fitted value) Y
R3 P1 R2 P3

P4
b1 b2 X Y

R4

Y b1 b 2 X

b1
b1

R1

P2

X1

X2

X3

X4

Note that the values of the residuals are not the same as the values of the disturbance term. The diagram now shows the true unknown relationship as well as the fitted line.
Christopher Dougherty 19992006 11

SIMPLE REGRESSION MODEL Y (actual value) Y

(fitted value) Y

P4
b1 b2 X Y

Y b1 b 2 X

P1

b1
b1

Q1

Q2 P2

Q3 P3

Q4

X1

X2

X3

X4

The disturbance term in each observation is responsible for the divergence between the nonrandom component of the true relationship and the actual observation.
Christopher Dougherty 19992006 12

SIMPLE REGRESSION MODEL Y (actual value) Y

(fitted value) Y
R3 P1 R2 P3

P4
b1 b2 X Y

R4

Y b1 b 2 X

b1
b1

R1

P2

X1

X2

X3

X4

The residuals are the discrepancies between the actual and the fitted values.

Christopher Dougherty 19992006

13

SIMPLE REGRESSION MODEL Y (actual value) Y

(fitted value) Y
R3 P1 R2 P3

P4
b1 b2 X Y

R4

Y b1 b 2 X

b1
b1

R1

P2

X1

X2

X3

X4

If the fit is a good one, the residuals and the values of the disturbance term will be similar, but they must be kept apart conceptually.
Christopher Dougherty 19992006 14

SIMPLE REGRESSION MODEL Y (actual value) Y

(fitted value) Y

P4
u4 Q4
b1 b2 X Y

Y b1 b 2 X

b1
b1

b1 b 2 X 4

X1

X2

X3

X4

Both of these lines will be used in our analysis. Each permits a decomposition of the value of Y. The decompositions will be illustrated with the fourth observation.
Christopher Dougherty 19992006 15

SIMPLE REGRESSION MODEL Y (actual value) Y

(fitted value) Y

P4
u4 Q4
b1 b2 X Y

Y b1 b 2 X

b1
b1

b1 b 2 X 4

X1

X2

X3

X4

Using the theoretical relationship, Y can be decomposed into its nonstochastic component b1 + b2X and its random component u.
Christopher Dougherty 19992006 16

SIMPLE REGRESSION MODEL Y (actual value) Y

(fitted value) Y

P4
u4 Q4
b1 b2 X Y

Y b1 b 2 X

b1
b1

b1 b 2 X 4

X1

X2

X3

X4

This is a theoretical decomposition because we do not know the values of b1 or b2, or the values of the disturbance term. We shall use it in our analysis of the properties of the regression coefficients.
Christopher Dougherty 19992006 17

SIMPLE REGRESSION MODEL Y (actual value) Y

(fitted value) Y

P4
e4 R4
b1 b2 X Y

Y b1 b 2 X

b1
b1

b1 b2 X 4

X1

X2

X3

X4

The other decomposition is with reference to the fitted line. In each observation, the actual value of Y is equal to the fitted value plus the residual. This is an operational decomposition which we will use for practical purposes.
Christopher Dougherty 19992006 18

SIMPLE REGRESSION MODEL

Least squares criterion:

Minimize RSS (residual sum of squares), where
2 2 RSS ei2 e1 ... en i 1 n

To begin with, we will draw the fitted line so as to minimize the sum of the squares of the residuals, RSS. This is described as the least squares criterion.
Christopher Dougherty 19992006 19

SIMPLE REGRESSION MODEL

Least squares criterion:

Minimize RSS (residual sum of squares), where
2 2 RSS ei2 e1 ... en i 1 n

Why not minimize

e
i 1

e1 ... en

Why the squares of the residuals? Why not just minimize the sum of the residuals?

Christopher Dougherty 19992006

20

SIMPLE REGRESSION MODEL

P4

P1 P2 P3

X1

X2

X3

X4

The answer is that you would get an apparently perfect fit by drawing a horizontal line through the mean value of Y. The sum of the residuals would be zero.
Christopher Dougherty 19992006 21

SIMPLE REGRESSION MODEL

P4

P1 P2 P3

X1

X2

X3

X4

You must prevent negative residuals from cancelling positive ones, and one way to do this is to use the squares of the residuals.
Christopher Dougherty 19992006 22

SIMPLE REGRESSION MODEL

P4

P1 P2 P3

X1

X2

X3

X4

Of course there are other ways of dealing with the problem. The least squares criterion has the attraction that the estimators derived with it have desirable properties, provided that certain conditions are satisfied.
Christopher Dougherty 19992006 23

SIMPLE REGRESSION MODEL

P4

P1 P2 P3

X1

X2

X3

X4

The next sequence shows how the least squares criterion is used to calculate the coefficients of the fitted line.
Christopher Dougherty 19992006 24

Copyright Christopher Dougherty 19992006. This slideshow may be freely copied for personal use.

Christopher Dougherty 19992006

17.06.06