Cribsheet 2 MAS2305/MAS3305

Proof that E(RSS) = (n 2) 2

Recall from the notes that the residual sum of squares, RSS, can be written
as:
S2
Syy xy
Sxx
and that
Sxy
=
Sxx
Now note that these can be put together to give
2 2
!
Sxy Sxy
RSS = Syy = Syy 2
Sxx = Syy 2 Sxx
Sxx Sxx

The expectation of this can therefore be found by finding in turn the expec-
tations of Syy and 2 Sxx . The second of these is the simpler. We know from
the notes that:
=
E()
2
=
var()
Sxx
We also know from the definition of the variance of a random variable, X,
that:
E(X 2 ) = var(X) + E(X)2
Putting these together shows that:
 
E(2 Sxx ) = + E()
var() 2 Sxx
2
!
= + 2 Sxx
Sxx
= + 2 Sxx
2
(1)

The expectation of Syy = ni=1 (yi y)2 might be thought to be (n 1) 2 , as

P

the expression is formally the same as that for a sample variance. However,
this result arises only for the case when the yi have the same distribution and
in the case of the regression model the expectations of the yi are different
for observations with different xi s. How do we proceed? The i do have
a common distribution and so we know from the case of the variance of a
simple random sample that the expectation of ni=1 (i )2 is (n 1) 2 . Can
P

we make use of this observation when working out E(Syy )?

www.mas.ncl.ac.uk/nag48/teaching/MAS2305/ 1
Cribsheet 2 MAS2305/MAS3305

As yi = + xi + i , then summing this from i = 1, . . . , n and dividing by n

shows that y = + x + . Therefore

yi y = ( + xi + i ) ( + x + )
= (xi x) + (i )

Squaring this gives:

(yi y)2 = 2 (xi x)2 + 2(xi x)(i ) + (i )2

and summing from i = 1, . . . , n gives:

n n
Syy = 2 Sxx + 2 (i )2
X X
(xi x)(i ) +
i=1 i=1

Since E(i ) = 0 it follows that E() = 0 and hence the expectation of the
second term on the RHS is 0. Therefore
n
!
E(Syy ) = 2 Sxx + E (i )2
X

i=1
= Sxx + (n 1) 2
2
(2)

Now subtract (1) from (2) to get

E(RSS) = (n 2) 2

as required.

www.mas.ncl.ac.uk/nag48/teaching/MAS2305/ 2