+

Stat 608 Chapter 4

+
Weighted Least Squares

In Chapter 3, we saw that it is sometimes possible to

overcome nonconstant error variance by transforming
Y and/or X . In this chapter we consider an alternative
way of coping with nonconstant error variance, namely
weighted least squares (WLS).

2
+The Use of Weighted Least Squares

Yi is the average or the Yi is the sum of ni

median of ni observations observations
w i = ni wi = 1/ni
Ex: Yi = Per capita spending Ex: Yi = GDP, Total
on education, Crimes per spending per
100,000 people, average department
income

These are the most common uses of WLS.

Warning: In many situations, the variance is not
constant, and it is not straightforward to determine the
correct model for the variance.
Many people use transformations in these situations.

3
+
Examples: Variance Proportional
to n and x
Suppose Yi = Per Capita GDP. Then

Then we could take:

Suppose That is, Var(ei) = 2 xi.

Then we could take:

4
+
Weighted Least Squares Model
Consider the straight line regression model

where the ei have mean 0 but variance 2 / wi.

wi large wi small
Variance of errors close to
0. Variance of errors large.

Points close to the line. Points far from line.

Parameter estimates such Parameter estimates dont

that the fitted line at xi take the values (xi, yi) into
close to yi. The ith case is account as much.
taken more into account. 5
+
Weighted Least Squares Model
What does the covariance matrix for the errors look
like?

6
+
Derivation
To take into account the weights when we estimate the
regression parameters, we consider the following
weighted version of the residual sum of squares:

We minimize WRSS with respect to the parameters or

b0 and b1.

7
+
Derivation

Then setting the derivative equal to 0, we find the

minimum:

8
+
Special Case: Multiple
measurements at each value of
x
70 50
Rooms
30 10

2 4 6 8 10 12 14 16
Crews
9
+
Special Case: Multiple
measurements at each value of
x
Consider the simple linear regression model

where the ei have mean 0 but variance 2/wi. In this case

we take

so that the Yi (given xi) have variance 1.

In the special case of the cleaning data set, we were able

to estimate the variances of Y at each value of x i
because we had multiple measurements at each x i.

10
+
Cleaning Example

Number n Standard
of deviation
Crews (Yi)
2 9 3.00
4 6 4.97
6 5 4.69
8 8 6.64
10 8 7.93
12 7 7.29
16 10 12.00

11
+
Cleaning Example: Old model
with square root
transformations
0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 Scale-Location
31
5
46
Standardized residuals

3 4 5 6 7 8
Fitted values
lm(y2 ~ x2)
12
+
Cleaning Example: New
weighted model, no
transformations
Scale-Location
1.5 31
10

34
Standardized residuals
0.5 0.01.0

10 20 30 40 50 60
Fitted values
lm(clean$Rooms ~ clean$Crews)
13
+
Case 2: X continuous

In general, we dont have multiple measurements at

every value of X.

If we have only one observed response at each value of

X, how many parameters do we need to estimate using
the previous method?

14
+
Case 2: X continuous
Calculating weighted least squares:

Consider the least squares regression model:

where the ei have mean 0 but variance 2wi. If we

multiply both sides of the previous equation by w i we
get:

where the wi ei have mean 0 but variance:

15
+
Case 2: X continuous
We can calculate the least squares fit of the first model
by calculating the least squares fit to the second, which
is a multiple linear regression model with two
predictors and no intercept. Use the following
substitutions:

Then we can rewrite the second model as:

16
+
Case 2: X continuous
In other words, we can rewrite:

17
+
Another Parameterization:

18