Part 8 Linear Regression

1. Simple Linear Regression Models 2. Multiple Linear Regression Models 3. F Tests 4. Prediction and Cautions 5. Further Inference about Coefficients of Individual Independent Variables Section 8.1 Simple Linear Regression Models Example (Reed Auto) Reed Auto periodically has a special weeklong sale. As part of the advertising campaign Reed runs one or more television commercials during the weekend preceding the sale. Data from a sample of 5 previous sales are shown below. Week 1 2 3 4 5 Number of TV Ads 1 3 2 1 3 Number of Cars Sold 14 24 18 17 27

Number of Cars Sold

30 20 10 0 1 2 3

Number of TV Ads

By looking at the scatter diagram, we can observe that there exists a strong linear relationship between the number of TV ads (the independent variable, explanatory variable or predictor) and the number of cars sold (the dependent variable or response variable). We may draw more than one straight line through the scatter diagram. Number of Cars Sold
30 20 10 0 1 2 3

Number of TV Ads
2

BUS1200-8

BUS1200-8

Which straight line do you think is the best? Simple linear regression model Assumption 1. For i = 1, 2, , n, Yi follows N(0 + 1xi , 2), where 0 and 1 are constants and xis are non-random. n is called the number of observations. 2. Y1, Y2, , Yn are independent. Density Same shape of Y1 Density of Y2 0 y x
1

1 n 1 n y2), , (xn, yn), let x = xi , y = yi , n i =1 n i =1 b1 =

(x
i =1 n

x )( yi y )
i

(x
i =1

x)

and b0 = y b1 x .

Then y = b0 + b1 x is called the estimated regression equation. Let yi = b0 + b1 xi for i = 1, 2, , n. y (xi, yi) yi

yi yi yi
y = b0 + b1 x

x2 x

Regression line y = 0 + 1x For n pairs of data (x1, y1), (x2,

3

xi

Definition
BUS1200-8

Theorem 1. b0 and b1 meet the least squares criterion.

BUS1200-8 4

That is, the value of

(y y )
i =1 i i

will be

increased if the value of b0 or b1 is made different from that in expression (*). 2. b0 and b1 are unbiased estimates of 0 and 1 respectively. Two formulae for calculation:

( xi x )( yi y ) = xi yi
i =1

(x x )
i =1 i

n 1 n xi yi , n i =1 i =1 i =1 2 n 1 n 2 = xi xi . n i =1 i =1

(The last row will be used later.) n n n 1 n ( xi x )( yi y ) = xi yi n xi yi i =1 i =1 i =1 i =1 1 = 220 (10)(100) = 20, 5 2 n n 1 n 2 2 ( xi x ) = xi n xi i =1 i =1 i =1 1 2 = 24 (10 ) = 4, 5 b1 = 20 / 4 = 5, b0 = y b1 x = 100 / 5 5 (10 / 5) = 10.
Number of Cars Sold
30 20

Example (Reed Auto) [Solution] 1 2 3 i 1 3 2 xi 14 24 18 yi 2 xi 1 9 4 xiyi 14 72 36 yi2 196 576 324

BUS1200-8

5 Sum 1 3 10 17 27 100 1 9 24 17 81 220 289 729 2114

5

y = 10 + 5x
1 2 3

10 0

Number of TV Ads

[Interpretation] For each increase of one in the number of TV ads, we may say that the number of cars sold is estimated to increase
BUS1200-8 6

by 5 on the average. We say that the regression relationship is significant if 1 is not zero. Hypothesis tests should be conducted before saying that the regression relationship is significant or not.

one-family houses sold recently in a certain large housing development.

Section 8.2 Multiple Linear Regression Models Example (Price of house) Number of Number of Price (in bedrooms, x1 baths, x2 dollars), y 3 2 78800 2 1 74300 4 3 83800 2 1 74200 3 2 79700 2 2 74900 5 3 88400 4 2 82900 The above data show the numbers of bedrooms, the numbers of baths and the prices at which a random sample of eight
BUS1200-8 7

Multiple linear regression model Assumption 1. For i = 1, 2, , n, Yi follows N(0 + 1xi1 + 2xi2 + + pxip, 2), where 0, 1, , p are constants and xiks are non-random for k = 1, 2, , p. n is called the number of observations. 2. Y1, Y2, , Yn are independent.
Comparing with a simple linear regression model, there are now p independent variables, although there is still only one dependent variable. The p independent variables are x1, x2, , xp, where the variable xk takes on values x1k, x2k, , xnk, k = 1, 2, , p. Definition

BUS1200-8

1 x11 K x1 p 0 y1 1 x K x y 21 2p 2 , = 1 . Y = , X = M M M M M M 1 x K x y n1 np n p T 1 T Let b = (X X) X Y. Then b is a (p + 1)1 matrix and can be written as b0 b 1 b = . M b p y = b0 + b1 x1 + b2 x2 + + bp xp is called the estimated regression equation. Let yi = b0 + b1 xi1 + b2 xi2 + + bp xip for i = 1, 2, , n. Theorem 1. b0, b1, , bp meet the least squares criterion. 2. b0, b1, , bp are unbiased estimates of 0, 1, , p respectively.
BUS1200-8 9

When p = 2, yi = b0 + b1 xi1 + b2 xi2. (xi1, xi2, yi) y y = b0 + b1 x1 + b2 x2

yi yi

x2

x1 ( xi1 , xi 2 , yi ) (xi1, xi2, 0)

Using Excel How to make data analysis tools available Menu: Tools > Add-Ins. If Analysis ToolPak is not selected, select it and click OK. The following example shows how to use data analysis tools to solve regression problem. Example (Price of house)
BUS1200-8 10

[Solution] Open the Excel file for regression. Tools > Data Analysis > Regression Click OK. Input Y Range: C2:C10 Input X Range: A2:B10 Check Labels. Do not check Constant is Zero. Check Confidence Level and then change 95 to 90. (No matter you check Confidence Level or not, a 95% confidence interval is always provided.) Output Range: E2 Click OK. Therefore the estimated regression equation is y = 65191.67 + 4133.33x1 + 758.33x2. [Interpretation] For each increase of one in the number of bedrooms, we may say that the price of a house is estimated to increase by 4133.33 dollars on the average. For each increase of one in the number of baths, we may say that the price of a house is estimated to increase by 758.33 dollars on the average.
BUS1200-8 11

Section 8.3 F Tests Definition 1. The total sum of squares, denoted by SST, is the variation in the y values around their mean:

(y
i =1

y) .
2

2. The regression sum of squares, denoted by SSR, is the variation in the y values attributable to the linear relationship between x values and y values:
(y
i =1 n i

y) .
2

For simple linear regression, it can be proved that

SSR = ( yi y ) = b
2 i =1

2 1

(x
i =1

x) .
2

3. The error sum of squares, denoted by SSE, is the variation in the y values attributable to factors other than the linear relationship between x values and y values:

(y
i =1
BUS1200-8

yi ) .
2
12

This is also called the sum of squares for error or the residual sum of squares. 4. The regression mean square, denoted by MSR, is SSR / p where p is the number of independent variables. 5. The error mean square, denoted by MSE, is SSE / (n p 1). This is also called the mean square for error or the residual mean square. 6. The coefficient of determination, denoted by r2, is SSR / SST. 2 r measures the proportion of the variation in y values that can be explained by the variation in x values. Theorem 1. It is always true that SST = SSR + SSE. Therefore 0 r2 1 (because all SSs are
BUS1200-8 13

non-negative). 2. The MSE is an unbiased estimate of 2. Therefore we denote the square root of MSE by se. Hypothesis test H0: 1 = 2 = = p = 0, H1: at least one of the 1, 2, , p is not equal to 0. ANOVA table Value Degrees Mean Source of Sum of of test of square variation squares statistic freedom Regression SSR MSR f = p MSR Error SSE n p 1 MSE MSE Total SST n1 At the significance level of , we reject H0 if and only if MSR / MSE f,p,np1. Example (Reed Auto)
BUS1200-8 14

Test H0: 1 = 0 against H1: 1 0 at the 5% level of significance. Find r2 and interpret your result. [Solution] n = 5, p = 1. SSR = b12 ( xi x ) = 52 4 = 100,
2 i =1 n

For hypothesis testing, computer statistical packages usually provide the p-value (a probability calculated according to the value of the test statistic based on the sample drawn) rather than the critical value(s) for each test. Decision rule Reject the null hypothesis if and only if the p-value is less than or equal to the given significance level . Example (Price of house) Test, at the 1% level of significance, H0: 1 = 2 = 0 against H1: at least one of the 1 and 2 is not 0. Find r2. [Solution] By using Excel, we obtain
Source of variation Sum of squares Value Degrees Mean of test of square statistic freedom

1 n SST = ( yi y ) = y yi n i =1 i =1 i =1 2 = 2114 100 / 5 = 114, SSE = SST SSR = 114 100 = 14. MSR 100 / 1 = = 21.43. MSE 14 /(5 1 1) Since 21.43 10.13 = f0.05,1,3, we reject H0 and conclude that 1 0. We say that the regression relationship is significant. From r2 = SSR / SST = 100 / 114 = 0.8772 we know that the regression relationship is very strong since about 88% of the variation in the number of cars sold may be explained by the linear relationship between the number of TV ads and the number of cars sold.
n 2

2 i

2 Regression 185269167 92634583 675.34 Error 685833 8 2 1 137167 Total 185955000 8 1

BUS1200-8 16

BUS1200-8

15

Moreover, the p-value is 8.26(107), which is below Significance F. Hence we reject H0 even at the 1% level of significance and conclude that at least one of the 1 and 2 is not 0. The value of r2 is 0.9963. [Remarks] 1. The critical value(s) depends on , which is determined subjectively. The p-value does not depend on . 2. It is more important to report p-values than to report values of test statistics in research results. Another example A multiple regression problem with two independent variables is considered. It is known that n = 6, se = 3.89142 and SST = 146. At the 5% level of significance, test whether both the coefficients for the two independent variables are zero. [Solution]
BUS1200-8 17

H0: 1 = 2 = 0 H1: at least one of the 1 and 2 is not 0. Degrees of freedom for regression = 2. Degrees of freedom for error = 6 2 1 = 3. MSE = se2 = 3.891422 = 15.1431, SSE = 3 (15.1431) = 45.4294, SSR = 146 45.4293 = 100.5706, MSR = 100.5707 / 2 = 50.2853, f = 50.28535 / 15.1431 = 3.321 < 9.552 = f0.05,2,3. Therefore we do not reject H0. We conclude that both the coefficients for the two independent variables are not significantly different from zero. Source SS df MS f Regression 100.5706 2 50.2853 3.321 Error 45.4294 3 15.1431 --Total 146 ----Definition For n pairs of data (x1, y1), (x2, y2), , (xn, yn), define the correlation coefficient as
BUS1200-8 18

r=

(x
i =1

x )( yi y )
.

r = 1

n n ( xi x )2 ( yi y )2 i =1 i =1

r = 0.75
0 x 0 x

It can be proved that 1 r 1.

y y

r=1

r = 0.75

If we apply a simple linear regression model to the n pairs of data (x1, y1), (x2, y2), , (xn, yn), it can be proved that the square of r is just r2, the coefficient of determination.
x

0 y

x 0

r = 0.2

r = 0.001

Section 8.4 Prediction and Cautions For given values x01, x02, and x0p of the independent variables, the estimated mean of Y is y0 = b0 + b1 x01 + b2 x02 + + bp x0p .
Example (Reed Auto) If Reed Auto plans to run 4 television commercials during a weekend, what will be
19 BUS1200-8 20

0
BUS1200-8

x 0

the average number of cars sold in the next week? [Solution] y0 = b0 + b1 x01 = 10 + 5 4 = 30. Example (Price of House) The predicted average price (in dollars) of houses having three bedrooms and one bath each is y0 = b0 + b1 x01 + b2 x02 = 65191.67 + 4133.33 3 + 758.33 1 = 78350. Cautions The linear relationship estimated for the given data may not be good outside the range of the values of independent variables actually used in the estimation. Rejecting H0: 1 = 2 = = p = 0 does not enable us to conclude that a cause-andeffect relationship is present between the independent variables and the dependent variable. If we do not reject H0: 1 = 2 = = p = 0,
BUS1200-8 21

then we can conclude that there is no linear relationship between the dependent variable and the independent variables, but we cannot say that there is no relationship between the dependent variable and the independent variables.

Section 8.5 Further Inference about Coefficients of Individual Independent Variables Example (Price of House) The following results are obtained by using Excel. 95% confidence interval for 1: (3546, 4721). 90% confidence interval for 2: (72, 1444). The p-value for testing H0: 1 = 0 against H1: 1 0 is 9.51(106). The p-value for testing H0: 2 = 0 against H1: 2 0 is 0.07644. At the 0.05 level of significance, we should reject 1 = 0 and accept 1 0;
BUS1200-8 22

not reject 2 = 0. Therefore we let the number of bedrooms be the only independent variable and obtain by Excel the new estimated regression equation y = 65373.24 + 4560.56x1. The p-value for testing H0: 1 = 0 against H1: 1 0 is 1.24(107). Therefore we accept 1 0 even at the 1% level of significance. (Note that this p-value appears twice in the Excel output.)

BUS1200-8

23