SIMPLE LINEAR

REGRESSION
MODEL
Dr. Indra, S.Si, M.Si
OUTLINE

• Review of least squares procedure

• Inference for least squares lines
Introduction
Regression analysis is used to predict the value of one
variable (the dependent variable) on the basis of other
variables (the independent variables).

Dependent variable: denoted Y

Independent variables: denoted X1, X2, …, Xk
If we only have ONE independent variable, the model is

which is referred to as simple linear regression. We would

be interested in estimating β0 and β1 from the data we collect.
Introduction

We will examine the relationship between

quantitative variables x and y via a
mathematical equation.
The motivation for using the technique:
– Forecast the value of a dependent variable
(y) from the value of independent variables
(x1, x2,…xk.).
– Analyze the specific relationships between
the independent variables and the dependent
variable.
5

The Model

The model has a deterministic and a

probabilistic components
House
Cost

Most lots sell

for $25,000

House size
6

The Model

However, house cost vary even among same size houses!

Since cost behave unpredictably,

House we add a random component.
Cost

Most lots sell

for $25,000

House size
7

The Model

The first order linear model

y = b 0 + b1x + e
b0 and b1 are unknown population
y parameters, therefore are estimated
y = dependent variable from the data.

x = independent variable
b0 = y-intercept Rise b1 = Rise/Run
b1 = slope of the line b0 Run
e = error variable x
8

Estimating the Coefficients

The estimates are determined by
– drawing a sample from the population of
interest,
– calculating sample statistics,
– producing a straight line that cuts into the
data.
y w
w Question: What should be
w considered a good line?
w
w w w w w
w w w w w
w
x
9

The Least Squares (Regression) Line

A good line is one that minimizes

the sum of squared differences between the
points and the line.
0

The Least Squares (Regression) Line

Sum of squared differences = (2 - 1)2 + (4 - 2)2 +(1.5 - 3)2 + (3.2 - 4)2 = 6.89
Sum of squared differences = (2 -2.5)2 + (4 - 2.5)2 + (1.5 - 2.5)2 + (3.2 - 2.5)2 = 3.99
Let us compare two lines
4 (2,4)
w The second line is horizontal
3 w (4,3.2)
2.5
2
(1,2) w
w (3,1.5)
1 The smaller the sum of
squared differences
the better the fit of the
1 2 3 4
line to the data.
1

The Estimated Coefficients

To calculate the estimates of the slope and

intercept of the least squares line , use the
formulas: The regression equation that estimates
the equation of the first order linear model
is:
S xy
b1 =
S xx
ŷ = b0 + b1 x
b0 = y − b1 x

S xy =  ( x i − x )( yi − y )
n −1
 ( )
2
x −x
=
i
S xx
n −1
2

Example:

The Simple Linear Regression Line

– A car dealer wants to find Car Odometer Price

1 37388 14636
the relationship between 2 44758 14122
the odometer reading and 3 45833 14016
the selling price of used cars. 4 30862 15590
– A random sample of 100 cars is 5 31705 15568
selected, and the data recorded. 6 34010 14718
. .
Independent .
Dependent
– Find the regression line. . .
variable .
x variable y
. . .
3

The Simple Linear Regression Line

• Solution
– Solving by hand: Calculate a number of statistics

x = 36,009.45;
S xx = 43,528,690
y = 14,822.823; S xy = −2,712,511

where n = 100.
S xy −2,712,511
b1 = = = −.06232
Sx 43,528,690
b0 = y − b1 x = 14,822.82 − (−.06232)(36,009.45) = 17,067

ŷ = b 0 + b1x = 17,067 − .0623x

4

The Simple Linear Regression Line

• Solution – continued
– Using the computer
1. Scatterplot
2. Trend function
3. Tools > Data Analysis > Regression
5

The Simple Linear Regression Line

SUMMARY OUTPUT

Regression Statistics
Multiple R 0.8063
R Square 0.6501
Adjusted R Square
0.6466
Standard Error 303.1
Observations 100 yˆ = 17 ,067 − .0623 x
ANOVA
df SS MS F Significance F
Regression 1 16734111 16734111 182.11 0.0000
Residual 98 9005450 91892
Total 99 25739561

CoefficientsStandard Error t Stat P-value

Intercept 17067 169 100.97 0.0000
Odometer -0.0623 0.0046 -13.49 0.0000
6
Interpreting the Linear Regression -E
quation
17067 Odometer Line Fit Plot

16000

15000
Price

14000

0 No data 13000
Odometer

yˆ = 17 ,067 − .0623 x

The intercept is b0 = $17067. This is the slope of the line.

For each additional mile on the odometer,
the price decreases by an average of $0.0623
Do not interpret the intercept as the
“Price of cars that have not been driven”
7

Error Variable: Required Conditions

The error e is a critical part of the regression model.

Four requirements involving the distribution of e must be
satisfied.
– The probability distribution of e is normal.
– The mean of e is zero: E(e) = 0.
– The standard deviation of e is se for all values of x.
– The set of errors associated with different values of y
are all independent.
8

The Normality of e

E(y|x3)
The standard deviation remains constant,
b0 + b1x3 m3
E(y|x2)
b0 + b1x2 m2
but the mean value changes with x E(y|x1)

b0 + b1x1 m1

From the first three assumptions we have:

y is normally distributed with mean x1 x2 x3
E(y) = b0 + b1x, and a constant standard d
eviation se
9

Assessing the Model

• The least squares method will produces a

regression line whether or not there is a linear
relationship between x and y.
• Consequently, it is important to assess how well
the linear model fits the data.
• Several methods are used to assess the model.
All are based on the sum of squares for errors,
SSE.
0

Sum of Squares for Errors

– This is the sum of differences between the p
oints and the regression line.
– It can serve as a measure of how well the lin
e fits the data. SSE is defined by

n
SSE = 
i =1
( y i − ŷ i ) 2 .

– A shortcut formula

SSE =  yi2 −b0  yi − b1  xi yi

1

Standard Error of Estimate

– The mean error is equal to zero.
– If se is small the errors tend to be close to
zero (close to the mean error). Then, the
model fits the data well.
– Therefore, we can, use se as a measure of
the suitability of using a linear model.
– An estimator of se is given by se

S tan dard Error of Estimate

SSE
se =
n−2
2

Standard Error of Estimate

Example:
– Calculate the standard error of estimate for the
previous example and describe what it tells you
about the model fit.
Solution
SSE = 9, 005, 450
SSE 9, 005, 450
se = = = 303.13
n−2 98

It is hard to assess the model based

on se even when compared with the
mean value of y.
s e = 303.1 y = 14,823
3

Testing the slope

– When no linear relationship exists between t

wo variables, the regression line should be
horizontal.
q
q
q
qq q
q
q
q q
q q

Linear relationship. No linear relationship.

Different inputs (x) yield Different inputs (x) yield
different outputs (y). the same output (y).
The slope is not equal to zero The slope is equal to zero
4

Testing the Slope

We can draw inference about b1 from b1 by testing

H0: b1 = 0
H1: b1 = 0 (or < 0,or > 0)
– The test statistic is

b1 − b1 se
t= s =
( n − 1) S xx
s b1 where b1

The standard error of b1.

– If the error variable is normally distributed,
the statistic is Student t distribution with d.f.
= n-2.
5

Testing the Slope

Example
– Test to determine whether there is enough e
vidence to infer that there is a linear relation
ship between the car auction price and the
odometer reading for all three-year-old Taur
uses in the previous example .
Use a = 5%.
6
Testing the Slope,
Example
Solving by hand
– To compute “t” we need the values of b1 and sb1.

𝑏1 = −.0623
𝑠𝜀 303.1
𝑠𝑏1 = = = .00462
(99)(43,528,690)
(𝑛 − 1)𝑆𝑥𝑥
𝑏1 − 𝛽1 −.0623 − 0
𝑡= = = −13.49
𝑠𝑏1 .00462

– The rejection region is t > t.025 or t < -t.025 with n =

n-2 = 98.
Approximately, t.025 = 1.984
7

Testing the Slope

• Using the computer

Price Odometer SUMMARY OUTPUT
14636 37388
14122 44758 Regression Statistics
14016 45833 Multiple R 0.8063
15590 30862 R Square 0.6501
15568 31705 Adjusted R Square
0.6466 There is overwhelming evidence to infer
14718 34010 Standard Error 303.1 that the odometer reading affects the
14470 45854 Observations 100 auction selling price.
15690 19057
15072 40149 ANOVA
14802 40237 df SS MS F Significance F
15190 32359 Regression 1 16734111 16734111 182.11 0.0000
14660 43533 Residual 98 9005450 91892
15612 32744 Total 99 25739561
15610 34470
14634 37720 Coefficients
Standard Error t Stat P-value
14632 41350 Intercept 17067 169 100.97 0.0000
15740 24469 Odometer -0.0623 0.0046 -13.49 0.0000
8

Coefficient of determination
– To measure the strength of the linear relationship
we use the coefficient of determination.

2
S
R =
2 xy

S xx S yy
SSE
or R = 1 −2

 ( yi − y ) 2

Note that the coefficient of determination is R2

9

Coefficient of determination

• To understand the significance of this coefficient

note:
The regression model
Overall variability in y

The error
0

Coefficient of determination

y2
Two data points (x1,y1) and (x2,y2)
of a certain sample are shown.

y1 Variation in y = SSR + SSE

x1 x2
Total variation in y = Variation explained by the + Unexplained variation (error)
regression line
(y1 − y ) 2 + ( y 2 − y ) 2 = (ŷ1 − y) 2 + (ŷ 2 − y) 2 + (y1 − ŷ1 ) 2 + (y 2 − ŷ 2 ) 2
1

Coefficient of determination

• R2 measures the proportion of the variation in y t

hat is explained by the variation in x.

2
R = 1−
SSE
=
 ( y i − y ) 2 − SSE
=
SSR
 (y i − y) 2
(y − y )
i
2
 (y i − y) 2

• R2 takes on any value between zero and one.

R2 = 1: Perfect match between the line and the data points.
R2 = 0: There are no linear relationship between x and y.
2

Coefficient of determination

Example
– Find the coefficient of determination for the
used car price –odometer example.what doe
s this statistic tell you about the model?
Solution
– Solving by hand;

R = .6501
2
3

Coefficient of determination
– Using the computer
From the regression output we have
SUMMARY OUTPUT

Regression Statistics 65% of the variation in the auction

Multiple R 0.8063
selling price is explained by the
R Square 0.6501
Adjusted R Square
0.6466 variation in odometer reading. The
Standard Error 303.1 rest (35%) remains unexplained by
Observations 100 this model.
ANOVA
df SS MS F Significance F
Regression 1 16734111 16734111 182.11 0.0000
Residual 98 9005450 91892
Total 99 25739561

CoefficientsStandard Error t Stat P-value

Intercept 17067 169 100.97 0.0000
Odometer -0.0623 0.0046 -13.49 0.0000
4

Using the Regression Equation

• Before using the regression model, we need to a

ssess how well it fits the data.

• If we are satisfied with how well the model fits th

e data, we can use it to predict the values of y.
• To make a prediction we use
– Point prediction, and
– Interval prediction
5

Point Prediction

• Example
– Predict the selling price of a three-year-old Taurus wi
th 40,000 miles on the odometer.
A point prediction
ŷ = 17067− .0623x = 17067− .0623(40,000) = 14,575

– It is predicted that a 40,000 miles car would sell for $14,

575.
– How close is this prediction to the real price?
6

Interval Estimates

Two intervals can be used to discover how closely the

predicted value will match the true value of y.
– Prediction interval – predicts y for a given value of x,
– Confidence interval – estimates the average y for a
given x.

– The prediction interval – The confidence interval

1 ( xg − x ) 2 1 ( xg − x ) 2
yˆ  ta 2 se 1 + + yˆ  ta 2 se +
n  ( xi − x ) 2  (x
2
n i − x)
7

Interval Estimates

Example - continued
– Provide an interval estimate for the bidding
price on a Ford Taurus with 40,000 miles on
the odometer.
– Two types of predictions are required:
• A prediction for a specific car
• An estimate for the average price per car
8

Interval Estimates
• Solution
– A prediction interval provides the price estimate for a sin
gle car:
1 ( xg − x ) 2
yˆ  ta 2 se 1 + +
n  ( xi − x ) 2
t.025,98
Approximately

1 (40, 000 − 36, 009) 2

[17, 067 − .0623(40000)]  1.984(303.1) 1 + + = 14,575  605
100 4,309,340,310
9

Interval Estimate

Solution – continued
– A confidence interval provides the estimate
of the mean price per car for a Ford Taurus
with 40,000 miles reading on the odometer.
• The confidence interval (95%) =
1 ( x g − x)2
ŷ  t a 2 s e +
n
 ( x i − x)2

1 (40, 000 − 36, 009) 2

[17, 067 − .0623(40000)]  1.984(303.1) + = 14,575  70
100 4,309,340,310
0
The effect of the given xg on the
length of the interval
– As xg moves away from x the interval becom
es longer. That is, the shortest interval is fo
und at x.

ŷ = b 0 + b1x g
1 ( x g − x)
2

ŷ  t a 2 s e +
n (n − 1)s 2x

x
1
The effect of the given xg on
the length of the interval
– As xg moves away from x the interval becom
es longer. That is, the shortest interval is fo
und at x.

ŷ = b 0 + b1x g
1 ( x g − x)
2

ŷ  t a 2 s e +
n (n − 1)s 2x
ŷ ( x g = x + 1)
ŷ ( x g = x − 1) 1 12
ŷ  t a 2 s e +
n (n − 1)s 2x

x −1 x +1
x
( x − 1) − x = −1 ( x + 1) − x = 1
2
The effect of the given xg on the
length of the interval
– As xg moves away from x the interval becomes long
er. That is, the shortest interval is found at x.

ŷ = b 0 + b1x g
1 ( x g − x)
2

ŷ  t a 2 s e +
n (n − 1)s 2x

1 12
ŷ  t a 2 s e +
n (n − 1)s 2x

x−2 x+2 1 22
x ŷ  t a 2 s e +
n (n − 1)s 2x
( x − 2) − x = −2 ( x + 2) − x = 2
3

Regression Diagnostics - I

The three conditions required for the validity of the

regression analysis are:
– the error variable is normally distributed.
– the error variance is constant for all values o
f x.
– The errors are independent of each other.
How can we diagnose violations of these condition
s?
4

Residual Analysis

Examining the residuals (or standardized residuals), help detect

violations of the required conditions.
Example – continued:
– Nonnormality.
• Use Excel to obtain the standardized residual
histogram.
• Examine the histogram and look for a bell
shaped. diagram with a mean close to zero.
5

Residual Analysis

ObservationPredicted Price Residuals Standard Residuals

1 14736.91 -100.91 -0.33
2 14277.65 -155.65 -0.52
3 14210.66 -194.66 -0.65
4 15143.59 446.41 1.48
5 15091.05 476.95 1.58
A Partial list of
For each residual we calculate Standard residuals
the standard deviation as follows:
s ri = s e 1 − hi where Standardized residual ‘i’ =
1 ( x i − x)2 Residual ‘i’
hi = + Standard deviation
n (n − 1)s x2
6

Residual Analysis

Standardized residuals

40
30
20
10
0
-2 -1 0 1 2 More

It seems the residual are normally distributed with mean zero

7

Heteroscedasticity
When the requirement of a constant variance is violated we
have a condition of heteroscedasticity.
Diagnose heteroscedasticity by plotting the residual against
the predicted y.

+
^y
++
Residual
+
+ + + ++
+
+ + + ++ + +
+ + + +
+ + + ++ +
+ + + + y^
+ + ++ +
+ + +
+ + ++
+ + ++

The spread increases with ^y

8

Homoscedasticity
When the requirement of a constant variance is no
t violated we have a condition of homoscedasticity.
Example - continued

1000

500
Residuals

0
13500 14000 14500 15000 15500 16000
-500

-1000
Predicted Price
9
Non Independence of
Error Variables
– A time series is constituted if data were coll
ected over time.
– Examining the residuals over time, no patter
n should be observed if the errors are indep
endent.
– When a pattern is detected, the errors are sa
id to be autocorrelated.
– Autocorrelation can be detected by graphin
g the residuals against time.
0

Non Independence of Error Variables

Patterns in the appearance of the residuals over time indicates
that autocorrelation exists.
Residual Residual

+
+ ++
+
+ + +
+ + +
0 + 0 + +
+ Time Time
+ + + + + +
+
+
+ ++ +
+

Note the runs of positive residuals, Note the oscillating behavior of the
replaced by runs of negative residuals residuals around zero.
1

Outliers

An outlier is an observation that is unusually small

or large.
Several possibilities need to be investigated when
an outlier is observed:
– There was an error in recording the value.
– The point does not belong in the sample.
– The observation is valid.
Identify outliers from the scatter diagram.
It is customary to suspect an observation is an outli
er if its |standard residual| > 2
2

An outlier An influential observation

+++++++++++
+ +
+ … but, some outliers
+ +
+ +
may be very influential
+
+ + + +
+
+ +
+

The outlier causes a shift

in the regression line
3
Procedure for Regression
Diagnostics

Develop a model that has a theoretical basis.

Gather data for the two variables in the model.
Draw the scatter diagram to determine whether a linear model
appears to be appropriate.
Determine the regression equation.
Check the required conditions for the errors.
Check the existence of outliers and influential observations
Assess the model fit.
If the model fits the data, use the regression equation.
Assignment No.1
Consider the following five observations. You are to do all the parts of this exercise
using only a calculator
Assignment No.1

(g) Compute R2
(f) Conduct significant test for b2.
Assignment No.2

In this case, we want to estimate the effect of government expenditure of

infrastructure (GEI, in billion Rp) to national income (GDP, in billion Rp)
during the period of 1984-2006. The data is provided in the file assignment
2.xlsx The regression model is expressed as follow:
GDPt = β1 + β2 GEIt + et
a) Estimate b1 and b2 using least square formula.
b) Calculate the confidence interval for b1 and b2.
c) Calculate the residual from point a)
d) What is the value of R2, and give an interpretation.
e) Does GEI have a significant effect on GDP?
f) If the government targets GDP of 75000 billion Rupiah, how much infra
structure budget should be provided?