MTH2212 – Computational Methods and

Statistics

Curve Fitting and Interpolation

Lecture 7:
Least Squares Regression
 Objectives

 Introduction
 Linear regression
 Polynomial regression
 Multiple linear regression
 General linear least squares
 Nonlinear regression

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 2

 Curve Fitting

 Experimentation
 Data available at discrete points or times
 Estimates are required at points between the discrete values
 Curves are fit to data in order to estimate the intermediate values

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 3

 Curve Fitting

 Two methods - depending on error in data

120

 Interpolation 100

- Precise data 80

Temperature (deg F)
- Force through each data point 60

40

20

0
0 1 2 3 4 5
Time (s)

 Regression 6

5
- Noisy data f(x)
4

- Represent trend of the data 3

0
0 2 4 6 8 10 12 14 16
x

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 4

 Least Squares Regression

 Experimental Data
 Noisy (contains errors or inaccuracies)
 x values are accurate, y values are not

 Find relationship between x and y = f(x)

 Fit general trend without matching individual points
 Derive a curve that minimizes the discrepancy between the data
points and the curve  Least-squares regression

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 5

 Linear Regression: Definition

Noisy Data From 8

Experiment 7

6
x y
2.10 2.90 5

y  a0  a1 x
f(x)

6.22 3.83 4

7.17 5.98
3
10.5 5.71
2
13.7 7.74
1

0
0 2 4 6 8 10 12 14 16
x

Straight line characterizes trend without

passing through particular point
Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 6
 Linear Regression: criteria for a “best” fit

 How do we measure goodness of fit of the line to the data?

9

y5
8

7
Data points
y3
6
e3
y4
5
f(x)

e2
4 Residual
y1 y2 e = y - (a 0 + a 1x )

Regression Model
2 y = a 0 + a 1x

0
0 2 4 6 8 10 12 14 16
x

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 7

 Linear Regression: criteria for a “best” fit

 Use the curve that minimizes the residual between the data
points and the line 9

y5

Model: y  a0  a1 x
8

7
Data points
y3
6
x y e3
y4

yi  a0  a1xi
5
2.10 2.90

f(x)
e2
6.22 3.83 4 Residual
y1 y2 e = y - (a 0 + a 1x )

7.17 5.98
ei = yi  a0  a1 xi
3

Regression Model
10.5 5.71 2 y = a 0 + a 1x

13.7 7.74 1

n 2 n 0

S r =  ei =   yi  a0  a1xi  2
0 2 4 6 8 10 12 14 16
x

i=1 i=1
 Find the values of a0 and a1 that minimize Sr
Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 8
 Linear Regression: Finding a0 and a1

S r  n 2
 Minimize Sr by taking =    yi  a0  a1 xi  
a0 a0  i=1 
derivatives WRT n

a0 and a1,    2
i=1 a0
 First a0
n
  
  2   
i=1  a0 
n
  2 y
i=1
i  a0  a1 xi (  1 )
 Finally  0
n  n
na0    xi  a1   yi
i=1  i=1
Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 9
 Linear Regression: Finding a0 and a1

S r  n 2
 Minimize Sr by taking = 
 i 0 1 i 
y  a  a x 
a1 a1  i=1
derivatives WRT
a0 and a1 n

   2

 Second a1 i=1 a1

n
 
  2   
i=1  a1 
n
  2 y
i=1
i  a0  a1 xi (  xi )
 Finally  0
n   n 2 n
  xi  a0    xi  a1   xi yi
i=1  i=1  i=1

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 10

 Linear Regression: Normal equations

 Set of two simultaneous linear equations with two unknowns

( a0 and a1):

n  n
na0    xi  a1   yi
i=1  i=1

n   n 2 n
  xi  a0    xi  a1   xi yi
i=1  i=1  i=1

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 11

 Linear Regression: Solution of normal equations

 The normal equations can be solved simultaneously for:

1 n n 2 1 n n
 yi  xi   xi  xi yi
n n i=1 i=1
a0  i=1 i=1
n 2 1  n 2
 xi    xi 
i=1 n i=1 
n 1 n n
 xi yi   xi  yi
i=1 n i=1 i=1
a1 
n 2 1  n 2
 xi    xi 
i=1 n i=1 

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 12

 Example 1

 Fit a straight line to the values in the following table

x y
2.10 2.90
6.22 3.83
7.17 5.98
10.5 5.71
13.7 7.74

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 13

 Example 1 - Solution

 The intercept and the slope can be calculated with:

1 n n 2 1 n n
 yi  xi   xi  xi yi
n i=1 i=1 n i=1 i=1
a0 
n 2 1  n 2 i xi yi xi2 xiyi
 xi    xi  1 2.10 2.90 4.41 6.09
i=1 n i=1 
2 6.22 3.83 38.69 23.82
n 1 n n
 xi yi   xi  yi 3 7.17 5.98 51.41 42.88
i=1 n i=1 i=1 4 10.5 5.71 110.25 59.96
a1 
n 2 1  n 2 5 13.7 7.74 187.69 106.04
 xi    xi  Σ 39.69 26.16 392.32 238.74
i=1 n i=1 

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 14

 Example 1 - Solution

 The values of the intercept and the slope:

1 1
(26.16)(392.3)  (39.69)( 238.7)
a0  5 5  2.038
1
392.3   39.69 2
5
1
238.7  (39.69)( 26.16)
a1  5  0.4023
1
392.3   39.69 2
5

 The equation of the straight line linear regression

y  2.038  0.4023x

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 15

 Linear Regression: Quantification of error

 Suppose we have data points (xi, yi) and modeled (or

predicted) points (xi, ŷi) from the model ŷ = f(x).
 Data {yi} have two types of variations;
(i) variation explained by the model and
(ii) variation not explained by the model.
 Residual sum of squares: variation not explained by the
model n
S r   ( yi  yˆ i ) 2
i 1

 Regression sum of squares: variation explained by the

model n
St   ( y  yˆ i ) 2
i 1
St  S r
 The coefficient of determination r2 r2 
St
Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 16
 Linear Regression: Quantification of error

 For a perfect fit

Sr=0 and r=r2=1, signifying that the line explains 100% of the
variability of the data.
 For r=r2=0, Sr=St, the fit represents no improvement.
y2

y1
x1 x2
Total variation in y = Variation explained by the model + Unexplained variation (error)

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 17

 Linear Regression: Another measure of fit

 In addition to r2, r
 Define Sr
S y| x 
n2
= standard error of the estimate
- Represents the distribution of the residuals around the
regression line
- Large Sy|x􀃆large residuals
- Small Sy|x􀃆small residuals

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 18

 Example 2

 Compute the total standard deviation, the standard error of

the estimate, and the correlation coefficient for the data in
Example 1.

x y
2.10 2.90
6.22 3.83
7.17 5.98
10.5 5.71
13.7 7.74

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 19

 Example 2 - Solution

 The standard deviation is

St 14.4819
Sy    1.9028
n 1 5 1
 The standard error of the estimate is
Sr 1.9643
S y| x    0.8092 i xi yi (yi-̅y)2 (yi-a0-a1xi)2
n2 52 1 2.1 2.9 5.4382 0.0003
 The correlation coefficient r is 2 6.22 3.83 1.9656 0.5045
3 7.17 5.98 0.5595 1.1183
2St  S r 14.4819  1.9643
r    0.8644 4 10.5 5.71 0.2285 0.3049
St 14.4819 5 13.7 7.74 6.2901 0.0363
 Σ 39.69 26.16 14.4819 1.9643
r  0.8644  0.9297

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 20

 Linearization of Nonlinear Relationships

 If relationship is an exponential function y  ae bx

To make it linear, take logarithm of both sides
ln (y)  ln (a) + bx
Now it’s a linear relation between ln(y) and x

 If relationship is a power function y  ax b

To make linear, take logarithm of both sides

ln (y)  ln (a) + b ln (x)

Now it’s a linear relation between ln(y) and ln(x)

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 21

 Linearization of Nonlinear Relationships

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 22

 Polynomial Regression

 Minimize the residual between the data points and the

curve -- least-squares regression

Linear yi  a0  a1xi

Quadratic yi  a0  a1 xi  a2 xi2

Cubic yi  a0  a1 xi  a2 xi2  a3 xi3

General yi  a0  a1 xi  a2 xi2  a3 xi3    am xim

Must find values of a0 , a1, a2, … am

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 23

 Polynomial Regression

 Residual

ei = yi  (a0  a1 xi  a2 xi2  a3 xi3    am xim )

 Sum of squared residuals

n 2 n
S r =  ei =  [ y  (a0  a1 x  a2 x 2  a3 x 3    am x m )]2
i=1 i=1

 Minimize by taking derivatives

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 24

 Polynomial Regression

 Normal Equations

 n n
2
n
m   n 
 n  xi  xi   xi    y i 
 n i=1 i=1 i=1   a0   ni=1 
n n n
 x  xi
2 3
 xi   xi   a1 
m 1  x y 
 i=1 i i=1 i=1 i=1    i=1 i i 
 n    n 2 
m  2  a2 
n n n
  xi2  xi
3 4
 xi   xi     xi yi 
 i=1 i=1 i=1 i=1     i=1 
       am    
n m n
m 1
n
m 2
n
2m  n m 
  xi  xi  xi   xi    xi yi 
i=1 i=1 i=1 i=1  i=1 

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 25

 Example 3

 Fit a third-order polynomial to the data given in the Table

below

x 0 1.0 1.5 2.3 2.5 4.0 5.1 6.0 6.5 7.0 8.1 9.0
y 0.2 0.8 2.5 2.5 3.5 4.3 3.0 5.0 3.5 2.4 1.3 2.0
x 9.3 11.0 11.3 12.1 13.1 14.0 15.5 16.0 17.5 17.8 19.0 20.0
y -0.3 -1.3 -3.0 -4.0 -4.9 -4.0 -5.2 -3.0 -3.5 -1.6 -1.4 -0.1

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 26

 Example 3 - Solution

 n n
2
n
3  n 
 n  xi  xi  i
x   i 
y
 n i=1 i=1 i=1   ni=1 
4  0 
n n n a
 x  xi
2 3
 xi  xi  a   x y 
 i=1 i i=1 i=1 i=1   1   i=1 i i   a0    0.3593
n   n 
n n n
5 a2  a   2.3051 
  xi2  xi
3 4
 xi  xi   
2
  xi yi  1
    
i=1 i=1 i=1 i=1   a3  i=1  a2   0.3532
n 3 n
4
n
5
n
6 n 3  a   0.0121 
  xi  xi  xi  i
x   xi yi   3  
i=1 i=1 i=1 i=1  i=1 

 24 229.6 3060.2 46342.8   a0    1.30 

 229.6 3060.2 46342.8 752835.2   a1    316.9 
     
 3060.2 46342.8 752835.2 12780147.7  a 2    6037.2 
46342.8 752835.2 12780147.7 223518116.8  a   9943.36
  3   

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 27

 Example 3 - Solution

Regression Equation
y = - 0.359 + 2.305x - 0.353x2 + 0.012x3
6

2
f(x)

-2

-4

-6
0 5 10 15 20 25
x

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 28

 Multiple Linear Regression

 y = a0 + a1x1 + a2x2 + e
 Again very similar.
 Minimize e
 Polynomial and
multiple regression fall
within definition of
General Linear Least
Squares.

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 29

 General Linear Least Squares

y  a0 z0  a1 z1  a2 z 2    am z m  e
z0 , z1,  , zm are m  1 basis functions
Y    Z  A   E
 Z   matrix of the calculated values of the basis functions
at the measured values of the independent variable
 Y  observed valued of the dependent variable
 A  unknown coefficients
 E  residuals
2 Minimized by taking its partial
n  m  derivative w.r.t. each of the
S r    yi   a j z ji  coefficients and setting the
i 1  j 0  resulting equation equal to zero
Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 30
 Nonlinear Regression

 Not all equations can be broken down into General Linear

ax
Least Squares model i.e. y  a0 (1  e 1 )  e
 Solve with nonlinear least squares using iterative methods
like Gauss-Newton method
 Equation could possibly be transformed into linear form
 Caveat: when fitting transformed data you minimize the residuals of the
data you are working with
 May not give you the exact same fit as nonlinear regression on
untransformed data

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 31

 Nonlinear Regression

y  a0 (1  e a1 x )  e

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 32