Linear Least Squares

Linear Least Squares refers to situations where known data points (xi,yi) are fitted with a function to achieve
best fit, namely that the sum of the squares (often called 2) of the vertical distances between the points and
the fitting function is minimized. Thus, 2 = (f(xi) - yi)2. Some may think that linear least squares means fitting
a straight line to a set of data points. In general, this isnt the case. Linear Least Squares simply means that
the fitting function is linear with respect to the coefficients of the fitting function (not the known xi). Examples
are:
y = mx + b
y = ae0.4x + bx2
y = acos(x) + bsin(x)
y = ax3 + bx2 + cx + d
This tutorial focuses on practical examples of linear least squares curve fitting. Therefore, no theory or
derivations are provided. Instead, the end result is stated along with some worked examples. The objective of
linear least squares analysis is to solve the equation to the
f1 ( x1 ) L f1 ( xn ) c1 y1
right, where [c] is a matrix of the coefficients of the fitting

functions, [A] is in general a non-square n x m matrix of basis f 2 ( x1 ) L f 2 ( xn ) c2 y2
=
[A][c ] = [ y ]
functions at the given points, and [y] is the matrix of given y M
O
M M M

values. The solution is:
1 T
T
f
x
L
f
x
(
)
(
)
c
=
A
A
A
y
T
-1
T
1
m
m
n

c n y n
where [A A] A is often
referred to as the pseudoinverse A+ to make it analogous to systems of linear equations:
System of Linear Equations: [x] = [A]-1[y]
Linear Least Squares: [c] = [A+]-1[y]

[]

[]

Special case: if the fitting functions is a straight line, it can be shown that the slope m and y intercept b can be
calculated directly from the below expressions, where n is the number of points, x is the mean of xi, and y is the
mean of yi.
Example: fit several functions to:

b=

( x ) x x y
2
i

2
i

nx 2

m=

( x y ) nxy
x nx
i

2
i

1
[x] = 5
7

1.4
[ y ] = 1.1
0.2

n =3
x = (1 + 5 + 7)/3 = 13/3
y = (1.4 + 1.1 + 0.2)/3 = 2.7/3
Fitting Fcn

[y]

y = mx + b

1.4
1.1
0.2

x1
x2
x3

1
1
1

y = ae x + be 0.2x

1.4
1.1
0.2

ex1
ex2
ex3

e0.2x1
e0.2x2
e0.2x3

y = ax2 + bx + c 1.4
1.1
0.2

x 12
x 22
x 32

x1
x2
x3

x 10.05
x 20.05
x 30.05

ex1
ex2
ex3

y = ax0.05 + be x

1.4
1.1
0.2

2.7/3(1 + 25 + 49) - 13/3(1.4 + 5.5 + 1.4) 2.7(25) (13 / 3)(8.3)

= 1.689
=
(1 + 25 + 49) - 3(13/3) 2
75 13(13 / 3)
(1.4 + 5.5 + 1.4) 3(13 / 3)(2.7 / 3) 8.3 13(2.7 / 3)
=
= 0.182
m=
(1 + 25 + 49) 3(13 / 3) 2
75 13(13 / 3)
b=

[A]

[A]T

[A]
1
5
7

1
1
1

1
1

2.718 1.221
148.4 2.718
1.E+03 4.055
1
1
1

1
25
5.E+01

1
5
7

1
1.084
1.102

2.718
148.4
1097

2.72
1.22

1
1
1

1
1
1
1.00
2.72

ATA

5
1

7
1

148.41 1.E+03
2.72
4.06

25
5
1

1.08 1.E+00
148.41 1097

1.5

1.5

0.5

0.5

y = 1.285x0.05 - 0.001ex
y=

-0.5

-.06x2

+ .3x + 1.16

13
3

1E+06 4854
4854 25.33

49
7
1

75
13

3027
469
75

469
75
13

3.389
1372

1372
1E+06

75
13
3

[c]

0.054 -0.232
-0.232 1.339

-0.1821
1.6893

0.286

0.000 -0.001
-0.001 0.164

-0.0023
0.6537

0.486

0.024 -0.188 0.205

-0.188 1.500 -1.813
0.205 -1.813 3.066

-0.0625 7E-29
0.3
1.1625

0.540 -0.001
-0.001 0.000

1.2851
-0.0011

0.03

y = -0.002ex + .654ex

-0.5

-1

ATA -1

y = -0.182x + 1.69

7.9

6.9

5.9

4.9

3.9

2.9

1.9

0.9

-0.1

-1
0