M IDDLE E AST T ECHNICAL U NIVERSITY N ORTHERN C YPRUS C AMPUS

M ATH 210 S PRING 2015

Week 9: Projections and Data Fitting

All linear combinations x 1 v1 + x 2 v2 + . . . + x n vn are called the linear span of the vectors v1 , v2 , , vn . If A = [v1 v2 . . . vn ]
is the matrix whose columns consists of the vectors, v1 , v2 , , vn , then A x = x 1 v1 + x 2 v2 + . . . + x n vn . The linear span of
v1 , v2 , , vn are all the vectors of the form A x.
If for a given vector b, one can solve Ax = b for some vector x, then b is a linear combination of v1 , v2 , , vn . More
precisely, b = x 1 v1 + x 2 v2 + . . . + x n vn .
What if you cannot solve Ax = b for x? What is the next best solution? One can try the minimize the length of Ax b.
Clearly, if the minimum length is zero, then Axb = 0 is solved. Typically, minimization problems related to lengths/distances,
it is better to deal with the square of the lengths/distances.
We denote the length of a vector w by ||w||. In fact, ||w||2 = wT w. In this sense, we would like to minimize ||Ax b||2 =
(Ax b)T (Ax b).
Intuition tells us that the vector b must split as a sum, b = p + e, where p = A x for some x and e is a vector perpendicular
to all A xs.
This leads us to the following formulae/equations:
(AT A) x = AT b,
p = A x

(The Normal Equation)

T

= A (A A)

A b,

(0.1)
(0.2)

e = b p = b A (AT A)1 AT b.

(0.3)

Typically, the vector p is called the projection of the vector b onto the plane determined by the vectors v1 , v2 , , vn . The
vector x is the coefficients needed to express the vector p as a linear combination of the v1 , v2 , , vn . The vector e is
orthogonal component of the vector b which is perpendicular to the plane determined by the vectors v1 , v2 , , vn .

1 P ROJECTION .
Project the vector b on the plane determined by the vectors v1 , v2 , , vn : Calculate the vectors p, e and x . Compute the
squared error ||e||2 = eT e.


3
2
, v2 =
,
2
6

5
.
8

(a) v1 = 2 ,

b= 6 .

(b) v1 = 0 ,

b= 5 .

1
2
(e) v1 =
, v2 =
,
1
2

5
b=
.
3

3
(c) v1 =
,
2

5
b=
.
8

1
2
(f) v1 =
, v2 =
,
1
2

5
b=
.
5

(d) v1 =

b=

1
1
(g) v1 = 0 , v2 = 1 ,
1
0


1
1
0
1


(h) v1 = , v2 = ,
1
0
1
0

1
b = 1 .

1

1
1

b = .
1
1

2 A PPROXIMATE S OLUTIONS OF A M ATRIX E QUATION .

For the equations Ax = b below, write the normal equation and find the best x so that ||b Ax|| is minimized.

(a) 4 x = 8 .

(d)

(b) 7 x = 0 .

(e)

2
8
(c)
x =
.
3
7

7 2
5
x =
.
2
4
8

1
(g) 1
0

1
1

(h)
1
1

4 6
5
x =
.
6 9
3

4 6
2
(f)
x =
.
6 9
3

0
1
1 x = 1.
1
1

1
1
3
2

x = .
10
3
4
14

v1 b

HINT: Remember that AT b = v2 b is the vector of dot products of b with the columns of A
..
.

v1 v1 v1 v2

and similarly AT A = v2 v1 v2 v2 is the matrix of dot products of columns of A with each other.
..
..
..
.
.
.

3 D ATA F ITTING ( T HE L EAST S QUARES M ETHOD ).

In this problem, you will be given a data set and the a function with some unknown coefficients. The aim is a) to formulate this problem to find the best coefficients for the function and b) find the best coefficients.
a) For the following data sets and the function f (t ), formulate how you can find the best coefficients for the function
f (t ). Setup your problem in the form Ax = b so that
Each row corresponds to a data point,
The vector x is the vector of unknown coefficients in f (t ),
The vector b consists of the observed values of f (t ).
Do not solve for x.
1) f (t ) = a cos(t ) + b sin(t ) + c.
t
f (t )

0
1

-1

2) f (t ) = a cos(t ) + b sin(t ) + ce t + d ln(t ).

t
f (t )

3
4

-1

b) Same as a) above, but also solve for x.

4) f (t ) = a 1t + b.

1) f (t ) = at + bt 3 .
t
f (t )

-1
-3

0
1

1
4

t
f (t )

2) f (t ) = a + bt 2 .
t
f (t )

-1
3

0
-1

-1
3.1

0
2.9

1
2

1
-6

5) f (t ) = a cos(t ) + b sin(t ).
t
f (t )

1
4

3) f (t ) = a.
t
f (t )

1
3

1
3

0
1

-1

3
2

-7