Vector Geometry

Topic and contents

Projections

Projection Revision
Ar

Definition (Projection onto a line)

School of Mathematics and Statistics

b a

MATH1151 Algebra

A/Prof Josef Dick

based on notes from A/Prof Rob Womersley

The projection of the vector a onto the non-zero vector b is OP where

Lecture 14 Projections and Least Squares

Planes in
Distances

Vector Geometry
Cross Product

AP is perpendicular to b AP b = 0 = (b a) b = 0

Length of b 6= 0 does not change OP

Projection of a onto b

lssq1.m

Projb a =

lssq2.m
L14 Projections and Least Squares
Vector Geometry
Projections

Session 1, 2015

1 / 12

Projection onto Planes 1

b1
r
O

Proj a =OP = 1 b1 + 2 b2 ,

1 , 2 R

L14 Projections and Least Squares

Vector Geometry
Projections

Session 1, 2015

2 / 12

d=

b1 a
b2 a

B symmetric, positive diagonal elements

Solve for 1 , 2
Proj a = 1 b1 + 2 b2

Find the projection of a = (6, 11, 4)T onto the plane spanned by
b1 = (1, 2, 5)T and b2 = (3, 1, 1)T .

b1 (1 b1 + 2 b2 a) = 0

Solution

b2 (1 b1 + 2 b2 a) = 0
Expand to get linear system

B=

1 (b1 b1 ) + 2 (b1 b2 ) = b1 a

1 (b2 b1 ) + 2 (b2 b2 ) = b2 a
L14 Projections and Least Squares

Linear system Bx = d




b1 b1 b1 b2
1
,
, B=
x=
b2 b1 b2 b2
2

Example

AP perpendicular to plane AP perpendicular to b1 , b2

MATH1151 (Algebra)

MATH1151 (Algebra)

(a b)
b
(b b)

Projection onto Planes 2

Plane = Span{b1 , b2 } = {x Rn : x = 1 b1 + 2 b2 , 1 , 2 R}
rA
a

b2
r
P

R3

Matlab M-files

MATH1151 (Algebra)

OP is parallel to b OP = b,

30 0
0 11

Proj a = ( 95 , 57 , 7)T
Session 1, 2015

3 / 12

MATH1151 (Algebra)

d=

36
11

= x =

65
1

B diagonal why?

L14 Projections and Least Squares

Session 1, 2015

4 / 12

Vector Geometry

Projections

Vector Geometry

Projection onto general span

Least Squares Problems

Linear system Ax = b, A Mm,n (R), b Rm , x Rn
m > n = more equations than variables over-determined

Proposition
Projection of a Rm onto = span{bj , j = 1, . . . , n} Rm is
Proj a =

n
X

Generally dont expect a solution

Residual r = Ax b Rm
Ax = b r = 0 |r| = 0 |r|2 = 0 r r = 0
P
2
Least Squares Problem Minimize |r|2 = m
j=1 rj

j bj

j=1

)T

where x = (1 , 2 , . . . , n solves n n linear system Bx = d,

b1 b1 b1 b2 b1 bn
b1 a
b2 b1 b2 b2 b2 bn
b2 a

B=
, d = ..
..
..
..

.
.
.

.
bn b1 bn b2 bn bn
bn a

|r|2 = rT r = (Ax b)T (Ax b) = xT AT Ax 2xT AT b + bT b

Normal equations

MATH1151 (Algebra)

Pn

j=1 (bj

a) bj

L14 Projections and Least Squares

Vector Geometry
Least squares problems

Session 1, 2015

AT A x = AT b

AT b Rn , square linear system

AT A Mn,n (R),

{ b1 , b2 , . . . , bn } orthogonal bi bj = 0, i 6= j B diagonal
{ b1 , b2 , . . . , bn } orthonormal also bi bi = 1 B = I
Proj a =

5 / 12

Line of best fit 1

Projection of b onto span of columns of A

Projection: r = Ax b orthogonal to columns of A
AT r = 0 AT (Ax b) = 0 AT Ax = AT b
MATH1151 (Algebra)

L14 Projections and Least Squares

Vector Geometry
Least squares problems

Example (Line of best fit to data)

n = 2 parameters , ,

Find the line y(t) = + t which best fits the data (ti , yi ), i = 1, . . . , m
in the least squares sense.

Linear system Ax = y where

1 t1
1 t2

A = . . Mm,2 (R),
.
.
. .

Line 1.81 + 0.60 t of best fit to data, ||r||22 = 553.95

9
Data
Line of best fit
8

+ ti yi ,

x=

Normal Equations AT Ax = AT y

A A=

10

12

MATH1151 (Algebra)

6 / 12

L14 Projections and Least Squares

Session 1, 2015

7 / 12

MATH1151 (Algebra)

1 1
t1 t2

i = 1, . . . , m

m > 2 data values

1 tm

1
2

Session 1, 2015

Line of best fit 2

Want line close to data points:

Least squares problems

1
tm

1
1
..
.

t1
t2
..
.

1 tm

y=

y1
y2
..
.
ym

Rm

m
X

ti
m


i=1
=
m
m
X
X

t2i
ti

L14 Projections and Least Squares

i=1

i=1

Session 1, 2015

8 / 12

Vector Geometry

Least squares problems

Vector Geometry

Line of best fit 3

Exponential fit 1

Normal Equations AT Ax = AT y


1

tm

1 1
t1 t2

Solve 2 by 2 linear system

m
X
ti
m

i=1

X
m
m
X

t2i
ti
i=1

i=1

Matlab x = A \ y,
MATH1151 (Algebra)

y1
y2
..
.

m
X

yi

i=1
= m
X
ti y i

ym

i=1

m
X

yi



i=1

= X
m

ti y i
i=1

x = (A*A) \ (A*y),

Example (MATH1151 past exam)

Total cost T of claims in million $ is

Year t
Cost T

2
2.5

3
2.8

Use a log transformation to solve this as a linear least squares problem.

Solution
Log transformation
log(T ) = log(ekt ) = log() + log(ekt ) = log() + kt
Linear in variables x = (log(), k)T

Session 1, 2015

9 / 12

Exponential fit 2

MATH1151 (Algebra)

L14 Projections and Least Squares

Vector Geometry
Least squares problems

Session 1, 2015

10 / 12

Exponential fit 3

Solution
Linear system Ax = y where

1 0
1 1
A=
1 2
1 3

Matlab lssq2.m

Normal equations AT Ax = AT y



4 6
T
T
A A=
, A y=
6 14

Cost of claims T(t) = 1.99 e0.11 t

log(2.0)
log(2.2)

y=
log(2.5)
log(2.8)

2.8
Data
Exponential fit
2.7

2.6

2.5

log(2.0)+log(2.2)+log(2.5)+log(2.8)
log(2.2)+2 log(2.5)+3 log(2.8)

Solution x = (AT A)1 (AT y)





1
14 6
0.6863
= ex1 = 1.9863
T
x=
(A y) =
=
0.1137
k = x2 = 0.1137
20 6 4
MATH1151 (Algebra)

1
2.2

T = ekt

lssq1.m

L14 Projections and Least Squares

Vector Geometry
Least squares problems

0
2.0

Model by exponential growth, with parameters , k R,

L14 Projections and Least Squares

Session 1, 2015

11 / 12

Claims T ($million)

AT y =

Least squares problems

2.4

2.3

2.2

2.1

1.9

1.8

MATH1151 (Algebra)

0.5

1.5
Time t (years)

L14 Projections and Least Squares

2.5

Session 1, 2015

12 / 12