Solving problems in inner product space

V inner product space

E, F bases of V

P change of basis matrix from E to F

1
Questions:

1. If E and F are orthonormal, what can we

say about P ?

Answer: P is orthogonal.

2. If E is orthonormal and P is orthogonal,

what can we say about F ?

Answer F is also orthonormal.

3. If F is orthonormal and P is orthogonal,

what can we say about E?

Answer E is also orthonormal,

2
Reason:

1. if E orthonormal and ~ y V , then

x, ~

h~ y i = h[~
x, ~ y ]E i = [~
x]E , [~ x]E [~
y ]E

 
u1]E [~
2. P = [~ un ]

3. ~ un orthonormal
u1 , . . . , ~
[~
u1]E , . . . , [~
un]E orthonormal

4. P orthogonal P 1 orthogonal

3
To see 1 on preceding slide, let

a1 b1
E = {~ un},
u1 , . . . , ~ x]E = ... ,
[~ y ]E = ...
[~

an bn
Then
* n n +
X X
h~ yi =
x, ~ ai ~
ui , bj ~
uj
i=1 j=1
X
= aibj h~ uj i
ui , ~
i,j
X
= aibih~ ui i
ui , ~
i
X
= aibi
i
a1 b1
.. ..
= . .
an bn

4
Rule of thumb

To solve problems in an inner product space

which involve the inner product, coordinatize
relative to an orthonormal basis!

5
Example
Extend an orthonormal set {~ uk } in V to
u1 , . . . , ~
an orthonormal basis of V .

Strategy:

1. Figure out how to do it do it in Rn.

2. Use the general (orthonormal) coordinati-

zation process.

6
Blast from the past:

How to extend an independent set {~ xk }

x1 , . . . , ~
in Rn to a basis of Rn:

form the matrix

   
x1 ~
A= ~ x1 ~
xk I = ~ xk ~e1 ~en

reduce to reduced echelon form, and take the

columns of A corresponding to the leading columns

there will be n leading columns, because the

columns of A span Rn

these columns will include the first k, because

~
x1 , . . . , ~
xk are independent

7
How to make the extended basis of Rn or-
thonormal?

Use Gram-Schmidt Process. Dont forget to

normalize after orthogonalizing.

The given vectors ~x1 , . . . , ~

xk will still survive,
because they are orthonormal.

8
Example
V = P2

Z 1
hf, gi = f (x)g(x) dx
1

Problemq1. 
Extend 3/8(1 + x), 1/ 8(1 3x)
to an orthonormal basis of P2

Problem 2.
Find the orthogonal projection of x onto
span{1 + x2, x + x2}

9
To use the coordinatization approach, we need
an orthonormal basis of P2. Heres one:
 q q 
E = 1/ 2, 3/2 x, 5/8(3x2 1)

(I got this by applying Gram-Schmidt to the

standard basis {1, x, x2}, then normalizing.)

For Problem 1, first we coordinatize: compute

the inner products of each of the vectors f, g
in the given orthonormal set with each vector
u1, u2, u3 in our orthonormal basis to get the
coordinate vectors.

After some integrating:

3/2 1/2

[f ]E = 1/2 , [g]E = 3/2

0 0

10
For example, the 2nd coordinate of [g]E is
Z 1
hg, u2i = g(x)u2(x) dx
1 s
Z 1
1 3
= (1 3x) x dx
s
1 8 2
3 1
Z
= (x 3x2) dx
s
16 1
3 2
 
= (3)
16 3

3
=
2

11
Next, let

  3/2 1/2 1 0 0
A = [f ]E [g]E I = 1/2 3/2 0 1 0

0 0 0 0 1

reduced echelon form:

1 0 3/2 1/2 0
0 1 1/2 3/2 0

0 0 0 0 1
leading columns: 1, 2, 5, so take correspond-
ing columns of A:


3/2 1/2
0
1/2 , 3/2 , 0



0 0 1

This is certainly a basis of R3, so the next

step would be to apply Gram-Schmidt to get
an orthogonal basis, but actually this basis is
already orthogonal (by accident).
12
Thus we only need to normalize. The first
two vectors are already unit vectors, since they
were orthonormal to begin with. The third
vector is also already a unit vector (again by
accident), so we already have an orthonormal
basis of R3.

Finally, translate back to P2: the desired or-

thonormal basis is
 q q 
{f, g, u3} = 1/ 2, 3/2 x, 5/8(3x2 1)

13
Actually, there is an alternative approach to
the above problem: put together the given or-
thonormal set {f, g} with any basis for ex-
ample, the standard one {1, x, x2}, and apply
Gram-Schmidt to the set {f, g, 1, x, x2}.

This seems weird at first, because were not

applying G-S to a basis, in fact were applying
it to a dependent set! But G-S will just give
the 0 vector at any step where the next vector
is in the span of the preceding ones, and so we
just discard these 0s, and get an orthogonal
basis, then we normalize to get an orthonormal
basis.

Even better, we could scale the given f, g if we

want, since we only expect orthogonality after
G-S anyway.

14
Anyway, the end result is that this alternative
approach has both pros and cons: it avoids
the coordinatization (and the need for an or-
thonormal basis of P2 to begin with), but on
the other hand it involves a significant amount
of algebraic manipulation with vectors in P2, as
opposed to (the more easily manageable) vec-
tors in R3. And it wouldnt really involve much
less work. However, its certainly a reasonable
approach.

Conclusion: sometimes it is at least as good,

maybe better, to work directly in the inner
product space rather than coordinatize, espe-
cially since coordinatizing relative to an or-
thonormal basis by hand can be a lot of work.

15
For Problem 2, again we start by coordina-
tizing: compute the inner product of each of
the vectors x, 1 + x2, x + x2 with each vector
u1, u2, u3 in our orthonormal basis to get the
coordinate vectors.

After some integrating (done on computer):

0

[x]E = 6/3

0

4 2/3
[1 + x2]E = 0


2 10/15

2/3
[x + x2]E =

6/3

2 10/15
Call these vectors ~
u, ~v1, ~v2.

16
The associated problem in R3 is to find the
orthogonal projection of ~
u onto the subspace
W spanned by ~v1, ~v2.

Recall that the general method is to find an

orthogonal basis {w ~ 2} of W , then compute
~ 1, w
2
X h~ ~ ii
u, w
projW ~
u= w
~i
i=1 hw ~ ii
~ i, w
To get the orthogonal basis, we apply the Gram-
Schmidt process to ~v1, ~v2: take

4 2/3
w
~ 1 = ~v1 = 0


2 2/15
then
h~v , w
~ i
~ 2 = ~v2 2 1 w
w ~1
hw~ ,w~ 1i
1
2/21

= 6/3

2 10/21
(done with computer)
17
Now we can compute the orthogonal projection
of ~
u onto W :
uw
~ ~1 uw
~ ~2
projW ~
u= w
~1 + w
~2
w
~ w
1 1
~
w
~ 2 w~ 2
2/24

=
7 6/24

10/12
(done with computer)

Finally, we translate back: the desired orthog-

onal projection is
     
2/24 ~ u1 + 7 6/24 ~ u2 + 10/12 ~u3
5 2 7 1
= x + x
8 8 4

18
Confession: this time it really would have
been less work to solve the problem directly
in the inner product space P2:

First use Gram-Schmidt to orthogonalize the

set {p1, p2} = {1 + x2, x + x2}:

q1 := p1
hp2, q1i 5 2
q2 := p2 q 1 = x2 + x
hq1, q1i 7 7
Next use the formula for the orthogonal pro-
jection:
hx, q1i hx, q2i
q1 + q2
hq1, q1i hq2, q2i
5 2 7 1
= x + x
8 8 4
(again all done on computer)

19
Conclusion: sometimes it is definitely bet-
ter to work directly in the inner product space
rather than coordinatizing relative to an or-
thonormal basis.

Thus, it is not always best to use the coordi-

natization method of solving problems in inner
product spaces.

Actually, this can be said about problems in

vector spaces generally: it is not always best
to use coordinatization!

The thing to remember is that the coordinati-

zation method always works, usually fairly ef-
ficiently, but it should not be regarded as a
required method!

20
Least Squares and Normal Equations

There is one more thing about orthogonal pro-

jections what we should say: there is a way to
do it in Rn without using an orthogonal basis
of the subspace!

A Mmn

W = Col A subspace of Rm

~b Rm

~c := projW ~b
approximation to ~b from W

~c = A~ x Rn
x for some ~

21
Consider the equation A~ x = ~b as a system
which we would like to solve. If the system
is consistent, we know what to do. But what
if the system is inconsistent?

Instead of just abandoning the system when it

is inconsistent, it is frequently useful to find
an approximate solution. More precisely, we
x Rn such that A~
find ~ x is as close to ~b as
possible. Namely, such that A~ x = ~c = projW ~b,
where W = Col A.

Such an ~ x is called a least squares solution

of the system A~ x = ~b (because were minimiz-
ing the distances, which means minimizing the
sum of the squares of the differences between
the coordinates).

22
Heres how we solve the least squares problem
without using an orthogonal basis of W :

The orthogonal projection ~c of ~b onto W is

characterized as the unique vector in W such
that ~b ~c is in the orthogonal complement of
W.

Thus A~ x W , that
x = ~c if and only if ~b A~
y Rn we have
is, if and only if for every ~

0 = h~b A~ y i = hAt~b AtA~

x, A~ yi
x, ~
Since At~b AtA~
x is a vector in Rn whose inner
product with every vector in Rn is 0, we must
have
At~b AtA~
x=~
0,
equivalently
AtA~
x = At~b.

23
Therefore ~x is a least squares solution of the
system A~x = ~b if and only if it is a solution of
the normal equations:

AtA~
x = At~b.
Weve seen that even if the original system
A~x = ~b might be inconsistent, the normal equa-
tions AtA~ x = At~b are always consistent.

24
Example

1 1
A = 2 0

2 1

1
~b =
1

1

x = ~b is inconsistent:
A~

1 0 0
reduce
 
A ~b
0 1 0

0 0 1

!
1 2 2
At =
1 0 1
!
9 3
AtA =
3 2
!
5
At~b =
2
25
normal equations:
! !
9 3 5
~
x=
3 2 2
Gauss-Jordan Elimination:
! !
9 3 5 reduce 1 0 4/9


3 2 2 0 1 1/3
So:

solution of normal equations

x = ~b
= least squares solution of A~
!
4/9
=
1/3
Thus:

orthogonal projection of ~b onto Col A

= best approximation to ~b from Col A

1 1 ! 7/9
4/9
= 2 0 = 8/9

1/3
2 1 11/9

26
More on extending independent sets to bases:

In Rn, another way to extend an independent

set {~ xk } to a basis is to find a basis of
x1 , . . . , ~
the orthogonal complement of the subspace
W := span{~ xk }.
x1 , . . . , ~

Even better, if {~ xk } is orthonormal and

x1 , . . . , ~
were trying to extend to an orthonormal ba-
sis of Rn, once we get a basis {~ ynk }
y1 , . . . , ~
of W all we have to do is apply the Gram-
Schmidt process to ~ y1 , . . . , ~
ynk , not to all the
vectors ~
x1 , . . . , ~
xk , ~
y1 , . . . , ~
ynk , because the ~ y s
are automatically orthogonal to the ~ xs.

Moreover, to find the basis of W , all we need

is any basis of W we do not have to use
the given orthonormal basis. For example, if
we wish we can scale the given ~ xs for conve-
nience.
27
Extend the set


1 1
1 1 1 1

,



3 1 7 2


0
1
to an orthonormal basis of R4.

First we scale the given vectors:

1 1
1 1
,

1 2


0 1
We want a basis of W , where W = Col A,
where in turn

1 1
1 1
A=

1 2


0 1

28
By the general theory,

(Col A) = Null At
So, we solve the homogeneous problem
At~
x=~ 0:
! !
1 1 1 0 reduce 1 1 0 1/3
At =

1 1 2 1 0 0 1 1/3
solution space:

x2 (1/3)x4 1 1/3
x2 1 0
= x2 + x4

(1/3)x4 0 1/3


x4 0 1
So:

basis of solution space

= basis of W


1 1/3


1
0
= ,

0 1/3





0 1

29
So to finish we need to apply Gram-Schmidt to
this latter basis (then normalize). But first we
scale the 2nd vector, getting a basis {~ u2 }
u1 , ~
of W :

1
1
~
u1 =

0


0

1
0
~
u2 =

1


3
Gram-Schmidt produces an orthogonal basis
{~v1, ~v2}, where

1
1
~v1 = ~
u1 =

0


0

1/2
u2 ~v1
~ 1/2
u2
~v2 = ~ ~v1 =

1

~v1 ~v1
3
30
Actually, for convenience we scale the second
vector ~v2, getting the following orthogonal ba-
sis of W :


1 1
1 1

,

0 2





0 6

Now we normalize:


1 1
1 1

1 1


,

0 2



2 42


0 6

31
Thus an orthonormal basis of R4 containing
the given vectors is

1 1 1 1
1 1 1 1 1 1 1 1


, ,

,

1 2 0 2

3 7 2 42

0 1 0 6

32