An Overview of Vector Spaces

EEL205 Signals and Systems

Prof. Shankar Prakriya
Indian Institute of Technology Delhi

February 4, 2015

Prof. Shankar Prakriya

An Overview of Vector Spaces EEL205 Signals and Systems

Contents

General Motivation
Introduction to Metric Spaces
Introduction to Vector Spaces, Inner Products and Norms Hilbert and Banach Spaces
Projections
Geometric Notions

Prof. Shankar Prakriya

An Overview of Vector Spaces EEL205 Signals and Systems

General Motivation I
Geometric Notions
In 3D geometry, we are used to notions of unit-length basis
vectors, dimensionality, angle between vectors, dot product,
etc. The need for orthogonal and unit-length basis vectors is
readily apparent (extension to N length vectors).
From DSP, we can see that these notions extend to signals we use orthogonal basis signals of unit energy to represent
signals.
In communications, an optimal receiver projects the received
signal on a space spanned by the basis of the modulated
signals - the same geometric notions as with vectors are used.
These notions extend to random variables too, and will play a
pivotal role in statistical signal processing for example.

Prof. Shankar Prakriya

An Overview of Vector Spaces EEL205 Signals and Systems

Familiar Notions From Geometry I

3D Geometry
Basis vectors i1 , i2 and i3 span any point in 3D, that is
a = a1i1 + a2i2 + a3i3
It is sufficient if basis vectors are linearly independent, that is,
3
X
akik = 0 iff a1 = a2 = a3 = 0.
k=1

What happens when i1 , i2 and i3 are not independent?

Why do we prefer to use i1 , i2 and i3 to be orthogonal? When
they are orthogonal, the Grammian matrix is diagonal.
When they are of unit length & orthogonal, the Grammian
matrix is an identity matrix.
Notions generalized to vectors of larger dimensions.
Why do we write one vector as the sum of other vectors?
Prof. Shankar Prakriya

An Overview of Vector Spaces EEL205 Signals and Systems

Familiar Notions From Geometry Contd. I

3D Geometry
Dot Product of a = a1i1 + a2i2 + a3i3 and
b = b1i1 + b2i2 + b3i3 denoted by a.b or ha, bi =

3
X

ak bk

k=1

ha, bi = ||a|| ||b|| cos() where is the angle between the

vectors.
Implies that |ha, bi| ||a|| ||b|| with equality holding only
when a and b are colinear
What is the physical meaning of ||a|| cos()? It is the
component of a along b.
b
What is the physical meaning of a ||a|| cos()
?
||b||

Prof. Shankar Prakriya

An Overview of Vector Spaces EEL205 Signals and Systems

Familiar Notions From Geometry Contd. I

3D Geometry
With a = a1i1 + a2i2 + a3i3 , what is the vector in the space
spanned by i1 and i2 that is closest to a?
The above is same as projection of a on subspace
spanned by i1 and i2 .
Having an orthogonal basis for the space onto which the
vector is to be projected is useful.
3
X
||a||2 = ha, ai =
|ak |2 is the squared norm of the vector
k=1

(squared distance from origin).

Is the basis for the space unique? Does the angle between a
and b depend on the choice of basis? Do the length of the
vectors change?
Prof. Shankar Prakriya

An Overview of Vector Spaces EEL205 Signals and Systems

Metric Spaces I
The notion of metric spaces will be very useful to understand
norms, inner products and other important concepts.
Metric Spaces - Definition
A Metric d: X X R is a function that measures the
distance between elements in set X.
Properties of a Metric
1

d(x, y ) = d(y , x)

d(x, y ) 0

d(x, y ) = 0 iff x = y

d(x, z) d(x, y ) + d(y , z) triangle inequality

A metric space (X,d) is a metric d(x,y) together with a set X.

Prof. Shankar Prakriya

An Overview of Vector Spaces EEL205 Signals and Systems

Metric Spaces II
Commonly Used Metrics
For M 1 vectors x and y
d1 (x, y)

PM

d2 (x, y)

i=1
P

dp (x, y)

P

d (x, y) =

|xi yi |

M
i=1 |xi

M
i=1 |xi
maxM
i=1 |xi

yi

|2

y i |p

1

1
p

yi |

dp is referred to as Minskowski distance. The metric to use

depends on the application, and ease of use - d2 (.) is used most
often because of its desirable properties. A very large number of
metrics are in use - a list of statistical and other distance measures
can be found in wikipedia for example.

Prof. Shankar Prakriya

An Overview of Vector Spaces EEL205 Signals and Systems

Metric Spaces III

Examples
A quantizer that maps vector x to
x uses d1 (.) or d2 (.)
To measure the distance between binary codewords, one based
on the hamming distance dH can be used:
dH (x, y) =

M1
X

xi yi

(modulo 2)

i=0

Does it satisfy all the requirements of a metric?

Prof. Shankar Prakriya

An Overview of Vector Spaces EEL205 Signals and Systems

Metric Spaces IV

Some Other Distance Measures

d(x, y) = max
P i (|xi yi |) Chebyshev
i xi yi
d(x, y) =
Cosine correlation
||x||2 ||y||2
X |xi yi |
d(x, y) =
Canberra Are all of them metrics?
|xi + yi |

(1)

Prof. Shankar Prakriya

An Overview of Vector Spaces EEL205 Signals and Systems

Metric Spaces V

Signal Representation
In signal representation, d2 (.) is often used (Root mean-squared
error).
The Fourier series representation of
x(t) = sin(2kt/T
)

sin(2kt/T ) t 6= T /3
y (t) =
3
t = T /3
will be exactly the same! Convergence is in the mean-square sense.
Convergence issues arise in description of the Gibbs Phenomenon

Prof. Shankar Prakriya

An Overview of Vector Spaces EEL205 Signals and Systems

Vector Spaces I
Linear Vector Space
Linear vector space S over aa set of scalars R is a collection of
vectors, together with additive operation + and a scalar
multiplication . such that:
1

x and y S then x + y S

An additive identity 0 such that x + 0 = 0 + x = x.

For every x S there is an additive inverse y such that

x+y =0

(x + y) + z + x + (y + z)

For a and b R, a.x S, (b.x) = (a.b.)x,

(a + b)x = a.x + b.x, a.(x + y) = a.x + a.y

There is a mutltiplicative identity 1 such that 1.x = x, and

a scalar 0 R, with 0.x = 0

Prof. Shankar Prakriya

An Overview of Vector Spaces EEL205 Signals and Systems

Vector Spaces II

Some Definitions
Let S be a vector space. If V S is a subset such that V itself is
a space, then V is a subspace of S.
This notion of subspaces will be useful when we deal with Hilbert
Spaces and Projections.

Prof. Shankar Prakriya

An Overview of Vector Spaces EEL205 Signals and Systems

Vector Spaces III

Signals as vectors
Under some simple assumptions, we can treat signals as
vectors
A signal x(t) can be considered as an infinite-sized vector
Similarly, a sequence x[n] can be consdired to be an infinitely
long vector
Some issues arise with basis signals (convergence etc)
Any inifinte set of basis signals cannot span every possible
signal x(t) - hence the need for Dirichlet conditions in Fourier
Transforms

Prof. Shankar Prakriya

An Overview of Vector Spaces EEL205 Signals and Systems

Normed Vector Spaces I

Norms
For vector spaces, notions of length is natural.
For any x S, a real valued k x k is a norm is:
1

k x k is real, and 0

k x k= 0 Iff x = 0

k cx k= |c| k x k

k x + y kk x k + k y k (triangle inequality)

Prof. Shankar Prakriya

An Overview of Vector Spaces EEL205 Signals and Systems

Normed Vector Spaces II

Common Norms
l1 norm:P
k x k1 = M
i=1 |xi |

&

lp norm:
P
1
M
p p
k x kp =
|x
|
i
i=1

k x(t) k1 =

&

Rb
a

|x(t)|dt

k x(t) kp =

R

b
a

|x(t)|p dt

1
p

l norm:
k x k = maxi |xi |
&
k x(t) k = sup[a,b] |x(t)|
All these satisfy all conditions of a norm.
The norm used depends on the application.

Prof. Shankar Prakriya

An Overview of Vector Spaces EEL205 Signals and Systems

Normed Vector Spaces III

Normed Linear Space

A normed linear space is a pair (S, ||.||) - a vector space with a
norm defined.
A space is said to be complete if all points arbitrarily close also lie
within the space.

Example - the set of rational numbers is not complete since 2 is

not in the space.
A complete normed linear space is referred to as a Banach Space

Prof. Shankar Prakriya

An Overview of Vector Spaces EEL205 Signals and Systems

Inner Product I
Defintion and Properties
For vector space S with elements R, the inner product h., .i
S S R satisfies:
1

hx, yi = (hy, xi)

hcx, yi = chx, yi

hx + y, zi = hx, zi + hy, zi

hx, xi = 0 iff x = 0, hx, xi > 0

Prof. Shankar Prakriya

An Overview of Vector Spaces EEL205 Signals and Systems

Inner Product II

Hilbert Space
A complete normed linear space with an inner product (with the
norm being the induced norm) is referred to as a Hilbert Space.
Orthogonal Subspaces Let S be a vector space, and let V and W
be subspaces of S. V and W are orthogonal if every vector in V is
orthogonal to every vector in W.

Prof. Shankar Prakriya

An Overview of Vector Spaces EEL205 Signals and Systems

Inner Product III

Examples
For signals hx(t), y (t)i =

Rb
a

x(t)y (t)dt

The induced norm is therefore

Rb
k x(t) k2 = hx(t), y (t)i = a |x(t)|2 dt = Ex
P 1
An FIR filter y [n] = M
l=0 h[m]x[n M] can be viewed as an
inner product y [n] = hH x[n] = hx[n], hi where
h = [h[0], . . . , h[M 1]]T , and
x[n] = [x[n], x[n 1], . . . , x[n M + 1]]T
For matrices A and B, hA, Bi = tr (BH A)

Prof. Shankar Prakriya

An Overview of Vector Spaces EEL205 Signals and Systems

Inner Product IV
Projection Theorem
Let S be a Hilbert space, and V a subspace of S. Then for every
vector x S, there exists a unique vector vx V that is closest to
x. ||x vx || is minimized only when x vx is othogonal to V.
This theorem plays a fundamental role in communications,
statistical signal processing and many other areas.

Prof. Shankar Prakriya

An Overview of Vector Spaces EEL205 Signals and Systems

Inner Product V

Weighted Inner Product

hx, yiW = yH Wx
For the induced norm to be proper, W should be positive
definite
Rb
For signals, hx(t), y (t)iw (t) = a x(t)w (t)y (t)dt (what are
the constraints on w (t)?)

Prof. Shankar Prakriya

An Overview of Vector Spaces EEL205 Signals and Systems

Inner Product VI
Cauchy-Schwartz Inequality
|hx, yi| k x kk y k
with equality iff y = x. Defined similarly for signals.Proof...
Using this, the angle between real x and y can be understood:
cos() =

hx, yi
k x k2 k y k2

For complex vectors x and y, we use:

cos() =

Prof. Shankar Prakriya

<{hx, yi}
k x k2 k y k2

An Overview of Vector Spaces EEL205 Signals and Systems

Inner Product VII

Signals as Points in Space

1

x(t) and y (t) are orthogonal if hx(t), y (t)i = 0

Commonly used signal representations use orthogonal basis

Fourier series uses e j2kt/T which are orthogonal over

t [0, T ], but of energy T
Fourier Transform uses e jt , t = . . .

sin(t + ) can be written as the sum of sin(t) and cos(t).

What is the angle between sin(t + ) and cos(t)?

We can consider cos(t) and sin(t) as basis for signals of

the form sin(t + ), which are points in this space

Prof. Shankar Prakriya

An Overview of Vector Spaces EEL205 Signals and Systems

Inner Product VIII

Signals as Points in Space

1

Similarly, (almost all) periodic signals x(t) can be considered

as points in space spanned by e j2kt/T for k = ldots,

Similarly, (almost all) aperiodic signals x(t) can be considered

as points in space spanned by e j

The Parsevals theorems should be understood in this context:

Z T
1 P
2
|x(t)|2 dt =
FS
k= |ak |
T
Z0
Z
1
|x(t)|2 dt =
|X (j)|2 d FT
2

Prof. Shankar Prakriya

An Overview of Vector Spaces EEL205 Signals and Systems