EE103 (Winter 2006-07)

1. Vectors and matrices

vectors
inner products and linear functions
angle between vectors
matrices
matrix-vector product
matrix-matrix product
block matrix notation
cost of matrix operations
11
Vectors
n-vector x:
x =
_

_
x
1
x
2
.
.
.
x
n
_

_
also written as x = (x
1
, x
2
, . . . , x
n
)
set of n-vectors is denoted R
n
x
i
: ith element or component of x
special vectors (size follows from context):
x = 0 (zero vector): x
i
= 0, i = 1, . . . , n
x = e
i
(ith basis vector or ith unit vector ): x
i
= 1, x
k
= 0 for k = i
Vectors and matrices 12
Vector operations
scalar multiplication of a vector x with a scalar
x =
_

_
x
1
x
2
.
.
.
x
n
_

_
addition and subtraction of two n-vectors x, y
x +y =
_

_
x
1
+y
1
x
2
+y
2
.
.
.
x
n
+y
n
_

_
, x y =
_

_
x
1
y
1
x
2
y
2
.
.
.
x
n
y
n
_

_
Vectors and matrices 13
Geometrical interpretation
for n 3: x is a point with coordinates x
i
example: x = (4, 0), y = (2, 2)
x
0.75x
y
1.5y
0.75x + 1.5y
0.75 x =
_
3
0
_
, 1.5 y =
_
3
3
_
, 0.75 x + 1.5 y =
_
6
3
_
Vectors and matrices 14
Examples of vectors
sampled signal (e.g., audio signal)
a function u(t) of time, sampled with period T for n 1 periods
u(t)
t
0 T 2T 3T
(n 1)T
x
1
x
2
x
3
x
4
xn
can be represented as an n-vector with
x
1
= u(0), x
2
= u(T), . . . , x
n
= u((n 1)T)
Vectors and matrices 15
image: N N pixels, numbered 1 to N
2
1 2 3 N
N + 1 N + 2 N + 3 2N
2N + 1 2N + 2 2N + 3 3N
N
2
N
+ 1
N
2
N
+ 2
N
2
N
+ 3
N
2
grayscale image: can be represented by a vector x of size n = N
2
;
x
i
is the grayscale level of pixel i
color image: color of each pixel is given by three values (red, green, blue
intensity); image can be represented by three vectors x, y, z of size N
2
Vectors and matrices 16
Inner products
the inner product of two n-vectors x, y is dened as
x
1
y
1
+x
2
y
2
+ +x
n
y
n
notation: x
T
y
properties
(x)
T
y = (x
T
y) for scalar
(x +y)
T
z = x
T
z +y
T
z
x
T
y = y
T
x
Vectors and matrices 17
Linear functions
denition: a function f : R
n
R is linear if
f(x +y) = f(x) +f(y)
for all scalars , and all n-vectors x, y
property: f is linear if and only if it can be written as
f(x) = a
T
x = a
1
x
1
+a
2
x
2
+ +a
n
x
n
for some a (a has to be independent of x!)
Vectors and matrices 18
f(x) =
1
3
(x
1
+x
2
+x
3
) is linear: choose
a =
_
_
1/3
1/3
1/3
_
_
f(x) = x
1
is linear: choose
a =
_
_
1
0
0
_
_
f(x) = max{x
1
, x
2
, x
3
} is not linear: apply the denition to
x =
_
_
1
0
0
_
_
, y =
_
_
0
0
0
_
_
, = 1, = 1
we have f(x) = 1, f(y) = 0, f(x +y) = 0, f(x) +f(y) = 1
Vectors and matrices 19
mechanics example
unit mass, with zero position/velocity at t = 0, subject to force F(t)
F(t)
t
F(t)
0
0 1 2 3 4 5 6 7 8 9 10
x
1
x
1
x
1
x
1
x
2
x
2
x
2
x
2
x
3
x
3
x
3
x
3
x
4
x
4
x
4
x
4
x
5
x
5
x
5
x
5
x
6
x
6
x
6
x
6
x
7
x
7
x
7
x
7
x
8
x
8
x
8
x
8
x
9
x
9
x
9
x
9
x
10
x
10
x
10
x
10
F(t) = x
j
for j 1 t < j, j = 1, . . . , 10 (x is sequence of forces)
f(x) is position of mass at t = 10
g(x) is velocity of mass at t = 10
are f and g linear functions of x?
Vectors and matrices 110
t
velocity s

(t)
g(x) g(x) g(x)
0
0 1 2 3 4 5 6 7 8 9 10
t
position s(t)
f(x) f(x) f(x)
0
0 1 2 3 4 5 6 7 8 9 10
let s(t) be the position at time t; from Newtons law: s

(t) = F(t)
g(x) = s

(10) =
_
10
0
F(t) dt =
10

i=1
x
i
f(x) = s(10) =
_
10
0
s

(t) dt =
10

i=1
(
1
2
+ 10 i) x
i
i.e., both functions are linear
Vectors and matrices 111
(Euclidean) norm
the Euclidean norm of an n-vector x is
x =
_
x
2
1
+x
2
2
+ +x
2
n
=

x
T
x
properties
x = ||x for scalar
x +y x +y (triangle inequality)
x 0 and x = 0 only if x = 0
interpretation:
x measures the magnitude or length of x
x y measures the distance between x and y
Vectors and matrices 112
Angle between vectors
Cauchy-Schwarz inequality
xy x
T
y xy
angle between nonzero vectors x, y
cos =
x
T
y
xy
(with 0 )
x
y

x = (3, 0), y = (2, 2)

= arccos(
x
T
y
xy
)
= arccos(
6
3

8
)
= /4 radians
Vectors and matrices 113
angle and inner product
orthogonal vectors
= /2, x
T
y = 0
aligned vectors
= 0, x
T
y = xy
anti-aligned (opposed) vectors
= , x
T
y = xy
acute angle
< /2, x
T
y > 0
obtuse angle
> /2, x
T
y < 0
Vectors and matrices 114
exercise: use the Cauchy-Schwarz inequality to prove that

nx
n

i=1
x
i

nx
for all n-vectors x; when does equality hold?
Vectors and matrices 115
Matrices
mn-matrix A:
A =
_

_
A
11
A
12
A
1n
A
21
A
22
A
2n
.
.
.
.
.
.
.
.
.
.
.
.
A
m1
A
m2
A
mn
_

_
A
ij
are the elements or coecients
set of mn-matrices is denoted R
mn
m, n are the dimensions
special matrices (dimensions follow from context):
A = 0 (zero matrix): A
ij
= 0 for i = 1, . . . , m, j = 1, . . . , n
A = I (identity matrix): m = n, A
ii
= 1, A
ij
= 0 for i = j
Vectors and matrices 116
Matrix transpose
A
T
=
_

_
A
11
A
21
A
m1
A
12
A
22
A
m2
.
.
.
.
.
.
.
.
.
.
.
.
A
1n
A
2n
A
mn
_

_
A
T
is n m if A is mn
(A
T
)
T
= A
a square matrix A is symmetric if A = A
T
, i.e., A
ij
= A
ji
Vectors and matrices 117
Scalar multiplication and addition
scalar multiplication of an mn-matrix A with a scalar
A =
_

_
A
11
A
12
A
1n
A
21
A
22
A
2n
.
.
.
.
.
.
.
.
.
.
.
.
A
m1
A
m2
A
mn
_

_
addition of two mn-matrices A and B
A+B =
_

_
A
11
+B
11
A
12
+B
12
A
1n
+B
1n
A
21
+B
21
A
22
+B
22
A
2n
+B
2n
.
.
.
.
.
.
.
.
.
.
.
.
A
m1
+B
m1
A
m2
+B
m2
A
mn
+B
mn
_

_
Vectors and matrices 118
Matrix-vector product
product of mn-matrix A with n-vector x
Ax =
_

_
A
11
x
1
+A
12
x
2
+ +A
1n
x
n
A
21
x
1
+A
22
x
2
+ +A
2n
x
n
.
.
.
A
m1
x
1
+A
m2
x
2
+ +A
mn
x
n
_

_
dimensions must be compatible:
# columns in A = # elements in x
Vectors and matrices 119
Linear functions
f : R
n
R
m
is linear if
f(x +y) = f(x) +f(y)
for all n-vectors x, y and all scalars ,
property: f is linear if and only if it can be expressed as
f(x) = Ax
where A is mn
Vectors and matrices 120
Operator interpretation
think of a function f : R
n
R
m
in terms of its eect on x
f x y = f(x)
systems interpretation: a system with n inputs x
i
, m outputs y
i
programming interpretation: a subroutine with n input arguments x
i
and m output arguments y
i
f is linear if we can represent its action on x as a a product f(x) = Ax
Vectors and matrices 121
some examples in R
3
reverse the order of the components of x: y
1
= x
3
, y
2
= x
2
, y
3
= x
1
a linear function:
A =
_
_
0 0 1
0 1 0
1 0 0
_
_
sort the components of x in decreasing order: not linear
scale x
1
by a given number d
1
, x
2
by d
2
, x
3
by d
3
a linear function:
A =
_
_
d
1
0 0
0 d
2
0
0 0 d
3
_
_
replace each x
i
by its absolute value |x
i
|: not linear
Vectors and matrices 122
geometrical examples of linear functions f : R
2
R
2
rotate x counterclockwise over an angle
x
f(x)

f(x) = Ax with
A =
_
cos sin
sin cos
_
project x on the vector (1, 1)
x
f(x)
e x
f(e x)
f(x) = Ax with
A =
1
2
_
1 1
1 1
_
Vectors and matrices 123
sampled signal (see p.15): dene (n 1)-vector y as
y
k
=
1
T
(u(kT) u((k 1)T)) =
1
T
(x
k+1
x
k
), k = 1, . . . , n 1
(k 1)T kT (k + 1)T
u(t) x
k
x
k+1 x
k+2
slope y
k
slope y
k+1
a linear function of x: y = Ax with
A =
1
T
_

_
1 1 0 0 0
0 1 1 0 0
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
0 0 0 1 0
0 0 0 1 1
_

_
Vectors and matrices 124
Matrix-matrix product
product of mn-matrix A with n p-matrix B:
(AB)
ij
= A
i1
B
1j
+A
i2
B
2j
+ +A
in
B
nj
dimensions must be compatible: # of columns in A = # of rows in B
AB = BA in general! (even if the dimensions make sense)
_
1 0
0 1
_ _
0 1
1 0
_
=
_
0 1
1 0
_ _
1 0
0 1
_
there are exceptions, e.g., AI = I A for all square A
(AB)
T
= B
T
A
T
Vectors and matrices 125
operator interpretation: y = (AB)x = A(Bx)
PSfrag
x
B A
z = Bx
y = Az = ABx
explains why in general AB = BA :
f(x) =
_
1 0
0 1
_ _
0 1
1 0
_ _
x
1
x
2
_
swaps the two elements of x; then changes the sign of the rst element
f(x) =
_
0 1
1 0
_ _
1 0
0 1
_ _
x
1
x
2
_
changes the sign of rst element of x; then swaps the two elements
Vectors and matrices 126
Block matrix notation
B =
_
2 2
1 3
_
, C =
_
0 2 3
5 4 7
_
, D =
_
1 0

, E =
_
1 6 0

A =
_
B C
D E
_
=
_
_
2 2 0 2 3
1 3 5 4 7
1 0 1 6 0
_
_
A is a block matrix; B, C, D, E are the blocks of A
dimensions of the blocks must be compatible!
(B and D have the same number of columns; B and C have the same
number of rows, etc.)
Vectors and matrices 127
adding block matrices
_
A B
C D
_
+
_

A

B

C

D
_
=
_
A+

A B +

B
C +

C D +

D
_
dimensions must conform (i.e., matrix additions make sense)
multiplying block matrices
_
A B
C D
_ _
W X
Y Z
_
=
_
AW +BY AX +BZ
CW +DY CX +DZ
_
dimensions must conform (i.e., multiplications and additions make sense)
Vectors and matrices 128
The cost of matrix/vector operations
usually evaluated by counting the number of oating-point operations
one oating-point operation (op): +, , /, or
other common denition: one multiplication (or division) followed by
one addition (or subtraction)
example: inner product of two n-vectors x, y
prod = x(1)*y(1);
for i = 2:n
prod = prod + x(i)*y(i);
end
cost (with our denition): 2n 1 ops ( 2n for large n)
other vector-vector operations (on n-vectors):
addition, subtraction: n ops; scalar multiplication: n ops
Vectors and matrices 129
matrix-vector product: y = Ax for A mn
for i = 1:m
y(i) = A(i,1)*x(1);
for j = 2:n
y(i) = y(i) + A(i,j)*x(j);
end
end
cost: (2n 1)m ops ( 2mn for large n)
special cases:
m = n, A diagonal: n ops
m = n, A lower triangular: n(n + 1) ops
A very sparse (lots of zero coecients): #ops 2mn
Vectors and matrices 130
matrix-matrix product: C = AB if A is mn, B is n p
for i = 1:m
for j = 1:p
C(i,j) = A(i,1)*B(1,j);
for k = 2:n
C(i,j) = C(i,j) + A(i,k)*B(k,j);
end
end
end
cost: mp(2n 1) ops ( 2mnp for large n)
Vectors and matrices 131
exercise: evaluate y = ABx two ways (A and B are n n, x is a vector)
1. y = (AB)x (e.g., coded in Matlab as: C = A*B; y = C*x;)
2. y = A(Bx) (e.g., coded in Matlab as: z = B*x; y = A*z;)
both methods give the same answer, but which method is faster?
exercise: evaluate y = (I +uv
T
)x where u, v, x are n-vectors
1. A = I +uv
T
followed by y = Ax (in Matlab: y = (eye(n)+u*v)*x)
2. w = (v
T
x)u followed by y = x +w (in Matlab: y = x + (v*x)*u)
Vectors and matrices 132