Vector Arithmetic

Prelude
• A pattern of activation in a NN is a vector

• A set of connection weights between units is
a matrix
• Vectors and matrices have well-understood
mathematical and geometric properties
• Very useful for understanding the properties
of NNs
Operations on Vectors and
Matrices
Outline
1) The Players: Scalars, Vectors and Matrices
2) Vectors, matrices and neural nets
3) Geometric Analysis of Vectors
4) Multiplying Vectors by Scalars
5) Multiplying Vectors by Vectors
a) The inner product (produces a scalar)
b) The outer product (produces a matrix)
6) Multiplying Vectors by Matrices
7) Multiplying Matrices by Matrices
Scalars, Vectors and Matrices
1) Scalar: A single number (integer or real)
2) Vector: An ordered list of scalars
[ 1 2 3 4 5 ] [ 0.4 1.2 0.07 8.4 12.3 ] [ 12 10 ] [ 2 ]
[ 1 2 3 4 5 ] [ 0.4 1.2 0.07 8.4 12.3 ] [ 12 10 ] [ 2 ]
[ 12 10 ] ≠ [ 10 12 ]
[ 1 2 3 4 5 ] [ 0.4 1.2 0.07 8.4 12.3 ] [ 12 10 ] [ 2 ]
Row vectors
[ 1 2 3 4 5 ] [ 0.4 1.2 0.07 8.4 12.3 ] [ 12 10 ] [ 2 ]
Row vectors
1 1.5
12.0
Column Vectors 2
0.3
3 17.1
4 6.2
5
3) Matrix: An ordered list of vectors:
1 2 6 1 7 8
2 5 9 0 0 3
3 1 5 7 6 3
2 7 9 3 3 1
1 2 6 1 7 8
2 5 9 0 0 3 Row vectors
3 1 5 7 6 3
2 7 9 3 3 1
1 2 6 1 7 8
2 5 9 0 0 3 Column vectors
3 1 5 7 6 3
2 7 9 3 3 1
1 2 6 1 7 8 Matrices are indexed (row, column)
2 5 9 0 0 3
M=
3 1 5 7 6 3
2 7 9 3 3 1
2 5 9 0 0 3 M(1,3) = 6 (row 1, column 3)
M=
3 1 5 7 6 3
2 7 9 3 3 1
2 5 9 0 0 3 M(1,3) = 6 (row 1, column 3)
M=
3 1 5 7 6 3 M(3,1) = 3 (row 3, column 1)
2 7 9 3 3 1
Variable Naming Conventions
1) Scalars: Lowercase, italics
x, y, z…
2) Vectors: Lowercase, bold
u, v, w…
• Matrices: Uppercase, bold
M, N, O …
• Constants: Greek
, , , ,  …
Transposing Vectors
If u is a row vector…
u=[12345]
…then u’ (“u-transpose”) is a column vector
1
2
u’ = 3
4
5
… and vice-versa.
Transposing Vectors
u=[12345]
1
2 Why in the world
u’ = 3 would I care??
4
5
… and vice-versa.
You
Transposing Vectors
u=[12345]
1
2
u’ = 3 Answer: It’ll matter
4 when we come to
5 vector multiplication.
… and vice-versa.
Transposing Vectors
u=[12345]
1
2
u’ = 3 OK.
4
5
… and vice-versa.
Vectors, Matrices & Neural Nets
j1 j2 j3 Input units, j
i1 i2 Output units, i
w11
w23 Connection
w12 w22 w13 weights, wij
w21
w11
w23 Connection
w21
0.2 0.9 0.5 Input units, j
The activations of the input nodes can be

represented as a 3-dimensional vector:
j = [ 0.2 0.9 0.5 ]
1.0 0.0 Output units, i
w11
w23 Connection
w21
The activations of the output nodes can be

represented as a 2-dimensional vector:
i = [ 1.0 0.0 ]
w11
0.1
w23 Connection
w12
1.0 w22 0.2
w13 weights, wij
w21
The weights leading into any output node can

be represented as a 3-dimensional vector:
w1j = [ 0.1 1.0 0.2 ]
w11
0.1
w23
-0.9 Connection
w12
1.0 w22
0.1 0.2
w13 weights, wij
w21
1.0
The complete set of

weights can be 0.1 1.0 0.2
represented as a 3 W=
1.0 0.1
(row) X 2 (column) -0.9
matrix:
w11
0.1
w23
-0.9 Connection
w12
1.0 w22
0.1 0.2
w13 weights, wij
w21
1.0
Why in the world
j1 j2 j3 Input
would units, j
I care??
The complete set of

weights can be 0.1 1.0 0.2
represented as a 2 W=
1.0 0.1
(row) X 3 (column) -0.9
matrix:
1. Because the
mathematics of vectors
and matrices is well-
understood.
2. Because vectors have a
very useful geometric
interpretation. Why in the world
would I care??
3. Because Matlab
“thinks” in vectors and
matrices.
W
4. Because you are going
to have to learn to think
in Matlab.
1. Because the
mathematics of vectors
and matrices is well-
understood.
2. Because vectors have a
very useful geometric
interpretation. OK.
3. Because Matlab
“thinks” in vectors and
matrices.
4. Because you are going
to have to learn to think
in Matlab.
Geometric Analysis of Vectors
Dimensionality: The number of numbers in a vector
x2 .3
.5 x2
.5
.2
x3 x3
.2
x1 x1
.3
Vector as a point Vector as arrow
x2 .3
.5 x2
.5
.2
x3 x3
.2
x1 x1
.3
Vector as a point Vector as arrow
Implications for neural networks
• Auto-associative nets
• State of activation at time t is a vector (a point in a space)
• As activations change, vector moves through that space
• Will prove invaluable in understanding Hopfield nets
• Layered nets (“perceptrons”)

• Input vectors activate output vectors
• Points in input space map to points in output space
• Will prove invaluable in understanding perceptrons and back-
propagation learning
Multiplying a Vector by a Scalar
[5 4]*2=
Multiplying a Vector by a Scalar
5 10
[ 5 4 ] * 2 = [ 10 8 ]
Lengthens the vector but does not change its orientation
Adding a Vector to a Scalar
[5 4]+2=
Adding a Vector to a Scalar
[ 5 4 ] + 2 = NAN
Is Illegal.
Adding a Vector to a Vector
6
4
3 5
[5 4]+[3 6]=
Adding a Vector to a Vector
10
6
4
3 5 8
[ 5 4 ] + [ 3 6 ] = [ 8 10 ]
Forms a parallelogram.
Multiplying a Vector by a Vector 1:
The Inner Product (aka “Dot Product”)
If u and v are both row vectors of the same

dimensionality…
u=[1 2 3]
v=[4 5 6]

dimensionality…
u=[1 2 3]
v=[4 5 6]
… then the product
u·v=

dimensionality…
u=[1 2 3]
v=[4 5 6]
u · v = NAN
Is undefined.

dimensionality…
u=[1 2 3]
Huh??
v=[4 5 6] Why??
That’s BS!
u · v = NAN
Is undefined.
I told you you’d eventually care about transposing

vectors…
?
• The Mantra: “Rows by Columns”

• Multiply rows (or row vectors) by columns (or
column vectors)

column vectors)
u=[1 2 3]
v=[4 5 6]

column vectors)
u=[1 2 3] 4
v=[4 5 6] v’ = 5
6

column vectors)
u=[1 2 3] 4
v’ = 5
6
u · v’ = 32

column vectors) v’
1 4 Imagine rotating your row
u=[1 2 3]
vector into a (pseudo)
2 5
column vector…
3 6
u · v’ = 32

Now multiply
u=[1 2 3] 1 4 4
corresponding
2 5 elements and
3 6 add up the
u · v’ = 32 products…

Now multiply
u=[1 2 3] 1 4 4
corresponding
2 5 10 elements and
3 6 add up the

Now multiply
u=[1 2 3] 1 4 4
corresponding
2 5 10 elements and
3 6 18 add up the

Now multiply
u=[1 2 3] 1 4 4
corresponding
2 5 10 elements and
3 6 18 add up the
u · v’ = 32 32 products…
• Inner product is commutative as long as you

transpose correctly
v’ v u’
u=[1 2 3] 4 4 4 1 4
5 10 5 2 10
v=[4 5 6]
6 18 6 3 18
u · v’ = 32 32 32
v · u’ = 32
The Inner (“Dot”) Product
• In scalar notation…
u v’
d 1 4 4
u  v'   u i v i 5
2 10
i 1
3 6 18
32
d
u  v'   u i v i
i 1
• Remind you of…

d
ni   wij a j … the net input to a unit
j 1
d
u  v'   u i v i
i 1
• Remind you of…

d
nij   wij a j … the net input to a unit
j 1
• In vector notation…
n  w a
What Does the Dot Product “Mean”?
• Consider u  u’
u = [ 3, 4 ]
3
u u’
3 3 9
u = [ 3, 4 ]
4 4 16
25
3
u u’
3 3 9
u = [ 3, 4 ]
4 4 16
25
u  u  u'
5
u   ui 2
4 
3
u u’
3 3 9
u = [ 3, 4 ]
4 4 16
25
u  u  u'
5
u   ui 2
4 
True for vectors of any
3 dimensionality
• So: u  u  u'

• What about u  v where u  v?

u v
Well… cos( uv ) 
u v

u v
Well… cos( uv ) 
u v
… and cos(uv) is a length-invariant measure of the
similarity of u and v

• What about u  v where u  v? cos( uv )  u  v

u v
cos(uv) is a length-invariant measure of the similarity of u and v
V’ = [ 1, 1 ]

U = [ 1, 0 ]

u v
V’ = [ 1, 1 ] U  V = (1 * 1) +
(1 * 0) = 1

uv = 45º; cos(uv) = .707
U = [ 1, 0 ]

u v
V’ = [ 1, 1 ] U  V = (1 * 1) +
(1 * 0) = 1

uv = 45º; cos(uv) = .707
U = [ 1, 0 ]
1
cos( uv ) 
u v

u v
V’ = [ 1, 1 ]
||u|| = sqrt(1) = 1
||v|| = sqrt(2) = 1.414

uv = 45º; cos(uv) = .707
U = [ 1, 0 ]
1
cos( uv ) 
u v

u v
V’ = [ 1, 1 ]
||u|| = sqrt(1) = 1
||v|| = sqrt(2) = 1.414

uv = 45º; cos(uv) = .707
U = [ 1, 0 ]
1
cos( uv )   .707
1*1.414

u v
V’ = [ 0, 1 ]

uv = 90º; cos(uv) = 0
U = [ 1, 0 ]

u v

uv = 270º; cos(uv) = 0

U = [ 1, 0 ]
V’ = [ 0, -1 ]

u v

uv = 180º; cos(uv) = -1
U = [ 1, 0 ]
V’ = [ -1,0 ]

u v

uv = 0º; cos(uv) = 1
U = [ 1, 0 ]
V’ = [ 2.2,0 ]

u v
In general…
cos(uv)  -1…1 
True regardless of dimensionality

u v
To see why, consider the cosine expressed in scalar notation…

cos( uv ) 
u v i i
 u v2
i
2
i

u v
… and compare it to the equation for the correlation coefficient…

cos( uv ) 
u v i i
r(u,v) 
 (u  u )(v  v )
i i
 u v2
i
2
i  (u  u )  (v  v )
i
2
i
2


u v

cos( uv ) 
u v i i
r(u,v) 
 (u  u )(v  v )
i i
 u v2
i
2
i  (u  u )  (v  v )
i
2
i
2
if u and v have means of zero, then cos(uv) = r(u,v)


u v

cos( uv ) 
u v i i
r(u,v) 
 (u  u )(v  v )
i i
 u v2
i
2
i  (u  u )  (v  v )
i
2
i
2
if u and v have means of zero, then cos(uv) = r(u,v)

The cosine is a special case of the correlation coefficient!

… and let’s compare the cosine to the dot product…
u v
cos( uv ) 
u v


u v
cos( uv ) 
u v
If u and v have lengths of 1, then the dot product is equal to the cosine.


u v
cos( uv ) 
u v
If u and v have lengths of 1, then the dot product is equal to the cosine.
The dot product is a special case of the cosine, which is a special case of
the correlation coefficient, which is a measure of vector similarity!

ni   wij a j  w i a
j
• The most common input rule is a dot product between unit i’s
vector of weights and the activation vector on the other end
• Such a unit is computing the (biased) similarity between what
it expects (wi) and what it’s getting (a).
• It’s activation is a positive function of this similarity
j
ai asymptotic
ni
j
ai Step (BTU)
ni
j
ai logistic
ni
The Outer Product
The two vectors need not have the same dimensionality.
Same Mantra: Rows by Columns.
This time, multiply a column vector by a row vector:
The Outer Product
M = u’ * v
1
u’ = v=[4 5 6]
2
The Outer Product
M = u’ * v
1
u’ = v=[4 5 6] M=
2
The Outer Product
M = u’ * v
1
u’ = v=[4 5 6] M=
2
Row 1
The Outer Product
M = u’ * v
1
u’ = v=[4 5 6] M=
2
Row 1 times column 1

The Outer Product
M = u’ * v
1 4
u’ = v=[4 5 6] M=
2
Row 1 times column 1 goes into row 1, column 1

The Outer Product
M = u’ * v
1 4 5
u’ = v=[4 5 6] M=
2

The Outer Product
M = u’ * v
1 4 5 6
u’ = v=[4 5 6] M=
2

The Outer Product
M = u’ * v
1 4 5 6
u= v=[4 5 6] M=
2 8

The Outer Product
M = u’ * v
1 4 5 6
u’ = v=[4 5 6] M=
2 8 10

The Outer Product
M = u’ * v
1 4 5 6
u’ = v=[4 5 6] M=
2 8 10 12

The Outer Product
M = u’ * v
A better way to visualize it…
v=[4 5 6]
u’ = 1 4 5 6
=M
2 8 10 12
The Outer Product
Outer product is not exactly commutative…
M = u’ * v M = v’ * u
v=[4 5 6] u=[1 2]
1 4 5 6 4 4 8
u’ = =M v’ = 5
2 8 10 12 5 10
6 6 12
Multiplying a Vector by a Matrix
• Same Mantra: Rows by Columns
Rows by Columns
A row vector:
[ .2 .6 .3 .7 .9 .4 .3 ]
Rows by Columns
A row vector: A matrix:
[ .2 .6 .3 .7 .9 .4 .3 ] .3 .4 .8 .1 .2 .3
.5 .2 0 .1 .5 .2
.1 .1 .9 .2 .5 .3
.2 .4 .1 .7 .8 .5
.9 .9 .2 .5 .3 .5
.4 .1 .2 .7 .8 .2
.1 .2 .2 .5 .7 .2
Rows by Columns
[ .2 .6 .3 .7 .9 .4 .3 ] .3 .4 .8 .1 .2 .3
.5 .2 0 .1 .5 .2
.1 .1 .9 .2 .5 .3
Multiply rows
.2 .4 .1 .7 .8 .5
.9 .9 .2 .5 .3 .5
.4 .1 .2 .7 .8 .2
.1 .2 .2 .5 .7 .2
Rows by Columns
[ .2 .6 .3 .7 .9 .4 .3 ] .3 .4 .8 .1 .2 .3
.5 .2 0 .1 .5 .2
.1 .1 .9 .2 .5 .3
Multiply rows by columns
.2 .4 .1 .7 .8 .5
.9 .9 .2 .5 .3 .5
.4 .1 .2 .7 .8 .2
.1 .2 .2 .5 .7 .2
Each such multiplication is a simple dot product

[ .2 .6 .3 .7 .9 .4 .3 ] .3 .4 .8 .1 .2 .3
.5 .2 0 .1 .5 .2
.1 .1 .9 .2 .5 .3
.2 .4 .1 .7 .8 .5
.9 .9 .2 .5 .3 .5
.4 .1 .2 .7 .8 .2
.1 .2 .2 .5 .7 .2

[ .2 .6 .3 .7 .9 .4 .3 ] .3 .4 .8 .1 .2 .3
.5 .2 0 .1 .5 .2
Make a proxy column .1 .1 .9 .2 .5 .3
vector… .2 .4 .1 .7 .8 .5
.9 .9 .2 .5 .3 .5
.4 .1 .2 .7 .8 .2
.1 .2 .2 .5 .7 .2

[ .2 .6 .3 .7 .9 .4 .3 ] .2 .3 .4 .8 .1 .2 .3
.6 .5 .2 0 .1 .5 .2
.3 .1 .1 .9 .2 .5 .3
.7 .2 .4 .1 .7 .8 .5
.9 .9 .9 .2 .5 .3 .5
.4 .4 .1 .2 .7 .8 .2
.3 .1 .2 .2 .5 .7 .2

[ .2 .6 .3 .7 .9 .4 .3 ] .2 .3 .4 .8 .1 .2 .3
.6 .5 .2 0 .1 .5 .2
Now compute the .3 .1 .1 .9 .2 .5 .3
dot product of the .2 .4 .1 .7 .8 .5
.7
(proxy) row vector
.9 .9 .9 .2 .5 .3 .5
with each column of
the matrix… .4 .4 .1 .2 .7 .8 .2
.3 .1 .2 .2 .5 .7 .2
[ 1.5 ]

[ .2 .6 .3 .7 .9 .4 .3 ] .2 .3 .4 .8 .1 .2 .3
.6 .5 .2 0 .1 .5 .2
.3 .1 .1 .9 .2 .5 .3
.7 .2 .4 .1 .7 .8 .5
.9 .9 .9 .2 .5 .3 .5
.4 .4 .1 .2 .7 .8 .2
.3 .1 .2 .2 .5 .7 .2
[ 1.5 1.4 ]

[ .2 .6 .3 .7 .9 .4 .3 ] .2 .3 .4 .8 .1 .2 .3
.6 .5 .2 0 .1 .5 .2
.3 .1 .1 .9 .2 .5 .3
.7 .2 .4 .1 .7 .8 .5
.9 .9 .9 .2 .5 .3 .5
.4 .4 .1 .2 .7 .8 .2
.3 .1 .2 .2 .5 .7 .2
[ 1.5 1.4 0.8 ]

[ .2 .6 .3 .7 .9 .4 .3 ] .2 .3 .4 .8 .1 .2 .3
.6 .5 .2 0 .1 .5 .2
.3 .1 .1 .9 .2 .5 .3
.7 .2 .4 .1 .7 .8 .5
.9 .9 .9 .2 .5 .3 .5
.4 .4 .1 .2 .7 .8 .2
.3 .1 .2 .2 .5 .7 .2
[ 1.5 1.4 0.8 1.5 ]

[ .2 .6 .3 .7 .9 .4 .3 ] .2 .3 .4 .8 .1 .2 .3
.6 .5 .2 0 .1 .5 .2
.3 .1 .1 .9 .2 .5 .3
.7 .2 .4 .1 .7 .8 .5
.9 .9 .9 .2 .5 .3 .5
.4 .4 .1 .2 .7 .8 .2
.3 .1 .2 .2 .5 .7 .2
[ 1.5 1.4 0.8 1.5 1.9 ]

[ .2 .6 .3 .7 .9 .4 .3 ] .2 .3 .4 .8 .1 .2 .3
.6 .5 .2 0 .1 .5 .2
.3 .1 .1 .9 .2 .5 .3
.7 .2 .4 .1 .7 .8 .5
.9 .9 .9 .2 .5 .3 .5
.4 .4 .1 .2 .7 .8 .2
.3 .1 .2 .2 .5 .7 .2
[ 1.5 1.4 0.8 1.5 1.9 1.2 ]
[ .2 .6 .3 .7 .9 .4 .3 ] .3 .4 .8 .1 .2 .3
.5 .2 0 .1 .5 .2
.1 .1 .9 .2 .5 .3
.2 .4 .1 .7 .8 .5
.9 .9 .2 .5 .3 .5
.4 .1 .2 .7 .8 .2
.1 .2 .2 .5 .7 .2
The result is a row
vector with as many
columns (dimensions) [ 1.5 1.4 0.8 1.5 1.9 1.2 ]
as the matrix (not the
vector)
[ .2 .6 .3 .7 .9 .4 .3 ] .3 .4 .8 .1 .2 .3
.5 .2 0 .1 .5 .2
.1 .1 .9 .2 .5 .3
7-dimensional vector .2 .4 .1 .7 .8 .5
.9 .9 .2 .5 .3 .5
.4 .1 .2 .7 .8 .2
.1 .2 .2 .5 .7 .2
[ 1.5 1.4 0.8 1.5 1.9 1.2 ]

[ .2 .6 .3 .7 .9 .4 .3 ] .3 .4 .8 .1 .2 .3
.5 .2 0 .1 .5 .2
.1 .1 .9 .2 .5 .3
.9 .9 .2 .5 .3 .5
.4 .1 .2 .7 .8 .2
.1 .2 .2 .5 .7 .2
6-dimensional vector [ 1.5 1.4 0.8 1.5 1.9 1.2 ]

[ .2 .6 .3 .7 .9 .4 .3 ] .3 .4 .8 .1 .2 .3
.5 .2 0 .1 .5 .2
.1 .1 .9 .2 .5 .3
.9 .9 .2 .5 .3 .5
7 (rows) X 6 (columns) matrix .4 .1 .2 .7 .8 .2
.1 .2 .2 .5 .7 .2

NOT Commutative!

[ .2 .6 .3 .7 .9 .4 .3 ] .3 .4 .8 .1 .2 .3
.5 .2 0 .1 .5 .2
.1 .1 .9 .2 .5 .3
.9 .9 .2 .5 .3 .5
7 (rows) X 6 (columns) matrix .4 .1 .2 .7 .8 .2
.1 .2 .2 .5 .7 .2

Multiplying a Matrix by a Matrix
• The Same Mantra: Rows by Columns
Rows by Columns
A 2 X 3 matrix A 3 X 2 matrix
1 2
1 2 3
1 2
1 2 3
1 2
Row 1 X Column 1
1 1 2 Column 1
Row 1 1 2 3
2 1 2
1 2 3
3 1 2
(proxy)
Row 1 X Column 1
1 1 2 Column 1
Row 1 1 2 3
2 1 2
1 2 3
3 1 2
Result = 6
Row 1 X Column 1
1 1 2 Column 1
Row 1 1 2 3
2 1 2
1 2 3
3 1 2
Result = 6
Place the result in

row 1,
Row 1 X Column 1
1 1 2 Column 1
Row 1 1 2 3
2 1 2
1 2 3
3 1 2
Result = 6
Place the result in

row 1, column 1
Row 1 X Column 1
1 1 2 Column 1
Row 1 1 2 3
2 1 2
1 2 3
3 1 2
Result = 6
Place the result in 6

row 1, column 1 of
a new matrix…
Row 1 X Column 2
1 1 2 Column 2
Row 1 1 2 3
2 1 2
1 2 3
3 1 2
Result = 12
6
Row 1 X Column 2
1 1 2 Column 2
Row 1 1 2 3
2 1 2
1 2 3
3 1 2
Result = 12
Place the result in 6 12

row 1, column 2 of
the new matrix…
Row 2 X Column 1
1 1 2 Column 1
1 2 3
Row 2 2 1 2
1 2 3
3 1 2
Result = 6
6 12
Row 2 X Column 1
1 1 2 Column 1
1 2 3
Row 2 2 1 2
1 2 3
3 1 2
Result = 6

row 2, column 1 of
the new matrix… 6
Row 2 X Column 2
1 1 2 Column 2
1 2 3
Row 2 2 1 2
1 2 3
3 1 2
Result = 12
6 12
6
Row 2 X Column 2
1 1 2 Column 2
1 2 3
Row 2 2 1 2
1 2 3
3 1 2
Result = 12

row 2, column 2 of
the new matrix… 6 12
So…
1 2
1 2 3
1 2 3 * 1 2
1 2
6 12
=
6 12
A 2 X 2 matrix
So…
1 2
1 2 3
1 2 3 * 1 2
1 2
The result has
the same
number of
6 12
rows as the
first matrix… =
6 12
A 2 X 2 matrix
So…
1 2
1 2 3
1 2 3 * 1 2
1 2
The result has …and the same
the same number of columns
number of as the second.
6 12
rows as the
first matrix… =
6 12
A 2 X 2 matrix
…and…
1 2
1 2 3
1 2 3 * 1 2
1 2
…and the
number of
columns in the
6 12
first matrix…
=
6 12
A 2 X 2 matrix
…and…
1 2
1 2 3
1 2 3 * 1 2
1 2
…and the …must be equal to
number of the number of rows
columns in the in the second.
6 12
first matrix…
=
6 12
A 2 X 2 matrix
This is basic (default) matrix
multiplication.
There’s other more complicated stuff, too.
You (probably) won’t need it for this class.

Vector Arithmetic

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Vector Arithmetic

Uploaded by

Copyright:

Available Formats

Prelude

• A pattern of activation in a NN is a vector

0.2 0.9 0.5 Input units, j

The activations of the input nodes can be

1.0 0.0 Output units, i

The activations of the output nodes can be

The weights leading into any output node can

The complete set of

The complete set of

• Layered nets (“perceptrons”)

If u and v are both row vectors of the same

If u and v are both row vectors of the same

If u and v are both row vectors of the same

If u and v are both row vectors of the same

I told you you’d eventually care about transposing

• The Mantra: “Rows by Columns”

• The Mantra: “Rows by Columns”

• The Mantra: “Rows by Columns”

• The Mantra: “Rows by Columns”

• The Mantra: “Rows by Columns”

• The Mantra: “Rows by Columns”

• The Mantra: “Rows by Columns”

• The Mantra: “Rows by Columns”

• The Mantra: “Rows by Columns”

• Inner product is commutative as long as you

• Remind you of…

• Remind you of…

• What about u  v where u  v?

• What about u  v where u  v?

• What about u  v where u  v?

• What about u  v where u  v? cos( uv )  u  v

• What about u  v where u  v? cos( uv )  u  v

• What about u  v where u  v? cos( uv )  u  v

• What about u  v where u  v? cos( uv )  u  v

• What about u  v where u  v? cos( uv )  u  v

• What about u  v where u  v? cos( uv )  u  v

• What about u  v where u  v? cos( uv )  u  v

uv = 270º; cos(uv) = 0

• What about u  v where u  v? cos( uv )  u  v

• What about u  v where u  v? cos( uv )  u  v

• What about u  v where u  v? cos( uv )  u  v

• What about u  v where u  v? cos( uv )  u  v

To see why, consider the cosine expressed in scalar notation…

• What about u  v where u  v? cos( uv )  u  v

… and compare it to the equation for the correlation coefficient…

• What about u  v where u  v? cos( uv )  u  v

… and compare it to the equation for the correlation coefficient…

if u and v have means of zero, then cos(uv) = r(u,v)

• What about u  v where u  v? cos( uv )  u  v

… and compare it to the equation for the correlation coefficient…

if u and v have means of zero, then cos(uv) = r(u,v)

• What about u  v where u  v?

… and let’s compare the cosine to the dot product…

• What about u  v where u  v?

… and let’s compare the cosine to the dot product…

• What about u  v where u  v?

… and let’s compare the cosine to the dot product…

Row 1 times column 1

Row 1 times column 1 goes into row 1, column 1

Row 1 times column 2 goes into row 1, column 2

Row 1 times column 3 goes into row 1, column 3

Row 2 times column 1 goes into row 2, column 1

Row 2 times column 2 goes into row 2, column 2