Inner Products with Waveforms

Brian Beckman, 8 Aug 2010

Setting
Reminder: Discrete Fourier Transform (DFT) is a reversible, composable transform for sampled data. Give it n data points, get back
n complex numbers encoding information about the data. What information? Each point of the DFT measures how much the data is
"similary to" a pure waveform. This note offers some explanation.
Each point of the DFT is an inner product of two, equal-length vectors of complex numbers: one vector is a sample of the kernel
function and the other vector is the data. The kernel function supplies waveforms; the inner product measures how "similar" the data
is to each waveform. If the data happens to be very like one of the waveforms, it will have a large-magnitude inner product with the
matching waveform and small-magnitude inner product with the others.

Example Data
As before, grab n temperature points in Seattle.

stAn_E := ðP2T & ž

Take@WeatherData@"Seattle", "Temperature", 82010, 3<D, nD;
In[1]:=

In[2]:= st60 = st@60D;

In[3]:= ListPlot@st60, Joined ® TrueD

16

14

12

Out[3]=
10

10 20 30 40 50 60

Transform-Kernel Function
The kernel function of the DFT takes a wave number 0 £ k < n and a sample point index 0 £ p < n and gives back a complex
number:

-2 Π ä
kp
ã n
In[4]:= kerAk_, p_, n_E :=
n
This definition is one of the conventional ones for the DFT kernel. Note the minus sign in the exponent -- a good explanation for it
emerges from considering the DFT as a collection of inner products.
Fix k and vary p, meaning sample the kernel function (putting the kernel function on the same footing as the data)
2 Waves001.nb

Fix k and vary p, meaning sample the kernel function (putting the kernel function on the same footing as the data)

In[5]:= eksAker_, k_, n_E := Table@ker@k, p, nD, 8p, 0, n - 1<D

The Wave Number

Note: the horizontal axes on linear plots are labeled from 1 to n, one ahead of the sample-point indices, which run from 0 to n - 1.

In[6]:= linearPlotComplexesAcomplexes_, showImaginary_: TrueE :=

ListPlot@If@showImaginary,
8Re ž complexes, Im ž complexes<, Re ž complexesD,
Joined ® True, PlotMarkers ® AutomaticD
See why k is the wave number: when k = 1, there is one wave in the sample of the kernel function (blue points for the real part,
purple points for the imaginary part):

In[7]:= linearPlotComplexes@eks@ker, 1, 60DD

ææææ àààààààà æææ
ææ àà à ææ
0.10 æ à à æ
æ à àæ
æ à æà
æ à æ à
æ à æ à
0.05 æ à æ à
æ à æ à
æ à æ à
æ à æ à
Out[7]= à æ à æ
à 10 æ 20 30
à 40 æ 50 60
à æ à æ
à æ à æ
-0.05 à æ à æ
à æ à æ
à æ à æ
à æà æ
-0.10 à àæ æ
à à æ æ
àà à ææ æ
àààààààà ææææææææ

When k is 3, there are three waves in the sample

In[8]:= linearPlotComplexes@eks@ker, 3, 60DD

ææ ààà æææ ààà æææ ààà æ
0.10 æ à àæ æ à àæ æ à àæ

æ à æà æ à æà æ à æà
0.05
æ à æ à æ à æ à æ à æ à

Out[8]= à æ à æ à æ à æ à æ à æ
10 20 30 40 50 60
à æ à æ à æ à æ à æ à æ
-0.05
à æà æ à æà æ à æà æ
-0.10 à àæ æ à àæ æ à àæ æ
ààà æææ ààà æææ ààà æææ
Waves001.nb 3

Wave Numbers Near n  2

The idea above - that the number of waves in the sample equals the wave-number k - only goes half-way to n, the number of
samples. For k greater than half n, the number of waves in the sample decreases again, but the imaginary part (in purple) is
inverted about the horizontal axis (flipped).
That observation implies that for a k exactly equal to n  2, the imaginary part must vanish: that's the only way for it to equal its own
negative.
We expect the real part of the sample for k = 60 - 3 = 57, n = 60 to be the same as the real part of the sample above for k = 3,
n = 60, but the imaginary parts to be flipped:

In[9]:= linearPlotComplexes@eks@ker, 57, 60DD

ææ ààà æææ ààà æææ ààà æ
0.10 æà à æ æà à æ æà à æ

àæ à æ àæ à æ àæ à æ
0.05
à æ à æ à æ à æ à æ à æ

Out[9]= à æ à æ à æ à æ à æ à æ
10 20 30 40 50 60
æ à æ à æ à æ à æ à æ à
-0.05
æ àæ à æ àæ à æ àæ à
-0.10 æ æà à æ æà à æ æà à
æææ ààà æææ ààà æææ ààà

Back to the Polar Plot

Plot complex numbers as points on the Euclidean plane, real parts specifying x-coordinates and imaginary parts specifying y-
coordinates.

In[10]:= ReImAc_E := 8Režc, Imžc<

polarPlotComplexesAcomplexes_, n_, showLines_: FalseE :=

Module@8points = ReIm ž complexes, lp<,
In[11]:=

lp = ListPlot@List ž points,
PlotMarkers ® Table@i, 8i, 0, n - 1<D,
H* NOTE:
Mathematica Bug: cannot use "Range" under PlotMarkers *L
Axes ® False, Frame ® True, AspectRatio ® 1D;
If@showLines, Show@lp, ListLinePlot@pointsDD, lpDD
For wave number k = 1, the values of the kernel sample distribute clockwise around the unit circle as closely and finely as possible.
4 Waves001.nb

In[12]:= polarPlotComplexes@eks@ker, 1, 60D, 60D

44 45 46 47
42 43 48
41 49
40 50
39 51
0.10 38 52
37 53
36 54
35 55
0.05 34 56
33 57
32 58
31 59
0.00 30 0
Out[12]=
29 1
28 2
27 3
-0.05 26 4
25 5
24 6
23 7
-0.10 22 8
21 9
20 10
19 11
18 17 12
16 15 14 13
-0.10 -0.05 0.00 0.05 0.10

As points progress around the unit circle clockwise with wave-number 1, the real part (horizontal coordinate) progresses from its
maximum positive value down to its minimum negative value while the imaginary part (vertical coordinate) is negative. Then the real
part increases again while the imaginary part is positive. With wave number 59 out of 60, adjacent points skip 58 points, clockwise,
which is the same as going counterclockwise by 1 point, meaning that as the real part decreases from its maximum positive down to
its minimum negative, the imaginary part is positive, and as the real part goes back up, the imaginary part is negative.
The general rule is that the real parts for wave number k are the same as the real parts for wave number n - k, and the imaginary
parts are inverted.
When k is just one either side of n  2, we can observe this transition with maximum detail. With n = 60, k = 29, the real part starts
positive and the imaginary part starts trending negative:

In[13]:= linearPlotComplexes@eks@ker, 29, 60DD

ææ àààà ææææ ààà æ
æ à à æ æ à à æ
0.10 æ à à æ
à àæ æ
æ à æà
à æ à æ
æ à æ à
0.05 à æ à æ
æ à æ à
à æ à æ
æ à æ à
Out[13]= à æ à æ
à 10 æ 20 30
à 40 æ 50 60
æ à æ à
à æ à æ
-0.05 æ à æ à
à æ à æ
æ à æ à
à æà æ
-0.10 æ à àæ
à à æ æ
æ à à æ æ à à æ
ææ ààà æææ àààà ææ

Notice the one full wave shape of "outer-envelope," then file away the observation. It's for another discussion.

With n = 60, k = 30, exactly half, the imginary part goes to zero, as expected: check the point 30 on the polar plot resting exactly on
the negative real axis:
Waves001.nb 5

In[14]:= linearPlotComplexes@eks@ker, 30, 60DD

ææææææææææææææææææææææææææææææ

0.10

0.05

Out[14]= àààààààààààààààààààààààààààààààààààààààààààààààààààààààààààà
10 20 30 40 50 60

-0.05

-0.10

ææææææææææææææææææææææææææææææ

With n = 60, k = 31, the real part begins positive and the imaginary part starts trending positive:

In[15]:= linearPlotComplexes@eks@ker, 31, 60DD

ææ ààà ææææ àààà æ
æ à à æ æ à à æ
0.10 æ à à æ
æ æà à
àæ à æ
æ à æ à
à æ à æ
0.05 æ à æ à
à æ à æ
æ à æ à
à æ à æ
Out[15]= à æ à æ
10 æ 20 30 à 40 æ 50 60
à
à æ à æ
æ à æ à
-0.05 à æ à æ
æ à æ à
à æ à æ
æ àæ à
-0.10 æà à æ
æ æ à à
æ à à æ æ à à æ
ææ àààà æææ ààà ææ

A Closer Look
With a smaller number of sample points, the effect is easier to see.

Even vs. Odd Wave Numbers

The cases of even n and odd n are distinct, because with even n, there is a point with k = n  2 exactly (imaginary part = 0), but with
odd n, we can only have a wave number just below n  2 and a wave number just above n  2.

Ÿ Even k
First, inspect the case of n = 6 and the three wave numbers k = 2, 3, 4. Expect the real parts of k = 2 and k = 4 to be the same,
imaginary parts inverted; and expect the imaginary part of k = 3 to vanish, just as with k = 29, 30, 31; n = 60:
6 Waves001.nb

In[16]:= linearPlotComplexes@eks@ker, 2, 6DD

0.4 æ æ
à à
0.3

0.2

0.1
Out[16]=
à à
1 2 3 4 5 6
-0.1

-0.2 æ æ æ æ

-0.3
à à

In[17]:= linearPlotComplexes@eks@ker, 3, 6DD

0.4 æ æ æ

0.2

Out[17]= à à à à à à
1 2 3 4 5 6

-0.2

-0.4 æ æ æ

In[18]:= linearPlotComplexes@eks@ker, 4, 6DD

0.4 æ æ
à à
0.3

0.2

0.1
Out[18]=
à à
1 2 3 4 5 6
-0.1

-0.2 æ æ æ æ

-0.3
à à

Ÿ Odd k
k = 2, 3; n = 5 exhibits the imaginary part's changing sign either side of the negative real axis, no point exactly on the negative real
axis, no case of non-zero k with vanishing imaginary part:
Waves001.nb 7

In[19]:= linearPlotComplexes@eks@ker, 2, 5DD

æ à
0.4

à
0.2
æ æ

Out[19]= à
1 2 3 4 5

-0.2
à

æ æ
-0.4
à

In[20]:= linearPlotComplexes@eks@ker, 3, 5DD

æ à
0.4

à
0.2
æ æ

Out[20]= à
1 2 3 4 5

-0.2
à

æ æ
-0.4
à

Common Factors in k and n

When k and n share common factors (as when k divides n) some kernel-function sample values clump together in the complex
plane. In the next plot, k = 4 and n = 20 share the common factor 4 and the number of clumps is 20  4 or 5.
8 Waves001.nb

In[21]:= polarPlotComplexes@eks@ker, 4, 20D, 20, TrueD

14
19
94
0.2

13
18
83

0.1

0.0 10
15
0
5
Out[21]=

-0.1

12
17
72

-0.2
11
16
61
-0.1 0.0 0.1 0.2

As expected, the polar plot for 16 = 20 - 4 looks the same, with just the points going around the unit-circle in the opposite direction
(counterclockwise):

In[22]:= polarPlotComplexes@eks@ker, 16, 20D, 20, TrueD

11
16
61
0.2

12
17
72

0.1

0.0 10
15
0
5
Out[22]=

-0.1

13
18
83

-0.2
14
19
94
-0.1 0.0 0.1 0.2

In the linear plots below, see that existence of common factors in k and n mean that points are sampled at similar positions on each
wave:
Waves001.nb 9

In[23]:= linearPlotComplexes@eks@ker, 4, 20DD

æ à æ à æ à æ à
0.2

à à à à
0.1
æ æ æ æ æ æ æ æ

Out[23]= à à à à
5 10 15 20

-0.1
à à à à

æ æ æ æ æ æ æ æ
-0.2
à à à à

When k and n don't have common factors, points are sampled at different positions on each wave:

In[24]:= linearPlotComplexes@eks@ker, 3, 20DD

æ à æ à æ à
0.2
æ à æ à

æ à à æ
0.1
æ æ à à

Out[24]= à æ à æ
5 10 15 20

æ à à æ
-0.1
æ æ à à

à æ à æ
-0.2
à æ à æ à æ

Interpretation of the Inner Product

A Certain Product of Individual Complex Numbers
Think of a complex number as a point on the Euclidean plane. Consider the special product a b of two complex numbers a and b,
where a is the complex conjugate of a, that is, a copy of a with the imaginary part flipped or inverted (remember the inverted
imaginary parts of the waveforms for wave numbers n - k: They're the complex conjugates of the waveforms for wave numbers k)
Symbolic example:

In[25]:= Remove@a, b, x, y, u, vD

In[26]:= a=x+äy
b=u+äv

Out[26]= x+äy

Out[27]= u +äv
10 Waves001.nb

In[28]:= Conjugate@aD b  Expand

Out[28]= u x­ - ä u y­ + ä v x­ + v y­

Real part of a b is the classic dot product

Observe that the real part of the product a b is x u + y v, which is the ordinary dot product a × b of a and b considered as Euclidean 2-
vectors. Ignore the stars above x and y, denoting nested conjugation -- Mathematica doesn't know that we stipulate x and y to be
real numbers.

Imaginary part of a b encodes the classic cross product

The imaginary part of a b is x v - y u, exactly the magnitude of the classic cross product a ´ b.

The one special product a b encodes both the classic dot and cross products of 2-dimensional vector geometry.

Visualization on the 2-Plane

Observe that if two complex numbers a and b as vectors in the 2-plane are parallel, then the real part of a b is maximal and positive,
and the imaginary part of a b vanishes. If they're anti-parallel, then the real part if minimal and negative. If a and b are perpendicular,
then the real part of a b vanishes and the imaginary part is maximal, positive or negative depending on angular orientation.

Extend to n-dimensional Complex Vectors

Our purpose is to extend the product a b to n-dimensional complex vectors representing waveforms and data so as to measure
similarity, but without getting lost in the heavenly beauties of geometry in spaces of high complex dimensionality. By the time we're
even at 2 complex dimensions, we must tackle 4 real numbers, out of the realm of direct visualization.
Notice that if we just computed a b componentwise and added up the results, we would preserve the extremal aspects of the one-
dimensional case: parallel vectors will be parallel in each of their components, and the result will have maximal positive real part and
vanishing imaginary part. Ditto anti-parallel and perpendicular cases. The intermediate cases will have incredible richness of
geometric interpretation, but that where we don't want to play today -- we'll just take the summary numbers so we can have insight
into the DFT.
Our candidate inner product for n-dimensional complex vectors a and b (now no longer just individual complex numbers, which are
just like 1-vectors) is just
a1 b1 + a2 b2 + º + an bn

Where ai and bi are the i-th complex components of the n-vectors a and b. Mathematica has an easy way to calculate just that using
a dot -- Mathematica extends dot product to complex vectors!

In[29]:= cInnerAa_, b_E := Conjugate@aD.b

Examples
Get a random n-vector of reasonably sized complex numbers

ranc := RandomComplex@8-1 - ä, 1 + ä<D

randomComplexVectorAn_E := Table@ranc, 8n<D
In[30]:=

Here's a five-dimensional example:

Waves001.nb 11

In[32]:= Hx = randomComplexVector@5DL  TableForm

Out[32]//TableForm=

0.40878 + 0.694519 ä
0.646311 - 0.643512 ä
0.883663 + 0.0648531 ä
-0.429352 - 0.497205 ä
-0.0404347 - 0.211491 ä

x inner x should have vanishing imaginary part if we're to interpret inner as measuring similarity in high-dimensional complex space:

In[33]:= x ~cInner~ x

Out[33]= 2.74427 + 0. ä

Mutual Inner Products of the Waveforms

How do the waveforms measure up? Look at the 17-th one versus itself:

In[34]:= eks@ker, 17, 60D~cInner~eks@ker, 17, 60D

Out[34]= 1

And the 17-th versus the 29-th?

In[35]:= eks@ker, 17, 60D~cInner~eks@ker, 29, 60D  N  Chop

Out[35]= 0

How about all-versus-all (this time with a smaller number of samples, so we can see)
12 Waves001.nb

In[36]:= Table@
Table@
Nžeks@ker, j, 15D~cInner~ Nžeks@ker, i, 15D,
8j, 0, 14<D,
8i, 0, 14<D  Chop

1. 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 1. 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 1. 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 1. 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 1. 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 1. 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 1. 0 0 0 0 0 0 0 0
Out[36]= 0 0 0 0 0 0 0 1. 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 1. 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 1. 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 1. 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 1. 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 1. 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 1. 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 1.

Our discretely sampled waveforms are orthonormal: maximally self-similar and having no cross-similarity, one to another. This
means they are suitable as basis vectors in n-dimensional complex vector space, at least with respect to our inner product.

DFT Measures Similarity of Data to Waveforms

Now we can see exactly what the DFT measures: similarity of the data to each waveform from wave number 0 to wave number
n - 1.
Leave for another time interpreting the of measuring data against wave number k and against its partner wave number n - k -- the
one with the same real part and flipped imaginary part.

The Minus Sign in the Kernel

The backbone of the inner product is the special complex product a b. The complex conjugate in the first factor accounts for the
minus sign that most authors automatically place in the kernel function. That's just a convention, and it fits well with other conven-
tions in linear algebra.

Since that minus sign is built-in, we'll just use Mathematica's dot going forward, instead of using our special cInner function.

Back to the Sample Data

Look at the real part of the DFT of the sample weather data (remember that wave number + 1 is the horizontal axis):

In[37]:= ipAdata_, kernel_, k_, n_E := eks@kernel, k, nD.data

In[38]:= dft = Table@ip@st60, ker, k, 60D  N, 8k, 0, 60 - 1<D;

Waves001.nb 13

In[39]:= linearPlotComplexes@dft, FalseD

æææ æ æ ææ
1.5
æ æ

1.0 æ æ
æ æ æ æ æ æ
æ æ ææ æ æ ææ æ
Out[39]= 0.5 æ æ ææ ææ æ
æ æ æ æ
æ æ ææ æ æ æ æ ææ æ æ
æ æ æ æ
æ
æ æ
10 20 30 40 50 60
æ æ æ æ
-0.5
ææ æ

Would be surprising that it's not symmetric about wave number k = 30 = n  2. Look deeper into the peaks at low wave numbers 0
through 6 (plot indices 1 through 7)

In[40]:= Re ž dftP1 ;; 7T H* wave numbers 0 through 6 *L

Out[40]= 883.0109, -4.92398, 0.200734, 5.04122, 0.741965, -0.318114, 1.58549<

This is telling us that, except for the wild peak at k = 0, the data looks a lot like the waveform with wave number 3. Check it out
visually:

Show@8ListPlot@st60, Joined ® TrueD,

ListPlot@10 + 32 * Re ž eks@ker, 3, 60D,
In[41]:=

Joined ® TrueD<D

16

14

12

Out[41]=
10

10 20 30 40 50 60

Interactive Wave-Number Explorer

Just in case you have Mathematica, here is an interactive gadget.
14 Waves001.nb

Manipulate@linearPlotComplexes@eks@ker, k, nDD,
88n, 5, "Number of Samples"<, 1, 60, 1<,
In[42]:=

88k, 0, "Wave Number"<, 0, n - 1, 1<D

Number of Samples

Wave Number

æ æ æ æ æ

0.4

Out[42]= 0.3

0.2

0.1

à à à à à
1 2 3 4 5