DOS Matlab

DIGITAL
SIGNAL
PROCESSING
with selected topics
ADAPTIVE SYSTEMS
TIME-FREQUENCY ANALYSIS
SPARSE SIGNAL PROCESSING
Ljubiša Stanković
2015
2
Library of Congress Cataloging-in-Publication Data

Library of Congress Control Number: 2015912465
ISBN-13: 978-1514179987
ISBN-10: 1514179989
c 2015 Ljubiša Stanković, All Rights Reserved

⃝
Printed by CreateSpace Independent Publishing Platform,

An Amazon.com Company
North Charleston, South Carolina, USA.
Available from Amazon.com and other online and bookstores
All right reserved. Printed and bounded in the United States of America.
No part of this book may be reproduced or utilized in any form or by any

means, electronic or mechanical, including photocopying, recording, or by
any information storage and retrieval system, without permission in writing
from the copyright holder.
Ljubiša Stanković Digital Signal Processing 3
To
my parents
Božo and Cana,
my wife Snežana,
and our
Irena, Isidora, and Nikola.
4
Contents
I Review 19
Chapter 1 Continuous-Time Signals and Systems 21
1.1 Continuous-Time Signals 21
1.2 Periodic Signals and Fourier Series 24
1.2.1 Fourier Series of Real-Valued Signals 28
1.2.2 Linear Systems 33
1.3 Fourier Transform 35
1.3.1 Fourier Transform and Linear Time-Invariant
Systems 37
1.3.2 Properties of the Fourier Transform 37
1.3.3 Relationship Between the Fourier Series and
the Fourier Transform 40
1.4 Fourier Transform and Stationary Phase Method 42
1.5 Laplace Transform 48
1.5.1 Linear Systems Described by Differential Equa-
tions 51
1.5.2 Table of the Laplace Transform 52
1.6 Butterworth Filter 53
II Discrete Signals and Systems 57

Chapter 2 Discrete-Time Signals and Transforms 59
2.1 Discrete-Time Signals 59
2.1.1 Discrete-Time Systems 64
2.2 Fourier Transform of Discrete-Time Signals 67
2.2.1 Properties 69
2.2.2 Spectral Energy and Power Density 74
5
6 Contents
2.3 Sampling Theorem in the Time Domain 75

2.4 Problems 80
2.5 Solutions 85
2.6 Exercise 105
Chapter 3 Discrete Fourier Transform 107

3.1 DFT Definition 107
3.2 DFT Properties 113
3.3 Zero-Padding and Interpolation 118
3.4 Relation among the Fourier Representations 122
3.5 Fast Fourier Transform 124
3.6 Sampling of Periodic Signals 130
3.7 Analysis of a Sinusoid by Using the DFT 133
3.7.1 Leakage Effect 134
3.7.2 Displacement 137
3.8 Discrete Cosine and Sine Transforms 140
3.9 Discrete Walsh-Hadamard and Haar Transforms 145
3.9.1 Discrete Walsh-Hadamard Transform 148
3.9.2 Discrete Haar Wavelet Transform 152
3.10 Problems 156
3.11 Solutions 158
3.12 Exercise 167
Chapter 4 z-Transform 169

4.1 Definition of the z-transform 169
4.2 Properties of the z-transform 171
4.2.1 Linearity 171
4.2.2 Time-Shift 171
4.2.3 Multiplication by exponential signal: Modulation172
4.2.4 Differentiation 172
4.2.5 Convolution in time 173
4.2.6 Table of the z-transform 173
4.2.7 Initial and Stationary State Signal Value 174
4.3 Inverse z-transform 174
4.3.1 Direct Power Series Expansion 174
4.3.2 Theorem of Residues Based Inversion 178
4.4 Discrete systems and the z-transform 180
4.5 Difference equations 183
4.5.1 Solution Based on the z-transform 183
4.5.2 Solution of Difference Equations in the Time
Domain 186
4.6 Relation of the z-transform to other Transforms 191
4.7 Problems 193

4.8 Solutions 197
4.9 Exercise 213
Chapter 5 From Continuous to Discrete Systems 217

5.1 Impulse Invariance Method 218
5.2 Matched z-transform method 223
5.3 Differentiation and Integration 226
5.4 Bilinear Transform 230
5.5 Discrete Filters Design 236
5.5.1 Lowpass filters 236
5.5.2 Highpass Filters 242
5.5.3 Bandpass Filters 244
5.5.4 Allpass Systems - System Stabilization 246
5.5.5 Inverse and Minimum Phase Systems 247
5.6 Problems 251
5.7 Solutions 254
5.8 Exercise 264
Chapter 6 Realization of Discrete Systems 267

6.1 Realization of IIR systems 267
6.1.1 Direct realization I 268
6.1.2 Direct realization II 268
6.1.3 Sensitivity of the System Poles/Zeros to Errors
in Coefficients 271
6.1.4 Cascade Realization 276
6.1.5 Parallel realization 280
6.1.6 Inverse realization 283
6.2 FIR Systems and their Realizations 284
6.2.1 Linear Phase Systems and Group Delay 285
6.2.2 Windows 287
6.2.3 Design of FIR System in the Frequency Domain 291
6.2.4 Realizations of FIR system 293
6.3 Problems 298
6.4 Solutions 302
6.5 Exercise 314
Chapter 7 Discrete-Time Random Signals 319

7.1 Basic Statistical Definitions 319
7.1.1 Expected Value 319
7.1.2 Probability and Probability Density Function 326
7.1.3 Median 329
8 Contents
7.1.4 Variance 331

7.2 Second-Order Statistics 336
7.2.1 Correlation and Covariance 336
7.2.2 Stationarity and Ergodicity 337
7.2.3 Power Spectral Density 338
7.3 Noise 340
7.3.1 Uniform Noise 340
7.3.2 Binary Noise 341
7.3.3 Gaussian Noise 344
7.3.4 Complex Gaussian Noise and Rayleigh Distri-
bution 349
7.3.5 Impulsive Noises 350
7.3.6 Noisy Signals 352
7.4 Discrete Fourier Transform of Noisy Signals 352
7.4.1 Detection of a Sinusoidal Signal Frequency 356
7.5 Linear Systems and Random Signals 360
7.5.1 Spectral Estimation of Narrowband Signals 366
7.6 Detection and Matched Filter 368
7.6.1 Matched Filter 369
7.7 Optimal Wiener Filter 372
7.8 Quantization effects 376
7.8.1 Input signal quantization 377
7.8.2 Quantization of the results 382
7.9 Problems 394
7.10 Solutions 400
7.11 Exercise 418
III Selected Topics 421

Chapter 8 Adaptive Systems 423
8.1 Introduction 423
8.2 Linear Adaptive Adder 427
8.2.1 Error Signal 429
8.2.2 Autocorrelation Matrix Eigenvalues and Eigen-
vectors 438
8.2.3 Error Signal Analysis 443
8.2.4 Orthogonality Principle 445
8.3 Steepest Descend Method 446
8.4 LMS Algorithm 457
8.4.1 Convergence of the LMS algorithm 458
8.5 LMS Application Examples 460
8.5.1 Identification of Unknown System 460

8.5.2 Noise Cancellation 464
8.5.3 Sinusoidal Disturbance Cancellation 466
8.5.4 Signal Prediction 468
8.5.5 Adaptive Antenna Arrays 473
8.5.6 Acoustic Echo Cancellation 478
8.6 Variations on the LMS Algorithm 481
8.6.1 Sign LMS 481
8.6.2 Block LMS 482
8.6.3 Normalized LMS Algorithm 483
8.6.4 LMS with Variable Step Size 485
8.6.5 Complex LMS 487
8.7 RLS Algorithm 489
8.8 Adaptive Recursive Systems 493
8.9 From the LMS algorithm to the Kalman filters 495
8.10 Neural Networks 500
8.10.1 Neuron 502
8.10.2 Network Function 502
8.10.3 Activation Function 503
8.10.4 Neural Network Topology 505
8.10.5 Network with Supervised Learning 507
8.10.6 One-Layer Network with Binary Output - Per-
ceptron 508
8.10.7 One-Layer Neural Network with Continuous
Output 512
8.10.8 Multilayer Neural Networks 515
8.10.9 Neural Networks with Unsupervised Learning 518
8.10.10 Voting Machines 519
Chapter 9 Time-Frequency Analysis 521

9.1 Short-Time Fourier Transform 522
9.2 Windows 529
9.2.1 Rectangular Window 529
9.2.2 Triangular (Bartlett) Window 530
9.2.3 Hann(ing) Window 531
9.2.4 Hamming Window 533
9.2.5 Blackman and Kaiser Windows 534
9.2.6 Discrete Form and Realizations of the STFT 535
9.2.7 Recursive STFT Realization 541
9.2.8 Filter Bank STFT Implementation 542
9.2.9 Signal Reconstruction from the Discrete STFT 546
10 Contents
9.2.10 Time-Varying Windows 556

9.2.11 Frequency-Varying Window 567
9.2.12 Hybrid Time-Frequency-Varying Windows 569
9.3 Wavelet Transform 569
9.3.1 Filter Bank and Discrete Wavelet 574
9.3.2 S-Transform 607
9.4 Local Polynomial Fourier Transform 610
9.4.1 Fractional Fourier Transform with Relation to
the LPFT 613
9.5 High-Resolution STFT 614
9.5.1 Capon’s STFT 614
9.5.2 MUSIC STFT 618
9.5.3 Capon’s LPFT 620
9.6 Wigner Distribution 622
9.6.1 Auto-Terms and Cross-Terms in the Wigner
Distribution 627
9.6.2 Wigner Distribution Properties 631
9.6.3 Pseudo and Smoothed Wigner Distribution 636
9.6.4 Discrete Pseudo Wigner Distribution 639
9.6.5 From the STFT to the Wigner Distribution via
S-Method 647
9.7 General Quadratic Time-Frequency Distributions 653
9.7.1 Reduced Interference Distributions 657
9.7.2 Kernel Decomposition Method 662
Chapter 10 Sparse Signal Processing 665

10.1 Illustrative Examples 666
10.2 Sparsity and Reduced Set of Samples/Observations 676
10.3 Transformation Matrix Parameters 680
10.3.1 Unitary Matrix 682
10.3.2 Isometry and Restricted Isometry Property 683
10.3.3 Coherence 684
10.3.4 Restricted Isometry and Coherence 687
10.3.5 Restricted Isometry and Eigenvalues 691
10.3.6 Unique Reconstruction Condition and RIP 701
10.3.7 Rank and Spark of a Matrix 704
10.3.8 Spark and the Solution Uniqueness 706
10.4 Norm-Zero Based Reconstruction 710
10.4.1 Direct Combinatorial Search 710
10.4.2 Pseudoinverse matrix 713
10.4.3 Estimation of Unknown Positions 714
10.4.4 Unavailable/Missing Samples Noise in Initial

Estimation 717
10.4.5 Iterative Procedure 727
10.4.6 Influence of Additive Input Noise 729
10.4.7 Nonsparse Signal Reconstruction 733
10.5 Norm-One Based Reconstruction 735
10.5.1 Illustrations in the Signal Domain 737
10.5.2 Illustration in the Sparsity Domain 742
10.5.3 Equivalence of the Norm-Zero and Norm-One
Based Minimization Solutions 756
10.6 Median Based Formulation 760
10.7 Norm-one Based Reconstruction Algorithms 766
10.7.1 LASSO- Minimization 767
10.7.2 Signal Domain Reconstruction with a Gradient
Algorithm 770
10.8 On the Uniqueness of the DFT of Sparse Signals 782
10.9 Indirect Measurements/Sampling 791
10.10Processing of Sparse Signals with Impulsive Noise 801
10.10.1 Direct Search Procedure 802
10.10.2 Criteria for Selecting Samples 803
10.10.3 Uniqueness of the Obtained Solution 806
10.11Image Reconstruction 808
Index 811
About the Author 820

12 Contents
Preface
book is a result of author’s thirty-three years of experience in
T
HIS
teaching and research in signal processing. It is written for students
and engineers as a first book in digital signal processing, assuming
that a reader is familiar with the basic mathematics, including integrals, dif-
ferential calculus, and linear algebra. Although a review of continuous-time
analysis is presented in the first chapter, a prerequisite for the presented
content is a basic knowledge about continuous-time signal processing.
The book consists of three parts. After an introductory review part, the
basic principles of digital signal processing are presented within Part two of
the book. This part starts with Chapter two which deals with basic defini-
tions, transforms, and properties of discrete-time signals. The sampling the-
orem, providing essential relation between continuous-time and discrete-
time signals, is presented in this chapter as well. Discrete Fourier transform
and its applications to signal processing are the topic of the third chapter.
Other common discrete transforms, like Cosine, Sine, Walsh-Hadamard,
and Haar are also presented in this chapter. The z-transform, as a power-
ful tool for analysis of discrete-time systems, is the topic of Chapter four.
Various methods for transforming a continuous-time system into a corre-
sponding discrete-time system are derived and illustrated in Chapter five.
Chapter six is dedicated to the forms of discrete-time system realizations.
Basic definitions and properties of random discrete-time signals are given
in Chapter six. Systems to process random discrete-time signals are consid-
ered in this chapter as well. Chapter six concludes with a short study of
quantization effects.
The presentation is supported by numerous illustrations and exam-
ples. Chapters within Part two are followed by a number of solved and
unsolved problems for practice. Theory is explained in a simple way with
a necessary mathematical rigor. The book provides simple examples and
13
14 Preface
explanations for each presented transform, method, algorithm or approach.

Sophisticated results in signal processing theory are illustrated by simple
numerical examples.
Part three of the book contains few selected topics in digital signal
processing: adaptive discrete-time systems, time-frequency signal analysis,
and processing of discrete-time sparse signals. This part could be studied
within an advanced course in digital signal processing, following the basic
course. Some parts from the selected topics may be included in tailoring a
more extensive first course in digital signal processing as well.
The author would like to thank colleagues: prof. Zdravko Uskoković,
prof. Srdjan Stanković, prof. Igor Djurović, prof. Veselin Ivanović, prof.
Miloš Daković, prof. Božo Krstajić, prof. Vesna Popović-Bugarin, prof. Slo-
bodan Djukanović, prof. Irena Orović, dr. Nikola Žarić, dr Marko Sime-
unović, and M.Sc. Predrag Raković for careful reading of the initial version
of this book and for many comments that helped to improve the presenta-
tion.
The author thanks the colleagues that helped in preparing the special
topics part of the book. Many thanks to Miloš Daković who coauthored all
three chapters of Part three of this book and to other coauthors of chapters
in this part: Thayaparan Thayananthan, Srdjan Stanković, and Irena Orović.
Special thanks to M.Sc. Miloš Brajović and M.Sc. Stefan Vujović for their
careful double-check of the presented theory and examples, numerous com-
ments, and for the help in proofreading the final version of the book.
London,
July 2013 - July 2015.
Author
Introduction
is a physical process, mathematical function, or any other
S
IGNAL
physical or symbolic representation of an information. Signal theory
and processing are the areas dealing with the efficient generation,
description, transformation, transmission, reception, and interpretation of
information. In the beginning, the most common physical processes used
for these purposes were the electric signals, for example, varying current
or electromagnetic waves. Signal theory is most commonly studied within
electrical engineering. Signal theory theory are strongly related to the ap-
plied mathematics and information theory. Examples of signals include
speech, music, image, video, medical, biological, geophysical, sonar, radar,
biomedical, car engine, financial, and molecular data. In terms of signal gen-
eration, the main topics are in sensing, acquisition, synthesis, and reproduc-
tion of information. Various mathematical transforms, representations, and
algorithms are used for describing signals. Signal transformations are a set
of methods for decomposition, filtering, estimation and detection. Modu-
lation, demodulation, detection, coding, and compression are the most im-
portant aspects of the signal transmission. In the process of interpretation,
various approaches may be used, including adaptive and learning-based
tools and analysis.
Mathematically, signals are presented by functions of one or more vari-
ables. Examples of one-dimensional signals are speech and music signals.
A typical example of a two-dimensional signal is an image while video se-
quence is a sample of a three-dimensional signal. Some signals, for example,
geophysical, medical, biological, radar, or sonar, may be represented and
interpreted as one-dimensional, two-dimensional, or multidimensional.
Signals may be continuous functions of independent variables, for ex-
ample, functions of time and/or space. Independent variables may also
be discrete, with the signal values being defined only over an ordered set
15
16 Introduction
continuous discrete-time digital

1 1
15 1111
14 1110
0.8 0.8 13 1101
12 1100
11 1011
10 1010
0.6 0.6
9 1001
xd(n)
x(n)
x(t)
8 1000
7 0111
0.4 0.4 6 0110
5 0101
4 0100
0.2 0.2 3 0011
2 0010
1 0001
0 0 0000
0 5 10 15 0 5 10 15 0 5 10 15
t n n
Figure 1 Illustration of a continuous signal and its discrete-time and digital version.
of discrete independent variable values. This is a discrete-time signal. The

discrete-time signals, after being stored in a general computer or special-
purpose hardware, are discretized (quantized) in amplitude as well, so that
they can be memorized within the registers of a finite length. These kinds
of signals are referred to as digital signals, Fig.1. A continuous-time and
continuous amplitude (analog) signal is transformed into a discrete-time
and discrete-amplitude (digital) signal by using analog-to-digital (A/D)
converters, Fig.2. Their processing is known as digital signal processing. In
modern systems, the amplitude quantization errors are very small. Com-
mon A/D converters are with sampling frequency of up to megasample
(some even up to few gigasample) per second with 8 to 24 bits of resolu-
tion in amplitude. The digital signals are usually mathematically treated as
continuous (nondiscretized) in amplitude, while the quantization error is
studied, if needed, as a small disturbance in processing, reduced to a noise
in the input signal. Digital signals are transformed back into analog form by
digital-to-analog (D/A) converters.
According to the nature of their behavior, all signals could be deter-
ministic or stochastic. For deterministic signals, the values are known in the
past and future, while the stochastic signals are described by probabilistic
methods. The deterministic signals are commonly used for theoretical de-
scription, analysis, and syntheses of systems for signal processing.
ANALOG SYSTEM
x(t) y(t)
ha(t)
DIGITAL SYSTEM
x(t) x(n) y(n) y(t)

A/D h(n) D/A
Figure 2 Illustration of an analog and a digital system used to process an analog signal.
Advantages of processing signals in digital form are in their flexibility

and adaptability with possibilities ranging up to our imagination to imple-
ment a transformation with an algorithm on a computer. The time required
for processing in real time (all calculations have to be completed between
two signal samples) is a limitation as compared to the analog systems that
are limited with a physical delay of electrical components and circuits only.
18 Introduction
Part I
Review
19
Chapter 1
Continuous-Time Signals and Systems
of discrete-time signals are obtained by sampling continuous-
M
OST
time signals. In many applications, the result of signal process-
ing is presented and interpreted in the continuous-time domain.
Throughout the course of digital signal processing, the results will be dis-
cussed and related to the continuous-time forms of signals and their param-
eters. This is the reason why the first chapter is dedicated to a review of
signals and transforms in the continuous-time domain. This review will be
of help in establishing proper correspondence and notation for the presen-
tation that follows in the next chapters.
1.1 CONTINUOUS-TIME SIGNALS
One-dimensional signals, represented by a function of time as a continuous

independent variable, are referred to as continuous-time signals (continu-
ous signals). Some simple forms of deterministic continuous-time signals
are presented next.
The unit-step signal (Heaviside function) is defined by
!
1, for t ≥ 0
u(t) = . (1.1)
0, for t < 0
In the Heaviside function definition, the value of u(0) = 1/2 is also used.
Note that the independent variable t is continuous, while the signal itself is
not a continuous function. It has a discontinuity at t = 0.
The boxcar signal (rectangular window) is formed as b(t) = u(t +
1/2) − u(t − 1/2), that is, b(t) = 1 for −1/2 ≤ t < 1/2 and b(t) = 0 else-
where. A signal obtained by multiplying the unit-step signal by t is called
the ramp signal, with notation R(t) = tu(t).
21
22 Continuous-Time Signals and Systems
The impulse signal (or delta function) is defined as

"∞
δ(t) = 0, for t ̸= 0 and δ(t)dt = 1. (1.2)
−∞
The impulse signal is equal to 0 everywhere, except at t = 0, where it

assumes an infinite value, so that its area is 1. From the definition of
the impulse signal, it follows δ( at) = δ(t)/ | a| . This function cannot be
implemented in real-world systems due to its infinitely short duration and
infinitely large amplitude at t = 0.
In theory, any signal can be expressed by using the impulse signal, as
"∞ "∞
x (t) = x (t − τ )δ(τ )dτ = x (τ )δ(t − τ )dτ. (1.3)
−∞ −∞
Using the previous relation, it is possible to relate the unit-step signal

and the impulse signal,
"∞ "t
u(t) = δ(τ )u(t − τ )dτ = δ(τ )dτ
−∞ −∞
or
du(t)
= δ ( t ). (1.4)
dt
A sinusoidal signal, with amplitude A, frequency Ω0 , and initial phase
ϕ, is a signal of the form
x (t) = A sin(Ω0 t + ϕ). (1.5)
This signal is periodic in time, since it satisfies the periodicity condition
x ( t + T ) = x ( t ). (1.6)
In this case, the period is T = 2π/Ω0 .

A signal periodic with a basic period T could also be considered as
periodic with periods kT, where k is an integer.
A complex sinusoidal signal
x (t) = Ae j(Ω0 t+ ϕ) = A cos(Ω0 t + ϕ) + jA sin(Ω0 t + ϕ) (1.7)
is also periodic with period T = 2π/Ω0 . Fig. 1.1 depicts basic continuous-
time signals.
u(t) 1 1
δ(t)
0 0
-1 (a) -1 (b)
-4 -2 0 2 4 -4 -2 0 2 4
1 1
sin(πt)
b(t)
0 0
-1 (c) -1 (d)
-4 -2 0 2 4 -4 -2 0 2 4
t t
Figure 1.1 Continuous-time signals: (a) unit-step signal, (b) impulse signal, (c) boxcar signal,
and (d) sinusoidal signal.
Example 1.1. Find the period of a signal

N
x (t) = ∑ An e jnΩ t . 0
n =0
⋆This signal consists of N + 1 components. The constant component A0

can be considered as periodic with any period. The remaining components
A1 e jΩ0 t , A2 e j2Ω0 t , A3 e j3Ω0 t , ..., A N e jNΩ0 t are periodic with periods, T1 =
2π/Ω0 , T2 = 2π/(2Ω0 ), T3 = 2π/(3Ω0 ), ...., TN = 2π/( NΩ0 ), respectively. A
sum of periodic signals is periodic with the period being equal to the small-
est time interval T containing all of the periods T1 , T2 , T3 ,..., TN an integer
number of times. In this case, it is T = 2π/Ω0 .
Example 1.2. Find the periods of signals: x1 (t) = sin(2πt/36), x2 (t) = cos(4πt/15 +
2), x3 (t) = exp( j0.1t), x4 (t) = x1 (t) + x2 (t), and x5 (t) = x1 (t) + x3 (t).
⋆Periods are calculated according to (1.6). For x1 (t) the period follows
from 2πT1 /36 = 2π, as T1 = 36. Similarly, T2 = 15/2 and T3 = 20π. The
period of x4 (t) is the smallest interval containing T1 and T2 . It is T4 = 180 (5
periods of x1 (t) and 24 periods of x2 (t)). For signal x5 (t), when the periods of
components are T1 = 36 and T3 = 20π, there is no common interval T5 such
that the periods T1 and T3 are contained an integer number of times. Thus,
the signal x5 (t) is not periodic.
Some parameters that can be used to describe a signal are:

• Maximum absolute value (magnitude) of a signal
Mx = max | x (t)| , (1.8)

−∞<t<∞
• Signal energy
"∞
Ex = | x (t)|2 dt, (1.9)
−∞
• Signal instantaneous power
Px (t) = | x (t)|2 . (1.10)
The average signal power is defined by
"T
1
PAV = lim | x (t)|2 dt.
T →∞ 2T
−T
The average power is a time average of energy. Energy signals are

signals with a finite energy, while power signals have finite and nonzero
power. The average signal power of energy signals is zero.
1.2 PERIODIC SIGNALS AND FOURIER SERIES
Consider a periodic signal x (t) with a period T. It can be expressed as a sum

of weighted periodic complex sinusoidal functions e j2πnt/T , −∞ < n < ∞,
x (t) = · · · + X−1 e− j2πt/T + X0 e− j0 + X1 e j2πt/T + · · · (1.11)

∞
= ∑ Xn e j2πnt/T
n=−∞
if the Dirichlet conditions are met: (1) the signal x (t) has a finite number of
discontinuities within the period T; (2) it has a finite average value in the
period T; and (3) the signal has a finite number of maxima and minima.
Since the signal analysis deals with real-world physical signals, rather than
with mathematical generalizations, these conditions are almost always met.
The set of basis functions {e j2πnt/T : −∞ < n < ∞}, is an orthonormal

set of functions since their inner product is
# $ 1 T/2"
e j2πmt/T , e j2πnt/T = e j2πmt/T e− j2πnt/T dt
T
− T/2
%
1 for m = n
= sin(π (m−n)) .
π (m−n)
= 0 for m ̸= n
It means that the inner product of any two different basis functions is zero
(orthogonal set), while the inner product of a function with itself is 1 (normal
set). In the case of orthonormal set of basis functions, it is easy to show that
the weighting coefficients Xn can be calculated as projections of x (t) onto
the basis functions e j2πnt/T ,
# $ 1 T/2
"
Xn = x (t), e j2πnt/T = x (t)e− j2πnt/T dt. (1.12)
T
− T/2
This relation follows after a simple multiplication of the right and left sides
& T/2
of (1.11) by e− j2πmt/T and an integration within the period T1 −T/2 (·) dt.
Example 1.3. Show that the Fourier series coefficients Xn of a periodic signal x (t)
can be obtained by minimizing the mean square error between the signal and
∑nN=− N Xn e j2πnt/T within the period T.
⋆The mean square value of error
N
e(t) = x (t) − ∑ Xn e j2πnt/T ,
n=− N
within the period, is

T/2
"
' '2
1 ' N '
' j2πnt/T '
I= ' x ( t ) − ∑ Xn e ' dt.
T ' n=− N
'
− T/2
From ∗
∂I/∂Xm = 0 follows
T/2
"
( )
N
1
e− j2πmt/T x (t) − ∑ Xn e j2πnt/T dt = 0
T n=− N
− T/2
T/2
"
1
Xm = x (t)e− j2πmt/T dt. (1.13)
T
− T/2
Note: A derivative of complex function F (z) = u + jv = u( x, y) + jv( x, y) with

z = x + jy is defined by
* +
∂F (z) ∂ ∂
= −j F ( x, y),
∂z ∂x ∂y
* +
∂F (z) ∂ ∂
= + j F ( x, y).
∂z∗ ∂x ∂y
Often a half of these values is used in the definition, what does not change
our results.
In order to prove the form ∂I/∂Xm ∗ = 0 let us denote X by z, all terms
m
N j2πnt/T
in x (t) − ∑n=− N Xn e = f (z) that not depend on z = x + jy = Xm by
a + jb and e j2πnt/T = e jα , then we have to show that
∂F (z) ∂ | f (z)|2
∗ = = 2e− jα f (z).
∂z ∂z∗
In our case
' '
' '
| f (z)|2 = 'a + jb + e jα ( x + jy)'
= ( a + x cos α − y sin α)2 + (b + x sin α + y cos α)2
For the minimization of a function of two variables x and y we need partial
derivatives
∂ | f (z)|2
= 2 cos α( a + x cos α − y sin α)+ (1.14)
∂x
2 sin α(b + x sin α + y cos α)
= 2 Re{e− jα f (z)}
and
∂ | f (z)|2
= 2 Im{e− jα f (z)}. (1.15)
∂y
Therefore, all calculations with two real-valued equations (1.14) and (1.15)
are the same as using one complex valued relation
* +
∂ | f (z)|2 ∂ | f (z)|2 ∂ ∂ ∂F (z)
+j = +j F ( x, y) = .
∂x ∂y ∂x ∂y ∂z∗
Since the signal and the basis functions are periodic with period T, in
all previous integrals, we can use
T/2
" " +Λ
T/2
1 − j2πnt/T 1
x (t)e dt = x (t)e− j2πnt/T dt (1.16)
T T
− T/2 − T/2+Λ
where Λ is an arbitrary constant.

The signal expansion (1.11) is known as the Fourier series, and the
coefficients Xn are the Fourier series coefficients.
Example 1.4. Calculate the Fourier series coefficients of a periodic signal x (t) =
cos2 (πt/4). What will be the coefficient values if period T = 8 is assumed?
⋆The signal x (t) can be written as x (t) = (1 + cos(πt/2))/2. The
period is T = 4. Assuming that the Fourier series coefficients are calculated
with T = 4, after transforming the signal into (1.11) form, we get
1 − jπt/2 1 1 jπt/2
x (t) = e + + e .
4 2 4
The Fourier series coefficients are recognized as X−1 = 1/4, X0 = 1/2 and
X1 = 1/4 (without the calculation defined by (1.12)). Other coefficients are
equal to zero. In the above transformation, the relation cos (πt/2) = (e jπt/2 +
e− jπt/2 )/2 is used. If the period T = 8 is used, then the signal is decomposed
into complex sinusoids e j2πnt/8 = e jπnt/4 , (1.11). The signal can be written as
1 − j2πt/4 1 1 j2πt/4
x (t) = e + + e . (1.17)
4 2 4
Thus, comparing the signal definition with the basis functions e jπnt/4 , we
may write X−2 = 1/4, X0 = 1/2, and X2 = 1/4. Other coefficients are equal
to zero.
Example 1.5. Calculate the Fourier series coefficients of a periodic signal x (t)
defined as
∞
x (t) = ∑ x0 (t + 2n)
n=−∞
with
x0 (t) = u(t + 1/4) − u(t − 1/4). (1.18)
⋆The signal x (t) is a periodic extension of x0 (t), with period T = 2.

This signal is equal to 1 for −1/4 ≤ t < 1/4, within its basic period. Thus,
1/4
"
1 sin(πn/4)
Xn = 1e− j2πnt/2 dt = , (1.19)
2 πn
−1/4
with X0 = 1/4. Values of Xn are presented in Fig. 1.2.

The signal x (t) can be reconstructed by using the Fourier series (1.11).
In calculations, a finite number of terms denoted by M should be used,
M
x M (t) = ∑ Xn e jπnt .
n=− M
The reconstructed signal, with M = 1, 2, 6, and 30, is shown in Fig. 1.3.

x(t) Xn
-1 -1/4 1/4 1
t 0 n
Figure 1.2 Periodic signal (left) and its Fourier series coefficients (right).
1.5 1.5
1 1
x (t)
x (t)
0.5 0.5
1
0 0
(a) (b)
-0.5 -0.5
-2 -1 0 1 2 -2 -1 0 1 2
t t
1.5 1.5
1 1
x (t)
x (t)
0.5 0.5
30
6
0 0
(c) (d)
-0.5 -0.5
-2 -1 0 1 2 -2 -1 0 1 2
t t
Figure 1.3 Illustration of signal reconstruction by using a finite Fourier series with: (a)
coefficients Xn within −1 ≤ n ≤ 1, (b) coefficients Xn within −2 ≤ n ≤ 2, (c) coefficients Xn
within −6 ≤ n ≤ 6, and (d) coefficients Xn within −30 ≤ n ≤ 30.
1.2.1 Fourier Series of Real-Valued Signals
For a real-valued signal x (t) the Fourier series coefficients can be written as
T/2
" T/2
"
1 2πnt 1 2πnt
Xn = x (t) cos( )dt − j x (t) sin( )dt
T T T T
− T/2 − T/2
An − jBn
= . (1.20)
2
∗
where An /2 and − Bn /2 are real and imaginary part of Xn . Since Xn = X− n
holds for real-valued signals, the values of An and Bn are
T/2
"
2 2πnt
A n = Xn + X− n = x (t) cos( )dt,
T T
− T/2
T/2
"
Xn − X− n 2 2πnt
Bn = = x (t) sin( )dt. (1.21)
−j T T
− T/2
The Fourier series form for real-valued signals is

−1 ∞
x (n) = ∑ Xn e j2πnt/T + X0 + ∑ Xn e j2πnt/T
n=−∞ n =1
∞
= X0 + ∑ (Xn e j2πnt/T + X−n e− j2πnt/T ) (1.22)
n =1
∞ ∞
A0 2πnt 2πnt
= + ∑ An cos( ) + ∑ Bn sin( ) (1.23)
2 n =1
T n =1
T
,
with | Xn | = A2n + Bn2 /2. For real-valued signals the integrals in (1.21),
corresponding to An and Bn , are even and odd functions with respect to
n. Therefore, it is possible to calculate
T/2
" - .
1 2πnt 2πnt
Hn = x (t) cos( ) + sin( ) dt (1.24)
T T T
− T/2
and to get
An = Hn + H−n
Bn = Hn − H−n .
The coefficients calculated by (1.24) are the Hartley series coefficients. For a
real-valued and even signal x (t) = x (−t) this transform reduces to
T/2
"
1 2πnt
Cn = x (t) cos( )dt
T T
− T/2
corresponding to the Fourier cosine series coefficients . Similar expression is

obtained for an odd real-valued signal x (n) when the Fourier series reduces
to the Fourier sine series coefficients.
Example 1.6. For a signal

x (t) = t[u(t) − u(t − 1/2)]
we are interested in its reconstruction based on the Fourier series coefficients.
The rate of coefficients convergence may depend on the way how the periodic
extension of this signal is formed.
(a) Calculate the Fourier series of the original signal periodically ex-
tended with period T = 1/2,
∞
1
x p (t) = ∑ x ( t + n ).
2
n=−∞
Write the reconstruction formula with M Fourier series coefficients.

(b) What are the Fourier transform coefficients and the reconstruction
formula for
∞
x p (t) = ∑ x ( t + n ),
n=−∞
when the period is T = 1.
(c) The signal is first extended with its reversed version
x c ( t ) = x ( t ) + x (1 − t )
and then periodically extended with period T = 1. Find the Fourier series
coefficients and the reconstruction formula.
(d) Comment the coefficients convergence in all cases.
⋆ (a) The Fourier series coefficients of this signal are
1/2
"
1 1
Xn = te− j2πn/(1/2)t dt =
1/2 − j4πn
0
with X0 = 1/4. The reconstructed signal with M coefficients is

M - .
1 1 j4πnt 1 − j4πnt
x M (t) = + ∑ − e + e
4 n =1 4jπn 4jπn
M
1 sin(4πnt)
= −∑ .
4 n =1 2πn
The reconstructed signal for some values of M is presented in Fig.1.4.

(b) The Fourier series coefficients of the signal extended with period 1
are
1/2
" '1/2 '1/2
1 1 ' 1 '
Xn = te− j2πnt dt = te− j2πnt '' + 2
e− j2πnt '
1 − j2πn 0 ( 2πn ) 0
0
(−1)n (−1)n − 1
= + ,
− j4πn (2πn)2
0.6 0.6
0.4 0.4
x1(t)
x (t)
2
0.2 0.2
0 0
(a) (b)
-1 -0.5 0 0.5 1 -1 -0.5 0 0.5 1
t t
0.6 0.6
0.4 0.4
x (t)
x6(t)
30
0.2 0.2
0 0
(c) (d)
-1 -0.5 0 0.5 1 -1 -0.5 0 0.5 1
t t
Figure 1.4 Reconstruction of a signal using the Fourier series. Reconstructed signal is denoted
by x M (t), where M indicates the number of coefficients used in reconstruction.
with X0 = 1/8. Note that the relation between the Fourier coefficients in (a)
(b) ( a)
and (b) is 2X2n = Xn . The reconstruction is presented in Fig.1.5.
(c) For the signal xc (t) extended with its reversed version follows
1/2
" "1
Cn = te− j2πnt dt + (1 − t)e− j2πnt dt
0 1/2
1/2
"
(−1)n − 1
=2 t cos(2πnt)dt =
2π 2 n2
0
with C0 = 1/4. The reconstruction formula is

M
1 1 − (−1)n
x M (t) = −2 ∑ 2 2
cos(2πnt)
4 n=1 2π n
M
1 1
= −2 ∑ 2 2
cos(2π (2n − 1)t).
n=1 π (2n − 1)
4
The reconstructed signal in this case is presented in Fig.1.6.

(d) The coefficients convergence in cases (a) and (b) is of order 1/n
while the convergence in the last case (c) is of order 1/n2 . The best signal
reconstruction with the given number of coefficients will be achieved in
the case (c). Also for a given reconstruction error we will need the smallest
0.6 0.6
0.4 0.4
x1(t)
x2(t)
0.2 0.2
0 0
(a) (b)
-1 -0.5 0 0.5 1 -1 -0.5 0 0.5 1
t t
0.6 0.6
0.4 0.4
x30(t)
x6(t)
0.2 0.2
0 0
(c) (d)
-1 -0.5 0 0.5 1 -1 -0.5 0 0.5 1
t t
Figure 1.5 Reconstruction of a periodic signal, with a zero interval extension before using the
Fourier series.
0.6 0.6
0.4 0.4
x1(t)
x2(t)
0.2 0.2
0 0
(a) (b)
-1 -0.5 0 0.5 1 -1 -0.5 0 0.5 1
t t
0.6 0.6
0.4 0.4
x30(t)
x6(t)
0.2 0.2
0 0
(c) (d)
-1 -0.5 0 0.5 1 -1 -0.5 0 0.5 1
t t
Figure 1.6 Reconstruction of a periodic signal after an even extension before using the Fourier
series (cosine Fourier series).
number of reconstruction terms M in case (c). This kind of signal extension

will be later used as a basis for a definition of the so called cosine signal
transforms.
1.2.2 Linear Systems
A system transforms one signal (input signal) into another signal (output
signal). Assume that x (t) is the input signal. The system transformation will
be denoted by an operator T {◦}. The output signal can be written as
y(t) = T { x (t)}. (1.25)
A system is linear if, for any two signals x1 (t) and x2 (t) and arbitrary
constants a1 and a2 , it holds
y(t) = T { a1 x1 (t) + a2 x2 (t)} = a1 T { x1 (t)} + a2 T { x2 (t)}. (1.26)
We say that a system is time-invariant if its properties and parameters do

not change in time. For a time-invariant system, it holds:
if y(t) = T { x (t)}, then T { x (t − t0 )} = y(t − t0 ), (1.27)
for any t0 .
Linear time-invariant (LTI) systems are fully described by their re-
sponse to the impulse signal. If we know the impulse response of these
systems,
h(t) = T {δ(t)},
then for arbitrary signal x (t) at the input, the output can be calculated, by
using (1.3), as
⎧ ⎫
⎨ "∞ ⎬
y(t) = T { x (t)} = T x (τ )δ(t − τ )dτ
⎩ ⎭
−∞
"∞ "∞
Linearity Time−invariance
= x (τ )T {δ(t − τ )}dτ = x (τ )h(t − τ )dτ.
−∞ −∞
The last integral is of particular significance. It is called a convolution in

time of x (t) and h(t). Its notation is
"∞
y(t) = x (t) ∗t h(t) = x (τ )h(t − τ )dτ. (1.28)
−∞
The convolution is a commutative operation
x ( t ) ∗ t h ( t ) = h ( t ) ∗ t x ( t ). (1.29)
Example 1.7. Find a convolution of signals x (t) = u(t + 1) − u(t − 1) and h(t) =
e − t u ( t ).
⋆By using the convolution definition, we get
"∞ "1
y(t) = x (τ )h(t − τ )dτ = 1 · e−(t−τ ) u(t − τ )dτ
−∞ −1
⎧ & t +1
⎪
⎪ e−λ dλ = e−t (e − 1/e), for t ≥ 1
t −1
" ⎨ t −1
⎪
& t +1 − λ
=− e−λ u(λ)dλ = −(t+1) ,
⎪ 0 e dλ = 1 − e
⎪
for − 1 ≤ t < 1
t +1 ⎪
⎩
0 for t < −1.
A system is causal if there is no response before the input signal

appears. For causal systems h(t) = 0 for t < 0. In general signals that satisfy
the property that they may be an impulse response of a causal system may
be referred to as causal signals.
A system is stable if any input signal with a finite magnitude Mx =
max−∞<t<∞ | x (t)| produces an output y(t) whose values are finite, |y(t)| <
∞. Sufficient condition that a linear time-invariant system is stable is
"∞
|h(τ )|dτ < ∞ (1.30)
−∞
since
"∞ "∞
|y(t)| = | x (t − τ )h(τ )dτ | ≤ | x (t − τ )h(τ )|dτ
−∞ −∞
"∞ "∞
= | x (t − τ )|| h(τ )|dτ ≤ Mx |h(τ )|dτ < ∞,
−∞ −∞
if (1.30) holds.
It can be shown that the absolute value integrability of the impulse
response is the necessary condition for a linear time-invariant system to be
stable as well.
1.3 FOURIER TRANSFORM
The Fourier series has been introduced and presented for periodic signals,
with a period T. Assume now that we extend the period to infinity, while not
changing the signal. This case corresponds to the analysis of an aperiodic
signal x (t). Its transform, the Fourier series coefficients normalized by the
period, is given by
T/2
" "∞
− j2πnt/T
lim Xn T = lim x (t)e dt = x (t)e− jΩt dt (1.31)
T →∞ T →∞
− T/2 −∞
with 2π/T = ∆Ω → dΩ (being infinitesimal) and 2πn/T → Ω becoming a

continuous variable, as T → ∞ and −∞ < n < ∞.
The function X (Ω), defined by
"∞
X (Ω) = x (t)e− jΩt dt, (1.32)
−∞
is called the Fourier transform (FT) of a signal x (t). For the Fourier trans-
form existence it is sufficient that a signal is absolutely integrable. There
are some signals that do not satisfy this condition, whose Fourier transform
exists in a form of generalized functions, such as delta function.
The inverse Fourier transform (IFT) can be obtained by multiplying
both sides of (1.32) by e jΩτ and integrating over Ω,
"∞ "∞ "∞

X (Ω)e jΩτ dΩ = x (t)e jΩ(τ −t) dtdΩ.
−∞ −∞ −∞
Using the fact that

"∞
e jΩ(τ −t) dΩ = 2πδ(τ − t)
−∞
we get the inverse Fourier transform
"∞
1
x (t) = X (Ω)e jΩt dΩ. (1.33)
2π
−∞
Example 1.8. Calculate the Fourier transform of x (t) = Ae−at u(t), a > 0.
⋆According to the Fourier transform definition we get
"∞
A
X (Ω) = Ae− at e− jΩt dt = .
( a + jΩ)
0
Example 1.9. Find the Fourier transform of

⎧
⎨ 1 for t > 0
x (t) = sign(t) = 0 for t = 0 . (1.34)
⎩
−1 for t < 0
⋆Since a direct calculation of the Fourier transform for this signal is

not possible, let us consider the signal
⎧ − at
⎨ e for t > 0
x a (t) = 0 for t = 0
⎩
−e at for t < 0
where a > 0 is a real-valued constant. It is obvious that
lim x a (t) = x (t).
a →0
The Fourier transform of x (t) can be obtained as

X (Ω) = lim Xa (Ω),
a →0
where
"0 "∞
2Ω
Xa (Ω) = −e at e− jΩt dt + e− at e− jΩt dt = . (1.35)
ja2 + jΩ2
−∞ 0
It results in
2
X (Ω) = . (1.36)
jΩ
Based on the definitions of the Fourier transform and the inverse

Fourier transform, it is easy to conclude that the duality property holds:
If X (Ω) is the Fourier transform of x (t), then the Fourier transform of
X (t) is 2πx (−Ω)
X (Ω) = FT{ x (t)}

2πx (−Ω) = FT{ X (t)}, (1.37)
where FT{◦} denotes the Fourier transform operator.

Example 1.10. Find the Fourier transform of δ(t), x (t) = 1 and u(t).
⋆The Fourier transform of δ(t) is
"∞
FT{δ(t)} = δ(t)e− jΩt dt = 1. (1.38)
−∞
According to the duality property,
FT{1} = 2πδ(Ω).
Finally,
! 6
sign(t) + 1 1
FT{u(t)} = FT = + πδ(Ω). (1.39)
2 jΩ
1.3.1 Fourier Transform and Linear Time-Invariant Systems
Consider a linear, time-invariant system with an impulse response h(t) and

the input signal x (t) = Ae j(Ω0 t+ ϕ) . The output signal is
"∞
y(t) = x (t) ∗t h(t) = Ae j(Ω0 (t−τ )+ ϕ) h(τ )dτ
−∞
"∞
= Ae j(Ω0 t+ ϕ) h(τ )e− jΩ0 τ dτ = H (Ω0 ) x (t), (1.40)
−∞
where
"∞
H (Ω) = h(t)e− jΩt dt (1.41)
−∞
is the Fourier transform of h(t). The linear time-invariant system does not
change the form of an input complex harmonic signal Ae j(Ω0 t+ ϕ) . It remains
complex harmonic signal after passing through the system, with the same
frequency Ω0 . The amplitude of the input signal x (t) is changed for | H (Ω0 )|
and the phase is changed for arg { H (Ω0 )}.
1.3.2 Properties of the Fourier Transform
The Fourier transform satisfies the following properties:

1. Linearity
FT{ a1 x1 (t) + a2 x2 (t)} = a1 X1 (Ω) + a2 X2 (Ω), (1.42)
where X1 (Ω) and X2 (Ω) are the Fourier transforms of signals x1 (t) and
x2 (t), separately.
2. Realness
The Fourier transform of a signal is real (i.e., X ∗ (Ω) = X (Ω)), if
x ∗ (−t) = x (t),
since
"∞ "∞
t→−t
X ∗ (Ω) = x ∗ (t)e jΩt dt = x ∗ (−t)e− jΩt dt = X (Ω), (1.43)
−∞ −∞
if x ∗ (−t) = x (t).
3. Modulation
"∞
FT{ x (t)e jΩ0 t } = x (t)e jΩ0 t e− jΩt dt = X (Ω − Ω0 ) (1.44)
−∞
FT{2x (t) cos(Ω0 t)} = X (Ω − Ω0 ) + X (Ω + Ω0 ).
4. Shift in time
"∞
FT{ x (t − t0 )} = x (t − t0 )e− jΩt dt = X (Ω)e− jt0 Ω . (1.45)
−∞
5. Time-scaling
"∞
1 Ω
FT{ x ( at)} = x ( at)e− jΩt dt = X ( ). (1.46)
| a| a
−∞
6. Convolution
"∞ "∞
FT{ x (t) ∗t h(t)} = x (τ )h(t − τ )e− jΩt dτdt (1.47)
−∞ −∞
"∞ "∞
t−τ →u
= x (τ )h(u)e− jΩ(τ +u) dτdu = X (Ω) H (Ω).
−∞ −∞
7. Multiplication
"∞ "∞
1
FT{ x (t)h(t)} = x (t) H (θ )e jθt dθe− jΩt dt (1.48)
2π
−∞ −∞
"∞
1
= H (θ ) X (Ω − θ )dθ = X (Ω) ∗Ω H (Ω) = H (Ω) ∗Ω X (Ω).
2π
−∞
Convolution in frequency domain is denoted by ∗Ω with a factor of 1/2π

being included.
8. Parseval’s theorem
"∞ "∞
∗ 1
x (t)y (t)dt = X (Ω)Y ∗ (Ω)dΩ (1.49)
2π
−∞ −∞
"∞ "∞
1
| x (t)|2 dt = | X (Ω)|2 dΩ.
2π
−∞ −∞
9. Differentiation
⎧ ⎛ ⎞⎫
! 6 ⎨d "∞ ⎬
dx (t) ⎝ 1
FT = FT X (Ω)e jΩt dΩ⎠ = jΩX (Ω). (1.50)
dt ⎩ dt 2π ⎭
−∞
10. Integration
The Fourier transform of
"t
x (τ )dτ
−∞
can be calculated as the Fourier transform of

"∞ "t
x (t) ∗t u(t) = x (τ )u(t − τ )dτ = x (τ )dτ.
−∞ −∞
Then,
⎧ ⎫
⎨ "t ⎬
FT = FT{ x (t)}FT{u(t)} =
x (τ )dτ (1.51)
⎩ ⎭
−∞
* +
1 1
X (Ω) + πδ(Ω) = X (Ω) + πX (0)δ(Ω).
jΩ jΩ
11. An analytic part of a signal x (t), whose Fourier transform is X (Ω), is a

signal with the Fourier transform defined by
⎧
⎨ 2X (Ω) for Ω > 0
Xa (Ω) = X (0) for Ω = 0 . (1.52)
⎩
0 for Ω < 0
It can be written as
Xa (Ω) = X (Ω) + X (Ω)sign(Ω) = X (Ω) + jXh (Ω) (1.53)
where Xh (Ω) is the Fourier transform of the Hilbert transform of the signal
x (t). From Example 1.9 with the signal x (t) = sign(t) and the duality prop-
erty of the Fourier transform pair, obviously the inverse Fourier transform
of sign(Ω) is j/(πt). Therefore, the analytic part of a signal, in the time
domain, reads as
"∞
j 1 x (τ )
x a (t) = x (t) + jxh (t) = x (t) + x (t) ∗t = x (t) + j dτ. (1.54)
πt π t−τ
−∞
p.v.
where p.v. stands for Cauchy principal value of the considered integral.
1.3.3 Relationship Between the Fourier Series and the Fourier Transform
Consider an aperiodic signal x (t), with the Fourier transform X (Ω). As-
sume that the signal is of a limited duration (i.e., x (t) = 0 for |t| > T0 /2).
Then,
T"0 /2
X (Ω) = x (t)e− jΩt dt. (1.55)
− T0 /2
If we make a periodic extension of x (t), with a period T, we get a signal

∞
x p (t) = ∑ x (t + nT ).
n=−∞
The periodic signal x p (t) can be expanded into Fourier series with the
coefficients
T/2
"
1
Xn = x p (t)e− j2πnt/T dt. (1.56)
T
− T/2
If T > T0 it is easy to conclude that
T/2
" T"0 /2
− j2πnt/T
x p (t)e dt = x (t)e− jΩt dt|Ω=2πn/T
− T/2 − T0 /2
or
1
Xn =
X (Ω)|Ω=2πn/T . (1.57)
T
It means that the Fourier series coefficients are the samples of the Fourier
transform, divided by T. The only condition in the derivation of this relation
is that the signal duration is shorter than the period of periodic extension
(i.e., T > T0 ). The sampling interval in frequency is
2π 2π
∆Ω = , ∆Ω < .
T T0
It should be smaller than 2π/T0 , where T0 is the signal x (t) duration. This
is a form of the sampling theorem in the frequency domain. The sampling
theorem in the time domain will be discussed later.
In order to write the Fourier series coefficients in the Fourier transform
form, note that a periodic signal x p (t), formed by a periodic extension of
x (t) with period T, can be written as
∞ ∞
x p (t) = ∑ x (t + nT ) = x (t) ∗t ∑ δ(t + nT ). (1.58)
n=−∞ n=−∞
The Fourier transform of this periodic signal is

% ;
∞
X p (Ω) = FT x (t) ∗t ∑ δ(t + nT ) (1.59)
n=−∞
* + * + * +
2π ∞ 2π 2π 2π 2π ∞
= X (Ω) · ∑ δ Ω− T n = T ∑ X T n δ Ω− T n
T n=− ∞ n=−∞
since
% ; "∞
∞ ∞
FT ∑ δ(t + nT ) = ∑ δ(t + nT )e− jΩt dt
n=−∞ n=−∞−∞
∞ * +
jΩnT 2π ∞ 2π
= ∑ e = ∑ δ Ω− T n .
T n=−
(1.60)
n=−∞ ∞
The Fourier transform of a periodic signal is a series of generalized

impulse signals at Ω = 2πn/T with weighting factors X ( 2π T n )/T being
equal to the Fourier series coefficients Xn . The relation between periodic
generalized impulse signals in the time and frequency domain will be
explained (derived) later, (see Example 2.8).
1.4 FOURIER TRANSFORM AND STATIONARY PHASE METHOD
When a signal
x (t) = A(t)e jφ(t) (1.61)
is not of a simple analytic form, it may be possible, in some cases, to obtain
an approximative expression for its Fourier transform by using the method
of stationary phase.
The method of stationary phase states that if the phase function φ(t) is
monotonous and the amplitude A(t) is sufficiently smooth function, then
<
"∞
2πj
A(t)e jφ(t) e− jΩt dt ≃ A(t0 )e jφ(t0 ) e− jΩt0 , (1.62)
|φ′′ (t0 )|
−∞
where t0 is the solution of

φ′ (t0 ) = Ω.
The most significant contribution to the integral on the left side of
(1.62) comes from the region where the phase φ(t) − Ωt of the exponential
function exp( j(φ(t) − Ωt)) is stationary in time, since the contribution of
the intervals with fast varying φ(t) − Ωt tends to zero. It means that locally
around an instant t the signal behaves as exp ( j(φ′ (t)t). Value
Ωi ( t ) = φ′ ( t )
is called instantaneous frequency of a signal. Around the stationary phase

instant t0 holds
d (φ(t) − Ωt)
=0
dt | t = t0
φ′ (t0 ) − Ω = 0.
Around this point the phase can be expanded into a Taylor series as
1
φ(t) − Ωt = [φ(t0 ) − Ωt0 ] + [φ′ (t0 ) − Ω] + φ′′ (t0 )t2 + ...
2
Since φ′ (t0 ) − Ω = 0 the integral (1.62) is
"∞ "∞
1 ′′ ( t 2
A(t)e j(φ(t)−Ωt) dt ∼
= A(t0 )e j(φ(t0 )−Ωt0 ) ej 2 φ 0 )t dt
−∞ −∞
where A(t) ∼
= A(t0 ) is also used. With
<
"∞
j 12 at2 2πj
e dt =
| a|
−∞
the stationary phase approximation follows.

If the equation φ′ (t0 ) = Ω has two (or more) solutions t0+ and t0− , then
the integral on the left side of (1.62) is equal to the sum of functions at both
(or more) stationary phase points. Finally, this relation holds for φ′′ (t0 ) ̸= 0.
If φ′′ (t0 ) = 0, then similar analysis may be performed, using the lowest
nonzero phase derivative at the stationary phase point.
Example 1.11. Consider signal
x (t) = exp(−(t2 − 1)t2 ) exp( j4πt2 + j10πt).
Find its Fourier transform approximation by using the stationary phase

method.
⋆According to the stationary phase method,
8πt0 + 10π = Ω
Ω − 10π
t0 =
8π
and
φ′′ (t0 ) = 8π. (1.63)
The amplitude of X (Ω) is
'< ' =
' 2π '' 2π
' 2 2
| X (Ω)| ≃ A(t0 ) ' ' = exp (−( t 0 − 1 ) t 0 )
' φ′′ (t0 ) ' 8π
( >* + ?* + )
1 Ω − 10π 2 Ω − 10π 2
= exp − −1 (1.64)
2 8π 8π
The signal, stationary phase approximation of the Fourier transform and the
numerical value of the Fourier transform amplitudes are shown in Fig.1.7
1
x(t)
-1
-4 -3 -2 -1 0 1 2 3 4
t
1
Stationary phase method

|X(Ω)|
0.5
0
-100 -80 -60 -40 -20 0 20 40 60 80 100
Ω
1
Numeric calculation
|X(Ω)|
0.5
0
-100 -80 -60 -40 -20 0 20 40 60 80 100
Ω
Figure 1.7 The signal (top), along with the stationary phase method approximation of its
Fourier transform and the Fourier transform obtained by a numeric calculation (bottom).
Example 1.12. Consider a frequency-modulated signal
x (t) = A(t) exp( jat2N ).
where A(t) is a slow-varying non-negative function. Find its Fourier trans-

form approximation by using the stationary phase method.
⋆According to the stationary phase method, we get that the stationary
−1
phase point is 2Nat2N
0 = Ω with
* +1/(2N −1)
Ω
t0 =
2Na
and * +
Ω (2N −2)/(2N −1)
φ′′ (t0 ) = 2N (2N − 1) a . (1.65)
2Na
The amplitude and phase of X (Ω), according to (1.62), are
' '
2 2
' 2π '
'
| X (Ω)| ≃ A (t0 ) ' ′′ ' (1.66)
φ ( t0 ) '
* +1/(2N −1) '' * +1/(2N −1) ''
Ω ' 2π Ω '
= A2 ( )' '
2Na ' (2N − 1)Ω 2aN '
* +1/(2N −1)
(1 − 2N ) Ω
arg { X (Ω)} ≃ φ(t0 ) − Ωt0 + π/4 = Ω + π/4
2N 2aN
for a large value of a.

For N = 1 and A(t) = 1, we get | X (Ω)|2 = |π/a| and arg { X (Ω)} =
−Ω2 /(4a) + π/4.
The method of stationary phase may be defined in the frequency

domain as well. For a Fourier transform
X (Ω) = B(Ω)e jθ (Ω) (1.67)
the method of stationary phase states that if the Fourier transform phase
function θ (t) is monotonous and the amplitude B(t) is sufficiently smooth
function, then
<
"∞
1 jθ (Ω) jΩt jθ (Ω0 ) jΩ0 t j
x (t) = B(Ω)e e dΩ ≃ B(Ω0 )e e , (1.68)
2π 2π |θ ′′ (Ω0 )|
−∞
where Ω0 is the solution of
−θ ′ (Ω0 ) = t,
and
t g = −θ ′ (Ω)
is the group delay.
Example 1.13. Consider a system with transfer function
H (Ω) = exp(−Ω2 ) exp(− jaΩ2 /2 − jbΩ).
Find its impulse response by using the stationary phase method.

⋆According to the stationary phase method,
aΩ0 + b = t
t−b
Ω0 =
a
and
θ ′′ (Ω0 ) = − a.
The impulse response is
<
2 j
h(t)) ≃ exp(−Ω20 )e− jaΩ0 /2− jbΩ0 + jΩ0 t
2π |θ ′′ (Ω0 )|
* + =
t − b 2 j((t−b)2 /(2a)+π/4) 1
= exp(− )e .
a 2πa
The signal amplitude is delayed for b. The second order parameter a in

the phase function scales time axis of the impulse response. This is an
undesirable effect in common systems.
Example 1.14. For a system with frequency response H (Ω) = | H (Ω)| e j0 the im-
pulse response is h(t). Find the impulse response of the systems with transfer
functions shown in Fig.1.8 with:
(a) Ha (Ω) = | H (Ω)| e− j4Ω ,
2
(b) Hb (Ω) = | H (Ω)|@e− j2πΩ , and A
3
(c) Hc (Ω) = | H (Ω)| 4 + 14 cos(2πΩ2 ) e j0 .
⋆(a) The impulse response is
"∞
1
h a (t) = H (Ω)e− j4Ω e jΩt = h(t − 4).
2π
−∞
It is delayed with respect to h(t) for t0 = 4.

(b) In this case
"∞
1 2
hb (t) = H (Ω)e− j2πΩ e jΩt dΩ.
2π
−∞
The group delay is t g = −θ ′ (Ω) = 4πΩ. According to the stationary phase

method, by replacing Ω with t/(4π ), we get
* + =
t j(t2 /8π +π/4) 1
hb (t) = H e .
4π 8π 2
|Hb(jΩ)|
|Ha(jΩ)|
|H (jΩ)|
c
Ω Ω Ω
arg{Hb(jΩ)}
arg{Ha(jΩ)}
arg{H (jΩ)}
c
Ω Ω Ω
|ha(t)|=|h(t-4)|
|hb(t)|
|h (t)|
c
t t t
Figure 1.8 Frequency response of systems (amplitude, top row, and phase, middle row) with
corresponding impulse responses (amplitude - bottom row).
Nonlinear group delay causes & ∞ time scaling

&∞and form change of the impulse
response. Note: Check that −∞ h2 (t)dt = −∞ h2b (t)dt. The impulse response
is calculated numerically as well. The agreement with the approximative
result is high, Fig.1.8(third row, middle column)
(c) Since 2 cos(2πΩ2 ) = exp( j2πΩ2 ) + exp(− j2πΩ2 )
3 1 1
hc (t) = h(t) + h (t) + hb (−t)
4 8 b 8
3 1
= h(t) + h ( t ).
4 4 b
Here, fast variations of the amplitude result in a two-component impulse
response, one being proportional to the impulse response from case (a) and
the other proportional to the form from (b).
1.5 LAPLACE TRANSFORM
The Fourier transform could be considered as a special case of the Laplace

transform. At the beginning, Fourier’s work was even not published as an
original contribution due to this fact. The Laplace transform is defined by
"∞
X (s) = L{ x (t)} = x (t)e−st dt, (1.69)
−∞
where s = σ + jΩ is a complex number. It is obvious that the Fourier

transform is the value of a Laplace transform along the imaginary axis, σ = 0
or s = jΩ. This form of the Laplace transform is also known as the bilateral
Laplace transform (in contrast to unilateral one, where the integration limits
are from 0− to ∞).
Example 1.15. Calculate the Laplace transform of x (t) = e−at u(t).
⋆According to the definition
'∞
'
"∞ e−(s+ a)t ' 1
X (s) = e− at e−st dt = − 0
=
s+a s+a
0
if
lim e−(s+ a)t = 0
t→∞
or σ + a > 0, that is, σ > − a. Therefore, the region of convergence of this
Laplace transform is the region where σ > − a. The point s = − a is the pole of
the Laplace transform. The region of convergence is limited by a vertical line

in the complex s-plane, passing through a pole.
The Laplace transform may be considered as a Fourier transform of a

signal x (t) multiplied by exp(−σt), with varying parameter σ,
"∞ "∞
−σt −σt − jΩt
FT{ x (t)e }= x (t)e e dt = x (t)e−st dt = X (s). (1.70)
−∞ −∞
In this way, we may calculate the Laplace transform of functions that

are not absolutely integrable
&∞ (i.e., do not satisfy condition for the Fourier
transform existence, −∞ | x (t)| dt < ∞) In these cases, for some values of
σ, the new signal x (t)e−σt may be absolutely integrable and the Laplace
transform could exist. In the previous example, the Fourier transform does
not exist for a < 0, while for a = 0 it exists in the generalized functions
sense only. Laplace transform of the considered signal always exists, with
the region of convergence σ > − a. If a > 0, then the region of convergence
σ > − a includes the line σ = 0, meaning that the Fourier transform exists.
The inverse Laplace transform is
γ"+ jT
1
x (t) = lim X (s)est ds
2πj T →∞
γ− jT
where the integration is performed along a path in the region of conver-

gence of X (s).
Example 1.16. Consider a signal x (t) such that x (t) = 0 for |t| > T (time-limited
signal). Its Fourier transform is X (Ω). Derive the relation to calculate the
Laplace transform X (s) for any σ within the region of convergence, based
on the value of X (Ω).
1
&∞ jΩt dΩ. The
⋆Based on X (Ω) the signal values are x (t) = 2π −∞ X (Ω)e
Laplace transform is
⎛ ⎞
"T "∞
1
X (s) = ⎝ X (Ω)e dΩ⎠ e−st dt
jΩt
2π
−T −∞
"∞ "T "∞
1 1 sinh(( jΩ − s) T )
= X (Ω) e−st+ jΩt dt dΩ = X (Ω) dΩ. (1.71)
2π π jΩ − s
−∞ −T −∞
within the region of convergence.

Properties of the Laplace transform may easily be generalized from

those presented for the Fourier transform, like for example
L{ ax (t) + by(t)} = aL{ x (t)} + bL{y(t)} = aX (s) + bY (s),
L{ x (t) ∗t h(t)} = L{ x (t)}L{ h(t)} = X (s) H (s).
Since the Laplace transform will be used to describe linear systems de-
scribed by linear differential equations we will consider only the relation of
the signal derivatives with the corresponding forms in the Laplace domain.
In general the Laplace transform of the first derivative dx (t)/d(t) of a signal
x (t) is
"∞ "∞
dx (t) −st
e dt = s x (t)e−st dt = sX (s).
dt
−∞ −∞
This relation follows by integration in part of the first integral, with the
assumption that the values of x (t)e−st are zero as t → ±∞.
In many applications it has been assumed that the systems are causal
with corresponding causal signals used in calculations. In these cases x (t) =
0 for t < 0, i.e., x (t) = x (t)u(t). Then the so called one-sided Laplace trans-
form (unilateral Laplace transform) is used. Its definition is
"∞
X (s) = x (t)e−st dt.
0
When dealing with the derivatives of causal signals we have to take

care about possible discontinuity at t = 0.
In general the first derivative of the function x (t)u(t) is
d( x (t)u(t)) dx (t)
= u ( t ) + x (0 ) δ ( t ).
dt dt
The Laplace transform of the first derivative of a causal signal is
"∞ "∞
dx (t) −st
e dt = x (t)e−st |0∞ + s x (t)e−st dt = sX (s) − x (0).
dt
0 0
Value of signal at t = 0, denoted by x (0), is the initial condition.

These relations can easily be generalized to higher order derivatives
"∞ n "∞
d x (t) −st
n
e dt = s n
x (t)e−st dt − sn−1 x (0) − sn−2 x ′ (0) − ... − x (n−1) (0)
dt
0 0
= s X (s) − sn−1 x (0) − sn−2 x ′ (0) + ... − x (n−1) (0).
n
The Laplace transform of an integral of x (t) is
"t
1
L{ x (τ )dτ } = L{u(t) ∗t x (t)} = X (s)},
s
0
&∞
since L{u(t)} = 0 e−st dt = 1/s.
The initial and final values of the signal are x (0) = lims→∞ sX (s) and
x (∞) = lims→0 sX (s), respectively.
1.5.1 Linear Systems Described by Differential Equations
After we have established the relation between the Laplace transform and
signals derivatives we may use it to analyze the systems described by
differential equations. Consider a causal system
d N y(t) dy(t) d M x (t) dx (t)

aN + ... + a 1 + a 0 y ( t ) = b M + ... + b1 + b0 x (t)
dt N dt dtn dt
with the initial conditions x (0) = x ′ (0) = x (n−1) (0) = 0. The Laplace trans-
form of both sides of this differential equation is
a N s N Y (s) + ... + a1 sY (s) + a0 Y (s) = b M s M X (s) + ... + b1 sX (s) + b0 X (s).
Transfer function of this system is of the form
Y (s) b s M + ... + b1 s + b0
H (s) = = M N .
X (s) a N s + ... + a1 s + a0
Example 1.17. A causal system is described by the differential equation

d2 y ( t ) dy(t)
+3 + 2y(t) = x (t)
dt2 dt
with the initial conditions y′(0) = 1 and y(0) = 0. Find the system output y(t)
for x (t) = e−4t u(t).
⋆The Laplace transform of both sides is

[s2 Y (s) − sy(0) − y′ (0)] + 3[sY (s) − y(0)] + 2Y (s) = X (s)
or
Y (s)(s2 + 3s + 2) = X (s) + sy(0) + y′ (0) + 3y(0).
With X (s) = 1/(s + 4) follows
s+5 A1 A2 A3
Y (s) = = + + .
(s + 4)(s2 + 3s + 2) s+4 s+1 s+2
The coefficients Ai are obtained from
A i = ( s − s i )Y ( s ) | s = s i .
For example,
s+5
A1 = ( s + 4) = 1/6.
(s + 4)(s2 + 3s + 2) |s=−4
The other two coefficients are A2 = −3/2 and A3 = 4/3.

The output signal is
1 −4t 3 4
y(t) = e u(t) − e−2t u(t) + e−t u(t).
6 2 3
1.5.2 Table of the Laplace Transform
Signal x (t) Laplace transform X (s)

δ(t) 1
u(t) 1/s
1
e at u(t) s− a
tu(t) 1/s2
s− a
e at cos(Ω0 t)u(t) (s− a)2 +Ω20
Ω0
e at sin(Ω0 t)u(t) (s− a)2 +Ω20
1
te at u(t) ( s − a )2
x ′ (t)u(t) sX (s) − x (0)
tx (t)u(t) −
&∞dX (s)/ds
x (t)u(t)/t s F (s )ds
e at x (t)u(t) X (s − a)
&t
x (t) ∗ u(t) = 0 x (t)dt X (s)/s
N=2 N=4 N=32

|H(jΩ)|2
|H(jΩ)|2
|H(jΩ)|
Ω Ω Ω
Figure 1.9 Squared amplitude of the frequency response of a Butterworth filter of order N.
1.6 BUTTERWORTH FILTER
The most common processing systems in communications and signal pro-

cessing are filters, used to selectively pass a part of the input signal in the
frequency domain and to reduce possible interferences. The basic form is a
lowpass filter. Here we will present a simple Butterworth lowpass filter.
The squared frequency response of the Butterworth lowpass filter is
1
| H ( jΩ)|2 = B C2N .
Ω
1+ Ωc
It is shown in Fig.1.9. This filter definition contains two parameters. Order of

the filter is N. It is a measure of the transition sharpness from the passband
to the stopband region. For N → ∞ the amplitude form of an ideal lowpass
filter is achieved. The second parameter is the critical frequency. At Ω = Ωc
we get | H ( jΩc )|2 = | H (0)|2 /2 = 1/2, corresponding to −3[dB] gain for any
filter order N.
The squared frequency response may be written as
1
H ( jΩ) H (− jΩ) = B C2N
jΩ
1+ jΩc
1
H (s) H (−s) = B C2N for s = jΩ.
s
1+ jΩc
N=3 N=4 N=5

Im{s}
Im{s}
Im{s}
Re{s} Re{s} Re{s}
Figure 1.10 Poles of a stable Butterworth filter for N = 3, N = 4, and N = 5.
Poles of the product of transfer functions H (s) H (−s) are

* +2N
sk
= −1 = e j(2πk+π )
jΩc
sk = Ωc e j(2πk+π )/2N + jπ/2 for k = 0, 1, 2, ..., 2N − 1.
Poles of the Butterworth filter are located on a circle whose radius is Ωc at

the angles
2πk + π π
αk = + for k = 0, 1, 2, ..., 2N − 1.
2N 2
For a given filter order N and frequency Ωc the only remaining decision is
to select a half of the poles sk that belong to H (s) and to declare that the
remaining half of the poles belong to H (−s). Since we want that a filter is
stable then we chose the poles
s0 , s1 , ..., s N −1
within the left side of the s plane, where Re{s} < 0, i.e., π/2 < αk < 3π/2.
The symmetric poles with Re {s} > 0 are the poles of H (−s). They are not
used in the filter design.
Example 1.18. Design a lowpass Butterworth filter with:
(a) N = 3 with Ωc = 1,
(b) N = 4 with Ωc = 3.
⋆(a) Poles for N = 3 with Ωc = 1 have the phases
2πk + π π
αk = + , for k = 0, 1, 2.
6 2
Their values are

√
2π 2π 1 3
s0 = cos( ) + j sin( ) = − + j
3 3 2 2
2π π 2π π
s1 = cos( + ) + j sin( + ) = −1
3 3 3 3
√
2π 2π 2π 2π 1 3
s2 = cos( + ) + j sin( + )=− −j
3 3 3 3 2 2
with
c 1
H (s) = √ √ =
(s + 1
−j 3 1 3 (s2 + s + 1)(s + 1)
2 2 )( s + 2 +j 2 )( s + 1)
where c = 1 is used to make H (0) = 1.

(b) Poles for N = 4 with Ωc = 3 are at angles
2πk + π π
αk = + , for k = 0, 1, 2, 3.
8 2
Their values are

π π π π
s0 = 3 cos( + ) + j3 sin( + )
2 8 2 8
π 3π π 3π
s1 = 3 cos( + ) + j3 sin( + )
2 8 2 8
π 5π π 5π
s2 = 3 cos( + ) + j3 sin( + )
2 8 2 8
π 7π π 7π
s3 = 3 cos( + ) + j3 sin( + )
2 8 2 8
with
c
H (s) =
(s2 + 2.296s + 9)(s2 + 5.543s + 9)
9
= 2
(s + 2.296s + 9)(s2 + 5.543s + 9)
where c = 9 is used to make H (0) = 1.
In practice we usually do not know the filter order, but its passband
frequency Ω p and stopband frequency Ωs , with a maximal attenuation in
the passband a p [dB] and a minimal attenuation in the stopband a p [dB], as
shown in Fig.1.11. Based on these values we can calculate the order N and
the critical frequency Ωc needed for a filter design.
1
A
p
|H(jΩ)|2
A
s
Ω Ω
p s
Figure 1.11 Specification of a Butterworth filter parameters in the passband and stopband.
The relations for N and Ωc are
1 2
B C2N ≥ A p (1.72)
Ωp
1+ Ωc
1 2
B C2N ≤ As .
Ωs
1+ Ωc
Using equality in both relations, it follows
ln( A12 − 1) − ln( A12 − 1)

1 p s
N= .
2 ln Ω p − ln Ωs
Nearest greater integer is assumed for the filter order N. Then we can
use any of the relations in (1.72) with equality sign to calculate Ωc . If we
' '2
choose the first one then Ωc will satisfy ' H ( jΩ p )' = A2p , while if we use
the second relation the value of Ωc will satisfy | H ( jΩs )|2 = A2s . These two
values differ. However both of them are within the defined criteria for the
transfer function.
The relation
a = 20 log A
or A = 10a/20 should be used for the attenuation given in [dB] .
All other filter forms, like passband and highpass, may be obtained
from a lowpass filter with appropriate signal modulations. These modula-
tions will be discussed for discrete-time filter forms in Chapter V.
Part II
Discrete Signals and Systems
57
Chapter 2
Discrete-Time Signals and Transforms
first step in numerical processing of signals is in their discretiza-
T
HE
tion in time. A continuous-time signal is converted into a sequence
of numbers, defining the discrete-time signal. The basic definitions
of discrete-time signals and their transforms are presented in this chapter.
The key fact in the conversion from a continuous-time signal into a sequence
of numbers is that these two signal representations are equivalent under cer-
tain conditions. The discrete-time signal may contain the same information
as the original continuous-time signal. The sampling theorem is fundamen-
tal for this relation between two signal forms. It is presented in this chapter,
after basic definitions of discrete-time signals and systems are introduced.
2.1 DISCRETE-TIME SIGNALS
Discrete-time signals (discrete signals) are represented in a form of an or-

dered set of numbers { x (n)}. Commonly, they are obtained by sampling
continuous-time signals. There exist discrete-time signals whose indepen-
dent variable is inherently discrete in nature as well.
In the case that a discrete-time signal is obtained by sampling a
continuous-time signal, we can write (Fig. 2.1),
x (n) = x (t)|t=n∆t ∆t. (2.1)
Discrete-time signals are defined for an integer value of the argument

n. We will use the same notation for continuous-time and discrete-time
signals, x (t) and x (n). However, we hope that this will not cause any
confusion since we will use different sets of variables, for example, t and
τ for continuous time and n and m for discrete time. Also, we hope that the
59
60 Discrete-Time Signals and Transforms
x(t) x(n) = x(t) Δt t = nΔt
Δt t n
Figure 2.1 Signal discretization: continuous-time signal (left) and corresponding discrete-
time signal (right).
context will always be clear, so that there is no doubt what kind of signal is
considered. Notation x [n] is sometimes used in literature for discrete-time
signals, instead of x (n).
Examples of discrete-time signals are presented next.

The discrete-time impulse signal is defined by
!
1, for n = 0
δ(n) = . (2.2)
0, for n ̸= 0
It is presented in Fig. 2.2.

In contrast to the continuous-time impulse signal, that cannot be prac-
tically implemented and used, the discrete-time unit impulse is a signal that
can easily be implemented and used in realizations. In mathematical nota-
tion, this signal corresponds to the Kronecker delta function
!
1, for m = n
δm,n = (2.3)
0, for m ̸= n.
Any discrete-time signal can be written in a form of a sum of shifted

and weighted discrete-time impulses,
∞
x (n) = ∑ x ( k ) δ ( n − k ), (2.4)
k=−∞
as illustrated in Fig.2.3.
The discrete unit-step signal is defined by
!
1, for n ≥ 0
u(n) = . (2.5)
0, for n < 0
1 1
x(n)=u(n)
δ(n)
0 0
-1 -1
(a) (b)
-10 0 10 -10 0 10
t n
1 1
x(n)=b(n)
sin(nπ/4)
0 0
-1 -1
(c) (d)
-10 0 10 -10 0 10
n n
Figure 2.2 Illustration of discrete-time signals: (a) unit-step function, (b) discrete-time im-
pulse signal, (c) boxcar signal b(n) = u(n + 2) − u(n − 3), and (d) discrete-time sinusoid.
4 4
2 2
-2 δ (n+2)
x(n)
0 0
-2 -2
-4 -4
-5 0 5 -5 0 5
n n
4 4
2 2
- δ(n-1 )
3δ(n)
0 0
-2 -2
-4 -4
-5 0 5 -5 0 5
n n
Figure 2.3 Signal x (n) along with corresponding discrete-time impulses.

The discrete-time impulse and the unit-step signal are related as
δ ( n ) = u ( n ) − u ( n − 1)
n
u(n) = ∑ δ ( k ).
k=−∞
The discrete-time complex sinusoidal signal is defined by
x (n) = Ae j(ω0 n+ ϕ) = A cos(ω0 n + ϕ) + jA sin(ω0 n + ϕ). (2.6)
A discrete-time signal is periodic if there exists an integer N such that
x ( n + N ) = x ( n ). (2.7)
Smallest positive integer N that satisfies this equation is called the

period of the discrete-time signal x (n). Note that the signal x (n) with
a period N is also periodic with any integer multiple of N. Some basic
discrete-time signals are presented in Fig. 2.2.
Example 2.1. Check the periodicity of discrete-time signals x1 (n) = sin(2πn/36),
x2 (n) = cos(4πn/15 + 2), x3 (n) = exp( j0.1n), x4 (n) = x1 (n) + x2 (n), and
x5 ( n ) = x1 ( n ) + x3 ( n ).
⋆Period of the discrete-time signal x1 (n) = sin(2πn/36) is obtained
from 2πN1 /36 = 2πk, where k is an integer. It is N1 = 36, for k = 1. The period
N2 follows from 4πN2 /15 = 2πk as N2 = 15 with k = 2. Period of signal x3 (n)
should be calculated from 0.1N3 = 2πk. Obviously, there is no integer k such
that N3 is an integer. This signal is not periodic. The same holds for x5 (n). The
period of x4 (n) is a common period for signals x1 (n) and x2 (n) with N1 = 36
and N2 = 15. It is N4 = 180.
A discrete-time signal is even if
x (n) = x (−n).
For an odd signal holds

x (n) = − x (−n).
Example 2.2. Show that a discrete-time signal may be written as a sum
x (n) = xe (n) + xo (n)
where xe (n) and xo (n) are its even and odd part, respectively.
⋆For a signal x (n) we can form its even and odd part as
x (n) + x (−n)
xe (n) =
2
x (n) − x (−n)
xo (n) = .
2
Summing these two parts, the signal x (n) is reconstructed. Note that xo (0) =
0.
A signal is Hermitian if
x (n) = x ∗ (−n).
Magnitude of a discrete-time signal is defined as the maximal value of

the signal amplitude
Mx = max | x (n)| .
−∞<n<∞
Energy of discrete-time signals is defined by
∞
Ex = ∑ | x (n)|2 . (2.8)
n=−∞
The instantaneous power of x (n) is Px (n) = | x (n)|2 , while the average

signal power is
1 N # $
PAV = lim ∑ | x (n)|2 = | x (n)|2 , (2.9)
N →∞ 2N + 1 n=− N
# $
where | x (n)|2 is used to denote an average over large number of signal
values, as N → ∞. The average power of signals with a finite energy (energy
signals) is PAV = 0. For power signals (when 0 < PAV < ∞) the energy is
infinite, Ex → ∞.
Example 2.3. The energy of signal x (n) is Ex = 10. The energy of its even part is
Exe = 3. Find the energy of its odd part.
⋆The energy of signal is
∞ ∞
Ex = ∑ | x (n)|2 = ∑ | xe (n) + xo (n)|2
n=−∞ n=−∞
∞
= ∑ [ xe (n) + xo (n)][ xe (n) + xo (n)]∗
n=−∞
∞ ∞ ∞
= ∑ | xe (n)|2 + ∑ | xo (n)|2 + ∑ [ xo (n) xe∗ (n) + xe (n) xo∗ (n)].
n=−∞ n=−∞ n=−∞
The terms xo (n) xe∗ (n) and xe (n) xo∗ (n) in the last sum correspond to odd
signals
xo (−n) xe∗ (−n) = − xo (n) xe∗ (n)

xe (−n) xo∗ (−n) = − xe (n) xo∗ (n).
Their sum is zero,

∞ ∞
∑ xo (n) xe∗ (n) = ∑ xe (n) x0∗ (n) = 0.
n=−∞ n=−∞
For the signals xe (n) and xo (n), satisfying the previous relation, we say that
they are orthogonal.
Therefore, for the energies Ex , Exe , and Exo , holds
Ex = Ex e + Ex o .
Obviously Ex0 = Ex − Exe = 7.
2.1.1 Discrete-Time Systems
Discrete-time (discrete) system transforms one discrete-time signal (input)

into the other (output signal)
y(n) = T { x (n)}. (2.10)
A discrete system T {·} is linear if for any two signals x1 (n) and x2 (n) and
any two constants a1 and a2 holds
y(n) = T { a1 x1 (n) + a2 x2 (n)} = a1 T { x1 (n)} + a2 T { x2 (n)}. (2.11)
A discrete system is time-invariant if for
y(n) = T { x (n)} (2.12)
holds
T { x (n − n0 )} = y(n − n0 ),
for any t0 .
For any input signal x (n) the signal at the output of a linear time-
invariant discrete system can be calculated if we know the output to the
impulse signal. The output to the impulse signal, h(n) = T {δ(n)}, is the
impulse response.
3 3
2 2
1 1
h(n)
x(n)
0 0
-1 -1
-2 -2
-4 -2 0 2 4 6 8 -4 -2 0 2 4 6 8
n n
Figure 2.4 Input signal and impulse response.
The output to an input signal x (n) is

% ;
∞
y(n) = T { x (n)} = T ∑ x (k )δ(n − k ) .
k=−∞
For a linear time-invariant discrete system we get

∞ ∞
y(n) = ∑ x (k )T {δ(n − k )} = ∑ x ( k ) h ( n − k ). (2.13)
k=−∞ k=−∞
This is a discrete-time convolution. Its notation is

∞
x (n) ∗n h(n) = ∑ x ( k ) h ( n − k ). (2.14)
k=−∞
Discrete-time convolution is a commutative operation,
x ( n ) ∗ n h ( n ) = h ( n ) ∗ n x ( n ). (2.15)
Example 2.4. Calculate discrete-time convolution of signals x (n) and h(n) shown
in Fig. 2.4.
⋆By definition, according to Fig. 2.5, we have
∞
y (0) = ∑ x (k)h(−k) = 1 − 1 + 2 = 2,
k=−∞
∞
y (1) = ∑ x (k)h(1 − k ) = −1 − 1 + 1 + 4 = 3.
k=−∞
In a similar way y(−2) = 2, y(−1) = −1, y(2) = 6, y(3) = 2, y(4) = −1,

y(5) = −1, and y(n) = 0, for all other n. The convolution y(n) is shown in
Fig. 2.6.
3 3
2 2
h(-k )
1 1
x(k)
0 0
-1 -1
-2 -2
-4 -2 0 2 4 6 8 -4 -2 0 2 4 6 8
n n
3 3
2 2
h(1- k)
h(2- k)
1 1
0 0
-1 -1
-2 -2
-4 -2 0 2 4 6 8 -4 -2 0 2 4 6 8
n n
Figure 2.5 Signals for the output y(0), y(1), and y(2) calculation.
8
6
4
y(n)
2
0
-2
-4 -2 0 2 4 6 8
n
Figure 2.6 Resulting output signal y(n).
Example 2.5. Calculate the convolution of signals x (n) = n[u(n) − u(n − 10)] and
h ( n ) = u ( n ).
⋆The convolution is
∞
y(n) = ∑ k[u(k) − u(k − 10)]u(n − k ) =
k=−∞
⎧ n
⎪
⎪ n +1
⎨ ∑ k=n 2 for 0≤n≤9
k =0
= ∑ k=
⎪ 9
0≤k ≤9 and k≤n ⎪
⎩ ∑ k = 45 for n>9
k =0
n+1
=n [u(n) − u(n − 10)] + 45u(n − 10).
2
Example 2.6. If the response of a linear time-invariant system to the unit-step is

y(n) = T {u(n)} = e−n u(n) find the impulse response h(n) of this system.
⋆The impulse response is
h(n) = T {δ(n)} = T {u(n) − u(n − 1)} = T {u(n)} − T {u(n − 1)}
= e−n u(n) − e−(n−1) u(n − 1) = e−n [u(n) − e u(n − 1)].
A discrete system is causal if there is no response before the input

signal appears. For causal discrete systems h(n) = 0 for n < 0 holds. For a
signal that may be an impulse response of a causal system we say that it is
a causal signal or one-sided signal.
A discrete system is stable if any input signal with finite magnitude
Mx = max−∞<t<∞ | x (n)| produces the output y(n) whose values are finite,
|y(n)| < ∞. A discrete linear time-invariant system is stable if
∞
∑ |h(m)| < ∞. (2.16)
m=−∞
The output of a linear time-invariant system is

∞ ∞
|y(n)| = | ∑ x (n − m)h(m)| ≤ ∑ | x (n − m)||h(m)|
m=−∞ m=−∞
∞
≤ Mx ∑ |h(m)|.
m=−∞
Therefore |y(n)| < ∞ if (2.16) holds. It can be shown that the absolute sum
convergence of the impulse response is the necessary condition for a linear
time-invariant discrete system to be stable as well.
2.2 FOURIER TRANSFORM OF DISCRETE-TIME SIGNALS
The Fourier transform of a discrete-time signal is defined by

∞
X (e jω ) = ∑ x (n)e− jωn . (2.17)
n=−∞
Notation X (e jω ) is used to emphasize the fact that it is a periodic function

of the normalized frequency ω. The period is 2π.
In order to establish the relation between the Fourier transform of

discrete-time signals and the Fourier transform of continuous-time signals,
"∞
X (Ω) = x (t)e− jΩt dt,
−∞
we will write an approximation of the Fourier transform of continuous-time
signal according to the rectangular rule of numerical integration,
∞
X (Ω) ∼
= ∑ x (n∆t)e− jΩn∆t ∆t. (2.18)
n=−∞
By using the notation
x (n∆t)∆t −→ x (n)
Ω∆t −→ ω, (2.19)
the previous approximation can be written as

∞
∑ x (n)e− jωn = X (e jω ). (2.20)
n=−∞
This is the Fourier transform of the discrete-time signal x (n).

Later we will show that, under certain conditions, the Fourier trans-
form X (e jω ) of discrete-time signals is not just an approximation of the
Fourier transform X (Ω) of continuous-time signals, but the equality holds
(i.e., X (Ω) = X (e jω )) with Ω∆t = ω and −π ≤ ω < π.
The inverse Fourier transform of discrete-time signals is obtained by
multiplying both sides of (2.20) by e jωm and integrating them over a period
of X (e jω )
∞ "π "π
− jω (n−m)
∑ x (n) e dω = X (e jω )e jωm dω.
n=−∞ −π −π
Since
"π
sin ((n − m)π )
e− jω (n−m) dω = 2 = 2πδ(n − m),
(n − m)
−π
we get
"π
1
x (n) = X (e jω )e jωn dω. (2.21)
2π
−π
Example 2.7. Find the Fourier transform of the discrete-time signal
x (n) = Ae−α|n|
where α > 0 is a real constant.

⋆The Fourier transform of this signal is
−1 ∞
X (e jω ) = A + ∑ Aeαn− jωn + ∑ Ae−αn− jωn
n=−∞ n =1
( )
e jω −α e− jω −α 1 − e−2α
= A 1+ + =A . (2.22)
1 − e jω −α 1 − e− jω −α 1 − 2e−α cos(ω ) + e−2α
Example 2.8. Find the inverse Fourier transform of a discrete-time signal if

X (e jω ) = 2πδ(ω ) for −π ≤ ω < π and X (e jω ) = 2π ∑∞
k=−∞ δ (ω + 2kπ ) for
any ω.
⋆By definition
"π
1
x (n) = 2πδ(ω )e jωn dω = 1.
2π
−π
Therefore, the Fourier transform of signal x (n) = 1 is

∞ ∞
∑ e− jωn = 2π ∑ δ(ω + 2kπ ). (2.23)
n=−∞ k=−∞
The equivalent form in the continuous-time domain is obtained (by using

ω = ΩT and δ( TΩ) = δ(Ω)/T) as
∞ ∞
2π ∞
∑ e jΩnT = 2π ∑ δ(ΩT + 2kπ ) = ∑ δ(Ω + 2kπ/T ). (2.24)
n=−∞ k=−∞
T k=− ∞
2.2.1 Properties
The Fourier transform of discrete-time signals is linear
FT{ ax (n) + by(n)} = aX (e jω ) + bY (e jω ), (2.25)
where X (e jω ) and Y (e jω ) are the Fourier transforms of the discrete-time

signals x (n) and y(n), respectively.
With respect to the signal shift and modulation the Fourier transform
of discrete-time signals behaves in the same way as the Fourier transform
of continuous-time signals,
FT{ x (n − n0 )} = X (e jω )e jn0 ω (2.26)
and
FT{ x (n)e jω0 n } = X (e j(ω −ω0 ) ). (2.27)
Example 2.9. The Fourier transform of a discrete-time signal x (n) is X (e jω ).
Find the Fourier transform of y(n) = x (2n).
⋆For y(n) = x (2n) the Fourier transform is
∞
FT{ x (2n)} = ∑ x (2n)e− jωn
n=−∞
∞
x (n) + (−1)n x (n)
= ∑ e− jωn/2
n=−∞ 2
∞
1
= ∑ [x(n) + e− jnπ x(n)]e− jωn/2
2 n=− ∞
1 1
= [ X (e jω/2 ) + X (e j(ω/2+π ) )] = [ X (e jω/2 ) + X (e j(ω +2π )/2 )]. (2.28)
2 2
The period of this Fourier transform is 2π. Period of X (e jω/2 ) is 4π.
Example 2.10. Calculate the Fourier transform of the discrete-time signal (rectan-
gular window),
w R ( n ) = u ( N + n ) − u ( n − N − 1 ). (2.29)
Write the Fourier transform of a Hann(ing) window
1
w H (n) = [1 + cos(nπ/N )] [u( N + n) − u(n − N − 1)] .
2
⋆By definition
N
1 − e− jω (2N +1) sin(ω 2N2+1 )
WR (e jω ) = ∑ e− jωn = e jωN = . (2.30)
n=− N 1 − e− jω sin(ω/2)
The Fourier transform of the Hann(ing) window can easily be written as

* +
1 N 1 1
WH (e jω ) = ∑ 1 + e jnπ/N + e− jnπ/N e− jωn = (2.31)
2 n=− N 2 2
π 2N +1 π 2N +1
sin(ω 2N2+1 ) sin((ω − N ) 2 ) sin((ω + N ) 2 )
= + π + π . (2.32)
2 sin(ω/2) 4 sin((ω − N )/2) 4 sin((ω + N )/2)
1
N=4
W (ejω)
w (n)
0.5
R
R
0
-10 0 10 -π ω π
n
1
N=8
W (ejω)
w (n)
0.5
R
R
0
-10 0 10 -π ω π
n
1
N=8
W (ejω)
w (n)
0.5
H
-10 0 10 -π ω π
n
Figure 2.7 Discrete-time signal in a form of rectangular window of the widths 2 N + 1 = 9 and
2N + 1 = 17 samples (top and middle), and a Hann(ing) window with 2N + 1 = 17 (bottom).
The time domain values are on the left while the Fourier transforms of these discrete-time
signals are on the right.
As the window width increases in the time domain the main lobe width in
the Fourier domain is narrowing. The first zero value of the Fourier transform
of a rectangular window is at ω (2N + 1)/2 = π, i.e., at ω = 2π/(2N + 1)
where 2N + 1 is the signal duration. In the case of a Hann(ing) window
the main lobe is wider as compared to the rectangular window of the same
width, but its convergence is much faster with very reduced oscillations in
the Fourier transform, Fig.2.7.
The Fourier transform of a convolution of discrete-time signals,
∞ ∞
FT{ x (n) ∗n h(n)} = ∑ ∑ x (k )h(n − k )e− jnω = X (e jω ) H (e jω ), (2.33)
n=−∞ k=−∞
is equal to the product of the Fourier transforms of corresponding discrete-

time signals.
The Fourier transform of the impulse response
∞
H (e jω ) = ∑ h(n)e− jωn
n=−∞
is called frequency response of a discrete linear time-invariant system.
Example 2.11. Find the output of a discrete linear time-invariant system with
frequency response H (e jω ) if the input signals are:
(a) x (n) = Ae jω0 n and (b) x (n) = A cos(ω0 n + ϕ). What is the output if
the impulse response h(n) is real-valued?
⋆(a) The output signal to the input x (n) is
∞ ∞
y(n) = ∑ h(k ) x (n − k ) = ∑ h(k) Ae jω0 (n−k)
k=−∞ k=−∞
∞
= Ae jω0 n ∑ h(k) Ae− jω0 k = Ae jω0 n H (e jω0 )
k =−∞
' '
' ' jω
= A 'H (e jω0 )' e j(ω0 n+arg{ H (e 0 }) .
(b) The input signal can be written as
A j ( ω0 n + ϕ ) A − j ( ω0 n + ϕ )
x (n) = A cos(ω0 n + ϕ) = e + e .
2 2
According to the linearity property, the output is
A '' '
' jω0
y(n) = 'H (e jω0 )' e j(ω0 n+ ϕ)+ j arg{ H (e )}
2
A '' '
' − jω
+ 'H (e− jω0 )' e− j(ω0 n+ ϕ)+ j arg{ H (e 0 )} .
2
For a real-valued impulse response holds
H (e jω ) = H ∗ (e− jω )
and
∞ ∞
H (e jω ) = ∑ h(n) cos(ωn) + j ∑ h(n) sin(ωn)
n=−∞ n=−∞
' '2 ' '2
' ' ' '
'H (e jω )' = 'H (e− jω )'
! ∞ 6
∑n=−∞ h(n) sin(ωn)
jω
arg{ H (e ) = arctan = − arg{ H (e− jω ).
∑∞ n=−∞ h (n ) cos( ωn )
The output for a real-valued impulse response is

' '
' '
y(n) = A 'H (e jω0 )' cos(ω0 n + ϕ + arg{ H (e jω0 )).
Example 2.12. Find the impulse response of an ideal discrete-time differentiator

H (e jω ) = jω for − π ≤ ω < π.
⋆The impulse response is

"π "π "π
1 j 1
h(n) = jωe jωn dω = ω cos(ωn)dω − ω sin(ωn)dω
2π 2π 2π
−π −π −π
"π ' "π
−1 1 cos(ωn) ''π 1
= ω sin(ωn)dω = ω ' + cos(ωn)dω.
π π n 0 πn
0 0
Since the last integral is equal to 0 the final result is

cos(πn) (−1)n
h(n) = =
n n
for n ̸= 0 and h(n) = 0 for n = 0. Using samples n = ±1, ±2, ...,± N the
approximation of the frequency response is
N N
sin(ωn)
HN (e jω ) = ∑ h(n)e− jωn = 2j ∑ (−1)n−1 .
n=− N n =1
n
Note that this system is not causal.
The Fourier transform of a product of discrete-time signals is equal to

the convolution of the Fourier transforms in frequency,
"π
1
FT{ x (n)h(n)} = X (e jθ ) H (e j(ω −θ ) )dθ = X (e jω ) ∗ω H (e jω ). (2.34)
2π
−π
This convolution is periodic with period 2π (circular convolution).

2.2.2 Spectral Energy and Power Density
Parseval’s theorem for discrete-time signals reads
∞ ∞ "π
1
∑ ∑ 2π X (e jω )e jωn y∗ (n)dω
x (n)y∗ (n) = (2.35)
n=−∞ n=−∞ −π
"π
( ) "π
∞
1 1
=
2π
jω
X (e ) ∑ (e y(n)) dω = 2π X (e jω )Y∗ (e jω )dω.
− jωn ∗
−π n=−∞ −π
For a signal x (n) Parseval’s theorem is
∞ "π ' '2

1 2 ' '
∑ |x(n)| = 2π 'X (e jω )' dω = Ex .
n=−∞ −π
' '2
Function ' X (e jω )' is the spectral energy density of signal x (n).
Since the average power of a signal x (n) is defined by
N
1
PAV = lim ∑ | x (n)|2 , (2.36)
N →∞ 2N + 1 n=− N
its power spectral density may be defined as
1 ' '2
' '
Pxx (e jω ) = lim 'X N (e jω )' , (2.37)
N →∞ 2N + 1
where the Fourier transform of x (n) within − N ≤ n ≤ N is denoted by

X N (e jω ). The power spectral density can be written as
N N
1
Pxx (e jω ) = lim ∑ ∑ x (n) x ∗ (m)e− jω (n−m) . (2.38)
N →∞ 2N + 1 n=− N m=− N
For a very specific signal x (n) = Ae j(ω0 n+ ϕ) which satisfies r (k ) =

x (n) x ∗ (n − k ) = A2 e jω0 k the power spectral density is
2N
1
Pxx (e jω ) = lim ∑ (2N + 1 − |k |)r (k )e− jωk
N →∞ 2N + 1
k=−2N
since the value r (0), for n − m = k = 0, appears 2N + 1 times along the

diagonal in n, m domain in (2.38). The value for n − m = k = ±1 appears 2N
times, and so on. The value r (k ), for n − m = k, appears 2N + 1 − |k | times

in double summation (2.38). Note that Pxx (e jω ) in this case is the Fourier
transform of r (k ) multiplied by a Bartlett window 1 − |k |/(2N + 1).
2.3 SAMPLING THEOREM IN THE TIME DOMAIN
Continuous-time signal x (t), whose Fourier transform X (Ω) is limited with

Ωm = 2π f m , where f m is a frequency in [Hz], while Ωm is frequency in
[rad/s], i.e.,
X (Ω) = 0 for |Ω| > Ωm , (2.39)
can be reconstructed, for any t, based on samples taken with a sampling
interval ∆t,
x (n) = x (n∆t)∆t,
such that
π 1
∆t < = .
Ωm 2 fm
Now we will prove this statement. Since we have assumed a limited
frequency duration of X (Ω) we can make its periodic extension
∞
X p (Ω) = ∑ X (Ω + 2Ω0 m) (2.40)
m=−∞
with a period 2Ω0 . It is very important to note that X p (Ω) = X (Ω) for
|Ω| < Ω0 if
Ω0 > Ω m .
In this case, it is possible to make transformation from X (Ω) to X p (Ω) and
back without losing any information.
Of course, that would not be the case if Ω0 > Ωm did not hold. By
periodic extension of X (Ω), in that case, overlapping (aliasing) would have
occurred in X p (Ω). It would not be reversible. Then it would not be possible
to recover X (Ω) from X p (Ω). The periodic extension is illustrated in Fig. 2.8.
The periodic function X p (Ω) can be expanded into Fourier series with
coefficients
"Ω0 "∞
1 jπΩn/Ω0 1
X− n = X p (Ω)e dΩ = X (Ω)e jπΩn/Ω0 dΩ.
2Ω0 2Ω0
− Ω0 −∞
The integration limits are extended to the infinity since X (Ω) = X p (Ω)
within the basic period interval and X (Ω) = 0 outside this interval.
X(Ω)
-Ω Ω Ω
m m
X (Ω) = X(Ω)
p
-Ω <Ω<Ω
0 0
- Ω0 - Ωm Ωm Ω0 Ω
Figure 2.8 The Fourier transform of a signal, with X (Ω) = 0 for |Ω| > Ωm (top) and its
periodically extended version, with period 2Ω0 > 2Ωm (bottom).
The inverse Fourier transform of continuous-time signal is
"∞
1
x (t) = X (Ω)e jΩt dΩ. (2.41)
2π
−∞
By comparing the last two equations, we easily conclude that

π
X− n = x (t)|t=πn/Ω0 = x (n∆t)∆t (2.42)
Ω0
π
∆t = ,
Ω0
meaning that the Fourier series coefficients of the periodically extended

Fourier transform of X (Ω) are the samples of the signal, taken with the
sampling interval ∆t = π/Ω0 .
Therefore, the samples of a signal and the periodically extended
Fourier transform are the Fourier series pair
∞
X−n = x (n∆t)∆t ←→ X p (Ω) = ∑ X (Ω + 2Ω0 m) (2.43)
m=−∞
with ∆t = π/Ω0 .
The reconstruction formula for x (t) then follows from
"∞ "Ω0
1 jΩt 1
x (t) = X (Ω)e dΩ = X p (Ω)e jΩt dΩ (2.44)
2π 2π
−∞ − Ω0
"Ω0
( )
∞
1
= ∑ Xn e jπnΩ/Ω0 e jΩt dΩ
2π n=−∞
− Ω0
"Ω0
( )
∞
1 jπnΩ/Ω0
= ∑ x (−n∆t)∆te e jΩt dΩ
2π n=−∞
− Ω0
as
∞ π
sin( ∆t (t − n∆t))
x (t) = ∑ x (n∆t) π . (2.45)
n=−∞ ∆t (t − n∆t )
The signal x (t), for any t, is expressed in terms of its samples x (n∆t).
Example 2.13. The last relation can be used to prove that X (Ω) = X (e jω ) with
Ω∆t = ω and |ω | < π for the signals sampled at the rate satisfying the
sampling theorem.
⋆Starting from
"∞
X (Ω) = x (t)e− jΩt dt
−∞
the signal x (t), satisfying the sampling theorem, can by written in terms of
samples, according to the third row of (2.44), as
"Ω0
( )
∞
1 − j∆tnθ
x (t) = ∑ x (n∆t)∆te e jθt dθ.
2π n=−∞
− Ω0
It follows
"∞ "Ω0
( )
∞
1 − j∆tnθ
X (Ω) = ∑ x (n∆t)∆te e jθt dθe− jΩt dt
2π n=−∞
−∞ − Ω0
∞ "Ω0
= ∑ x (n∆t)∆t δ(θ − Ω)e− j∆tnθ dθ
n=−∞
− Ω0
∞
= ∑ x (n∆t)∆te− j∆tnΩ for |Ω| < Ω0 (2.46)
n=−∞
resulting in
∞
X (Ω) = ∑ x (n)e− jωn for |ω | < π
n=−∞
with ω = Ω∆t and x (n) = x (n∆t)∆t.
Example 2.14. If the highest frequency in a signal x (t) is Ωm1 and the highest
frequency in a signal y(t) is Ωm2 what should be the sampling interval for the
signal x (t)y(t) and for the signal x (t − t1 )y∗ (t − t2 )? The highest frequency
Ωm in a signal is used in the sense that the Fourier transform of the signal is
zero for |Ω| > Ωm .
⋆The Fourier transform of a product x (t)y(t) is a convolution of the

Fourier transforms X (Ω) and Y (Ω). Since these functions are of limited
duration |Ω| < Ωm1 and |Ω| < Ωm2 , respectively, in general their convolution
is limited to the interval |Ω| < Ωm1 + Ωm2 . Therefore, the sampling interval
for y(t) should be
π
∆t < .
Ωm1 + Ωm2
Shifts in time and complex conjugate operation do not change the Fourier
transform width. Therefore the conclusion remains the same for x (t −
t1 ) y ∗ ( t − t2 ).
Example 2.15. If the signal

x (t) = e−|t|
is sampled with ∆t = 0.1, write the Fourier transform of the obtained discrete-
time signal: (a) by a periodical extension of the continuous-time Fourier
transform and (b) by a direct calculation based on the discrete-time signal.
Comment on the expected error due to the discretization.
⋆The Fourier transform of this signal is
"∞ "0 "∞

X (Ω) = x (t)e− jΩt dt = et e− jΩt dt + e−t e− jΩt dt
−∞ −∞ 0
1 1 2
= + = . (2.47)
1 − jΩ 1 + jΩ 1 + Ω2
After sampling with ∆t = 0.1, the Fourier transform is periodically extended

with period 2Ω0 = 2π/∆t = 20π.
(a) The periodic Fourier transform is
2 2 2
X p (Ω) = ... + + + + ...
1 + (Ω + 20π )2 1 + Ω2 1 + (Ω − 20π )2
Thus, the value of X p (Ω) at the period ending points ±10π will approxi-
mately be X p (±10π ) = 2/(1 + 100π 2 ) ∼ = 0.002. Comparing with the maxi-
mum value X p (0) = 2, it means that the expected error due to the discretiza-
tion of this signal (since it does not strictly satisfy the sampling theorem) will
be of a 0.1% order.
(b) The discrete-time signal obtained by sampling x (t) = exp(− |t|)
with ∆t = 0.1 is x (n) = 0.1e−0.1|n| . Its Fourier transform is already calculated
with A = 0.1 and α = 0.1, eq.(2.22). The result is
1 − e−0.2
X (e jω ) = 0.1 . (2.48)
1 − 2e−0.1 cos(ω ) + e−0.2
Therefore, the exact value of an infinite sum in X p (Ω) is X (e jω ) with ω =

Ω∆t = 0.1Ω
∞
2 1 − e−0.2
X p (Ω) = ∑ 1 + ( Ω + 20kπ ) 2
= 0.1
1 − 2e − 0.1 cos(0.1Ω) + e−0.2
.
k=−∞
Note that in this way we solve an interesting mathematical problem of

finding a sum of an infinite series.
For Ω = 0, the original value of the Fourier transform is X (0) = 2.
In the signal that could be reconstructed based on the discretized signal
X p (0) = 0.1(1 + e−0.1 )/(1 − e−0.1 ) = 2.00167. The increase of 0.00167 is due
to periods overlapping. It manifests as an aliasing error in X (0). The value of
error corresponds to our previous conclusion of about a 0.1% error order.
Example 2.16. A continuous-time signal x (t) = cos(25πt + π/4) + sin(50πt −

π/3) is sampled with a step ∆t = 1/100 and a discrete-time signal x (n) =
x (n∆t)∆t is formed. The discrete-time signal is processed using the system
whose impulse response is
1 1 1
h(n) = δ ( n ) + δ ( n − 2) + δ ( n + 2)
2 4 4
Find the output signal y(n) and the corresponding continuous-time signal
y a (t) as the result of convolution.
⋆The discrete-time input signal is
x (n) = [cos(nπ/4 + π/4) + sin(nπ/2 − π/3)]∆t
with the Fourier transform

∞
π π π
X (e jω ) = ∑ [δ(ω + + 2kπ )e− jπ/4 + δ(ω − + 2kπ )e jπ/4 ]
100 k=−∞ 4 4
∞
π π π
+ ∑ [δ(ω + + 2kπ )e− jπ/6 + δ(ω − + 2kπ )e jπ/6 ].
100 k=−∞ 2 2
The frequency response of the discrete system is

∞
1
H (e jω ) = ∑ h(n)e− jωn = (1 + cos(2ω )).
n=−∞ 2
The frequency response values at the frequencies of X (e jω ), within the basic

period −π ≤ ω < π, are H (e± jπ/4 ) = 1/2 and H (e± jπ/2 ) = 0. Therefore the
output signal Fourier transforms is
π π π
Y (e jω ) = H (e jω ) X (e jω ) = [δ(ω + )e− jπ/4 + δ(ω − )e jπ/4 ]
200 4 4
for − π ≤ ω < π.
The output discrete-time signal is
1
y(n) = cos(nπ/4 + π/4)∆t
2
corresponding to the continuous-time signal
1 π 1
y(t) = cos(n + π/4) = cos(25πt + π/4).
2 4∆t 2
Find the output signal for h(n) = ∑2i=−2 δ(n − i ).
2.4 PROBLEMS
Problem 2.1. Check the periodicity and find the period of signals:
(a) x (n) = sin(2πn/32), (b) x (n) = cos(9πn/82), (c) x (n) = e jn/32 , and (d)
x (n) = sin(πn/5) + cos(5πn/6) − sin(πn/4).
Problem 2.2. Check the linearity and time-invariance of the discrete system
described by equation
y(n) = x (n) + 2.
Problem 2.3. The output of a linear time-invariant discrete system to the

input signal x (n) = u(n) is y(n) = 2−n u(n). Find the impulse response h(n).
Is the system stable?
Problem 2.4. Find the convolution
y(n) = x (n) ∗ x (n)
for x (n) = u(n) − u(n − 5).

h(n) = h1(n)*h2(n)*h2(n)
12 3
10 2
8
1
x(n)
6
4 0
2 -1
0
-2
-2 0 2 4 6 8 -2 0 2 4 6 8
n n
Figure 2.9 Problem 2.7, impulse response h(n) (left) and Problem 2.14, discrete signal x (n)
(right).
Problem 2.5. Find the convolution of signals x (n) = e−|n| and h(n) = u(n +
5) − u ( n − 6).
Problem 2.6. A discrete system consists of systems with impulse responses
h1 (n) = e− an u(n), h2 (n) = e−bn u(n), and h3 (n) = u(n). Find the impulse
response of the resulting system for:
(a) Systems h1 (n), h2 (n), and h3 (n) connected in parallel,
(b) System h1 (n) connected in parallel with a cascade of systems h2 (n)
and h3 (n).
Problem 2.7. Consider three causal linear time-invariant systems in cas-
cade. Impulse responses of these systems are h1 (n), h2 (n), and h2 (n), re-
spectively. The impulse response of the second and the third system is
h2 (n) = u(n) − u(n − 2), while the impulse response of the whole system,
h ( n ) = h 1 ( n ) ∗ n h 2 ( n ) ∗ n h2 ( n ),
is shown in Fig. 2.9 (left).

Find h1 (n) and y(n) = h(n) ∗n x (n), with x (n) = δ(n) − δ(n − 1).
Problem 2.8. Find the output of discrete system whose impulse response is
h(n) = ne−n/2 u(n)
to the input signal
x (n) = 5 sin(πn/5) − 3 cos(πn/3 + π/6).
Find
∞
S= ∑ ne−n/2 .
n =0
Problem 2.9. Find the Fourier transform of:

(a) x (n) = u(n),
(b) x (n) = 2 cos(ω0 n)u(n), and
(c) y(n) = ∑∞ k=−∞ x ( n + kN ) if the Fourier transform of discrete-time
signal x (n) is X (e jω ).
Problem 2.10. In implementing an approximation of a signal derivative we
may use a system with impulse response
h(n) = a[δ(n + 1) − δ(n − 1)] + b[δ(n + 2) − δ(n − 2)].
Find the constants a and b such that
H (e jω ) ∼
= jω for small ω,
i.e., ' '
dH (e jω ) '' d2 H (e jω ) ''
= j and = 0.
dω 'ω =0 dω 2 'ω =0
Problem 2.11. Find the Fourier transform of the following discrete-time
signal (triangular window)
* +
|n|
wT (n) = 1 − [u(n + N ) − u(n − N − 1)].
N+1
with N being an even number.
Problem 2.12. Find the value of integral
"π
1 sin2 (( N + 1)ω/2)
I= dω.
2π sin2 (ω/2)
−π
Problem 2.13. A window is formed as
w(n) = w H (n + N ) + w H (n) + w H (n − N )
where w H (n) is the Hann(ing) window

1
2
Plot the window w(n) and express its Fourier transform as a function of the
Fourier transform of the Hann(ing) window WH (e jω ). Generalize the results
for
K
w(n) = ∑ w H (n + kN ).
k=−K
Problem 2.14. A discrete-time signal x (n) is given in Fig. 2.9 (right). Without
calculating its Fourier transform X (e jω ) find
"π "π ' '2

j0 jπ jω ' '
X ( e ), X ( e ), X (e )dω, 'X (e jω )' dω,
−π −π
and a signal whose Fourier transform is the real part of X (e jω ), denoted by

Re{ X (e jω )}.
Problem 2.15. Find the Fourier transform of discrete-time signal
y(n) = ne−n/4 u(n).
Using this Fourier transform find the center of gravity of signal x (n) =
e−n/4 u(n) defined by
∞
∑ nx (n)
n=−∞
ng = ∞ .
∑ x (n)
n=−∞
Problem 2.16. Impulse response of a discrete system is given as:
sin(nπ/3)
(a) h(n) = , with h(0) = 1/3,
nπ
sin2 (nπ/3)
(b) h(n) = ,
(nπ )2
sin((n − 2)π/4)
(c) h(n) = .
( n − 2) π
Show that the frequency response of the system with h(n) = sin(nπ/3)/nπ
is H (e jω ) = 1 for |ω | ≤ π/3 and H (e jω ) = 0 for π/3 < |ω | < π. Find the
frequency responses in other two cases. Find the systems output to the input
signal x (n) = sin(nπ/6).
Problem 2.17. A continuous-time signal x (t) = cos(20πt + π/4) + sin(90πt)

is sampled with a step ∆t and a discrete-time signal x (n) = x (n∆t)∆t is
formed. The signal is convolved with h(n) = sin(nπ/2)/(nπ ). (a) What is
the result of this convolution for ∆t = 1/100? (b) If the signal is sampled
with ∆t = 1/50 what is the output signal? (c) Find the result of convolution
for ∆t = 3/100.
Problem 2.18. Analytic part x a (n) of a discrete-time signal x (n) is defined

in the frequency domain by
⎧
⎨ 2X (e jω ) for 0<ω<π
jω
Xa (e ) = X (e jω ) for ω=0 .
⎩
0 for −π ≤ ω < 0
In the time domain the analytic part can be written as
x a (n) = x (n) + jxh (n),
where xh (n) is the Hilbert transform of x (n). Find the impulse response of
the system that transforms a signal x (n) into its Hilbert transform (Hilbert
transformer).
Problem 2.19. The Fourier transform of a continuous signal x (t) is nonzero

only within 3Ω1 < Ω < 5Ω1 . Find the maximum possible sampling interval
∆t such that the signal can be reconstructed based on the samples x (n∆t).
Problem 2.20. For a signal whose Fourier transform is zero for frequencies
Ω ≥ Ωm = 2π f m = π/∆t show that
"∞
sin(π (t − τ )/∆t)
x (t) = x (τ ) dτ.
π (t − τ )
−∞
Write a discrete-time version of this relation.
Problem 2.21. Sampling of a signal is done twice, with the sampling interval
∆t = 2π/Ωm that is twice larger than the sampling interval required by the
sampling theorem (∆t = π/Ωm is required). After first sampling process,
the discrete-time signal x1 (n) = ∆tx (n∆t) is formed, while after the second
sampling process signal x2 (n) = ∆tx (n∆t + a) is formed. Show that we can
reconstruct continuous-time signal x (t) based on x1 (n) and x2 (n) if a ̸= k∆t,
that is, if samples x1 (n) and x2 (n) do not overlap in continuous-time.
Problem 2.22. In general, a sinusoidal signal x (t) = A sin(Ω0 t + ϕ) is de-

scribed with three parameters A, Ω0 and ϕ. Thus, generally speaking, three
points of x (t) would be sufficient to find three signal parameters. If we know
the signal x (t) at t = t0 , t = t0 + ∆t and t = t0 − ∆t what is the relation
and conditions to reconstruct, for example, Ω0 , which is usually the most
important parameter of a sinusoid?
Problem 2.23. Show that the relation among the amplitudes of a signal
x (n) and its even and odd parts xe (n) = [ x (n) + x (−n)]/2 and xo (n) =
[ x (n) − x (−n)]/2 is
√
As (n) ≤ | xe (n)| + | xo (n)| ≤ 2As (n)
@ A
with As (n) > 0 defined by A2s (n) = | x (n)|2 + | x (−n)|2 /2.
2.5 SOLUTIONS
Solution 2.1. (a) Calculate x (n + N ) = sin(2π (n + N )/32). For 2πN/32 =

2kπ, k = 1, 2, ..., x (n + N ) = x (n) holds. The smallest integer N satisfying the
previous condition is N = 32 with k = 1. The period of signal is N = 32.
(b) For this signal x (n + N ) = cos(9πn/82 + 9πN/82) = x (n) for
9πN/82 = 2kπ, k = 1, 2, .... The period follows from N = 164k/9. The period
of this signal is N = 164 for k = 9.
(c) In this case x (n + N ) = e j(n/32+ N/32) . The relation N/32 = 2kπ,
k = 1, 2, ..., produces N = 64kπ. This is not an integer for any k, meaning
that the signal is not periodic.
(d) The periods of signal components are obtained from N1 = 10k,
N2 = 12k/5, and N3 = 8k. The smallest value of N when N1 = N2 = N3 = N
is N = 120 containing 12 periods of sin (πn/5), 50 periods of cos(5πn/6),
and 15 periods of sin(πn/4).
Solution 2.2. For linearity we have to check the system output to the linear
combination of input signals x1 (n) and x2 (n),
T { a1 x1 (n) + a2 x2 (n)} = a1 x1 (n) + a2 x2 (n) + 2.
This not equal to
a1 y1 (n) + a2 y2 (n) = a1 x (n) + 2a1 + a2 x2 (n) + 2a2 .
System is not linear.

This system is time-invariant since
T { x ( n − N ) = x ( n − N ) + 2 = y ( n − N ).
Solution 2.3. The impulse response is defined by
h(n) = T {δ(n)}.
It can be written as
h(n) = T {u(n) − u(n − 1)}.
For a linear time-invariant discrete system holds
h(n) = T {u(n)} − T {u(n − 1)}.
In this case it means
h(n) = T { x (n)} − T { x (n − 1)}

= y(n) − y(n − 1) = 2−n u(n) − 2−(n−1) u(n − 1)
= δ(n) + 2−n u(n − 1) − 2−(n−1) u(n − 1)
= δ ( n ) + 2− n (1 − 2 ) u ( n − 1 ) = δ ( n ) − 2− n u ( n − 1 ).
For this system
∞ ∞
2−1
∑ |h(n)| = 1 + ∑ 2− n = 1 + = 2.
n=−∞ n =1 1 − 2−1
The system is stable since the sum of absolute values of impulse response is
finite.
Solution 2.4. The convolution is calculated sample by sample as
∞
y (0) = ∑ x (k ) x (−k ) = x (0) x (0) = 1
k=−∞
∞
y (1) = ∑ x ( k ) x (1 − k ) = x (0 ) x (1 ) + x (1 ) x (0 ) = 2
k=−∞
∞
y(−1) = ∑ x (k ) x (−1 − k ) = 0
k=−∞
∞
y (2) = ∑ x ( k ) x (2 − k ) = 3
k=−∞
...
The calculation, along with the final result y(n), is presented in

Fig.2.10.
1.5 1.5
1 1
x(-k )
x(k)
0.5 0.5
0 0
-0.5 -0.5
-15 -10 -5 0 5 10 15 -15 -10 -5 0 5 10 15
k k
1.5 1.5
1 1
x(1-k )
x(2-k )
0.5 0.5
0 0
-0.5 -0.5
-15 -10 -5 0 5 10 15 -15 -10 -5 0 5 10 15
k k
1.5 6
x(n)* x(n)
1
x(-1-k )
4
0.5
2
0
-0.5 0
-15 -10 -5 0 5 10 15 -15 -10 -5 0 5 10 15
k n
Figure 2.10 Illustration of a discrete-time signal convolution calculation.
Solution 2.5. Based on the convolution definition

∞
y(n) = x (n) ∗ h(n) = ∑ x (k )h(n − k ) = (2.49)
k =−∞
∞
= ∑ e−|k| (u((n − k ) + 5) − u((n − k ) − 6))
k=−∞
with !
1, for k ≤ n + 5
u((n − k ) + 5) =
0, for k > n + 5
and !
1, for k ≤ n − 6
u((n − k ) − 6) =
0, for k > n − 6
we get
!
1, for n − 6 < k ≤ n + 5
(u((n − k) + 5) − u((n − k) − 6)) =
0, elsewhere.
The infinite sum in (2.49) reduces to the terms for n − 5 ≤ k ≤ n + 5

n +5
y(n) = ∑ e−|k| .
k = n −5
Since !
k, for k ≥ 0
|k | = ,
−k, for k < 0
we have three cases:
1) For n + 5 ≤ 0, i.e., n ≤ −5, we have k ≤ 0 for all terms. Therefore |k | = −k,
n +5
1 − e11 e −5 − e 6
y(n) = ∑ e k = e n −5 = en
k = n −5
1−e 1−e
e0.5 e−5.5 − e5.5 sinh 5.5
= en 0.5 −0.5 = en
e e − e0.5 sinh 0.5
2) For n − 5 ≥ 0, the lowest k = n − 5 is greater than 0. Then k ≥ 0 for all
terms and |k | = k with
n +5
1 − e−11 5
−n e − e
−6
y(n) = ∑ e − k = e − n +5 = e
k = n −5
1 − e −1 1 − e −1
e−0.5 e5.5 − e−5.5 sinh 5.5
= e−n −0.5 0.5 = e−n .
e e − e−0.5 sinh 0.5
3) For −5 < n < 5, index k can assume positive and negative values. The
convolution is split into two sums as
n +5 −1 n +5 5− n n +5
y(n) = ∑ e−|k| = ∑ ek + ∑ e−k = ∑ e−k + ∑ e−k
k = n −5 k = n −5 k =0 k =1 k =0
1−e −( 5 − n ) 1 − e n +6)
−(
= e −1 + =
1 − e −1 1 − e −1
1 − e n −5 1/2 1 − e
−(n+6)
= e−1/2 1/2 + e
e − e−1/2 e1/2 − e−1/2
1
= 0.5 (e−0.5 − en−5.5 + e0.5 − e−n−5.5 ) =
e − e−0.5
−e−5.5 (en + e−n ) + e−0.5 + e0.5
=
e0.5 − e−0.5
cosh 0.5 − e−5.5 cosh(n)
= .
sinh 0.5
Finally we can write

⎧
⎪
⎨ e−|n| sinh 5.5
sinh 0.5 for |n| ≥ 5
y(n) =
⎪
⎩ cosh 0.5−e−5.5 cosh(n)
sinh 0.5 for |n| < 5.
Solution 2.6. (a) For a parallel connection of systems
y ( n ) = y1 ( n ) + y2 ( n ) + y3 ( n )
∞ ∞ ∞
= ∑ h1 ( k ) x ( n − k ) + ∑ h2 ( k ) x ( n − k ) + ∑ h3 ( k ) x ( n − k )
k =−∞ k=−∞ k=−∞
∞
= ∑ [h1 (k) + h2 (k) + h3 (k )] x (n − k).
k =−∞
The resulting impulse response is

h ( n ) = h1 ( k ) + h2 ( k ) + h3 ( k )
= [e−an + e−bn + 1]u(n).
(b) For a cascade of systems with h2 (n) and h3 (n) holds
∞ ∞
y2 ( n ) = ∑ h2 ( k ) x ( n − k ) = ∑ h2 ( n − k ) x ( k )
k=−∞ k =−∞
∞ ∞ ∞
y3 ( n ) = ∑ h3 ( m ) y2 ( n − m ) = ∑ h3 ( m ) ∑ h2 ( n − m − k ) x ( k )
m=−∞ m=−∞ k =−∞
∞ ∞ ∞
= ∑ ∑ h3 ( m ) h2 ( n − m − k ) x ( k ) = ∑ h23 (n − k ) x (k )
k=−∞ m=−∞ k=−∞
where
∞
h23 (n) = ∑ h 3 ( m ) h2 ( n − m ) = h 2 ( n ) ∗ h 3 ( n ).
m=−∞
The impulse response of the whole system is
h(n) = h1 (n) + h23 (n) = h1 (n) + h2 (n) ∗ h3 (n),
with
∞
h2 ( n ) ∗ h3 ( n ) = ∑ e−b(n−m) u (n − m )u (m )
m=−∞
n
1 − e b ( n +1) e−bn − eb
= u(n) ∑ e−b(n−m) = e−bn 1 − eb
u ( n ) =
1 − eb
u ( n ).
m =0
Solution 2.7. Since we know h2 (n), we can calculate

h2 (n) ∗n h2 (n) = δ(n) + 2δ(n − 1) + δ(n − 2).
Therefore, the total impulse response
h(n) = h1 (n) ∗n [h2 (n) ∗n h2 (n)]
= h1 (n) + 2h1 (n − 1) + h1 (n − 2)
h1 (n) = h(n) − 2h1 (n − 1) − h1 (n − 2).
From the last relation it follows h1 (n) = 0 for n < 0, h1 (0) = h(0) = 1,
h1 (1) = h(1) − 2h1 (0) = 3, h1 (2) = h(2) − 2h1 (1) − h1 (0) = 3, h1 (3) = 2,
h1 (4) = 1, h1 (5) = 0, and h1 (n) = 0 for n > 5.
Output to x (n) = δ(n) − δ(n − 1) can be easily calculated as
y ( n ) = h ( n ) − h ( n − 1 ).
Solution 2.8. Instead of a direct convolution we will calculate the frequency
response of discrete system as
∞
H (e jω ) = ∑ h(n)e− jωn .
n=−∞
Find first the transform of e−n/2 u(n),

∞
1
H1 (e jω ) = ∑ e−n/2 e− jωn = 1 − e−(1/2+ jω)
n =0
and differentiate both sides with respect to ω

∞
− je−(1/2+ jω )
−j ∑ ne−n/2 e− jωn = (1 − e−(1/2+ jω) )2 .
n =0
It follows
∞
e−(1/2+ jω )
H (e jω ) = ∑ ne−n/2 e− jωn = .
n =0 (1 − e−(1/2+ jω ) )2
The output for a real-valued h(n) is
' '
' '
y(n) = 5 'H (e jπ/10 )' sin(πn/5 + arg{ H (e jπ/10 })
' ' ' '
' ' ' '
− 3 'H (e jπ/6 )' cos(πn/3 + π/6 + 'H (e jπ/6 )')
=14.1587 sin(πn/5 − 1.1481)
− 5.7339 cos(πn/3 + π/6 − 1.6605).
Value of the sum S is

∞ √
−n/2 j0 e
S= ∑ ne = H (e ) = √
( e − 1)2
.
n =0
Solution 2.9. (a) The unit step signal can be written as

- .
1 −an 1 1 − an
x (n) = u(n) = lim e u(n) + − e u(−n − 1) = lim x a (n).
a →0 2 2 2 a →0
The Fourier transform of x a (n) is

∞ - .
1 − an 1 1 −an
jω
Xa (e ) = ∑ e u(n) + − e u(−n − 1) e− jωn
n=−∞ 2 2 2
1 ∞ 1 a+ jω
2 2e
=
1 − e−a− jω
+ ∑ πδ(ω + 2kπ ) −
1 − e a+ jω
k=−∞
∞
1
X (e jω ) = lim Xa (e jω ) = + ∑ πδ(ω + 2kπ ).
a →0 1 − e− jω k=−∞
The result from (2.23) is used to transform the constant signal equal to 1/2.
(b) This signal is
x (n) = 2 cos(ω0 n)u(n) = (e jω0 n + e− jω0 n )u(n).

Its Fourier transform is
∞
1
X (e jω ) = + ∑ πδ(ω − ω0 + 2kπ )
1 − e − j ( ω − ω0 ) k=−∞
∞
1
+ + ∑ πδ(ω + ω0 + 2kπ )
1 − e− j(ω +ω0 ) k=−∞
1 − e− jω cos(ω0 )
=2
1 − 2 cos(ω0 )e− jω + e− j2ω
∞
+ ∑ π [δ(ω − ω0 + 2kπ ) + δ(ω + ω0 + 2kπ )] .
k =−∞
(c) For a periodic signal y(n) the Fourier transform is

∞ ∞ ∞
Y (e jω ) = ∑ ∑ x (n + kN )e− jωn = ∑ X (e jω )e jωkN
k=−∞ n=−∞ k=−∞
∞
jω
= X (e ) ∑ e jωkN .
k=−∞
Using (2.23) we get

∞
2π ∞ 2kπ
Y (e jω ) = X (e jω )2π ∑ δ(ωN + 2kπ ) = X (e jω ) ∑ δ ( ω + N ).
k=−∞
N k=− ∞
Solution 2.10. For the impulse response h(n) the frequency response is
H (e jω ) = 2aj sin(ω ) + 2jb sin(2ω ).
The first derivative of H (e jω ) at ω = 0 is

'
dH (e jω ) ''
= 2aj + 4jb = j,
dω 'ω =0
while the second derivative at ω = 0 is

'
d2 H (e jω ) ''
= −2aj − 8jb = 0.
dω 2 'ω =0
The constants a and b follow from the system
a + 2b = 1/2
a + 4b = 0
as b = −1/4 and a = 1 with the impulse response
1
h(n) = δ(n + 1) − δ(n − 1) − (δ(n + 2) − δ(n − 2)).
4
Solution 2.11. Note that
1
wT (n) = w R (n) ∗n w R (n)
N+1
where w R (n) = u(n + N/2) − u(n − N/2 − 1) is the rectangular window.

Since
sin(ω N2+1 )
WR (e jω ) = ,
sin(ω/2)
we have
1 1 sin2 (ω N2+1 )
WT (e jω ) = WR (e jω )WR (e jω ) = .
N+1 N + 1 sin2 (ω/2)
Solution 2.12. The integral represents the energy of a discrete-time signal

with Fourier transform
sin(ω N2+1 )
X (e jω ) = .
sin(ω/2)
This signal is the rectangular window, x (n) = u(n + N/2) − u(n − N/2 − 1).
Its energy is
"π N/2 N/2

1 sin2 (( N + 1)ω/2) 2
I=
2π sin2 (ω/2)
dω = ∑ x ( n ) = ∑ 1 = N + 1.
−π n=− N/2 n=− N/2
This integral is also equal to wT (0) multiplied by N + 1.

Solution 2.13. The Hann(ing) window
1
2
is of duration − N ≤ n ≤ N − 1. Thus the windows w H (n) and w H (n − N )

overlap within 0 ≤ n ≤ N − 1. Within this interval the new window is
w(n) = w H (n) + w H (n − N )
1 1
= [1 + cos(nπ/N )] + [1 + cos((n − N )π/N )]
2 2
1 1
= 1 + cos(nπ/N ) + cos(nπ/N − π ) = 1.
2 2
The same holds for − N ≤ n ≤ −1 when
w(n) = w H (n + N ) + w H (n) = 1.
The resulting window is

⎧
⎪
⎪ 0 for n < −2N
⎪
⎪
⎨ 12 [1 − cos(nπ/N )] for −2N + 1 ≤ n ≤ − N + 1
w(n) = 1 for −N ≤ n ≤ N − 1
⎪
⎪ 1
⎪
⎪ [1 − cos(nπ/N )] for N ≤ n ≤ 2N − 1
⎩ 2
0 for n > 2N − 1
since 12 [1 + cos((n ± N )π/N ) = 12 [1 − cos(nπ/N )] . The Fourier transform

of the resulting window, in terms of the Fourier transform of the Hann(ing)
window WH (e jω ), is
W (e jω ) = WH (e jω )e− jωN + WH (e jω ) + WH (e jω )e jωN

= WH (e jω )[1 + 2 cos(ωN )].
For
K
w(n) = ∑ w H (n + kN )
k=−K
we get
⎧
⎪
⎪ 0D E for n < −(K + 1) N
⎪
⎪ π
⎨ 12 1 + cos((n + KN ) N ) for −(K + 1) N + 1 ≤ n ≤ −KN + 1
w(n) = 1D E for −KN ≤ n ≤ KN − 1
⎪
⎪ 1 π
⎪
⎪ 2 1 + cos((n − KN ) N ) for KN ≤ n ≤ (K + 1) N − 1
⎩
0 for n > ( K + 1) N − 1
with
K
1 − e− jω (2K +1) N
W (e jω ) = WH (e jω ) ∑ e− jωkN = e jωKN
k=−K 1 − e− jωN
sin(ω (2K + 1) N/2)
= WH (e jω ) .
sin(ωN/2)
Similar results hold for the Hamming and triangular window. The results
can be generalized for shifts of N/2, N/4,...
For very large K the second term variations in W (e jω ) are much faster
than the variations of WH (e jω ). Thus, for large K the Fourier transform
W (e jω ) approaches to the Fourier transform of a rectangular window of the
width (2K + 1) N.
Solution 2.14. Based on the definition of the Fourier transform of discrete-
time signals,
∞
X (e j0 ) = ∑ x (n) = 7,
n=−∞
∞
X (e jπ ) = ∑ x (n)(−1)n = 1,
n=−∞
"π
X (e jω )dω = 2πx (0) = 4π,
−π
"π ' '2 ∞

' '
'X (e jω )' dω = 2π ∑ | x (n)|2 = 30π.
−π n=−∞
Finally, X (e jω ) = Re{ X (e jω )} + j Im{ X (e jω )} and X ∗ (e jω ) = Re{ X (e jω )} −

j Im{ X (e jω )}. Thus,
1B C
Re{ X (e jω )} = X (e jω ) + X ∗ (e jω ) .
2
The inverse Fourier transform of Re { X (e jω )} is
1
y(n) = ( x (n) + x ∗ (−n)).
2
Solution 2.15. The Fourier transform of y(n) is
> ?
∞ ∞
d
Y (e jω ) = ∑ ne−n/4 u(n)e− jωn = j ∑ e−n/4 e− jωn
n=−∞ dω n =0
d 1 e − 1/4 − jω
=j = .
dω 1 − e 1/4− jω
− (1 − e−1/4− jω )2
The center of gravity of x (n) = e−n/4 u(n) is

∞
e−1/4− jω
∑ nx (n)
n=−∞ Y (e j0 ) (1−e−1/4− jω )2 |ω =0 1
ng = ∞ = = 1
= = 3.52.
X (e j0 ) e1/4 − 1
∑ x (n) 1−e−1/4− jω |ω =0
n=−∞
Solution 2.16. (a) The inverse Fourier transform of

!
1 for |ω | ≤ π/3
H (e jω ) =
0 for π/3 < |ω | < π
is
π/3
" 'π/3
1 e jωn '' sin(πn/3)
h(n) = e jωn dω = = .
2π 2jπn '−π/3 πn
−π/3
The value of frequency response at the input signal frequency ω = ±π/6 is
H (e± jπ/6 ) = 1. The output signal is, y(n) = sin(nπ/6).
(b) The frequency response, in this case, is H (e jω ) ∗ω H (e jω ), resulting in
y(n) = 0.25 sin(nπ/6).
(c) Output signal in this case is y(n) = sin((n − 2)π/6) = sin(nπ/6 − π/3).
Solution 2.17. For the signal
x (t) = cos(20πt + π/4) + sin(90πt),
corresponding discrete-time signal is
x (n) = cos(20πn∆t + π/4)∆t + sin(90πn∆t)∆t.
(a) For ∆t = 1/100
x (n) = cos(0.2πn + π/4)/100 + sin(0.9πn)/100
π ∞
X (e jω ) = ∑ [δ(ω − 0.2π + 2kπ )e jπ/4 + δ(ω + 0.2π + 2kπ )e− jπ/4 ]
100 k=− ∞
∞
π
+ ∑
j100 k=−∞
[δ(ω − 0.9π + 2kπ ) − δ(ω + 0.9π + 2kπ )].
Since the Fourier transform of h(n) = sin(nπ/2)/(nπ ) is H (e jω ) = 1 for

|ω | ≤ π/2 and H (e jω ) = 0 for π/2 < |ω | < π, the result of a convolution
is equal to the output of system with transfer function H (e jω ) to the input
signal x (n). In this case
x (n) = cos(0.2πn + π/4)/100.
Continuous-time signal corresponding to the output discrete-time signal is

y(t) = cos(20πt + π/4), Fig.2.11(top).
(b) If the signal is sampled with ∆t = 1/50 the discrete-time signal is
x (n) = cos(0.4πn + π/4)/50 + sin(1.8πn)/50,
π ∞
X (e jω ) = ∑ [δ(ω − 0.4π + 2kπ )e jπ/4 + δ(ω + 0.4π + 2kπ )e− jπ/4 ]
50 k=− ∞
π ∞
+ ∑ [δ(ω − 1.8π + 2kπ ) − δ(ω + 1.8π + 2kπ )].
j50 k=− ∞
H(ejω), X(ejω)
1
-2 π -π - π/2 0 π/2 π 3π/2 2π ω
H(ejω), X(ejω)
1
-2 π -π - π/2 0 π/2 π 3π/2 2π ω
jω jω
H(e ), X(e )
1
-2 π -π - π/2 0 π/2 π 3π/2 2π ω
Figure 2.11 Illustration of the system output with various sampling intervals (a)-(c).
The Fourier transform components within −π ≤ ω < π are
π
X (e jω ) = [δ(ω − 0.4π )e jπ/4 + δ(ω + 0.4π )e− jπ/4 ]
50
π
+ [δ(ω − 1.8π + 2π ) − δ(ω + 1.8π − 2π )]
j50
π
= [δ(ω − 0.4π )e jπ/4 + δ(ω + 0.4π )e− jπ/4 ]
50
π
+ [δ(ω + 0.2π ) − δ(ω − 0.2π )].
j50
The result of convolution is x (n) = cos(0.4πn + π/4)/50 − sin(0.2πn)/50.

Corresponding continuous-time signal is x (t)= cos(20πt + π/4)− sin(10πt).
The component − sin(10πt) does not correspond to any frequency in the
input signal, , Fig.2.11(middle). This effect is illustrated in Fig.2.12.
(c) For ∆t = 3/100
x (n) = 3 cos(0.6πn + π/4)/100 + 3 sin(2.7πn)/100.

x(n)
Figure 2.12 Illustration of the aliasing caused frequency change, from signal sin (90πt) to
signal − sin(10πt).
The Fourier transform components within −π ≤ ω < π are

3π
X (e jω ) = [δ(ω − 0.6π )e jπ/4 + δ(ω + 0.6π )e− jπ/4 ]
100
3π
+ [δ(ω − 2.7π + 2π ) − δ(ω + 2.7π − 2π )].
j100
The result of convolution is y(n) = 0, , Fig.2.11(bottom).
Solution 2.18. The Fourier transform of an analytic part of a signal is
⎧
⎨ 2X (e jω ) for 0 < ω < π
jω
Xa (e ) = X (e jω ) for ω = 0
⎩
0 for −π ≤ ω < 0
= X (e jω ) + sign(ω )( X (e jω ) = X (e jω ) + Xh (e jω ).
The frequency response of the discrete Hilbert transformer is
⎧
⎨ 1 for 0 < ω < π
H (e jω ) = 0 for ω = 0 = sign(ω )
⎩
−1 for −π ≤ ω < 0
for −π ≤ ω < π. The impulse response is
"π
2 sin2 (πn/2)
h(n) = sign(ω )e jωn dω = .
πn
−π
H(ejω) h(n)
1 2/π
-2 π -π 0 π 2π ω
0 n
Figure 2.13 Frequency and impulse response of the discrete-time Hilbert transformer.
1.5 1.5
1 1
X (Ω)
X(Ω)
p
0.5 0.5
0 0
-3 -2 -1 0 1 2 3 4 5 6 7 -3 -2 -1 0 1 2 3 4 5 6 7
Ω/Ω1 Ω/Ω1
Figure 2.14 Problem 2.19: illustration of the Fourier transform periodic extension.
For n = 0 the impulse response is h(0) = 0, Fig.2.13.
Solution 2.19. By a direct application of the sampling theorem we could

conclude that the sampling interval should be related to the maximum
frequency 5Ω1 as ∆t = π/(5Ω1 ), corresponding to the periodical extension
of the Fourier transform X (Ω) with period 10Ω1 . However, in this case,
there is no need to use such a large period in order to achieve that two
periods do not overlap. It is sufficient to use the period equal to 2Ω1 , as
shown in Fig. 2.14. We will be able to reconstruct the signal, with some
additional processing.
It is obvious that after signal sampling with ∆t = π/Ω1 (periodic ex-
tension of Fourier transform with 2Ω1 ) the basic period −Ω1 < Ω < Ω1 will
contain the original Fourier transform shifted for 4Ω1 . The reconstructed
signal is
∞
sin(π (t − n∆t)/∆t)
x (t) = e j4Ω1 t ∑ x (n∆t) with ∆t = π/Ω1 .
n=−∞ π (t − n∆t)/∆t
Solution 2.20. For signal whose Fourier transform is zero for frequencies
Ω ≥ Ωm = 2π f m = π/∆t hods
X (Ω) = X (Ω) H (Ω)
where !
1 for |Ω| < π/∆t
H (Ω) = .
0 for |Ω| ≥ π/∆t
The impulse response of H (Ω) is
1 π/∆t
& jΩt sin(πt/∆t)
h(t) = e dΩ = .
2π −π/∆t πt
Then x (t) = x (t) ∗ h(t) produces
"∞ "∞
sin(π (t − τ )/∆t)
x (t) = x (τ )h(t − τ )dτ = x (τ ) dτ.
π (t − τ )
−∞ −∞
In order to write this relation in discrete-time form note that
X (Ω) = X p (Ω) H (Ω) (2.50)
holds if the Fourier transform of signal X (Ω) is periodically extended with

π
period 2 ∆t ≥ 2Ωm to produce
∞ * +
2π
X (Ω) ∗Ω ∑ 2πδ Ω − k = X p ( Ω ).
k=−∞
∆t
Convolution in the frequency domain corresponds to the product of signals

in the time domain
∞
x (t) ∑ δ(t + n∆t)∆t = IFT{ X p (Ω)} = x p (t). (2.51)
n=−∞
Relation (1.60)
* + % ; % ;
2π ∞ 2π ∞ ∞
∑ δ Ω − ∆t k = FT ∑ δ(t + n∆t) = FT ∑ δ(t − n∆t)
∆t k=− ∞ n=−∞ n=−∞
is used.
From (2.50) and then (2.51) follows
"∞ ∞
x (t) = x p (t) ∗t h(t) = x (τ ) ∑ δ(τ − n∆t)h(t − τ )∆tdτ
−∞ n=−∞
∞ ∞ π
sin( ∆t (t − n∆t))
= ∑ x (n∆t)h(t − n∆t)∆t = ∑ x (n∆t) π . (2.52)
n=−∞ n=−∞ ∆t ( t − n∆t )
The convergence of function sin (t)/t is very slow.

The previous derivation provides a possibility that a smooth transition
of H (Ω) is used for Ωm ≤ |Ω| ≤ Ωm + ∆Ωm . This region of smooth changes
from H (Ω) = 1 for |Ω| < Ωm to H (Ω) = 0 for |Ω| ≥ Ωm + ∆Ωm improves
the convergence of h(t), Fig.2.15. The sampling step should be (Ωm +
∆Ωm
2 ) = π/∆t so that the periodic extension of X (Ω ) H (Ω ) does not include
overlapped X (Ω) values. The impulse response h(t) can be then used in the
reconstruction formula
∞
x (t) = ∑ x (n∆t)h(t − n∆t),
n=−∞
∆Ωm
with a reduction of the sampling interval to ∆t = π/(Ωm + 2 ) with
respect to ∆t = π/Ωm .
Solution 2.21. The Fourier transforms of discrete-time signals, in continu-
ous frequency notation, are periodically extended versions of X (Ω) with the
period 2π/∆t,
∞
X1 ( Ω ) = ∑ X (Ω + 2πn/∆t),
n−−∞
∞
X2 ( Ω ) = ∑ X (Ω + 2πn/∆t)e j(Ω+2πn/∆t)a .
n−−∞
Within the basic period (considering positive frequencies 0 ≤ Ω < Ωm ), only

two periods overlap
X1 (Ω) = X (Ω) + X (Ω − 2π/∆t),

X2 (Ω) = X (Ω)e jΩa + X (Ω − 2π/∆t)e j(Ω−2π/∆t)a .
The second term X (Ω − 2π/∆t) in these relations is the overlapped period

(aliasing) that should be eliminated using these two equations. The original
H(Ω)
X(Ω)
- Ωm Ωm Ω
X (Ω) = X(Ω)
p
H(Ω) -Ω <Ω<Ω
0 0
X(Ω)
-Ω -Ω Ω Ω Ω
0 m m 0
Xp(Ω) = X(Ω)
- Ω0 < Ω < Ω0
X(Ω)
-Ω -Ω Ω Ω Ω
0 m m 0
Figure 2.15 Smoothed filter in the sampling theorem illustration (first two graphs) versus
original sampling theorem relation within filtering framework.
signal’s Fourier transform X (Ω) follows as
X1 (Ω)e− j2πa/∆t − X2 (Ω)e− jΩa

X (Ω) = for a ̸= k∆t.
e− j2πa/∆t − 1
Similarly for negative frequencies, within the basic period −Ωm < Ω < 0,
follows
X (Ω)e j2πa/∆t − X2 (Ω)e− jΩa
X (Ω) = 1 for a ̸= k∆t.
e j2πa/∆t − 1
Therefore, the signal can be reconstructed from two independent

discrete-time signals undersampled with factor of two. A similar result
could be derived for N independently sampled, N times undersampled sig-
nals.
Solution 2.22. It is easy to show that
x (t0 + ∆t) + x (t0 − ∆t)

2x (t0 )
A sin(Ω0 t0 + ϕ + Ω0 ∆t) + A sin(Ω0 t0 + ϕ − Ω0 ∆t)
=
2A sin(Ω0 t0 + ϕ)
2 sin(Ω0 t0 + ϕ) cos(Ω0 ∆t)
= = cos(Ω0 ∆t),
2 sin(Ω0 t0 + ϕ)
with * +
1 x (t0 + ∆t) + x (t0 − ∆t)
Ω0 = arccos .
∆t 2x (t0 )
The condition for a unique solution is that the argument of cosine is 0 ≤
Ω0 ∆t ≤ π, limiting the approach to small values of ∆t.
In addition, here we will discuss the discrete complex-valued signal.
For a complex sinusoid x (n) = A exp( j2πk0 n/N + φ0 ), with available two
samples x (n1 ) = A exp( jϕ(n1 )) and x (n2 ) = A exp( jϕ(n2 )), from
x ( n1 )
= exp( j2πk0 (n1 − n2 )/N )
x ( n2 )
follows
2πk0 (n1 − n2 )/N = ϕ(n1 ) − ϕ(n2 ) + 2kπ,
where k is an arbitrary integer. Then
ϕ ( n1 ) − ϕ ( n2 ) k
k0 = N+ N. (2.53)
2π (n1 − n2 ) n1 − n2
Let us analyze the ambiguous term kN/(n1 − n2 ) role in the determination

of k0 . For n1 − n2 = 1, this term is kN, meaning that any frequency k0
would be ambiguous with kN. Any value k0 + kN for k ̸= 0, in this case,
will be outside the basic period 0 ≤ k ≤ N − 1. Thus, we may find k0
in a unique way, within 0 ≤ k0 ≤ N − 1. However, for n1 − n2 = L > 1,
the terms kN/(n1 − n2 ) = kN/L produce shifts within the frequency basic
period. Then several possible solutions for the frequency k0 are obtained.
For example, for N = 16 and k0 = 5 if we use n1 = 1 and n2 = 5, a possible
solution of (2.53) is k0 = 5, but also
k0 = 5 + 16k/4,
or k0 = 9, k0 = 13, and k0 = 1, for k0 within 0 ≤ k0 ≤ 15, are possible solutions

for frequency.
Solution 2.23. For the absolute values of even and odd part it holds
' ' ' '
' x (n) + x (−n) '2 ' x (n) − x (−n) '2
| xe (n)|2 + | xo (n)|2 = '' ' +'
' '
' .
'
2 2
From this relation follows
| x (n)|2 + | x (−n)|2
| xe (n)|2 + | xo (n)|2 = = A2s (n).
2
Obviously
F | xe (n)|2 ≤ A2s (n) and | xo (n)|2 ≤ A2s (n). Replacing | xo (n)| =
A2s (n) − | xe (n)|2 into | xe (n)| + | xo (n)| we get
F
| xe (n)| + | xo (n)| = | xe (n)| + A2s (n) − | xe (n)|2 .
Now we have to check the function

F
f (χ) = χ + A2s (n) − χ2
for variable χ within 0 ≤ χ ≤ | As (n)|. Variable χ stands for |,

xe (n)|. By dif-
ferentiating f (χ) with respect to χ we get d f (χ)/dχ = 1 − χ/ A2s (n) −√ χ2 .
The stationary point is obtained from d f (χ)/dχ = 0√for χ = As (n)/ 2.
√ d f (χ)/dχ is positive for χ < As (n)/ 2 and negative for
The derivative
χ > As (n)/ 2. Thus the stationary point√is the position of the function
maximum. The maximal function value is 2As (n) since
=
As (n) A2s (n) √
| xe (n)| + | xo (n)| ≤ √ + A2s (n) − = 2As (n).
2 2
The minimal value is achieved at the interval ending points for χ = 0 or

χ = As (n), producing
√
As (n) ≤ | xe (n)| + | xo (n)| ≤ 2As (n).
2.6 EXERCISE
Exercise 2.1. Calculate the convolution of signals x (n) = n[u(n) − u(n − 3)]
and h(n) = δ(n + 1) + 2δ(n) − δ(n − 2).
Exercise 2.2. Find the convolution of signals x (n) = e−|n| and h(n) = u(3 −
n ) u (3 + n ).
Exercise 2.3. The output of a linear time-invariant discrete system to the
input signal x (n) = u(n) is y(n) = ( 31n + n)u(n). Find the impulse response
h(n). Is the system stable?
Exercise&2.4. For signal x (n&) = nu(5 − n)u(n + 5) find the values of X (e j0 ),
π π
X (e jπ ), −π X (e jω )dω, and −π | X (e jω )|2 dω without the Fourier transform
calculation. Check the results by calculating the Fourier transform.
Exercise 2.5. For a signal x (n) at an instant m a signal y(n) = x (m −
n) x ∗ (m + n) is formed. Show that the Fourier transform of y(n) is real-
valued. What is the Fourier transform of y(n) if x (n) = A exp( jan2 /4 +
j2ω0 n)? Find the Fourier transform of z(m) = x (m − n) x ∗ (m + n) for a given
n.
Note: The Fourier transform of y(n) is the Wigner distribution of x (n)
for a given m, while the Fourier transform of z(m) is the Ambiguity function
of x (n) for a given n.
Exercise 2.6. For a signal x (n) with Fourier transform X (e jω ) find the
Fourier transform of x (2n). Find the Fourier transform of y1 (2n) = x (2n)
and y1 (2n + 1) = 0. What is the Fourier transform of x (2n + 1) and what is
the Fourier transform of y2 (2n) = 0 and y2 (2n + 1) = x (2n + 1). Check the
result by showing that Y1 (e jω ) + Y2 (e jω ) = X (e jω ).
Exercise 2.7. For a real-valued signal find the relation between the Fourier
transform of signal X (e jω ) and the Hartley transform
∞
H (e jω ) = ∑ x (n)[cos(ωn) + sin(ωn)].
n=−∞
Write this relation if the signal is real-valued and even, x (n) = x (−n).
Exercise 2.8. Systems with impulse responses h1 (n), h2 (n) and h3 (n) are
connected in cascade. If the impulse responses h2 (n) = h3 (n) = u(n) −
u(n − 2) and the resulting impulse response is h(n) = δ(n) + 5δ(n − 1) +
10δ(n − 2) + 11δ(n − 3) + 8δ(n − 4) + 4δ(n − 5) + δ(n − 6). Find the impulse
response h1 (n).
Exercise 2.9. Continuous-time signal x (t) = sin(100πt) + cos(180πt) +

sin(200πt + π/4) is sampled with ∆t = 1/125 and used as an input to the
system with transfer function H (e jω ) = 1 for |ω | < 3π/4 and H (e jω ) = 0
for |ω | ≥ 3π/4. What is the discrete-time output of this system? What is
the corresponding continuous-time output signal? What should be the sam-
pling interval so that the continuous-time output signal y(t) is equal to the
input signal x (t)?
Chapter 3
Discrete Fourier Transform
ISCRETE - TIME signals can be processed on digital computers in the
D time domain. Their Fourier transform is a function of continuous

frequency. For numeric processing of discrete-time signals in the
frequency domain their Fourier transform should be discretized as well.
Discretization in the frequency domain will enable numeric processing of
discrete-time signals in both time and frequency domain.
3.1 DFT DEFINITION
The discrete Fourier transform (DFT) is defined by

N −1
DFT{ x (n)} = X (k ) = ∑ x (n)e− j2πkn/N (3.1)
n =0
for k = 0, 1, 2, ..., N − 1.
In order to establish the relation between the DFT with the Fourier
transform of discrete-time signals, consider a discrete-time signal x (n) of
limited duration. Assume that nonzero samples of x (n) are within 0 ≤ n ≤
N0 − 1. Its Fourier transform is
N0 −1
X (e jω ) = ∑ x (n)e− jωn .
n =0
The DFT values can be considered as the frequency domain samples of the
Fourier transform of discrete-time signals, taken at ∆ω = 2π/N. There are
N frequency samples within the period −π ≤ ω < π,
'
'
X (k ) = X (e j2πk/N ) = X (e jω )' . (3.2)
ω =k∆ω =2πk/N
107
108 Discrete Fourier Transform
x(n)
x(n) = x(t) Δt t = nΔt
0 N0 n
xp(n)
0 N n
Figure 3.1 Periodic extension of a discrete-time signal.
In order to examine how the Fourier Transform sampling in the fre-

quency domain influences the signal in the time domain, we will form a
periodic extension of x (n) with a period N ≥ N0 , Fig.3.1.
With N being greater or equal to the signal duration N0 , we will
be able to reconstruct the original signal x (n) from its periodic extension
x p (n). Furthermore, we will assume that the periodic signal x p (n) is formed
from the samples of periodic continuous-time signal x p (t) with a period T
(corresponding to N signal samples within the period, T = N∆t). Its Fourier
series coefficients are
"T
1
Xk = x p (t)e− j2πkt/T dt.
T
0
Assuming that the sampling theorem is satisfied, the integral can be re-
placed by a sum (in the sense of Example 2.13)
N −1
1
Xk = ∑ x (n∆t)e− j2πkn∆t/T ∆t
T n =0
with x p (t) = x (t) within 0 ≤ t < T. Using T/∆t = N, x (n∆t)∆t = x (n) and
X (k ) = TXk this sum can be written as
N −1
X (k ) = ∑ x (n)e− j2πkn/N . (3.3)
n =0
Therefore, the relation between the DFT and the Fourier series coeffi-
cients is
X (k ) = TXk . (3.4)
Sampling the Fourier transform of a discrete-time signal corresponds to
the periodical extension of the original discrete-time signal in time by the
period N. The period N in time is equal to the number of samples of the
Fourier transform within one period in frequency. We can conclude that this
periodic extension in time (discretization in frequency) will not influence
the possibility to recover the original signal if the original discrete-time
signal duration was not longer than N (the number of samples in the Fourier
transform of discrete-time signal).
The inverse DFT is obtained by multiplying both sides of the DFT
definition (3.1) by e j2πkm/N and summing over k
N −1 N −1 N −1
∑ X (k )e j2πmk/N = ∑ x (n) ∑ e j2πk(m−n)/N
k =0 n =0 k =0
with
N −1
1 − e j2π (m−n)
∑ e j2πk(m−n)/N = = Nδ(m − n),
k =0 1 − e j2π (m−n)/N
for 0 ≤ m, n ≤ N − 1. The inverse discrete Fourier transform (IDFT) of signal
x (n) is
1 N −1
X (k )e j2πnk/N .
N k∑
x (n) = (3.5)
=0
for 0 ≤ n ≤ N − 1.
The signal calculated by using the IDFT is, by definition, periodic with
the period N since
N −1
1
x (n + N ) = ∑ X (k )e j2π (n+ N )k/N = x (n).
N k =0
Therefore the DFT of a signal x (n) calculated using the signal samples
within 0 ≤ n ≤ N − 1 assumes that the signal x (n) is periodically extended
with period N as
∞
IDFT{DFT{ x (n)}} = ∑ x (n + mN )
m=−∞
∞
with ∑ x (n + mN ) = x (n) for 0 ≤ n ≤ N − 1.
m=−∞
The values of this periodical extension within the basic period are equal to
x (n). This is a circular extension of signal x (n). The following notations are
also used for this kind of the signal x (n) extension
IDFT{DFT{ x (n)}} = x (n mod N ) = x ((n)) N .
The original aperiodic signal is then
x (n) = IDFT{DFT{ x (n)}} (u(n) − u(n − N )) ,
assuming that the initial DFT was calculated for signal samples x (n) within
0 ≤ n ≤ N − 1.
In literature it is quite common to use the same notation for both x (n)
and IDFT{DFT{ x (n)}} having in mind that any DFT calculation with N
signal samples implicitly assumes a periodic extension of the original signal
x (n) with period N. Thus, we will use this kind of notation, except in the
cases when we want to emphasize a difference in the results when the
inherent periodicity in the signal (when the DFT is used) is not properly
taken into account.
Example 3.1. For the signals x (n) = 2 cos(2πn/8) for 0 ≤ n ≤ 7 and x (n) =
2 cos(2πn/16) for 0 ≤ n ≤ 7 plot the periodic signals IDFT {DFT{ x (n)}} with
N = 8 without calculating the DFTs.
Example 3.2. ⋆The periodic extensions of these signals resulting from
∞
IDFT{DFT{ x (n)}} = ∑ x (n + 8m)
m=−∞
are shown in Fig.3.2.
Example 3.3. For a signal x (n) whose values are x (0) = 1, x (1) = 1/2, x (2) = −1,
and x (3) = 1/2 find the DFT with N = 4. What is the IDFT for n = −2?
x(n) x(n)
0 N=8 n 0 N=8 n
...x(n-N)+x(n)+x(n+N)+.. ...x(n-N)+x(n)+x(n+N)+..
0 N=8 n 0 N=8 n
Figure 3.2 Signals x (n) = 2 cos(2πn/8) for 0 ≤ n ≤ 7 (left) and x (n) = 2 cos(2πn/16) for
0 ≤ n ≤ 7 (right) along with their periodic extensions IDFT {DFT{ x (n)}} with N = 8.
⋆The DFT of this signal is

3
1 1
X (k) = ∑ x(n)e− j2πnk/4 = 1 + 2 e− j2πk/4 − e− jπk + 2 e j2πk/4
n =0
= 1 + (−1)k+1 + cos(2πk/4).
The IDFT is
1 3
[1 + cos(2πk/4) + (−1)k+1 ]e j2πnk/4 ,
4 k∑
x (n) =
=0
for 0 ≤ n ≤ 3. The DFT and IDFT inherently assume the signal and its Fourier
transform periodicity. Thus the result for n = −2 is
1 3 k 1 3 k
x (−2) = ∑ X (k )e j2π (−2) 4 = ∑ X (k )e j2π (4−2) 4 = x (4 − 2) = x (2) = −1.
4 k =0 4 k =0
Example 3.4. Assume that there is a routine to calculate the DFT of x (n) for
0 ≤ n ≤ N − 1 as X (k ) = DFT{ x (n)} = R{ x (n)}. How to use it to calculate the
DFT of a signal x (n) whose values are given within − N/2 ≤ n ≤ N/2 − 1?
⋆A periodic extension of the signal x (n) is assumed when the DFT
is calculated. It means that in the DFT calculation the signal x (n), defined
within − N/2 ≤ n ≤ N/2 − 1, will be extended with the period N. Here, we
have a routine to calculate the DFT of a signal using samples within 0 ≤ n ≤

N − 1. Samples of the periodic extension of x (n) within 0 ≤ n ≤ N − 1 should
be used in the calculation. They are xs (n) = x (n) within 0 ≤ n ≤ N/2 − 1 and
xs (n) = x (n − N ) for N/2 ≤ n ≤ N − 1. Then the DFT is obtained as
DFT{ x (n)} = DFT{ x (ns )} = R{ xs (n)}

x (n) = IDFT{DFT{ xs (n)}}(u(n + N/2) − u(n − N/2)).
Here, we have used the property that for a signal y(n) periodic with a period
N holds ∑nN=−01 y(n) = ∑nM=+MN −1 y(n) for any M (Generalize the result for the
DFT calculation and inversion for a signal x (n) defined within M ≤ n ≤
M + N − 1, using the given routine R{ x (n)}).
In a matrix form, the DFT can be written as

⎡ ⎤ ⎡ ⎤⎡ ⎤
X (0) 1 1 ··· 1 x (0 )
⎢ X (1) ⎥ ⎢ 1 e − j 2π ··· e − j 2π ( N −1) ⎥⎢
x (1 ) ⎥
⎥ ⎢ ⎥⎢
N N
⎢ ⎥
⎢ .. ⎥=⎢ .. .. .. .. ⎥⎢ .. ⎥
⎣ . ⎦ ⎢
⎣ . . . .
⎥⎣
⎦ . ⎦
2π ( N −1) 2π ( N −1)( N −1)
X ( N − 1) 1 e− j N ··· e− j N x ( N − 1)
(3.6)
or
X = Wx, (3.7)
where X and x are the vectors containing the signal and its DFT values
X=[ X (0) X (1) ... X ( N − 1)] T

x=[ x (0) x (1) ... x ( N − 1)] T ,
respectively, while W is the discrete Fourier transform matrix with coeffi-

cients
⎡ ⎤
1 1 ··· 1
⎢ 1 1 N −1 ⎥
⎢ W N · · · W N ⎥
W = ⎢ .. .. .. .. ⎥, (3.8)
⎣ . . . . ⎦
( N −1) ( N −1)( N −1)
1 WN · · · WN
where
k
WN = e− j2πk/N
is used to simplify the notation, especially in graphical illustrations.
The number of additions to calculate a DFT is N − 1 for each X (k )
in (3.1). Since there are N DFT coefficients the total number of additions is
N ( N − 1). From the matrix from (3.6) we can see that the multiplications
are not needed for calculation of X (0). There is non need for multiplication
in the first term of each coefficient calculation as well. If we neglect the fact
that some other terms in matrix (3.6) may also assume values 1, −1, j, or − j
then the number of multiplications is ( N − 1)2 . The order of the number of
multiplications and the number of additions for the DFT calculation is N 2 .
The inverse DFT in a matrix form is
x = W−1 X, (3.9)
with W−1 = N1 W∗ , where ∗ denotes complex-conjugate operation. The same

calculation complexity analysis holds for the inverse DFT as for the DFT.
3.2 DFT PROPERTIES
Most of the DFT properties can be derived in the same way as in the Fourier
transform and the Fourier transform of discrete-time signals.
1. Consider a signal x (n) shifted in time x (n − n0 ). If the DFT of signal

x (n) is X (k ) = DFT{ x (n)} then X (k )e− j2πkn0 /N will represent a signal
N −1
1 2π
IDFT{ X (k )e− j2πkn0 /N } = ∑ X (k )e− j2πkn0 /N e j N kn
N k =0
N −1
1 2π
= ∑ X ( k ) e j N k ( n − n0 ) = x ( n − n 0 ). (3.10)
N k =0
Here x (n − n0 ) is the signal obtained when x (n) is periodically ex-

tended with N first and then this periodic signal is shifted for n0 .
The basic period of the original signal x (n) is now within n0 ≤ n ≤
N − n0 − 1.
This kind of shift in periodic signals, used in the above relation,
is also referred to as a circular shift. Thus, with the circular shift
DFT{ x (n − n0 )} = X (k )e− j2πkn0 /N . (3.11)
2. For a modulated signal x (n)e j2πnk0 /N we easily get

M N
DFT x (n)e j2πnk0 /N = X (k − k0 ). (3.12)
3. The DFT is real-valued if
x ∗ ( n ) = x ( N − n ).
For a real-valued DFT holds
X (k ) = X ∗ (k )
or
N −1 N −1 N −1
∑ x (n)e− j2πnk/N = ∑ x ∗ (n)e j2πnk/N = ∑ x ∗ ( N − n)e j2π ( N −n)k/N ,
n =0 n =0 n =0
where x ∗ ( N )e j2πNk/N = x ∗ (0)e j2π0k/N is used. Since e j2πn( N −n)/N =

e− j2πnk/N we get
N −1 N −1
∑ x (n)e− j2πnk/N = ∑ x ∗ ( N − n)e− j2πnk/N .
n =0 n =0
It means that if X (k ) = X ∗ (k ) then x ∗ (n) = x ( N − n) = x (−n).

In the same way for a real-valued signal x (n) the DFT satisfies
X ∗ ( k ) = X ( N − k ).
4. Parseval’s theorem for discrete-time periodic signals relates the en-

ergy in the time and the frequency domain
N −1 N −1 N −1 N −1
1
∑ | x (n)|2 = ∑ ∑ ∑ X (k1 ) X ∗ (k2 )e j2πn(k1 −k2 )/N
n =0 N2 n =0 k 1 =0 k 2 =0
N −1 N −1 N −1
1 1
= 2 ∑ ∑ X (k1 ) X ∗ (k2 ) Nδ(k1 − k2 ) = ∑ | X (k)|2 .
N k =0 k =0 N k =0
1 2
5. Convolution of two periodic signals x (n) and h(n), whose period is

N, is defined by
N −1
y(n) = ∑ x ( m ) h ( n − m ).
m =0
The DFT of this signal is
N −1 N −1
Y (k ) = DFT{y(n)} = ∑ ∑ x (m)h(n − m)e− j2πnk/N = X (k ) H (k ).
n =0 m =0
(3.13)
Thus, the DFT of a convolution of two periodic signals is equal to

the product of DFTs of individual signals. Since the convolution is
performed on periodic signals (the DFT inherently assumes signals
periodicity), a circular shift of signals is assumed in the calculation.
This kind of convolution is called circular convolution.
Relation (3.13) indicates that we can calculate convolution of two
aperiodic discrete-time signals of a limited duration in the following
way:
• Calculate DFTs of x (n) and h(n) and obtain X (k ) and H (k ). At
this point, we inherently make periodic extension of x (n) and
h(n) with period N.
• Multiply these two DFTs to obtain DFT of the output signal
Y ( k ) = X ( k ) H ( k ).
• Calculate the inverse DFT to get the convolution
y(n) = IDFT{Y (k )}.
This procedure looks more complex than the direct calculation

of convolution by definition. However, due to very efficient and fast
routines for the DFT and the IDFT calculation, this way of calculation
could be more efficient than the direct one.
In using this procedure, we have to take care about the length of
signals and their DFTs that assume periodic extension.
Example 3.5. Consider a discrete-time signal
x ( n ) = u ( n ) − u ( n − 5).
Calculate the convolution x (n) ∗ x (n). Extend signals with period N = 7 and
calculate the circular convolution (corresponding to the DFT based convolu-
tion calculation with N = 7, which is longer than the signal duration). Com-
pare the results. What value of N should be used for the period so that the
direct convolution corresponds to one period of the circular convolution?
⋆Signal x (n) and its reversed version x (−n), along with the
shifted signal used in the convolution calculation, are presented in Fig.3.3.
In the circular (DFT) calculation, for example, at n = 0, the con-
volution value is
6
x p (n) ∗ x p (n) = ∑ x p (m) x p (0 − m) = 1 + 1 + 1 = 3.
m =0
In addition to the term x (0) x (0) = 1 which exists in the aperiodic convolution,
two terms for m = 3 and m = 4 appeared due to the periodic extension of
1.5 6
x(n)* x(n)
1 4
x(n)
0.5
2
0
-0.5 0
-15 -10 -5 0 5 10 15 -15 -10 -5 0 5 10 15
n n
1.5 1.5
xp(-m+1 )
1 1
x (m)
0.5 0.5
p
0 0
-0.5 -0.5
-15 -10 -5 0 5 10 15 -15 -10 -5 0 5 10 15
n n
1.5 1.5
xp(-m+3 )
1 1
x (-m )
0.5 0.5
p
0 0
-0.5 -0.5
-15 -10 -5 0 5 10 15 -15 -10 -5 0 5 10 15
n n
1.5 6
xp(n)* xp(n)
x (-m+5 )
1 4
0.5
2
0
p
-0.5 0
-15 -10 -5 0 5 10 15 -15 -10 -5 0 5 10 15
n n
Figure 3.3 Illustration of the discrete-time signal convolution and circular convolution for
signals whose length is 5 and the circular convolution is calculated with N = 7.
the signal. They made that the circular convolution value differs from the
convolution of original aperiodic signals. The same situation occurred for
n = 1 and n = 2. For n = 3, 4, and 5 the correct result for aperiodic convolution
is obtained using circular convolution. It could be concluded that if the signal
in circular convolution were separated by at least two more zero values (if the
period N were N ≥ 9) this difference would not occur, Fig.3.4 for N = 9. Then
one period of circular convolution 0 ≤ n ≤ N − 1 would correspond to the
original aperiodic convolution.
If a signal x (n) is of length M, then we can calculate its DFT with

any N ≥ M, so that the signal will not overlap with its periods, added
1.5
1
xp(-m )
xp(m)
0.5
0
-0.5
-15 -10 -5 0 5 10 15 n
n
1.5 6
xp(n)* xp(n)
xp(-m+8 )
1 4
0.5
2
0
-0.5 0
-15 -10 -5 0 5 10 15 -15 -10 -5 0 5 10 15
n n
Figure 3.4 Illustration of the discrete-time signal circular convolution for signals whose
length is 5 and the circular convolution is calculated with N = 9.
using the DFT. If a signal h(n) is of length L, then we can calculate

its DFT with any N ≥ L. However, if we want to use their DFTs for
a convolution calculation (to use circular convolution), then from the
previous example we see that the length of convolution y(n) is M +
L − 1. Therefore, for the DFT-based calculation of y(n), we have to use
at least N ≥ M + L − 1. It means that both DFTs, X (k ) and H (k ), whose
product results in Y (k ), must be at least of N ≥ M + L − 1 duration.
Otherwise, aliasing (overlapping of the periods) will appear. Then
the circular convolution calculated in this way would not correspond
(within the basic period) to the convolution of the original discrete-
time (aperiodic) signals.
Duration of the input signal x (n) may be much longer that the dura-
tion of the impulse response h(n). For example, an input signal may have
tens of thousands of samples, while the impulse response of a discrete sys-
tem duration is, for example, tens of samples, M ≫ L. A direct convolution
would be calculated (after first L − 1 output samples) as
n
y(n) = ∑ x ( m ) h ( n − m ).
m = n − L +1
For each output sample, L multiplications would be used. For a direct DFT
application in the convolution calculation we should wait until the end of
the signal and then zero-pad both the input signal and the impulse response
up to M + L − 1. This kind of calculation is not efficient. Instead of using

a direct DFT calculation, the signal is split into nonoverlapping sequences
whose duration N is of the order of impulse response duration L,
K −1
x (n) = ∑ x k ( n ),
k =0
where xk (n) = x (n)[u(n − kN ) − u(n − (k + 1) N ] and M = KN (the input

signal can always be zero-padded up to the nearest KN duration, where K
is an integer). The output signal is
( )
K −1 n K −1
y(n) = ∑ ∑ xk (m)h(n − m) = ∑ y k ( n ). (3.14)
k =0 m = n − L +1 k =0
For the convolutions yk (n) = xk (n) ∗n h(n) calculation the signals xk (n)
and h(n) should be of duration N + L − 1 only. These convolutions can be
calculated after each N ≪ M input signal samples. The output sequence
yk (n) duration is N + L − 1. Since yk (n), k = 0, 1, . . . , K − 1, are calculated
with step N in time, they overlap, although the input signals xk (n) are
nonoverlapping. For two successive yk (n) and yk+1 (n) and L ≤ N, L − 1
samples within kN + N ≤ n < kN + N + L − 1 overlap. This should be taken
into account, by summing the overlapped output samples in y(n), after the
individual convolutions yk (n) = xk (n) ∗n h(n) are calculated using the DFTs,
Fig.3.5.
3.3 ZERO-PADDING AND INTERPOLATION
The basic period of the DFT X (k ), calculated for k = 0, 1, 2, ..., N − 1, should

be considered as having two parts: one part for 0 ≤ k ≤ N/2 − 1, that
corresponds to the positive frequencies
2π 2π
ω= k or Ω = k, for 0 ≤ k ≤ N/2 − 1, (3.15)
N N∆t
and the other part being a shifted version of the negative frequencies (in the
original aperiodic signal)
2π 2π
ω= (k − N ) or Ω = (k − N ), for N/2 ≤ k ≤ N − 1. (3.16)
N N∆t
Illustration of the frequency correspondence to the frequency index in the
DFT is given in Fig.3.6
x(n)
0 n
h(n)
0 n
x (n)
1
0 n
x (n)
2
0 n
x (n)
3
0 n
y1(n)
0 n
y2(n)
0 n
y (n)
3
0 n
y(n)
0 n
Figure 3.5 Illustration of the convolution calculation when the input signal duration is much
longer then the duration of the system impulse response.
X(Ω)|Ω=2πk/(NΔt)
-N/2 0 N/2-1 k
Ω=2πk/(NΔt)
X(k)
0 N k
Figure 3.6 Relation between the frequency in continuous-time and the DFT frequency index.
We have seen that the DFT of a signal whose duration is limited to

M samples can be calculated by using any N ≥ M. In practice, this means
that we can add (use) as many zeros, after the nonzero signal x (n) values,
as we like. By doing this, we increase the calculation complexity, but we
also increase the number of samples within the same frequency range of the
Fourier transform.
If we recall that
X (k ) = X (e jω )|ω =k∆ω =2πk/N = X (Ω)|Ω=k∆Ω=2πk/( N∆t) , (3.17)
holds in the case when the sampling theorem is satisfied, then we see that by
increasing N in the DFT calculation, the density of sampling (interpolation)
in the Fourier transform of the original signal increases. The DFT interpo-
lation by zero padding the signal in the time domain is illustrated in Fig.
3.7.
The same holds for the frequency domain. If we calculate DFT with N
samples and then add, for example, N zeros after the region corresponding
to the highest frequencies, then by the IDFT of this 2N point DFT, we will
interpolate the original signal in time. All zero values in the frequency
domain should be inserted between two parts (regions) of the original DFT
corresponding to positive and negative frequencies.
x(n)
n
X(k)
k
x(n)
n
X(k)
k
x(n)
n
X(k)
Figure 3.7 Discrete-time signal and its DFT (top two subplots). Discrete-time signal zero-
padded and its DFT interpolated (two subplots in the middle). Zero-padding (interpolation)
factor was 2. Discrete-time signal zero-padded and its DFT interpolated (two bottom subplots).
Zero-padding (interpolation) factor was 4. According to the duality property, the same holds if
X (k) were signal in the discrete-time and x (−n) was its Fourier transform.
Example 3.6. The Hann(ing) window for a signal within − N/2 ≤ n ≤ N/2 − 1, is
1 2πn
w(n) = [1 + cos( )], for − N/2 ≤ n ≤ N/2 − 1. (3.18)
2 N
If the original signal values are within 0 ≤ n ≤ N − 1 then the Hann(ing)
window form is
1 2πn
w(n) = [1 − cos( )], for 0 ≤ n ≤ N − 1. (3.19)
2 N
Present the zero-padded forms of Hann(ing) windows with 2N samples.
⋆The zero-padded form of the Hann(ing) windows used for window-
ing data within the intervals − N/2 ≤ n ≤ N/2 − 1 and 0 ≤ n ≤ N − 1 are
shown in Fig.3.8. The DFTs of windows (3.18) and (3.19) are W (k ) = N [δ(k) +
δ(k − 1)/2 + δ(k + 1)/2]/2 and W (k ) = N [δ(k) − δ(k − 1)/2 − δ(k + 1)/2]/2,
respectively. After the presented zero-padding the window DFT realness
property w pz (n) = w pz (n − 2N ) is preserved (for an even N in the case
− N/2 ≤ n ≤ N/2 − 1 and for an odd N for data within 0 ≤ n ≤ N − 1).
3.4 RELATION AMONG THE FOURIER REPRESENTATIONS
Presentation of the DFT will be concluded with an illustration (Fig. 3.9) of

the relation among four forms of the Fourier domain signal representations
for the cases of:
1. Continuous-time aperiodic signal (Fourier transform):
"∞ "∞
x (t) = 1
2π X (Ω)e jΩt dΩ, X (Ω) = x (t)e− jΩt dt.
−∞ −∞
2. Continuous-time periodic signal (Fourier series):
∞
x p (t) = ∑ x (t + mT )
m=−∞
∞ T/2
"
x p (t) = ∑ Xn e j2πnt/T
, Xn = 1
T x (t)e− j2πnt/T dt,
n=−∞
− T/2
1
Xn = X (Ω)|Ω=2πn/T .
T
w(n)
-N/2 0 N/2-1 n
wp(n)
0 N n
w (n)
p
0 N 2N n
w(n)
0 N n
wp(n)
0 N n
w (n)
p
0 2N n
Figure 3.8 Zero-padding of the Hann(ing) windows used to window data within − N/2 ≤
n ≤ N/2 − 1 and 0 ≤ n ≤ N − 1.
If the periodic signal is formed by a periodic extension of an aperiodic

signal x (t) then there is no signal overlapping (aliasing) in the periodic
signal if the original aperiodic signal duration is shorter than the
extension period T.
3. Discrete-time aperiodic signal (Fourier transform of discrete-time
signals)
x (n) = x (n∆t)∆t,
"∞ ∞
x (n) = 1
2π X (e jω )e jωt dω, X (e jω ) = ∑ x (n)e− jωn ,
−∞ n=−∞
∞
2π
X (e jω ) = ∑ X (Ω + m ) .
m=−∞ ∆t |Ω=ω/∆t
The Fourier transform of the discrete-time signal is a periodic exten-
sion X (e jω ), ω = Ω∆t, of the Fourier transform X (Ω) of a continuous-
time signal. There is no overlapping (aliasing) if the width of the
Fourier transform of the original continuous-time signal is shorter
than the extension period 2π/∆t.
4. Discrete-time periodic signal (discrete Fourier transform)
∞
x p (n) = ∑ x (n + mN ) = x p (t)|t=n∆t ,
m=−∞
N −1 N −1
x p (n) = 1
N ∑ X (k )e j2πnk/N , X (k ) = ∑ x (n)e− j2πnk/N ,
k =0 n =0
X (k ) = X (e jω )|ω =2πk/N = X (Ω)|Ω=2πk/( N∆t) = TXk .
In the periodic discrete-time signal x p (n) it has been assumed that

there is no overlapping of the original aperiodic discrete-time signal
x (n) samples, i.e, that its duration is shorter than the period N, x (n) =
x p (n) for 0 ≤ n ≤ N − 1.
3.5 FAST FOURIER TRANSFORM
Algorithms that provide efficient calculation of the DFT, with a reduced

number of arithmetic operations, are called the fast Fourier transform (FFT).
A unified approach to the DFT and the inverse DFT, (3.5), is used. The only
x(t) X(Ω)
X(Ω)
x(t)
t Ω
x(n) = x(t) Δt t = nΔt X(e jω ) = X(Ω)

Ω = ω/Δt
-π≤ω<π
)
jω
x(n)
X(e
-π π
n ω
xp(t) = x(t) Xn = X(Ω)/T Ω = 2πn/T

- T/2 ≤ t < T/2
x (t)
Xn
p
- T/2 T/2
t n
j2πk/T
xp(n) = x(n) X(k) = X(e ) = TX
k
- N/2 ≤ n < N/2 - N/2 ≤ k < N/2
x (n)
X(k)
p
- N/2 N/2 - N/2 N/2
n k
Figure 3.9 Aperiodic continuous-time signal and its Fourier transform (first row). Discrete-
time signal and its Fourier transform (second row). Periodic continuous-time signal and its
Fourier series coefficients (third row). Periodic discrete-time signal and its discrete Fourier
transform (DFT), (fourth row).
differences between the DFT and inverse DFT calculation are in the sign of
the exponent and the division of the final result by N.
Here we will present an algorithm based on splitting the signal x (n),
with N samples, into two signals x (n) for 0 ≤ n ≤ N/2 − 1 and x (n) for
N/2 ≤ n ≤ N − 1, whose duration is N/2. It is assumed that N is an even
number. By definition, a DFT of a signal with N samples is
N −1
DFT N { x (n))} = X (k ) = ∑ x (n)e− j2πnk/N
n =0
N/2−1 N −1
= ∑ x (n)e− j2πnk/N + ∑ x (n)e− j2πnk/N
n =0 n= N/2
N/2−1 @ A
= ∑ x (n) + x (n + N/2)(−1)k e− j2πnk/N
n =0
since e− j2π (n+ N/2)k/N = e− j2πnk/N e− jπk = (−1)k e− j2πnk/N .

For an even number k = 2r we have
N/2−1
DFT N/2 { g(n)} = X (2r ) = ∑ g(n)e− j2πnr/( N/2)
n =0
with
g(n) = x (n) + x (n + N/2).
For an odd number k = 2r + 1, follows
N/2−1
DFT N/2 {h(n)} = X (2r + 1) = ∑ h(n)e− j2πnr/( N/2)
n =0
where
h(n) = ( x (n) − x (n + N/2))e− j2πn/N .
In this way, we split one DFT of N elements into two DFTs of N/2
elements. Having in mind that the direct calculation of a DFT with N
elements requires an order of N 2 operations, it means that we will reduce
the calculation complexity, since N 2 > ( N/2)2 + ( N/2)2 . An illustration of
this calculation, with N = 8, is shown in Fig. 3.10. We can continue and
split N/2 DFTs into N/4 DFTs, and so on. A complete calculation scheme is
shown in Fig. 3.11. We can conclude that in the FFT algorithms an order of
N log2 N of operations is required. Here it is assumed that log 2 N = p is an
integer, i.e., N = 2 p . This a decimation-in-frequency algorithm.
x(0) X(0)
x(1) X(2)
DFT
4
x(2) X(4)
x(3) X(6)
x(4) 0 X(1)
-1 W8
x(5) 1 X(3)
-1 W8 DFT
4
x(6) 2 X(5)
-1 W8
x(7) X(7)
-1 W3
8
Figure 3.10 DFT of length 8 calculation using two DFTs of length 4.
x(0) X(0)
x(1) 0 X(4)
-1 W8
x(2) 0 X(2)
-1 W8
x(3) 2 0 X(6)
-1 W8 -1 W8
x(4) 0 X(1)
-1 W8
x(5) 1 0 X(5)
-1 W8 -1 W8
x(6) 2 0 X(3)
-1 W8 -1 W8
x(7) X(7)
-1 W3 -1 W8
2 -1 W8
0
8
Figure 3.11 FFT calculation scheme obtained by decimation-in-frequency for N = 8.

If we want to be precise the number of additions is exactly
Nadditions = N log2 N.
For the number of multiplications we can see that in the first stage there are
( N/2 − 1) multiplications. In the second stage there are 2 ( N/4 − 1) mul-
B be 4C( N/8 − 1) multiplications. Finally
tiplications. In the next stage would
in the last stage would be 2 p−1 2Np − 1 = N2 ( N N − 1) = 0 multiplications
p
(N = 2 or p = log2 N). The total number of multiplications, in this algo-
rithm, is
* + * + * + * +
N N N p −1 N
Nmultiplicat. = −1 +2 −1 +4 − 1 + ... + 2 −1
2 4 8 2p
N N N N N
= − 1 + − 2 + − 4 + ... + −
2 2 2 2 2
N N 1 − 2p
= p − (1 + 2 + 22 + ... + 2 p−1 ) = p −
2 2 1−2
N N
= log2 N − ( N − 1) = [log2 N − 2] + 1.
2 2
If the multiplications by j and − j were excluded the number of multiplica-
tions would be additionally reduced.
Example 3.7. Consider a signal x (n) within 0 ≤ n ≤ N − 1. Assume that N is an
even number. Show that the DFT of x (n) can be calculated as two DFTs, one
using the even samples of x (n) and the other using odd samples of x (n).
⋆By definition
N −1
X (k ) = ∑ x (n)e− j2πkn/N
n =0
N/2−1 N/2−1
= ∑ x (2m)e− j2πk2m/N + ∑ x (2m + 1)e− j2πk(2m+1)/N
m =0 m =0
N/2−1 N/2−1
= ∑ xe (m)e− j2πkm/( N/2) + e− j2πk/N ∑ xo (m)e− j2πkm/( N/2) , (3.20)
m =0 m =0
where xe (m) = x (2m) and xo (m) = x (2m + 1) are even and odd samples of the
signal, respectively. Thus, a DFT of N elements is split into two DFTs of N/2
elements. Two DFTs of N/2 elements require an order of 2 ( N/2)2 = N 2 /2
operations. It is less than N 2 . In this way, if N/2 is an even number, we can
continue and split two DFTs of N/2 elements into four DFTs of N/4 elements,
and so on. This is a decimation-in-time algorithm, Fig.3.12.
x(0) X(0)
x(4) X(1)
W0 -1
8
x(2) X(2)
W0 -1
8
x(6) X(3)
W0 -1 W2 -1
8 8
x(1) X(4)
W0 -1
8
x(5) X(5)
W0 -1 W1 -1
8 8
x(3) X(6)
W0 -1 W2 -1
8 8
x(7) X(7)
W0 -1 W2 -1 W3 -1
8 8 8
Figure 3.12 Decimation-in-time FFT algorithm for N = 8.
Example 3.8. Consider a signal x (n) within 0 ≤ n ≤ N − 1. Assume that N = 3M.

Show that the DFT of x (n) can be calculated using three DFTs of M samples.
⋆The DFT of x (n) is
3M−1
X (k) = ∑ x (n)e− j2πkn/(3M)
n =0
M −1 2M −1 3M −1
= ∑ x (m)e− j2πkm/(3M) + ∑ x (m)e− j2πkm/(3M) + ∑ x (m)e− j2πkm/(3M)
m =0 m= M m=2M
M −1 @ 2πkM 2πk2M
A 2πmk
= ∑ x (m) + x (m + M )e− j 3M + x (m + 2M)e− j 3M e− j 3M .
m =0
Now we can consider three cases for frequency index k
M −1
X (3k) = ∑ g(n)e− j2πmk/M
m =0
with g(n) = x (m) + x (m + M) + x (m + 2M )
M −1
X (3k + 1) = ∑ r (n)e− j2πmk/M
m =0
D E
with r (n) = x (m) + ax (m + M ) + a2 x (m + 3M ) e− j2πm/(3M) ,
M −1
X (3k + 2) = ∑ p(n)e− j2πmk/M
m =0
D E
with p(n) = x (m) + a2 x (m + M ) + ax (m + 3M ) e− j2π2m/(3M) , where a =
e− j2π/3 . Thus, a DFT of N = 3M elements is split into three DFTs of N/3 = M
elements. Three DFTs of N/3 elements require an order of 3 ( N/3)2 = N 2 /3
operations. If, for example, M = N/3 is an even number, we can continue
and split three DFTs of N/3 elements into six DFTs of N/6 elements, and so
on.
3.6 SAMPLING OF PERIODIC SIGNALS
A periodic signal x (t), with a period T, can be reconstructed if its Fourier se-
ries is with limited number of nonzero coefficients so that Xk = 0 for k > k m
corresponding to frequencies greater than Ωm = 2πk m /T. The periodic sig-
nal can be reconstructed from the samples taken at ∆t < π/Ωm = 1/(2 f m ).
The number of samples within the period is N = T/∆t.
The reconstructed signal is
N −1 t
sin[(n − ∆t )π ]
x (t) = ∑ x (n∆t)
N sin[(n − t
n =0 ∆t )π/N ]
for and odd N and
N −1 t
sin[(n − ∆t )π ]
x (t) = ∑ x (n∆t)e j(n−t/∆t)π/N t
n =0 N sin[(n − ∆t ) π/N ]
for an even N.
Example 3.9. Samples of a signal x (t) are taken with step ∆t = 1. Obtained discrete-
time values are x (n) = [0, 2.8284, − 2, 2.8284, 0, − 2.8284, 2, − 2.8284] for 0 ≤
n ≤ N − 1 with N = 8. Assuming that the signal satisfies the sampling
theorem find its value at t = 1.5. Check the accuracy if the original signal
values were known, x (t) = 3 sin(3πt/4) + sin(πt/4).
⋆Using the reconstruction formula for an even N we get
7
sin[(n − 1.5)π ]
x (1.5) = ∑ x(n)e j(n−1.5)π/8 8 sin[(n − 1.5)π/8] = −0.2242.
n =0
x(t), x(n) with Δt=1

2
-2
-4
0 2 4 6 8
time
Figure 3.13 Periodic signal reconstructed from its samples at ∆t = 1.
This result is equal to the original signal value. Calculation is repeated with
0 ≤ t ≤ 8, with step 0.01. The reconstructed values of x (t) are presented in
Fig.3.13.
In order to prove the sampling theorem of periodic signals write the

signal in a form of the Fourier series
km
x (t) = ∑ Xk e j2πkt/T . (3.21)
k=−k m
Using N samples of x (t) within the signal period (assuming that N is an

odd number), i.e., by sampling the signal at ∆t = T/N, we get
km
x (n∆t) = ∑ Xk e j2πkn/N .
k =−k m
With ( N − 1)/2 ≥ k m we can write
( N −1)/2
T km T
x (n∆t)∆t = ∑ Xk e j2πkn/N = N
N k=− ∑ Xk e j2πkn/N .
k m k=−( N −1)/2
With x (n∆t)∆t = x (n) and TXk = X (k ) this form reduces to the DFT and the
inverse DFT
( N −1)/2 N −1
1
X (k )e j2πkn/N , x (n)− j2πkn/N .
N k=−(∑ ∑
x (n) = X (k ) =
N −1)/2 n =0
Substituting the Fourier series coefficients Xk , expressed in terms of X (k )

and x (n), into signal (3.21), with k m = ( N − 1)/2, we get
N −1 N −1
2 N −1 N −1 2
1 n j2πk t
− j2πk N 1 t n
x (t) = ∑ ∑ x (n)e e T = ∑ ∑ x (n∆t)e j2πk( T − N )
T N
k=− N2−1 n=0 n=0 k=− N −1
2
N −1 n )( N −1)/2 1 − e
j2π ( Tt − N
n )N
1 − j2π ( Tt − N
=
N ∑ x (n∆t)e t n
n =0 1 − e j2π ( T − N )
N −1 π
sin[ ∆t (t − n∆t)]
= ∑ x (n∆t) π
N sin[ N∆t (t − n∆t)]
.
n =0
This is the reconstruction formula that can be used to calculate x (t) for any
t based on the signal values at x (n∆t) with ∆t < π/Ωm = 1/(2 f m ).
In a similar way the reconstruction formula for an even number of
samples N can be obtained.
The sampling theorem reconstruction formula of aperiodic signals
follows as a special case as N → ∞, since for a small argument
π π
sin[ (t − n∆t)] → (t − n∆t)
N∆t N∆t
and
∞ π
sin[ ∆t (t − n∆t)]
x (t) → ∑ x (n∆t) π .
n=−∞ ∆t (t − n∆t )
Example 3.10. For a signal x (t) whose period is T it is known that the signal has
components corresponding to the nonzero Fourier series coefficients at k1 , k2 ,
..., k K . What is the minimal number of signal samples needed to reconstruct
the signal? What condition the sampling instants and the frequencies should
satisfy for the reconstruction?
⋆The signal x (t) can be reconstructed by using the Fourier series
(1.11). In calculations, a finite number of K nonzero terms will be used,
K
x (t) = ∑ Xkm e j2πkm t/T .
m =1
Since there are K unknown values Xk1 , Xk2 ,...,XkK the minimal number of
equations to calculate their values is K. The equations are written for K time
instants
K
∑ Xkm e j2πkm ti /T = x (ti ), for i = 1, 2, ..., K
m =1
or
Xk1 e j2πk1 t1 /T + Xk2 e j2πk2 t1 /T + ... + XkK e j2πkK t1 /T = x (t1 )

Xk1 e j2πk1 t2 /T + Xk2 e j2πk2 t2 /T + ... + XkK e j2πkK t2 /T = x (t2 )
...
j2πk1 tK /T j2πk2 tK /T j2πk K tK /T
Xk 1 e + Xk 2 e + ... + XkK e = x ( t K ).
In a matrix from
ΦX= y, X = Φ −1 y
where
X = [ Xk1 Xk2 ... XkK ] T , y = [ x (t1 ) x (t2 ) ... x (tK )] T
⎡ ⎤
e j2πk1 t1 /T e j2πk2 t1 /T ... e j2πkK t1 /T
⎢ e j2πk1 t2 /T e j2πk2 t2 /T ... e j2πkK t2 /T ⎥
Φ=⎢
⎣ ...
⎥
⎦
... ... ...
e j2πk1 tK /T e j2πk2 tK /T ... e j2πkK tK /T
The reconstruction condition is det ∥Φ∥ ̸= 0 for selected time instants ti and
given frequency indices k i .
3.7 ANALYSIS OF A SINUSOID BY USING THE DFT
Analysis and estimation of frequency and amplitude of pure sinusoidal

signals is of great importance in many applications.
Consider a simple continuous-time sinusoidal signal
x (t) = Ae jΩ0 t (3.22)
whose Fourier transform is X (Ω) = 2πAδ(Ω − Ω0 ). The whole signal en-

ergy is concentrated just in one frequency point at Ω = Ω0 . Obviously, the
position of maximum is equal to the signal frequency. For this operation we
will use the notation
! 6
Ω0 = arg max | X (Ω)| . (3.23)
−∞<Ω<∞
Assume that the signal x (t) is sampled with ∆t. The discrete-time form
of this signal is
x (n) = Ae jω0 n ∆t,
with ω0 = Ω0 ∆t.
In order to compute the DFT of this signal, we will assume a value of
N and calculate
N −1
X (k ) = ∑ Ae jω0 n e− j2πnk/N ∆t.
n =0
In general, the DFT is of the form
N −1
1 − e jω0 N e− j2πk
X (k ) = A ∑ e jω0 n e− j2πnk/N ∆t = A ∆t (3.24)
n =0 1 − e jω0 e− j2πk/N
sin( N (ω0 − 2πk/N )/2)
= Ae j(( N −1)(ω0 −2πk/N )/2) ∆t (3.25)
sin((ω0 − 2πk/N )/2)
' '
' sin( N (ω0 − 2πk/N )/2) '
| X (k)| = | A| '' ' ∆t. (3.26)
sin((ω0 − 2πk/N )/2) '
3.7.1 Leakage Effect
Two cases may appear in the analysis of a sinusoidal signal:

1. The signal frequency is such that
ω0 = 2πk0 /N
or Ω0 = 2πk0 /( N∆t), where 0 ≤ k0 ≤ N − 1 is an integer. The discretization

step ∆t is such that it is contained an integer number of times within the
signal period T0 = 2π/Ω0 . Then
N −1
X (k ) = A ∑ e j2πk0 n/N e− j2πnk/N ∆t = N Aδ(k − k0 )∆t. (3.27)
n =0
Obviously we can find the signal frequency index from
k0 = arg{ max | X (k )|}. (3.28)

0≤ k ≤ N −1
Frequency is calculated as Ω0 = 2πk0 /( N∆t) for 0 ≤ k0 ≤ N/2 − 1 and

Ω0 = 2π (k0 − N )/( N∆t) for N/2 ≤ k0 ≤ N − 1. This case is illustrated in
Fig. 3.14, top row. Noisy signals will be considered later in the book.
X(k)
x(n)
n k
X(k)
x(n)
n k
Figure 3.14 Sinusoid x (n) = cos(8πn/64) and its DFT with N = 64 (top row) and sinusoid
x (n) = cos(8.8πn/64) and its DFT absolute value, with N = 64 (bottom row).
Estimation of amplitude is easy in this case as

1
A= X ( k 0 ).
N∆t
2. In reality, we never know the signal period (or Ω0 ) in advance (if we knew
it, then this analysis would not be needed). So, it is highly unlikely to have a
case with frequency on the grid, when Ω0 = 2πk0 /( N∆t) as in Fig. 3.14, top
row. More common is the case illustrated in Fig. 3.14, bottom row, when the
true signal frequency does not correspond to any DFT sample position. A
simple sinusoidal signal produces then DFT components at all frequencies.
This effect is known as leakage effect.
Estimation of frequency, based on
! ' '6
' sin( N (ω0 − 2πk/N )/2) '
k̂0 = arg max ' ' ' ,
0≤k≤ N −1 sin(( ω0 − 2πk/N )/2)
'
will produce an estimation error

2π
e = Ω0 − k̂0 .
N∆t
The estimation error could be up to a half of the discretization period in
frequency, ∆Ω = 2π/( N∆t),
π π 2π π 2π π
− ≤e< and k̂0 − ≤ Ω0 < k̂0 + . (3.29)
N∆t N∆t N∆t N∆t N∆t N∆t
Two ways to improve the estimation will be described here.

1. The simplest way to reduce the estimation error is to increase the number
of samples and to reduce the discretization interval in frequency ∆Ω =
2π/( N∆t). This could be achieved by appropriate zero-padding in the time
domain, before the DFT calculation (corresponding to the interpolation in
the frequency domain). This way increases the calculation complexity.
2. The other way is based on a window function application in the DFT
calculation
N −1 B 2πk C
X (k ) = ∑ w(n) Ae jω0 n e− j2πnk/N ∆t = W e j( N −ω0 ) ∆t,
n =0
where W (e jω ) is the Fourier transform of the window function. Windows,

like for example Hann(ing) or Hamming window, smooth the transition and
reduce discontinuities at the ending calculation points that cause leakage. A
simple realization with, for example, the Hann(ing) window (relation (2.31)
and Fig.2.7)
1
w(n) = [1 − cos(2nπ/N )] [u(n) − u(n − N − 1)] .
2
adjusted to the time interval 0 ≤ n ≤ N − 1, produces
N −1
1
X H (k ) = ∑ [1 − cos(2nπ/N )] Ae jω0 n e− j2πnk/N ∆t
n =0 2
- .
A N −1 1 j2nπ/N 1 − j2nπ/N
Ae jω0 n e− j2πnk/N ∆t
2 n∑
= 1− e − e
=0 2 2
- .
1 1 1
= X R ( k ) − X R ( k − 1) − X R ( k + 1) ,
2 2 2
where XR (k ) would be the DFT if the rectangular window were used. It is

defined by (3.24). The DFT of sinusoids on the grid and outside of the grid,
multiplied by a Hann(ing) window, are shown in Fig.3.15.
The leakage effect is reduced. However the DFT is spread over two
additional consecutive samples even in the case when the frequency is on
the DFT grid, Fig.3.15(top). In this case the amplitude is estimated as
1
A= [ X (k0 ) + X (k0 + 1) + X (k0 − 1)].
N∆t
x(n)w(n)
XH(k)
n k
x(n)w(n)
XH(k)
n k
Figure 3.15 Sinusoid x (n) = cos(8πn/64) multiplied by a Hann(ing) window and its DFT
with N = 64 (top row) and sinusoid x (n) = cos(8.8πn/64) multiplied by a Hann(ing) window
and its DFT absolute value, with N = 64 (bottom row).
3.7.2 Displacement
A relation of the maximum DFT value with the few surrounding values of
the windowed DFT is used to calculate correction, the displacement bin of
the estimated frequency. If we apply a window function w(n) in the DFT
calculation, we get
B 2πk C
X (k ) = W e j( N −ω0 ) ∆t.
For a given window function it is possible to derive the exact displacement

formula.
Instead of deriving an exact formula for each window form, here we
will present an approach that combines the interpolation and a general
fitting polynomial form. It can be used with any window.
We can always interpolate the DFT, so that there are several DFT
samples within the main lobe. Then for any symmetric window we can
approximate the Fourier transform around the maximum by a quadratic
function (in analog domain X (Ω) = aΩ2 + bΩ + c). Let us denote the largest
sample, following from
! 6
k̂0 = arg max | X (k)| ,
0≤ k ≤ N −1
by
X0 = | X (k̂0 )|
and two neighboring samples by
X−1 = | X (k̂0 − 1)|

X1 = | X (k̂0 + 1)|.
By using the Lagrange polynomial interpolation of the second-order, at a

point x = d, taking the bin index as the independent variable k −1 = −1,
k0 = 0, k1 = 1, with the function values at these points being denoted by
X−1 , X0 and X1 , we have the Lagrange second-order polynomial
(d − 0)(d − 1) (d + 1)(d − 1) (d − 0)(d + 1)

X (k̂0 + d) = X−1 + X0 + X1
(−1 − 0)(−1 − 1) (0 + 1)(0 − 1) (1 − 0)(1 + 1)
= d2 [− X0 + X−1 /2 + X1 /2)] + d[ X1 − X−1 ]/2 + X0 . (3.30)
This function reaches maximum at
∂X (k̂0 + d)/∂d = 0,
resulting in the displacement bin for the frequency correction
| X (k̂0 + 1)| − | X (k̂0 − 1)|

d = 0.5 , (3.31)
2| X (k̂0 )| − | X (k̂0 + 1)| − | X (k̂0 − 1)|
with frequency as in (3.32). The displacement procedure is illustrated in Fig.

3.16.
Thus, the best frequency estimation is
2π
Ω0 = (k̂0 + d) (3.32)
N∆t
2π
for 0 ≤ k̂0 ≤ N/2 − 1 and Ω0 = N∆t (( k̂ 0 + d) − N ) for N/2 ≤ k̂0 ≤ N − 1.
Example 3.11. A sinusoidal signal x (t) = A exp( jΩ0 t) is sampled with a sampling
interval ∆t = 1/128 and N0 = 64 samples are considered. Prior to the DFT
calculation, the signal is zero padded four times. The DFT maximum is
detected at k̂0 = 95. The maximum DFT value is X (95) = 0.9. Neighboring
values are X (96) = 0.7 and X (94) = 0.3. Calculate the displacement bin d and
estimate the value of Ω0 .
X(0) X(1)
X(Ω), X(k)
X(Ω), X(k)
X(-1)
Ω, k Ω, k
Figure 3.16 Illustration of the displacement bin correction for a true maximum position
calculation based on three neighboring values (full range – left and zoomed graph – right)
.
⋆The displacement bin value is

0.7 − 0.3
d = 0.5 = 0.25.
1.8 − 0.7 − 0.3
The total number of samples in the DFT calculation was N = 4N0 = 256,
meaning that the value k̂0 = 95 is within the first half of the samples (cor-
responding to positive frequency Ω0 ). Therefore, we can use (3.32) for the
frequency calculation
2π
Ω0 = (k̂ + d) = 95.25π.
N∆t 0
It is possible to derive the exact displacement formula for some win-

dows, based on their Fourier transform function. For example, for the
Hann(ing) window the exact displacement formula is
1.5[| X (k̂0 + 1)| − | X (k̂0 − 1)|]

dH = B C . (3.33)
| X (k̂0 − 1)| 1 + |X (k̂0 +1)| + | X (k̂0 )| + | X (k̂0 + 1)|
| X (k̂0 )|
After the displacement is calculated the signal can also be modulated

for the displacement frequency shift in order to produce a signal with the
frequency on the frequency grid. This is especially important if we expect
that the signal contains much smaller higher order harmonics that were
masked with strong values of the dominant harmonic. If we detected that
the k0 th harmonic is dominant and displaced for d then this harmonic
should be removed from a signal modulated by the displacement frequency.
The DFT of the new signal is used for the analysis of the second largest
harmonic and so on.
3.8 DISCRETE COSINE AND SINE TRANSFORMS
The DFT of signal satisfies many desirable properties. Also its calculation is
simple and efficient using the FFT algorithm. With the DFT calculation the
signal periodic extension is assumed and embedded in the discrete trans-
form. However, in the DFT case the periodic signal extension will, in gen-
eral, introduce significant signal change (corresponding to discontinuities in
continuous time) at the period ending points Fig.3.17 (first and second row).
This change will significantly worsen the DFT coefficients convergence and
increases number of coefficients in signal reconstruction. In order to reduce
this effect and to improve convergence of the signal transform coefficients
the signal could be extended in an appropriate way.
The discrete cosine transforms (DCT) and discrete sine transforms
(DST) are used to analyze real-valued discrete signals, periodically extended
to produce even or odd signal forms, respectively. However, this extension
is not straightforward for discrete-time signals. Consider a discrete-time
signal of duration N, when x (n) assumes nonzero values for 0 ≤ n ≤ N − 1.
If we try with a direct extension (using all signal values) and form a periodic
signal y(n), whose basic period is of duration 2N, as
!
x (n) for 0 ≤ n ≤ N − 1
y(n) =
x (2N − n − 1) for N ≤ n ≤ 2N − 1
the obtained signal is not even, Fig.3.17(third row). It is obvious that y(n)
does not satisfy the condition y(n) = y(−n) = y(2N − n), required for a
real-valued DFT.
The same holds for an odd extension, Fig.3.17(fourth row),
!
x (n) for 0 ≤ n ≤ N − 1
y(n) = .
− x (2N − n − 1) for N ≤ n ≤ 2N − 1
Thus we have not achieved one of our goals to have a real-valued transform
after a real-valued signal periodic extension. However from Fig.3.17(third
and fourth row) we can see that the signals y(n) are even (or odd) with
respect to the vertical line at n = −1/2. Thus, if we add zeros between each
sample of y(n) and assume that the position which was at n = −1/2 in the
initial signal is the new coordinate origin n = 0 in the new signal z(n), then
these signals will be even and odd, respectively, Fig.3.17(last two rows).
This is just one of possible extensions to make the original discrete-
time signal even (or odd). Several forms of the DCT and DST are defined
based on other ways of getting an even (odd) signal extension.
The most commonly used is the so called DCT-II or just DCT. It will
be presented here. Signal extension for this transform corresponds to the
x(n)
x(n+N) x(n) x(n-N )
x(-n-1 ) y(n) x(n) x(2N-n-1 )
- x(-n-1 ) y(n) x(n) - x(2N-n-1 )
z(n)
z(n)
Figure 3.17 Illustration of a signal x (n), its periodic extension corresponding to the DFT, an
even and odd discrete-time signal extension corresponding to the DCT and DST of type II.
already described one, Fig.3.17. If no form of the DCT is referred to in its

name, then it is assumed that DCT-II form is used. The DCT definition is
N −1
π (2n + 1)
C (k ) = ∑ 2x(n) cos( 2N
k ), 0 ≤ k ≤ N − 1.
n =0
There are two main advantages of this transform over the standard DFT
calculation. The DCT coefficients are real-valued for a real-valued signal.
This transform can produce a better energy concentration than the DFT. In
order to understand why a better energy concentration can be obtained we
will compare the DCT to the standard DFT
N −1
X (k ) = ∑ x (n)e− j2πnk/N ), 0≤k≤ N−1
n =0
calculation procedures for a real-valued signal x (n) with N samples.

In the DCT calculation it is assumed that the signal is extended as an
even function, by creating a sequence
!
x (n) for 0 ≤ n ≤ N − 1
y(n) = .
x (2N − n − 1) for N ≤ n ≤ 2N − 1
This extension eliminates the signal discontinuity at the period ending

points. Thus, in general the Fourier transform of such a signal will converge
faster, requiring fewer coefficients in the reconstruction.
A zero value is then inserted between each pair of samples and a 4 N
even signal z(n) is formed
z(2n + 1) = y(n), z(2n) = 0.
The 4N-sample DFT of z(n) is calculated

4N −1
XC (k ) = DFT{z(n)} = ∑ z(n)e− j2πnk/(4N )
n =0
2N −1 2N −1
= ∑ z(2n + 1)e− j2π (2n+1)k/(4N ) = ∑ y(n)e− j2π (2n+1)k/(4N )
n =0 n =0
N −1
2π (2n + 1)k
= ∑ 2x(n) cos( 4N
) = C ( k ).
n =0
Only N terms of the transform are used and the DCT values are obtained.
Since the basis functions are orthogonal the inverse DCT is obtained
2π (2m+1)k
by multiplying both sides of the DCT by cos ( 4N ) and summing over
0 ≤ k ≤ N − 1,
N −1 N −1
2π (2n + 1)k 2π (2m + 1)k
∑ 2x(n) ∑ wk cos(
4N
) cos(
4N
)
n =0 k =0
N −1
2π (2m + 1)k
= ∑ wk C (k ) cos(
4N
),
k =0
where w0 = 1/2 and wn = 1 for n ̸= 1. Since

N −1
2π (2n + 1)k 2π (2m + 1)k N
∑ wk cos(
4N
) cos(
4N
) = δ(m − n)
2
k =0
we get
N −1
1 2π (2n + 1)k
x (n) =
N ∑ wk C (k ) cos(
4N
). (3.34)
k =0
A symmetric relation, with the same coefficients in the time and fre-
quency domain, is
N −1
2π (2n + 1)k
C (k ) = vk ∑ x (n) cos(
4N
)
n =0
N −1
2π (2n + 1)k
x (n) = ∑ vk C (k ) cos(
4N
),
k =0
√ √
where v0 = 1/N and vn = 2/N for n ̸= 1.
In a similar way the discrete sine transforms are defined. The most
common form is the DST of type II (DST-II), whose definition is
N −1
2π (2n + 1)
S(k ) = ∑ 2x(n) sin( 2N
(k + 1))
n =0
for ≤ k ≤ N − 1. Its relation to the DFT can be established by creating a

sequence
!
x (n) for 0 ≤ n ≤ N − 1
y(n) =
− x (2N − n − 1) for N ≤ n ≤ 2N − 1
Zero values are inserted and a signal z(n) is formed
z(2n + 1) = y(n)
z(2n) = 0.
Again a 4N-sample DFT is calculated

4N −1 2N −1
XS ( k ) = ∑ z(n)e− j2πnk/(4N ) = ∑ y(n)e− j2π (2n+1)k/(4N )
n =0 n =0
% ;
N −1
2π (2n + 1)k
= Im ∑ 2jx(n) sin( 4N
) = S ( k ),
n =0
X(k)
x(n)
n k
[x(n) x(n)]
X (k)
2
n k
C(k)
y(n)
n k
Figure 3.18 Illustration of the DCT calculation.
with N terms of the transform being used. The DST is the imaginary part of
this DFT.
Example 3.12. Consider a signal
x (n) = cos(2π (2n + 1)/64) + 0.75 cos(7π (2n + 1)/64).
Calculate its DFT with N = 32. Plot the periodic extension of this signal. Plot
the even extension y(n) of x (n). Calculate the DFT (the DCT) of such a signal
and discuss the results.
⋆Signal x (n), along with its extended versions and corresponding
transforms, is presented in Fig.3.18. Better energy concentration in the DCT
is due to the introduced symmetry in y(n). The artificial discontinuity in the
DFT, which causes its slow convergence, is eliminated in the DCT.
By using periodic extensions in cosine transform the convolution

property of the DFT is lost. Thus, this kind of transforms may be used for a
signal reconstruction and compression but not in the realization of discrete
systems, unless they are properly related to the corresponding DFT values
(see Problem 3.10).
Example 3.13. For signal

x (n) = sin(2π (2n + 1)/64) − 0.5 cos(7π (2n + 1)/64).
Calculate its DFT with N = 32. Plot the periodic extension of this signal. Plot
even and odd extensions y(n) of x (n). Calculate the DCT and DST. Comment
the results.
⋆Signal with its periodic extensions, corresponding to the DFT, DCT,
and DST, is presented in Fig.3.19(left), as x (n), [ x (n) x (n)], yc (n), and ys (n),
respectively. The corresponding transforms are shown in the right side of
this figure. Note that the convergence of the DFT and DCT is similar. Here
the DST converges faster, since its extension is smoother.
3.9 DISCRETE WALSH-HADAMARD AND HAAR TRANSFORMS
Here we will present two discrete signal transform that can be calculated
without using multiplications. One of them will be used to explain the basic
principle of the wavelet transform calculation as well.
Let us consider a two-sample signal x (n), with N = 2. The correspond-
ing two-sample DFT is
1
X (k) = ∑ x(n)e− j2πnk/2 = x(0) + (−1)k x(1).
n =0
It can be calculated without using multiplications, X (0) = x (0) + x (1) and
X (1) = x (0) − x (1). Now we can show that it is possible to define basis
functions for any signal duration in such a way that the multiplications
are not used in the signal transformation. These transform values will be
denoted by H (k ). For two-sample signal case
H (0) = x (0) + x (1), for k = 0 and H (1) = x (0) − x (1), for k = 1.
The whole frequency interval is represented by a low-frequency value X (0)

and a high-frequency value X (1). In a matrix form
- . - .- .
H (0) 1 1 x (0 )
= . (3.35)
H (1) 1 −1 x (1 )
The transformation matrix is

- .
1 1
T2 = . (3.36)
1 −1
X(k)
x(n)
n k
[x(n) x(n)]
X2(k)
n k
y (n)
C(k)
c
n k
y (n)
S(k)
s
n k
Figure 3.19 Signal and its periodic extensions, corresponding to: the DFT (second row), the
cosine transform (third row), and the sine transform (fourth row). Positive frequencies for the
DFT are shown.
Example 3.14. For the signal shown in Fig. 3.20 calculate the two-sample The DFT
for each pair of signal samples
Hn (0) = y L (n) = x (2n) + x (2n + 1)

Hn (1) = y H (n) = x (2n) − x (2n + 1)
for 0 ≤ n ≤ N/2 − 1. Discuss the results.

⋆The values of Hn (0) = y L (n) and Hn (1) = y H (n) are calculated and
are presented in Fig. 3.20. Signal y L (n) is a low-frequency, smoothed version
of the original signal, while the signal y H (n) contains the details that are
lost in the smoothed version y L (n). The original signal values may easily
x(n)
n
y (n)
L
n
y (n)
H
Figure 3.20 Original signal x (n) and its two-sample lowpas part y L (n) and highpass part
y H ( n ).
be reconstructed from Hn (0) = y L (n) and Hn (1) = y H (n) as

- . - .- .
x (2n) 1 1 1 Hn (0)
=
x (2n + 1) 2 1 −1 Hn (1)
for 0 ≤ n ≤ N/2 − 1.
In some cases the smoothed version y L (n), with a half of the samples of
the original signal, (3.20), is quite good representative of the original signal,
so there is no need to use corrections. Note that for many instants correction
is zero as well.
There are two possibilities to continue and apply the two-point DFT
scheme to a signal with N samples. One of them is in further splitting of
both y L (n) and y H (n) into their low and highpass parts. It leads to a discrete
Walsh-Hadamard transform Fig.3.21. In the other case the splitting is done
for the lowpass part only, while the highpass correction is kept as it is.
It leads to the Haar wavelet transform, Fig.3.22. These two forms will be
explained in details next.
x(n)
yH(n)
y (n)
L
n n
yHL(n)
y (n)
LL
n n
yHH(n)
y (n)
LH
n n
Figure 3.21 Illustration of the procedure leading to the Walsh-Hadamard transform calcula-
tion.
3.9.1 Discrete Walsh-Hadamard Transform
Let us continue the idea of splitting both (lowpass and highpass) parts of the
signal and define a transformation of a four-sample signal. For this signal
form two auxiliary two-sample signals y L (n) and y H (n) as
y L (0 ) = x (0 ) + x (1 ), y L (1 ) = x (2 ) + x (3 ) (3.37)
y H (0 ) = x (0 ) − x (1 ), y H (1 ) = x (2 ) − x (3 ). (3.38)
They represent low-frequency and high-frequency parts of the pairs: x (0),

x (1) and x (2), x (3) of two-sample signals. The lowpass part of the auxiliary
yH(n)
x(n)
n n
yLH(n)
y (n)
L
n n
y (n)
LL
Figure 3.22 Illustration of the procedure leading to the Haar wavelet transform calculation.
two-sample lowpass signal y L (n) is
H (0 ) = y L (0 ) + y L (1 ) = x (0 ) + x (1 ) + x (2 ) + x (3 ).
The highpass part of the auxiliary two-sample lowpass signal y L (n) is
H (1 ) = y L (0 ) − y L (1 ) = x (0 ) + x (1 ) − x (2 ) − x (3 ).
Then we calculate the lowpass part of the auxiliary highpass signal as
H (3 ) = y H (0 ) + y H (1 ) = x (0 ) − x (1 ) + x (2 ) − x (3 ).
Finally the highpass part of the auxiliary highpass signal is
H (4 ) = y H (0 ) − y H (1 ) = x (0 ) − x (1 ) − x (2 ) + x (3 ).
The transformation matrix, for the case of four-sample transform, is

⎡ - .- . ⎤
⎡ ⎤ 1 1 y L (0 )
H (0) ⎢ ⎥
⎢ H (1) ⎥ ⎢ 1 −1 y L (1 ) ⎥
⎢ ⎥=⎢ ------------------- ⎥. (3.39)
⎣ H (2) ⎦ ⎢ - .- . ⎥
⎣ 1 1 y H (0 ) ⎦
H (3)
1 −1 y H (1 )
By replacing the values of y L (n) and y H (n) with signal values x (n),
we get the transformation equation
⎡ ⎤ ⎡ ⎤⎡ ⎤ ⎡ ⎤
H (0) 1 1 1 1 x (0 ) x (0 )
⎢ H (1) ⎥ ⎢ 1 −1 −1 ⎥ ⎢ ⎥ ⎢ ⎥
⎢ ⎥=⎢ 1 ⎥ ⎢ x (1 ) ⎥ = T 4 ⎢ x (1 ) ⎥ , (3.40)
⎣ H (2) ⎦ ⎣ 1 −1 1 −1 ⎦ ⎣ x (2 ) ⎦ ⎣ x (2 ) ⎦
H (3) 1 −1 −1 1 x (3 ) x (3 )
with the transformation matrix T4 .

The next step would be in grouping two four-sample transforms into
an eight-sample-based analysis. The transformation equation in the case of
eight signal samples is
⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤
H (0) 1 1 1 1 1 1 1 1 x (0) x (0)
⎢ H (1) ⎥ ⎢ 1 1 1 1 −1 −1 −1 −1 ⎥ ⎢ x (1) ⎥ ⎢ x (1) ⎥
⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥
⎢ H (2) ⎥ ⎢ 1 1 −1 −1 1 1 −1 −1 ⎥ ⎢ x (2) ⎥ ⎢ x (2) ⎥
⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥
⎢ H (3) ⎥ ⎢ 1 1 −1 −1 −1 −1 1 1 ⎥ ⎢ x (3) ⎥ ⎢ x (3) ⎥
⎢ ⎥=⎢ ⎥ ⎢ ⎥ = T8 ⎢ ⎥, (3.41)
⎢ H (4) ⎥ ⎢ 1 −1 1 −1 1 −1 1 −1 ⎥ ⎢ x (4) ⎥ ⎢ x (4) ⎥
⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥
⎢ H (5) ⎥ ⎢ 1 −1 1 −1 −1 1 −1 1 ⎥ ⎢ x (5) ⎥ ⎢ x (5) ⎥
⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥
⎣ H (6) ⎦ ⎣ 1 −1 −1 1 1 −1 −1 1 ⎦ ⎣ x (6) ⎦ ⎣ x (6) ⎦
H (7) 1 −1 −1 1 −1 1 1 −1 x (7) x (7)
where the transformation matrix is denoted by T8 .

The inverse Walsh-Hadamard transform is x = T− 1 1 T
8 H = 8 T8 H.
The four-sample transformation matrix could be written as
> D E ? > ?
T2 ⊗ 1 1 T2 ⊗ [T2 (1, :)]
T4 = D E =
T2 ⊗ 1 −1 T2 ⊗ [T2 (2, :)]
where ⊗ denotes Kronecker multiplication of two submatrices in T2 (its

rows) with T2 , defined by (3.36). Notation T2 (i, :) is used for the ith row
of T2 . The transformation matrix of order N is obtained by a Kronecker
product of N/2-order transformation matrix rows and T2 ,

⎡ ⎤
T2 ⊗ [T N/2 (1, :)]
⎢ ⎥
⎢ T2 ⊗ [T N/2 (2, :)] ⎥
TN = ⎢⎢ ⎥. (3.42)
... ⎥
⎣ ⎦
T2 ⊗ [T N/2 ( N/2, :)]
In this way, although we started from a two-point DFT, in splitting

the frequency domain, we did not obtain the Fourier transform of a signal,
but a form of the Walsh-Hadamard transform. In ordering the coefficients
(matrix rows) in our example, we followed the frequency region order from
the Fourier domain (for example, in the four-sample case, low-low, low-
high, high-low, and high-high frequency region).
Three ways of ordering transform coefficients in the Walsh-Hadamard
transform (ordering or transformation matrix rows) are used. They produce
the same result with different coefficients order and different recursive
formulae for constructing transformation matrices. The presented way of
ordering coefficients, as in (3.41), is known as the Walsh transform with
dyadic ordering . It will be used in examples and denoted as the Walsh-
Hadamard transform.
The Hadamard transform would correspond to the so called natural
ordering of rows from the transformation matrix T8 ,
⎡ ⎤
1 1 1 1 1 1 1 1
⎢ 1 −1 1 −1 1 −1 1 −1 ⎥
⎢ ⎥
⎢ 1 1 −1 −1 1 1 −1 −1 ⎥
⎢ ⎥
⎢ 1 −1 −1 1 1 −1 −1 1 ⎥
H8 = ⎢ ⎥
⎢ 1 1 1 1 −1 −1 −1 −1 ⎥
⎢ ⎥
⎢ 1 −1 1 −1 −1 1 −1 1 ⎥
⎢ ⎥
⎣ 1 1 −1 −1 −1 −1 1 1 ⎦
1 −1 −1 1 −1 1 1 −1
It would correspond to [ H (0), H (4), H (2), H (6), H (1), H (5), H (3), H (7)]T
order of coefficients in the Walsh transform with dyadic ordering (3.41).
Recursive construction of a Hadamard transform matrix H2N is easy
using the Kronecker product of T2 defined by (3.36) and HN ,
- .
HN HN
H2N = T2 ⊗ HN = .
HN −HN
Order [ H (0), H (1), H (3), H (2), H (6), H (7), H (5), H (4)] T in (3.41) would
correspond to a Walsh transform with sequency ordering.
Calculation of the Walsh-Hadamard transforms requires only addi-
tions. For an N-order transform the number of additions is ( N − 1) N.
3.9.2 Discrete Haar Wavelet Transform
Consider again two pairs of signal samples, x (0), x (1) and x (2), x (3). The
high frequency parts of these pairs are calculated as y H (n) = x (2n) − x (2n +
1), for n = 0, 1. They are used in the Haar transform without any further
modification. Since they represent highpass Haar transform coefficients
they will be denoted, in this case, by W (2) = y H (0) = x (0) − x (1) and
W (3) = y H (1) = x (2) − x (3). The lowpass coefficients of these pairs are
y L (0) = x (0) + x (1) and y L (1) = x (2) + x (3). The highpass and lowpass
parts of these signals are calculated as y LH (0) = [ x (0) + x (1)] − [ x (2) + x (3)]
and y LL (0) = [ x (0) + x (1)] + [ x (2) + x (3)]. For a four-sample signal the
transformation ends here with W (1) = y LH (0) and W (0) = y LL (0). Note
that the order of coefficients is such that the lowest frequency coefficient
corresponds to the transform index k = 0. Matrix form for a four-sample
signal is
⎡ ⎤ ⎡ ⎤⎡ ⎤
W (0) 1 1 1 1 x (0)
⎢ W (1) ⎥ ⎢ 1 1 −1 −1 ⎥ ⎢ ⎥
⎢ ⎥ ⎢ ⎥ ⎢ x (1) ⎥ .
⎣ W (2) ⎦ = ⎣ 1 −1 0 0 ⎦ ⎣ x (2) ⎦
W (3) 0 0 1 −1 x (3)
For an eight-sample signal the highpass coefficients would be kept with-

out further modification in each step (scale), while for the lowpass parts of
signal their highpass and lowpass parts would be calculated. The transfor-
mation matrix in the case of a signal with eight samples is
⎡ ⎤ ⎡ ⎤⎡ ⎤
W (0) 1 1 1 1 1 1 1 1 x (0)
⎢ W (1) ⎥ ⎢ 1 1 1 1 −1 −1 −1 −1 ⎥ ⎢ x (1) ⎥
⎢ ⎥ ⎢ ⎥⎢ ⎥
⎢ W (2) ⎥ ⎢ 1 1 −1 −1 0 0 0 0 ⎥ ⎢ x (2) ⎥
⎢ ⎥ ⎢ ⎥⎢ ⎥
⎢ W (3) ⎥ ⎢ 0 0 0 0 1 1 −1 −1 ⎥ ⎢ x (3) ⎥
⎢ ⎥=⎢ ⎥⎢ ⎥. (3.43)
⎢ W (4) ⎥ ⎢ 1 −1 0 0 0 0 0 0 ⎥ ⎢ x (4) ⎥
⎢ ⎥ ⎢ ⎥⎢ ⎥
⎢ W (5) ⎥ ⎢ 0 0 1 −1 0 0 0 0 ⎥ ⎢ x (5) ⎥
⎢ ⎥ ⎢ ⎥⎢ ⎥
⎣ W (6) ⎦ ⎣ 0 0 0 0 1 −1 0 0 ⎦ ⎣ x (6) ⎦
W (7) 0 0 0 0 0 0 1 −1 x (7)
This is the Haar transform or Haar wavelet transform of a signal with eight
samples.
The Haar transform is useful in the analysis of signals when we can
expect that in a slow-varying signal there are few details.
The Haar wavelet transform is computationally very efficient. The
efficiency comes from the fact that the Haar wavelet transform almost does
not transform the signal at high frequencies. It leaves it almost as it is, using
a very simple two-sample transform. For lower frequencies the number of
operations is increased.
In specific, for the highest N/2 coefficients the Haar transform does
only one addition (of two signal values) for each coefficient. For next N/4
coefficients the Haar wavelet uses 4 signal values with 3 additions and so
on. The total number of additions is for a Haar transform is
N N N N
Nadditions = (2 − 1) + (4 − 1) + (8 − 1) + ... + ( N − 1).
2 4 8 N
For N of the form N = 2m we can write
1 1 1 1
Nadditions = N log2 N − N ( + 2 + 3 + ... + m )
2 2 2 2
1 1 − 21m
= N log2 N − N = N log2 N − ( N − 1) = N [log2 N − 1] + 1.
2 1 − 12
This is the same order of additions as in the FFT algorithms.

x (n) = [2, 2, 12, −8, 2, 2, 2, 2, −3, −3, −3, −3, 3, −9, −3, −3].
Calculate its Haar and Walsh-Hadamard transform with N = 16. Discuss the
results.
⋆Signal x (n) is presented in Fig.3.23. In full analogy with (3.43) a
Haar transformation matrix of order N = 16 is formed. For example, higher
coefficients are just two-sample signal transforms,
W (k ) = x (2(k − 8)) − x (1 + 2(k − 8)), k = 8, 9, ..., 15.
Although there are some short duration pulses (x (2), x (3), x (13)), the Haar
transform coefficients W (2), W (3), ..., W (8), W (10), W (11), W (12), W (13),
W (15) are zero-valued, Fig.3.23. This is the result of its property to decom-
pose the high frequency signal region into short duration (two-sample) basis
functions. Then a short duration pulse is contained in high frequency part
of only one Haar coefficient. That is not the case in the Fourier transform
(or Walsh-Hadamard transform) where a single delta pulse will cause that
all coefficients are nonzero, Fig.3.24. Transformation matrix T16 is obtained
from T8 using (3.42).
Property that high-frequency coefficients are well localized in time and
they represent a short duration signal components is used in image compres-
sion where adding high frequency coefficients adds details into an image,
with important property that one detail in the image corresponds to one (a
few) nonzero coefficient. Reconstruction with the Haar transform with dif-
ferent number of coefficients is presented in Fig.3.23. As explained it can be
considered as "a zooming" a signal toward the details when the higher fre-
quency coefficients are added. Since a half of the coefficients are zero-valued
W(k)
x(n)
n k
x0-1 (n)
x (n)
0
n n
x0-1,9,14 (n)
(n)
0-1,9
x
n k
Figure 3.23 Signal x (n) and its discrete Haar transform H (k ). Reconstructed signals: using
H (0) presented by x0 (n), using two coefficients H (0) and H (1) denoted by x0−1 (n), using
H (0), H (1), and H (9) denoted by x0−1,9 (n), and using H (0), H (1), H (9), and H (14) denoted
by x0−1,9,14 (n). Vertical axes scales for the signal and transform are different.
H(k)
x(n)
n k
Figure 3.24 Signal x (n) and its Walsh-Hadamard transform HD (k).
a significant compression ratio can be achieved by storing or transmitting

the nonzero coefficients only. This is a basic idea for multiresolution wavelet
based image representations and compression.
x(n)
x(n)
n n
W(k)
W(k)
k k
H(k)
H(k)
k k
Figure 3.25 The Haar wavelet transform (second row) and the Walsh-Hadamard transform
(third row) for high frequency long duration signals (first row). Vertical axes scales for the
signal and transform are different.
Example 3.16. For long duration signals with high-frequency components:

(a) x (n) = [1, −1, 1, −1, 1, −1, 1, −1, 1, −1, 1, −1, 1, −1] and
(b) x (n) = [2, 0, −2, 0, 0, −2, 0, 2, 0, 2, 0, −2, −2, 0, 2, 0], calculate the Haar
wavelet transform and the Walsh-Hadamard transform with N = 16.
⋆The Haar wavelet transform and the Walsh-Hadamard transform are

shown in Fig.3.25. We can see that for a signal of long duration on high
frequencies the number of nonzero coefficients in the Haar wavelet trans-
form is large. Just one such component in the Walsh-Hadamard transform
can require a half of the available coefficients in the Haar wavelet transform,
Fig.3.25(left). In addition to the fact that a much smaller number of coef-
ficients is used for the Walsh-Hadamard transform based reconstruction, a
very large number of coefficients in the Haar wavelet transform reconstruc-
tion may annul its calculation complexity advantage in this case.
3.10 PROBLEMS
Problem 3.1. Calculate the DFT of signals using the smallest possible value
of N: a) x (n) = δ(n), b) x (n) = δ(n) + δ(n − 1) − 2jδ(n − 2) + 2jδ(n − 3) +
δ(n − 4), and c) x (n) = an (u(n) − u(n − 10)).
Problem 3.2. If the signals g(n) and f (n) are real-valued show that their
DFTs, G (k ) and F (k ), can be obtained from the DFT Y (k ) of the signal
y(n) = g(n) + jh(n).
Problem 3.3. The relationship between the DFT index and the continuous
signal frequency is given by
%
2πk/( N∆t) for 0 ≤ k ≤ N/2 − 1
Ω=
2π (k − N )/( N∆t) for N/2 ≤ k ≤ N − 1.
This mapping is achieved in programs by using shift functions. Show that

the shift will not be necessary if we use the signal x (n)(−1)n . The DFT
values of this signal will start from the lowest negative frequency, toward
the highest positive frequency.
Problem 3.4. If the DFT of signal x (n) with period N is X (k ) find the DFT
of signals !
x (n) for n = 2m
y(n) =
0 for n = 2m + 1
and !
0 for n = 2m
z(n) = .
x (n) for n = 2m + 1
Problem 3.5. Using the DFT find a convolution of signals x (n) and h(n)
whose nonzero values are x (0) = 1, x (1) = −1 and h(0) = 2, h(1) = −1,
h(2) = 2.
Problem 3.6. Find a circular convolution of x (n) = e j4πn/N + sin(2πn/N )
and h(n) = cos(4πn/N ) + e j2πn/N within the common period of signals.
Problem 3.7. Find the signal whose DFT is Y (k ) = | X (k )|2 and X (k ) is the
DFT of x (n) = u(n) − u(n − 3) with period N = 10.
Problem 3.8. What is the relation between the discrete Hartley transform
(DHT) of real-valued signals
N −1 * +
2πnk 2πnk
H (k ) = ∑ x (n) cos + sin
n =0
N N
and the DFT? Express the DHT in terms of the DFT and the DFT in terms of
the DHT.
Problem 3.9. Show that the DCT of a signal x (n) with N samples, defined
by
N −1
2πk 1
C (k ) = ∑ 2x(n) cos( 2N (n + 2 ))
n =0
can be calculated using an N-sample DFT of the signal

!
2x (2n) for 0 ≤ n ≤ N/2 − 1
y(n) =
2x (2N − 2n − 1) for N/2 ≤ n ≤ N − 1
as
πk
N −1 2πk n πk
C (k ) = Re{e− j 2N ∑ y(n)e− j N } = Re{e− j 2N DFT{y(n)}}.
n =0
Problem 3.10. A real-valued signal x (n) of a duration shorter than N,

defined for 0 ≤ n ≤ N − 1, has the Fourier transform X (k ). A signal y(n)
is formed as
!
2x (n) for 0 ≤ n ≤ N − 1
y(n) = , (3.44)
0 for N ≤ n ≤ 2N − 1
with the DFT Y (k ), then a signal z(n) is formed using
z(2n + 1) = y(n)
z(2n) = 0.
(a) What are the real and imaginary parts of Z (k ) = DFT{z(n)}? How
they are related to the DCT and DST of x (n)? (b) The signal x (n) is applied
as an input to a system with impulse response h(n) such that h(n) is of
duration shorter than N, defined within 0 ≤ n ≤ N − 1, and x (n) ∗n h(n) is
also within 0 ≤ n ≤ N − 1. The DCT of the output signal is calculated. How
it is related to the DCT and DST of x (n)?
Problem 3.11. Consider a signal x (n) whose duration is N, with nonzero

values within the interval 0 ≤ n ≤ N − 1. Define a system with the output
N −1
yk (n + ( N − 1)) = ∑ x (n + m)e− j2πmk/N
m =0
so that its value yk ( N − 1) at the last instant of the signal duration is equal
to the DFT of signal, for a given k,
N −1
y k ( N − 1) = ∑ x (m)e− j2πmk/N = DFT{ x (n)} = X (k ).
m =0
Note that the system is causal since yk (n) uses only x (n) at instant n and
previous instants.
Show that the output signal yk (n) is related to previous output value
yk (n − 1) by the equation
yk (n) = e j2πk/N yk (n − 1) + e j2πk/N [ x (n) − x (n − N )].
This equation can be used for a recursive DFT calculation.

Problem 3.12. Show that the discrete Hartley transform (DHT) coefficients
of a signal x (n) with an even number of samples N can be calculated, for an
even frequency index k = 2r, as a DHT with N/2 samples.
√
Problem 3.13. Find the DFT of signal x (n) = exp( j4π 3n/N ), for n =
0, 1, . . . , N − 1 with N = 16. If the DFT is interpolated four times (signal zero-
padded), find the displacement bin, estimate the frequency, and compare it
with the true frequency value. What is the displacement bin if the general
formula is applied without interpolation?
3.11 SOLUTIONS
Solution 3.1. The DFT assumes that the signals are periodic. In order to
calculate the DFT we have to assume a period of signals first. Period N
should be greater or equal to the duration of signal, so that the signal values
do not overlap. Larger values of N will increase the density of the frequency
domain samples, but will also increase the computation time.
a) For this signal any N ≥ 1 is acceptable, producing
X (k ) = 1, k = 0, 1, ..., N − 1,
with period N.
b) We may use any N ≥ 5. Using N = 5 we get:
5−1
X (k ) = ∑ x(n)e− j2πnk/5 = 1 + e− j2πk/5 − 2je− j4πk/5 + j2e− j6πk/5 + e− j8πk/5
n =0
= 1 + 2 cos(2πk/5) − 4 sin(4πk/5).
c) For a period N ≥ 10
9
1 − a10 e− j2πk(10/N )
X (k ) = ∑ (ae− j2πk/N )n = 1 − ae− j2πk/N
.
n =0
Solution 3.2. From y(n) = g(n) + j f (n) the real and imaginary parts g(n)
and f (n) can be obtained as
y(n) + y∗ (n) y(n) − y∗ (n)

g(n) = , and f (n) = .
2 2j
Since the DFT of y∗ (n) is equal to

( )∗
N −1 N −1
∗ ∗ − j2πnk/N j2πnk/N
DFT{y (n)} = ∑ y (n)e = ∑ y(n)e
n =0 n =0
with e j2πnk/N = e j2πn(k− N )/N = e− j2πn( N −k)/N , it follows
DFT{y∗ (n)} = Y ∗ ( N − k ).
Then the DFTs of signals g(n) and f (n) are
Y (k ) + Y ∗ ( N − k ) Y (k ) − Y ∗ ( N − k )
G (k) = and F (k ) = .
2 2j
Solution 3.3. The DFT of x (n)(−1)n is
N −1
X1 ( k ) = ∑ x (n)(−1)n e− j2πnk/N .
n =0
For 0 ≤ k ≤ N/2 − 1
N −1 N −1
N
X1 ( k ) = ∑ x (n)e− jπn e− j2πnk/N = ∑ x (n)e− j2πn(k+ N/2)/N = X (k + ).
n =0 n =0 2
For N/2 ≤ k ≤ N − 1
N −1 N −1
N
X1 ( k ) = ∑ x (n)e jπn e− j2πnk/N = ∑ x (n)e− j2πn(k− N/2)/N = X (k − ).
n =0 n =0
2
Solution 3.4. The DFT of signal y(n) is
N −1 N −1
Y (k ) = ∑ y(n)e− j2πnk/N = ∑ [x(n) + (−1)n x(n)]e− j2πnk/N
n =0 n =0
N −1
N
= ∑ [x(n) + x(n)e− jπnN/N ]e− j2πnk/N = X (k) + X (k + 2
)
n =0
with X (k + N/2) = X (k − N/2) for k > N/2.

For z(n) the DFT is
N −1 N −1
Z (k ) = ∑ z(n)e− j2πnk/N = ∑ [x(n) − (−1)n x(n)]e− j2πnk/N
n =0 n =0
N
= X ( k ) − X ( k + ).
2
Obviously Y (k ) + Z (k ) = X (k ).
Solution 3.5. For the convolution calculation, using the DFT, the minimal
number N is N = K + L − 1 = 4, where K = 2 is the duration of x (n) and
L = 3 is the duration of h(n). With N = 4 follows
X (k ) = 1 − e− j2πk/4
H (k ) = 2 − e− j2πk/4 + 2e− j4πk/4
Y (k ) = X (k ) H (k ) = (1 − e− j2πk/4 )(2 − e− j2πk/4 + 2e− j4πk/4 )
= 2 − 3e− j2πk/4 + 3e− j4πk/4 − 2e− j6πk/4 .
The signal is
y(n) = IDFT{Y (k )} = 2δ(n) − 3δ(n − 1) + 3δ(n − 2) − 2δ(n − 3).
Solution 3.6. The circular convolution of y(n) = x (n) ∗ h(n) has the DFT
Y (k ) = X (k) H (k ) with
N −1
1 1
X (k ) = ∑ [e j4πn/N + 2j e j2πn/N − 2j e− j2πn/N ]e− j2πnk/N
n =0
N N
= Nδ(k − 2) + δ ( k − 1) − δ ( k + 1)
2j 2j
and
N −1
1 1
H (k) = ∑ [ 2 e j4πn/N + 2 e− j4πn/N + e j2πn/N ]e− j2πnk/N
n =0
N N
= δ(k − 2) + δ(k + 2) + Nδ(k − 1).
2 2
The value of Y (k ) is
N2 N2
Y (k ) = δ ( k − 2) + δ ( k − 1 ).
2 2j
The inverse DFT is

N j4πn/N N j2πn/N
y(n) = e + e .
2 2j
Solution 3.7. The DFT can be written as Y (k ) = X (k ) X ∗ (k ) with
y(n) = IDFT{ X (k )} ∗n IDFT{ X ∗ (k )}.
Since
( )∗
N −1 N −1
IDFT{ X ∗ (k )} = ∑ X ∗ (k )e j2πnk/N = ∑ X (k )e− j2πnk/N
k =0 k =0
( )∗
N −1
= ∑ X (k )e j2πk( N −n)/N = x∗ ( N − n)
k =0
we get
y(n) = ( x (n))10 ∗n ( x ∗ (10 − n))10

= (u(n) − u(n − 3))10 ∗n (u(10 − n) − u(7 − n))10
= (δ(n + 2) + 2δ(n + 1) + 3δ(n) + 2δ(n − 1) + δ(n − 2))10
where ( x (n)) N indicates that the signal is periodically extended with N.

Solution 3.8. For a real-valued signal holds
N −1
2πnk 2πnk
X (k ) = ∑ [x(n) cos N
− jx (n) sin
N
]
n =0
N −1
2πnk 2πnk
X ( N − k) = ∑ [x(n) cos N
+ jx (n) sin
N
].
n =0
Thus,
N −1
2πnk X (k ) + X ( N − k ) H (k ) + H ( N − k )
∑ x (n) cos
N
=
2
=
2
n =0
N −1
2πnk X ( N − k ) − X (k ) H (k ) − H ( N − k )
∑ x (n) sin
N
=
2j
=
2
.
n =0
The DHT can be calculated as a sum of these terms,
2H (k ) = X (k ) + X ( N − k ) − j[ X ( N − k ) − X (k )].
The DFT is obtained using the DHT in the same way as
2X (k ) = H (k ) + H ( N − k ) − j[ H (k ) − H ( N − k )].
Solution 3.9. We can split the DCT sum into an even and odd part
N −1
2πk 1
C (k ) = ∑ 2x(n) cos( 2N (n + 2 )) =
n =0
N/2−1 N/2−1
2πk 1 2πk 1
∑ 2x (2n) cos(
2N
(2n + )) + ∑ 2x (2n + 1) cos(
2 2N
(2n + 1 + )).
2
n =0 n =0
By reverting the summation index in the second sum using n = N/2 − 1 − m

the summation in m is from m = N/2 − 1 for n = 0 down to m = 0 for
n = N/2 − 1. Then
N/2−1
2πk 1
∑ 2x (2n + 1) cos(
2N
(2n + 1 + ))
2
n =0
N/2−1
2πk 1
= ∑ 2x ( N − 2m − 1) cos(
2N
( N − 2m − 1 + )).
2
m =0
Shifting now the summation index in this sum for N/2 + m = n follows
N/2−1
2πk 1
∑ 2x ( N − 2m − 1) cos(
2N
( N − 2m − 1 + ))
2
m =0
N −1
2πk 1
= ∑ 2x (2N − 2n − 1) cos(
2N
(2N − 2n − )).
2
n= N/2
Now we can go back to the DCT and to replace the second sum, to get
N/2−1
2πk 1
C (k ) = ∑ 2x (2n) cos(
2N
(2n + ))
2
n =0
N −1 N −1
2πk 1 2πk 1
+ ∑ 2x (2N − 2n − 1) cos(
2N
(2n + )) = ∑ y(n) cos(
2 2N
(2n + ))
2
n= N/2 n =0
with cos( 2πk 1 2πk 1

2N (2N − 2n − 2 )) = cos( 2N (2n + 2 )) and
!
2x (2n) for 0 ≤ n ≤ N/2 − 1
y(n) =
2x (2N − 2n − 1) for N/2 ≤ n ≤ N − 1
or
N −1 2πk 1 πk
C (k ) = Re{ ∑ y(n)e− j 2N (2n+ 2 ) } = Re{e− j 2N DFT{y(n)}}.
n =0
Solution 3.10. (a) For the signal z(n) we can write

4N −1 2N −1
DFT{z(n)} = ∑ z(n)e− j2πnk/(4N ) = ∑ z(2n + 1)e− j2π (2n+1)k/(4N )
n =0 n =0
2N −1 N −1
= ∑ y(n)e− j2π (2n+1)k/(4N ) = ∑ 2x(n)e− j2π(2n+1)k/(4N ) .
n =0 n =0
The real and imaginary parts of DFT {z(n)} are

N −1
2π (2n + 1)k
Re{DFT{z(n)}} = ∑ 2x(n) cos( 4N
) = C (k )
n =0
N −1
2π (2n + 1)k
Im{DFT{z(n)}} = − ∑ 2x(n) sin( 4N
) = −S(k )
n =0
DFT{z(n)} = C (k ) − jS(k ),
and
N −1
Z (k ) = DFT{z(n)} = e− j2πk/(4N ) ∑ 2x(n)e− j2πnk/(2N )
n =0
jπk/(2N )
Z (k )e = Y (k) = 2X (k/2).
Note that X (k/2) is just a notation for 2X ( 2k ) = Y (k ), where Y (k ) =

DFT{y(n)} and y(n) is zero-padded version of 2x (n) defined by (3.44).
b) If the signal x (n) is input to a system then the DCT is calculated for
xh (n) = x (n) ∗n h(n)

X h ( k ) = X ( k ) H ( k ).
It has been assumed that all x (n), h(n), and x (n) ∗n h(n) are zero-valued
outside 0 ≤ n ≤ N − 1 (it means that the duration of x (n) and h(n) should
be such that their convolution is within 0 ≤ n ≤ N − 1) . Then for a signal
zh (n) related to xh (n) = x (n) ∗n h(n) in the same way as z(n) to x (n) in a)
we can write
k k k k
DFT{zh (n)}e jπk/(2N ) = 2Xh ( ) = 2X ( ) H ( ) = Y (k ) H ( ).
2 2 2 2
Then
k
Ch (k ) = DCT{ xh (n)} = Re{Y (k ) H ( )e− jπk/(2N ) }
2
− jπk/(4N ) k k
= Re{Y (k)e } Re{ H ( )} − Im{Y (k)e− jπk/(4N ) } Im{ H ( )}
2 2
k k
= C (k) Re{ H ( )} + S(k) Im{ H ( )}.
2 2
The system output is x (n) ∗n h(n) = xh (n) = IDCT{Ch )k )}, (3.34). Transform
H (k/2) is the DFT of zero-padded h(n) with factor 2. Only first half of the
DFT samples are then used.
Solution 3.11. For the signal yk (n) we may write
N −1
yk (n) = ∑ x (n − N + 1 + m)e− j2πmk/N .
m =0
Now let us shift the summation

N 2π
N
yk (n) = ∑ x(n − N + m)e− j2π(m−1)k/N = e j N k ∑ x(n − N + m)e− j2πmk/N
m =1 m =1
2π
N −1
= e j N k [ ∑ x (n − N + m)e− j2πmk/N − x (n − N )e− j2π0k/N + x (n)e− j2πNk/N ]
m =0
= e j2πk/N [yk (n − 1) − x (n − N ) + x (n)].
For 0 ≤ n ≤ N − 1
yk (n) = e j2πk/N [yk (n − 1) + x (n)]

since x (n − N ) = 0. This proves the problem statement.

If the signal x (n) continues as a periodic signal after n = 0,
∞
x p (n) = ∑ x(n − lN ) (3.45)
l =0
then, for n ≥ N, it holds x p (n − N ) = x p (n) and yk (n) = e j2πk/N yk (n − 1),

⎧
⎨ 0 for n<0
yk (n) = y (n) = e j2πk/N [yk (n − 1) + x (n)] for 0≤n≤ N−1
⎩ k
yk (n) = e j2πk/N yk (n − 1) for n≥N
for x p (n) defined by (3.45).

Solution 3.12. For k = 2r the DHT can be written as
N/2−1 @ A N −1 @ A
H (2r ) = ∑ x (n) cos 2πrn
N/2 + sin 2πrn
N/2 + ∑ x (n) cos 2πrn
N/2 + sin 2πrn
N/2
n =0 n= N/2
N/2−1 @ A
2πrn
= ∑ ( x (n) + x (n + N/2)) cos N/2 + sin 2πrn
N/2 .
n =0
Therefore H (2r ) is
N/2−1 - .
2πrn 2πrn
H (2r ) = ∑ g(n) cos
N/2
+ sin
N/2
n =0
where g(n) = x (n) + x (n + N/2). This is a DHT of g(n) with N/2 samples.
Note: For odd frequency indices k = 2r + 1 we can write
N −1 - .
2π (2r + 1)n 2π (2r + 1)n
H (2r + 1) = ∑ x (n) cos + sin .
n =0
N N
After some lengthy, but straightforward transformations, we get
N/2−1 - .
2πnr 2πnr
H (2r + 1) = ∑ f (n) cos
N/2
+ sin
N/2
n =0
where
N 2πn N 2πn
f (n) = [ x (n) − x (n + )] cos + [ x ( − n) − x ( N − n)] sin .
2 N 2 N
This is again a DHT of a signal f (n) with N/2 samples.
Solution 3.13. The DFT is

' B C '
15 √ ' sin π (2√3 − k ) '
' '
| X (k)| = | ∑ e j2π (2 3−k)n/16 '
|=' B √ C '' , with (3.46)
n =0 ' sin π (2 3 − k )/16 '
|X| = (1.5799, 2.1361, 3.5045, 10.9192, 9.4607, 3.3454,
2.0805, 1.5530, 1.2781, 1.1225, 1.0362, 0.99781, 0.9992, 1.0406, 1.1310, 1.2929),
where |X| is the vector whose elements are the DFT values | X (k )|, k =
0, 1, ..., 15. Maximal value is at k = 3, meaning that the frequency estimation
without displacement √ bin would be (2π · 3)/16 = 1.1781, while the true
frequency is (2π · 2 3)/16 = 1.3603. The error is 13.4%.
For the zero-padded signal (interpolated DFT), with a factor of 4,
15 √ 15 √
3n/16 − j2πnk/64
| X (k)| = | ∑ e j4π e | =| ∑ e j2π (8 3−k)n/64
|
n =0 n =0
' B C'
' sin π (8√3 − k )/4 '
' '
= '' B √ C '' .
' sin π (8 3 − k )/64 '
@ √ A @ √ A
Maximal value is obtained for k = 8 3 = 14, where 8 3 denotes the
nearest integer value. Then
' B C'
' sin π (8√3 − 14)/4 '
' '
| X (14)| = '' B √ C '' = 15.9662,
' sin π (8 3 − 14)/64 '
' B C'
' sin π (8√3 − 15)/4 '
' '
| X (15)| = '' B √ C '' = 13.9412
' sin π (8 3 − 15)/64 '
' B C'
' sin π (8√3 − 13)/4 '
' '
| X (13)| = '' B √ C '' = 14.8249,
' sin π (8 3 − 13)/64 '
and the displacement bin is
| X (15)| − | X (13)|
d = 0.5 = −0.1395.
2 | X (14)| − | X (15)| − | X (15)|
√
The true frequency index would be 8 3 = 13.8564, with the true frequency
2π · 13.8564/64 = 1.3603. The correct value of frequency index is shifted
4
3
2
x( n)
1
0
-1
-2
-3
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
n
Figure 3.26 Discrete signal x (n) (Exercise 3.4)
from the nearest integer k = 14 (on the frequency grid) for 14 − 13.8564 =
−0.1436, when the interpolation is done. Thus, the obtained displacement
bin value −0.1395 is close to the true shift value −0.1436. The estimated
frequency, using the displacement bin, is 1.3608. As compared to the true
frequency the error is 0.03%.
If the displacement formula is applied on √ the DFT values, without
interpolation, we would get d = 0.3356, while 2 3 = 3.4641 is displaced
from the nearest integer for 0.4641.
3.12 EXERCISE
Exercise 3.1. Find the DFT of x (n) = δ(n) − δ(n − 3) with N = 4 and N = 8.
Exercise 3.2. Calculate the DFT of signal x (n) = sin(nπ/4) for 0 ≤ n < N
with N = 8 and N = 16.
Exercise 3.3. For a real-valued signal the DFT is calculated with N = 8 and
the following DFT values are known: X (0) = 1, X (2) = 2 − j, X (5) = j,
X (7) = 3. Find the remaining values. What are the values of x (0) and
∑7n=0 x (n)?
Exercise 3.4. Signal x (n) is presented in Fig. 3.26. Find X (0), X (4), and X (8),
where X (k ) is the DFT of x (n) calculated with N = 16.
Exercise 3.5. Prove that for an arbitrary real-valued signal x (n), defined for
0 ≤ n < N, where N is an even integer, the DFT value X ( N/2) is real-valued.
4
3
2
X( k)
1
0
-1
-2
-3
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
k
Figure 3.27 DFT of the discrete signal x (n) (Exercise 3.6)
Exercise 3.6. Consider a signal x (n) whose DFT values X (k ), calculated

with N = 16, are presented in Fig. 3.27.
1. Find the DFT of signal y1 (n) = x (n) + (−1)n x (n).
2. Find the DFT of signal y2 (n) = x (n) − (−1)n x (n).
3. Find the DFT of signal y3 (n) = x (n) ∗ x ( N − n) where ∗ denotes a
circular convolution with period N.
4. Find the DFT of signal
%
x (n) for n ̸= 8
y4 ( n ) =
0 for n = 8.
5. Find x (0).
6. Calculate ∑15
n =0 x ( n ).
7. Calculate ∑15 2
n=0 | x ( n )| .
8. Calculate ∑15 n
n=0 (−1) x (n ).
Exercise 3.7. Prove that if | x (n)| ≤ A for 0 ≤ n < N then | X (k )| ≤ N A for

any k, where X (k ) is the DFT of x (n) calculated with N points.
Exercise 3.8. Prove that if ∑nN=−01 | x (n)| ≤ A then ∑kN=−01 | X (k )| ≤ N A where

X (k ) is the DFT of x (n) calculated with N points.
Chapter 4
z-Transform
Fourier transform of discrete signals and the DFT are used for
T
HE
direct signal processing and calculations. A transform that gener-
alizes these transforms, in the same way as the Laplace transform
generalizes the Fourier transform of continuous signals, is the z-transform.
This transform provides an efficient tool for qualitative analysis and design
of the discrete systems.
4.1 DEFINITION OF THE Z-TRANSFORM
The Fourier transform of a discrete-time signal x (n) can be considered as a

special case of the z-transform defined by
∞
X (z) = ∑ x (n )z−n , (4.1)
n=−∞
where z = r exp( jω ) is a complex number. Value of the z-transform along

the unit circle |z| = 1 or z = exp( jω ) is equal to the Fourier transform of
discrete-time signals.
The z-transform, in general, converges only for some values of the
complex argument z. The region of values of z where X (z) is finite is the
region of convergence (ROC) of the z-transform.
x ( n ) = a n u ( n ) + b n u ( n ),
where a and b are complex numbers, | a| < |b|. Find the z-transform of this
signal and its region of convergence.
169
170 z-Transform
a a
Im{z}
Im{z}
Im{z}
b b
Re{z} Re{z} Re{z}
Figure 4.1 Regions of convergence (gray area)
⋆The z-transform of x (n) is

∞ ∞ ∞
X (z) = ∑ (an z−n + bn z−n ) = ∑ (a/z)n + ∑ (b/z)n
n =0 n =0 n =0
1 1 z z
= + = + .
1 − a/z 1 − b/z z−a z−b
Infinite geometric series with progression coefficient ( a/z) converges for
| a/z| < 1. Thus, the region of convergence for the first part of the z-transform
is |z| > | a|. The other series converges for |b/z| < 1, i.e., for |z| > |b|. The
resulting transform is finite if both parts are finite (or do not cancel out to
produce a finite value). Since | a| < |b|, the region of convergence for X (z) is
|z| > |b|, Fig.4.1.

x (n) = an u(n − 1) − bn u(−n − 1) + 2δ(n − 2),
where a and b are complex numbers, |b| > | a|. Find the z-transform of x (n)
and its region of convergence.
⋆The z-transform is
∞ −1 ∞ ∞
X (z) = ∑ an z−n − ∑ bn z−n + 2z−2 = ∑ an z−n − ∑ b−n zn + 2z−2
n =1 n=−∞ n =1 n =1
a/z z/b a z
= − + 2z−2 = + + 2z−2 .
1 − a/z 1 − z/b z−a z−b
Infinite geometric series with progression coefficient ( a/z) converges for
| a/z| < 1. The other series converges for |z/b| < 1. The last term has pole
at z = 0. Since |b| > | a| the region of convergence is | a| < |z| < |b|, Fig.4.2.
Note that in this example and the previous one two different signals
bn u(n) and −bn u(−n − 1) produced the same z-transform Xb (z) = z/(z − b),
but with different regions of convergence.
a a
Im{z}
Im{z}
Im{z}
b b
Re{z} Re{z} Re{z}
Figure 4.2 Regions of convergence (gray area)
4.2 PROPERTIES OF THE Z-TRANSFORM
4.2.1 Linearity
The z-transform is linear since
∞
Z{ ax (n) + by(n)} = ∑ [ ax (n) + by(n)]z−n = aX (z) + bY (z)
n=−∞
with the region of convergence being at least the intersection of the regions
of convergence of X (z) and Y (z). In special cases the region can be larger
than the intersection of the regions of convergence of X (z) and Y (z) if
some poles, defining the region of convergence, cancel out in the linear
combination of transforms.
4.2.2 Time-Shift
For a shifted signal x (n − n0 ) the z-transform is
∞ ∞
Z{ x (n − n0 )} = ∑ x ( n − n0 ) z − n = ∑ x (n)z−(n+n0 ) = X (z)z−n0 .
n=−∞ n=−∞
Additional pole at z = 0 is introduced for n0 > 0. The region of convergence

is the same except for z = 0 or z → ∞, depending on the value of n0 .
Example 4.3. For a causal signal x (n) = x (n)u(n) find the z-transform of x (n +
n0 )u(n), for n0 ≥ 0.
172 z-Transform
⋆ The signal x (n + n0 )u(n) has a z-transform

∞ ∞
Z{ x (n + n0 )u(n)} = ∑ x(n + n0 )z−n = ∑ x(n + n0 )z−(n+n ) zn 0 0
n =0 n =0
> ?
∞
= z n0 ∑ x (n)z−n − x (0) − x (1)z−1 − ... − x (n0 − 1)z−n0 +1
n =0
@ A
= zn0 X (z) − x (0) − x (1)z−1 − ... − x (n0 − 1)z−n0 +1 .
For n0 = 1 follows Z{ x (n + 1)u(n)} = zX (z) − x (0). Note that for this signal
x ( n + n0 ) u ( n ) ̸ = x ( n + n0 ) u ( n + n0 ).
4.2.3 Multiplication by exponential signal: Modulation
For a signal multiplied by an exponential signal the z-transform is

∞
z
Z{ an x (n)} = x ( n ) a n z − n = X ( ),
∑
n=−∞ a
with region of convergence being scaled by | a|. In a special case when

a = e jω0 , the z-transform plane is just rotated
∞
Z{e jω0 n x (n)} = ∑ x (n)e jω0 n z−n = X (ze− jω0 )
n=−∞
with the same region of convergence as X (z).
4.2.4 Differentiation
Consider the z-transform of a causal signal x (n)

∞ ∞
dX (z)
X (z) = ∑ x(n)z−n and dz
= ∑ −nx (n)z−n−1 .
n =0 n =0
We can conclude that

dX (z)
Z{nx (n)u(n)} = −z .
dz
This kind of the z-transform derivations can be generalized to
d N X (z)
Z{n(n − 1)...(n − N − 1) x (n)u(n)} = (−1) N z N .
dz N
4.2.5 Convolution in time
The z-transform of a convolution of signals x (n) and y(n) is
∞
Z{ x (n) ∗ y(n)} = Z{ ∑ x (m)y(n − m)}
m=−∞
∞ ∞ ∞ ∞
= ∑ ∑ x (m )y (n − m )z−n = ∑ ∑ x ( m ) y ( l ) z − m − l = X ( z )Y ( z )
n=−∞ m=−∞ l =−∞ m=−∞
with the region of convergence being at least the intersection of the re-
gions of convergence of X (z) and Y (z). In the case of a product of two z-
transforms it may happen that some poles are canceled out causing that the
resulting region of convergence is larger than the intersection of the individ-
ual regions of convergence.
4.2.6 Table of the z-transform
Signal x (n) z-transform X (z)

δ(n) 1
z
u(n) 1− z , | z | > |1 |
z
an u(n) a−z , |z | > | a |
nan−1 u(n) −z , z > a
( a − z )2
| | | |
z
− an u(−n − 1) a−z ,|z| < | a|
an x (n) X (z/a)
z (1− a2)
a|n| , | a | < 1 (z− a)(1− az)
, | a| < |z| < |1/a|
x ( n − n0 ) z − n 0 X (z)
nx (n)u(n) −zdX (z)/dz
n ( n − 1) x ( n ) u ( n ) z2 d2 X (z)/dz2
1−z−1 cos(ω0 )
cos(ω0 n)u(n) 1−2z−1 cos(ω0 )+z−2
1−z−1 sin(ω0 )
sin(ω0 n)u(n) 1−2z−1 cos(ω0 )+z−2
1
n! u ( n ) exp(z)
[ x (n)u(n)] ∗ u(n) = ∑nm=−∞ x (m) z
z −1 X ( z )
174 z-Transform
4.2.7 Initial and Stationary State Signal Value
The initial value of a causal signal may be calculated as
x (0) = lim X (z). (4.2)

z→∞
According to the z-transform definition all terms with z−n vanishes as

z → ∞. The term which does not depend on z follows then. It is the term
with x (0).
The stationary state value of a causal signal x (n) is
lim x (n) = lim (z − 1) X (z). (4.3)

n→∞ z →1
This relation follows from
Z{ x (n + 1)u(n))} − Z{ x (n)u(n))} = zX (z) − x (0) − X (z)

> ?
N N
−n −n
Z{ x (n + 1)u(n))} − Z{ x (n)u(n))} = lim ∑ x ( n + 1) z
N → ∞ n =0
− ∑ x (n)z
z →1 n =0
= lim [ x ( N + 1) − x (0)].
N →∞
Thus,
lim [ x ( N + 1) − x (0)] = zX (z) − x (0) − X (z),
N →∞
produces the stationary state value (4.3).
4.3 INVERSE Z-TRANSFORM
4.3.1 Direct Power Series Expansion
Most common approach to the z-transform inversion is based on a direct

expansion of the given transform into power series with respect to z−1
within the region of convergence. After the z-transform is expanded into
series
∞
X (z) = ∑ Xn z − n
n=−∞
the signal is identified as x (n) = Xn for −∞ < n < ∞.

In general various techniques may be used to expand a function into

power series. Most of the cases in signal processing, after some transforma-
tions, reduce to a simple form of an infinite geometric series
∞
1
= 1 + q + q2 + ... = ∑ qn
1−q n =0
for |q| < 1.

Example 4.4. For the z-transform
1 1
X (z) = +
1 − 12 z−1 1 − 3z
identify possible regions of convergence and find the inverse z-transform for
each of them.
⋆Obviously the z-transform has the poles z1 = 1/2 and z2 = 1/3. Since
there are no poles in the region of convergence there are three possibilities
to define the region of convergence: 1) |z| > 1/2, 2) 1/3 < |z| < 1/2, and 3)
|z| < 1/3. The signals are obtained by using power series expansion for each
case.
1) For the region of convergence |z| > 1/2 the z-transform should be
written in the form
1 1
X (z) = 1
+ 1
.
1− 2z −3z(1 − 3z )
Now we have two sums of the geometric series

∞ * + ' '
1 1 n ∞
1 −n '1'
' ' < 1 or |z| > 1
1
1 − 2z
= ∑ 2z
= ∑ 2 n
z for ' 2z ' 2
n =0 n =0
* + ' '
1 ∞
1 n ∞
1 −n '1'
' ' < 1 or |z| > 1
1
1 − 3z
= ∑ 3z
= ∑ 3 n
z for ' 3z ' 3
.
n =0 n =0
Both of these sums converge for |z| > 1/2. The resulting power series expan-
sion of X (z) is
∞
1 −n 1 ∞ 1 −n
X (z) = ∑ 2n
z −
3z n∑ n
z
n =0 =0 3
∞ ∞
1 1
= ∑ n z−n − ∑ n z−n .
n =0
2 n =1
3
The inverse z-transform, for this region of convergence, is

1 1
x (n) = u ( n ) − n u ( n − 1).
2n 3
176 z-Transform
2) For 1/3 < |z| < 1/2 the z-transform should be written in the form
−2z 1
X (z) = + 1
.
1 − 2z −3z(1 − 3z )
The corresponding geometric series are
∞ 0
1 1
= ∑ (2z)n = ∑ 2−n z−n for |2z| < 1 or |z| <
1 − 2z n=0 n=−∞ 2
∞ * + ' '
1 1 n ∞
1 −n '1' 1
1
= ∑ = ∑ n z for '' '' < 1 or |z| > .
1 − 3z n =0
3z n =0
3 3z 3
They converge for 1/3 < |z| < 1/2. The resulting power series expansion is
0
1 ∞ 1 −n
∑ 2− n z − n −
3z n∑
X (z) = −2z n
z
n=−∞ =0 3
−1 1 −n ∞
1
=− ∑ n
z − ∑ n z−n .
n=−∞ 2 n =1
3
The inverse z-transform for this region of convergence is
1 1
x (n) = − u(−n − 1) − n u(n − 1).
2n 3
3) For |z| < 1/3 we can write
−2z 1
X (z) = + .
1 − 2z 1 − 3z
The corresponding geometric series are
∞ 0
1 1
= ∑ (2z)n = ∑ 2−n z−n for |2z| < 1 or |z| <
1 − 2z n=0 n=−∞ 2
∞ 0
1 1
= ∑ (3z)n = ∑ 3−n z−n for |3z| < 1 or |z| < .
1 − 3z n=0 n=−∞ 3
Both series converge for |z| < 1/3. The expansion is

0 0
X (z) = −2z ∑ 2− n z − n + ∑ 3− n z − n
n=−∞ n=−∞
−1 1 −n 0
1
=− ∑ n
z + ∑ n z−n .
n=−∞ 2 n=−∞ 3
The inverse z-transform, in this case, is
1 1
x (n) = − u(−n − 1) + n u(−n).
2n 3

X (z) = e a/z
identify the region of convergence and find the inverse z-transform.
⋆ Expanding e a/z into a complex Taylor (Laurant) series
1 1
X (z) = e a/z = 1 + ( a/z) + ( a/z)2 + ( a/z)3 + ...
2! 3!
follows
1 2 1
x (n) = δ(n) + aδ(n − 1) + a δ(n − 2) + a3 δ(n − 3)+
2! 3!
1
= an u ( n ).
n!
The series converges for any z except z = 0.
z2 + 1
X (z) =
(z − 1/2)(z2 − 3z/4 + 1/8)
find the signal x (n) if the region of convergence is |z| > 1/2.
⋆ The denominator of X (z) will be rewritten in the form
z2 + 1 z2 + 1
X (z) = =
(z − 1/2)(z − z1 )(z − z2 ) (z − 1/2)2 (z − 1/4)
where z1 = 1/2 and z2 = 1/4. Writing X (z) in the form of partial fractions
A B C
X (z) = + +
(z − 12 )2 z − 1
2 z− 1
4
the coefficients A, B, and C follow from
( z2 + 1) A(z − 14 ) + B(z − 12 )(z − 14 ) + C (z − 12 )2

=
(z − 12 )2 (z − 14 ) (z − 12 )2 (z − 14 )
or from
1 1 1 1
(z2 + 1) = A(z − ) + B(z − )(z − ) + C (z − )2 . (4.4)
4 2 4 2
For z = 1/4 we get 17/16 = C/16 or C = 17. Value of z = 1/2 gives
1 1 1
( + 1) = A ( − )
4 2 4
178 z-Transform
and A = 5 is obtained. Finally if the highest order coefficients in the relation

(4.4) with z2 are equated
z2 = Bz2 + Cz2
we get 1 = B + C, producing B = −16. The z-transform is
5 −16 17
X (z) = 1 2
+ 1
+ .
(z − 2) z− 2 z − 1/4
For the region of convergence |z| > 1/2 and a parameter | a| ≤ 1/2 holds
∞
1 1 −1 −1
= a = z (1 + az + a 2 z −2 + . . . ) = ∑ a n −1 z − n .
z−a z (1 − z ) n =1
Differentiating both sides of the previous equation with respect to a we get

∞
d 1 1
( )= 2
= ∑ ( n − 1 ) a n −2 z − n
da z − a (z − a) n =2
Using this relation with a = 1/2 the inverse z-transform of X (z) is

n−1 1 1
x (n) = 5 u(n − 2) − 16 n−1 u(n − 1) + 17 n−1 u(n − 1).
2n −2 2 4
Note: In general, the relation
1 1 dm 1
m + 1
= m
( )=
(z − a) m! da z − a
( )
∞
1 dm (n − 1)(n − 2)..(n − m) ∞ n−m−1 −n
= m ∑ a n −1 z − n = ∑a z
m! da n =1
m! n =1
produces the inverse z-transform

(n − 1)(n − 2)..(n − m) n−m−1
x (n) = a u(n)
m!
(n − 1)(n − 2)..(n − m) n−m−1
= a u ( n − m − 1)
m!
* +
n
= a n − m −1 u ( n − m − 1 ).
m
4.3.2 Theorem of Residues Based Inversion
In general the inversion is calculated by using the Cauchy relation from the
complex analysis
O
1
zm−1 dz = δ(m),
2πj
C
where C is any closed contour line within the region of convergence. The
complex plane origin is within the contour. By multiplying both sides of
X (z) by zm−1 , after integration along the closed contour within the region
of convergence we get
O ∞ O
1 1
zm−1 X (z)dz = ∑ 2πj zm−1 x (n)z−n dz = x (m).
2πj n=−∞
C C
The integral is calculated by using the theorem of residues

O
% ;
1 n −1 1 d(k−1) [zn−1 X (z)(z − zi )k ]
x (n) = z X (z)dz = ∑ ,
2πj zi ( k − 1) ! dzk−1 | z = zi
C
where zi are the poles of zn−1 X (z) within the integration contour C that is
in the region of convergence and k is the pole order. If the signal is causal,
n ≥ 0, and all poles of zn−1 X (z) within contour C are simple (first-order
poles with k = 1) then, for a given instant n,
M N
x (n) = ∑ [zn−1 X (z)(z − zi )]|z=zi . (4.5)
zi

2z + 3
X (z) =
(z − 1/2)(z − 1/4)
find a causal signal x (n).
⋆ According to the residuum theorem for n ≥ 1
M N
x (n) = ∑ [zn−1 X (z)(z − zi )]|z=zi
zi
zn−1 (2z + 3) 1 zn−1 (2z + 3) 1

= 1 1
(z − )|z=1/2 + 1 1
(z − )|z=1/4
(z − 2 )( z − 4 )
2 (z − 2 )(z − 4 ) 4
1 1 7
n −1 4 4n −1 2 1 1
= 2 1 + −1
= 16 − 14 n−1 .
4 4
2n −1 4
For n = 0 additional pole at z = 0 exists
z−1 (2z + 3) z−1 (2z + 3) 1

x (0) = 1 1
z | z =0 + 1 1
(z − )|z=1/2
(z − 2 )( z − 4 ) (z − 2 )( z − 4 )
2
z−1 (2z + 3) 1
+ 1 1
(z − )|z=1/4 = 0.
(z − 2 )( z − 4 )
4
180 z-Transform
An easy way to get x (0) is x (0) = limz→∞ X (z).

The resulting inverse z-transform is
1 1
x (n) = 16 u(n − 1) − 14 n−1 u(n − 1).
2n −1 4
It has been assumed that the signal is causal. Using the theorem of
residuum prove that x (n) = 0 for n < 0 with |z| > 1/2.
Hint: Since for each n < 0 there is a pole at z = 0 of the order n + 1, to
avoid different derivatives for each n we can make a substitution of variables
z = 1/p, with dz = −dp/p2 . New region of convergence in the complex
plane p will be p < 2. All poles are now outside this region and outside the
integration contour, producing the zero-valued integral.
4.4 DISCRETE SYSTEMS AND THE Z-TRANSFORM
For a linear time-invariant discrete system described by

∞
y(n) = x (n) ∗ h(n) = ∑ x (m)h(n − m)
m=−∞
the z-transform is given by
Y ( z ) = X ( z ) H ( z ).
The output signal z-transform is obtained by multiplying the input signal

z-transform by the transfer function
∞
H (z) = ∑ h (n )z−n .
n=−∞
It is possible to relate two important properties of a system with the transfer

function properties.
The system is stable if
∞
∑ |h(m)| < ∞.
m=−∞
It means that the z-transform exists at |z| = 1, i.e., that the circle
|z| = 1
a a a
Im{z}
Im{z}
Im{z}
1 1 1
c
b b
Re{z} Re{z} Re{z}
2 4
60 h (n) 1.5 h2(n) 3 h3(n)
1
40 1 2
20 0.5 1
0 0 0
-10 0 10 -10 0 10 -10 0 10
Figure 4.3 Regions of convergence (gray) with corresponding signals. Poles are denoted by
"x".
belongs to the region of convergence for a stable system.

The system is causal if h(n) = 0 for n < 0. Since H (z) = h(0) +
h(1)z−1 + h(2)z−2 + ... it is obvious that z → ∞ belongs to the region of
convergence for a causal system.
From the previous two properties we can conclude that a linear time-
invariant system is stable and causal if the unit circle |z| = 1 and z → ∞
belong to the region of convergence. Since there are no poles within the
region of convergence one may conclude that a transfer function H (z) may
correspond to a stable and causal system only if all of its poles are inside the
unit circle.
Example 4.8. For the systems whose transfer functions are
1
H1 (z) = , |z| > 3/2
(z − 1/3)(z − 3/2)
1
H2 (z) = , 1/3 < |z| < 3/2
z(z − 1/3)(z − 3/2)
1
H3 (z) = , |z| > 3/4
(z − 1/3)(z − 3/4)
plot the regions of convergence and discuss the stability and causality. Find
and plot the impulse response for each case.
182 z-Transform
⋆ The regions of convergence are shown in Fig.4.3. The system de-

scribed by H1 (z) is causal but not stable. The system H2 (z) is stable but not
causal, while the system H3 (z) is both stable and causal. Their impulse re-
sponses are presented in Fig.4.3 as well.
Amplitude of the frequency response (gain) of a discrete system is

related to the transfer function as
| H (e jω )| = | H (z)||z=e jω .
Consider a discrete system whose transfer function assumes the form of a

ratio of two polynomials
B0 + B1 z−1 + ... + B M z− M B (z − z01 )(z − z02 )...(z − z0M )

H (z) = = 0 zN−M
A0 + A1 z−1 + ... + A N z− N A0 (z − z p1 )(z − z p2 )...(z − z pN )
where z0i are zeros and z pi are poles of the transfer function. For the
amplitude of frequency response we my write
' '
' B ' TO1 TO2 ...TO M
| H (e jω )| = '' 0 ''
A0 TP1 TP2 ...TPN
where TOi are the distances from point T at a given frequency z = e jω to

zero Oi at z0i . Distances from point T to poles Pi at z pi are denoted by TPi .
Example 4.9. Plot the frequency response of the causal notch filter with the transfer
function
z − e jπ/3
H (z) =
z − 0.95e jπ/3
⋆ The transfer functions calculation is illustrated in Fig.4.4. Its value is
TO1
| H (e jω )| =
TP1
where O1 is positioned at z01 = e jπ/3 and pole P1 is at z p1 = 0.95e jπ/3 . For

any point T at z = e jω , ω ̸= π/3, the distances TO1 and TP1 from T to O1 and
from T to P1 are almost the same, TO1 ∼ = TP1 . Then | H (z)||z=e jω ∼
= 1 except
at ω = π/3, when TO1 = 0 and TP1 ̸= 0 resulting in | H (z)||z=e jπ/3 = 0. The
frequency response | H (e jω )| is shown in Fig.4.4.
O1
1.5
T P
1
ω
π/3
|H(ejω)|
Im{z}
0.5
0
Re{z} -2 0 π/3 2 ω
Figure 4.4 Poles and zeros of a first-order notch filter (left). The frequency response of this
notch filter (right).
4.5 DIFFERENCE EQUATIONS
An important class of discrete systems can be described by difference equa-

tions. They are obtained by converting corresponding differential equations
or by describing an intrinsically discrete system relating the input and out-
put signal in a recursive way. A general form of a linear difference equation
with constant coefficients, that relates the output signal at an instant n with
the input signal x (n) and the previous input and output samples, is
y(n)+ A1 y(n − 1) + ... + A N y(n − N ) = B0 x (n)+ B1 x (n − 1) + ... + B M x (n − M ).
4.5.1 Solution Based on the z-transform
The z-transform of the linear difference equation, assuming zero-valued

initial conditions, is
[1 + A1 z−1 + ... + A N z− N ]Y (z) = [ B0 + B1 z−1 + ... + B M z− M ] X (z),
since Z{ x (n − i )} = X (z)z−i and Z{y(n − k )} = Y (z)z−k . The solution y(n)

of the difference equation is obtained as an inverse z-transform of
B0 + B1 z−1 + ... + B M z− M
Y (z) = X ( z ).
1 + A1 z−1 + ... + A N z− N
184 z-Transform
Example 4.10. A causal discrete system is described by the difference equation
5 1
y ( n ) − y ( n − 1) + y ( n − 2) = x ( n ). (4.6)
6 6
If the input signal is x (n) = 1/4n u(n) find the output signal.
⋆The z-transform domain form of the system is
1
Y (z) = X ( z ).
1 − 12 z−1 + 16 z−2
The z-transform of the input signal is X (z) = 1/(1 − 14 z−1 ) for |z| > 1/4. The
output signal z-transform is
z3
Y (z) = .
(z − 12 )(z − 13 )(z − 14 )
For a causal system the region of convergence is |z| > 1/2. The output signal
is the inverse z-transform of Y (z). For n > 0 it is
M N
y(n) = ∑ [zn−1 Y (z)(z − zi )]|z=zi
zi =1/2,1/3,1/4
z n +2 z n +2 z n +2
= 1 1
+ 1 1
+
(z − 3 )( z − 4 ) |z=1/2 (z − 2 )( z − 4 ) |z=1/3 (z − 12 )(z − 13 ) |z=1/4
1 8 3
=6 − n + n.
2n 3 4
For n = 0 there is no pole at z = 0. Thus, the above expressions hold for n = 0
as well. The output signal is
- .
6 8 3
y(n) = − + u ( n ).
2n 3n 4n
Note: This kind of solution assumes the initial values from the system causal-
ity and x (n) as y(0) = x (0) = 1 and y(1) − 5y(0)/6 = x (1), i.e., y(1) =
13/12.
Example 4.11. A first-order causal discrete system is described by the following

difference equation
y(n) + A1 y(n − 1) = B0 x (n) + B1 x (n − 1). (4.7)
Find its impulse response and discuss its behavior in terms of the system
coefficients.
⋆For the impulse response calculation the input signal is x (n) = δ(n)
with X (z) = 1. Then we have
(1 + A1 z−1 )Y (z) = ( B0 + B1 z−1 )

B0 + B1 z−1
Y (z) = .
1 + A 1 z −1
The pole of this system is z = − A1 . The are two possibilities for the region
of convergence |z| > | A1 | and |z| < | A1 |. For a causal system the region of
convergence is |z| > | A1 |. Thus, the z-transform Y (z) can be expanded into a
geometric series with q = A1 z−1 = ( A1 /z) < 1
B CB C
Y (z) = B0 + B1 z−1 1 − A1 z−1 + A21 z−2 − A31 z−3 + ... + (− A1 z−1 )n + ...
∞ ∞
= B0 + B0 ∑ (− A1 )n z−n + B1 ∑ (− A1 )(n−1) z−n
n =1 n =1
with
y(n) = B0 δ(n) + (− A1 )n−1 (− A1 B0 + B1 )u(n − 1).
We can conclude that, in general, the impulse response has an infinite dura-
tion for any A1 ̸= 0. It is a result of the recursive relation between the output
y(n) and its previous value(s) y(n − 1). This kind of systems are referred to
as infinite impulse response (IIR) systems or recursive systems. If the value
of coefficient A1 is A1 = 0 then there is no recursion and
y(n) = B0 δ(n) + B1 δ(n − 1).
Then we have a system with a finite impulse response (FIR). This kind of
system produces an output to a signal x (n) as
y(n) = B0 x (n) + B1 x (n − 1).
They are called moving average (MA) systems. Systems without recursion
are always stable since a finite sum of finite signal values is always finite.
Systems that would contain only x (n) and the output recursions, in this
case,
y(n) + A1 y(n − 1) = B0 x (n)
are auto-regressive (AR) systems or all pole systems. This kind of systems
could be unstable, due to recursion. In our case the system is obviously
unstable if | A1 | > 1. Systems (4.7) are in general auto-regressive moving
average (ARMA) systems.
186 z-Transform
If the region of convergence were |z| < | A1 | then the function Y (z)
would be expanded into series with q = z/A1 < 1 as
* + ∞
B0 + B1 z−1 B0 B1
(− A1−1 z)n
A1 n∑
Y (z) = = z +
A1 z−1 (z/A1 + 1) A1 =0
0
B1 0
= B0 ∑ (− A1 )n−1 z−(n−1) + ∑ (− A1 )n z−n
n=−∞ A1 n=− ∞
−1 B1 0
= B0 ∑ (− A1 )n z−n + ∑ (− A1 )n z−n
n=−∞ A1 n=− ∞
with
B1
y(n) = B0 (− A1 )n u(−n − 1) + (− A1 )n u(−n).
A1
This system would be stable if |1/A1 | < 1 and unstable if |1/A1 | > 1, having
in mind that y(n) is nonzero for n < 0. This is an anticausal system since it
has impulse response satisfying h(n) = 0 for n ≥ 1.
Here, we have just introduced the notions. These systems will be
considered in Chapter 5 in details.
4.5.2 Solution of Difference Equations in the Time Domain
A direct way to solve a linear difference equation with constant coefficients

of the form
y(n) + A1 y(n − 1) + ... + A N y(n − N ) = x (n) (4.8)
in the time domain will be described next.

A homogeneous part of this difference equation is
y(n) + A1 y(n − 1) + ... + A N y(n − N ) = 0. (4.9)
Solution for the homogeneous equation is of the form
yi (n) = Ci λin .
Replacing yi (n) into (4.9), the characteristic polynomial equation follows
Ci λin + Ci A1 λin−1 + ... + Ci A N λin− N = 0,

or λiN + A1 λiN −1 + ... + A N = 0.
This is a polynomial of the Nth order. In general, it has N solutions λi ,

i = 1, 2, . . . , N. All functions yi (n) = λin , i = 1, 2, . . . , N are the solutions of
equation (4.9). Since the equation is linear, a linear combination of these

solutions,
N
yh (n) = ∑ Ci λin
i =1
is also a solution of the homogeneous equation (4.9). This solution is called
homogeneous part of the solution of (4.8).
Next a particular solution y p (n), corresponding to the form of input
signal x (n), should be found using the form of x (n). The solution of equa-
tion (4.8) is then
y ( n ) = y h ( n ) + y p ( n ).
The constants Ci , i = 1, 2, . . . , N are calculated based on initial conditions
y(i − 1), i = 1, 2, . . . , N.
Example 4.12. Find the output of a causal discrete system
5 1
y ( n ) − y ( n − 1) + y ( n − 2) = x ( n ) (4.10)
6 6
to the input signal x (n) = (n + 11/6)u(n) by solving the difference equation
in the discrete-time domain. The initial conditions are y(0) = 1 and y(1) = 5.
⋆Solution of the homogeneous part of (4.10)
5 1
y ( n ) − y ( n − 1) + y ( n − 2) = 0
6 6
is of the form yi (n) = Ci λin . Its replacement into the equation results in the
characteristic polynomial
5 1
λ2i − λi + = 0,
6 6
producing λ1 = 1/2 and λ2 = 1/3. The homogeneous part of the solution is
1 1
yh (n) = C1 + C2 n .
2n 3
Since x (n) is a linear function of n, a particular solution is of the form
y p (n) = An + B. Replacing y p (n) into (4.10) we obtain
5 1
y p (n) − y p (n − 1) + y p (n − 2) = n + 11/6
6 6
5 1
An + B − ( An − A + B) + ( An − 2A + B) = n + 11/6,
6 6
and A = 3, B = 1 follow. The solution of (4.10) is a sum of homogeneous and
particular solutions,
1 1
y(n) = yh (n) + y p (n) = C1 + C2 n + 3n + 1.
2n 3
188 z-Transform
Using the initial conditions
y(0) = C1 + C2 + 1 = 1
C1 C
y (1) = + 2 +4=5
2 3
the constants C1 = 6 and C2 = −6 follow. The final solution is
- .
6 6
y(n) = n − n + 3n + 1 u(n).
2 3
Note: The z-transform based solution would assume y(0) = x (0) =
11/6 and y(1) = 5y(0)/6 + x (1) = 157/36. The solution with the initial
conditions y(0) = 1 and y(1) = 5 could be obtained from this solution with
appropriate changes of the first two samples of the input signal in order to
take into account the previous system state and to produce the given initial
conditions y(0) = 1 and y(1) = 5 .
If multiple polynomial roots are obtained, for example λi = λi+1 , then
yi (n) = λin and yi+1 (n) = nλin .
Example 4.13. Goertzel algorithm: Show that a discrete-time signal
y(n) = e j(2πk0 n/N + ϕ)
is a solution of the homogeneous difference equation
y(n) − e j2πk0 /N y(n − 1) = 0. (4.11)
Consider a periodic signal x (n) with a period N and its DFT values X (k ),
1 N −1
x (n) = ∑ X (k )e j2πnk/N . (4.12)
N k =0
If the signal within one of its periods, for 0 ≤ n ≤ N − 1, is applied as the

input to the system described by difference equation (4.11) show that the
output signal at n = N − 1 is equal to the DFT of signal at frequency k = k0 ,
i.e.,
y ( N − 1) = X ( k 0 ).
⋆For the signal y(n) holds
y(n) = e j(2πk0 n/N + ϕ) = e j(2πk0 (n−1+1)/N + ϕ)

= e j(2πk0 /N ) y(n − 1).
Consider now the case when the input signal x (n) is applied to the system.
Since the system is linear, consider one component of the input signal (4.12)
1
xk (n) = X (k)e j2πnk/N ,
N
for an arbitrary 0 ≤ k ≤ N − 1. Then the difference equation for this input

signal reads
yk (n) − e j2πk0 /N yk (n − 1) = xk (n)
Z{ xk (n)}
Yk (z) = . (4.13)
1 − e j2πk0 /N z−1
The z-transform of xk (n), for 0 ≤ n ≤ N − 1, is
1
Z{ xk (n)} = Z{ X (k )e j2πnk/N } (4.14)
N
1 N −1 1 1 − e j2πk z− N
= X (k) ∑ e j2πnk/N z−n = X (k) .
N n =0 N 1 − e j2πk/N z−1
The transform Z{ xk (n)}, for a given k, has zeros at
z0N = e j2πk+ j2lπ , l = 0, 1, 2, ...,N − 1
or
z0 = e j2π (k+l )/N , l = 0, 1, 2, ..., N − 1.
Note that the zero
z0 = e j2πk/N , obtained for l = 0
is canceled with the pole z p = e j2πkn/N in (4.14). Therefore the remaining
zeros are at
z0 = e j2π (k+l )/N , l = 1, 2, ...,N − 1
The output z-transform Yk (z), defined by (4.13), has a pole at
z p = e j2πk0 /N
- If k ̸= k0 then one of zeros z0 = e j2π (k+l )/N , l = 1, 2,...,N − 1 will coincide

with the pole z p = e j2πk0 /N and will cancel it. Thus for k ̸= k0 the function
Yk (z) will not have any poles. Then
O
1
y k ( N − 1) = z N −2 Yk (z)dz = 0 (4.15)
2πj
C
since there are no poles, Fig.4.5.
- If k = k0 then the pole at k = k0 is already canceled in Z{ xk (n)}
and z p = e j2πk0 /N remains as a pole of Y (z). In this case the signal value
at n = N − 1 is equal to the residuum of function in (4.15) at the pole
z p = e j2πk0 /N , relation (4.5),
'
'
yk0 ( N − 1) = z N −2 Yk0 (z)(z − e j2πk0 /N )' j2πk /N
' z=e
0
1 1 − e j2πk0 z− N '
'
= z N −1 X ( k 0 ) '
N 1 − e j2πk0 /N z−1 ' j2πk0 /N
z=e
1 z N − e j2πk0
= X (k0 ) lim j2πk0 /N
= X ( k 0 ).
N z→e j2πk0 /N z − e
190 z-Transform
j2π k n/N -1 j2π k n/N -1

Z {xk(n)} 1/(1-e 0 z ), k≠ k0 1/(1-e 0 z ), k=k0
z=ej2πk/N z=e
j2πk/N
k0=k
Im{z}
Im{z}
Im{z}
z=ej2πk0/N
k0≠ k
Re{z} Re{z} Re{z}
Figure 4.5 Zeros and the pole in Z{ xk (n)} (left), the pole in 1/ (1 − e j2πk0 n/N z−1 ) for k ̸= k0
(middle), and the pole in 1/ (1 − e j2πk0 n/N z−1 ) for k = k0 (right). Illustration is for N = 16.
Therefore the output of the system , at n = N − 1, is
y k ( N − 1) = X ( k ) δ ( k − k 0 ).
Note: The difference relation
y(n) − e j2πk0 n/N y(n − 1) = x (n) (4.16)
with the z-transform domain form

X (z)
Y (z) =
1 − e j2πk0 n/N z−1
is often extended to
X (z) 1 − e− j2πk0 n/N z−1

Y (z) =
1 − e j2πk0 n/N z−1 1 − e− j2πk0 n/N z−1
1 − e− j2πk0 n/N z−1
Y (z) = X (z)
1 − 2 cos(2πk0 n/N )z−1 + z−2
In the discrete-time domain the system
y(n) − 2 cos(2πk0 /N )y(n − 1) + y(n − 2) = x (n) − e− j2πk0 n/N x (n − 1)

(4.17)
is called Goertzel algorithm for the DFT calculation at a given single fre-
quency X (k0 ).
It is interesting to note that the computation of (4.17) is more efficient
than the computation of (4.16). For the calculation of (4.16), for one k0 , we
need one complex multiplication (4 real multiplications) and one complex
addition (2 real additions). For N instants and one k0 we need 4N real
multiplications and 2N real additions. For the calculation of (4.17)we can use
linear property and calculate only
y1 (n) − 2 cos(2πk0 /N )y1 (n − 1) + y1 (n − 2) = x (n) (4.18)
at each instant. It requires a multiplication of complex signal with a real

coefficient. It means 2 real multiplications for each instant or 2N in total for
N instants. The resulting output, at the instant N − 1, is
y( N − 1) = T { x ( N − 1)} − e− j2πk0 ( N −1)/N T { x ( N − 1)}

= y1 ( N ) − e j2πk0 y1 ( N − 1).
It requires just one additional complex multiplication for the last instant
and for one frequency. The total number of multiplications is 2N + 4. It
is reduced with respect to the previously needed 4N real multiplications.
The total number of additions is 4N + 2. It is increased. However the time
needed for a multiplication is much longer than the time needed for an
addition. Thus, the overall efficiency is improved. The efficiency is even more
improved having in mind that (4.18) is the same for calculation of X (k0 ) and
X (−k0 ) = X ( N − k0 ).
4.6 RELATION OF THE Z-TRANSFORM TO OTHER TRANSFORMS
By sampling a signal x (t), the Laplace transform integral can be approxi-

mated by a sum
"∞ ∞ ∞
X (s) = x (t)e−st dt ∼
= ∑ x (n∆t)e−sn∆t ∆t = ∑ x (n)e−sn∆t
−∞ n=−∞ n=−∞
with x (n) = x (n∆t)∆t. Comparing this relation with the z-transform defini-
tion we can conclude that the Laplace transform of x (t) corresponds to the
z-transform of its samples with
z = exp(s∆t),
that is,
X (s) ↔ X (z)|z=exp(s∆t) . (4.19)
A point s = σ + jΩ from the Laplace domain maps into the point
z = re jω with r = eσ∆t and ω = Ω∆t. Points from the left half-plane in the
s domain, σ < 0, map to the interior of unit circle in the z domain, r < 1.
192 z-Transform
According to the sampling theorem, for the Laplace transform of

discrete-time signal holds X (s)|σ=0 = X ( jΩ) = X ( j(Ω + 2kπ/∆t)).
The Fourier transform of a discrete-time signal is
∞
X (e jω ) = X (z)|z=e jω = ∑ x (n)z−n jω .
n=−∞
|z=e
Example 4.14. A causal discrete-time signal x (n) has the Fourier transform X (e jω ).
Write its z-transform in terms of the Fourier transform of the discrete-time
signal, i.e., write the z-transform value based on its values on the unit circle.
⋆The signal can be expressed in term of its Fourier transform as
"π
1
x (n) = X (e jω )e jωn dω
2π
−π
∞ "π ∞
−n 1
X (z) = ∑ x (n)z =
2π
X (e jω ) ∑ e jωn z−n dω
n =0 −π n =0
"π
1 X (e jω )
= dω,
2π 1 − e jω z−1
−π
for |z| > 1.
The DFT of discrete-time signal with N nonzero samples is
N −1
X (k ) = X (e jω )|ω =2πk/N = X (z)|z=e j2πk/N = ∑ x (n)z−n j2πk/N .
n =0
|z=e
Example 4.15. Consider a discrete-time signal with N samples different from zero
within 0 ≤ n ≤ N − 1. Show that all values of X (z), for any z, can be calculated
based on its N samples on the unit circle in the z-plane.
⋆If the signal has N nonzero samples, then it can be expressed in term
of its DFT as
N −1
1 N −1
∑ x (n)e− j2πnk/N and x (n) = X (k )e j2πnk/N .
N k∑
X (k ) =
n =0 =0
Thus, the z-transform of x (n), using only the values of the IDFT where the
original signal is nonzero, 0 ≤ n ≤ N − 1,
1 N −1 N −1 1 N −1 1 − z− N e j2πk
∑ ∑ X (k )e j2πnk/N z−n =
N k∑
X (z) = −1 j2πk/N
X (k)
N k =0 n =0 =0 1 − z e
N=16
jω j2π k/16
z=e z=e
π/Δt
Im{s}=Ω
Im{z}
Im{z}
0
- π/Δt
Re{s}=σ Re{z} Re{z}
Figure 4.6 Illustration of the z-transform relation with the Laplace transform (left), the Fourier
transform of discrete signals (middle), and the DFT (right).
with X (k ) = X (z) at z = exp( j2πk/N ), k = 0, 1, 2, ..., N − 1.

For a periodic signal, including all periods in the z-transform calcula-
tion, holds
1 N −1 ∞ 1 N −1 1
∑ ∑ X (k )e j2πnk/N z−n =
N k∑
X (z) = −1 e j2πk/N
X ( k ).
N k =0 n =0 =0 1 − z
4.7 PROBLEMS
Problem 4.1. Find the z-transform and the region of convergence for the
following signals:
(a) x (n) = δ(n − 2),
(b) x (n) = a|n| u(n),
(c) x (n) = 21n u(n) + 31n u(n)
Problem 4.2. Find the z-transform and the region of convergence for the
following signals:
(a) x (n) = δ(n + 1) + δ(n) + δ(n − 1),
(b) x (n) = 21n [u(n) − u(n − 10)].
Problem 4.3. Using the z-transfrom property that
dX (z)
Y (z) = −z
dz
corresponds to
y(n) = nx (n)u(n)
194 z-Transform
in the discrete-time domain, with the same region of convergence for X (z)
and Y (z), find a causal signal whose z-transform is
(a) X (z) = e a/z , |z| > 0.
(b) X (z) = ln(1 + az−1 ), |z| > | a|.
Problem 4.4. (a) How the z-transform of x (−n) is related to the z-transform
of x (n)?
(b) If the signal x (n) is real-valued show that its z-transfrom satisfies
X ( z ) = X ∗ ( z ∗ ).
Problem 4.5. If X (z) is the z-transform of a signal x (n) find the z-transform
of
∞
y(n) = ∑ x ( k ) x ( n + k ).
k =−∞
Problem 4.6. Find the inverse z-transform of

1 2
X (z) = , |z| > .
2 − 3z 3
Problem 4.7. The z-transform of a causal signal x (n) is
z+1
X (z) = .
(2z − 1)(3z + 2)
Find the signal x (n).

Problem 4.8. The transfer function of a discrete system is
3 − 56 z−1
H (z) = .
(1 − 14 z−1 )(1 − 13 z−1 )
Find the impulse response if:

(a) System is stable,
1
(b) Region of convergence is 4 < |z| < 13 ,
(c) System is anticausal.
Problem 4.9. For the z-transform
1
H (z) = √
3
(1 − 4z)( 14 − 2
2 z+z )
identify possible regions of convergence. In each case comment stability and

causality of the system whose transfer function is H (z). What is the output
of the stable system to the input x (n) = 2 cos(nπ/2)?
Problem 4.10. Find the impulse response of a causal system whose transfer
function is
z+2
H (z) = .
( z − 2) z2
Problem 4.11. Find the inverse z-transform of
z2
X (z) = .
z2 + 1
Problem 4.12. The system is described by a difference equation
5 1 5 3
y ( n ) − y ( n − 1) + y(n − 2) − y(n − 3) = 3x (n) − x (n − 1) + x (n − 2).
16 16 4 16
Find the impulse response of a causal system.
Problem 4.13. Show that the system defined by
3 1
y ( n ) = x ( n ) − x ( n − 1) + x ( n − 2)
4 8
has a finite output duration for an infinite duration input x (n) = 1/4n u(n) .
Problem 4.14. A linear time-invariant system has impulse response
h(n) = 1/3n u(n).
Using the z-transform find the output to the input signal x (n) = u(n) −
u ( n − 6) .
Problem 4.15. Find the output of a causal discrete system
11 1 3
y(n) − y(n − 1) + y(n − 2) = 2x (n) − x (n − 1)
6 2 2
if the input signal is x (n) = δ(n) − 32 δ(n − 1).
Problem 4.16. Solve the difference equation using the z-transform
x (n + 2) + 3x (n + 1) + 2x (n) = 0
with the initial condition x (0) = 0 and x (1) = 1. Signal x (n) is causal.
196 z-Transform
Problem 4.17. Solve the difference equation
x ( n + 1) = x ( n ) + a n
using the z-transform with the initial condition x (0) = 0.

Problem 4.18. Find the output of a causal discrete system
√
2 1
y(n) − y ( n − 1) + y ( n − 2) = x ( n ) (4.20)
2 4
to the input signal x (n) = 31n u(n) by a direct solution of the differential
equation in the discrete-time domain and by using the z-transform. The
initial conditions are y(n) = 0 for n < 0.
Problem 4.19. The first backward difference is defined as
∇ x ( n ) = x ( n ) − x ( n − 1 ),
and the mth backward difference is defined by
∇ m x ( n ) = ∇ m −1 x ( n ) − ∇ m −1 x ( n − 1 ).
The first forward difference is
∆x (n) = x (n + 1) − x (n),
with the mth forward difference being
∆ m x ( n ) = ∆ m −1 x ( n + 1 ) − ∆ m −1 x ( n ).
Find the z-transforms of these differences.

Problem 4.20. Based on the poles-zero geometry plot the amplitude of the
frequency response of system
√ √
y(n) = x (n) − 2x (n − 1) + x (n − 2) + r 2y(n − 1) − r2 y(n − 2)
for r = 0.99. Based on the frequency response, find approximative values of

the output signal if the input is a continuous-time signal
x (t) = 2 cos(10πt) − sin(15πt) + 0.5e j20πt
sampled at ∆t = 1/60.
Problem 4.21. Plot the frequency response of the discrete system (comb
filter)
1 − z− N
H (z) =
1 − rz− N
with r = 0.9999 and r 1/N ∼
= 1. Show that this system has the same transfer
function as
(1 − z−2 ) N/2−1 1 − 2 cos(2kπ/N )z−1 + z−2

(1 − r2 z−2 ) k∏
H (z) = −1 + z −2
.
=1 1 − 2r cos(2kπ/N )z
4.8 SOLUTIONS
Solution 4.1. (a) The z-transform is

∞
X (z) = ∑ δ ( n − 2 ) z − n = z −2
n=−∞
for any z ̸= 0.
(b) For this signal
∞ −1 ∞
(1 − a 2 ) z
X (z) = ∑ a|n| z−n = ∑ a−n z−n + ∑ an z−n = (1 − az)(z − a)
n=−∞ n=−∞ n =0
for |z| < 1/a and |z| > a. If | a| < 1 then the region of convergence is
a < |z| < 1/a.
(c) In this case
∞ ∞
1 −n 1 1 1
X (z) = ∑ n
z + ∑ n z−n = 1 −1
+ 1 −1
n =0 2 n =0 3 1 − 2 z 1 − 3z
2 − 56 z−1 z(2z − 56 )
X (z) = =
(1 − 12 z−1 )(1 − 13 z−1 ) (z − 12 )(z − 13 )
for |z| > 1/2 and |z| > 1/3. The region of convergence is |z| > 1/2.
Solution 4.2. (a) The z-transform is
∞
X (z) = ∑ (δ(n + 1) + δ(n) + δ(n − 1)) z−n =
n=−∞
1
= z + 1 + z −1 = z + 1 + .
z
198 z-Transform
pole-zero cancellation at z=1/2
j2π/10
z=e /2
Im{z}
z=1/2
Re{z}
Figure 4.7 Pole-zero cancellation at z = 1/2.
The region of convergence excludes z = 0 and z −→ ∞.

(b) For x (n) = 21n [u(n) − u(n − 10)] we know that
!
1, n = 0, 1, ..., 9
u(n) − u(n − 10) =
0, elsewhere.
The z-transform is
∞ 9 9
1 −n −n 1 − (2z)−10
X (z) = ∑ x (n )z−n = ∑ z = ∑ ( 2z ) = =
n=−∞ n =0 2
n
n =0 1 − (2z)−1
z−10 z10 − ( 12 )10 z10 − ( 12 )10
= =
z −1 z − 12 z9 (z − 12 )
The expression for X (z) is written in this way in order to find the region of
convergence, observing the zero-pole locations in the z-plane, Fig.4.7. Poles
are at z p1 = 0 and z p2 = 1/2. Zeros are z0i = e j2iπ/10 /2, Fig.4.7. Since the z-
transform has a zero at z0 = 1/2, it will cancel out the pole z p2 = 1/2. The
resulting region of convergence will include the whole z plane, except the
point at z = 0.
Solution 4.3. (a) For X (z) = e a/z holds
dX (z) a a
−z = z 2 e a/z = X (z)
dz z z
The inverse z-transform of left and right side of this equation is
nx (n)u(n) = ax (n − 1)u(n)
dX (z)
since Z [nx (n)] = −z dz and z−1 X (z) = Z [ x (n − 1)]. It means that
a
x (n) = x ( n − 1)
n
for n > 0. According to the initial value theorem
x (0) = lim X (z) = 1.

z→∞
It means that
a2 a3
x (1) = a, x (2) = , x (3 ) = ,...
2 2·3
or
an
x (n) = u ( n ).
n!
(b) For X (z) = ln(1 + az−1 )
dX (z) d(ln(1 + az−1 )) az−2 az−1

Y (z) = −z = −z =z = .
dz dz 1 + az−1 1 + az−1
Therefore
dX (z) az−1
Z [nx (n)] = −z =
dz 1 + az−1
nx (n) = a(− a)n−1 u(n − 1),
producing
−(− a)n
x (n) = u ( n − 1 ).
n
Solution 4.4. (a) The z-transform of signal x (−n) is
∞
X1 ( z ) = ∑ x (−n)z−n .
n=−∞
With a substitution −n = m it follows

∞
X1 ( z ) = ∑ x (n)zm = X (1/z).
m=−∞
200 z-Transform
The region of convergence is complementary to the one of the original

signal. If the region of convergence for x (n) is |z| > a, then the region of
convergence for x (−n) is |z| < a .
(b) For a real-valued signal holds x ∗ (n) = x (n). Then we can write
∗ ∗
X (z ) as
∞ P Q∗
X ∗ (z∗ ) = ∑ x ∗ (n ) (z∗ )−n .
n=−∞
Since (z∗ )−n = ( z−n )∗ we get

∞ ∞
X ∗ (z∗ ) = ∑ x ∗ (n )z−n = ∑ x ( n ) z − n = X ( z ),
n=−∞ n=−∞
for a real-valued signal x (n).

Solution 4.5. From
∞ ∞ ∞
Y (z) = ∑ y (n )z−n = ∑ ∑ x (k ) x (n + k )z−n ,
n=−∞ n=−∞ k=−∞
using the substitution n + k = m, follows
1
Y ( z ) = X ( z ) X ( ).
z
Solution 4.6. A direct expansion of the given transform into power series,
within the region of convergence, will be used. In order to find the signal
x (n) whose z-transform is X (z) = 2−13z , it should be written in a form of
' '
power series with respect to z−1 . Since the condition ' 3z '
2 < 1 does not
correspond to the region of convergence given in the problem formulation
we have to rewrite X (z) as
1 1
X (z) = − 2
.
3z 1 − 3z
'2'
Now the condition ' 3z ' < 1, that is |z| > 2 , corresponds to the problem for-
3
mulation region of convergence. In order to obtain the inverse z-transform,
write
1 1 1
X (z) = − 2
= − X1 ( z ) ,
3z 1 − 3z 3z
where
1
X1 ( z ) = 2
.
1 − 3z
For X1 (z) holds
∞ * +n ∞ * +n
2 2
X1 ( z ) = ∑ = ∑ z−n .
n =0 3z n =0 3
It can be concluded that X (z) can be written as

* +
1 ∞ 2 n −n
3z n∑
X (z) = − z .
=0 3
Comparing the z-transform definition
∞
X (z) = ∑ x (n ) z−n (4.21)
n=−∞
and the last expression it follows

* +
1 ∞ 2 n − n −1
3 n∑
X (z) = − z z =
=0 3
* +
1 ∞ 2 n −(n+1)
=− ∑ z .
3 n =0 3
With the substitution n → n + 1 we get

* +
1 ∞ 2 n −1 − n
X (z) = − ∑ z .
3 n =1 3
Finally, comparing this result with (4.21) we get

⎧ P Q n −1
⎨ − 13 23 , for n = 1, 2, ..., ∞
x (n) = ,
⎩
0 , elsewhere,
or
* + n −1
1 2
x (n) = − u ( n − 1 ).
3 3
Solution 4.7. Since the signal is causal the region of convergence is outside
the pole with the largest radius (outside the circle passing through this pole).
202 z-Transform
Poles of the z-transform are

1 2
z p1 = and z p2 = − .
2 3
The region of convergence is |z| > 23 .
The z-transform is
z+1 A B
X (z) = = +
(2z − 1)(3z + 2) 2z − 1 3z + 2
3 1
A= , B=− .
7 7
The terms in X (z) should be written in such a way that they represent sums
of geometric series for the given region of convergence. From the solution
of the previous problem, we conclude that
A 1 B 1
X (z) = 1
+ 2
.
2z 1 − 2z 3z 1 + 3z
Now we can write

* + * +
A 1 A ∞ 1 n −n A ∞ 1 n − n −1 1
2z n∑ 2 n∑
1
= z = z , |z| >
2z 1 − 2z =0 2 =0 2 2
and
* + * +
B 1 B ∞ 2 n −n B ∞ 2 n − n −1 2
3z n∑
2
= − z = ∑ − z , |z| > .
3z 1 + 3z =0 3 3 n =0 3 3
The z-transform, with m = n + 1, assumes the form

* + * +
A ∞ 1 m −1 − m B ∞ 2 m −1 − m
2 m∑ 3 m∑
X (z) = z + − z .
=1 2 =1 3
Replacing the values for A and B it follows

* + * +
3 ∞ 1 m −m 1 ∞ 2 m −m
X (z) = ∑
14 m∑
z + − z .
7 m =1 2 =1 3
The signal x (n) is obtained by comparing this transform with the z-

transform definition,
* * +n * + +
3 1 1 2 n
x (n) = + − u ( n − 1 ).
7 2 14 3
Solution 4.8. The transfer function may be written as
3 − 56 z−1 A B
H (z) = 1 −1 1 −1
= 1 −1
+
(1 − 4 z )(1 − 3 z ) 1− 4z 1 − 13 z−1
with A = 1, B = 2.
(a) The region of convergence must contain |z| = 1, for a stable system. It is
|z| > 13 .
From
1 2
H (z) = + =
1 − 14 z−1 1 − 13 z−1
∞ * +n ∞ * +n
1 −n 1 1 1
= ∑ z +2 ∑ z−n , |z| > and |z| >
n =0
4 n =0
3 3 4
the impulse response is obtained as
h ( n ) = ( 4− n + 2 × 3− n ) u ( n ).
(b) The region of convergence is 14 < |z| < 13 . The first term in H (z) is the
same as in (a), since it converges for |z| > 14 . It corresponds to the signal
4−n u(n). The second term must be rewritten in such a way that its geometric
series converges for |z| < 13 . Then
∞ −1
2 3z 1
1 −1
= −2 = −2 ∑ (3z)n = −2 ∑ (3z)−m with |z| < .
1− 3z
1 − 3z n =1
m=−n
m=−∞ 3
Signal corresponding to this z-transform is −2 × 3−n u(−n − 1). Then the

impulse response of the system with the region of convergence 14 < |z| < 13
is obtained in the form
h(n) = 4−n u(n) − 2 × 3−n u(−n − 1).
c) For an anticausal system the region of convergence is |z| < 14 . Now the
second term in H (z) is the same as in (b). For |z| < 14 the first term in H (z)
should be written as:
∞ −1
1 4z 1
=− = − ∑ (4z)n = − ∑ (4z)−m with |z| < .
1 − 14 z−1 1 − 4z n =1
m=−n
m=−∞ 4
204 z-Transform
The signal corresponding to this term is −4−n u(−n − 1). The impulse
response of the anticausal discrete system with given transfer function is
h(n) = −4−n u(−n − 1) − 2 × 3−n u(−n − 1).
Solution 4.9. The z-transform

1
H (z) = √
3
(1 − 4z)( 14 − 2
2 z+z )
can be written as
1
H (z) = √ √
3 3
(1 − 4z)(z − 4 + j 14 )(z − 4 − j 14 )
√ √
with poles z1 = 1/4, z2 = 43 − j 14 , and z3 = 43 + j 14 . Since |z2 | = |z3 | = 1/2
possible regions of convergence are: 1) |z| < 1/4, 2) 1/4 < |z| < 1/2, and 3)
|z| > 1/2. In the first two cases the system is neither causal nor stable, while
in the third case the system is causal and stable since |z| = 1 and |z| → ∞
belong to the region of convergence.
The output to x (n) = 2 cos(nπ/2) = 1 + cos(nπ ) = 1 + (−1)n is y(n) =
H (e )|ω =0 × 1 + H (e jω )|ω =π × (−1)n = H (z)|z=1 + H (z)|z=−1 (−1)n =
jω
−0.8681 + 0.0945(−1)n .
Solution 4.10. The transfer function can be written as
z+2 A B C
H (z) = = + + 2.
z2 ( z − 2) z−2 z z
Multiplying both sides by z2 (z − 2) yields
Az2 + Bz(z − 2) + C (z − 2) = z + 2
( A + B)z2 + (−2B + C ) − 2C = z + 2.
The coefficients follow from
A+B=0
−2B + C = 1
−2C = 2,
as A = 1, B = −1, and C = −1. The transfer function is
z −1 1 1
H (z) = − 1
− 2− .
1 − 2z z z
The region of convergence for a causal system is |z| > 2. The inverse z-
transform for a causal system is the system impulse response
h ( n ) = 2n −1 u ( n − 1 ) − δ ( n − 2 ) − δ ( n − 1 ) = δ ( n − 2 ) + 2n −1 u ( n − 3 ).
The system is not stable.

Solution 4.11. The z-transform X (z) can be written in the form
1 1
z2 2z 2z
X (z) = 2
= + .
z +1 z+j z−j
For the region of convergence defined by |z| > 1 the signal is causal and
1 1
x (n) = [1 + (−1)n ] jn u(n) = [1 + (−1)n ]e jπn/2 u(n).
2 2
For n = 4k, where k ≥ 0 is an integer, x (n) = 1 , while for n = 4k + 2 the
signal values are x (n) = −1. For other n the signal is x (n) = 0.
For |z| < 1 the inverse z-transform is
1
x (n) = − [1 + (−1)n ] jn u(−n − 1).
2
Solution 4.12. The transfer function of this system is
3 − 54 z−1 + 3 −2
16 z 3 − 54 z−1 + 3 −2
16 z
H (z) = 5 −2 1 −3
=
1 − z −1 + 16 z − 32 z (1 − 12 z−1 + 1 −2 1 −1
16 z )(1 − 2 z )
1 1 1
= 1 −1
+B C2 + .
1 − 4z 1 − 14 z−1 (1 − 12 z−1 )
For a causal system the region of convergence is outside of the pole z = 1/2,
that is |z| > 1/2. Since
* +'
1 d z '
B C2 = '
da 1 − az − 1 '
1 − 14 z−1 a=1/4
' '
d ∞ n −(n−1) '' ∞ '
n−1 −(n−1) '
∞
1
= ∑ a z ' = ∑ na z ' = ∑ ( n + 1) n z − n ,
da n=0 ' n =0
' n =0
4
a=1/4 a=1/4
the inverse z-transform is

1 1 1
h(n) = n
u ( n ) + ( n + 1 ) n u ( n ) + n u ( n ).
4 4 2
206 z-Transform
Solution 4.13. The transfer function of the system defined by
3 1
y ( n ) = x ( n ) − x ( n − 1) + x ( n − 2)
4 8
is
3 1
H ( z ) = 1 − z −1 + z −2 .
4 8
The z-transform of the input signal x (n) = 1/4n u(n) is
1
X (z) = ,
1 − 14 z−1
with the region of convergence |z| > 1/4. The output signal z-transform is
(1 − 12 z−1 )(1 − 14 z−1 ) 1

Y (z) = H (z) X (z) = 1 −1
= 1 − z −1 .
(1 − 4 z ) 2
Its inverse is a finite duration output signal
y(n) = δ(n) − δ(n − 1)/2.
Solution 4.14. The system transfer function is
1
H (z) =
1 − 13 z−1
and the input signal z-transform is
1 − z −6
X ( z ) = 1 + z −1 + z −2 + z −3 + z −4 + z −5 = .
1 − z −1
The z-transform of the output signal is
1 − z −6
Y (z) = = Y1 (z) − Y1 (z)z−6
(1 − z−1 )(1 − 1/3z−1 )
with
1 3/2 1/2
Y1 (z) = = − .
(1 − z−1 )(1 − 1/3z−1 ) 1 − z−1 1 − 13 z−1
Its inverse is - * +n .
3 1 1
y1 ( n ) = − u ( n ).
2 2 3
Im{z}
Im{z}
Im{z}
1/3 1/3
3/2 3/2 3/2
Re{z} Re{z} Re{z}
Figure 4.8 Poles and zeros of the system (left), input signal z-transform (middle), and the
z-transform of the output signal (right).
Thus the system output is

- * +n . > * + ?
3 1 1 3 1 1 n −6
y(n) = − u(n) − − u ( n − 6).
2 2 3 2 2 3
Solution 4.15. The transfer function is obtained from

11 −1 1 −2 3
Y (z)(1 − z + z ) = X (z)(2 − z−1 )
6 2 2
as
2 − 32 z−1
H (z) = 11 −1
.
1− 6 z + 12 z−2
The poles are at z p1 = 1/3 and z p2 = 3/2 with the region of convergence
|z| > 3/2. It means that the system is not stable, Fig.4.8.
The z-transform of the input signal is
3
X (z) = 1 − z−1 for |z| > 0.
2
The output signal transform is
* +
2 − 32 z−1 3 2 − 32 z−1
Y (z) = 11 −1
1 − z −1 = .
1− 6 z + 12 z−2 2 1 − 13 z−1
The output signal transform does not have a pole z = 3/2 since this pole is
canceled out. The output signal is
1 3 1
y(n) = u(n) − u ( n − 1).
3 n 2 3n −1
208 z-Transform
Solution 4.16. The z-transform of signal x (n + 2) is
X2 (z) = z2 X (z) − z2 x (0) − zx (1)
while for x (n + 1) the transform is
X1 (z) = zX (z) − zx (0).
The z-transform domain form is
z2 X (z) − z2 x (0) − zx (1) + 3zX (z) − 3zx (0) + 2X (z) = 0
with
z 1 1
X (z) = = − .
z2 + 3z + 2 1 + z −1 1 + 2z−1
The inverse z-transform of X (z) is
x (n) = [(−1)n − (−2)n ]u(n).
Solution 4.17. The z-transforms of the left and right side of the equation are
z
zX (z) − zx (0) = X (z) +
z−a
- .
z 1 1 a
X (z) = = − .
(z − a)(z − 1) 1 − a z − 1 z − a
The inverse z-transform is
1 1 − an
x (n) = [u(n − 1) − an u(n − 1)] = u ( n − 1)
1−a 1−a
or
n −1
x (n) = ∑ ak , n > 0.
k =0
Solution 4.18. For a direct solution in the discrete-time domain we assume

a solution of the homogenous part of the equation
√
2 1
y(n) − y ( n − 1) + y ( n − 2) = 0 (4.22)
2 4
in the form yi (n) = Ci λin . The characteristic polynomial is
√
2 2 1
λ − λ+ =0
2 4
√ √
2 2
with λ1,2 = 4 ±j 4 . The homogenous solution is
√ √ √ √
2 2 n 2 2 n
yh (n) = C1 ( +j ) + C2 ( −j )
4 4 4 4
1 1
= C1 n e jnπ/4 + C2 n e− jnπ/4 .
2 2
A particular solution is of the input signal x (n) = 31n u(n) form. It is y p (n) =
A 31n u(n). The constant A is obtained by replacing this signal into (4.20)
√
1 2 1 1 1 1
A n− A + A n −2 = n
3 2 3n −1 4 3 3
√
3 2 9
A (1 − + ) = 1.
2 4
Its value is A = 0.886. The general solution is
1 jnπ/4 1 1
y(n) = yh (n) + y p (n) = C1 n
e + C2 n e− jnπ/4 + 0.886 n .
2 2 3
Since the system is causal with y(n) = 0 for n < 0 then the constants C1
and
√
C2 may be obtained from the initial condition following from
√
y(n) −
2 2
2 y(n − 1) + 14 y(n − 2) = x (n) as y(0) = x (0) = 1 and y(1) = 2 y (0 ) +
√
x (1) = 22 + 13 ,
C1 + C2 + 0.886 = 1 (4.23)
√ √ √ √ √
2 2 2 2 1 2 1
C1 ( +j )/2 + C2 ( −j )/2 + 0.886 = + ,
2 2 2 2 3 2 3
as C1 = 0.057 − j0.9967 = 0.9984 exp(− j1.5137) = C2∗ . The final solution is
1 1
y(n) = 2 × 0.9984 cos(nπ/4 − 1.5137) + 0.886 n .
2n 3
For the z-domain we write
√
2 1
Y (z) − Y ( z ) z −1 + Y ( z ) z −2 = X ( z )
2 4
with
1 1
Y (z) = √
1− 2 −1
+ 1 −2 1 − 13 z−1
2 z 4z
210 z-Transform
with
z3
Y (z) = √ √ √ √
2 2 2 2 1
(z − ( 4 +j 4 ))( z − ( 4 −j 4 ))( z − 3 )
Using, for example, the residual value based inversion of the z-transform,
M N
n −1
y (n) = ∑ [
√
z
√
Y ( z )( z − z )]
i | z = zi
2 2
z1,2,3 = 4 ± j 4 ,1/3
' '
' '
1 ' '1
= z n +2 √ √ '
' + z n +2 √ √ '
'
2− j 2 1 '√ √ 2+ j 2 1 ' √ √
(z − 4 )(z − 3 ) 2+ j 2 (z − 4 )(z − 3 ) z= 2− j 2
4 4
'
'
1 '
+ z n +2 √ √ √ √ '
'
(z − 2+4 j 2 )(z − 2−4 j 2 ) 'z=1/3
(√ √ ) n +2 (√ √ ) n +2
1 2+j 2 1 1 2−j 2 1
= √ √ √ − √ √ √
j 2 4 2+ j 2
− 1 j 2 4 2− j 2
− 1
2 4 3 2 4 3
1 1
+ √
3n +2 ( 1 − 1 2 + 1 )
9 3 2 4
√ √
1 −j 2 1 j 2 1
= e j(n+2)π/4 √ √ + e− j(n+2)π/4 √ √ + 0.886
2n +2 2+ j 2
− 13 2n +2 − 13 2− j 2 3n
4 4
√ √
1 2 1 2 1
= n e jnπ/4 √ √ 4
+ n e− jnπ/4 √ √ 4
+ 0.886 n
2 2+j 2− 3 2 2−j 2− 3 3
1 1
= 2 × 0.9984 n cos(nπ/4 − 1.5137) + 0.886 n ,
2 3
for n ≥ 1. For n = 0 there is no additional pole at z = 0 the previous result
holds for n ≥ 0.
Solution 4.19. The z-transform of the first backward difference is
Z [∇ x (n)] = Z [ x (n)] − Z [ x (n − 1)] = (1 − z−1 ) X (z).
The second backward difference may be written as
∇2 x (n) = ∇[∇ x (n)] = ∇[ x (n) − x (n − 1)] = ∇ x (n) − ∇ x (n − 1)

= x (n) − 2x (n − 1) + x (n − 2).
Its z-transform is
Z [∇2 x (n)] = (1 − z−1 )2 X (z).
In the same way we get
Z [∇m x (n)] = (1 − z−1 )m X (z).
The z-transform of the first forward difference is
Z [∆x (n)] = Z [ x (n + 1) − x (n)] = zX (z) − zx (0) − X (z)

= (z − 1) X (z) − zx (0).
The second forward difference is
Z [∆2 x (n)] = x (n + 2) − 2x (n + 1) + x (n)
with the z-transform
Z [∆2 x (n)] = ( z − 1)2 X (z) − z(z − 1) x (0) − z∆x (0).
In a recursive way, the z-transform of the mth forward difference is
m −1
Z [∆m x (n)] = ( z − 1)m X (z) − z ∑ ( z − 1 ) m − j −1 ∆ j x (0 ).
j =0
Solution 4.20. The transfer function of this system is

√ −1 √ √ √ √
1− 2z + z−2 [1 − ( 22 + j 22 )z−1 ][1 − ( 22 − j 22 )z−1 ]
H (z) = √ = √ √ √ √
1 − r 2z−1 + r2 z−2 [1 − r ( 22 + j 22 )z−1 ][1 − r ( 22 − j 22 )z−1 ]
√ √ √ √
2 2 2 2
[z − ( +j )][ z − ( − j 2√ )]
= √2 √2 2√
[z − r ( 22 + j 22 )][z − r ( 22 − j 22 )]
√ √ √ √
The zeros and poles are z01,02 = 22 ± j 22 and z p1,p2 = r 2
2
± jr 2
2 .They are
located as in Fig.4.9.
The amplitude of the frequency response is
' '
' B ' TO1 TO2 TO1 TO2
| H (e jω )| = '' 0 '' = .
A0 TP1 TP2 TP1 TP2
The values of TP1 and TO1 , and TP2 and TO2 , are almost the same for
any ω except ω = ±π/4 where the distance to the transfer function zero is
212 z-Transform
O1
1.5
T
P
1
|H(ejω)|
Im{z}
P
2
0.5
O
2
0
Re{z} -2 - π/4 0 π/4 2 ω
Figure 4.9 Location of zeros and poles for a second order system.
0, while the distance to the corresponding pole is small but finite. Based on
this analysis the amplitude of frequency response is presented in Fig.4.9.
The input discrete-time signal is
x (n) = x (n∆t)n∆t = [2 cos(πn/6) − sin(πn/4) + 0.5e jπn/3 ]/60.
This system will filter out signal components at ω = ±π/4. The output
discrete-time signal is
y(n) = [2 cos(nπ/6) + 0.5e jnπ/3 ]/60.
Corresponding continuous-time output signal is
y(t) = 2 cos(10πt) + 0.5e j20πt .
Solution 4.21. The zeros of the system are
z−
o
N
= 1 = e− j2πm
zom = e j2πm/N , m = 0, 1, ..., N − 1
Similarly, the poles are zmp = r1/N e j2πm/N , m = 0, 1, ..., N − 1. The frequency
response of the comb filter is
N −1 N −1
z − zom z − e j2πm/N
H (z) = ∏ = ∏ 1/N e j2πm/N
.
m=0 z − z pm m =0 z − r
With r = 0.9999 and r1/N ∼

= 1 follows
| H (e jω )| ∼
= 1 for z ̸= e j2πm/N
| H (e jω )| = 0 for z = e j2πm/N .
The same holds for
(1 − z−1 )(1 + z−1 ) N/2−1 1 − 2 cos(2kπ/N )z−1 + z−2

(1 − rz−1 )(1 + rz−1 ) k∏
H (z) = −1 + r 2 z −2
=1 1 − 2r cos(2kπ/N )z
since for 1 ≤ k ≤ N/2 − 1 we can group the terms
(1 − e2kπ/N z−1 )(1 − e2( N −k)π/N z−1 ) 1 − 2 cos(2kπ/N )z−1 + z−2

= .
(1 − re2kπ/N z−1 )(1 − re2( N −k)π/N z−1 ) 1 − 2r cos(2kπ/N )z−1 + r2 z−2
4.9 EXERCISE
Exercise 4.1. Find the z-transform and the region of convergence for the
following signals:
(a) x (n) = δ(n − 3) − δ(n + 3),
(b) x (n) = u(n) − u(n − 20) + 3δ(n),
(c) x (n) = 1/3|n| + 1/2n u(n),
(d) x (n) = 3n u(−n) + 2−n u(n),
(e) x (n) = n(1/3)n u(n).
(f) x (n) = cos(n π2 ).
Exercise 4.2. Find the z-transform and the region of convergence for the
signals:
(a) x (n) = 3n u(n) − (−2)n u(n) + n2 u(n).
(b) x (n) = ∑nk=0 2k 3n−k ,
(c) x (n) = ∑nk=0 3k .
Exercise 4.3. Find the inverse z-transform of:
−8
(a) X (z) = 1z−z + 3, if X (z) is the z-transform of a causal signal x (n).
(b) X (z) = (zz−+22)z2 , if X (z) is the z-transform of a causal signal x (n).
2+3z−2 , if X (z ) is the z-transform of an unlimited-duration
(c) X (z) = 6z
6z2 −5z+1
signal x (n). Find ∑∞ n=−∞ x ( n ) in this case.
Exercise 4.4. Find the inverse z-transforms of:

z5 (5z−3)
(a) X (z) = (3z−1)(2z−4) , if x (n) is causal,
214 z-Transform
(b) Y (z) = X ( 2z ), for a causal signal y(n),

(c) Y (z) = z−2 X (z), for a causal signal y(n).
Exercise 4.5. Find the inverse z-transforms of X (z) = cosh( az) and X (z) =
sinh( az).
Exercise 4.6. If X (z) is the z-transform of a signal x (n), with the region of
convergence |z| > 12 , find the z-transforms for the following signals:
(a) y(n) = x (n) − x (n − 1),
∞
(b) y(n) = ∑ x (n − kN ), where N is an integer,
k=−∞
(c) y(n) = x (n) ∗ x (−n), where ∗ denotes convolution.
d
(d) find the signal whose z-transform is Y (z) = dz X ( z ).
Exercise 4.7. If X (z) is the z-transform of a signal x (n) find the z-transform
of
∞
y(n) = ∑ x ∗ ( n − k ) x ( n + k ).
k=−∞
Exercise 4.8. For the z-transform

(2 − z )
H (z) =
(1 − 4z)(1 − 3z)
identify possible regions of convergence and find the inverse z-transform

for each of them. For each case comment stability and causality. What is the
output of the stable system to x (n) = 1 + (−1)n ?
Exercise 4.9. Find the output of a causal discrete system
3 1
y ( n ) − y ( n − 1) + y ( n − 2 ) = x ( n ). (4.24)
4 8
to the input signal x (n) = nu(n) by:
(a) a direct solution in the time domain.
(b) using the z-transform.
The initial conditions are y(n) = 0 for n < 0, that is y(0) = x (0) = 0 and
y(1) = 3y(0)/4 + x (1) = 1.
Exercise 4.10. A causal discrete system is described by the difference equa-
tion
5 1
y ( n ) − y ( n − 1) + y ( n − 2 ) = x ( n ). (4.25)
6 6
If the input signal is x (n) = 1/4n u(n) find the output signal if the initial
value of the output was y(0) = 2.
Hint: Since y(0) does not follow from (4.25) obviously the system
output was "preloaded" before the input is applied. This fact can be taken
into account by changing the input signal at n = 0 to produce the initial
output. It is x (n) = 1/4n u(n) + δ(n). Now the initial conditions are y(0) = 2
and y(1) = 5/3 + 1/4 = 23/12 and we can apply the z-transform with this
new input signal.
Exercise 4.11. Solve the difference equation using the z-transform
1
x ( n + 2) − x ( n + 1) + x ( n ) = 0
2
with initial condition x (0) = 0 and x (1) = 1/2. The signal x (n) is causal.
Exercise 4.12. Using the basic trigonometric transformations show that a

real-valued signal y(n) = cos(2πk0 n/N + ϕ) is a solution of the homoge-
neous difference equation
y(n) − 2 cos(2πk0 /N )y(n − 1) + y(n − 2) = 0.
with similar conclusions as in the complex-valued signal case.
Exercise 4.13. For the system
(1 − z−1 )(1 + z−1 ) 3 1 − 2 cos(2kπ/8)z−1 + z−2

(1 − rz−1 )(1 + rz−1 ) k∏
H (z) = −1 + z −2
=1 1 − 2r cos(2kπ/8)z
and r = 0.9999 plot the amplitude of the frequency response and find the
output to the signal
x (n) = cos(nπ/3 + π/4) + sin(nπ/2) + (−1)n .

216 z-Transform
Chapter 5
From Continuous to Discrete Systems
of continuous-time systems into corresponding
T
RANSFORMATION
discrete-time systems is of high importance. Some discrete-time sys-
tems are designed and realized in order to replace or perform as
equivalents of continuous-time systems. It is quite common to design a
continuous-time system with desired properties, since the designing pro-
cedures in this domain are simpler and well developed. In the next step
the obtained continuous-time system is transformed into an appropriate
discrete-time system.
Consider an Nth order linear continuous-time system described by a
differential equation with constant coefficients
d N y(t) dy(t) d M x (t) dx (t)

aN + ... + a1 + a0 y ( t ) = b M + ... + b1 + b0 x (t).
dt N dt dtn dt
The Laplace transform domain equation for this system is
[ a N s N + ... + a1 s + a0 ]Y (s) = [b M s M + ... + b1 s + b0 ] X (s),

assuming zero-valued initial conditions. The topic of this chapter is to find
a corresponding discrete-time system, described by
A0 y(n) + A1 y(n − 1) + ... + A N y(n − N )
= B0 x (n) + B1 x (n − 1) + ... + B M x (n − M).
The z-transform domain form of this system is
[ A0 + A1 z−1 + ... + A N z− N ]Y (z) = [ B0 + B1 z−1 + ... + B M z− M ] X (z).

There are several approaches to establish a relation between continuous-
time and discrete-time systems represented by their impulse responses or
transfer functions.
217
218 From Continuous to Discrete Systems
h(t) h(n) = h(t) Δt

t = nΔt
Δt t n
Figure 5.1 Sampling of the impulse response for the impulse invariance method.
5.1 IMPULSE INVARIANCE METHOD
A natural approach to transform a continuous-time system into a discrete-

time system is based on the relation between the impulse responses of these
systems. Assume that the impulse response of the continuous-time system is
hc (t). The impulse response h(n) of the corresponding discrete-time system,
according to this approach, is equal to the samples of hc (t),
h(n) = hc (n∆t)∆t.
Obviously this relation can be used only if the sampling theorem is satisfied
for the sampling interval ∆t. It means that the frequency response of the
continuous-time system satisfies the condition
H (Ω) = FT{hc (t)} = 0

for |Ω| > Ωm
and ∆t < π/Ωm . Otherwise the discrete-time version will not correspond to
the continuous-time version of the frequency response. Here, the discrete-
time system frequency response is related to a periodically extended form
of the continuous-time system frequency response H (Ω) as
∞
∑ H (Ω + 2kπ/∆t) = H (e jω ), Ω = ω/∆t.
k=−∞
Transfer function of the continuous-time system may be written as
a N s N + ... + a1 s + a0 k1 k2 kM
H (s) = = + + ··· + , (5.1)
b M s M + ... + b1 s + b0 s − s1 s − s2 s − sM
where only simple poles of the transfer function are assumed. The case of
multiple poles will be discussed later. The inverse Laplace transform of a
causal system, described by the previous transfer function, is
h c ( t ) = k 1 e s1 t u ( t ) + k 2 e s2 t u ( t ) + · · · + k M e s M t u ( t ).
The impulse response of the corresponding discrete-time system is equal to

the the samples of hc (t),
h(n) = hc (n∆t)∆t = [k1 ∆tes1 n∆t u(n) + k2 ∆tes2 n∆t u(n) + ... + k M ∆tes M n∆t u(n)],
since u(n∆t) = u(n). The z-transform of the impulse response h(n) of the
discrete-time system is
k1 ∆t k2 ∆t k M ∆t
H (z) = −
+ −
+ ··· + . (5.2)
1−e s 1 ∆t z 1 1−e s 2 ∆t z 1 1 − es M ∆t z−1
By comparing (5.1) and (5.2) it can be concluded that the terms in the
transfer functions are transformed from the continuous-time to the discrete-
time case as
ki k i ∆t
→ . (5.3)
s − si 1 − esi ∆t z−1
If a multiple pole, of an (m + 1)th order, exists in the continuous-time
system transfer function then it holds
ki 1 dm k i
= .
( s − si ) m + 1 m! dsim s − si
A term in the discrete-time system, corresponding to this continuous-time

system term, is
! 6
1 dm k i 1 dm k i ∆t
m → . (5.4)
m! dsi s − si m! dsim 1 − esi ∆t z−1
In the impulse invariance method the poles are mapped according to
si → esi ∆t .
This mapping relation does not hold for zeros, Fig.5.2.

In the case when the continuous-time impulse response hc (t) has a
discontinuity at t = 0, i.e., when hc (t)|t=−0 ̸= hc (t)|t=+0 then the previous
s=jΩ
j2π/Δt
jπ/Δt jω
z=e
Im{z}
Im{s}
-j π/Δt
-j2 π/Δt
Re{s} Re{z}
Figure 5.2 Illustration of the impulse invariance method mapping.
forms assume that the discrete-time impulse response h(n) = hc (t)|t=+0 . Re-
mind that the theory of Fourier transforms in this case states that the inverse
Fourier transform IFT P { H ( jΩ)} = hc (t) where
Q the signal hc (t) is continuous
and IFT{ H ( jΩ)} = hc (t)|t=−0 + hc (t)|t=+0 /2 at the discontinuity points,
in this case at t = 0. The special case of discontinuity at t = 0 can be easily
detected by mapping H (s) into H (z) and by checking, for a causal system,
is the following relation satisfied
0 = hc (t)|t=−0 = hc (t)|t=+0 = h(n)|n=0 = lim H (z).

z→∞
If limz→∞ H (z) ̸= 0 then a discontinuity existed and we should use
h(0) = lim H (z)/2

z→∞
since hc (t)|t=−0 = 0 and hc (t)|t=+0 ∆t = limz→∞ H (z). The resulting fre-

quency response is
H (z) − lim H (z)/2.
z→∞
Example 5.1. A continuous-time system has a transfer function of the form

3
s+ 2
H (s) = .
s2 + 32 s + 1
2
What is the corresponding discrete-time system according to the impulse

invariance method with ∆t = 1?
⋆The transfer function should be written as

3
s+ k1 k
H (s) = 2
= + 21
(s + 1)(s + 12 ) s+1 s+ 2
with
k1 = H (s)(s + 1)|s=−1 = −1,
'
1 ''
k2 = H (s)(s + )' = 2.
2 s=−1/2
Thus, we get
−1 2
H (s) = + .
s+1 s + 12
According to (5.3) the discrete-time system is
−1 2
H (z) = + .
1 − e −1 z −1 1 − e−1/2 z−1
Since limz→∞ H (z) = 1 obviously there is a discontinuity in the impulse
response and the resulting transfer function should be corrected as
−1 2
H (z) = + − 1/2.
1 − e −1 z −1 1 − e−1/2 z−1
Impulse and frequency responses of the systems with uncorrected and cor-
rected discontinuity effect are presented in Fig.5.3.
Example 5.2. A continuous-time system has a transfer function of the form

(1 − 3s/2)
H (s) = .
(6s2 + 5s + 1)(s + 1)2
What is the corresponding discrete-time system according to the impulse

invariance method with ∆t = 1?
⋆The transfer function should be written as
1 − 3s/2
H (s) =
6(s + 12 )(s + 13 )(s + 1)2
k1 k2 k3 k
= + + + 4
s+ 1
2 s+ 1
3
( s + 1)2 s + 1
with
k1 = H (s)(s + 1/2)|s=−1/2 = −7,
k2 = 27/8,
1.5 h (t) 1.5 h (t)

c c
h(n) h(n)
1 1
0.5 0.5
0 0
-5 0 5 10 15 -5 0 5 10 15
4 jω
4 jω
|H(e )| |H(e )|
3 |H(jΩ)| 3 |H(jΩ)|
2 2
1 1
0 0
-2 0 2 -2 0 2
Figure 5.3 Impulse responses of systems in continuous and discrete-time domains (top). Am-
plitude of the frequency response of systems in continuous and discrete-time domains (bot-
tom). System without discontinuity correction (left) and system with discontinuity correction
(right).
'
'
k3 = H (s)(s + 1)2 ' = 5/4.
s=−1
The coefficient k4 follows, for example, from
H (0) = 1 = 2k1 + 3k2 + k3 + k4 ,
as
k4 = 29/8.
Thus, we get
−7 27/8 5/4 29/8
H (s) = 1
+ 1
+ 2
+ .
s+ 2 s+ 3 ( s + 1) s+1
According to (5.3) and (5.4) the discrete-time system is
−7 27/8
H (z) = +
1 − e−1/2 z−1 1 − e−1/3 z−1
'
d 5/4 ' 29/8
+ { } ' +
s
dsi 1 − e i z − 1 ' 1 − e −1 z −1
si =−1
−7z 27z/8 5e−1 z/4 29z/8
= + + + .
z−e − 1/2 z−e − 1/3 ( z − e −1 )2 z − e −1
s=jΩ
jω
z=e
Im{z}
Im{s}
1
-1 2/32/3 1.9894
Re{s} Re{z}
Figure 5.4 Pole-zero locations in the s-domain and the z-domain using the impulse invariance
method.
Since h(0) = limz→∞ H (z) = 0 there no need to consider possible impulse

response correction due to discontinuity.
Writing the transfer function in the form
0.0341z(z − 1.9894)(z + 0.3259)

H (z) = −
(z − 0.7165) (z − 0.6065) (z − 0.3679)2
we can easily see that the poles are mapped according to s pi → es pi ∆t , Fig.5.4,
while there is no direct correspondence among zeros of the transfer functions.
Impulse responses of continuous-time system and discrete-time system are
presented in Fig.5.5.
5.2 MATCHED Z-TRANSFORM METHOD
The matched z-transform method is based on a discrete-time approximation

of the Laplace transform derived in the previous chapter as
"∞ ∞
X (s) = x (t)e−st dt ∼
= ∑ x (n)e−sn∆t = X (z)|z=es∆t .
−∞ n=−∞
This approximation leads to a relation between the Laplace domain and the
z-domain in the form of
z = es∆t .
0.3
h (t), h(n)
c
0.2
0.1
-0.1
0 5 10 15 20 25 30 35 40
jω
|H(jΩ)|, |H(e )|
1
0.5
0
-3 -2 -1 0 1 2 3
1
10
20log|H(jΩ)|
jω
0 20log|H(e )|
10
-1
10
-2
10
-3 -2 -1 0 1 2 3
Figure 5.5 Impulse responses of systems in continuous and discrete-time domains (top).
Amplitude of the frequency response of systems in continuous and discrete-time domains
(middle). Amplitude of the frequency response of systems in continuous and discrete-time
domains in logarithmic scale (bottom).
If we use this relation to map all zeros and poles of a continuous system
transfer function
b M s M + ... + b1 s + b0 b (s − s01 )(s − s02 )...(s − s0M )

H (s) = N
= M
a N s + ... + a1 s + a0 a N (s − s p1 )(s − s p2 )...(s − s pN )
into the corresponding z-plane locations
z0i = es0i ∆t
z pi = es pi ∆t ,
s=jΩ
j2π/Δt
jπ/Δt jω
z=e
Im{z}
Im{s}
-j π/Δt
-j2 π/Δt
Re{s} Re{z}
Figure 5.6 Illustration of the zeros and poles mapping in the matched z−transform method.
the matched z-transform method of the system follows. The discrete-time

system transfer function is
(z − es01 ∆t )(z − es02 ∆t )...(z − es0M ∆t )

H (z) = C .
(z − es p1 ∆t )(z − es p2 ∆t )...(z − es pN ∆t )
Constant C follows from the amplitude condition. For example, it can be

calculated from H (s)|s=0 = H (z)|z=1 .
Example 5.3. For the continuous-time system with a transfer function of the form
1−s
H (s) =
8s2 + 6s + 1
find the corresponding discrete-time system according to the matched z-
transform method and ∆t = 1?
⋆The transfer function of discrete-time system is obtained from
1−s
H (s) = ,
8(s + 12 )(s + 14 )
using the mapping z0i = es0i ∆t and z pi = es pi ∆t , as

z−e
H (z) = k .
8(z − e−1/2 )(z − e−1/4 )
Since H (s)|s=0 = 1 if we want that H (z)|z=e j0 = 1 then k = −1/2.4678 =

−0.4052.
5.3 DIFFERENTIATION AND INTEGRATION
The first-order backward difference is a common method to approximate

the first-order derivative of a continuous-time signal
dx (t)
y(t) =
dt
∼ n∆t) − x ((n − 1)∆t) .
y(n∆t) =
x (
∆t
The Laplace transform domain of the continuous-time first derivative is
Y (s) = sX (s). (5.5)
In the discrete-time domain, with y(n) = y(n∆t)∆t and x (n) = x (n∆t)∆t,

this derivative approximation results in the first-order linear difference
equation
x ( n ) − x ( n − 1)
y(n) = .
∆t
In the z-transform domain this equation is
1 − z −1
Y (z) = X ( z ). (5.6)
∆t
Based on (5.5) and (5.6) we can conclude that a mapping of the correspond-
ing differentiation operators from the continuous-time to the discrete-time
domain is
1 − z −1
s= . (5.7)
∆t
With a normalized discretization step ∆t = 1 this mapping is of the form
s = 1 − z −1 .
The same result could be obtained by considering a rectangular rule

approximation of a continuous-time integral
"t −∆t
t"
y(t) = x (t)dt ∼
= x (t)dt + x (n∆t)∆t.
−∞ −∞
At an instant t = n∆t, the value of integral can be approximated as
y(n∆t) ∼
= y(n∆t − ∆t) + x (n∆t)∆t.
In the discrete-time domain this relation reads
y(n) = y(n − 1) + x (n)∆t.
The Laplace and the z-transform domain forms of the previous integral
equations are
1
Y (s) = X (s)
s
∆t
Y (z) = X ( z ).
1 − z −1
The same mapping of the z-plane to the s-plane as in (5.7) follows.

Consider the imaginary axis from the s-plane (the Fourier transform
line). According to (5.7) the mapping, with ∆t = 1, is defined by
1 − s → z −1 . (5.8)
Now we will consider the region that corresponds to the imaginary axis and
the left semi-plane of the s-domain (containing poles of a stable system),
Fig.5.7(left). The aim is to find the corresponding region in the z-domain.
If we start from the s-domain and the region in Fig.5.7(left), the first
mapping is to reverse the s-domain to −s and shift it for +1, as
1 − s → p.
The corresponding domain, after this mapping, is shown in Fig.5.7(middle).

The next step is to map the region from p-domain into the z-domain,
according to (5.8), as
p → z −1 .
By denoting Re{z} = x and Im{z} = y we get that the line Re { p} = 1
in the p−domain, corresponding to the imaginary axis in the s-plane, is
transformed into the z-domain according to
1
Re{ p} = Re{ }
z
1
1 = Re{ }
x + jy
1 x − jy
1 = Re{ }
x + jy x − jy
s=0+jΩ p=1
z=ejω
Im{p}
Im{z}
Im{s}
-1
1-s → p p→ z
Re{s} Re{p} Re{z}
Figure 5.7 Illustration of the differentiation based mapping of the left s−semi-plane with the
imaginary axis (left), translated and reversed p−domain (middle), and the z−domain (right).
resulting in
x
1=
x2 + y2
or in * +2
1 1
( x − )2 + y2 = . (5.9)
2 2
Therefore, the imaginary axis in the s-plane is mapped onto a circle defined
by (5.9), Fig.5.7(right) in the z-plane. From the mapping relation 1 − s → z−1
it is easy to conclude that the origin s = 0 + j0 maps into z = 1 and that
s = 0 ± j∞ maps into z = ±0, according to 1/ (1 − s) → z.
Mapping of the imaginary axis into z-domain can also be analyzed
from
1 − (re jω )−1 1 − r −1 cos ω r −1
σ + jΩ → = +j sin ω.
∆t ∆t ∆t
For σ = 0 follows
1 − r −1 cos ω = 0 (5.10)
r = cos ω,
with
r −1 tan ω
Ω=sin ω = .
∆t ∆t
Obviously ω = 0 maps to Ω = 0 (with Ω ∼ = ω/∆t for small ω), and ω =
±π/2 maps into Ω → ±∞. Thus, the whole imaginary axis maps onto
−π/2 ≤ ω ≤ π/2. These values of ω could be used within the basic period.
Relation (5.10), with −π/2 ≤ ω ≤ π/2, is a circle defined by (5.9) if we
, ,
replace r = x2 + y2 and cos ω = x/ x2 + y2 with σ < 0 (semi-plane with
negative real values) being mapped into r < cos ω (interior of unit circle).
Example 5.4. A continuous-time system is described by a differential equation
3 1
y′′ (t) + y′ (t) + y(t) = x (t),
4 8
with zero initial conditions and the transfer function
1
H (s) = .
s + 4 s + 18
2 3
What is the corresponding transfer function of a discrete-time system using

the first-order backward difference approximation with ∆t = 1/2? What is
the solution of the differential equation for x (t) = u(t). Compare it with the
solution of difference equation y(n) with ∆t = 1/8.
P ⋆A discrete-time
Q system transfer function is obtained by replacing
s = 1 − z−1 /∆t in H (s) as
1
H (z) = B C2
1 − z −1 3 1 − z −1 1
∆t + 4 ∆t + 8
(∆t)2
= 2
1 + 34 ∆t + 18 (∆t) − [2 + 34 ∆t]z−1 + z−2
with
y(n) = B0 x (n) + A1 y(n − 1) + A2 y(n − 2)
(∆t)2
B0 = = 0.1778
1 + 34 ∆t + 18 (∆t)2
[2 + 34 ∆t]
A1 = 3 1 2
= 1.6889
1+ 4 ∆t + 8 (∆t )
1
A2 = − = −0.7111,
1 + 34 ∆t + 18 (∆t)2
where ∆t = 1/2. For x (t) = u(t) in the continuous-time case
1
Y (s) = H (s) X (s) =
s(s2 + 34 s + 18 )
8 8 16
= + 1
− 1
s s+ 2 s+ 4
with
y(t) = [8 + 8e−t/2 − 16e−t/4 ]u(t).
The results of the difference equation for y(n) are compared with the exact
solution y(t) in Fig.5.8. The agreement is high. It could be additionally
improved by reducing the sampling interval, for example, to ∆t = 1/8.
10
y(t), y(n)
0 5 10 15
Figure 5.8 Exact solution of the difference equation y(t) in solid line and the discrete-time
system output y(n) in large dots for ∆t = 1/2 and in small dots for ∆t = 1/8..
5.4 BILINEAR TRANSFORM
In the case of a differentiator based mapping the imaginary axis in the

s−domain, corresponding to the Fourier transform values, has been mapped
onto a circle with radius 1/2 and the center at z = 1/2 in the z−domain. It
does not correspond to the Fourier transform of discrete-time signals posi-
tion in the z−plane, that is along |z| = 1. A transformation that will map the
imaginary axis from the s−domain onto the unit circle in the z−domain is
presented next.
Consider numerical integration by using the trapezoid rule
"t −∆t
t"
x (n∆t) + x ((n − 1)∆t)
y(t) = x (t)dt ∼
= x (t)dt + ∆t
2
−∞ −∞
x ( n ) + x ( n − 1)
y ( n ) = y ( n − 1) + ∆t.
2
In the Laplace and the z-transform domain, these relations have the forms
1
Y (s) = X (s)
s
∆t 1 + z−1
Y (z) = X ( z ).
2 1 − z −1
Mapping from the s−domain to the z−domain follows as
2 1 − z −1
s→ . (5.11)
∆t 1 + z−1
In the complex analysis this mapping is known as a bilinear transform.

Within the derivatives framework the bilinear transform can be under-
stood as the following derivative approximation. Consider the first-order
backward derivative approximation
y ( n ) = x ( n ) − x ( n − 1 ).
The same signal samples can used for the first-order forward derivative
approximation
y ( n − 1 ) = x ( n ) − x ( n − 1 ).
If we assume that the difference x (n) − x (n − 1) fits better to the mean
of y(n) and y(n − 1) than to any single one of them, then the derivative
approximation by using the difference equation
y ( n ) + y ( n − 1)
= x ( n ) − x ( n − 1 ),
2
produces the bilinear transform.
In order to prove that the imaginary axis in the s−domain corresponds
to the unit circle in the z−domain we may simply replace z = e jω into (5.11)
and obtain
1 − e− jω e jω/2 − e− jω/2 ω
2 −
= 2 jω/2 = 2j tan( ) → s∆t.
1+e jω e + e− jω/2 2
For s = σ + jΩ follows
σ=0
2 ω
Ω= tan( ).
∆t 2
Therefore, the unit circle z = e jω maps onto the imaginary axis σ = 0.

The frequency points ω = 0 and ω = ±π map into Ω = 0 and Ω → ±∞,
respectively. The linearity of frequency mapping Ω → ω is lost. It holds for
small values of ω only
2 ω ω
Ω= tan( ) ∼
= , for |ω | ≪ 1.
∆t 2 ∆t
From
s∆t
1+ 2
z= s∆t
1− 2
F
(1 + σ∆t 2
2 ) + ( Ω∆t
2 )
2
|z| = F
(1 − σ∆t 2
2 ) + ( Ω∆t
2 )
2
it may easily be concluded that σ < 0 maps into |z| < 1, since 1 + σ∆t
2 <
σ∆t
1 − 2 for σ < 0.
The bilinear transform mapping can be derived by using a series of
complex plane mappings. Since
s∆t
1+ 2 2
z= s∆t
= − 1,
1− 2 1 − s∆t
2
we can write
s∆t
1− → p1 ,
2
1
→ p2 ,
p1
2p2 − 1 → z.
This series of mappings from the s-domain to the z-domain is illustrated in

Fig.5.9, with ∆t = 1. The fact that p1 → p2 maps the line Re{ p1 } = 1 into the
1
B C2
1 2 2 1
circle ( x − 2 ) + y = 2 in p2 -domain is proven in the previous section.
Since the bilinear transform introduces a nonlinear transformation
of the frequency axis from the continuous-time domain to the discrete-
time domain, Ω = ∆t 2
tan( ω2 ), in system design it is necessary to take into
account this nonlinearity. Usually it is done by pre-modifying the projected
important frequency values from the analog domain Ωc → ∆t 2
tan( ω2d ), and
ωd → Ωd ∆t. The frequencies Ωd will, after the bilinear transformation, be
returned to the desired values Ωc = ωc /∆t.
Example 5.5. A continuous-time system
2QΩ1 2QΩ2
H (s) = + 2
s2 + 2Ω1 Qs + Ω21 + Q2 s + 2Ω2 Qs + Ω22 + Q2
is designed to pass the signal
x (t) = A1 cos(Ω1 t + ϕ1 ) + A2 cos(Ω2 t + ϕ2 ).
s=0+jΩ p=1
Im{p1}
Im{s}
-1
p1 → p2
Re{s} Re{p1}
jω
z=e
Im{p }
Im{z}
1 1
2
Re{p2} 2p2-1 → z
Re{z}
Figure 5.9 Bilinear mapping illustration trough a series of elementary complex plane map-
pings.
and to stop all other possible signal components. The parameters are Q =
0.01, Ω1 = π/4, and Ω2 = 3π/5. The signal is sampled with ∆t = 1 and
the discrete-time signal x (n) is formed. Using the bilinear transform, design
the discrete system that corresponds to the continuous-time system with the
transfer function H (s).
⋆For the beginning just use the bilinear transform relation
1 − z −1
s→2 (5.12)
1 + z −1
and map H (s) to HB (z) without any pre-modification. The result is presented
in the first two subplots of Fig.5.10. The discrete frequencies are shifted since
the bilinear transform (5.12) made a nonlinear frequency mapping from the
continuous-time to discrete-time domain, according to

ω
Ω = 2 tan( ).
2
Thus, obviously, the system HB (z) is not a system that will filter the corre-
sponding frequencies in x (n) in the same way as H (s) filters x (t).
In order to correct the shift introduced by the bilinear transform map-
ping the continuous-time system should be pre-modified as
2QΩ1d 2QΩ2d
Hd (s) = 2
+ 2
s2 + 2Ω1d Qs + Ω1d + Q 2 s + 2Ω2d Qs + Ω22d + Q2
with
2 Ω ∆t
Ω1d = tan( 1 ) = 0.8284 = 0.2637π
∆t 2
2 Ω2 ∆t
Ω2d = tan( ) = 2.7528 = 0.8762π.
∆t 2
We see that the shift of Ω1 = 0.25π to Ω1d = 0.2637π is small since the bilin-
ear transform frequency mapping for small frequency values is almost linear.
However for Ω2 = 0.6π the shift to Ω2d = 0.8762π is significant due to a high
nonlinearity of mapping in that region. The modified system Hd (s) is pre-
sented in subplot 3 of Fig.5.10. Next, by using the bilinear transform mapping
−z−1 the modified frequencies will map to the desired ones ω = Ω ∆t
s → 2 11+ z −1 1 1
and ω2 = Ω2 ∆t. The obtained discrete-time system transfer function
2QΩ1d
H (z) = B C2 +
1 − z −1 −1
− z + Ω2 + Q2
2 1 + z −1
+ 4Ω1d Q 11+ z −1 1d
2QΩ2d
+B C2
− z −1
2 11+ + 4Ω2d Q 11−
−1
z
+ Ω22d + Q2
z −1 + z −1
0.016569
=B C2
1 − z −1 −1
−z + 0.68641
2 1 + z −1
+ 0.0331375 11+ z −1
0.0551
+B C2
1 − z −1 −1
−z + 7.5778
2 1 + z −1
+ 0.1101 11+ z −1
0.016569(1 + z−1 )2
=
4.65327z−2 − 6.6272z−1 + 4.7195
0.0551(1 + z−1 )2
+
11.4677z 2 + 7.1556z−1 + 11.6879
−
0.003567(1 + z−1 )2
= −1
(z − 1.0071e j0.25π )(z−1 − 1.0071e− j0.25π )
0.0048(1 + z−1 )2
+
( z −1 − 1.0096e j0.6π )(z−1 − 1.0096e− j0.6π )
Ω1 Ω2
1
H(s)
s → 2(1-z
0.5
0
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1
-1
)/(1+z )
1
HB(z)
-1
0.5
0
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1
frequency ω/π or ΩΔt/π
modified H(s) Ω1d=2tan(Ω1Δt/2)/Δt Ω =2tan(Ω Δt/2)/Δt

2d 2
1
H (s)
d
s → 2(1-z
0.5
0
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1
-1
)/(1+z )
1
H(z) ω =Ω Δt ω2=Ω2Δt
1 1
-1
0.5
0
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1
frequency ω/π or ΩΔt/π
Figure 5.10 Amplitude of the continuous-time system with transfer function H (s) and the
amplitude of the transfer function HB (z) of the discrete-time system obtained by the bilinear
transform (first two subplots). A premodified system to take into account the frequency map-
ping nonlinearity in the bilinear transform Hd (s) and the amplitude of the transfer function
H (z) of the discrete-time system obtained by the bilinear transform of Hd (s) (last two subplots).
is shown in subplot 4 of Fig.5.10.

This is the desired discrete-time system corresponding to the continuous-
time system in subplot 1 of this figure. In calculations the coefficients are
rounded to four decimal places.
Comparison of the mapping methods presented in this section is

summarized in the next table.
Sampling
Fourier transform
Method theorem
H (s)|s= jΩ → H (z)|z=e jω
condition
Impulse Invariance Yes, Ω = ω/∆t Yes
Matched z-transform No No
First-oder difference No No
tan(ω/2)
Bilinear transform Yes, Ω = ∆t/2 No
5.5 DISCRETE FILTERS DESIGN
5.5.1 Lowpass filters
An ideal discrete lowpass filter is defined by the frequency response

!
jω 1 for |ω | < ωc
H (e ) = .
0 for ωc < |ω | < π
The frequency response is periodic in ω with period 2π.

The implementation of an ideal lowpass filter in the DFT domain is
obvious, by multiplying all DFT coefficients corresponding to ωc < |ω | < π
by zero. In on-line implementations in the discrete-time domain the ideal
filter should be approximated by a corresponding transfer function form
that can be implemented, since the impulse response h(n) = 2 sin(ωc n)/n
of the ideal filter is noncausal signal with a slow convergence.
There are several methods to approximate the ideal lowpass filter
frequency response. One of them is the Butterworth approximation. Some of
commonly used approximations are Chebyshev and elliptic forms as well.
A lowpass filter of the Butterworth type is shown in Fig.5.11, along
with the ideal one.
|H(ejω)|2
|H(ejω)|2
ω ω
-2 π -π - ωc ωc π 2π -2 π -π 0 π 2π
Figure 5.11 Lowpass filter frequency response: ideal case (left) and Butterworth type (right).
Example 5.6. Implement a Butterworth discrete filter of order N = 4 with a critical

frequency corresponding to the continuous domain filter with the critical
frequency f c = 4[kHz] and the sampling interval ∆t = 31.25[µ sec]. Using:
(a) The impulse invariance method and
(b) the bilinear transform.
⋆Note that the discrete-time frequency is ωc = Ωc ∆t = 2π f c ∆t = π/4.
The poles of the fourth order Butterworth filter in the continuous-time
domain (Chapter I, Subsection 1.6) are
@ π π π π A
s0 = Ωc cos( + ) + j sin( + ) = Ωc (−0.3827 + j0.9239)
- 2 8 2 8 .
π 3π π 3π
s1 = Ωc cos( + ) + j sin( + ) = Ωc (−0.9239 + j0.3827)
2 8 2 8
- .
π 5π π 5π
s2 = Ωc cos( + ) + j sin( + ) = Ωc (−0.9239 − j0.3827)
2 8 2 8
- .
π 7π π 7π
s3 = Ωc cos( + ) + j sin( + ) = Ωc (−0.3827 − j0.9239).
2 8 2 8
The transfer function is
Ω4c
H (s) = 2
. (5.13)
( s2 + 0.7654Ωc s + Ωc )(s2 + 1.8478Ωc s + Ω2c )
(a) For the impulse invariance method the transfer function (5.13) should be
written in the form
k0 k1 k2 k3
H (s) = + + + ,
s − s0 s − s1 s − s2 s − s3
with
k i = H (s)(s − si )|s=si
k0 = (−0.3628 + j0.1503)/∆t,
k1 = (0.3628 − j0.8758)/∆t,
k2 = (0.3628 + j0.8758)/∆t,
k3 = (−0.3628 − j0.1503)/∆t.
Using the impulse invariance method we get the transfer function of the
discrete-time fourth order Butterworth filter
k0 ∆t k1 ∆t k2 ∆t k3 ∆t
H (z) = + + +
1 − es0 ∆t z−1 1 − es1 ∆t z−1 1 − es2 ∆t z−1 1 − es3 ∆t z−1
−0.3628 + j0.1503 0.3628 − j0.8758
= +
1 − eωc (−0.3827+ j0.9239) z−1 1 − eωc (−0.9239+ j0.3827) z−1
0.3628 + j0.8758 −0.3628 − j0.1503
+ + .
1 − eωc (−0.9239− j0.3827) z−1 1 − eωc (−0.3827− j0.9239) z−1
It can be seen that the discrete-time filter is a function of ωc . Thus, for a

given continuous domain frequency and sampling interval, it is possible to
calculate ωc = Ωc ∆t and to to use this frequency in the continuous filter
design with normalized ∆t = 1. Replacing the value for ωc = π/4 we get
−0.3628 + j0.1503 −0.3628 − j0.1503
H (z) = +
1 − (0.5539 + j0.4913)z−1 1 − (0.5539 − j0.4913)z−1
0.3628 − j0.8758 0.3628 + j0.8758
+ + .
1 − (0.4623 + j0.1433)z−1 1 − (0.4623 − j0.1433)z−1
Grouping the complex-conjugate terms, a system form with real-valued co-
efficients is obtained
−0.7256 + 0.2542z−1 0.7256 − 0.084z−1
H (z) = − −
+ .
1
1 − 1.1078z + 0.5482z 2 1 − 0.9246z−1 + 0.2343z−2
(b) For the bilinear transform the critical frequency ωc has to be pre-
modified according to
2 ωc 0.8284
Ωd = tan( ) = .
∆t 2 ∆t
Then the frequency Ωd is used for the design in (5.13) instead Ωc . It will be
transformed back to Ωc = ωc /∆t after the bilinear transform is used. Using
2 1 − z −1
substitutions s → ∆t 1 + z −1
and
ωd = Ωd ∆t = 0.8284
in (5.13) the transfer function follows as
H (z) =
ωd 4
z −1 2 − z −1 −1 −1
[4( 11−
+ z −1
) + 2ωd 0.7654 11+ z −1
−z )2 + 2ω 1.8478 1−z + ω 2 ]
+ ωd2 ][4( 11+ z −1 d 1 + z −1 d
0.4710
= z −1 2 − z −1 −1 −1
[4( 11−
+ z −1
) + 0.6863][4( 11−
+ 1.2626 11+ z −1
z
+ z −1
−z + 0.6863]
)2 + 3.0481 11+ z −1
P − 1
Q 4
0.4710 1 + z
=P QP Q
−
3.4237z − 6.6274z + 5.9484 1.6382z−2 − 6.6274z−1 + 7.7704
2 − 1
P Q4
0.084 1 + z−1
P
= −2 Q P Q
z − 1.9357z−1 + 1.7343 z−2 − 4.0455z−1 + 4.7433
0.084z−4 + 0.336z−3 + 0.504z−2 + 0.336z−1 + 0.084
= .
z−4 − 5.9810z−3 + 14.3z−2 − 16.1977z−1 + 8.2263
The transfer function (amplitude and phase) of the continuous-time
filter and the discrete-time filters obtained by using the impulse invariance
method and the bilinear transform are presented in Fig.5.12, within one
1.5
arg{H(e }
1
jω
|H(e |
jω
0
0.5
-π
0 ω ω
-π - π/2 0 π/2 π -π - π/2 0 π/2 π
Figure 5.12 Amplitude and phase of the fourth order Butterworth filter frequency response
obtained by using the impulse invariance method and bilinear transform.
frequency period. Agreement between amplitude and phase functions is

high. The difference equation describing this Butterworth filter is
y(n) = 1.969y(n − 1) − 1.7383y(n − 2) + 0.7271y(n − 3) − 0.1216y(n − 4)
+0.0102x (n) + 0.0408x (n − 1) + 0.0613x (n − 2)
+0.0408x (n − 3) + 0.0102x (n − 4).
In calculations the coefficients are rounded to four decimal places, what
may cause small quantization error (that will be discussed within the next
chapter).
Example 5.7. Design a continuous-time lowpass filter whose parameters are:
- passband frequency Ω p = 2π f p , f p = 3 kHz,
- stopband frequency Ωs = 2π f s , f s = 6 kHz,
- maximal attenuation in the passband a p = −2 dB, and
- minimal attenuation in the stopband as = −15 dB.
Find the corresponding discrete-time filter using the bilinear transform
and ∆t = 0.05 × 10−3 sec.
⋆The maximal attenuation in the passband and the minimal attenua-
tion in the stopband are
a p = 20 log( A p )
A p = 10a p /20 = 0.7943
As = 10as /20 = 0.1778.
The relations for the filter order N and frequency Ωc are (Chapter I, Subsec-
tion 1.6)
1 2 1 2
B C2N ≥ A p , B C2N ≤ As . (5.14)
Ωp Ωs
1+ Ωc 1+ Ωc
Using the equality in both of these relations it follows

1
−1
A2p
ln 1
1 −1
A2s
N= = 2.8551.
2 ln Ω p
Ωs
The first greater integer is assumed for the filter order as

N = 3.
Then we can use any relation in (5.14) with equality sign in order to calculate
' '2
Ωc . For the first one, the value of Ωc will be such to satisfy ' H ( jΩ p )' = A2p .
Then
Ωp
Ωc = F = 2π × 3.2805 kHz,
1
2N
A2p
−1
ωc = Ωc ∆t = 1.0306.
Poles of the Butterworth filter in the continuous domain are
sk = Ωc e j(2πk+π )/6+ jπ/2 , k = 0, 1, 2
- .
2π 2π
s0 = 2π × 3.2805 cos + j sin × 103
3 3
s1 = −2π × 3.2805 × 103
- .
2π 2π
s2 = 2π × 3.2805 cos − j sin × 103 .
3 3
(2π3.2805 × 103 )3
H (s) = .
(s + 2π3.2805 × 103 )(s2 + 2π3.2805 × 103 s + (2π3.2805 × 103 )2 )
In the design we will not use this transfer function.
For the bilinear transform we have to pre-modify the frequency Ωc so
that it will be returned back to the desired value when the bilinear transform
is applied. This frequency is
2 ωc 1.1327
Ωd = tan( ) = 2π × 3.6054 kHz = .
∆t 2 ∆t
The modified transfer function in the continuous domain is
(2π3.6054 × 103 )3
Hd (s) = .
(s + 2π3.6054 × 103 )(s2 + 2π3.6054 × 103 s + (2π3.6054 × 103 )2 )
The discrete-time Butterworth filter transfer function H (z) follows with
2 1 − z −1
s=
∆t 1 + z−1
as
1.13273
H (z) = B C2
1 − z −1 − z −1 −z−1 + 1.13272 )
(2 1 + z −1
+ 1.1327)( 2 11+ z − 1 + 2.2653 11+ z −1
1.4533(1 + z−1 )3
=
(−0.8673z−1 + 3.1327)(3.0177z−2 − 5.434z−1 + 7.54)
−0.5553z−3 − 1.6658z−2 − 1.6658z−1 − 0.5553
=
z−3 − 5.4127z−2 + 9.0028z−1 − 9.0249
0.0615z3 + 0.1846z2 + 0.1846z + 0.0615
= .
z3 − 0.9975z2 + 0.5998z − 0.1108
The corresponding difference equation is
y(n) = 0.9975y(n − 1) − 0.5998y(n − 2) + 0.1108y(n − 3)
+ 0.0615x (n) + 0.1846x (n − 1) + 0.1846x (n − 2) + 0.0615x (n − 3).
Example 5.8. A continuous-time signal

22π 8π π
t) + 4 sin(πt) + 4 cos(
x (t) = 8 cos( t+ )
3 3 4
is sampled with ∆t = 1/4. The discrete-time signal is passed through an ideal
lowpass filter with frequency ωc = π/3. Find the output signal. What is the
corresponding continuous-time output signal?
⋆The discrete-time signal is
11π π 2π π
x (n) = 2 cos( n) + sin( n) + cos( n+ )
6 4 3 4
∞
11π 11π
X (e jω ) = 2π ∑ [δ(ω − + 2kπ ) + δ(ω + + 2kπ )]
k=−∞
6 6
∞
π π π
+ ∑ [δ(ω − 4 + 2kπ ) − δ(ω + 4 + 2kπ )]
j k=− ∞
∞
2π 2π
+π ∑ [δ(ω − + 2kπ )e jπ/4 + δ(ω + + 2kπ )e− jπ/4 ].
k =−∞
3 3
Within the basic period −π ≤ ω ≤ π the Fourier transform value is
11π 11π
X (e jω ) = 2π [δ(ω − + 2π ) + δ(ω + − 2π )]
6 6
π π π
+ [δ(ω − ) − δ(ω + )]
j 4 4
2π jπ/4 2π − jπ/4
+ π [δ(ω − )e + δ(ω + )e ].
3 3
In addition to the last two components that have frequencies corre-

sponding to the analog signal there is the first component
11π 12π 11π 12π
2π [δ(ω − + ) + δ(ω + − )]
6 6 6 6
corresponding to
π
x1 (n) = 2 cos( n ).
6
The lowpass filter output is
π π
y(n) = 2 cos( n) + sin( n).
6 4
It corresponds to the continuous-time signal
π
y(t) = 8 cos( t) + 4 sin(πt).
6
One component at frequency ω = 2π/3 > π/3 is filtered out. The component
at ω = π/4 is unchanged. One more component appeared at ω = π/6 due to
the periodic extension of the Fourier transform of a discrete-time signal.
In general a signal component x (t) = exp( jΩ0 t), Ω0 < 0, with a sam-
pling interval ∆t such that
Kπ ≤ Ω0 ∆t < (K + 1)π
will, after sampling, result into a component within the basic period of the
Fourier transform of discrete-time signal, corresponding to the continuous
K
signal at exp( j(Ω0 t − ∆t πt) This effect is known as aliasing. The most
obvious visual effect is when a wheel rotating with f 0 = 25 [Hz], Ω0 = 50π, is
sampled in a video sequence at ∆t = 1/50 [sec]. Then Ω0 ∆t = π corresponds
to exp( j(Ω0 t − 50πt)) = e j0 , i.e., the wheel looks as a static (nonmoving)
object.
5.5.2 Highpass Filters
Highpass filters can be obtained by transforming corresponding continuous-

time filters into the discrete-time domain. In the discrete-time domain a
highpass filter frequency response is equal to a lowpass filter response
shifted in frequency for π, Fig.5.13,
HH (e jω ) = H (e j(ω −π ) ).
A frequency shift corresponds to the impulse response modulation
h H (n) = e jπn h(n) = (−1)n h(n).

|H (ejω)|2
jω 2
1
|H(e )|
H
ω ω
-2 π -π -ω ω π 2π -2 π -π 0 π 2π
c c
Figure 5.13 Highpass filter as a shifted version of the lowpass filter.
x(n) y(n)
× h(n) ×
n n
(-1) (-1)
Figure 5.14 Highpass filter realization using lowpass filter.
Thus, if we have a lowpass filter, the corresponding highpass filter is ob-

tained by multiplying the impulse response values h(n) by (−1)n . The out-
put of a highpass filter to any input signal x (n) is, Fig.5.14,
∞
y(n) = x (n) ∗n h H (n) = ∑ x (m)(−1)n−m h(n − m)
m=−∞
∞
= (−1)n ∑ (−1)m x (m)h(n − m) = (−1)n × [(−1)n x (n)] ∗n h(n) (5.15)
m=−∞
Example 5.9. For the lowpass Butterworth discrete-time filter

P Q4
0.1236 1 + z−1
H ( z ) = P −2 QP Q
z − 1.9389z−1 + 1.7420 z−2 − 4.0790z−1 + 4.7686
from Fig.5.15 plot the frequency response if z is replaced by −z.
⋆The impulse response is obtained by changing the sign for each
other sample in h(n). In the z-transform definition that means using (−z)−n
instead of z−n . The frequency response of
P Q4
0.1236 1 − z−1
H H ( z ) = P −2 QP Q
z + 1.9389z−1 + 1.7420 z−2 + 4.0790z−1 + 4.7686
is shown in Fig.5.15.
1.5 1.5
1 1
|HH(ejω|
|H(e |
jω
0.5 0.5
0 ω 0 ω
-π - π/2 0 π/2 π -π - π/2 0 π/2 π
Figure 5.15 Amplitude of frequency response of a lowpass Butterworth filter (left) and a filter
obtained from the lowpass Butterworth filter when z is replaced by −z (right).
jω 2
|H(ejω)|2
|HB(e )|
ω ω
-2 π -π - ωc ωc π 2π -2 π - π - ω0 0 ω0 π 2π
Figure 5.16 Bandpass filter as shifted version of a lowpass filter.
5.5.3 Bandpass Filters
A bandpass filter is obtained from a lowpas filter by shifting its frequency

response for ω0 and −ω0 , as shown in Fig.5.16. The frequency response is
HB (e jω ) = H (e j(ω −ω0 ) ) + H (e j(ω +ω0 ) ).
In the discrete-time domain this frequency shift corresponds to
h B (n) = e jω0 n h(n) + e− jω0 n h(n) = 2 cos(ω0 n)h(n).

× h(n) ×
x(n) sin(ω n) 2sin(ω n) y(n)

0 0
× h(n) ×
cos(ω0n) 2cos(ω0n)
Figure 5.17 Bandpass system realization using corresponding lowpass systems and signal
modulation.
In general for an input signal x (n) the output of a bandpass filter is
∞ ∞
y(n) = h B (n) ∗ x (n) = ∑ h B (m) x (n − m) = 2 ∑ cos(ω0 m)h(m) x (n − m)
m=−∞ m=−∞
∞
=2 ∑ cos(ω0 n + ω0 m − ω0 n)h(m) x (n − m)
m=−∞
∞
=2 ∑ [cos(ω0 n) cos(ω0 m − ω0 n) − sin(ω0 n) sin(ω0 m − ω0 n)]h(m) x (n − m)
m=−∞
∞
= 2 cos(ω0 n) ∑ cos(ω0 (n − m)) x (n − m)h(m)
m=−∞
∞
+2 sin(ω0 n) ∑ sin(ω0 (n − m)) x (n − m)h(m).
m=−∞
Finally we may write the output of a bandpass filter as a function of the

lowpass impulse response as
y(n) = 2 cos(ω0 n) {[cos(ω0 n) x (n)] ∗ h(n)}

+ 2 sin(ω0 n) {[sin(ω0 n) x (n)] ∗ h(n)} .
This relation leads to a realization of a bandpass filter using lowpass filters,

as shown in Fig.5.17.
5.5.4 Allpass Systems - System Stabilization
A system (filter) with unit (constant) amplitude of the frequency response is
z−1 − ae− jθ 1 − zae− jθ z − 1a e jθ − j2θ

H A (z) = −
= = e ,
jθ
1 − ae z 1 z − ae jθ
1 − 1a e− jθ z
where 0 < a < 1 and θ is an arbitrary phase. For this system

' '
' '
'H A (e jω )' = 1.
To prove the statement consider
' '
' ' '' e− jω − ae− jθ '' ' e j(θ −ω ) − a '
' jω ' ' '
'H A (e )' = '' '=' '
1 − ae jθ e− jω ' ' 1 − ae jθ e− jω '
< <
(cos(θ − ω ) − a)2 + sin2 (θ − ω ) a2 − 2a cos(θ − ω ) + 1
= 2
= = 1.
2 2
(1 − a cos(θ − ω )) + a sin (θ − ω ) 1 − 2a cos(θ − ω ) + a2
Example 5.10. Given a system

z+2
H (z) = 1 1
.
(z − 2 )( z − 3 )( z − 2)
System cannot be causal and stable since there is a pole at z = 2. Define an

allpass system to be connected to H (z) in cascade such that the resulting sys-
tem is causal and stable, with the same amplitude of the frequency response
as H (z).
⋆The system is
z+2 z − 1a e jθ
Hs (z) = H (z) H A (z) = e− j2θ .
(z − 12 )(z − 13 )(z − 2) 1 − 1a e− jθ z
For a = 1/2 and θ = 0 we get
z+2 z−2
Hs (z) =
(z − 12 )(z − 13 )(z − 2) 1 − 2z
z+2
=− .
2(z − 12 )2 (z − 13 )
This system has the same frequency response amplitude as the initial system
' ' ' ' ' '
' ' ' ' ' '
'Hs (e jω )' = 'H (e jω ) H A (e jω )' = 'H (e jω )' .
The allpass system can be generalized to the form
z−1 − a1 e− jθ1 z−1 z−1 − a2 e− jθ2 z−1 z−1 − a N e− jθ N z−1

H A (z) = ...
1 − a1 e jθ1 z−1 1 − a2 e jθ2 z−1 1 − a N e jθ N z−1
where 0 < ai < 1 and θi , i = 1, 2, ..., N are arbitrary constants and phases. The
resulting frequency response amplitude is
' '
' '
'H A (e jω )' = 1.
This system can be used for multiple poles cancellation and phase correc-
tion.
5.5.5 Inverse and Minimum Phase Systems
An inverse system to the system H (z) is defined as
1
Hi (z) = .
H (z)
It is obvious that
H (z) Hi (z) = 1
h ( n ) ∗ h i ( n ) = δ ( n ).
This kind of system can be used to reverse the signal distortion. For ex-
ample, assume that the Fourier transform of a signal x (n) is distorted dur-
ing transmission by a transfer function H (z), i.e., the received signal z-
transform is R(z) = H (z) X (z). In that case the distortion can be compen-
sated by processing the received signal using the inverse system. The output
signal is obtained as
1
Y (z) = R ( z ) = X ( z ).
H (z)
The system Hi (z) = 1/H (z) should be stable as well. It means that the poles
of the inverse system should be within the unit circle. The poles of the
inverse system are equal to the zeros of H (z).
The system H (z) whose both poles and zeros are within the unit circle
is called a minimum phase system.
Example 5.11. (a) Which of these two systems

5
z2 + z − 16
H1 (z) = 3
z2 + z + 16
3
z2 − z + 16
H2 (z) = 3
z2 + z + 16
is a minimum phase system?
(b) If the amplitude of the Fourier transform of the discrete-time re-
ceived signal is distorted as R(z') = H1'(z) X' (z) what' is a stable and causal
' ' ' '
system HD (z) that will produce 'Y (e jω )' = 'X (e jω )' at its output if the input
is the received signal r (n)?
⋆ a) The systems can be written as
(z − 14 )(z + 54 )
H1 (z) =
(z + 14 )(z + 34 )
(z − 14 )(z − 34 )
H2 (z) =
(z + 14 )(z + 34 )
The first system is causal and stable for the region of convergence |z| > 3/4.
However one of its zeros is at |z| = 5/4 > 1 and the system is not a minimum
phase system, since its causal inverse form is not stable. The second system is
causal and stable. The same holds for its inverse since all poles of the inverse
system are within |z| < 1. Thus, the system H2 (z) is a minimal phase system.
(b) In this case
5
z2 + z − 16 (z − 14 )(z + 54 )
R(z) = 3
X (z) = X ( z ).
z2 + z + 16 (z + 14 )(z + 34 )
An inverse system to H1 (z) cannot be used since it will not be stable. How-
ever the inverse can be stabilized with an allpass system H A (z) so that the
amplitude is not changed
1 1
Y (z) = R(z) H (z) = H1 (z) X (z) H (z)
H1 (z) A H1 (z) A
where
5
z+ 4
H A (z) = 5
1+ 4z
and
1 (z + 14 )(z + 34 ) (z + 54 )
HD ( z ) = H A (z) =
H1 (z) (z − 14 )(z + 54 ) (1 + 54 z)
(z + 14 )(z + 34 )
=
(z − 14 )(1 + 54 z)
' ' ' '

' ' ' '
This system is stable and causal and will produce 'Y (e jω )' = 'X (e jω )'.
If a system is the minimum phase system (with all poles and zeros
within |z| < 1) then this system has a minimum group delay out of all
systems with the same amplitude of the frequency response. Thus, any
nonminimum phase system will have a more negative phase compared
to the minimum phase system. The negative part of the phase is called
the phase-lag function. The name minimum phase system comes from the
minimum phase-lag function.
In order to prove this statement consider a system H (z) with the sam-
ple amplitude of the frequency response as a nonminimum phase system
Hmin (z). Its frequency response can be written as
z−1 − ae− jθ
H (z) = Hmin (z) H A (z) = Hmin (z)
1 − ae jθ z−1
Here we assumed the first-order allpass system without any loss of gener-
ality, since the same proof can be used for any number of allpass systems
that multiply Hmin (z). Since 0 < a1 < 1 and the system Hmin (z) is stable the
system H (z) has a zero at |z| = 1/a1 > 1.
The phases of the system are related as
arg{ H (e jω )} = arg{ Hmin (e jω )} + arg{ H A (e jω )}.
The phase of allpass system is
e− jω − ae− jθ
arg{ H A (e jω )} = arg{ }
1 − ae jθ e− jω
1 − ae− jθ e jω
= arg{e− jω } = −ω + arg{1 − ae− jθ e jω }
1 − ae jθ e− jω
a sin(ω − θ )
− arg{1 − ae jθ e− jω } = −ω − 2 arctan .
1 − a cos(ω − θ )
Its derivative (group delay) is
d arg{ H A (e jω )} a cos(ω − θ ) − a2
τgA (ω ) = − =1+2
dω 1 − 2a cos(ω − θ ) + a2
1 − a2 1 − a2
= 2
=' ' .
1 − 2a cos(ω − θ ) + a '1 − ae j(ω −θ ) '2
Since a < 1 then the group delay is always positive and
τg (ω ) = τg min (ω ) + τgA (ω )
τg (ω ) ≥ τg min (ω ),
with τg (ω ) and τg min (ω ) being the phase derivatives (group delays) of

systems H (z) and Hmin (z), respectively.
The phase behavior of all pass system is
1 − ae− jθ
arg{ H A (e j0 )} = arg{ }=0 (5.16)
1 − ae jθ
"ω
arg{ H A (e jω )} = − τg (ω )dω ≤ 0 (5.17)
0
since τg (ω ) > 0 for 0 ≤ ω < π.
We can conclude that the minimum phase systems satisfy the following
conditions.
1. A minimum phase system is system of minimum group delay out
of the systems with the same amplitude of frequency response. A system
containing one or more allpass parts with uncompensated zeros outside of
the unit circle will have larger delay than the system which does not contain
zeros outside the unit circle.
2. The phase of a minimal phase system will be lower than the phase
of any other system with the same amplitude of frequency response since,
according to (5.17),
arg{ H (e jω ) = arg{ Hmin (e jω )} + arg{ H A (e jω )}

≤ arg{ Hmin (e jω )}.
This proves the fact that the phase of any system arg { H (e jω ) is always
lower than the phase of minimal phase system arg { Hmin (e jω )}, having the
same amplitude of the frequency response.
3. Since the group delay is minimal we can conclude that
n n
∑ |hmin (m)|2 ≥ ∑ |h(m)|2
m =0 m =0
This relation may be proven in a similar way like minimal phase property,
by considering the outputs of a minimum phase system and a system
H (z) = Hmin (z) H A (z).
Example 5.12. A system has absolute squared amplitude of the frequency response
equal to
B C2
' '2 5
' ' 2 cos(ω ) + 2
'H (e jω )' =
(12 cos(ω ) + 13)(24 cos(ω ) + 25)
Find the corresponding minimal phase system.
⋆ For the system we can write
' '2
' '
'H (e jω )' = H (e jω ) H ∗ (e jω ) = H (e jω ) H (e− jω )
In the z−domain the system with this amplitude of the frequency response
(with real-valued coefficients) satisfies
' ' ' '2
∗ 1 '
' 1 '' ' '
H (z) H ( ∗ )' = H (z) H ( )' = 'H (e jω )' = H (e jω ) H (e− jω ).
z z=e jω z z=e jω
In this sense
B C2
'
'
'2
' e jω + e− jω + 5
2
'H (e jω )' =
(6e jω + 6e− jω + 13)(12e jω + 12e− jω + 25)
and
B C2
1 z+ 5
2 + z −1
H (z) H ( ) =
z (6z + 13 + 6z−1 )(12z + 25 + 12z−1 )
B C2
z2 + 52 z + 1
=
(6z2 + 13z + 6)(12z2 + 25z + 12)
1 (z + 2)2 (z + 12 )2 1 ( 1z + 12 )2 (z + 12 )2
= = .
36 (z + 23 )(z + 32 )(z + 34 )(z + 43 ) 36 (z + 23 )( 1z + 23 )(z + 34 )( 1z + 34 )
The minimum phase system, with the desired amplitude of the frequency
response, is a part of H (z) H ∗ ( z1∗ ) with zeros and poles inside the unit circle
1 (z + 12 )2
H (z) = .
6 (z + 23 )(z + 34 )
The other poles and zeros then belong to H ∗ (1/z∗ ).
5.6 PROBLEMS
Problem 5.1. An RLC circuit transfer function is

1
LC
H (s) =
s2 + s RL + LC
1
with R/L = 8 and 1/ ( LC ) = 25. Find the difference equation describing

the corresponding discrete-time system obtained by the impulse invariance
method. What is the impulse response of the discrete-time system. Use
∆t = 1.
Problem 5.2. Could the method of impulse invariance be used to map the
system
s2 − 3s + 3
H (s) = 2
s + 3s + 3
to the discrete-time domain. What is the corresponding discrete-time system
obtained by the bilinear transform with ∆t = 1?
Problem 5.3. A continuous-time system is described by a differential equa-
tion
3 1
y′′ (t) + y′ (t) + y(t) = x (t)
2 2
with zero initial conditions. What is the corresponding transfer function
of discrete-time system using the first-order backward difference approxi-
mation with ∆t = 1/10? Write the difference equation of the system whose
output approximates the output of the continuous-time system.
Problem 5.4. Transfer function of a continuous-time system is
2s
H (s) = − .
s2 + 2s + 2
What is the corresponding discrete-time system using the invariance im-

pulse method and the bilinear transform with ∆t = 1?
Problem 5.5. A continuous-time system has a transfer function of the form
(1 + 4s)
H (s) = .
(s + 1/2)(s + 1)3
What is the corresponding discrete-time system according to:

(a) the impulse invariance method,
(b) the bilinear transform,
(c) the matched z-transform?
Use ∆t = 1.
Problem 5.6. A continuous-time system
2QΩ1
H (s) =
s2 + 2Ω1 Qs + Ω21 + Q2
is designed to pass the signal
x (t) = A1 cos(Ω1 t + ϕ1 )
and to stop all other possible signal components. The parameters are Q =
0.01, Ω1 = π/2. The signal is sampled with ∆t = 1 and a discrete-time signal
x (n) is formed. Using bilinear transform design the discrete system that
corresponds to the continuous-time system with transfer function H (s).
Problem 5.7. (a) By using the bilinear transform find the transfer function of
the second-order Butterworth filter with f ac = 4kHz. The sampling interval
is ∆t = 50µ sec.
(b) Translate the discrete-time transfer function to obtain a highpass filter.
Find its corresponding critical frequency in the continuous-time domain.
Problem 5.8. Design a discrete-time lowpass Butterworth filter for the sam-
pling frequency 1/∆t = 10 kHz. The passband should be from 0 to 1 kHz,
maximal attenuation in the passband should be 3 dB and the attenuation
should be more than 10 dB for frequencies above 2 kHz.
Problem 5.9. Using the impulse invariance method design a Butterworth
filter with the passband frequency ω p = 0.1π and stopband frequency
ωn = 0.3π in the discrete domain. Maximal attenuation in the passband
region should be less than 2dB, and the minimal attenuation in the stopband
should be 20dB.
Problem 5.10. Highpass filter can be obtained from a lowpass by using
HH (s) = H (1/s). Using the bilinear transform with ∆t = 2 we can trans-
form the continuous-time domain function into discrete domain using the
relation s = (z − 1)/(z + 1). If we have a design of a lowpass filter how to
change its coefficients in order to get a highpass filter.
Problem 5.11. For filtering of a continuous-time signal a discrete-time filter
is used. Find the corresponding continuous-time filter frequencies if the
discrete-time filter is: a) a lowpass with ω p = 0.15π, b) bandpass within
0.2π ≤ ω ≤ 0.25π, c) a highpass with ω p = 0.35. Consider cases when
∆t = 0.001s and ∆t = 0.1s.
What should be the starting frequencies to design these systems in the
continuous-time domain if the impulse invariance method is used and what
are the design frequencies if the bilinear transform is used?
Problem 5.12. A transfer function of the first-order lowpass system is
1−α
H (z) = .
1 − αz−1
Find the corresponding bandpass system transfer function with frequency

shifts for ±ωc .
Problem 5.13. Using allpass system find stable systems with the same
amplitude of the frequency response as the systems:
(a)
2 − 3z−1 + 2z−2
H1 (z) =
1 − 4z−1 + 4z−2
(b)
z
H2 (z) = .
(4 − z)(1/3 − z)
Problem 5.14. The z-transform
(z − 14 )(z−1 − 14 )(z + 12 )(z−1 + 12 )

R(z) =
(z + 45 )(z−1 + 45 )(z − 37 )(z−1 − 37 )
can can be written as

1
R(z) = H (z) H ∗ ( ).
z∗
Find H (z) for the minimum phase system.
Problem 5.15. A signal x (n) has passed trough a media whose influence
can be described by the transfer function
√
(4 − z)(1/3 − z)(z2 − 2z + 14 )
H (z) = .
z − 12
Signal r (n) is obtained. Find a causal and stable

' system
' ' to process ' r (n) in
order to obtain output signal y(n) such that 'Y (e jω )' = ' X (e jω )'.
5.7 SOLUTIONS
Solution 5.1. For this system we can write
1
LC 25
H (s) = =
s2 + s RL + LC
1 s2 + 8s + 25
25
=
(s + 4 + 3j)(s + 4 − j3)
j 25
6 − j 25
6
H (s) = + .
s + 4 + j3 s + 4 − j3
The poles are mapped using
s i → zi = e si .
The discrete-time system is
j 25
6 − j 25
6
H (z) = +
1 − e−(4+ j3) z−1 1 − e−(4− j3) z−1
25 −4 −1
3 e z sin 3
= ,
1 − 2e cos 3z 1 + e−8 z−2
− 4 −
with corresponding difference equation
25 −4
y(n) = e sin(3) x (n − 1) + 2e−4 cos(3)y(n − 1) − e−8 y(n − 2).
3
The output signal values can be calculated for any input signal using
this difference equation. For x (n) = δ(n) the impulse response would follow.
The impulse response can be obtained in a closed form from
25 ∞ −(4+ j3)n −n 25 ∞ −(4− j3)n −n

H (z) = j ∑
6 n =0
e z −j
6 n∑
e z
=0
as
25 −4n − j3n
h(n) = e ( je − je j3n )u(n) =
6
25
= e−4n sin(3n)u(n).
3
Solution 5.2. The system is not of lowpass type. For s → ∞ we get H (s) → 1.
Thus, the impulse invariance method cannot be used. The bilinear trans-
form can be used. It produces
(1 − z −1 )2 −z + 3−1
4 (1+z−1 )2 − 6 11+ z −1 13z−2 − 2z−1 + 1
H (z) = = .
(1 − z −1 )2 −z + 3
4 (1+z−1 )2 + 6 11+
−1 z−2 − 2z−1 + 13
z −1
Solution 5.3. For the system
3 1
y′′ (t) + y′ (t) + y(t) = x (t)
2 2
the transfer function is
1
H (s) = 3 1
.
s2 + 2s + 2
Corresponding discrete system is obtained using
1 − z −1
s→ = 10(1 − z−1 )
∆t
as
1
H (z) =
100(1 − z−1 )2 + 32 10(1 − z−1 ) + 1
2
1
= 231
.
100z−2 − 215z−1 + 2
The difference equation of this system is
2 430 200
y(n) = x (n) + y ( n − 1) − y ( n − 2 ).
231 231 231
1+j 1−j
H (s) = − − .
s+1−j s+1+j
Using the invariance impulse method it follows
2 − 2(cos(1) + sin(1))e−1 z−1

H (z) = − .
1 − 2 cos(1)e−1 z−1 + e−2 z−2
The bilinear transform produces
1 − z −2
H ( z ) = −2 .
5 − 2z−1 + z−2
Solution 5.5. (a) The transfer function
(1 + 4s)
H (s) =
(s + 1/2)(s + 1)3
is written to a form appropriate for the impulse invariance method
k1 k2 k3 k4
H (s) = + + +
s + 1/2 (s + 1) (s + 1)2 ( s + 1 )3
'
with k1 = H (s)(s + 1/2)|s=−1/2 = −8 and k4 = H (s)(s + 1)3 's=−1 = 6. By
equating the coefficients with s3 to 0 we get the relation k1 + k2 = 0. Similar
relation follows for the coefficients with s2 as 3k1 + 5k2 /2 + k3 = 0 or
k1 /2 + k3 = 0. Then k2 = 8 and k3 = 4. With
ki ki
→
s − si 1 − e s i z −1
and
1 dm k i 1 dm ki
m → m { }
m! dsi s − si m! dsi 1 − esi z−1
we get the discrete system
−8 8
H (z) = +
1 − e−1/2 z−1 1 − e −1 z −1
* +' * +'
d 4 '
' d2 6 '
'
+ s − 1 ' + 2 s − 1 '
ds1 1 − e 1 z s1 =−1 ds1 1 − e 1 z s1 =−1
−8 8 4e−1 z−1 3e−2 z−2 + 3e−1 z−1
= + − −
+ − −
+
1 − e−1/2 z−1 1−e z1 1 1
(1 − e z ) 1 2 (1 − e −1 z −1 )3
−5.83819z−3 − 9.68722z−2 + 22.0531z−1
=
(z−1 − e)3 (z−1 − e1/2 )
(b) Discrete system obtained using the bilinear transform is
−1
(1 + 8 11− z
+ z −1
)
H (z) = −1 −1
−z + 1/2)(2 1−z
(2 11+ + 1 )3
z −1 1 + z −1
−14z−4 − 24z−3 + 12z−2 + 40z−1 + 18
= .
3z−4 − 32z−3 + 126z−2 − 216z−1 + 135
(c) The matched z-transform produces
4(1 − e−1/4 z−1 )

H (z) = P Q 3
.
1 − e−1/2 z−1 (1 − e−1 z−1 )
Solution 5.6. Since we use the bilinear transform we have to pre-modify the
system according to
2 Ω ∆t
Ωd = tan( 1 ) = 2.0 = 0.6366π.
∆t 2
The frequency value is shifted from Ω1 = 0.5π to Ωd = 0.6366π. The modi-
fied system is
2QΩd
Hd (s) = 2 .
s + 2Ωd Qs + Ω2d + Q2
−z −1
Now using s = 2 11+ z −1
the corresponding discrete- system is obtained,
2QΩd
H (z) = B C2 B C .
− z −1
2 11+ + 2Ωd Q 2 11− z −1
+ Ω 2 + Q2
z −1 + z −1 d
The bilinear transform returns the pre-modified frequency to the desired

one.
Solution 5.7. The poles of H (s) H (−s) for a continuous-time second order
(N = 2) Butterworth filter are
sk = Ωc e j(2πk+π )/2N + jπ/2 = 2π f c e j(2πk+π )/4+ jπ/2 ,

2
where f c = ∆t tan(2π f ac ∆t/2)/(2π ) = 4.6253 kHz. With k = 0, 1, 2, 3 follows
√ √
2 2
sk = 2π f c (± ±j ).
2 2
For a stable system the poles satisfy Re {s p } < 0, thus
√ √
2 2
s1,2 = 2π f c (− ±j ).
2 2
The transfer function H (s) is
s1 s2 4π 2 f c2
Ha ( s ) = = √ .
(s − s1 )(s − s2 ) s + 2π f c 2s + 4π 2 f c2
2
Using the bilinear transform with ∆t = 50 · 10−6 we get the corresponding

discrete system transfer function
1.0548(1 + z−1 )2
H (z) = .
5.1066 − 1.8874z−1 + z−2
This filter has −3 dB attenuation at ω = 0.4π corresponding to Ω =

0.4π/∆t = 2π × 4 × 103 .
b) The discrete highpass filter is obtained by a shift corresponding to
Hh (e jω ) = H (e j(ω +π ) ). It corresponds to the impulse response modulation
hh (n) = (−1)n h(n) or substitution of z by −z in the transfer function,
1.0548(1 − z−1 )2
H (z) = .
5.1066 + 1.8874z−1 + z−2
The critical frequency of highpass filter is ωc = 0.6π or f ac = 6 kHz.
Solution 5.8. For the continuous-time system the design frequencies are
f p = 1 kHz
f s = 2 kHz.
They correspond to
Ω p = 2π 103 rad/s
Ωs = 4π 103 rad/s.
The discrete-time frequencies are obtained from ω = Ω∆t = Ω/104 as
ω p = 0.2π
ωs = 0.4π.
The frequencies for the filter design, that will be mapped to ωs and ω p by
using the bilinear transform, are
2 0.6498
Ω pd = tan(0.2π/2) =
∆t ∆t
2 1.4531
Ωsd = tan(0.4π/2) = .
∆t ∆t
The filter order follows from
1−100.1a p
1 log 1−100.1as
N= = 1.368.
2 log Ω pd
Ωsd
We assume N = 2.
Since the frequency for −3 dB attenuation is given the design critical

frequency is
0.6498
Ωcd = Ω pd = .
∆t
The poles of the filter transfer function are
√ √
0.6498 2 2
s p1/2 = (− ±j )
∆t 2 2
with the transfer function

1
s p1 s p2 ∆t2
0.4223
H (s) = = 1
.
(s − s p1 )(s − s p2 ) s2 + 0.919s ∆t + 0.4223 ∆t1 2
Mapping this system into the discrete-time domain by using the bilinear
transform,
2 1 − z −1
s= ,
∆t 1 + z−1
produces
0.067569(1 + z−1 )2
H (z) = .
1 − 1.14216z−1 + 0.412441z−2
Solution 5.9. The Butterworth filter order is
1−100.1a p
1 log 1−100.1as
N= = 2.335.
2 log Ω p
Ωs
with Ω p = ω p /∆t, Ωs = ωs /∆t, and ∆t = 1. Assume N = 3.

Critical frequency Ωc , where the amplitude of the frequency response
is attenuated for 3 dB, is
Ωp
Ωc = 2N
, = 0.109345π = 0.3435.
100.1a p − 1
The transfer function H (s) poles are
s p2/3 = −0.17175 ± j0.29748

s p1 = −Ωc = −0.3435.
The transfer function form is

−s p1 s p2 s p3 0.0405
H (s) = =
(s − s p1 )(s − s p2 )(s − sp3 ) (s + 0.3435)(s3 + 0.3435s + 0.1178
k1 k2 k3
= + +
s − s p1 s − s p2 s − s p3
0.3435 0.17175 − j0.09916 0.17175 + j0.09916
= − − .
s + 0.3435 s + 0.17175 + j0.29748 s + 0.17175 − j0.29748
The coefficients k i are calculated from

'
'
k i = H (s)(s − s pi ) 's=s pi .
Using the impulse invariance method, mapping from the continuous-time

domain to the discrete-time domain, is done according to
ki ∆tk i
→ .
s − spi 1 − es pi ∆t z−1
The discrete-time system transfer function is
−0.0253z−2 − 0.0318z−1
H (z) = .
−1.98774 + 4.61093z−1 − 3.68033z−2 + z−3
Solution 5.10. The transfer function is
1
HH (s) = H ( )
s
2 1− z −1 2 z −1
with s = ∆t 1+z−1 = ∆t z+1 and ∆t = 2. Corresponding lowpass filter would
be
z−1
HL (z) = H (s)|s= z−1 = H ( ).
z +1 z+1
The discrete highpass filter is
'
1 '
HH (z) = HH (s)|s= z−1 = H ( )''
z +1 s s = z −1
z +1
z+1
HH (z) = H (
).
z−1
Obviously HH (z) = HL (−z). It means that a discrete highpass system can be
realized by replacing z with −z in the transfer function. For ∆t ̸= 2 a scaling
is present as well.
Solution 5.11. a) The mapping with ∆t = 0.001 s produces a lowpass filter

with Ω p = ω p /∆t = 150 π rad/s. For ∆t = 0.1 s the frequency is Ω p =
ω p /∆t = 1.5 π rad/s.
b) For ∆t = 0.001 s a bandpass filter is obtained for the range 200π
rad/s ≤ Ω ≤ 250π rad/s, while ∆t = 0.1 s produces a bandpass filter with
2π rad/s ≤ Ω ≤ 2.5π rad/s.
c) For ∆t = 0.001 s a highpass filter has the frequency Ω p = 350 rad/s,
while for ∆t = 0.1 s the highpass filter has critical frequency Ω p = 3.5 rad/s.
For the impulse invariance method starting design frequencies should
be equal to the calculated analog frequencies. If the bilinear transform
is used calculated analog frequencies Ω p should be pre-modified to Ωm
2 Ω p ∆t
according to Ωm = ∆t tan 2 .
Solution 5.12. The impulse response of the passband filter is h B (n) =
2h(n) cos(ωc n). The z-transform of the impulse response is
∞ ∞ ∞
HB ( z ) = ∑ 2h(n) cos(ωc n)z−n = ∑ h(n)(e− jωc z)−n + ∑ h(n)(e jωc z)−n
n=−∞ n=−∞ n=−∞
2(1 − α)(1 − α cos ωc z−1 )

HB (z) = H (e− jωc z) + H (e jωc z) = .
1 − 2α cos ωc z−1 + α2 z−2
Solution 5.13. The causal system
2 − 3z−1 + 2z−2
H1 (z) =
(1 − 2z−1 )2
is not stable since it has a second-order pole at z = 2. This system may be
stabilized, keeping the same amplitude of the frequency response, using a
second-order allpass system with zero at z = 2
( )2
z −1 − 1
2
H A (z) = 1 −1
.
1− 2 z
The new system has a transfer function
2 − 3z−1 + 2z−2
H1 (z) = .
( z −1 − 2 )2
Causal system H2 (z) has a pole at z = 4. It can be stabilized by using allpass
system
z−1 − 14 4−z
H A (z) = 1 −1
= .
1 − 4z 4z − 1
The transfer function of a stable system is

z
H2 (z) = .
(4z − 1)(1/3 − z)
Solution 5.14. For the z-transform
(z − 14 )(z−1 − 14 )(z + 12 )(z−1 + 12 )
R(z) =
(z + 45 )(z−1 + 45 )(z − 37 )(z−1 − 37 )
and
1
R(z) = H (z) H ∗ (
).
z∗
the minimum phase system is a part of R(z) whose all zeros and poles
are inside the unit circle, meaning that H (z) system and its inverse system
1/H (z) can be a causal and stable. Therefore,
(z − 14 )(z + 12 )
H (z) = .
(z + 45 )(z − 37 )
It is easy to check that H ∗ ( z1∗ ) is equal to he remaining terms in R(z), since
1 ( z1∗ − 14 )∗ ( z1∗ + 12 )∗ (z−1 − 14 )(z−1 + 12 )

H∗ ( ) = = .
z∗ ( z1∗ + 45 )∗ ( z1∗ − 37 )∗ (z−1 + 45 )(z−1 − 37 )
B C∗
Here we used, for example, ( z1∗ − 14 )∗ = z1∗ − 14 = 1z − 14 .
Solution 5.15. The received signal should be processed by the inverse
system
1 z − 12
Hi (z) = = √ .
H (z) (4 − z)(1/3 − z)(z2 − 2z + 14 )
However this system has two poles outside the unit circle since
z − 12
Hi (z) = .
(4 − z)(1/3 − z)(z − 1.2071)(z − 0.2071)
These poles have to be compensated, keeping the same amplitude, by using
two first-order allpass systems. The resulting system transfer function is
z − 4 z − 1.2071
Hi (z)
1 − 4z 1 − 1.2071z
z − 12
= .
(1/3 − z)(z − 0.2071)(1 − 4z) (1 − 1.2071z)
5.8 EXERCISE
Exercise 5.1. Transfer function of a continuous-time system is
( s + 2)
H (s) = .
4s2 + s + 1
What is the corresponding discrete-time system obtained with ∆t = 1 by
using the impulse invariance method and the bilinear transform.
Exercise 5.2. A continuous system is described by a differential equation
1
y′′ (t) + 6y′ (t) − y(t) = x (t) + x ′ (t)
2
with zero initial conditions. What is the corresponding transfer function
of a discrete system obtained by using the first-order backward difference
approximation with ∆t = 1?
Exercise 5.3. (a) A continuous system
2QΩ0
H (s) =
s2 + 2Ω0 Qs + Ω20 + Q2
with Q = 0.01 is designed to pass the signal
x (t) = A cos(Ω0 t + ϕ)
for Ω0 = 3π/4 and to stop all other possible signal components.

The signal is sampled with ∆t = 1 and a discrete-time signal x (n) is
formed. Using the bilinear transform, design a discrete system that corre-
sponds to the continuous system with transfer function H (s).
(b) What is the output r (n) of the obtained discrete-time system to the
samples y(n) of the analog signal
y(t) = 1 + 2 sin(250πt) − cos(2750πt) + 2 sin(750πt)
sampled with the sampling interval ∆t = 10−3 s. What would be the corre-
sponding continuous-time output signal after an ideal D/A converter.
Exercise 5.4. (a) By using the bilinear transform find the transfer function
of a third-order Butterworth filter with f ac = 3.4 kHz. The sampling step is
∆t = 40 µ sec.
(b) Translate the discrete transfer function to obtain a bandpass system
with corresponding central frequency f ac = 12.5 kHz in the continuous
domain.
Exercise 5.5. Design a continuous lowpass filter whose parameters are:

- passband frequency Ω p = 2π f p , f p = 3.5 kHz,
- stopband frequency Ωs = 2π f s , f s = 6 kHz,
- maximal attenuation in passband a p = 2 dB, and
- minimal attenuation in the stopband as = 16 dB.
Find the corresponding discrete-time filter using:
(a) the impulse invariance method and
(b) the bilinear transform,
with ∆t = 0.05 × 10−3 sec.
(c) Write the corresponding highpass filter transfer functions, obtained
by a frequency shift in the discrete domain for π, for both cases.
Exercise 5.6. Using allpass system find a stable and causal system with the
same amplitude of the frequency response as the systems:
2 − 5z−1 + 2z−2
H1 (z) = ,
1 − 4z−1 + z−2
z−1
H2 (z) = .
(2 − z)(1/4 − z)
Exercise 5.7. The z-transform
(z − 13 )(z−1 − 13 )
R(z) =
(z + 12 )(z−1 + 12 )
can can be written as
1
R(z) = H (z) H ∗ ().
z∗
Find H (z) for the minimum phase system. If h(n) is the impulse response
of H (z) and h1 (n) is the impulse response of
z−1 − a1 e− jθ1
H1 (z) = H (z)
1 − a1 e jθ1 z−1
show that |h(0)| ≤ |h1 (0)| for any θ1 and | a1 | < 1. All systems are causal.
Exercise 5.8. A signal x (n) has passed trough a media whose influence can
be described by the transfer function
(1 − z/3)(1 − 5z)(z2 − z + 34 )
H (z) =
z2 − 2/3
and the signal r (n) = x (n) ∗ h(n) is' obtained.
' Find
' a causal
' and stable system
to process r (n) in order to obtain 'Y (e jω )' = ' X (e jω )'.
Chapter 6
Realization of Discrete Systems
discrete-time systems may, in general, be described by a differ-
L
INEAR
ence equation relating the output signal with the input signal at the
considered instant and the previous values of the output and input
signal. The transfer function can be written in various forms producing dif-
ferent system realizations. Some of them will be presented next. Symbols
that are used in the realizations are presented in Fig.6.1.
6.1 REALIZATION OF IIR SYSTEMS
A system that includes recursions of the output signal values results in an

infinite impulse response (IIR). These systems will be presented first.
a
z -1
x(n) ax(n) x(n) x(n-1) x(n) x(n)
x(n)
+ + ×
x(n) x(n)+y(n) x(n) - x(n)- y(n) x(n) x(n)y(n)
y(n) y(n) y(n)
Figure 6.1 Symbols and their function in the realization of discrete-time systems.
267
268 Realization of Discrete Systems
x(n) B0 y(n)
+ +
z-1 z-1
x(n-1) + + y(n-1)
B A1
-1 1 -1
z z
x(n-2) y(n-2)
B A
2 2
Figure 6.2 Direct form implementation of a second order system.
6.1.1 Direct realization I
Consider a discrete system described by a linear difference equation
y(n) = A1 y(n − 1) + ... + A N y(n − N ) (6.1)

+ B0 x (n) + B1 x (n − 1) + ... + B M x (n − M).
A second-order system, as a special case, will be presented first. Its imple-

mentation is shown in Fig.6.2.
A general system described by (6.1) can be implemented as in Fig.6.3.
This form is a direct realization I of a discrete-time system.
6.1.2 Direct realization II
Direct realization I, presented in Fig.6.3, consists of two blocks in cascade.

The first block realizes
y1 (n) = B0 x (n) + B1 x (n − 1) + ... + B M x (n − M )
and the second block corresponds to the recursive relation
y(n) = A1 y(n − 1) + ... + A N y(n − N ) + y1 (n).
These two blocks have transfer functions
H1 (z) = B0 + B1 z−1 + ... + B M z− M

x(n) B y (n) y(n)

0
1
+ +
z-1 z-1
+ +
B A
-1 1 1
z z-1
+ +
B A
2 2
-1 -1
z z
B A
M N
Figure 6.3 Direct form I implementation of a discrete-time system.
and
1
H2 (z) = .
1 − A1 z−1 − ... − A N z− N
The overall transfer function is
H (z) = H1 (z) H2 (z) = H2 (z) H1 (z).
It means that these two blocks can interchange their positions. After the
positions are interchanged, then by using the same delay systems, we get
the resulting system in the direct realization II form, presented in Fig.6.4.
This system uses a reduced number of delay blocks in the realization.
Example 6.1. Find the transfer function of a discrete system presented in Fig.6.5.
⋆The system can be recognized as a direct realization II form. After
its blocks are separated and interchanged the system in a form presented in
Fig.6.6 is obtained.
The output of the first block is
1 1
y1 ( n ) = x ( n ) − x ( n − 1) + x ( n − 2). (6.2)
2 3
Its transfer function is
1 1
H1 (z) = 1 − z−1 + z−2 .
2 3
x(n) B0 y(n)
+ +
z-1
+ +
A B
1 -1 1
z
+ +
A B
2 2
z-1
A B
N M
Figure 6.4 Direct realization II of a discrete-time system.
x(n) y(n)
+ +
-1
z
+
-1/2
-1
z
+
1/2 1/3
z-1
-1/6
Figure 6.5 A discrete-time system.
The output of the second block is described by the following difference

equation
1 1
y ( n ) = y ( n − 2) − y ( n − 3) + y1 ( n ). (6.3)
2 6
The transfer function of this block is
1
H2 (z) = .
1 − 12 z−2 + 16 z−3
x(n) y (n) y(n)

1
+ +
-1 -1
z z
+
-1/2
-1 -1
z z
+
1/3 1/2
-1
z
-1/6
Figure 6.6 System with interchanged blocks.
The difference equation for the whole system is obtained after y1 (n) from
(6.2) is replaced into (6.3)
1 1 1 1
y(n) = y ( n − 2) − y ( n − 3) + x ( n ) − x ( n − 1) + x ( n − 2).
2 6 2 3
The system transfer function is
1 − 12 z−1 + 13 z−2
H (z) = H1 (z) H2 (z) = .
1 − 12 z−2 + 16 z−3
6.1.3 Sensitivity of the System Poles/Zeros to Errors in Coefficients
Systems with a large number of elements in a recursion may be sensitive to

the errors due to the coefficients deviations. Deviations of the coefficients
from the true values are caused by finite order registers used to memorize
them in a computer. Influence of the finite register lengths to the signal and
system realization will be studied later, as a part of random disturbance.
Here, we will only consider influence of this effect to the system coefficients
since it may influence the way how to realize a discrete system.
For the first-order system with a real-valued pole
1 1
H (z) = −
=
1 + A1 z 1 1 − z p1 z−1
the error in coefficient A1 is the same as the error in the system pole z p1 . If
the coefficient is quantized with a step ∆ then the error in the pole location
is of order ∆. The same holds for the system zeros.
For a second-order system with real-valued coefficients and a pair of
complex-conjugated poles
1 1
H (z) = =
1 + A 1 z −1 + A 2 z −2 (1 − z p1 z−1 )(1 − z p2 z−1 )
the relation between the coefficients and the real and imaginary parts of the
poles z p1/2 = x p ± jy p is
1
H (z) =
1 − 2x p z −1 + ( x2p + y2p )z−2
A1 = −2x p
A2 = x2p + y2p .
The error in coefficient A1 defines the error in the real part of poles x p .
When the coefficient A2 assumes discrete values A2 = m∆, with A1 ∼
x p = n∆ then the imaginary part of poles may assume the values y p =
F √
± A2 − x2p = ± m∆ − n2 ∆2 with n2 ≤ mN. For small n, i.e., for small real
√
part of a pole, y p = ± ∆m. For N discretization levels, assuming that the
poles are within the unit circle x2p + y2p ≤ 1, the first discretization step is
√
changed from 1/N order to 1/ N order. The error, in this case, could be
significantly increased. The changes in y p due to the discretization of A2
may be large.
The quantization of x p and y p as a result of quantization of − A1 /2
and A2 = x2p + y2p is shown in Fig.6.7 for the case of N = 16 and N = 32
quantization levels. We see that the error in y p , when it assumes small
values, can be very large. We can conclude that the poles close to the unit
circle with larger imaginary values y p are less sensitive to the errors. The
highest error could appear if a second order real-valued pole (with y p = 0)
were implemented by using a second order system.
We have concluded that the poles close to the real axis (small y p )
are sensitive to the error in coefficients even in the second order systems.
The sensitivity increases with the system order, since the higher powers in
polynomial increase the maximal possible error.
Consider a general form of a polynomial in the transfer function,
written in two forms
P(z) = z M + z M−1 A1 + ... + A M

y =Im{z } y =Im{z }
p p p p
1 1
0.5 0.5
0 0
-0.5 -0.5
-1 -1
-1 -0.5 0 0.5 1 -1 -0.5 0 0.5 1
x =Re{z } x =Re{z }
p p p p
Figure 6.7 Quantization of the real and imaginary parts x p = Re{z p } and y p = Im{z p } of
poles (zeros) as a result of the quantization in 16 levels (left) and 32 levels (right) of the
coefficients A1 = −2x p and A2 = x2p + y2p .
and
P(z) = (z − z1 )(z − z2 )...(z − z M ).
If the coefficients A1 , A2 , ..., A M are changed for small ∆A1 , ∆A2 , ..., ∆A M
(due to quantization) then the pole position (without loss of generality and
for notation simplicity consider the pole z1 ) is changed for
- .
∼ ∂z1 ∂z1 ∂z1
∆z1 = ∆A1 + ∆A2 + ... + ∆A M . (6.4)
∂A1 ∂A2 ∂A M | z = z1
Since there is no a direct relation between z1 and A1 we will find ∂z1 /∂Ai
using
∂P(z) ∂P(z) ∂z1
= .
∂Ai |z=z1 ∂z1 ∂Ai |z=z1
From this relation it follows
∂P(z)
∂z1 ∂Ai |z=z1 z1M−i
= ∂P(z)
= .
∂Ai |z=z1 −(z1 − z2 )(z1 − z3 )...(z1 − z M )
∂z1 |z=z1
The coefficients ∂z1 /∂Ai|z=z1 could be large, especially in the case when
there are close poles, with a small distance (zi − zk ).
Example 6.2. Consider a discrete system

1
H (z) =
P(z)
with
12 7 111 95
P(z) = (z − )(z − )(z − )(z − )
27 29 132 101
∼
= (z − 0.4444)(z − 0.2414)(z − 0.8409)(z − 0.9406)
In the realization of this system the coefficients are rounded to two decimal
positions, with absolute error up to 0.005. Find the poles of the system with
rounded coefficients.
⋆The system denominator is
P(z) ∼
= z4 − 2.4673z3 + 2.1200z2 − 0.7336z + 0.0849.
With rounded coefficients to two decimal positions we get
P̂(z) = z4 − 2.47z3 + 2.12z2 − 0.73z + 0.08
with poles
P̂(z) = (z − 0.2045)(z − 0.5370)(0.7285)(z − 1).
The poles of the function with rounded coefficients can differ significantly
from the original pole values. Maximal error in poles is 0.115. One pole is on
the unit circle making the system with rounded coefficients unstable, in this
case.
Note that if the system is written as a product of the first-order func-
tions in the denominator and each pole value is rounded to two decimals
1
H (z) = 7 12 111 95
(z − 29 )( z − 27 )( z − 132 )( z − 101 )
P(z) ∼
= (z − 0.24)(z − 0.44)(z − 0.84)(z − 0.94)
the poles will differ from the original ones for no more than 0.005.
If the poles are grouped into the second-order terms (what should be
done if the coefficients were complex-conjugate in order to avoid calculation
with complex valued coefficients), then
P(z) ∼
= (z − 0.6858z + 0.1073)(z − 1.7815z + 0.7910).
If the coefficients are rounded to two decimal positions
P̂(z) = (z − 0.69z + 0.11)(z − 1.78z + 0.79)

P(z)
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Re{z}
P1(z)P2(z)
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Re{z}
Figure 6.8 Poles for a system with errors in coefficients
we will get
P̂(z) = (z − 0.25)(z − 0.44)(z − 0.8442)(z − 0.9358)
with maximal error of 0.01.

The pole values are illustrated in Fig.6.8.
The sensitivity analysis for this example can be done for each pole.
Assume that the poles are denoted as z1 = 12/27, z2 = 7/29, z3 = 111/132
and z4 = 95/101. Then
(z1 − z2 )(z1 − z3 )(z1 − z4 ) = 0.0399

∂z1 z41−1
= = −2.1979
∂A1 |z=z1 −(z1 − z2 )(z1 − z3 )(z1 − z4 )
∂z1 z41−2
= = −4.9452,
∂A2 |z=z1 −(z1 − z2 )(z1 − z3 )(z1 − z4 )
∂z1
= −11.1267,
∂A3 |z=z1
∂z1
= −25.0350
∂A4 |z=z1
with the errors in the coefficients
∆A1 = −2.4673 − (−2.47) = 0.0027,

∆A2 = 2.12 − 2.12 = 0,
∆A3 = −0.7336 − (−0.73) = −0.0036,
∆A4 = 0.0849 − 0.08 = 0.0049.
Replacing these values into (6.4) the approximation of the error is
∆z1 ∼
= 0.0878.
The true error is ∆z1 = 0.0926. A small difference is due to the linear approx-
imation, assuming small ∆Ai . The obtained result is a good estimate of an
order of error for the pole z1 . The error in z1 is about 18.5 time greater than
the maximal error in the coefficients Ai , that is of order 0.005.
6.1.4 Cascade Realization
A transfer function of discrete-time system might be written as a product of

the first-order subsystems
1 − zo0 z−1 1 − zo1 z−1 1 − zoN z−1

H (z) = k × ... × .
1 − z p0 z−1 1 − z p1 z−1 1 − z pN z−1
Commonly real-valued signals are processed and the poles and zeros in the
transfer function are in complex-conjugated pairs. In that case it is better to
group these pairs into second order systems to avoid complex calculations.
The transfer function is of the form
B00 + B10 z−1 + B20 z−2 B0K + B1K z−1 + B2K z−2
H (z) = × ... ×
1 − A10 z−1 − A20 z−2 1 − A1K z−1 − A2K z−2
= H0 (z) H1 (z)...HK (z),
where
B0i + B1i z−1 + B2i z−2
Hi (z) =
1 − A1i z−1 − A2i z−2
are second-order systems with real-valued coefficients. The whole system
may be realized as a cascade of lower-order (first or second-order) systems,
Fig.6.9. Of course, if there are some real-valued poles then there is no need
to group them. It is better to keep the realization order of the subsystems as
low as possible.
x(n) B00 B0K y(n)

+ + + +
z-1 z-1
+ + + +
A B A B
10 -1 10 1K -1 1K
z z
A B A B
20 20 2K 2K
Figure 6.9 Cascade realization of a discrete-time system.
x(n) r(n) y(n)

+ H(z)
-
H(z)
Figure 6.10 System with a feedback.
In the realization the second-order subsystems are commonly used.

It is possible to realize these second-order subsystems using the first-order
systems with real-valued coefficients x pL and y pL that are real and imag-
inary parts of the complex-conjugated pair of poles, z pL = x pL ± jy pL , re-
spectively. To this aim consider first an example.
Example 6.3. Find the transfer function of a system with a feedback shown in
Fig.6.10.
⋆The z-transform of the signal at the output of adder is
R ( z ) = X ( z ) − H ( z )Y ( z ) .
The output signal z-transform is
Y ( z ) = H ( z ) R ( z ) = H ( z ) X ( z ) − H 2 ( z )Y ( z ) .
The transfer function of this system is
Y (z) H (z)
He (z) = = .
X (z) 1 + H 2 (z)
H2(z)
x(n) y(n)
+ H(z) +
- +
z -1
xp
H(z)
Figure 6.11 Complete second-order subsystem with complex-conjugate pair of poles realized
using the first-order systems.
Let us now consider a realization of the second-order subsystem of the

form
y pL z−1
Qi ( z ) = .
1 + A1i z−1 + A2i z−2
Using the real and imaginary parts of the poles z pL = x pL + jy pL the transfer
function can be expressed as
y pL z−1
Qi ( z ) =
1 − 2x pL z−1 + x2pL z−2 + y2pL z−2
y pL z−1
=
(1 − x pL z−1 )2 + y2pL z−2
1 1
= y pL z−1 * +2
(1 − x pL z−1 )2 y pL z−1
1+ 1− x pL z−1
H (z) H2 (z)
=
1 + H 2 (z)
where
y pL z−1 1
H (z) = −
and H2 (z) = .
1 − x pL z 1 1 − x pL z−1
Therefore the second-order system can be implemented as in Fig.6.11, by

using the first-order systems as in Fig.6.12. In this case there is no grouping
of the coefficients into second or higher order polynomial.
H(z)
x(n) yp y(n)
-1
+ z
+
xp
Figure 6.12 First-order system for the realization of the second-order system with complex-
conjugate pair of poles.
The error in one coefficient (real or imaginary part of a pole) does not
influence the other coefficients. However if an error in the signal calculation
happens in one cascade, then it will propagate as an input to the following
cascades. In that sense it would be the best to order cascades in such a way
that the lowest probability of an error appears in early cascades. From the
analysis of error we can conclude that the cascades with the poles and zeros
close to the origin are more sensitive to the error and should be used in later
stages.
Example 6.4. For the system
1.4533(1 + z−1 )3
H (z) =
(−0.8673z−1 + 3.1327)(3.0177z−2 − 5.434z−1 + 7.54)
1 + z −1 1 + 2z−1 + z−2
= 0.0615 ×
1 − 0.2769z−1 1 − 0.7207z−1 + 0.4002z−2
present the cascade realization using:

(a) both the first and the second-order systems;
(b) the first-order systems with real-valued coefficients only.
⋆(a) Realization of the system H (z) when both the first and the
second-order subsystems can used is done according to system transfer func-
tion as in Fig.6.13.
(b) For the first-order systems the realization should be done based on
1 + z −1
H (z) = 0.0615 × (1 + z −1 ) × (1 + z −1 )
1 − 0.2769z−1
1
× ,
1 − 0.7207z−1 + 0.4002z−2
x(n) 0.0615 y(n)

+ + + +
-1 -1
z z
+ +
1 0.2769 0.7207 -1 2
z
-0.4002 1
Figure 6.13 Cascade realization of a system.
with
1
1 − 0.7207z−1 + 0.4002z−2
1
=
(1 − (0.3603 + j0.5199)z−1 )(1 − (0.3603 − j0.5199)z−1 )
1
=
1 − 2 × 0.3603z−1 + 0.36032 z−2 + 0.51992 z−2
1 1 1
= = .
0.51992 z−2 + (1 − 0.3603z−1 )2 (1 − 0.3603z−1 )2 1 + ( 0.5199z−1−1 )2
1−0.3603z
In this way the system can be written and realized in terms of the first-order
subsystems
1 + z −1 1 + z −1
H (z) = 0.0615 ×
1 − 0.2769z−1 1 − 0.3603z−1
1+z − 1 1
× × −1 .
1 − 0.3603z−1 −1
1 + 0.5199z −1 × 0.5199z −1
1−0.3603z 1−0.3603z
6.1.5 Parallel realization
This realization is implemented based on a transfer function written in the

form
B00 + B10 z−1 + B20 z−2 B + B1K z−1 + B2K z−2

H (z) = − −
+ ... + 0K
1
1 − A10 z − A20 z 2 1 − A1K z−1 − A2K z−2
= H0 (z) + H1 (z) + ... + HK (z).
x(n) 0.0615
+ + + +
-1 -1
z z
0.2769 0.3603
y(n)
+ + +
-1
z yp=0.5199
y yp
p
-1 + -1 +
z z
0.3603
0.3603 0.3603
Figure 6.14 Discrete-time system realized using first-order subsystems.
In the case of a parallel realization the error in one subsystem does not
influence the other subsystems. If an error in the signal calculation appears
in one parallel subsystem, then it will influence the output signal, but will
not influence the outputs of other parallel subsystems.
Example 6.5. For the system
−0.7256 + 0.2542z−1 0.7256 − 0.084z−1

H (z) = +
1 − 1.1078z−1 + 0.5482z−2 1 − 0.9246z−1 + 0.2343z−2
present a parallel and a cascade realization using the second-order subsys-

tems.
⋆Parallel realization follows directly from the system transfer function

definition. It is presented in Fig.6.16.
For the cascade realization the system transfer function should be
written in a form of the product of second-order transfer functions,
x(n) y(n)
+ + +
B00
z-1
+ +
A10 B10
-1
z
A B
20 20
+ +
B01
z-1
+ +
A B
11 11
z-1
A21 B21
+ +
B0K
z-1
+ +
A1K B1K
-1
z
A2K B2K
Figure 6.15 Parallel realization of a discrete-time system.

x(n) -0.7256 y(n)

+ +
z-1
1.1078 -1 0.2542
z
-0.5482
0.7256
+ +
-1
z
0.9246 -0.084
z-1
-0.2343
0.0373z−1 + 0.0858z−2 + 0.0135z−3

H (z) = P QP Q
1 − 1.1078z−1 + 0.5482z−2 1 − 0.9246z−1 + 0.2343z−2
z −1
=
1 − 1.1078z−1 + 0.5482z−2
0.0373 + 0.0858z−1 + 0.0135z−2
× .
1 − 0.9246z−1 + 0.2343z−2
Cascade realization is presented in Fig.6.17.
6.1.6 Inverse realization
For each of the previous realization an inverse form may be implemented by

switching the input and the output signal and changing the flow directions
of the signal. As an example consider the direct realization II from Fig.6.4.
This realization, with separated delay circuits is shown in Fig.6.18. Its in-
verse form is presented in Fig.6.19. It is easy to conclude that the inverse
realization of the direct realization II has the same transfer function as the
direct realization I. Since both realization I and realization II have the same
x(n) 0.0373 y(n)

+ + +
-1 -1
z z
+ + +
1.1078 1 0.9246 0.0858
z-1 z
-1
-0.5482 -0.2343 0.0135
Figure 6.17 Cascade realization of a discrete system.
x(n) B y(n)
0
+ +
z-1 z-1
+ +
A B
1 -1 -1 1
z z
+ +
A B
2 2
-1 -1
z z
AN BM
Figure 6.18 Direct realization II with separated delay circuits.
transfer functions it follows that the inverse realization has the same transfer
function as the original realization.
6.2 FIR SYSTEMS AND THEIR REALIZATIONS
In general, transfer functions of discrete systems are obtained in the form

of a ratio of two polynomials. The polynomial in the transfer function
denominator defines poles. In the time domain it means a recursive relation,
relating the output signal at the current instant with the previous output
signal values. Realization of this kind of system is efficient, as described
y(n) B x(n)
0
+ +
z-1 z-1
+ +
A B
1 1
z-1 z-1
+ +
A B
2 2
-1 -1
z z
A B
N M
Figure 6.19 Inverse realization of the direct realization II.
in the pervious section. Systems that would not have recursions, when the
output signal is a linear combination of the input signal and its delayed
versions only,
y(n) = B0 x (n) + B1 x (n − 1) + ... + B M x (n − M )
are the FIR systems. These systems are always stable. The FIR systems can
also have a linear phase.
6.2.1 Linear Phase Systems and Group Delay
In an implementation of a discrete system it is important to modify the

amplitude of the Fourier transform of the input signal in a desired way. At
the same time we should take care about the phase function changes in the
input signal. In an ideal case of signal filtering the phase function should
remain the same, meaning a zero-phase transfer function. A linear phase
form of the transfer function
Im{ H (e jω }
arg{ H (e jω } = arctan{ } = −ωq (6.5)
Re{ H (e jω }
is also acceptable in these systems. They will have a constant group delay
d(arg{ H (e jω })
τg = − =q
dω
and will not distort the impulse response with respect to the zero-phase
system. The impulse response will only be delayed in time for q.
Example 6.6. Consider an input signal of the form
M
x (n) = ∑ A m e j ( ωm n + θ m ).
m =1
After passing through a system with frequency response H (e jω ) this signal is

changed to
M jωm
y(n) = ∑ Am | H (e jωm )|e j(ωm n+θm +arg{ H (e })
.
m =1
In general the phase of each signal component is changed in a different

way for arg{ H (e jωm }, causing the signal distortion due to different delays
corresponding to different frequencies. If the phase function of the frequency
response is linear then all signal component phases are changed in the same
way for arg{ H (e jωm } = −ωm q. They corresponding to a constant delay for
all components. A delayed signal, without distortion, is obtained
M
y(n) = ∑ Ai | H (e jωm )|e j(ωm (n−q)+θm ) .
m =1
In the case of a linear phase arg { H (e jω } = −ωq the phase delay
arg{ H (e jω }
τϕ = − =q
ω
and the group delay τg are the same. In general, the group delay and the
phase delays are different. The group delay, as notion dual to the instanta-
neous frequency, is introduced and discussed in the first chapter.
Consider a system with a real-valued impulse response h(n). Its fre-

quency response is
N −1 N −1 N −1
H (e jω ) = ∑ h(n)e− jωn = ∑ h(n) cos(ωn) − j ∑ h(n) sin(ωn). (6.6)
n =0 n =0 n =0
Combining the linear phase condition (6.5) with form (6.6), we get
Im{ H (e jω } ∑nN=−01 h(n) sin(ωn)

− tan(ωq) = = − ,
Re{ H (e jω } ∑nN=−01 h(n) cos(ωn)
or
N −1
∑ h(n)[sin(ωq) cos(ωn) − cos(ωq)sin (ωn)] = 0.
n =0
The last equation can be written as
N −1
∑ h(n) sin(ω (n − q)) = 0. (6.7)
n =0
The middle point of interval where h(n) ̸= 0 is n = ( N − 1)/2. If q = ( N −

1)/2, then sin(ω (n − q)) is an odd function with respect to n = ( N − 1)/2.
The summation (6.7) is zero if the impulse response h(n) is an even function
with respect to n = ( N − 1)/2. Hence, the solution of (6.7) is
N−1
q=
2
h(n) = h( N − 1 − n), 0 ≤ n ≤ N − 1.
Since the Fourier transform is unique, this is the unique solution for the
linear phase condition. It is illustrated for an even and odd N in Fig.6.20.
From the symmetry condition it is easy to conclude that there is no a causal
linear phase system with infinite impulse response.
6.2.2 Windows
When a system obtained from the design procedure is an IIR system and
the requirement is to implement it as an FIR system, in order to get a linear
phase or to guaranty the system stability (when small changes of coefficients
are possible), then the most obvious way is to truncate the desired impulse
response hd (n) of the resulting IIR system. The impulse response of the FIR
system is
!
hd (n) for 0 ≤ n ≤ N − 1
h(n) =
0 elsewhere.
This form can be written as
h ( n ) = h d ( n ) w ( n ),
where !
1 for 0≤n≤ N−1
w(n) =
0 elsewhere
q=16
h(n)
N=32
0 16 32
n
q=16.5
h(n)
N=33
0 16.5 33
n
Figure 6.20 Impulse response of a system with a linear phase for an even and odd N.
is the rectangular window function. In the Fourier domain the desired im-
pulse response truncation by a window function will mean a convolution of
the desired frequency response with the frequency response of the window
function
H (e jω ) = Hd (e jω ) ∗ W (e jω ).
Since the rectangular window function has a Fourier transform of the

form
N −1
sin(ωN/2)
W (e jω ) = ∑ e− jωn = e− jω ( N −1)/2
n =0
sin(ω/2)
its convergence is slow with significant oscillations. It will cause oscilla-

tions in the resulting frequency response H (e jω ), Fig.6.21. By increasing the
number of samples N the convergence speed will increase. However the
oscillations amplitude will remain the same, Figs.6.21 (d) and (f). Even with
N → ∞ the amplitude oscillations will remain, Figs.6.21 (b). This effect is
called the Gibbs phenomenon.
Example 6.7. A desired frequency response of a system is Hd (e jω ) with the IIR

hd (n) for −∞ < n < ∞. Find the FIR system impulse response hc (n) that
approximates the desired transfer function with a minimal mean absolute
squared error.
⋆The mean squared absolute error is

"π ' '2
1 ' '
e2 = 'Hd (e jω ) − Hc (e jω )' dω.
2π
−π
According to Parseval’s theorem

"π ' '2 ∞
1 ' '
2
e = 'Hd (e jω ) − Hc (e jω )' dω = ∑ |hd (n) − hc (n)|2
2π n=−∞
−π
Without loss of generality, assume that the most significant values of hd (n)
are within − N/2 ≤ n ≤ N/2 − 1. The impulse response hc (n) can assume
nonzero values only within − N/2 ≤ n ≤ N/2 − 1. Therefore,
N/2−1 − N/2−1 ∞
e2 = ∑ |hd (n) − hc (n)|2 + ∑ |hd (n)|2 + ∑ |hd (n)|2 .
n=− N/2 n=−∞ n= N/2
Since the last two terms are hc (n) independent and all three terms are
non negative, the error e2 is minimal if
hc (n) = hd (n), − N/2 ≤ n ≤ N/2 − 1.
If we want to have a causal realization of the FIR system then
h(n) = hc (n − N/2).
A shift in time' does not ' change

' the
' amplitude of the desired frequency
' ' ' '
response, since 'H (e jω )' = 'Hc (e jω )'.
In order to reduce the oscillations in frequency response amplitude

other windows are introduced. They are presented within the introductory
chapters, trough the examples. Here we will list the basic windows (for
more details see Section 9.2).
Triangular (Bartlett) window is defined as
%
|n+1− N/2|
1− for 0≤n≤ N−1
w(n) = N/2 .
0 elsewhere
Avoiding window discontinuities at the ending points the convergence

of its transform is improved. Since this window may be considered as a
convolution of two rectangular windows
1
w(n) = [u(n) − u(n − N/2)] ∗n [u(n) − u(n − N/2)]
N/2
(a)
0.2
h (n)
d
0
-60 -40 -20 0 20 40 60 n
(b)
H ( ejω )
1
0.5
d
0
-π - π/4 0 π/4 π ω
(c)
0.2
h(n)
0
-40 -20 0 20 40 60 n
|H( ejω )|
1 (d)
0.5
0
-π - π/4 0 π/4 π ω
(e)
0.2
h(n)
0
-20 0 20 40 60 n
|H( ejω )|
1 (f)
0.5
0
-π - π/4 0 π/4 π ω
(g)
0.2
h(n)
0
-20 0 20 40 60 n
|H( ejω )|
1 (h)
0.5
0
-π - π/4 0 π/4 π ω
Figure 6.21 Impulse response of a FIR system obtained by truncating the desired IIR response
(a), (b) using two rectangular window of different widths (c)-(f), and using a Hann(ing)
window (g),(h).
its Fourier transform is a product of corresponding rectangular window

Fourier transforms
1 − jω ( N/2−1) sin2 (ωN/4)

W (e jω ) = e .
N/2 sin2 (ω/2)
Hann(ing) window defined by

! 1P Q
1 + cos((n − N/2) 2π for 0≤n≤ N−1
w(n) = 2 N
0 elsewhere
would be continuous in the continuous-time domain. In that domain its first

derivative would be continuous as well. Thus, its Fourier domain conver-
gence is further improved with respect to the rectangular and the Bartlett
windows. The Fourier transform of this window is related to the Fourier
transform of the rectangular window as W (e jω )/2 + W (e j(ω +2π/N )/4 +
W (e j(ω −2π/N )/4.
Hamming window is a slight modification of the Hann(ing) window
!
0.52 + 0.48 cos((n − N/2) 2π
N for 0 ≤ n ≤ N − 1
w(n) =
0 elsewhere.
It loses the continuity property (in the continuous-time domain). Its con-
vergence for very large values of ω will be slower than in the Hann(ing)
window case. However, as it will be shown later, its coefficients are derived
in such a way that the first side-lobe is canceled out at its mid point. Then
the immediate convergence, after the main lobe, is much better than in the
Hann(ing) window case.
Other windows are derived with various constraints. Some of them
will be reviewed in Part three of this book as well.
6.2.3 Design of FIR System in the Frequency Domain
Suppose that the desired system frequency response is given in the fre-
quency domain. If we want to get an N point FIR system that approximates
the desired frequency response then it can be obtained by sampling the fre-
quency response Hd (e jω ) at
2π
ω= k, k = 0, 1, 2, ..., N − 1
N
H (k ) = Hd (e jω )|ω =2πk/N
h(n) = IDFT{ H (k )}.
1.5 1.5
H (ejω), H (k)
H (ejω), H (k)
1 1
d
d
0.5 0.5
d
d
0 0
-0.5 -0.5
-2 0 2 -2 0 2
ω ω
h(n)
h(n)
n n
1.5 1.5
1 1
H(ejω), H(k)
H(ejω), H(k)
0.5 0.5
0 0
-0.5 -0.5
-2 0 2 -2 0 2
ω ω
Figure 6.22 Realization of a FIR system with N samples in time, obtained by sampling the
desired frequency response with N samples. A direct sampling (left) and the sampling with
smoothed transition (right),
This procedure is illustrated on a lowpass filter design, Fig.6.23. Note

that at the discontinuity points high oscillation will occur in the resulting
H (e jω ). The oscillations can be avoided by smoothing the transition inter-
vals. Smoothing by a Hann(ing) window in the frequency domain is shown
in Fig.6.23.
0.5
W (k)
w(n)
H
0.25
-16 -12 -8 -4 0 4 8 12 n
k
Figure 6.23 A Hann(ing) window for smoothing the frequency response in the frequency
domain (left) and in the time domain (right).
x(n) x(n-1) x(n-2) x(n-N+1)

z-1 z-1 z-1
h(0) h(1) h(2) h(N-1)
+ + +
y(n)
Figure 6.24 Direct realization of a FIR system.
6.2.4 Realizations of FIR system
The FIR systems can be realized in the same way as the IIR systems pre-
sented in the previous section, without using the recursive coefficients. A
common way of presenting a direct realization of FIR system is shown in
Fig.6.24. It is often referred to as an adder with weighted coefficients h(n).
A realization of liner phase FIR system that uses the coefficients sym-
metry h(0) = h( N − 1), h(2) = h( N − 2),... is shown in Fig.6.25.
Realization of a frequency sampled FIR filter may be done using the
relation between the z−transform and the DFT of a signal.
If we want to realize a FIR system with N nonzero samples, then it
can be expressed in term of the DFT of frequency response (samples of the
transfer function H (z) along the unit circle) as follows. For a FIR filter we
x(n) x(n-1) x(n-2) x(n-(N/2-1))

-1 -1 -1
z z z
h(0) h(1) h(2) h(N/2-1)

+ + + + -1
z
z-1 z-1 z-1

x(n-N+1)
y(n)
+ + +
Figure 6.25 Direct realization of a FIR system with a linear phase.
may write
N −1
H (k) = ∑ h(n)e− j2πnk/N
n =0
N −1
1
h(n) = ∑ H (k )e j2πnk/N .
N k =0
Then the transfer function H (z) using the values of h(n), 0 ≤ n ≤ N − 1, is
N −1 N −1 N −1
1 1 1 − z− N e j2πk
H (z) = ∑ ∑ H (k )e j2πnk/N z−n = ∑ H (k )
N k =0 n =0
N k =0 1 − z−1 e j2πk/N
with H (k ) = H (z) for z = exp( j2πk/N ), k = 0, 1, 2, ..., N − 1.
Example 6.8. For a system whose impulse response is the Hamming window
function of the length N = 32 present the FIR filter based realization.
⋆For the Hamming window with N = 32 the impulse response is

given by
π
h(n) = 0.52 + 0.48 cos((n − 16) )
16
0 ≤ n ≤ 31.
The DFT values are H (0) = 0.52 × 32, H (1) = −0.24 × 32, H (31) = H (−1) =
−0.24 × 32 and H (k) = 0 for other k within 0 ≤ k ≤ 31. Therefore
1 1 − z−32 1 1 − z−32 e j2π 1 1 − z−32 e− j2π

H (z) = −
H (0) − H (1) − H (31)
32 1 − z 1 −
32 1 − z e1 j2π/32 32 1 − z−1 e− j2π/32
( )
1 H (0) 2H (1)(1 − cos(π/16)z−1 )
= (1 − z−32 ) −
− .
32 1−z 1 1 − 2 cos(π/16)z−1 + z−2
This is a cascade of
H1 (z) = (1 − z−32 )/32
and a system H2 (z) + H3 (z) where
H2 (z) = H (0)/(1 − z−1 )
and
1 − cos(π/16)z−1
H3 (z) = −2H (1) .
1 − 2 cos(π/16)z−1 + z−2
Example 6.9. For a system whose frequency response Hd ( jΩ) in the continuous-
time domain is
Hd ( jΩ) = π − |Ω|
for |Ω| ≤ π, with corresponding Hd (e jω ) in the discrete-time domain (∆t = 1
is assumed, Fig.6.26) find the FIR filter impulse response with N = 7 and
N = 8 using:
(a) Sampling the desired frequency response Hd (e jω ) in the frequency
domain.
(b) Calculating hd (n) = IFT{ Hd (e jω )} and taking its N the most signif-
icant values, h(n) = hd (n) for − N/2 ≤ n ≤ N/2 − 1 and h(n) = 0 elsewhere.
(c) Comment the error in both cases.
⋆(a) The sampling in frequency domain is illustrated in Fig.6.26. The
values of the FIR system, in this case, are the samples of Hd (e jω ),
' !
' π (1 − 2 Nk ) for 0 ≤ k < N/2
H (k ) = Hd (e jω )' = .
ω =2πk/N π (2 Nk − 1) for N/2 ≤ k ≤ N − 1
The sampling is illustrated in the second row of Fig.6.26 for N = 7 and N = 8.

Impulse response of the FIR filter is
h(n) = IDFT{ H (k )}
1 N −1
H (k)e j2πnk/N .
N k∑
=
=0
For N = 7
π 10π 2π 6π 2π 2π 2π
h(n) = + cos( n) + cos(2 n) + cos(3 n)
7 49 7 49 7 49 7
0 ≤ n ≤ 6.
For N = 8
π 3π 2π π 2π π 2π
h(n) = + cos( n) + cos(2 n) + cos(3 n)
8 16 8 8 8 16 8
0 ≤ n ≤ 7.
It is shown in Fig.6.26 (third row). The frequency response of the FIR filter is
H (e jω ) = FT{h(n)}.
Its values are equal to the desired frequency response at the sampling points
' '
' '
H (e jω )' = Hd (e jω )' .
ω =2πk/N ω =2πk/N
Outside these point the frequency responses significantly differ (calculate,

for example the values H (e j0 ), H (e jπ/2 ), and H (e jπ )). Here, there is no sig-
nificant discontinuity in the frequency response. It means that the frequency
response smoothing, by using a window (Hann(ing) or Hamming window
in the time domain), would not improve the result.
(b) The impulse response of the desired system is
"π
1
hd (n) = IFT{ Hd (e jω )} = (π − |ω |) e jωn dω
2π
−π
"π
2
= (π − ω ) cos(ωn)dω =
2π
0
1 − cos(nπ )
= .
πn2
Using the first N = 7 samples in the time domain we get
%
1−cos(nπ )
for −3 ≤ n ≤ 3
h(n) = πn2
0 elsewhere.
or for N = 8 %
1−cos(nπ )
h(n) = πn2
for −4 ≤ n ≤ 3 .
0 elsewhere.
The frequency response of this FIR filter is
H (e jω ) = FT{h(n)}.
jω
H (jΩ) H (e )
d d
3 3
2 2
1 1
0 0
-5 -π 0 π 5 -5 -π 0 π 5
H(k), N=7 H(k), N=8

3 3
2 2
1 1
0 0
-5 0 5 -5 0 5
2 2
h(n), N=7 h(n), N=8
1 1
0 0
-1 -1
0 2 4 6 0 2 4 6
H(ejω), N=7 H(ejω), N=8

3 3
2 2
1 1
0 0
-5 0 5 -5 0 5
Figure 6.26 Design of a FIR filter by frequency sampling of the desired frequency response.
It is shown in Fig.6.27.
(c) The error in frequency sampling (a) is zero at the desired frequency
points. However, since the frequency response is equal to the samples of
the impulse response of an infinite duration there will be aliasing of the
impulse response, resulting in the error outside the sampling points. For the
case of windowing the impulse response (b), the aliasing in the frequency
response is avoided since the impulse response is truncated. However, the
truncation causes an error in the resulting frequency response. In this case
the error distribution is not the same as in case (a). The mean square error Er
2
hd(n)
-1
-15 -10 -5 0 5 10 15
2 2
h(n), N=7 h(n), N=8
1.5 1.5
1 1
0.5 0.5
0 0
-4 -2 0 2 4 -4 -2 0 2 4
jω jω
H(e ), N=7 H(e ), N=8
3 3
2 2
1 1
0 0
-5 0 5 -5 0 5
Figure 6.27 Design of a FIR filter by windowing the impulse response of an IIR filter.
is calculated and presented in Fig.6.28, along with the errors in the absolute
value of the frequency responses. As expected from the theory, the impulse
response truncation produced lower mean square error in the estimation.
6.3 PROBLEMS
Problem 6.1. For the system whose transfer function is
16(z + 1)z2
H (z) =
(4z2 − 2z + 1)(4z + 3)
plot the cascade, parallel and direct realization.

0.2
E = 0.008092
r
0.1
-0.1
-0.2
0 1 2 3 4 5 6
0.2
Er= 0.0018945
0.1
-0.1
-0.2
0 1 2 3 4 5 6
Figure 6.28 Error in the case of the frequency response sampling (top) and the IIR impulse
response truncation (bottom), along with the corresponding mean square error (Er ) value.
Problem 6.2. Given a discrete system with
y(n) = x (n) + x (n − 1) + x (n − 2) + y(n − 1) − y(n − 2) − 3y(n − 3).
Plot the direct realization I and II, parallel and cascade realization.
Problem 6.3. Find the transfer function of a discrete system presented in
Fig.6.29.
Problem 6.4. Find the transfer function of a discrete system presented in
Fig.6.30.
Problem 6.5. For the system
1 − 0.2z−1 + 0.02z−2 1 − 1.8z−1 + 1.45z−2

H (z) = ,
1 − 1.7z−1 + 1.285z−2 1 − 0.1z−1 + 0.125z−2
present a cascade realization. Order the system so that the subsystem which
is less sensitive to possible quantization comes first.
Problem 6.6. If the transfer function of a system is
4z2 4z + 4
H (z) = 2
,
4z − 2z + 1 4z + 3
x(n) y(n)
+ + + +
z-1 z-1
+ + + +
2 1/2 1/3
z-1 z
-1
1/3 -1/3 -1/4
Figure 6.29 Discrete-time system
-1 -1
z z
x(n) y(n)
+ + + + +
-1 -1
z z
+ + + +
2 -1 1/2 -1 1/3
z z
1/3 -1/3 -1/4
plot the cascade and parallel realization. Write down the difference equation
which describes this system.
Problem 6.7. For the system defined by the transfer function
1 + z −2
H (z) =
1 + 2z−1 + 2z−2 + z−3
plot the cascade realization.
Problem 6.8. System is defined by
1 1 2
y ( n ) + y ( n − 1) + w ( n ) + w ( n − 1) = x ( n )
4 2 3
H1 ( z )
x(n) rsinθ
+
+
-1
rcosθ z
y(n)
+
-r sinθ
-1
rcosθ z
H ( z)
2
5 5
y(n) − y(n − 1) + 2w(n) − 2w(n − 1) = − x (n),
4 3
where x (n) is the input signal, y(n) is the output, and w(n) is a signal within
the system. What is the frequency and impulse response of the system?
Problem 6.9. For the system presented in Fig.6.31 find the transfer function.
Problem 6.10. Show that the FIR system
1 + 2z − z2 + 4z3 − z4 + 2z5 + z6
H (z) =
z6
has a linear phase function. Find its group delay.
Problem 6.11. Let h(n) be an impulse response of a causal system with the
Fourier transform H (e jω ). A real-valued output signal y1 (n) = x (n) ∗ h(n)
of this system is reversed, r (n) = y1 (−n), and passed through the same
system, resulting in the output signal y2 (n) = r (n) ∗ h(n). The final output
is reversed again y(n) = y2 (−n). Find the phase of the frequency response
function of the overall system.
Problem 6.12. For a system whose frequency response in the continuous-
time domain is
⎧
⎨ 2 for |ω | < π2
Hd ( jΩ) = 1 for π2 < |ω | < 3π
4
⎩
0 elsewhere,
x(n) y(n)
+ +
-1 -1
z z
+
-1/4 -1
z
+
1/8
z-1
-3/16
Figure 6.32 Direct realization I of a discrete-time system.
with the corresponding Hd (e jω ) in the discrete-time domain obtained with

∆t = 1, find the FIR filter impulse response with N = 15 and N = 14 using:
domain,
(c) Comment the sources of error in both cases.
6.4 SOLUTIONS
Solution 6.1. In order to plot the direct form of realization, transfer function
should be written in a form suitable for this type of realization,
16(z + 1)z2 1 + z −1
H (z) = =
(4z2 − 2z + 1)(4z + 3) (1 − 12 z−1 + 14 z−2 )(1 + 34 z−1 )
1 + z −1
= .
1 + 14 z−1 − 18 z−2 + 3 −3
16 z
According to the previous relation, direct realization form I and II

follows. They are presented in Fig.6.32 and Fig.6.33, respectively.
x(n) y(n)
+ +
-1
z
+
-1/4 -1
z
+
1/8
z-1
-3/16
x(n) y(n)
+ + +
-1 -1
z z
+
1/2 -3/4
z-1
-1/4
For a cascade realization, the transfer function is written as
1 + z −1
H (z) =
(1 − 12 z−1 + 14 z−2 )(1 + 34 z−1 )
1 + z −1 1
= = H1 (z) H2 (z).
1 − 12 z−1 + 14 z−2 1 + 34 z−1
The cascade realization, implemented as a product of two blocks, will have

the form as shown in Fig.6.34.
x(n) y(n)
+ +
22/19
z-1
+
1/2 1/19
z-1
-1/4
+
-3/19
-1
z
-3/4
In order to plot a parallel realization, the transfer function should be

written in a suitable form for this type of realization,
1 + z −1 Az−1 + B C
H (z) = = + .
(1 − 1 −1
2z + 4 z )(1 + 4 z ) 1 − 12 z−1 + 14 z−2 1 + 34 z−1
1 −2 3 −1
Calculating the coefficients A = 1/19, B = 22/19 and C = −3/19, we get

22 1 −1 3
19 + 19 z − 19
H (z) = + .
1 − 12 z−1 + 14 z−2 1 + 34 z−1
It is used to plot the parallel realization, Fig.6.35.

Solution 6.2. Using the z-transform properties, the given difference equa-
tion can be written as
Y (z) = X (z) + X (z)z−1 + X (z)z−2 + Y (z)z−1 − Y (z)z−2 − 3Y (z)z−3 .
According to the definition of transfer function, follows:
Y (z) 1 + z −1 + z −2
H (z) = = .
X (z) 1 − z−1 + z−2 + 3z−3
Direct realizations I and II, presented in Fig.6.36 and Fig.6.37, respectively,

follow from the previous equation.
x(n) y(n)
+ +
-1 -1
z z
x(n-1) y(n-1)
+ +
1
-1 -1
z z
x(n-2) y(n-2)
+
-1
-1
z
y(n-3)
-3
Figure 6.36 Direct realization I of a discrete-time system.
x(n) y(n)
+ +
-1
z
+ +
1
-1
z
+
-1
z-1
-3
For a cascade realization, the transfer function should be written as a

product of two blocks
1 + z −1 + z −2 1
H (z) = = H1 (z) H2 (z).
1 − 2z−1 + 3z−2 1 + z−1
This form is suitable for the cascade realization given in Fig.6.38.

For a parallel realization, we will write the transfer function as
1 1 −1
6 2z + 56
H (z) = + .
1 + z −1 1 − 2z−1 + 3z−2
x(n) y(n)
+ + +
-1 -1
z z
+ +
2 -1 -1
z
-3
Its realization is now straightforward.

Solution 6.3. The system can be recognized as a cascade realization form. It
can be written as a product of two blocks
H (z) = H1 (z) H2 (z).
where H1 (z) denotes the first block. It can be considered as a direct realiza-
tion II, with
1 1 1
y1 (n) = 2y1 (n − 1) + y1 (n − 2) + x (n) + x (n − 1) − x (n − 2),
3 2 3
presented in Fig.6.39. Using the z-transform properties, its transfer function
is
Y (z) 1 + 12 z−1 − 13 z−2
H1 (z) = 1 = .
X (z) 1 − 2z−1 − 13 z−2
Now consider the second block whose transfer function is H2 (z). This
block can be considered as a parallel realization of two blocks, H2 (z) =
H21 (z) + H22 (z) where
H21 (z) = 1.
The second transfer function is the transfer function corresponding to a
direct realization II, of a subsystem described by
1 1
y 2 ( n ) = y 2 ( n − 1 ) + y 2 ( n − 2 ) + x 1 ( n ) + x 1 ( n − 1 ) − x 1 ( n − 2 ).
3 4
Thus, the transfer function of this subsystem is
Y2 (z) 1 + 13 z−1 − 14 z−2

H22 (z) = = .
X1 ( z ) 1 − z −1 − z −2
x(n) y1(n) x (n)

1 y2(n) y(n)
+ + + + +
-1 -1
z z
+ + + +
2 -1 1/2 -1 1/3
z z
1/3 -1/3 -1/4
Figure 6.39 A discrete-time system.
It means that
1 + 13 z−1 − 14 z−2
H2 (z) = H21 (z) + H22 (z) = 1 + .
1 − z −1 − z −2
The transfer function of the whole system is
( )
1 + 12 z−1 − 13 z−2 1 + 13 z−1 − 14 z−2
H (z) = H1 (z) H2 (z) = 1+ .
1 − 2z−1 − 13 z−2 1 − z −1 − z −2
Solution 6.4. This realization can be considered as a cascade realization of

two blocks H1 (z) and H2 (z),
H (z) = H1 (z) H2 (z).
First block is a direct realization II, whose transfer function is
1 + ( 12 + 1)z−1 − 13 z−2
H1 (z) = .
1 − 2z−1 − 13 z−2
Previous relation holds since the upper delay block (above the obvious
direct realization II block) has the same input and output as the first delay
block below it.
The block with transfer function H2 (z) can be considered as a parallel
realization of two blocks, similarly as in previous example, with, H21 (z) and
H22 (z), defined by
1 + 13 z−1 − 14 z−2
H21 (z) = ,
1 − z −1 − z −2
and
H22 (z) = z−1 .
Hence, the transfer function of the right block is
1 + 13 z−1 − 14 z−2
H2 (z) = H21 (z) + H22 (z) = + z −1 .
1 − z −1 − z −2
Now, the resulting transfer function can be written in the form
H (z) = H1 (z) H2 (z) =

( )
1 + ( 12 + 1)z−1 − 13 z−2 1 + 13 z−1 − 14 z−2
= + z −1 .
1 − 2z−1 − 13 z−2 1 − z −1 − z −2
H (z) = H1 (z) H2 (z).
It can be expressed, having in mind roots of the numerator and denominator

polynomials, as
P QP Q
1 − (0.1 + j0.1)z−1 1 + (0.1 + j0.1)z−1
H (z) =
(1 − (0.85 − j0.75)z−1 ) (1 + (0.85 − j0.75)z−1 )
P QP Q
1 − (0.9 + j0.8)z−1 1 + (0.9 − j0.8)z−1
× .
(1 − (0.05 − j0.1)z−1 ) (1 + (0.05 + j0.1)z−1 )
The subsystems should be positioned as
1 − 1.8z−1 + 1.45z−2
H1 (z) =
1 − 1.7z−1 + 1.285z−2
1 − 0.2z−1 + 0.02z−2
H2 (z) =
1 − 0.1z−1 + 0.125z−2
since the zero-pole pairs with small values of imaginary parts should come
later. They are more sensitive to the quantization of coefficients and they
will more probably cause this kind of error. Larger imaginary parts of roots
are less sensitive to these effects. The cascade realization is presented in
Fig.6.40.
Solution 6.6. For a cascade realization, form of the transfer function is
1 1 + z −1
H (z) = .
1 − 12 z−1 + 14 z−2 1 + 34 z−1
x(n) y(n)
+ + + +
-1 -1
z z
+ + + +
1.7 -1 -1.8 0.1 -1 -0.2
z z
-1.285 1.45 -0.125 0.02
Figure 6.40 Cascade realization less sensitive to possible quantization error
x(n) y(n)
+ + +
-1 -1
z z
+
-3/4 1/2 -1
z
-1/4
Figure 6.41 A cascade realization of a system
Its realization is presented in Fig.6.41

For a parallel realization, the transfer function can be written as
22 1 −1 3 −1
19 + 19 z − 19 z
H (z) = + 3 −1
.
1 − 12 z−1 + 14 z−2 1 + 4z
This realization is shown in Fig.6.42.

The transfer function can be written in the form
1 + z −1
H (z) = 1 −1
.
1+ 4z − 18 z−2 + 3 −3
16 z
A difference equation describing this system is
1 1 3
y ( n ) = x ( n ) + x ( n − 1 ) − y ( n − 1 ) + y ( n − 2 ) − y ( n − 3 ).
4 8 16
x(n) 22/19 y(n)

+ + +
z-1
1/2 1/19
z-1
-1/4
-3/19
+
-1
z
-3/4
Solution 6.7. The transfer function form corresponding to a cascade real-

ization is
(1 + z −2 )
H ( z ) = −1 .
(z + 1)(1 + z−1 + z−2 )
In order the use the smallest number of delay circuits, it can be ex-
pressed in the form
1 (1 + z −2 )
H (z) = H1 (z) H2 (z) = .
(1 + z −1 ) (1 + z −1 + z −2 )
This form corresponds to the cascade realization presented in Fig.6.43.
Solution 6.8. The z-transforms of these equations are
1 1 2
Y (z)(1 + z−1 ) + W (z)(1 + z−1 ) = X (z)
4 2 3
5 5
Y (z)(1 − z−1 ) + 2W (z)(1 − z−1 ) = − X (z).
4 3
By eliminating W (z) we get
1 5 1
Y (z)[(2 + z−1 )(1 − z−1 ) − (1 − z−1 )(1 + z−1 )]
2 4 2
4 5 1
= X (z)[ (1 − z−1 ) + (1 + z−1 )].
3 3 2
x(n) y(n)
+ + +
-1 -1
z z
+
-1 -1 -1
z
-1
Y (z) 3 − 12 z−1
H (z) = = ,
X (z) 1 − 34 z−1 + 18 z−2
with the difference equation describing this system
3 1 1
y(n) − y(n − 1) + y(n − 2) = 3x (n) − x (n − 1).
4 8 2
The frequency response is
3 − 12 e− jω
H (e jω ) = .
1 − 34 e− jω + 18 e− j2ω
Based on
Y (z) 3 − 12 z−1 4 1
H (z) = = 3 −1 1 −2
= 1 −1
− ,
X (z) 1 − 4z + 8z 1 − 2z 1 − 14 z−1
the impulse response is
h(n) = [4(1/2)n − (1/4)n ]u(n).
Solution 6.9. The transfer function of subsystem denoted by H1 (z) follows

from
y(n) = r sin θx1 (n − 1) + r cos θy(n − 1)
where x1 (n) is the input to this subsystem. Its transfer function is
Y (z) z−1 r sin θ

H1 (z) = = .
X1 ( z ) 1 − r cos θz−1
The transfer function of the other subsystem is
z−1 r sin θ
H2 (z) = − .
1 − r cos θz−1
For the feedback holds
H1 (z)( X (z) + Y (z) H2 (z)) = Y (z).
It produces
Y (z) H1 (z) z−1 r sin θ (1 − r cos θz−1 )

H (z) = = = .
X (z) 1 − H1 (z) H2 (z) 1 − 2r cos θz−1 + r2 z−2
Solution 6.10. The system impulse response is
h(n) = δ(n) + 2δ(n − 1) − δ(n − 2) + 4δ(n − 3) − δ(n − 4) + 2δ(n − 5) + δ(n − 6).
It satisfies the property
h ( n ) = h ( N − 1 − n ), 0 ≤ n ≤ N − 1
with N = 7, which implies phase function linearity. Thus, the group delay q
is
N−1
q= = 3.
2
Solution 6.11. We have that:
Y1 (e jω ) = H (e jω ) X (e jω )
R(e jω ) = Y1∗ (e jω ) = H ∗ (e jω ) X ∗ (e jω )
Y2 (e jω ) = R(e jω ) H (e jω ) = H ∗ (e jω ) H (e jω ) X ∗ (e jω )
Y (e jω ) = Y2∗ (e jω ) = H (e jω ) H ∗ (e jω ) X (e jω ).
So we get
Y (e jω ) = | H (e jω )|2 X (e jω ).
Obviously, the phase function of the system is equal to zero, for all ω.
Solution 6.12. (a) Values of the FIR filter, obtained by sampling frequency
response in the frequency domain are
'
'
H (k ) = Hd (e jω )' .
ω =2πnk/N
This sampling is illustrated in the second row of Fig.6.44 for N = 15 and

N = 14. The impulse response of the FIR filter is calculated as
h(n) = IDFT{ H (k )}
N −1
1
= ∑ H (k )e j2πnk/N .
N k =0
It is shown in Fig.6.44 (third row). Frequency response of the FIR filter is
H (e jω ) = FT{h(n)}.
Its values are equal to the desired frequency response at the sampling points
' '
' '
H (e jω )' = Hd (e jω )' .
ω =2πk/N ω =2πk/N
(b) The impulse response of the desired system is
sin(nπ/2) sin(3nπ/4)
hd (n) = IFT{ Hd (e jω )} = + .
πn πn
Using the first N = 15 samples in the discrete-time domain we get
!
hd (n) for −7 ≤ n ≤ 7
h(n) =
0 elsewhere
or for N = 16 !
hd (n) for −8 ≤ n ≤ 7
h(n) =
0 elsewhere.
The frequency response of this FIR filter is
H (e jω ) = DFT{h(n)}.
It is shown in Fig.6.45.
(c) The errors along with the mean square absolute errors Er are
jω
H (jΩ) H (e )
d d
3 3
2 2
1 1
0 0
-5 -π 0 π 5 -5 -π 0 π 5
H(k), N=15 H(k), N=14

3 3
2 2
1 1
0 0
-5 0 5 -5 0 5
2 2
h(n), N=15 h(n), N=14
1 1
0 0
-1 -1
-5 0 5 -6 -4 -2 0 2 4 6
H(ejω), N=15 H(ejω), N=14

3 3
2 2
1 1
0 0
-5 0 5 -5 0 5
Figure 6.44 Design of a FIR filter by frequency sampling of the desired frequency response.
6.5 EXERCISE
Exercise 6.1. Given a discrete system with
1 1 1
y ( n ) = x ( n ) − x ( n − 1 ) + x ( n − 2 ) + y ( n − 1 ) − y ( n − 2 ) − y ( n − 3 ),
2 3 4
plot the direct realization I and II, parallel and cascade realization.
2
hd(n)
-1
-15 -10 -5 0 5 10 15
h(n), N=15 h(n), N=14
1.5 1.5
1 1
0.5 0.5
0 0
-0.5 -0.5
-10 -5 0 5 10 -10 -5 0 5 10
jω jω
H(e ), N=15 H(e ), N=14
3 3
2 2
1 1
0 0
-5 0 5 -5 0 5
Figure 6.45 FIR filter design using N the most significant values of the impulse response.
Exercise 6.2. For a system whose transfer function is
z2 − 2
H (z) =
(z − 1)(z − 2)
plot the direct I and II realization, cascade realization, and parallel realiza-
tion.
Exercise 6.3. For a system whose transfer function is
3z−2 + 6
H (z) =
z−3 − 2z−2 + 3z−1 − 6
a) plot the direct realizations I and II, the cascade realization, and the parallel
realization.
b) Find ∑∞ n=−∞ h ( n ), where h ( n ) is the impulse response of the system.
E = 0.037954
0.5 r
-0.5
0 1 2 3 4 5 6
Er= 0.028921
0.5
-0.5
0 1 2 3 4 5 6
Figure 6.46 Error in the case of the frequency response sampling (top) and the IIR impulse
response truncation (bottom), along with the corresponding mean square error (Er ) value.
Exercise 6.4. Find the impulse response of the discrete system presented in
Fig.6.47.
Exercise 6.5. Using the impulse invariance method with the sampling step
∆t = 0.1, transform the analog system given with the transfer function
1 + 5s
H (s) =
8 + 2s + 5s2
into discrete, and plot the direct and cascade realization of the system. Is the
obtained discrete system stable?
Exercise 6.6. Using the bilinear transform with the sampling step ∆t = 1,
transform the system given with the transfer function
2+s
H (s) =
8 + 2s + 5s2
into discrete, and plot the direct and cascade realization of the system. Is the
obtained discrete system stable?
Exercise 6.7. Using the bilinear transform, with the sampling step ∆t = 0.2
transform the analog system given with the transfer function
3s + 6
H (s) =
(s + 1)(s + 3)
x(n) y(n)
+ +
z-1
4 -1
z-1
-5
+ +
0
-1
z
1/2 2
Figure 6.47 Discrete-time system.
jω
Hd(jΩ) Hd(e )
3 3
2 2
1 1
0 0
-5 - π - π/2 0 π/2 π 5 -5 - π - π/2 0 π/2 π 5
Figure 6.48 Desired system in the continuous-time and discrete-time domains.
into discrete, and plot the direct realization II of the discrete system.
Exercise 6.8. For a system whose frequency response in the continuous-time

domain is %
|Ω|
Hd ( jΩ) =
2− π/2 for |ω | < π2
0 elsewhere
with the corresponding Hd (e jω ) in the discrete-time domain obtained for

∆t = 1, and presented in Fig.6.48, find the FIR filter impulse response with
N = 7 and N = 8 using:

domain,
(c) Comment the sources of error in both cases.
Chapter 7
Discrete-Time Random Signals
signals cannot be described by simple mathematical func-
R
ANDOM
tions. Their values are not known in advance. These signals can be
described by stochastic tools only. Here we will restrict the analysis
to the discrete-time random signals. The first-order and the second-order
statistics will be considered.
7.1 BASIC STATISTICAL DEFINITIONS
7.1.1 Expected Value
The first-order statistics is the starting point in describing random signals.

The expected value, or the mean value, of a random signal is one of its basic
parameters. If we have a set of signal samples,
{ x (n)}, n = 1, 2, ..., N, (7.1)
the mean value of this set of signal values is calculated as
1
µx = ( x (1) + x (2) + ... + x ( N )).
N
Example 7.1. Consider a random signal x (n) whose one realization is given
in Table 7.1. Find the mean value of this signal. Find how many samples of
the signal are within the intervals [1, 10], [11, 20],...,[91, 100]. Plot the number
of occurrences of signal x (n) samples within these intervals as a function of
the interval range.
⋆The realization of signal x (n) defined in Table 7.1 is presented in
Fig.7.1.
319
320 Discrete-Time Random Signals
Table 7.1
A realization of random signal
54 62 58 51 70 43 99 52 57 57
56 53 38 61 28 69 87 41 72 72
23 26 66 47 69 71 69 81 68 68
31 55 52 23 60 34 83 39 66 66
37 12 54 42 67 95 89 67 42 42
35 55 54 55 49 77 18 64 73 73
67 56 42 66 50 47 49 25 50 50
61 84 48 67 71 74 35 59 60 60
40 77 52 63 57 42 44 64 36 36
66 39 50 31 11 75 45 62 60 60
120
x(n)
110
100
90
80
70
60 mean(x)
50
40
30
20
10
0
0 10 20 30 40 50 60 70 80 90 100
Figure 7.1 A realization of random signal x (n).
The mean value of all signal samples is
1 100
100 n∑
µx = x (n) = 55.76.
=1
25
20
15
10
0
0 10 20 30 40 50 60 70 80 90 100
Figure 7.2 Histogram of random signal x (n) with 10 intervals [10i + 1, 10i + 10], i = 0, 1, 2, ..., 9.
From Table 7.1 or the graph in Fig. 7.1 we can count that, for example,
there is no a signal sample whose value is within the interval [1, 10]. Within
[11, 20] there are two signal samples (x (42) = 12 and x (95) = 11). In a similar
way, the number of signal samples within other intervals are counted and
presented in Fig.7.2. This kind of random signal presentation is called a
histogram of x (n), with defined intervals.
Example 7.2. For the signal x (n) from the previous example assume that a new
random signal y(n) is formed as
! 6
x (n) + 5
y(n) = int ,
10
where int {◦} denotes the nearest integer. It means that y(n) = 1 for 1 ≤
x (n) ≤ 10, y(n) = 2 for 11 ≤ x (n) ≤ 20, ..., y(n) = i for 10(i − 1) + 1 ≤ x (n) ≤
10i up to i = 10. Plot the new signal y(n). What is the set of possible values of
y(n). Present on a graph how many times each of the possible values of y(n)
appeared in this signal realization. Find the mean value of the new signal
y(n) and discuss the result.
⋆ The signal y(n) is shown in Fig.7.3. This signal assumes values from
the set {2, 3, 4, 5, 6, 7, 8, 9, 10}.
For the signal y(n), instead of histogram we can plot a diagram of the
number of occurrences of each value that y(n) can assume. It is presented in
11
y(n)
10
7
mean(y)
6
0
0 10 20 30 40 50 60 70 80 90 100
Figure 7.3 Random signal y(n).
Fig.7.4. The mean value of y(n) is
1 100
100 n∑
µy = y(n) = 6.13.
=1
The mean value can also be written, by grouping the same values of y(n), as
1
µy = (1 · n1 + 2 · n2 + 3 · n3 + ... + 10 · n10 ) =
100
n n n n
= 1 · 1 + 2 · 2 + 3 · 3 + ... + 10 · 10 ,
N N N N
where N = 100 is the total number of signal values and ni is the number
showing how many times each of the values i appeared in y(n). If there is a
sufficient number of occurrences for each outcome value i then
n
Py (i ) = i
N
can be considered as the probability that the value i appears. In that sense
µy = 1 · Py (1) + 2 · Py (2) + 3 · Py (3) + ... + 10 · Py (10)

10
= ∑ y(i) Py (i)
i =1
25 0.25
P (i)
y
20 0.2
15 0.15
10 0.1
5 0.05
0 0
0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10
Figure 7.4 Number of appearances of each possible value of y(n) (left) and the probabilities
that the random signal y(n) takes a value i = 1, 2, . . . , 10 (right).
with
10
∑ Py (i) = 1.
i =1
Values of probability Py (i ) are shown in Fig.7.4.
In general, the mean for each signal sample could be different. For
example, if the signal values represent the highest daily temperature during
a year then the mean value is highly dependent on the considered sample.
In order to calculate the mean value of temperature, we have to have several
realizations of these random signals (measurements over M years), denoted
by { xi (n)}, where argument n = 1, 2, 3, ... is the cardinal number of the day
within a year and i = 1, 2, ..., M is the index of realization (year index). The
mean value is then calculated as
1 1 M
M i∑
µ x (n) = ( x1 (n) + x2 (n) + ... + x M (n)) = x i ( n ), (7.2)
M =1
for each n. In this case we have a set (a signal) of mean values {µ x (n)}, for
n = 1, 2, ..., 365.
Example 7.3. Consider a signal x (n) with realizations given in Table 7.2. Its values
are equal to the monthly average of maximal daily temperatures in a city
measured from year 2001 to 2015. Find the mean temperature for each month
over the considered period of years. What is the mean value of temperature
over all months and years? What is the mean temperature for each year?
Table 7.2
Average of maximal temperatures value within months over 15 years, 2001-2015.
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
10 4 18 17 22 29 30 28 27 17 17 5
6 7 11 23 22 32 35 33 22 26 22 8
10 11 10 16 21 26 32 31 23 19 17 4
3 11 13 19 22 26 34 29 26 22 12 9
7 10 13 21 27 29 30 34 24 20 16 11
7 11 17 17 27 25 37 34 33 22 14 14
7 12 13 19 23 32 34 38 21 21 12 10
12 5 9 20 21 37 34 34 27 22 20 7
7 12 13 23 27 33 29 31 25 21 6 11
8 12 10 17 27 33 38 32 23 20 15 9
8 10 13 24 23 33 33 31 27 21 16 8
4 6 15 18 25 26 27 33 23 23 13 11
3 6 16 17 27 28 30 32 29 24 12 10
11 12 14 18 22 29 34 34 23 21 20 11
6 13 8 22 22 29 30 34 23 18 15 8
⋆The signal for years 2001 to 2007 is presented in Fig.7.5. The mean
temperature for the nth month, over the considered years, is
1 15
15 i∑
µ x (n) = x20i (n),
=1
where the notation 20i is symbolic in the sense, 2001, 2002, ... 2015, for
i = 01, 02, ..., 15. The mean-value signal µ x (n) is presented in the last subplot
of Fig. 7.5. The mean value over all months and years is
12 15
1
15 · 12 n=1 i∑
∑
µx = x20i (n) = 19.84.
=1
The mean value for each of the considered years is
1 12
12 n∑
µ x (20i ) = x20i (n).
=1
45 45
x2001(n) x2002(n)
35 35
25 25
15 15
5 5
-5 -5
1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12
45 45
x2003(n) x2004(n)
35 35
25 25
15 15
5 5
-5 -5
1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12
45 45
x2005(n) x2006(n)
35 35
25 25
15 15
5 5
-5 -5
1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12
45 45
x (n) µx(n)
2007
35 35
25 25
15 15
5 5
-5 -5
1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12
Figure 7.5 Several realizations of a random signal x20i (n), for i = 01, 02, ..., 07 and the mean
value µ x (n) for each sample (month) over 15 available realizations.
7.1.2 Probability and Probability Density Function
If the probabilistic description of a random signal is known, then we can

calculate the mean value and other parameters of random signals. For the
first-order statistics calculation, it is sufficient to know the probabilities or
the probability density function.
If a random signal assumes only discrete values in amplitude {ξ 1 , ξ 2 , ...},
then we deal with probabilities,
Probability { x (n) = ξ i } = Px(n) (ξ i ). (7.3)
Probability function Px(n) (ξ ) satisfies the following properties:

1) 0 ≤ Px(n) (ξ ) ≤ 1 for any ξ.
2) For the events x (n) = ξ i and x (n) = ξ j , i ̸= j, which exclude each
other
R S
Probability x (n) = ξ i or x (n) = ξ j = Px(n) (ξ i ) + Px(n) (ξ j ).
3) The sum of probabilities that x (n) takes any value ξ i over the set A
of all possible values of ξ is a certain event. Its probability is 1,
∑ Px(n) (ξ ) = 1.
ξ∈A
An impossible event has the probability 0.

If x (n) and x (m) are statistically independent random samples then
R S
Probability x (n) = ξ i and x (m) = ξ j = Px(n) (ξ i ) Px(m) (ξ j ).
An example of a signal when the probabilities are calculated after the

experiment (a posteriori) is already presented within the first example. A
posteriori probability that the signal x (n) assumes value ξ i is defined as a
ratio of the number Nξ i of appearances of the event x (n) = ξ i and the total
number of signal values (experiments) N
Nξ i
Px(n) (ξ i ) =
N
for a sufficiently large N and Nξ i .
In some cases it is possible to find the probability of an event before the
experiment is performed. For example, if a signal is equal to the numbers
appearing in die tossing, then the signal may assume one of the values from
the set ξ i ∈ {1, 2, 3, 4, 5, 6}. In this case, the probability of each event is known
in advance (a priori). It is P(ξ i ) = 1/6.
Example 7.4. Consider a random signal whose values are equal to the numbers ap-
pearing in a die tossing. The set of possible signal values is ξ i ∈ {1, 2, 3, 4, 5, 6}.
Find
Probability { x (n) = 2 or x (n) = 5}
and
Probability { x (n) = 2 and x (n + 1) = 5} .
⋆Events that x (n) = 2 and x (n) = 5 are obviously mutually exclusive. Thus,
1 1 1
Probability { x (n) = 2 or x (n) = 5} = Px(n) (2) + Px(n) (5) = + = .
6 6 3
The events that x (n) = 2 and x (n + 1) = 5 are statistically independent. In
this case
11 1
Probability { x (n) = 2 and x (n + 1) = 5} = Px(n) (2) Px(n) (5) = = .
66 36
Example 7.5. Assume that a signal x (n) length is N and that the number of samples
disturbed by an extremely high noise is I. The observation set of signal
samples is taken as a set of M < N randomly positioned signal samples. What
is the probability that within M randomly selected signal samples there are
no samples affected by the high noise? If N = 128, I = 16, and M = 32 find
how many sets of M samples without high noise can be expected in 1000
realizations (trials).
⋆Probability that the first randomly chosen sample is not affected by
the high noise could be calculated as a priori probability,
N−I
P (1) =
N
since there are N samples in total and N − I of them are noise-free. Probability
that the first randomly chosen sample is not affected by high noise and that,
at the same time, the second randomly chosen sample is not affected by high
noise is equal to to the product of their probabilities,
N− I N−1− I
P (2) = .
N N−1
Here we used so called conditional probability property stating that
the probability that both events A and B occur is
Probability { A and B} = P( A) P( B/A),
where P( A) is the probability that event A occurs, while P( B/A) denotes the
probability that event B occurs subject to the condition that event A already
occurred.
Then we continue the process of random samples selection. In the same

way we can calculate the probability that all of M randomly chosen samples
are not affected by the high noise as
M −1
N−I−i
P( M) = ∏ N−i
.
i =0
For N = 128, I = 16, and M = 32 we get
P(32) = 0.0112.
It means that if we repeat the whole procedure 1000 times (1000 realizations)
we can expect
P(32) × 1000 = 11.2,
i.e., about 11 realizations when none of M signal samples is disturbed by the
high noise.
The mean value is calculated as a sum over the set of possible ampli-
tudes, weighted by the corresponding probabilities,
∞
µ x (n) = E{ x (n)} = ∑ ξ i Px(n) (ξ i ). (7.4)
i =1
If a random signal can assume continuous values in amplitude then

we cannot define a probability that one exact signal amplitude value is
assumed. In that case the probability density function p x(n) (ξ ) is used. It
defines the probability that the nth signal sample x (n) takes a value within
an infinitesimally small interval dξ around ξ,
Probability {ξ ≤ x (n) < ξ + dξ )} = p x(n) (ξ )dξ. (7.5)
Properties of the probability density function are:

1) It is nonnegative, p x(n) (ξ ) ≥ 0 for any ξ
2) Since Probability {−∞ < x (n) < ∞} = 1, then
"∞
p x(n) (ξ )dξ = 1.
−∞
The probability of an event that a value of signal x (n) is within a ≤ x (n) < b
is
"a
Probability { a ≤ x (n) < b} = p x(n) (ξ )dξ.
b
Cumulative distribution F (χ) function is the probability that a signal x (n)

value is lower than χ,
"χ
F (χ) = Probability { x (n) < χ} = p x(n) (ξ )dξ.
−∞
Obviously limχ→−∞ F (χ) = 0, limχ→+∞ F (χ) = 1, and F ( a) ≥ F (b) if a > b.

Note that
dF (ξ )
p x (n) (ξ ) = .
dξ
The expected value of a random variable x (n) in terms of the proba-
bility density function, is
"∞
µ x (n) = E{ x (n)} = ξ p x(n) (ξ )dξ. (7.6)
−∞
7.1.3 Median
In addition to the mean value, a median is used for description of a set of

random values. The median is a value in the middle of the set, after the
members of the set are sorted. If we denote the sorted values of x (n) as s(n)
s(n) = sort{ x (n)}, n = 1, 2, ..., N
then the median value is

* +
N+1
median{ x (n)} = s , for an odd N.
2
If N is an even number then the median is defined as the mean value of two
samples nearest to ( N − 1)/2,
B C B C
N N
s 2 +s 2 +1
median{ x (n)} = , for an even N.
2
The median will not be influenced by a possible small number of big outliers
(signal values being significantly different from the values of the rest of
data).
120
sort(x)
110
100
90
80
70
60 median(x)
50
40
30
20
10
0
0 10 20 30 40 50 60 70 80 90 100
Figure 7.6 Sorted values and the median of x (n).
Example 7.6. Find the median of sets

(a) A = {−1, 1, −2, 4, 6, −9, 0}, (b) B = {−1, 1, −1367, 4, 35, −9, 0}, and (c)
of the signal x (n) from Example 7.1.
⋆(a) After sorting the values in set A we get A = {−9, −2, −1, 0, 1, 4, 6}.
Thus, median( A) = 0. (b) In a similar way median ( B) = 0. The mean values
of these data would significantly differ. (c) The sorted values of x (n) are
presented in Fig. 7.6. Since the number of samples of signal x (n) is N = 100
there is no single sample in the middle of the sorted sequence. The middle
is between sorted samples 50 and 51. Thus the median in this situation is
defined as the mean value of the 50th and 51st sorted sample.
In some cases the number of big outliers is small. Thus the median will
neglect many signal values that could produce a good estimate of the mean
value. In that cases, the best choice would be to use not only the mid-value
in the sorted signal, but several samples of the signal around its median and
to calculate their mean, for odd N, as
L * +
1 N+1
LSmean{ x (n)} = ∑
2L + 1 i=−
s
2
+ i .
L
With L = ( N − 1)/2 all signal values are used and LSmean { x (n)} is the
standard mean of a signal. With L = 0 the value of LSmean{ x (n)} is the
standard median. In general, this way of signal parameters estimation is the

L-statistics based estimation.
7.1.4 Variance
For random signals that take values from a discrete set, with known proba-
bilities, the variance is defined as
σx2(n) = E{| x (n) − µ x (n)|2 }

' '2
' '
= ∑ 'ξ − µ x(n) ' Px(n) (ξ ).
ξ
For a random signal x (n) whose values are available in M realizations the
variance can be estimated as a mean square deviation of the signal values
from their corresponding mean values µ x (n),
1 B C
σx2 (n) = | x1 (n) − µ x (n)|2 + ... + | x M (n) − µ x (n)|2 .
M
The standard deviation is a square root of the variance. The standard

deviation can be estimated as a square root of the mean of squares of the
centered data,
=
1 B C
σx (n) = | x1 (n) − µ x (n)|2 + ... + | x M (n) − µ x (n)|2 . (7.7)
M
For a small number of samples, this estimate tends to produce lower values
of the standard deviation. Thus, an adjusted version, the sample standard
deviation, is also used. It reads
=
1 B C
σx (n) = |( x1 (n) − µ x (n))|2 + ... + | x M (n) − µ x (n)|2 .
M−1
This form confirms the fact that in the case when only one sample is
available, M = 1, we should not be able to estimate the standard deviation.
For the case of random signals whose amplitude is continuous the
variance, in terms of the probability density function p x(n) (ξ ), is
"∞ ' '2

' '
σx2(n) = 'ζ − µ x(n) ' p x(n) (ξ )dξ.
−∞
Table 7.3
Random signal z(n)
55 57 56 54 59 52 66 54 56 56
55 55 51 56 48 59 63 52 59 59
47 48 58 53 58 59 59 61 58 58
49 55 54 47 56 50 62 51 58 58
50 44 55 50 58 58 63 58 52 52
50 55 55 55 53 60 46 57 59 59
58 55 58 58 54 53 54 48 54 54
57 62 53 58 59 60 50 56 56 56
51 60 54 57 55 52 52 57 50 50
58 51 54 49 44 60 52 57 56 56
120
z(n)
110
100
90
80
70
60 mean(z)
50
40
30
20
10
0
0 10 20 30 40 50 60 70 80 90 100
Figure 7.7 Random signal z(n).
Example 7.7. For the signal x (n) from Example 7.1 calculate the mean and vari-
ance. Compare it with the mean and variance of the signal z(n) given in Table
7.3.
⋆The mean value and variance for signal x (n) are µ x = 55.76 and
σx2 = 314.3863. The standard deviation is σx = 17.7309. It is a measure of signal
value deviations from the mean value. For the signal z(n) the mean value
is µz = 55.14 (very close to µ x ), while the variance is σz2 = 18.7277 and the
standard deviation is σz = 4.3275. Deviations of z(n) from the mean value
are much smaller. If signals x (n) and z(n) were measurements of the same
physical value, then the individual measurements from z(n) would be much
more reliable than the individual measurements from x (n).
Example 7.8. A random signal x (n) can take values from the set {0, 1, 2, 3, 4, 5}. It
is known that for k = 1, 2, 3, 4 the probability of x (n) = k is twice higher than
the probability of x (n) = k + 1. Find the probabilities P{ x (n) = k }. Find the
mean value and variance of signal.
⋆Assume that P{ x (n) = 5} = A. Then the probabilities that x (n) takes
a value k are
k 0 1 2 3 4 5
P{ x (n) = k} 32A 16A 8A 4A 2A A
Constant A can be found from ∑k P{ x (n) = k } = 1. It results in A = 1/63.
Now we have
19
µ x(n) = ∑ kP{ x (n) = k} =
k
21
* +2
19 626
σx2(n) = ∑ k − P{ x (n) = k } = .
k
21 441
Example 7.9. Consider a real-valued random signal x (n) with samples whose
values are uniformly distributed over interval −1 ≤ x (n) ≤ 1. a) Find the
mean value and variance of the signal samples. b) Signal y(n) is obtained as
y(n) = x2 (n). Find the mean value and variance of signal y(n).
⋆Since the random signal x (n) is uniformly distributed, its probability
density function is of the form
!
A for |ξ | ≤ 1
p x (n) (ξ ) = .
0 for |ξ | > 1
&∞
Constant A = 1/2 is obtained from −∞ p x(n) (ξ )dξ = 1. Now we have
"∞ "1
1
µ x (n) = ξ p x(n) (ξ )dξ = ξdξ = 0
2
−∞ −1
"∞ "1
1 2 1
σx2(n) = (ξ − µ x(n) )2 p x(n) (ξ )dξ = ξ dξ = .
2 3
−∞ −1
The probability that y(n) is not higher than ξ is

, ,
Fy (ξ ) = P{y(n) ≤ ξ } = P{ x2 (n) ≤ ξ } = P{− ξ < x (n) ≤ ξ }
⎧ ⎧
⎪
⎨ &√ 0 for ξ ≤ 0 ⎨ √0 for ξ ≤ 0
ξ
= √ p x (n) ( ξ ) dξ for 0 < ξ < 1 = ξ for 0 < ξ < 1
⎩ − ξ
⎪ ⎩
1 for ξ ≥ 1
1 for ξ ≥ 1
since y(n) ≤ ξ when x2 (n) ≤ ξ. The probability density function is

%
1
dF (ξ ) √
2 ξ
for 0 < ξ ≤ 1
py(n) (ξ ) = =
dξ 0 otherwise.
The mean value and variance of signal y(n) are
"1
1 1
µy(n) = ξ √ dξ =
2 ξ 3
0
"1
1 1 4
σy2(n) = (ξ − )2 √ dξ = .
3 2 ξ 45
0
Note: Generalize for z(n) = f ( x (n)).
As an introduction to the second-order statistics consider two signals

x (n) and y(n) with continuous amplitude values. Probability that the nth
signal sample x (n) takes a value within ξ ≤ x (n) < ξ + dξ and that y(m)
takes a value within ζ ≤ y(m) < ζ + dζ is
Probability{ξ ≤ x (n) < ξ + dξ ), ζ ≤ y(m) < ζ + dζ )} = p x(n),y(m) (ξ, ζ )dξdζ,
where p x(n),y(m) (ξ, ζ ) is the joint probability density function. The probabil-
ity of an event a ≤ x (n) < b and c ≤ y(m) < d is
"a "d
Probability { a ≤ x (n) < b, c ≤ y(m) < d} = p x(n),y(m) (ξ, ζ )dξdζ.
b c
For mutually independent signals p x(n),y(m) (ξ, ζ ) = p x(n) (ξ ) py(m) (ζ ). A spe-

cial case of the previous relations is obtained when y(m) = x (m).
Example 7.10. Signal x (n) is defined as x (n) = a(n) + b(n) + c(n) where a(n),
b(n), and c(n) are mutually independent random signals with a uniform
probability density function over the range [−1, 1). Find the probability
density function of signal x (n), its mean µ x , and variance σx2 .
⋆Consider a sum of two independent random signals s(n) = a(n) +

b(n). The probability that s(n) = a(n) + b(n) < θ can be calculated from the
joint probability distribution of a(n) and b(n) as
F (θ ) = P{s(n) < θ }
= Probability{−∞ < a(n) < ∞, −∞ < a(n) + b(n) ≤ a < θ }
"∞ θ"−ζ "∞ θ"−ζ
= p a(n),b(n) (ξ, ζ )dξdζ = pb(n) (ζ ) p a(n) (ξ )dξdζ.
−∞ −∞ −∞ −∞
Now we can calculate the probability density function of s(n) as a derivative
"∞ θ"−ζ
dF (θ ) d
ps(n) (θ ) = = pb(n) (ζ ) p a(n) (ξ )dξdζ
dθ dθ
−∞ −∞
"∞
= pb(n) (ζ ) p a(n) (θ − ζ )dζ = pb(n) (θ ) ∗θ p a(n) (θ ),
−∞
meaning that the probability density function of a sum of two independent

random variables is a convolution of the individual probability density func-
tions. In a similar way we can include the third signal and obtain
p x ( n ) ( θ ) = p c ( n ) ( θ ) ∗ θ p b ( n ) ( θ ) ∗ θ p a ( n ) ( θ ),
⎧
⎪ ( θ +3)2
⎪
⎪ 16 for − 3 ≤ θ ≤ −1
⎪
⎪
⎪
⎨ 3−8θ
2
for − 1 < θ ≤ 1
p x (n) (θ ) = ( θ −3)2 .
⎪
⎪ for 1 < θ ≤ 3
⎪
⎪ 16
⎪
⎪
⎩ 0 for |θ | > 3
The mean value and variance can be calculated from p x(n) (θ ), or in direct
way, as
µ x = E{ x (n)} = E{ a(n)} + E{b(n)} + E{c(n)} = 0

σx2 = E{( x (n) − µ x )2 } = E{( a(n) + b(n) + c(n))2 }
= E{ a ( n )2 } + E{ b ( n )2 } + E{ c ( n )2 } + 2 ( µ a µ b + µ a µ c + µ b µ c )
1 1 1
= + + = 1.
3 3 3
7.2 SECOND-ORDER STATISTICS
7.2.1 Correlation and Covariance
Second-order statistics deals with two samples of random signals.

For a signal { xi (n)}, n = 1, 2, ..., N and i = 1, 2, ..., M, being the number
of realizations of this signal, the autocorrelation function is defined by
1 M
r xx (n, m) = E{ x (n) x ∗ (m)} = xi (n) xi∗ (m).
M i∑
(7.8)
=1
If the probability that a real-valued random signal x (n) assumes a

value ξ 1 and that x (m) assumes ξ 2 is Px(n),x(m) (ξ 1 , ξ 2 ) then
r xx (n, m) = ∑ ∑ ξ 1 ξ 2 Px(n),x(m) (ξ 1 , ξ 2 ). (7.9)

ξ1 ξ2
For a real-valued random signal with continuous amplitudes and the

second-order probability density function p x(n),x(m) (ξ 1 , ξ 2 ), the autocorrela-
tion is
"∞
r xx (n, m) = ξ 1 ξ 2 p x(n),x(m) (ξ 1 , ξ 2 )dξ 1 dξ 2 . (7.10)
−∞
If the real-valued random variables x (n) and x (m) are statistically

independent, then p x(n),x(m) (ξ 1 , ξ 2 ) = p x(n) (ξ 1 ) p x(m) (ξ 2 ) and r xx (n, m) =
µ x ( n ) µ x ( m ).
The autocovariance function is defined by
c xx (n, m) = E{( x (n) − µ x (n)) ( x (m) − µ x (m))∗ }

1 M
( xi (n) − µ x (n)) ( xi (m) − µ x (m))∗ .
M i∑
= (7.11)
=1
It may be easily shown that
c xx (n, m) = E{( x (n) − µ x (n)) ( x (m) − µ x (m))∗ } = r xx (n, m) − µ x (n)µ∗x (m).
Value of the autocovariance for m = n is the variance
σx2 (n) = E{| x (n) − µ x (n)|2 } = r xx (n, n) − |µ x (n)|2 . (7.12)

The cross-correlation and the cross-covariance of two signals x (n) and

y(n) are defined as
r xy (n, m) = E{ x (n)y∗ (m)}
and
c xy (n, m) = E{( x (n) − µ x (n)) (y(m) − µy (m))∗ } (7.13)

= r xy (n, m) − µ x (n)µy∗ (m).
7.2.2 Stationarity and Ergodicity
Signals whose first-order and second-order statistics are invariant to a shift

in time are called wide sense stationary (WSS) signals. For the WSS signals
holds
µ x (n) = E{ x (n)} = µ x
r xx (n, m) = E{ x (n) x ∗ (m)} = r xx (n − m). (7.14)
A signal is stationary in the strict sense (SSS) if all order statistics are
invariant to a shift in time. The relations introduced for the second-order
statistics may be extended to the higher-order statistics. For example, the
third-order moment of a signal x (n) is defined by
Mxxx (n, m, l ) = E{ x (n) x ∗ (m) x ∗ (l )}. (7.15)
For stationary signals it assumes the form
Mxxx (m, l ) = E{ x (n) x ∗ (n − m) x ∗ (n − l )}.
In order to calculate the third-order moment we should know the third-

order statistics, like the third-order probability Px(n),x(m),x(l ) (ξ 1 , ξ 2 , ξ 3 ) or
probability density function.
For a random process, as collection of all realizations of a random sig-
nal along with its probabilistic description, we say that it is ergodic if its
parameters can be estimated by averaging over time instead of over real-
izations. The process is ergodic in parameter β if that particular parameter
can be estimated by averaging over time instead of over realizations. If a
random signal x (n) is a realization of a process ergodic in mean then
1
µ x (n) = lim ( x (n) + x2 (n) + ... + x M (n))
M 1
M→∞
1
= lim ( xi (n) + xi (n − 1) + ... + xi (n − N + 1)).
N →∞ N
7.2.3 Power Spectral Density
For stationary signals the autocorrelation function is
r xx (n) = E{ x (n + m) x ∗ (m)} = r xx (n).
The Fourier transform of the autocorrelation function of a WSS signal is the

power spectral density
∞
Sxx (e jω ) = ∑ r xx (n)e− jωn (7.16)
n=−∞
"π
1
r xx (n) = Sxx (e jω )e jωn dω. (7.17)
2π
−π
Integral of Sxx (e jω ) over frequency,

"π
1
Sxx (e jω )dω = r xx (0) = E{| x (n)|2 }, (7.18)
2π
−π
is equal to the average power of the random signal.

Example 7.11. Find the mean, autocorrelation, and power spectral density of the
random signal
K
x (n) = ∑ ak e j(ω n+θ ) ,
k k
k =1
where θk are random variables uniformly distributed over −π < θk ≤ π.
All random variables are statistically independent. Frequencies ωk are −π <
ωk ≤ π for each k.
⋆The mean value is
K K "π
j ( ωk n + θ k ) 1 j ( ωk n + θ k )
µx = ∑ a k E{ e }= ∑ ak 2π
e dθk = 0.
k =1 k =1 −π
The autocorrelation is
K K K
r xx (n) = E{ ∑ ak e j(ωk (n+m)+θk ) ∑ ak e− j(ω m+θ ) } = ∑ a2k e jω n ,
k k k
k =1 k =1 k =1
while the power spectral density for −π < ω ≤ π is
K
Sxx (e jω ) = FT{r xx (n)} = 2π ∑ a2k δ(ω − ωk ).
k =1
Remind that the average signal power of a signal x (n) has been de-
fined as
1 N # $
PAV = lim ∑ | x (n)|2 = | x (n)|2 .
N →∞ 2N + 1 n=− N
This relation leads to another definition of the power spectral density

of random discrete-time signals
1 ' '2
' '
Pxx (e jω ) = lim E{'X N (e jω )' } (7.19)
N →∞ 2N + 1
' '2
1 ' N '
' '
= lim E{' ∑ x (n)e− jωn ' }.
N →∞ 2N + 1 'n=− N '
Different notation is used since the previous two definitions, (7.16) and
(7.19) of power spectral density, will not produce the same result, in general.
We can write
N N
1
Pxx (e jω ) = lim E{ ∑ ∑ x (m) x ∗ (n)e− jω (m−n) }.
N →∞ 2N + 1 m=− N n=− N
For a stationary signal

N N
1
Pxx (e jω ) = lim ∑ ∑ r xx (m − n)e− jω (m−n) .
N →∞ 2N + 1 m=− N n=− N
Double summation is performed within a square in the two-dimensional

domain defined by − N ≤ m ≤ N, − N ≤ n ≤ N. Since the terms within
double sum are functions of (m − n) only, then the summation could be
performed along the lines where (m − n) = k is constant. For (m − n) = k = 0
the summation line is the main diagonal of area − N ≤ m ≤ N, − N ≤ n ≤ N.
Along this diagonal there are 2N + 1 points where r xx (m − n)e− jω (m−n) =
r xx (0). For the nearest subdiagonals of − N ≤ m ≤ N, − N ≤ n ≤ N
when (m − n) = k = ±1 there are 2N points where r xx (m − n)e− jω (m−n) =
r xx (±1)e± jω . For arbitrary lines (m − n) = ±k, with |k | ≤ 2N, there are
2N + 1 − |k | terms with r xx (m − n)e− jω (m−n) = r xx (±k )e± jkω . It means that
we can write
2N
1
Pxx (e jω ) = lim
N →∞ 2N + 1
∑ (2N + 1 − |k|)rxx (k)e− jωk
k =−2N
2N 2N
|k |
= lim ∑ (1 − )r xx (k)e− jωk = lim ∑ w B (k )r xx (k )e− jωk .
N →∞
k =−2N
2N + 1 N →∞
k=−2N
Function w B (k) corresponds to a Bartlett window over the calculation in-

terval. If the values of autocorrelation function r xx (k ) are such that the sec-
ond part of the sum ∑k |k |/(2N + 1)r xx (k )e− jωk is negligible as compared to
∑k r xx (k )e− jωk then
2N
Pxx (e jω ) = lim ∑ r xx (k )e− jωk = FT{r xx (n)} = Sxx (e jω ).
N →∞
k=−2N
This is true for r xx (k ) = Cδ(k ). Otherwise Pxx (e jω ) is a smoothed version

of Sxx (e jω ). Note that Pxx (e jω ) is always nonnegative, by definition (for a
numeric illustration see Example 7.23).
7.3 NOISE
In many applications, the desired signal is disturbed by various forms of

random signals, caused by numerous factors in the signal sensing, trans-
mission, and/or processing. Often, a cumulative influence of these factors,
disturbing useful signal, is described by an equivalent random signal, called
noise. In most cases we will use a notation ε(n) for these kinds of signals.
They model a random, multiple source, disturbance.
A noise is said to be white if its values are uncorrelated
rεε (n, m) = σε2 δ(n − m) (7.20)

jω
Sεε (e ) = FT{r xx (n)} = σε2 .
Spectral density of this kind of noise is constant (like it is the case in the
white light). If this property is not satisfied, then the power spectral density
is not constant. Such a noise is referred to as colored.
Regarding to the distribution of noise ε(n) amplitudes the most com-
mon types of noise in signal processing are: uniform, binary, Gaussian, and
impulsive noise.
7.3.1 Uniform Noise
The uniform noise is a signal with the probability density function
1
pε(n) ( ξ ) = , for − ∆/2 ≤ ξ < ∆/2 (7.21)
∆
1.5 1.5
p ( ξ)
x
1 1
0.5 0.5
0 0
-0.5 -0.5
-1 -1
-1.5 -1.5
0 10 20 30 40 50 60 0 0.5 1 1.5
Figure 7.8 A realization of uniform noise (left) with probability density function (right) with
∆ = 0.5.
and pε(n) (ξ ) = 0 elsewhere, Fig.7.8. Its variance is
∆/2
"
∆2
σε2 = ξ 2 pε(n) (ξ )dξ = .
12
−∆/2
This kind of noise is used to model rounding errors in the amplitude

quantization of a signal. It indicates that all errors within −∆/2 ≤ ξ < ∆/2
are equally probable.
7.3.2 Binary Noise
Random binary sequence, or binary noise, is a stochastic signal which

randomly assumes one of two fixed signal values. Assume that the noise
ε(n) values are, for example, {−1, 1} and that the probability that ε(n)
assumes value 1 is p. The mean of this noise is
µε = ∑ ξPx (ξ ) = (−1)(1 − p) + 1 · p = 2p − 1.
ξ =−1,1
The variance is
σε2 = ∑ (ξ − µε )2 Px (ξ ) = 4p(1 − p).

ξ =−1,1
A special case is when the values from the set {−1, 1} are equally probable,
that is when p = 1/2. Then we get µε = 0 and σε2 = 1.
Example 7.12. Consider a set of N → ∞ balls. Equal number of balls is marked with
1 (or white) and 0 (or black). A random signal x (n) corresponds to drawing of
four balls in a row. It has four values x (0), x (1), x (2), and x (3). Signal values
x (n) are equal to the marks on the drawn balls. Write all possible realizations
of x (n). If k is the number of appearances of value 1 in the signal, write the
probabilities for each value of k.
⋆Signal realizations, with the number k being the number of appear-

ances of digit 1 in each signal realization, are given in the next table.
x (0) 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1
x (1) 0 0 0 0 1 1 1 1 0 0 0 0 1 1 1 1
x (2) 0 0 1 1 0 0 1 1 0 0 1 1 0 0 1 1
x (2) 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1
k 0 1 1 2 1 2 2 3 1 2 2 3 2 3 3 4
Possible values of k are 0, 1, 2, 3, 4 with corresponding probabilities

B C B C B C
1111
P (0) = 1 · 2222 P(1) = 4 · 12 12 12 12 ,
, P(2) = 6 · 12 12 12 12 ,
B C
P(3) = 4 · 12 12 12 12 , and P(4) = 1 · 12 12 12 12 .
These probabilities can be considered as the terms of a binomial expression
( a + b )4
B C B C B C B C B C
4 4
= 0 a4 + 1 a3 b + 42 a2 b2 + 43 ab3 + 44 b4
with a = 1/2 and b = 1/2. For the case when N is a finite number see Problem
7.6.
An interesting form of the random variable that can assume only two
possible values {−1, 1} or {No, Yes} or { A, B} is the binomial random
variable. It has been introduced through the previous simple example. In
general, if a signal x (n) assumes value B from the set { A, B} with probability
p, then the probability that there is exactly k values of B in a sequence of N
samples of x (n) is
B C
P(k) = N
k p k (1 − p ) N − k
N!
= p k (1 − p ) N − k .
k!( N − k )!
This is a binomial coefficients form.

The expected value of the number of appearances of event B in N

samples, denoted by y, is
N
µ y = E{ y } = ∑ kP(k)
k =0
N
N!
= ∑ k k!( N − k)! pk (1 − p) N −k .
k =0
Since the first term in summation is 0 we will shift the summation for one
and reindex it to
N −1
N ( N − 1) !
µ y = E{ y } = ∑ (k + 1) (k + 1)!(( N − (k + 1))! pk+1 (1 − p) N−(k+1)
k =0
N −1
( N − 1) !
= Np ∑ pk (1 − p)( N −1)−k .
k =0
k!(( N − 1) − k )!
The sum in the last expression is equal to
N −1 B C
1 = ( p + (1 − p)) N −1 = ∑ N −1
k pk (1 − p)( N −1)−k
k =0
N −1
( N − 1) !
= ∑ pk (1 − p)( N −1)−k
k =0
k!(( N − 1) − k )!
resulting, with p + (1 − p) = 1, into
µy = E{y} = N p.
As we could write from the beginning , the expected value of the number
of appearances of an event B, whose probability is p, in N realizations is
E{y} = N p. This derivation was performed not only to prove this fact, but
it will lead us to the next step in deriving the variance of the event y, by
using the expected value of the product of y and y − 1,
N
E{y(y − 1)} = ∑ k ( k − 1) P ( k )
k =0
N
N!
= ∑ k(k − 1) k!( N − k)! pk (1 − p) N −k .
k =0
Since the first two terms are 0 we can reindex the summation into
N −2
N!
E{y(y − 1)} = ∑ (k + 2)(k + 1) (k + 2)!( N − 2 − k)! pk+2 (1 − p) N−2−k
k =0
N −2
( N − 2) !
= N ( N − 1) p2 ∑ p k (1 − p ) N −2− k .
k =0
k!( N − 2 − k )!
The relation
N −2
( N − 2) !
∑ pk (1 − p) N −2−k = ( p + (1 − p)) N −2 = 1
k =0
k!( N − 2 − k )!
is used to get
E{y(y − 1)} = N ( N − 1) p2 .
The variance of y follows from
σy2 = E{y2 } − (E{y})2

= E{y(y − 1)} + E{y} − (E{y})2
= N p (1 − p ).
Therefore, in a sequence of N values of signal x (n) that can assume values

{ A, B} the mean value and variance of appearances of B divided by N will
be
Np
µy = =p
N
N p (1 − p ) p (1 − p )
σy2 = 2
=
N N
Increasing the number of the total values N the variance will be lower and
a finite set x (n) will produce a more reliable mean value p.
7.3.3 Gaussian Noise
The Gaussian (normal) noise is used to model a disturbance caused by many

small independent factors. Namely, the central limit theorem states that a
sum of a large number of statistically independent random variables, with
any distribution, obeys to the Gaussian (normal) distribution.
The Gaussian zero-mean noise has the probability density function
1 2 / (2σ2 )
pε(n) (ξ ) = √ e−ξ ε . (7.22)
σε 2π
3 3
ε(n) p ( ξ)
x
2 2
1 1
0 0
-1 -1
-2 -2
-3 -3
0 10 20 30 40 50 60 0 0.25 0.5
Figure 7.9 A realization of Gaussian noise (left) with probability density function (right).
Variance of this noise is σε2 . It is left to reader to prove this by evaluating

corresponding integral. For the Gaussian noise with mean µ and variance
σε2 we can use notation N (µ, σε2 ).
The probability that the amplitude of a zero-mean Gaussian random
variable takes a value smaller than λ is
"λ * +
1 2 / (2σ2 ) λ
Probability{|ε(n)| < λ} = √ e−ξ ε dξ = erf √ (7.23)
σε 2π 2σε
−λ
where
"λ
2 2
erf(λ) = √ e−ξ dξ
π
0
is the error function.
Commonly used probabilities that the absolute value of the noise is
within the standard deviation, two standard deviations (two-sigma rule),
or three standard deviations are:
√
Probability{−σε < ε(n) < σε } = erf(1/ 2) = 0.6827, (7.24)
√
Probability{−2σε < ε(n) < 2σε } = erf( 2) = 0.9545,
√
Probability{−3σε < ε(n) < 3σε } = erf(3/ 2) = 0.9973.
Example 7.13. Given 12 measurements of a Gaussian zero-mean noise {−0.7519,

1.5163, −0.0326, −0.4251, 0.5894, −0.0628, −2.0220, −0.9821, 0.6125, −0.0549,
−1.1187, 1.6360}, estimate the probability that the absolute value of this noise
will be smaller than 2.5.
0.5
px( ξ)
0.4
0.3
0.2
0.1
0
-4 -3 -2 -1 0 1 2 3 4
Figure 7.10 Probability density function with intervals corresponding to −σε < ε(n) < σε ,
−2σε < ε(n) < 2σε , and −3σε < ε(n) < 3σε . Value of σε = 1 is used.
⋆The standard deviation of this noise could be estimated by using (7.7)

with µ = 0 and N = 12. It is σ = 1.031. Thus, the absolute value of this noise
will be smaller than 2.5 with probability
"2.5 √
1 2 2
P= √ e−ξ /(2·1.031 ) dξ = erf(2.5/( 2 · 1.031)) = 0.9847.
1.031 2π
−2.5
Example 7.14. Consider a signal s(n) = Aδ(n − n0 ) and a zero-mean Gaussian

noise ε(n) with variance σε2 within the interval 0 ≤ n ≤ N − 1, where n0 is a
constant integer within 0 ≤ n0 ≤ N − 1. Find the probability of event A that
a maximum value of x (n) = s(n) + ε(n) is obtained at n = n0 .
⋆Probability density function for any sample x (n), n ̸= n0 , is
1 2 2
p x(n),n̸=n0 (ξ ) = √ e−ξ /(2σε ) .
σε 2π
The probability that any of these samples is smaller than a value of λ could
be defined by using (7.23)
P− (λ) = Probability{ x (n) < λ, n ̸= n0 }

Probability{ x (n) < 0, n ̸= n0 } + Probability{0 ≤ x (n) < λ, n ̸= n0 }
√
= 0.5 + 0.5 erf(λ/( 2σε )).
Since the random variables x (n), 0 ≤ n ≤ N − 1, n ̸= n0 , are statistically

independent, then the probability that all of them are smaller than a value
of λ is
−
PN −1 (λ ) = Probability{All N − 1 values of x (n ) < λ, n ̸ = n0 }
@ √ A N −1
= 0.5 + 0.5 erf(λ/( 2σε )) .
The probability density function of the sample x (n0 ) is a Gaussian function

with the mean value A,
1 2 2
p x ( n0 ) ( ξ ) = √ e−(ξ − A) /(2σε ) .
σε 2π
The probability that the random variable x (n0 ) takes a value around λ,
λ ≤ x (n0 ) < λ + dλ, is
1 2 2
Pn+0 (λ) = Probability{λ ≤ x (n0 ) < λ + dλ} = √ e−(ξ − A) /(2σε ) dλ (7.25)
σε 2π
The probability that all values of x (n), 0 ≤ n ≤ N − 1, n ̸= n0 are smaller than
λ and that, at the same time, λ ≤ x (n0 ) < λ + dλ is
- * +. N −1
− + λ 1 2 2
PA (λ) = PN −1 ( λ ) Pn 0
( λ ) = 0.5 + 0.5 erf √ √ e−(ξ − A) /(2σε ) dλ,
2σε σε 2π
while the total probability that all x (n), 0 ≤ n ≤ N − 1, n ̸= n0 are bellow
x (n0 ) is an integral over all possible values of λ
"∞ "∞ - * +. N −1
λ 1 2 2
PA = PA (λ) = 0.5 + 0.5 erf √ √ e−(ξ − A) /(2σε ) dλ.
2σε σε 2π
−∞ −∞
(7.26)
Example 7.15. Random signal x (n) is a Gaussian noise with the mean µ x = 1
and variance σx2 = 1. A random sequence y(n) is obtained by omitting
samples from signal x (n) that are either negative or higher that 1. Find the
probability density function of sequence y(n). Find its µy and σy .
⋆The probability density function for the sequence y(n) is
% ( ζ −1)2
py(n) (ζ ) = B √1 e− 2 for 0 < ζ ≤ 1
2π
0 otherwise
&∞
Constant B can be calculated from −∞ py(n) (ζ )dζ = 1, resulting in
1
B = 2/ erf( √ ).
2
Now we have
"1 √
2 1 − ( ζ −1)2 2(1 − e−1/2 )
µy(n) = ζ √ e 2 dζ = 1 − √ ≈ 0.54
erf( √1 ) 2π π erf( √1 )
0 2 2
"1 ( ζ −1)2
2 1
σy2(n) = ( ζ − µ y ( n ) )2 √ √ e− 2 dζ ≈ 0.08.
erf( 2) 2π
0
Example 7.16. Consider a random signal x (n) that can assume values
{No, Yes} with probabilities 1 − p and p. If a random realization of this
signal is available with N = 1000 samples and we obtained that the event
Yes appeared 555 times find the interval where the true p will be with
probability of 0.95. Denote by y the number of observed Yes values divided
by N. We can assume that the mean value estimates for various realizations
are Gaussian distributed.
⋆This is a binomial random variable with the mean p and the variance
555 555
p (1 − p ) ∼ 1000 (1 − 1000 ) 0.2470
σy2 = = =
N 1000 1000
σy = 0.0157.
Therefore the estimated value

555
p̂ =
1000
is within the range
D E
p̂ = 0.555 ∈ p − 2σy , p + 2σy
= [ p − 0.0314, p + 0.0314]
with probability 0.95, i.e.,
−0.0314 ≤ 0.555 − p ≤ 0.0314

|0.555 − p| ≤ 0.0314.
with the same probability. The true value is around 55.5% within 3.14%
range (from 52.36% to 58.64%) with probability 0.95. By increasing the value
of N we can reduce the margin of estimation error. However, about 1000
values are commonly used for various opinion poll estimations.
7.3.4 Complex Gaussian Noise and Rayleigh Distribution
In many application the complex-valued Gaussian noise is used as a model

for disturbance. Its form is
ε(n) = ε r (n) + jε i (n)
where ε r (n) and ε i (n) are real-valued Gaussian noises. Commonly it is

assumed that they are zero-mean, independent, with identical distributions
(i.i.d.), and variance σ2 /2.
The mean value of this noise is
µε = E{ε(n)} = E{ε r (n)} + jE{ε i (n)} = 0 + j0.
The variance is
σε2 = E{|ε(n)|2 } = E{ε(n)ε∗ (n)}

= E{ε r (n)ε r (n)} + E{ε i (n)ε i (n)} + j(E{ε i (n)ε r (n)} − E{ε r (n)ε i (n)}}
= E{ε r (n)ε r (n)} + E{ε i (n)ε i (n)} = σ2 .
The amplitude of Gaussian noise |ε(n)| is an important parameter in

many detection problems. The probability density function of the complex-
Gaussian noise amplitude is
2ξ −ξ 2 /σ2
p|ε(n)| (ξ ) = e u ( ξ ).
σ2
The probability density function p|ε(n)| (ξ ) is called the Rayleigh distribu-
tion.
In order to prove the previous relation consider the probability den-
sity function of ε r (n) and ε i (n). Since they are independent and equally
distributed then
1 −(ξ 2 +ς2 )/σ2
pε r ε i (ξ, ζ ) = pε r (ξ ) pε i (ζ ) = e .
σ2 π
F
The probability that |ε(n)| = ε2r (n) + ε2i (n) < χ is
F ""
P{ ε2r (n) + ε2i (n) < χ} = pε r ε i (ξ, ζ )dξdζ
ξ 2 + ς2 < χ2
""
1 2 + ς2 ) /σ2
= 2
e−(ξ dξdζ.
σ π
ξ 2 + ς2 < χ2
With ξ = ρ cos α and ζ = ρ cos α (the Jacobian of the polar coordinate trans-
formation is J = |ρ|) we get
F "χ "2π
1 2 /σ2
P{ ε2r (n) + ε2i (n) < χ} = 2 e−ρ ρdρdα
σ π
0 0
2 /σ2
"χ χ"
2 −ρ2 /σ2 2 /σ2
= 2 e ρdρdα = e−λ dλ = (1 − e−χ )u(χ) = F|ε(n)| (χ).
σ
0 0
The probability density function is
dF|ε(n)| (ξ ) 2ξ −ξ 2 /σ2
p|ε(n)| (ξ ) = = e u ( ξ ). (7.27)
dξ σ2
Example 7.17. A random signal is defined as y(n) = |ε(n)|, where ε(n) is

the Gaussian complex zero-mean i.i.d. noise with variance σ2 . What is the
probability that y(n) ≥ A? Calculate this probability for A = 2 and σ2 = 1.
⋆The probability density function for sequence y(n) is
2ξ − ξ 22
py ( x ) = e σ u(ξ )
σ2
The probability that y(n) ≥ A is
2
− A2
P{ξ > A} = 1 − P{ξ ≤ A} = e σ .
For A = 2 and σ2 = 1 we get P{ξ > A} ≈ 0.0183.
7.3.5 Impulsive Noises
This noise is used to model disturbances when strong impulses occur more
often than in the case of a Gaussian noise. Due to possible stronger pulses,
their probability density function decay toward ±∞ is slower than in the
case of Gaussian noise.
The Laplacian noise has the probability density function
1 −|ξ |/α
pε(n) (ξ ) = e .
2α
Gaussian distribution
pε(ξ)
0.6
0.4
0.2
0
-5 -4 -3 -2 -1 0 1 2 3 4 5
Laplacian distribution
pε(ξ)
0.6
0.4
0.2
0
-5 -4 -3 -2 -1 0 1 2 3 4 5
Figure 7.11 The Gaussian and Laplacian noise histograms (with 10000 realizations), with
corresponding probability density function (dots).
It decays much slower as |ξ | increases than in the Gaussian noise case.

The Laplacian noise can be generated as
ε(n) = ε 1 (n)ε 2 (n) + ε 3 (n)ε 4 (n)
where ε i (n), i = 1, 2, 3, 4 are real-valued Gaussian independent zero-mean

noises, Fig.7.11 (for variance see Problem 7.13).
The impulsive noise could be distributed in other ways, like, for
example, the Cauchy distributed noise, whose probability density function
is
1
pε(n) (ξ ) = .
π (1 + ξ 2 )
The Cauchy distributed noise ε(n) is a random signal that can be obtained
as a ratio of two independent Gaussian random signals ε 1 (n) and ε 2 (n), i.e.,
as
ε 1 (n)
ε(n) = .
ε 2 (n)
7.3.6 Noisy Signals
In the case of noisy signals the noise could added to the signal s(n). Then
we have
x ( n ) = s ( n ) + ε ( n ).
This is an additive noise. For a deterministic signal s(n)
E{ x (n)} = E{s(n) + ε(n)} = s(n) + µε (n),

E{| x (n) − µε (n)|2 } = σε2 (n).
Noise can also be multiplicative, when
x (n) = (1 + ε(n))s(n).
In this case
E{ x (n)} = E{s(n) + ε(n)s(n)} = s(n)(1 + µε (n)),

E{| x (n) − µε (n)|2 } = |s(n)|2 σε2 (n).
Both the mean and the variance are signal dependent in the case of multi-
plicative noise.
7.4 DISCRETE FOURIER TRANSFORM OF NOISY SIGNALS
Consider a noisy signal

x (n) = s(n) + ε(n) (7.28)
where s(n) is a deterministic useful signal and ε(n) is an additive noise. The
DFT of this signal is
N −1
X (k) = ∑ (s(n) + ε(n))e− j2πkn/N = S(k) + Ξ(k). (7.29)
n =0
The mean value of X (k ) is

N −1 N −1
E{ X (k )} = ∑ s(n)e− j2πkn/N + ∑ E{ε(n)}e− j2πkn/N = S(k) + DFT{µε (n)}.
n =0 n =0
In the case of a zero-mean noise ε(n), when µε (n) = 0, follows
µ X (k ) = E{ X (k )} = S(k ). (7.30)
The variance of X (k ), for a zero-mean noise, is
σX2 (k ) = E{| X (k ) − µ X (k )|2 } = E{ X (k ) X ∗ (k ) − S(k )S∗ (k )}

N −1 N −1
= ∑ ∑ E{(s(n1 ) + ε(n1 ))(s∗ (n2 ) + ε∗ (n2 ))}e− j2πk(n1 −n2 )/N
n1 =0 n2 =0
N −1 N −1
− ∑ ∑ s(n1 )s∗ (n2 )e− j2πk(n1 −n2 )/N
n1 =0 n2 =0
N −1 N −1
= ∑ ∑ E{ε(n1 )ε∗ (n2 )}e− j2πk(n1 −n2 )/N . (7.31)
n1 =0 n2 =0
For a white noise, with the autocorrelation
rεε (n1 , n2 ) = E{ε(n1 )ε∗ (n2 )} = σε2 δ(n1 − n2 ),
we get
σX2 (k ) = σε2 N. (7.32)
If the deterministic signal is a complex sinusoid,
s(n) = Ae j2πk0 n/N , (7.33)
with a frequency adjusted to the grid ω0 = 2πk0 /N, then its DFT is
S(k ) = ANδ(k − k0 ).
Peak signal-to-noise ratio, being relevant parameter for the DFT based
estimation of frequency, is
maxk |S(k )|2 A2 N 2 A2

PSNRout = 2
= 2 = 2 N. (7.34)
σX σε N σε
It increases as N increases. We have expected this result since the signal

values are added in phase, increasing the DFT amplitude N times (its power
N 2 times), while the noise values are added in power. Noise influence to
the DFT of a real-valued sinusoid s(n) = A cos(2πk0 n/N ) = ( Ae j2πk0 n/N +
Ae− j2πk0 n/N )/2 is illustrated in Fig. 7.12.
X(k)
x(n)
n k
X(k)
x(n)
n k
Figure 7.12 Illustration of a signal x (n) = cos(6πn/64) and its DFT (top row); the same signal
corrupted with additive zero-mean real-valued Gaussian noise of variance σε2 = 1/4, along
with its DFT (bottom row).
The input signal-to-noise ratio (SNR) for signal 7.33 is
N −1
2
∑ | x (n)|
Ex n =0 N A2 A2
SNRin = = N = Nσ2 = σ2 . (7.35)
Eε N −1 M
2 ε ε
∑ E |ε(n)|
n =0
If the maximal DFT value is detected then only its value could be used for
the signal reconstruction (equivalent to the notch filter at k = k0 being used).
The DFT of output signal is then
Y ( k ) = X ( k ) δ ( k − k 0 ).
The output signal in the discrete-time domain is

N −1
1 1
y(n) =
N ∑ Y (k)e j2πkn/N = N X (k0 )e j2πk0 n/N .
n =0
Since X (k0 ) = AN + Ξ(k0 ), according to (7.29) and (7.32), where Ξ(k ) is a

noise with variance σε2 N, we get
Ξ(k0 ) j2πk0 n/N

y(n) = Ae j2πk0 n/N + e = x ( n ) + ε X ( n ).
N
The output signal-to-noise ratio is
N −1
2
∑ | x (n)|
Ex n =0
SNRout = = !' '2 6
Eε X N −1 ' Ξ(k ) '
∑ E ' N0 e j2πk0 n/N '
n =0
N A2 A2
= =N = N · SNRin .
N Nσε2 σε2
N2
Taking 10 log(◦) of both sides we get the signal-to-noise ratio relation in dB,
SNRout [dB] = 10 log N + SNRin [dB]. (7.36)
Example 7.18. If the DFT of a noisy signal s(n) + ε(n) is calculated using a window
function w(n), find its mean and variance. Noise is white, rεε = σε2 δ(n), with
zero-mean.
⋆Here,
N −1
X (k) = ∑ w(n) [s(n) + ε(n)] e− j2πkn/N .
n =0
For this DFT, the mean value is

N −1
µ X (k) = E{ X (k )} = ∑ w(n)s(n)e− j2πkn/N = W (k) ∗k S(k)
n =0
where W (k) = DFT{w(n)}.

The variance of X (k ) is
N −1 N −1
2
σXX (k ) = ∑ ∑ w(n1 )w∗ (n2 )σε2 δ(n1 − n2 )e− j2πk(n1 −n2 )/N
n1 =0 n2 =0
N −1
= σε2 ∑ |w(n)|2 = σε2 Ew , (7.37)
n =0
where Ew is the window energy.
Example 7.19. The DFT definition, for a given frequency index k, can be under-
stood as
N −1
X (k) = ∑ (s(n) + ε(n))e− j2πkn/N
n =0
M N
=N mean (s(n) + ε(n))e− j2πkn/N (7.38)
n=0,1,...,N −1
Based on the definition of median, discuss when the DFT estimation

M N
XR (k ) = N median Re (s(n) + ε(n))e− j2πkn/N (7.39)
n=0,1,...,N −1
M N
+ jN median Im (s(n) + ε(n))e− j2πkn/N
n=0,1,...,N −1
can produce better results than (7.38). Calculate the value X (0) using (7.38)
and estimate it by (7.39) for s(n) = exp( j4πn/N ) with N = 8 and noise
ε(n) = 2001δ(n) − 204δ(n − 3). Which one is closer to the noise-free DFT
value?
⋆If we can expect strong impulsive noise then the mean value will be
highly sensitive to this noise. The median based calculation is less sensitive
to strong impulsive noise. For the given signal
s(n) = exp( jπn/2) = [1, j, −1, − j, 1, j, −1, − j]
and noise ε(n) the value of X (0) is
X (0) = 0 + 2001 − 204 = 805.
The median-based estimation is
XR (0) = 8 median {2002, 0, −1, −204, 1, 0, −1, 0}+ (7.40)

n=0,1,..,N −1
j8 median {0, 1, 0, −1, 0, 1, 0, −1} = 0 + j0.
n=0,1,..,N −1
Obviously the median-based estimate is not influenced by this impulsive

noise. In this case it produced better estimate of the noise-free DFT.
7.4.1 Detection of a Sinusoidal Signal Frequency
Consider a set of data x (n), for 0 ≤ n ≤ N − 1. Assume that this set of data
are noisy samples of signal s(n) = Ae j2πk0 n/N . Additive noise ε(n) is white,
complex-valued Gaussian with zero-mean independent real and imaginary
parts and variance σε2 . The aim is to find the signal s(n) parameters from
the noisy observations x (n). Since the signal form is known we look for
a solution of the same form, using the model be j2πkn/N where b and k are
parameters that have to determined, and α = {b, k } is the set of parameters.
Parameter b is complex-valued. It includes amplitude and initial phase of
the model. For each value of x (n) we may define an error as a difference of
the given value x (n) and the assumed model, at the considered instant n,
e(n, α) = x (n) − be j2πkn/N . (7.41)

Since the noise is Gaussian, the probability density function of the error is
1 −|e(n,α)|2 /(2σε2 )
p(e(n, α)) = e .
2πσε2
The joint probability density function for all samples from the data set is
equal to the product of individual probability density functions
1 N −1 2 2
pe (e(0, α), e(1, α), ..., e( N − 1, α)) = 2 N
e− ∑n=0 |e(n,α)| /(2σε ) .
(2πσε )
The maximum-likelihood solution for parameters α = {b, k } in obtained

by maximizing this function for given values of x (n). Maximization of
pe (e(0, α), e(1, α), ..., e( N − 1, α)) is the same as the minimization of the total
square error,
N −1 N −1 ' '2
' '
ϵ(α) = ∑ |e(n, α)|2 = ∑ 'x (n) − be j2πkn/N ' . (7.42)
n =0 n =0
The solution of this problem is obtained from ∂ϵ(α)/∂b∗ = 0 (see Example

1.3). It is in the form of a standard DFT of signal x (n),
1 N −1 M N 1
b= ∑ x (n)e− j2πkn/N = mean x (n)e− j2πkn/N = X (k ).
N n =0 N
A specific value of parameter k that minimizes ϵ(α) and gives the estimate
of the signal frequency index k0 is obtained by replacing the obtained b back
into relation (7.42) defining ϵ(α),
( )
N −1 N −1
j2πkn/N 2 2
ϵ(α) = ∑ | x (n) − be | = ∑ | x (n)| − N | b |2 .
n =0 n =0
Minimal value of ϵ(α) is achieved when |b|2 (or | X (k )|2 ) is maximal,
k̂0 = arg{max| X (k )|2 } = arg{max| X (k )|}.
If there is no noise | x (n)| = A , k̂0 = k0 , b = A or X (k0 ) = N A, and ϵ(k0 ) = 0.

The same approach can be used for a signal s(n) = Ae jω0 n . Assuming
the solution in the form be jωn , the Fourier transform of discrete-time signals
would follow.
If the additive noise were, for example, Laplacian then the probability
density function would be p(e(n, α)) = 2σ1 ε e−|e(n,α)|/σε , and the solution of
ϵ(α) = ∑nN=−01 |e(n, α)| minimization would follow from
M N
X (k ) = Nmedian x (n)e− j2πkn/N .
Note that the absolute value of error can be written as

' ' ' '
' ' ' '
|e(n, α)| = 'x (n) − be j2πkn/N ' = 'x (n)e− j2πkn/N − b' .
Minimization of a sum of this kind of terms is discussed in (10.72).

Now we will analyze the signal frequency estimation for a single
component sinusoidal signal s(n) with unknown discrete frequency ω0 =
2πk0 /N using the DFT. Since a frequency on the frequency grid is assumed
this case can be understood as a frequency position detection. Available
observations of the signal are
x (n) = s(n) + ε(n), for 0 ≤ n ≤ N − 1,
where ε(n) is a complex zero mean Gaussian white noise with independent
real and imaginary parts, with variance σε2 . Its DFT is
N −1
X (k ) = ∑ (s(n) + ε(n))e− j2πkn/N = N Aδ(k − k0 ) + Ξ(k),
n =0
with σX2 (k ) = σε2 N and E{Ξ(k )} = 0. The real and imaginary parts of the DFT
X (k0 ) at the signal position k = k0 are Gaussian random variables, with total
variance σε2 N, or
N ( N A, σε2 N/2), N (0, σε2 N/2), (7.43)
respectively, where a real-valued A is assumed without any loss of general-

ity.
Real and imaginary parts of the noise only DFT values X (k ) for k ̸= k0
are zero-mean random variables with the same variance
N (0, σε2 N/2).
Next, we will find the probability that a DFT value of noise at any
k ̸= k0 is higher than the signal DFT value at k = k0 . This case corresponds
to a false detection of the signal frequency position, resulting in an arbitrary

large and uniform estimation error (within the considered frequency range).
The probability density function for the absolute DFT values outside
the signal frequency is Rayleigh-distributed (7.27)
2ξ −ξ 2 /(σε2 N )
q(ξ ) = e , ξ ≥ 0.
σε2 N
The DFT at a noise only position takes a value greater than Ξ, with proba-
bility
"∞
2ξ −ξ 2 /(σε2 N ) Ξ2
Q(Ξ) = e dξ = exp (− ). (7.44)
σε2 N σε2 N
Ξ
The probability that a DFT of noise only is lower than Ξ is [1 − Q(Ξ)]. The
total number of noise only points in the DFT is M = N − 1. The probability
that M independent DFT noise only values are lower than Ξ is [1 − Q(Ξ)] M .
Probability that at least one of M DFT noise only values is greater than Ξ, is
G (Ξ) = 1 − [1 − Q(Ξ)] M . (7.45)
The probability density function for the absolute DFT values at the
position of the signal (whose real and imaginary parts are described by
(7.43)) is Rice-distributed
2ξ −(ξ 2 + N 2 A2 )/(σε2 N )
p(ξ ) = e I0 (2N Aξ/(σε2 N )), ξ ≥ 0, (7.46)
σε2 N
where I0 (ξ ) is the zero-order modified Bessel function (for A = 0, when

I0 (0) = 1 the Rayleigh distribution is obtained).
When a noise only DFT value surpasses the DFT signal value, then an
error in estimation occurs. To calculate this probability, consider the absolute
DFT value of a signal at and around ξ. The DFT value at the signal position
is within ξ and ξ + dξ with the probability p(ξ )dξ , where p(ξ ) is defined
by (7.46). The probability that at least one of M DFT noise only values is
above ξ in amplitude is G (ξ ) = 1 − [1 − Q(ξ )] M . Thus, the probability that
the absolute DFT signal component value is within ξ and ξ + dξ and that at
least one of the absolute DFT noise only values exceeds the DFT signal value
is G (ξ ) p(ξ )dξ. Considering all possible values of ξ, from (7.44) and (7.45), it
follows that the probability of the wrong signal frequency detection is
"∞ "∞
( - .M )
ξ2
PE = G (ξ ) p(ξ )dξ = 1 − 1 − exp(− 2 )
σε N
0 0
2ξ 2 2 2 2
× 2 e−(ξ + N A )/(σε N ) I0 (2N Aξ/(σε2 N ))dξ. (7.47)
σε N
Approximation of this expression can be calculated by assuming that

the DFT of the signal component is not random and that it is equal to
N A (positioned at the mean value of the signals DFT). The form of error
probability is then very simple
- .M
N A2
PE ∼
= 1 − 1 − exp(− 2 ) . (7.48)
σε N
This expression can be used for a simple rough approximative analysis.

Analysis can easily be generalized to the case with K signal compo-
nents, s(n) = ∑kK=1 Ak e jωk n .
In many cases, the discrete frequency of the deterministic signal does
not satisfy the relation ω0 = 2πk0 /N, where k0 is an integer. In these cases,
when ω0 ̸= 2πk0 /N, the frequency estimation result can be improved , for
example, by zero-padding before the Fourier transform calculation or using
finer grid around the detected maximum. Comments on the estimation of
signal frequency outside the grid are given in Chapter III as well.
7.5 LINEAR SYSTEMS AND RANDOM SIGNALS
If a random signal x (n) passes through a linear time-invariant system, with

an impulse response h(n), then the mean value of the output signal y(n) is
∞
µy (n) = E{y(n)} = ∑ h(k )E{ x (n − k )} (7.49)
k=−∞
∞
= ∑ h ( k ) µ x ( n − k ) = h ( n ) ∗ n µ x ( n ). (7.50)
k=−∞
For a stationary signal

∞
µy = µ x ∑ h(k ) = µ x H (e j0 ). (7.51)
k =−∞
The cross-correlation of the output and input signal is

∞
ryx (n, m) = E{y(n) x ∗ (m)} = ∑ E{ x (k ) x ∗ (m)}h(n − k )
k=−∞
∞
= ∑ r xx (k, m)h(n − k ). (7.52)
k=−∞
For a stationary signal, with n − m = l and k − m = p, we get

∞
ryx (l ) = ∑ r xx ( p)h(l − p) = r xx (l ) ∗l h(l ).
p=−∞
The z-transform of both sides gives
Ryx (z) = R xx (z) H (z).
The cross-correlation of the input and output signal is

∞
r xy (n, m) = E{ x (n)y∗ (m)} = ∑ E{ x (n) x ∗ (k )}h∗ (m − k )
k=−∞
∞
= ∑ r xx (n, k )h∗ (m − k ). (7.53)
k =−∞
For a stationary signal, with n − m = l and n − k = p, we get

∞
r xy (l ) = ∑ r xx ( p)h∗ ( p − l ).
p=−∞
The z-transform of both sides are

∞ ∞ ∞
∑ r xy (l )z−l = ∑ ∑ r xx ( p)h∗ ( p − l )z−l
l =−∞ l =−∞ p=−∞
∞ ∞ B C−k
= ∑ ∑ r xx ( p)h∗ (k )z− p z−1
k=−∞ p=−∞
1
R xy (z) = R xx (z) H ∗ ( ).
z∗
If we calculate the Fourier transform of both sides, we get
Sxy (e jω ) = Sxx (e jω ) H ∗ (e jω ). (7.54)

Similarly, starting from
ryy (n, m) = E{y(n)y∗ (m)}

∞ ∞
= ∑ ∑ E{ x (l ) x ∗ (k )}h(n − l )h∗ (m − k ), (7.55)
k =−∞ l =−∞
after some straightforward calculations, we get the relation
1
Ryy (z) = R xx (z) H (z) H ∗ ( ).
z∗
The Fourier transform of output signal autocorrelation function is

' '2
' '
Syy (e jω ) = Sxx (e jω ) 'H (e jω )' , (7.56)
proving that Sxx (e jω ) is indeed a power density function. By taking a

' '2
narrow-pass filter with unit amplitude ' H (e jω )' = 1 for ω0 ≤ ω < ω0 + dω,
we will get the spectral density of signal x (n) for that small frequency range.
Example 7.20. A linear time-invariant system is defined by
y(n) = x (n) + ax (n − 1) + a2 x (n − 2).
The input signal is a zero-mean white noise ε(n) with variance σε2 . Find the
cross-correlation of the input and output signal and the autocorrelation of
the output signal. For a = −1 find the power spectral density of the output
signal.
⋆The system transfer function is
H (z) = 1 + az−1 + a2 z−2 .
Since the input signal is a white noise of variance σε2 its autocorrelation, by
definition, is
r xx (n) = rεε (n) = σε2 δ(n).
The power spectral density of the input signal is
∞
Sxx (ω ) = ∑ r xx (n)e− jωn = σε2 .
n=−∞
The z-transform of the input signal autocorrelation function is

∞ ∞
R xx (z) = ∑ r xx (n)z−n = σε2 ∑ δ(n)z−n = σε2 .
n=−∞ n=−∞
The z-transform of the autocorrelation function of the output signal, for linear
time-invariant system, is
Ryy (z) = R xx (z) H (z) H ∗ (1/z∗ )

@ A
= σε2 1 + a2 + a4 + a(1 + a2 )(z + z−1 ) + a2 (z2 + z−2 ) .
The autocorrelation function of the output signal is equal to the inverse z-

transform of Ryy (z),
ryy (n) = σε2 (1 + a2 + a4 )δ(n) + σε2 a(1 + a2 )(δ(n + 1) + δ(n − 1))

+ σε2 a2 (δ(n + 2) + δ(n − 2)).
The power spectral density of the output signal is
Syy (ω ) = Ryy (e jω )
= σε2 (1 + a2 + a4 + 2a(1 + a2 ) cos ω + 2a2 cos(2ω )),
while the z-transform of the cross-correlation of the input and output signal
is
Ryx (z) = H (z) R xx (z) = (1 + az−1 + a2 z−2 )σε2 .
Its inverse z-transform is the cross-correlation,
ryx (n) = σε2 (δ(n) + aδ(n − 1) + a2 δ(n − 2)).
For a = −1 the power spectral density function of the output signal is
Syy (ω ) = σε2 (3 − 4 cos ω + 2 cos(2ω ))

= σε2 (1 − 4 cos ω + 4 cos2 ω ) = σε2 (1 − 2 cos ω )2 .
Example 7.21. For a discrete-time system defined by

y(n) − 1.3y(n − 1) + 0.36y(n − 2) = x (n)
with the input signal x (n) = ε(n), µε = 0 and rεε (n) = δ(n), find:
a) Mean value µy (n) and autocorrelation ryy (n) of the output signal,
b) Power spectral density functions Syy (ω ) and Syx (ω ).
⋆a) The mean value of output signal is
µy = µ x H (e j0 ) = µε H (e j0 ) = 0.
The z-transform of the output signal autocorrelation is
Ryy (z) = R xx (z) H (z) H (1/z)

since H (z) is the z-transform of a real-valued signal. The autocorrelation of

the input signal is
R xx (z) = 1.
The transfer function has the form
1
H (z) =
1 − 1.3z−1+ 0.36z−2
1
= .
(1 − 0.9z−1 )(1 − 0.4z−1 )
Therefore, the autocorrelation of the output signal is
1
Ryy (z) =
(1 − 0.9z−1 )(1 − 0.4z−1 )(1 − 0.9z)(1 − 0.4z)
or - .
25 z z
Ryy (z) = − .
8 (z − 0.4)(z − 1/0.4) (z − 0.9)(z − 1/0.9)
The inverse z-transform of Ryy (z) is
- .
25 0.9 0.4
ryy (n) = (0.9)|n| − (0.4)|n| .
8 0.19 0.84
b) The power spectral density of the output signal is obtained as
1
Syy (ω ) = Ryy (z)|z=e jω = ,
(1.16 − 0.8 cos ω )(1.81 − 1.8 cos ω )
while the cross-power spectral density function Syx (ω ) can be defined as the
value of Ryx (z) at z = e jω
Syx (ω ) = Ryx (z)|z=e jω = H (z) R xx (z)|z=e jω

1
= .
1 − 1.3 cos ω + 0.36 cos 2ω + j(1.3 sin ω − 0.36 sin 2ω )
Example 7.22. A white noise ε(n) with variance σε2 and zero mean is an input to
a linear time-invariant system. If the impulse response of the system is h(n)
show that
E { x (n)y(n)} = h(0)σε2
and
∞
σy2 = σε2 ∑ |h(n)|2 = σε2 Eh ,
n=−∞
where y(n) is the output of this system.

⋆The mean value of the product of input and output signal is

% ;
∞
E { x (n)y(n)} = E ∑ h(k ) x (n) x (n − k ) .
k=−∞
Since the impulse response is a deterministic signal

∞ ∞
E { x (n)y(n)} = ∑ h(k )E { x (n) x (n − k )} = ∑ h(k )r xx (k )
k =−∞ k=−∞
and
r xx (n) = σε2 δ(n)
we get
∞
E { x (n)y(n)} = ∑ h(k )σε2 δ(k ) = h(0)σε2 .
k=−∞
The variance of output signal is defined by
σy2 = E {y(n)y∗ (n)} − E {y(n)} E {y∗ (n)}
or % ;
∞ ∞
σy2 = E ∑ h(k ) x (n − k ) ∑ h∗ (k ) x ∗ (n − k ) −
% k=−∞ ; k% =−∞ ;
∞ ∞
−E ∑ h(k ) x (n − k) E ∑ h∗ (k ) x ∗ (n − k ) .
k=−∞ k=−∞
The output signal is zero-mean signal,

∞
E {y(n)} = E {y∗ (n)} = ∑ h(k)E { x (n − k )} = 0.
k=−∞
Thus, we get
∞ ∞
σy2 = ∑ ∑ h(k)h∗ (l ) E { x (n − k) x ∗ (n − l )}
k=−∞ l =−∞
∞ ∞
= ∑ ∑ h(k)h∗ (l )r xx (l − k).
k=−∞ l =−∞
Since r xx (n) = σε2 δ(n) , i.e., r xx (l − k ) = σε2 δ(l − k ) , only the terms with l = k
remain in the double summation expression for the variance σy2 , producing
∞
σy2 = σε2 ∑ |h(k)|2 = σε2 Eh .
k=−∞
7.5.1 Spectral Estimation of Narrowband Signals
A narrowband random signal with Np components around frequencies ω1 ,

ω2 , and ω Np can be considered, from a spectral point of view, as an output
of a system whose transfer function is of the form
G
H (z) =
(1 − r1 e jω1 z−1 )(1 − r2 e jω2 z−1 )...(1 − r Np e jω Np z−1 )
G
= .
1 + a1 z −1 + a2 z −2 + ... + a Np z− Np
when the input is a white noise. The amplitudes of the poles ri are inside
(and close to) the unit circle. The discrete-time domain description of this
system is
y(n) + a1 y(n − 1) + a2 y(n − 2) + ... + a Np y(n − Np ) = Gx (n),
where x (n) is a white noise with variance σx2 = 1, autocorrelation r xx (k ) =

δ(k ), and spectral energy density Sxx (ω ) = 1. For a given narrowband
random signal y(n), the task is to find coefficients ai and G.
The autocorrelation of the output signal is obtained after the multipli-
cation of the difference equation by y(n + k ),
y(n + k )y(n) + a1 y(n + k )y(n − 1) + ... + a Np y(n + k )y(n − Np )

= Gy(n + k) x (n),
and expected value calculation,
E{y(n + k )y(n) + a1 y(n + k )y(n − 1) + ... + a Np y(n + k )y(n − Np )}

= E{ Gy(n + k ) x (n)}.
For k = 0 it follows
ryy (0) + a1 ryy (0 − 1) + a2 ryy (0 − 2) + ... + a Np ryy (0 − Np ) = G2 .
For k > 0 we get
ryy (k ) + a1 ryy (k − 1) + a2 ryy (k − 2) + ... + a Np ryy (k − Np ) = 0.

The previous equations are known as the Yule-Walk equations. The matrix
form of this system is
⎡ ⎤ ⎡ 2 ⎤
⎡ ⎤ 1 G
ryy (0) ryy (1) ... ryy ( Np ) ⎢ a1 ⎥ ⎢ 0 ⎥
⎢ ryy (1) ryy (0) ... ryy ( Np − 1) ⎥ ⎢ ⎥ ⎢ ⎥
⎢ ⎥ ⎢ a2 ⎥ = ⎢ 0 ⎥ . (7.57)
⎣ ... ... ... ... ⎦ ⎢ ⎥ ⎢ ⎥
⎣ ... ⎦ ⎣ ... ⎦
ryy ( Np ) ryy ( Np − 1) ... ryy (0)
a Np 0
The system is solved for unknown system coefficients [ a0 , a1 , a2 ,...,a Np ] with

G = 1. Then the coefficients are normalized as [ a0 , a1 , a2 ,...,a Np ]/a0 with
G = 1/a0 . The spectral energy density of y(n) follows with Sxx (ω ) = 1 as
' '2
' G '
' '
Syy (ω ) = ' ' . (7.58)
' 1 + a1 e− jω + a2 e− j2ω + ... + a Np e− jNp ω '
This is the autoregressive (AR) spectral estimation.

Note that the autocorrelation functions can be estimated, for real-
valued y(n), defined within 0 ≤ n ≤ N − 1, as
1 N −1− k
N − k n∑
ryy (k ) = y(n + k )y(n) for 0 ≤ k ≤ N − 1, (7.59)
=0
and ryy (k ) = ryy (−k ) for − N + 1 ≤ k < 0. These values are then used in
(7.57) for the autoregressive spectral estimation.
Next we will comment the estimated autocorrelation within the basic
definition of the power spectral density, Section 7.2.3. Relation (7.59) corre-
sponds to the unbiased estimation of the autocorrelation function. Power
spectral density, according to (7.17), is calculated as Syy (ω ) = FT{ryy (k )}.
Since the autocorrelation estimates for a large k use only a small
number of signal samples in averaging, they are not reliable. It is common
to apply a triangular (Bartlett) window function (w(k ) = ( N − |k |)/N) to
reduce the weight of these estimates in the Fourier transform calculation
1 N −1− k
N − k n∑
w(k )ryy (k ) = w(k ) y(n + k )y(n)
=0
( N − k ) 1 N −1− k
N − k n∑
= y(n + k )y(n)
N =0
N −1− k
1
=
N ∑ y(n + k )y(n) (7.60)
n =0
for 0 ≤ k ≤ N − 1. Since the window is used this autocorrelation function

estimate is biased. The Fourier transform of biased autocorrelation func-
tion w(k )ryy (k) = (1 − |k |/N )ryy (k ) is the power spectral density Pyy (ω ) =
FT{(1 − |k |/N )ryy (k )} defined by (7.19).
Example 7.23. Consider a random signal
√
y(n) = 2 cos(0.95n + ϕ1 ) + 2 sin(1.05n + ϕ2 ) + 0.5
within 0 ≤ n ≤ 127, where ϕ1 and ϕ2 are random variables. Plot the power
spectral density calculated using:
(a) The Fourier transform of ryy (k )
N −1
Syy (ω ) = FT{ryy (k )} = ∑ ryy (k )e− jωk .
k=− N +1
(b) The Fourier transform of signal

' '2
1 '' N −1 '
'
Pyy (ω ) = ' ∑ y(n)e− jωn ' .
N ' n =0 '
This form corresponds to FT {w B (k )ryy (k )} where w B (k) is a Bartlett window

whose width is equal to the width of the autocorrelation function.
(c) Applying the Fourier transform to K = 7 shorter intervals of signal
of duration M = 32 with step R = M/2
1 M −1
Yi (e jω ) = y(iR + n)e− jωn
M n∑
=0
for i = 0, 1, ..., 6 and averaging the power spectral density over these intervals
(Welch periodogram)
A 1 K −1 '' '2
'
Syy (ω ) = ∑ 'Yi (e jω )' .
K i =0
(d) Using (7.58) with appropriately estimated coefficients ai and G

using (7.57) and (7.59).
⋆The results are shown in Fig.7.13., in order from (a) to (d).
7.6 DETECTION AND MATCHED FILTER
Detection of an unknown deterministic signal in a high noise environment is

of crucial interest in many real-world applications. In this case the problem
1
0.5
0
(a)
-0.5
-3 -2 -1 0 1 2 3
1
0.5
0
(b)
-0.5
-3 -2 -1 0 1 2 3
1
0.5
0
(c)
-0.5
-3 -2 -1 0 1 2 3
1
0.5
0
(d)
-0.5
-3 -2 -1 0 1 2 3
Figure 7.13 Spectral analysis of sinusoidal signals with random phases (normalized values).
is in testing the hypothesis
H0 : Signal is not present in the observed noisy signal (7.61)

H1 : Signal is present in the observed noisy signal
Here we will present the of detection of a known signal in a white

noise using the matched filter.
7.6.1 Matched Filter
Consider a general signal form
x ( n ) = s ( n ) + ε ( n ),
where s(n) is a known function with the Fourier transform S(e jω ) and ε(n)
is a white noise with power spectral density σε2 . The problem is to find a
system with a maximal output if the input x (n) contains the signal s(n).
The output signal is used to test the hypothesis H1 : presence of the signal
s(n) in x (n).
The output of a system with impulse response h(n), with the fre-
quency response H (e jω ), to the signal x (n) is of the form
y(n) = ys (n) + yε (n)
where ys (n) and yε (n) are the system outputs to the inputs s(n) and ε(n),
respectively. For the output signal ys (n) holds
Ys (e jω ) = H (e jω )S(e jω ).
Power spectral density of ys (n) is

' '2 ' '2 ' '2
' ' ' ' ' '
'Ys (e jω )' = 'H (e jω )' 'S(e jω )' .
The power of output noise is
"π ' '2

1 ' '
E{|yε (n)|2 } = 'H (e jω )' σε2 dω.
2π
−π
The output signal y(n), at an instant n0 , is
"π
1
y s ( n0 ) = H (e jω )S(e jω )e jωn0 dω
2π
−π
' '2
' "π '
' 1 '
|ys (n0 )|2 = '' H (e jω )S(e jω )e jωn0 dω '' .
' 2π
−π
'
The aim is to maximize the output signal at an instant n0 if the input signal
contains s(n). According to Schwartz’s inequality (for its discrete form see
Section 10.3.3)
' '
' π ' "π ' "π '
' 1 " jω jω jωn0
' 1 ' jω '2
' 1 ' jω '
'2
' H ( e ) S ( e ) e dω ' ≤ 'S ( e ) ' dω 'H ( e ) ' dω,
' 2π ' 2π 2π
' −π
' −π −π
the peak output signal-to-noise ratio is
"π ' jω '2 "π ' '

1 'S(e )' dω 1 ' H (e jω )'2 dω
2π 2π
|ys (n0 )|2 −π −π
PSNR = ≤ .
E{|yε (n)|2 } "π ' '
1 ' H (e jω )'2 σε2 dω
2π
−π
This ratio is maximal when the equality sign holds
"π ' '

1 ' jω '2 Es
PSNRmax = 'S(e )' dω = 2 .
2πσε2 σε
−π
The maximal ratio in Schwartz’s inequality is achieved for
H (e jω ) = kS∗ (e jω )e− jωn0 .
In the time domain the impulse response is
h(n) = ks∗ (n0 − n).
This system is called matched filter. Its impulse response is matched

to the signal form. It maximizes the ratio of the output signal and the noise.
Thus, it is used in the detection, i.e., used to make a decision if the known
signal s(n) exists in the noisy signal x (n).
The matched filter is illustrated on detection of a chirp signal
2
s(n) = e−2(n/128) cos(8π (n/128)2 + πn/8)
in a Gaussian white noise of variance σε2 = 1. The output of the matched

filter is calculated for n0 = 0 by using the known signal as
y(n) = x (n) ∗n s(−n).
Two cases are presented in Fig.7.14: 1) When the input signal contains s(n)
and 2) when the input signal does not contain s(n). We can see that the
output of the matched filter has an easily detectable peak at n = 0 for the
case then the input signal contains s(n). There is no such a peak in y(n)
when the input signal x (n) is noise only.
s(n)
1
-1
-2 -1 0 1 2
5 5
x(n)=s(n)+ε(n) x(n)=ε(n)
0 0
-5 -5
-2 -1 0 1 2 -2 -1 0 1 2
100 100
y(n) y(n)
50 50
0 0
-50 -50
-2 -1 0 1 2 -2 -1 0 1 2
Figure 7.14 Illustration of the matched filter: Signal s(n). Input noisy signal x (n) = s(n) + ε(n)
containing signal s(n). Input signal x (n) = ε(n) does not contain signal s(n). Corresponding
outputs from the matched filter y(n) = x (n) ∗ s(−n) are presented bellow the input signal
subplots.
7.7 OPTIMAL WIENER FILTER
Assume that the input signal is x (n) and that it contains an information
about the desired signal d(n). The output signal is y(n) = h(n) ∗n x (n).
The task here is to find the impulse response h(n) of system such that the
difference of the desired signal and the output signal, denoted as error
e ( n ) = d ( n ) − y ( n ),
is minimal in the mean square sense, i.e.,
h(n) = min{E{|e(n)|2 }}.

h(n)
The mean square error is

' '2
' ∞ '
2 ' '
E{|e(n)| } = E{'d(n) − ∑ h(m) x (n − m)' }.
' m=−∞ '
The minimal value is obtained from

% ( ) ;
∂E{|e(n)|2 } ∞
= E 2 d(n) − ∑ h(m) x (n − m) x ∗ (n − k ) = 0. (7.62)
∂h∗ (k ) m=−∞
This relation states that expected value of the product of error signal e(n) =
d(n) − y(n) and the input signal x ∗ (n − k ) is zero
E {2e(n) x ∗ (n − k )} = 0
for any k. For signals satisfying this relation we say that they are normal to
each other.
Relation (7.62) can be written as
% ;
∞
E ∑ h(m) x (n − m) x ∗ (n − k ) = E {d(n) x ∗ (n − k )}
m=−∞
or
∞
∑ h(m)r xx (k − m) = rdx (k ).
m=−∞
Taking the z-transform of both sides we get
H (z) R xx (z) = Rdx (z).
Transfer function of the optimal filter is
Rdx (z)
H (z) = .
R xx (z)
For a special case when the input signal is the desired signal d(n) with an
additive noise
x (n) = d(n) + ε(n)
where ε(n) is uncorrelated with the desired signal, the optimal Wiener
filtering relation follows
Rdd (z)
H (z) =
Rdd (z) + Rεε (z)
since
rdx (k ) = E {d(n) x ∗ (n − k )}
= E {d(n)[d∗ (n − k) + ε∗ (n − k)]}
= rdd (k).
Here we used E {d(n)ε∗ (n − k )} = 0 since d(n) and ε(n) are uncorrelated.

Also
r xx (k ) = E {[d(n) + ε(n)][d∗ (n − k ) + ε∗ (n − k )]}

= rdd (k) + rεε (k).
The frequency response of the optimal filter is
Sdd (ω )
H (e jω ) = .
Sdd (ω ) + Sεε (ω )
Example 7.24. A signal x (n) = d(n) + ε(n) is processed by an optimal filter. Power
spectral density of d(n) is Sdd (ω ). If the signal d(n) and the additive noise
ε(n), whose power spectral density is Sεε (ω ), are independent find the output
signal-to-noise ratio.
⋆For this signal and noise, according to (7.56), we have
' '2
' '
Syy (e jω ) = 'H (e jω )' Sxx (e jω )
' '2
' Sdd (ω ) '
Syy (e jω ) = '' ' Sxx (e jω )
Sdd (ω ) + Sεε (ω ) '
2 (ω )
Sdd
=
Sdd (ω ) + Sεε (ω )
since Sxx (e jω ) = Sdd (ω ) + Sεε (ω ). The output signal-to-noise ratio is
&π ' '2

1 ' jω '
2π −π Sdd (ω ) 'H (e )' dω
SNR = &π ' ' .
1 ' jω '2
2π −π Sεε ( ω ) H ( e ) dω
Note that the input signal-to-noise ratio is

1
&π
2π −π Sdd (ω )dω
SNRi = 1 & π .
2π −π Sεε ( ω ) dω
The optimal prediction system follows with the input signal x (n) =
d(n − 1) + ε(n − 1) and the desired signal d(n). Transfer function of the
optimal predictor is obtained from
rdx (k ) = E {d(n) x ∗ (n − k )}
= E {d(n)[d∗ (n − 1 − k) + ε∗ (n − 1 − k )]} = rdd (k + 1)
and
r xx (k ) = E {[d(n − 1) + ε(n − 1)][d∗ (n − 1 − k ) + ε∗ (n − 1 − k )]}

= rdd (k ) + rεε (k).
as
zSdd (z)
H (z) =
Sdd (z) + Sεε (z)
since
∞ ∞
∑ rdd (k + 1)z−k = ∑ rdd (k )z−k+1 = zSdd (z).
k =−∞ k=−∞
The optimal smoothing is the case when the desired signal is d(n) and
we can use its future value(s). It follows with x (n) = d(n + 1) + ε(n + 1) as
z−1 Sdd (z)

H (z) = .
Sdd (z) + Sεε (z)
Example 7.25. The input signal is x (n) = s(n) + ε(n), where d(n) = s(n) is the
desired signal and ε(n) is a noise. If the autocorrelation functions of the signal
and noise are rss (n) = 4−|n| and rεε (n) = 2δ(n), respectively, and the cross-
correlation of the signal and noise is rsε (n) = δ(n), design the optimal filter.
⋆The optimal filter transfer function is
Rdx (z)
H (z) =
R xx (z)
where are
Rdx (z) = Rss (z) + Rsε (z)
R xx (z) = Rss (z) + 2Rsε (z) + Rεε (z).

Based on the correlation functions we can calculate the z-transforms
∞ ∞ −1 ∞
Rss (z) = ∑ rss (n)z−n = ∑ 4−|n| z−n = ∑ 4n z − n + ∑ 4− n z − n =
n=−∞ n=−∞ n=−∞ n =0
z/4 1 −3.75z
= + =
1 − z/4 1 − 1/(4z) (z − 0.25)(z − 4)
and
Rsε (z) = 1
Rεε (z) = 2.
The transfer function of the optimal filter is
Rss (z) + Rsε (z) 0.25z2 − 2z + 0.25

H (z) = = 2 .
Rss (z) + 2Rsε (z) + Rεε (z) z − 5.1875z + 1
The optimal systems with FIR filters will be presented within the
introductory part of the chapter dealing with adaptive discrete systems.
7.8 QUANTIZATION EFFECTS
In order to process continuous signals using computers they have to be con-

verted into numbers stored into registers of a finite precision. Continuous
signals are transformed into digital by using analog-to-digital (A/D) con-
verters. A continuous-time signal is converted into a discrete-time signal by
taking samples of the continuous-time signal at discrete-time instants
x (n) = x (n∆t)∆t.
Next the discrete-time signal, with continuous amplitudes, is converted into

a digital signal
x Q (n) = Q[ x (n)]
with discrete-valued amplitudes (quantization). This process is illustrated
in Fig.7.15. Error caused by this process is called quantization noise.
Quantization noise influences results of signal processing in several
ways:
-Input signal quantization error, described by an additive quantization
noise. This influence (additive input noise that depends on quantization
continuous discrete-time digital

1 1
15 1111
14 1110
0.8 0.8 13 1101
12 1100
11 1011
10 1010
0.6 0.6
9 1001
x (n)
x(n)
x(t)
8 1000
d
7 0111
0.4 0.4 6 0110
5 0101
4 0100
0.2 0.2 3 0011
2 0010
1 0001
0 0 0000
0 5 10 15 0 5 10 15 0 5 10 15
t n n
Figure 7.15 Illustration of a continuous signal and its discrete-time and digital version.
step ∆) can be modeled as uniform noise with values between −∆/2 and
∆/2.
-Quantization of the results of arithmetic operations. It depends on the
way how the calculations are performed.
-Quantization of the coefficients in the algorithm. Usually this kind
of error is neglected in analysis since it is deterministic (comments on the
errors in the coefficients are given in the chapter dealing with realizations
of discrete systems).
In order to make appropriate analysis, common assumptions are:
1) random variables corresponding to the quantization errors are un-
correlated, i.e., the quantization error is a white noise process with a uniform
distribution,
2) the error sources are uncorrelated with one another, and
3) all the errors are uncorrelated with the input signal and, conse-
quently, with all signals in the system.
7.8.1 Input signal quantization
For registers with b bits the digital signal values xQ (n) are coded into binary
format.
Assume that registers with b bits are used and that all input signals
are normalized to the range 0 ≤ x (n) < 1. The binary numbers are written
within the register as
a −1 a −2 a −3 ... a−b .
The value of xQ (n) is
xQ (n) = a−1 2−1 + a−2 2−2 + ... + a−b 2−b .
The maximal number that can be written within this format is 0.111...11
representing 1 − 2−(b+1) . Common number of bits b ranges from 8 to 24.
For reducing the signal number of digits to b bits rounding or trun-
cation is used. An example of quantization with b = 4 bits is presented in
Fig.7.15, where the maximal value of xd (n) = xQ (n) is denoted by 1111
meaning 2−1 + 2−2 + 2−3 + 2−4 = 15/16.
For the case with positive an negative numbers, one extra bit is used
for the sign. The registers are now with b + 1 bits. The first bit is the sign bit
and the remaining b bits represent the signal absolute value
s a −1 a −2 a −3 ... a−b .
In computers negative numbers are commonly represented in a comple-

ment of 2 form.
In order to distinguish these two cases we will use register of length b
meaning no sign bit exists and register of length b + 1, where the sign bit is
used.
Example 7.26. In a register with b = 8 bits, the binary number xQ (n)
1 0 1 1 0 0 1 0
has the decimal value

89
x Q ( n ) = 1 · 2−1 + 1 · 2−3 + 1 · 2−4 + 1 · 2−7 = = 0.6953.
128
The decimal point assumes position just before the first digit. The values of
x Q (n) in this register are
255
0 ≤ xQ (n) ≤
256
with the quantization step 1/256.
The quantization error is a difference in the amplitude
e ( n ) = x ( n ) − x Q ( n ).
For rounding, the maximum absolute error can be a half of the last digit
weight
1 1 −b
− 2− b ≤ x ( n ) − x Q ( n ) < 2
2 2
1 1
− ∆ ≤ x (n) − x Q (n) < ∆
2 2
where
∆ = 2− b .
We can also write
1
|e(n)| ≤ 2−(b+1) = ∆.
2
In the example from Fig.7.15, obviously the quantization step is 2 −4 = 1/16
and the error is within |e(n)| ≤ 12 16
1
.
The error values are equally probable within the defined interval. Its
probability density function is
⎧ 1
⎨ ∆ for − 12 ∆ ≤ ξ < 12 ∆
pe (ξ ) = ,
⎩
0 elsewhere.
The quantization error of signal x (n) may be described as an additive

uniform white noise.
Mean of the quantization error, with rounding, is
∆/2
"
µe = E{e(n)} = ξ pe (ξ )dξ = 0.
−∆/2
Its variance is
∆/2
"
1 1
σe2 = (ξ − µe )2 dξ = ∆2 .
∆ 12
−∆/2
When the truncation is used, the error is within
0 ≤ x (n) − x Q (n) < ∆

or
0 ≤ e(n) < ∆
with mean value
∆
µe = E{e(n)} =
2
and variance
"∆
1 ∆ 1
σe2 = (ξ − )2 dξ = ∆2 .
∆ 2 12
0
Example 7.27. The DFT of a signal x (n) is calculated by using its quantized
version
xQ (n) = Q[ x (n)] = x (n) + e(n).
Quantization is done in an A/D convertor with b + 1 = 8 bits using round-
ing. The DFT is calculated on a high precision computer with N = 1024
signal samples. Find the mean and variance of the calculated DFT.
⋆The DFT of quantized signal is
N −1
XQ (k ) = ∑ [ x (n) + e(n)] e− j2πkn/N .
n =0
Its mean is
N −1
µ XQ (k ) = E{ XQ (k )} = ∑ x (n)e− j2πkn/N = X (k ).
n =0
The variance is
N −1 N −1
σX2 Q (k ) = ∑ ∑ σe2 δ(n1 − n2 )e− j2πk(n1 −n2 )/N
n1 =0 n2 =0
1 2 1
= σe2 N = ∆ N = 2−2b N
12 12
1 −14 1 −14 10 1
= 2 N= 2 2 = .
12 12 192
The noise in the DFT is a sum of many independent noises from the
input signal and coefficients. Thus it is Gaussian distributed with standard
deviation σXQ = 0.072. It may significantly influence the signal DFT values,
especially if they are not well concentrated or if there are signal components
with small amplitudes.
Example 7.28. How the input quantization error influences the results of:
(a) Weighted sum
N −1
Xs = ∑ an x (n)
n =0
(b) Product
N −1
XP = ∏ x ( n ).
n =0
⋆If the quantized values xQ (n) = Q[ x (n)] = x (n) + e(n) of signal x (n)
are used in calculation instead of the signal true values then:
(a) The estimator of a weighted sum is
N −1 N −1 N −1
X̂s = ∑ an x Q (n) = ∑ an x (n) + ∑ a n e ( n ).
n =0 n =0 n =0
Obviously the total error is
N −1
e Xs = ∑ a n e ( n ).
n =0
It is Gaussian distributed since there are many small errors e(n). It has been
assumed that the weighting coefficients are such that they allow many signal
values to influence result with similar weights.
The mean value is
N −1
µ Xs = E { e Xs } = ∑ an E{e(n)} = 0,
n =0
for rounding. The variance is
N −1
2 1 2 N −1 2
σX = ∑ a2n var{e(n)} = ∆ ∑ an .
s
n =0
12 n =0
(b) The estimator of the product is
N −1
X̂P = ∏ (x(n) + e(n)).
n =0
Assuming that the individual errors are small so that all higher order error
terms containing e(n)e(m), e(n)e(m)e(l ), ... could be neglected we get
N −1 N −1 N −1
X̂P ∼
= ∏ x ( n ) + ∑ ∏ x ( n ) e ( m ).
n =0 m =0 n =0
n̸=m
The quantization effect caused error is

N −1 N −1
e XP = ∑ ∏ x ( n ) e ( m ).
m =0 n =0
n̸=m
It is interesting to note that the relative error is additive since

N −1 N −1
∑ ∏ x (n)e(m)
m =0 n =0 N −1 N −1
e XP n̸=m e(m)
r XP =
XP
=
N −1
= ∑ = ∑ r x ( m ).
x ( m ) m =0
m =0
∏ x (n)
n =0
The mean value is zero if rounding is used. The variance is signal dependent,
N −1 N −1 N −1 N −1
2 1
σX p
= ∑ ∏ x2 (n)var{e(n)} = 12 ∆2 ∑ ∏ x2 (n).
m =0 n =0 m =0 n =0
n̸=m n̸=m
7.8.2 Quantization of the results
In the quantization of results after the basic arithmetic operations are per-
formed we can distinguish two cases. One is with fixed point arithmetic. In
that case the register assumes that the decimal point is at the fixed place. All
data are written with respect to this position. In the floating point arithmetic
numbers are written in the sign-mantissa-exponent format. The quantiza-
tion error is then produced on mantissa only.
7.8.2.1 Fixed point arithmetic
Fixed point arithmetic assumes that the decimal point position is fixed.
Common assumption is that the all input values and the mid-results, in this
case, are normalized so that 0 ≤ x (n) < 1 or −1 < x (n) < 1 if sign bit is used.
In multiplications, the result of a multiplication
x Q (n) x Q (m)
will, in general, produce a result of 2b digits. It should be quantized in the

same way as the input signal
Q[ xQ (n) xQ (m)] = xQ (n) xQ (m) + e(n, m)

where e(n, m) is the quantization error satisfying all the previous properties
with
1 1
− ∆ ≤ e(m, n) ≤ ∆.
2 2
Example 7.29. Find the mean of quantization error for
N −1
r (n) = ∑ x (n + m) x (n − m)
m =0
where x (n) is quantized and the product of signals is quantized as well to b

bits. Assume that the signal values are such that their additions will not cause
overflow.
⋆For this calculation the model is
N −1 D E
r̂ (n) = ∑ x Q (n + m) x Q (n − m) + e(n + m, n − m)
m =0
N −1
= ∑ {[x(n + m) + e(n + m)] [(x(n − m) + e(n − m)] + e(n + m, n − m)}.
m =0
The mean value is

N −1 N −1
E{r̂ (n)} = ∑ x ( n + m ) x ( n − m ) + E{ ∑ e(n + m)e(n − m)}
m =0 m =0
1 2
= r (n) + E{e2 (n)} = r (n) + ∆ ,
12
since it is assumed that errors for two different signal samples are not
correlated E{e(n + m)e(n − m)} = 0 for m ̸= 0 and the signal and errors are
not correlated, E{ x (n + m)e(n − m)} = 0 for any m and n.
In general the additions cause quantization error as well. Namely in
adding two values 0 ≤ x (n) < 1 the result could be greater than 1. In order
to avoid the overflow the input values are shifted in the register to the left
(appropriately divided), causing quantization error.
I the case that complex-valued numbers are used in calculation then

the quantization of real and imaginary parts is done separately,
xQ (n) = Q[ x (n)] = Q[Re{ x (n)} + j Im{ Q[ x (n)]}] = x (n) + er (n) + jei (n).
Since the real and imaginary part are independent, with the same variance,
the variance of quantization error for a complex-valued signal is
1 2 1 2
σe2 = 2 ∆ = ∆ .
12 6
For the additions the variance is doubled as well.

In case of multiplications one complex-valued multiplication requires
four real-valued multiplications, introducing four errors. The quantization
variance of a complex-valued multiplication is
1 2 1 2
σe2 = 4 ∆ = ∆ .
12 3
If the values of a signal x (n) are not small we have to ensure that no
overflow occurs during the calculations using the fixed point arithmetic.
Consider a real-valued random white signal whose samples are within
−1 < x (n) < 1, with variance σx2 . Registers of b + 1 bits are assumed, with
one bit being used for the sign. As an example consider the mean value
calculation
1 N −1
N n∑
XN = x ( n ).
=0
We have to be sure that an overflow will not occur during the mean value
calculation. All sums should stay within the interval (−1, 1).
One approach to calculate X N is in dividing the input signal values by
N and summing them
x (0) x (1 ) x ( N − 1)
XN = + + ... + .
N N N
Then we are sure that no result will be outside the interval (−1, 1). By divid-
ing the signal samples by N an additive quantization noise is introduced,
x (0 ) x (1) x ( N − 1)
X̂ N = + e (0 ) + + e(1) + ... + + e ( N − 1 ).
N N N
Variance of the equivalent noise e(0) + e(1) + · · · + e( N − 1) is
1 2 1
σe2 = ∆ N = 2−2b N.
12 12
Since the variance of x (n)/N is σx2 /N 2 , the variance of X̂ N is
σx2 1
σX2 N = N + ∆2 N.
N2 12
Ratio of variances corresponding to the signal and noise in the result is
σ2
N Nx2 1 σx2 1 σx2
SNR = 1 2
= 2 1
= 2 1 −2b
12 ∆ N
N 12 ∆ 2 N 12 2
or in [dB]
1 σx2
SNR = 10 log( )
N2 1 −2b
12 2
= 20 log σx − 20 log N − 20 log 2−b + 10 log(12)
log2 N log 2−b
= 20 log σx − 20 − 20 2 + 10.8
log2 10 log2 10
= 20 log σx − 6.02(m − b) + 10.8,
where N = 2m . Obviously increasing the number of samples N to 2N will

keep the same SNR if b is increased for one bit, since (m + 1 − (b + 1)) =
m − b.
Another way to calculate the mean is in performing the summation
step by step, according to the scheme presented, for example for N = 8, as
x (0) x (1) x (2) x (3) x (4) x (5) x (6) x (7)

2 + 2 2 + 2 2 + 2 2 + 2
+ +
XN = 2 2 + 2 2 .
2 2
Here two adjunct signal values x (n) are divided by 1/2 first. They are
added then, avoiding possible overflow. The error in one step is
x (n) x ( n + 1) x ( n ) + x ( n + 1) (2)
+ e(n) + + e ( n + 1) = + en .
2 2 2
The error
(2)
en = e ( n ) + e ( n + 1)
has the variance
M N 1 1 1
(2)
var en = ∆2 + ∆2 = ∆2 .
12 12 6
After each division by 2 the result is shifted in the register to the right and
a quantization error is created. Thus the error model, due to the addition
quantization, is
x (0) x (1) (2 ) x (2) x (3) (2)
2 + 2 + e0 2 + 2 + e2 (4)
2 + 2 + e0
X̂ N = (7.63)
2
x (4) x (5) (2) x (6) x (7) (2)
2 + 2 + e4 2 + 2 + e6 (4)
2 + 2 + e4 (8)
+ + e0
2
x (0 ) x (1 ) x ( N − 1)
= + + ... +
N N N
(2) (2) (2)
e0 e2 e
+ + + ... + N −2 +
N/2 N/2 N/2
(4) (4)
e0 e
+ + ... + N −4 +
N/4 N/4
....
(N)
e0
+ .
N/N
The variance of all qunatization noises is the same
1 1
σe2 = ∆2 = 2−2b .
6 6
Note that the noises in the first stage are divided by N/2, due to divisions
by 2 in the next stages of summation. Their variance is reduced for N 2 /4.
The value of variance of errors in these stages is
(2) (2) (2)

e0 e e 1 1 N 1 2
var{ + 2 + ... + N −2 } = ∆2 2 = ∆2
N/2 N/2 N/2 6 N /4 2 6 N
(4) (4)
e0 e 1 1 N 1 4
var{ + ... + N −4 } = ∆2 2 = ∆2
N/4 N/4 6 N /16 4 6 N
...
(N)
e0 1 1 N 1 N 1 2m
var{ } = ∆2 2 2 = ∆2 = ∆2 .
N/N 6 N /N N 6 N 6 N
The total variance of X̂ N is
σx2 1 2 1 4 1 2m
σX2 N = N 2
+ ∆2 + ∆2 + ... + ∆2 (7.64)
N 6 N 6 N 6 N
σ2 1 2 σ2 1 2 1 − 2m
= x + ∆2 (1 + 2 + ... + 2m−1 ) = x + ∆2
N 6 N N 6 N 1−2
σx2 1 22 σx2 1 2 1
= + ∆ (N − 1 ) = + ∆ (1 − ).
N 6 N N 3 N
Ratio of the variances, corresponding to the output signal-to-noise ratio, is

σx2
σx2 ∼ 1 σx2
SNR = 1 2
N
1
= 1 2 = = 3σx2 22(b−m/2) .
3 ∆ (1 − N ). 3 ∆ ( N − 1)
N 13 2−2b
Significant improvement (for an order of N) is obtained using this way of

the summation, instead of the direct one. In dB the ratio is
P Q m
SNR ∼= 10 log 3σx2 22(b−m/2) = 20 log σx − 6.02( − b) + 4.8.
2
If the signal values were complex then 2 −2b /12 would be changed to
2−2b /6.
The previous results are common in literature. They are derived as-
suming that the variances of the errors are the same and obtained assum-
ing unform nature of the quantization errors. However these results differ
from the ones obtained by statistical analysis. The reason is in the quan-
tization error distribution and variance. Namely, after the high precision
signal x (n) is divided by 2 and stored into b + 1 bit registers, the errors
in x (n)/2 + e(n) are unform with −∆/2 ≤ e(n) < ∆/2. When these val-
ues are stored into registers, then in each next stage when we calculate
[ x (n)/2 + e(n)] + [ x (n + 1)/2 + e(n + 1)]/2 the input values x (n)/2 + e(n)
and x (n + 1)/2 + e(n + 1) are already stored in the b + 1 bit registers. Di-
vision by 2 is just a one bit shift to the right. This shift cases one bit error.
Therefore this one bit error is discrete in amplitude
ed ∈ {−∆/2, 0, ∆/2},
with probabilities Pd (±∆/2) = 1/4 and Pd (0) = 1/2. Mean value of this
error kind of is zero, provided that the rounding is done in such a way that it
takes values ±∆/2 with equal probability (various tie-breaking algorithms
for rounding exist). Its variance is
M N 1 ∆ 1 ∆ 1
(i )
var en = 2var {ed } = 2[ (− )2 + ( )2 ] = ∆2 , for i > 2.
4 2 4 2 4
The total variance of X̂ N is then of form
σx2 1 2 1 4 1 2m σ2 1 4
σX2 N = N 2
+ ∆2 + ∆2 + ... + ∆2 = x + ∆ 2 (1 − ),
N 6 N 4 N 4 N N 2 3N
instead of (7.64). Signal-to-noise ratio is
σx2
SNR = N ∼
= 2σx2 22(b−m/2) .
1 2 4
2 ∆ (1 − 3N )
The previous analysis corresponds to the calculation of the DFT coeffi-

cient X (0) when the input signal is a random uniform signal whose values
are −1 < x (n) < 1 with variance σx2 . A model for a coefficient X (k ), with all
quantization errors included, is
N −1 M N N −1
1 nk
X̂ (k ) =
N ∑ [ x (n) + ei (n)] WN + e m ( n ) = ∑ y ( n ),
n =0 n =0
where ei (n) is the input signal quantization error and em (n) is the multipli-
cation quantization error. The variances for complex-valued signals are
1 2 1 2 1 1
var{ei (n)} = 2 ∆ = ∆ , var{em (n)} = 4 ∆2 = ∆2 .
12 6 12 3
In addition, we have to provide that additions do not produce an overflow.
If we use the calculation scheme, presented for N = 8, as
y (0) y (1) (2) y (2) y (3) (2)
2 + 2 + e0 2 + 2 + e2 (4)
2 + 2 + e0
X̂ (k ) =
2
y (4) y (5) (2) y (6) y (7) (2)
2 + 2 + e4 2 + 1 + e6 (4)
2 + 2 + e4 (8)
+ + e0 ,
2
then in each addition the terms should be divided by 2. This division
introduces a quantization error. In the first step
y(n) y ( n + 1) 1 nk
+ e(n) + + e(n + 1) = {[ x (n) + ei (n)] WN + em (n)+
2 2 2
( n +1) k
[ x (n + 1) + ei (n + 1)]WN + em (n + 1)} + e(n) + e(n + 1).
The total error in this step is
nk + e (n ) + e (n + 1)W ( n +1) k
(2) ei (n)WN m i N + em ( n + 1)
en = + e ( n ) + e ( n + 1)
2
with variance
* +
(2) 1 1 2 1 2 1 2 1 2 1 7
var{en } = ∆ + ∆ + ∆ + ∆ + 2 ∆2 = ∆2 .
4 6 3 6 3 6 12
(4) (N)
In all other steps, within the errors e0 to e0 , just the addition errors
appear. Their variance, for complex-valued terms, is
(i ) 1
var{en } = 2 ∆2 .
6
Therefore, the variance of

(2) (2) (2)
x (0) x (1) x ( N − 1) e e e
X̂ N = + + ... + + 0 + 2 + ... + N −2 +
N N N N/2 N/2 N/2
(4) (4) (N)
e0 e e
+ + ... + N −4 + .... + 0 (7.65)
N/4 N/4 N/N
is obtained using
(2) (2) (2)
e0 e e 7 1 N 7 2
var{ + 2 + ... + N −2 } = ∆2 2 = ∆2
N/2 N/2 N/2 12 N /4 2 12 N
(4) (4)
e e 1 1 N 1 4
var{ 0 + ... + N −4 } = ∆2 2 = ∆2
N/4 N/4 3 N /16 4 3 N
...
(N)
e0 1 1 N 1 2m
var{ } = ∆2 2 2 = ∆2 .
N/N 3 N /N N 3 N
The total variance of X̂ N is
σx2 1 2 3
σX2 N = + ∆2 ( + 1 + 2 + ... + 2m−1 )
N 3 N 4
σ2 2 N − 14 ∼ σx2 2
= x + ∆2 = + ∆2
N 3 N N 3
with
m
SNR = 20 log σx − 6.02( − b) + 1.76.
2
If the described discrete nature of the quantization error amplitude,
after the first quatization step, is taken into account (provided that the
rounding is done in such a way that the error takes values ±∆/2 with equal
probability), then with
M N 1
(i )
var en = 4var {ed } = ∆2 ,
2
for i > 2, the variance of X̂ N follows
σx2 ∆2 7
σX2 N = + ( + 2 + 4 + ... + 2m−1 )
N N 6
σ2 N − 54 ∼ σx2
= x + ∆2 = + ∆2 .
N N N
If the FFT is calculated using the fixed point arithmetic and the signal
is uniform, distributed within −1 < x (n) < 1 with variance σx2 , then in order
to avoid an overflow the signal could be divided at the input with N and
the standard FFT could be used, as in Fig.7.16.
An improvement in the SNR can be achieved if the scaling is done
not to the input signal x (n) by N but by 1/2 in each butterfly, as shown in
Fig.7.17. The improvement is here due to the fact that the quantization errors
appearing in the early butterfly stages are divided by 1/2 and reduced at the
output as in (7.63). Improvement of an order of N is obtained in the output
signal-to-noise ratio.
7.8.2.2 Floating point arithmetic
Fixed point arithmetic is simple, but could be inefficient if the signal values
within wide range of amplitudes may be expected. For example, if we can
expect signal values
x Q (n1 ) = 1011111110101.010
x Q (n2 ) = 0.0000000000110101
then obviously fixed point arithmetic would require large registers so that
both values can be stored without loosing their significant digits. However,
we can represent these signal values into the exponential form as
x Q (n1 ) = 1.011111110101010 × 212

x Q (n2 ) = 1.10101 × 2−11
The exponential format of numbers is then written within the register in the
following format
sn se e1 e2 e3 e4 e5 e6 e7 m −1 m −2 m −3 ... m−b
where:
sn is the sign of number (1 for positive number and 0 for negative
number)
se is the sign of exponent (1 for positive exponent and 0 for negative
exponent)
e1 e2 ...e7 is the binary format of exponent, and
m−1 m−2 ...m−b is the mantissa, assuming that the integer value is
always 1, it is omitted.
x(0)/N X(0)
x(1)/N X(4)
-1 W0
8
x(2)/N X(2)
-1 W0
8
x(3)/N 2 0 X(6)
-1 W8 -1 W8
x(4)/N 0 X(1)
-1 W8
x(5)/N X(5)
-1 W1 -1 0
W8
8
x(6)/N 2 0 X(3)
-1 W8 -1 W8
x(7)/N 3 2 0 X(7)
-1 W8 -1 W8 -1 W8
Figure 7.16 FFT calculation scheme obtained by decimation in frequency for N = 8 with
signal being divided by N in order to avoid overflow when the fixed point arithmetic is used.
1/2 1/2 1/2

x(0) X(0)
2
1/
2
1/
2
1/
2
1/
1/2 1/2
x(1) -1/2 X(4)
W0
2
1/
2
8
1/
1/
2
1/2 1/2
x(2) -1/2 X(2)
W0
1/
2
2
1/
2
1/
8
1/
2
1/2
x(3) -1/2 2 -1/2 0 X(6)
W8 W8
1/
2
1/
2
1/2 1/2
x(4) -1/2 0 X(1)
W8
2
1/
2
1/
1/
1/
2
2
1/2
x(5) -1/2 1 -1/2 0 X(5)
W8 W8
2
1/
1/
2
1/
2
1/2
x(6) -1/2 2 -1/2 0 X(3)
W8 W8
1/
2
1/
1/
2
2
1/
2
x(7) -1/2 3 -1/2 2 -1/2 0 X(7)

W8 W8 W8
Figure 7.17 FFT calculation scheme obtained by decimation in frequency for N = 8 with
signal being divided in each butterfly by 1/2 in order to avoid overflow when the fixed point
arithmetic is used.
Within this format, the previous signal value xQ (n1 ), with a register of
19 bits in total, is
1 1 0 0 0 1 1 0 0 0 1 1 1 1 1 1 1 0 1
while x Q (n2 ) is
1 0 0 0 0 1 0 1 1 1 0 1 0 1 0 0 0 0 0 .
If the exponent cannot be written within the defined number of bits (here
7) the computer has to stop the calculation and indicate "overflow", that
is, the number cannot fit into the register. For mantissa the values are just
rounded to the available number of bits. In the implementations based on
the floating-point arithmetic, the quantization affects the mantissa only. The
relative error in mantissa is again
1
|e(n)| ≤ 2−(b+1) = ∆.
2
The error in signal is multiplied by the exponent. Since we can say that the
exponent value is of the signal order, we can write
Q[ x (n)] = xQ (n) = x (n) + e(n) x (n) = x (n)(1 + e(n)).
The error behaves here as a multiplicative uniform noise. Thus, for the
floating-point representation, multiplicative errors appear.
The floating-point additions also produce the quantization errors,
which are represented by a multiplicative noise. During additions the num-
ber of bits may increase. This increase in the number of bits requires man-
tissa shift, what causes multiplicative error.
In addition to the IEEE standard when the total number of bits is 32
(23 for mantissa and 7 for exponent) we will mention two standard formats
for the telephone signal coding. The µ-law pulse-coded modulation (PCM)
is used in the North America and the A-law PCM is used in European tele-
phone networks. They use 8-bit representations with a sign bit, 3 exponent
bits, and 4 mantissa bits
s e1 e2 e3 m1 m2 m3 m4 .
The µ-law encoding takes a 14-bit signed signal value (its two’s complement
representation) as input, adds 33 (binary 100001) and converts it to an 8 bit
value. The encoding formula in the µ-law is
@ A
(−1)s 2e+1 (m + 16.5) − 33 .
This is a 14-bit signed integer from −8031 to +8031.

The sign bit s is set to 1 if the input sample is negative. it is set to 0 if
the input sample is positive. Number 0 is written as
0 0 0 0 0 0 0 0 .
As an example consider the positive numbers from +1 to +30. They

are written as +21 (m + 16.5) − 33 with 15 quantization step 2 (starting
from m = 1 to m = 15). Then the numbers from +31 to +94 are written
as +22 (m + 16.5) − 33 with 16 quantization steps 4 (with m from 0 to 15).
The last interval for positive numbers is from +4063 to +8158 written as
+28 (m + 16.5) − 33 with 16 quantization intervals (with m from 0 to 15)
of 256. The range of input values is from −8159 to +8159 (±213 ) with the
minimum step size 2 for the smallest amplitudes.
Compression function corresponding to this format of signal 0 ≤ | x | ≤
1 is
ln(1 + µ | x |)
F ( x ) = sign( x ) .
ln(1 + µ)
with µ = 255.
Example 7.30. Write the number a = 456 in the binary µ-law format.
⋆The number to be represented by 2 e+1 (m + 16.5) is 456 + 33 = 489.
The mantissa range is 0 ≤ m ≤ 15. It means that the exponent (e + 1) should
be such that
489
0 + 16.5 ≤ e+1 ≤ 15 + 16.5
2
for the range 16.5 ≤ m + 16.5 ≤ 31.5. It is easy to conclude that 489/16 =
30.5625, meaning e + 1 = 4 with m + 16.5 = 30.5625. The nearest integer value
of m is m = 14. Therefore â = 23+1 × (14 + 16.5) − 33 = 455 is the nearest µ-
law format number to a. The binary form is
0 0 1 1 1 1 1 0 .
Quantization step for this range of numbers is 2 4 = 16. It means that the
previous possible number is 439, while the next possible number would be
471. It is the last number with 2 e+1 = 16.
Example 7.31. Write a model for calculation of

r (n, m) = x (n + m) x (n − m)
if the quantization error is caused by floating point registers with b bits for
mantissa. What is the mean value? Write the model for
y ( n ) = x ( n ) + x ( n + 1).
Signals are real-valued.

⋆For this calculation the model is
r̂ (n, m) = x (n + m)(1 + e(n + m)) x (n − m)(1 + e(n − m))(1 + e(n + m, n − m)).
The mean value is
E{r̂ (n)} = x (n + m) x (n − m) + E{e(n + m)e(n − m)}

1 2
= r (n) + E{e2 (n)}δ(m) = r (n) + ∆ δ ( m ).
12
For y(n) the model is
ŷ(n) = [ x (n)(1 + e(n)) + x (n + 1)(1 + e(n + 1))](1 + e(n, n + 1))
where e(n, n + 1) is the is the multiplicative noise modeling the addition error.
7.9 PROBLEMS
Problem 7.1. Signal x20i (n), for i = 01, 02, .., 15, is the monthly average of
maximal daily temperatures in a city measured from year 2001 to 2015.
Values are given in Table 7.2. If we can assume that the signal for individual
month is Gaussian find the probability that the average of maximal daily
temperatures: (a) in January is lower than 2, (b) in January is higher than 12.
Problem 7.2. Cumulative probability distribution function F (χ) is given as

⎧
⎪
⎪ 0 χ≤0
⎪
⎪
⎨ χ/2 0<χ≤1
F (χ) = 1/2 1<χ≤2
⎪
⎪
⎪
⎪ (χ − 1)/2 2 < χ ≤ 3
⎩
1 χ > 3.
Find the probability density function p(ξ ) and the probability that x (n) <
2.5.
Problem 7.3. Probability density function of x (n) is
p x (ξ ) = ae−b|ξ | , −∞ < ξ < ∞,
where a and b are constants. Find the relation between a and b. What is the
cumulative probability distribution function for a = 1?
Problem 7.4. Random signal x (n) has a probability density function
λ −λ|ξ |
p x (ξ ) = e , λ > 0.
2
Find the mean and variance of x (n).

Problem 7.5. Joint probability density function of signals x (n) and y(n) is
!
kξe−ξ (ζ +1) 0≤ξ<∞ 0≤ζ<∞
p xy (ξ, ζ ) =
0 elsewhere.
Find the value of constant k.

Problem 7.6. A set of N = 10 balls is considered, with equal number of
balls being marked with 1 (or white) and 0 (or black). A random signal x (n)
corresponds to drawing four balls in a row. It has four values x (0), x (1),
x (2), and x (3). Signal values are equal to the number on randomly drawn
ball. If k is the number of values 0 that appear in the signal (number of black
balls), write the probability for k = 0. Generalize the result for N ballas and
M signal samples.
Problem 7.7. Random signal x (n) is such that x (n) = 0 with probability
0.8. In all other cases x (n) is Gaussian random variable with mean 3 and
variance 2. Find the mean and variance of x (n).
Problem 7.8. Signal ε(n) is a Gaussian noise with mean µε = 0 and variance
σε2 . Find probability that |ε(n)| > A. If the signal length is N = 2000 find the
expected number of samples with amplitudes higher than A = 10 assuming
that σε2 = 2. What is the result for A = 4 and σε2 = 2.
Problem 7.9. Random signal x (n) is a Gaussian noise with mean 0 and
variance σx2 . Signal has a large number N of samples. Random sequence
y(n) is obtained by using M samples from signal x (n) with the lowest
amplitudes. Find µy and σy .
Problem 7.10. Random signal x (n) is a Gaussian noise with mean 0 and
variance σx2 . Random sequence y(n) is obtained by omitting samples from
signal x (n) whose amplitude is higher than A. Find the probability density
function of sequence y(n). Find µy and σy .
Problem 7.11. Signal samples x (n) are such that
!
A + ε(n) for n ∈ N x
x (n) =
ε(n) otherwise
where ε(n) is a Gaussian noise with mean µε = 0 and variance σε2 , A > 0
is a constant and N x is nonempty set of discrete time instants. A threshold
based criterion is used to detect if an arbitrary time instant n belongs to the
set N x
n ∈ N x if x (n) > T,
where T is threshold. Find threshold T if the probability of false detection is
0.01.
Problem 7.12. Signal x (n) is a random Gaussian sequence with mean
µ x = 5 and variance σx2 = 1. Signal y(n) is a random Gaussian sequence,
independent from x (n), with mean µy = 1 and variance σy2 = 1. If we
consider N = 1000 samples of these signals find the expected number of
time instants where x (n) > y(n) holds.
Problem 7.13. Let x (n) and y(n) be independent real-valued white Gaus-
sian random variables with means µ x = µy = 0 and variances σx2 and σy2 .
Show that the random variable
1 M
M n∑
z= x (n)y(n)
=1
has the variance

1 2 2
σz2 = σ σ .
M x y
Problem 7.14. A random signal ε(n) is stationary and Cauchy distributed
with probability density function
a
pε(n) (ξ ) = .
1 + ξ2
Find the coefficient a, mean, and variance of signal.

Problem 7.15. A causal system is defined by
y(n) = x (n) + 0.5y(n − 1).
Input signal is x (n) = aδ(n) with a random amplitude a. Random variable a

is uniformly distributed within the interval from 4 to 5. Find the mean and
autocorrelation of the output signal. Is the output signal WSS?
Problem 7.16. Consider a Hilbert transformer with the impulse response
% 2
2 sin (nπ/2)
h(n) = π n , n ̸= 0 .
0, n=0
Input signal is a white noise with variance 1.

a) Find the autocorrelation function of the output signal.
b) Find the cross-correlation of the input and output signal. Show that
it is an antisymmetric function.
c) Find the autocorrelation and the power spectral density function of
an analytic signal ε a (n) = ε(n) + jε h (n), where ε h (n) = ε(n) ∗n h(n).
Problem 7.17. Consider a causal system
y(n) − ay(n − 1) = x (n).
If the input signal is white noise x (n) = ε(n), with the autocorrelation
function rεε (n) = σε2 δ(n), find the autocorrelation and the power spectral
density of the output signal.
Problem 7.18. Consider a linear time-invariant system whose input is
x (n) = ε(n)u(n)
and the impulse response is
h ( n ) = a n u ( n ),
where ε(n) is a stationary real-valued noise with mean µε and autocorrela-

tion rεε (n, m) = σε2 δ(n − m) + µ2ε . Find the mean and variance of the output
signal.
Problem 7.19. Find the mean, autocorrelation, and power spectral density
of random signal
N
x (n) = ε(n) + ∑ a k e j ( ωk n + θk ) ,
k =1
where ε(n) is a stationary real-valued noise with mean µε and autocorre-
lation rεε (n, m) = σε2 δ(n − m) + µ2ε and θk are random variables uniformly
distributed over −π < θk ≤ π. All random variables are statistically inde-
pendent.
Problem 7.20. Find a stable optimal filter if the correlation functions for the
signal and noise are rss (n) = 0.25|n| , rsε (n) = 0 and rεε (n) = δ(n). Discuss
filter causality.
Problem 7.21. Calculate the DFT value X (2) for x (n) = exp( j4πn/N ) with
N = 8 and noise ε(n) = 2001δ(n) − 204δ(n − 3) using
N −1
X (k ) = ∑ (s(n) + ε(n))e− j2πkn/N
n =0
and estimate the DFT using

M N
XR (k) = N median Re (s(n) + ε(n))e− j2πkn/N
n=0,1,..,N −1
M N
+ jN median Im (s(n) + ε(n))e− j2πkn/N .
n=0,1,..,N −1
Problem 7.22. The spectrogram is one of the most commonly used tools in
time-frequency analysis. Its form is
' '2
' N −1 '
' 2π '
Sx (n, k ) = ' ∑ x (n + i )w(i )e− j N ik '
' i =0 '
where the signal is x (n) = s(n) + ε(n), with s(n) being the desired deter-
ministic signal and ε(n) a complex-valued, zero-mean white Gaussian noise
with variance σε2 and independent and identically distributed (i.i.d.) real
and imaginary parts. Window function is w(i ). Using a rectangular window
of the width N find:
a) the mean of Sx (n, k ),
b) the variance of Sx (n, k ).
Note: For a Gaussian random signal ε(n), it holds
E{ε(l )ε∗ (m)ε∗ (n)ε( p)} = E{ε(l )ε∗ (m)} E{ε∗ (n)ε( p)}
+ E{ε(l )ε∗ (n)} E{ε∗ (m)ε( p)} + E{ε(l )ε( p)} E{ε∗ (m)ε∗ (n)}.
Problem 7.23. The basic time-frequency distribution is the Wigner distribu-

tion, whose discrete-time form reads
L
Wx (n, ω ) = ∑ x (n + k ) x ∗ (n − k )e− j2ωk
k =− L
where the signal is x (n) = s(n) + ε(n), with s(n) being the desired deter-
ministic signal and ε(n) complex-valued, zero-mean white Gaussian noise
with variance σε2 and independent and identically distributed (i.i.d.) real
and imaginary parts. Find:
a) the mean value of Wx (n, ω ),
b) the variance of Wx (n, ω ).
Use the previous problem note. Write the variance form for an FM
signal when |s(n)| = A.
Problem 7.24. A random signal s(n) carries an information. Its autocorre-
lation function is rss (n) = 4(0.5)|n| . A noise with variance of autocorrelation
rεε (n) = 2δ(n) is added to the signal. Find the optimal filter for:
1.5 jω
S (e )
dd
0.5
0
-3 -2 -1 0 1 2 3
1.5
S (ejω)
εε
0.5
0
-3 -2 -1 0 1 2 3
1.5
H(ejω)
1
0.5
0
-3 -2 -1 0 1 2 3
' '2
Figure 7.18 Power spectral densities of the signal 'S(e jω )' and input noise Sεε (e jω ) along
jω
with the frequency response of an optimal filter H (e ).
a) d(n) = s(n) - optimal filtering,

b) d(n) = s(n − 1) - optimal smoothing,
c) d(n) = s(n + 1) - optimal prediction.
Problem 7.25. Design an optimal filter if the autocorrelation function of the

signal is rss (n) = 3(0.9)|n| . The autocorrelation of noise is rεε (n) = 4δ(n),
while the cross–correlation of the signal and noise is rsε (n) = 2δ(n) .
Problem 7.26. The power spectral densities of the signal Sdd (e jω ) and input
noise Sεε (e jω ) are given in Fig.7.18. Show that the frequency response of the
optimal filter H (e jω ) is presented in Fig.7.18(bottom). Find the SNR at the
input and output of the optimal filter.
Problem 7.27. Find the mean of quantization error of the Wigner distribu-
tion (its pseudo form over-sampled in frequency)
N −1
Wx (n, k ) = ∑ x (n + m) x (n − m)e− j2πmk/N
m =0
where x (n) is real-valued quantized signal. The product of signals is quan-

tized to b bits as well. Neglect the quantization of the coefficients e− j2πmk/N
and the quantization of their products with the signal.
7.10 SOLUTIONS
Solution 7.1. (a) The mean value for January, Table 7.2, is
µ x (1) = 7.2667.
The standard deviation for January, calculated over 15 years, is σx (1) =

2.7115. Probability that the average maximal temperature in January is
lower than 2 is
"2 (ξ −µ x (1)) 2
1 −
P ( x (1) < 2) = √ e 2σx2 (1) dξ =
σ (1) 2π
−∞ x
- .
7.2667 − 2
= 0.5 1 − erf( √ ) = 0.0260.
2.7115 2
It means that this even will occur once in about 40 years.

(b) The average maximal temperature is higher than 12 with probabil-
ity
"∞ (ξ −µ x (1)) 2
1 −
P( x (1) > 12) = √ e 2σx2 (1) dξ =
σx (1) 2π
12
- .
12 − 7.2667
0.5 1 − erf( √ ) = 0.0404.
2.7115 2
It means that this will happen once in about 25 years.

Solution 7.2. For the cumulative probability distribution function

⎧
⎪
⎪ 0 χ≤0
⎪
⎪
⎨ χ/2 0<χ≤1
F (χ) = 1/2 1<χ≤2
⎪
⎪
⎪
⎪ (χ − 1)/2 2 < χ ≤ 3
⎩
1 χ>3
the probability density function is the derivative of F (χ),

⎧
⎪
⎪ 0 ξ≤0
⎪
⎪
⎨ 1/2 0 < ξ ≤ 1
dF (ξ )
p x (ξ ) = = 0 1<ξ≤2
dξ ⎪
⎪
⎪
⎪ 1/2 2 <ξ≤3
⎩
0 ξ > 3.
The probability of x (n) < 2.5 is P( x (n) < 2.5) = F (2.5) = 0.75.
Solution 7.3. Integral of the probability density function over (−∞, ∞) is

" ∞
p x (ξ )dξ = 1.
−∞
Therefore
" ∞ -" 0 " ∞ .
2a
ae−b|ξ | dξ = a ebξ dξ + e−bξ dξ = = 1,
−∞ −∞ 0 b
resulting in b = 2a.
For a = 1 the probability density function is p x (ξ ) = e−2|ξ | for −∞ <
ξ < ∞ . The probability distribution function is
" χ " χ
e2χ
Fx (χ) = p x (ξ )dξ = e2ξ dξ =
−∞ −∞ 2
for −∞ < χ < 0, and

" χ " χ
1 1 e−2χ
Fx (χ) = + pϵ (ξ )dξ = + e−2ξ dξ = 1 −
2 0 2 0 2
for 0 < χ < ∞.

Solution 7.4. The mean is

" ∞ " 0 " ∞
Ex = ξ p x (ξ )dξ = ξeλξ dξ + ξe−λξ dξ
−∞ −∞ 0
" ∞ " ∞
=− ξe−λξ dξ + ξe−λξ dξ = 0.
0 0
The variance of x is obtained from

M N " ∞ 2
σx2 (n) = E | x (n) − E{ x (n)}|2 = E{ x2 (n)} = ξ 2 p x (ξ )dξ = .
−∞ λ2
Solution 7.5. Since " ∞ " ∞

p xy (ξ, ζ )dξdζ = 1
−∞ −∞
" ∞" ∞ " ∞ " ∞

kxe−ξ (ζ +1) dξdζ = kξe−ξ dξ e−ξζ dζ
0 0 0 0
" ∞ " ∞
1
= kξe−ξ dξ = ke−ξ dξ = k
0 ξ 0
the value of constant k is 1.

Solution 7.6. There are 5 out of 10 black balls. Probability that x (0) = 0 is
5
P0 = .
10
If the first ball was 0 then we have 9 balls for the second draw, with 4 balls
marked with 0. The probability that x (1) = 0 if x (0) = 0 is
4
P1 = .
9
If x (0) = 0 and x (1) = 0 then there are 8 remaining balls with 3 of them being
marked with 0. The probability that x (2) = 0, with x (0) = 0 and x (1) = 0, is
3
P2 = .
8
The probability for k = 0 is
5 432
P ( k = 0) = .
10 9 8 7
In general, if there were N balls and we considered M signal samples

(drawings), the probability P(k = 0) would be
M −1
N/2 − i
P ( k = 0) = ∏ N−i
.
i =0
Solution 7.7. Let us find probability that x (n) < ξ for arbitrary ξ. Consider
the case when ξ < 0,
* * ++
0.2 ξ −3
P{ x (n) < ξ } = 1 + erf .
2 2
It has been taken into account that the considered sample is Gaussian (with
probability 0.2), along with the probability that the sample value is smaller
than ξ.
For ξ > 0 we should take into account that the signal assumes x (n) = 0
with probability 80% as well as that in the remaining 20% cases, Gaussian
random value could be smaller than ξ. So we get
* * ++
0.2 ξ −3
P{ x (n) < ξ } = 0.8 + 1 + erf .
2 2
Now we have
⎧ B B CC
⎪ 0.2 ξ −3
⎪
⎨ 2 1 + erf 2 forξ < 0
P{ x (n) < ξ } = B B CC
⎪
⎪ ξ −3
⎩ 0.8 + 0.2
1 + erf for ξ > 0
2 2
This function has a discontinuity at ξ = 0. It is not differentiable at this

point as well. Derivative of P{ x (n) < ξ } can be expressed in a form of the
generalized functions (Dirac delta function) as
d 0.2 ( ξ −3)2
P{ x (n) < ξ } = py(n) (ξ ) = √ e− 4 + 0.8δ(ξ ).
dξ 2 π
The mean and variance are
"∞
µy(n) = ξ py(n) (ξ )dξ = 0.2 × 3 + 0.8 × 0 = 0.6
−∞
"∞
σy2(n) = (ξ − 0.6)2 py(n) (ξ )dξ = 0.2 × 7.76 + 0.8 × (0.6)2 = 1.84.
−∞
Solution 7.8. Probability that |ε(n)| > A is
P{|ε(n)| > A} = P{ε(n) < − A} + P{ε(n) > A}

−A
" ζ2 "∞ ζ2
1 −
2σε2
1 −
2σε2
= e√ dζ + e √ dζ
σε 2π σε 2π
−∞ A
* +
A
= 1 − erf √ .
2σε
For A = 10 and σε2 = 2 we get
P{|ε(n)| > 10} = 1 − erf(5) ≈ 1.5 × 10−12 .
For N = 2000 the expected number of samples with amplitude above A is

P{|ε(n)| > 10} × 2000 ≈ 3 × 10−9 ≈ 0. It means that we do not expect any
sample with amplitude higher than 10.
For A = 3 we have
P{|ε(n)| > A} = 1 − erf (2) ≈ 4.7 × 10−3
with 2000 × 4.7 × 10−3 = 9.4 ≈ 9 samples among considered 2000 assuming
an amplitude higher than 3.
Solution 7.9. If we are in position to use a reduced set of signal samples
for processing, then the ideal scenario would be to eliminate signal samples
with higher noise values and to keep for processing the samples with lower
noise values. For the case of N signal samples and signal processing based
on M samples we can find the interval of amplitudes A for the lowest M
noisy samples. The probability that | x (n)| < Aσε is
"Aσε
1 2 / (2σ2 )
P{| x (n)| < Aσε } = √ e−ξ ε dξ.
σε 2π
− Aσε
Since we use M out of N samples this probability should be equal to M/N,
"A * +
1 2 /2 A M
√ e−ξ dξ = erf √ = .
2π 2 N
−A
√( x ) function
The calculation of A value is easily related to the inverse erf
denoted by erfinv( x ). For a given M/N, the amplitude is A = 2erfinv( M N ).
For example, for M = N/2 a half of the lowest √ noise samples will be within
the interval [−0.6745σε , 0.6745σε ] since A = 2erfinv(0.5) = 0.6745.
The probability density function of the new noise is
% 2 2
√k e−ξ /(2σε ) for |ξ | < Aσε
py (ξ ) = σε 2π
0 for |ξ | ≥ Aσε .
&∞
The constant k is obtained from the condition that py (ξ )dξ = 1. It is
−∞
k = N/M.
The variance of this new noise, formed from the Gaussian noise after
the largest N − M values are removed, is much lower than the variance of
the whole noise. It is
√
2erfinv( M
N ) σε
N "
2 / (2σ2 )
σy2 = M
√ ξ 2 e−ξ ε dξ. (7.66)
σε 2π √
− 2erfinv( M
N ) σε
Solution 7.10. The probability density function for sequence y(n) is

⎧ 2
⎨ 1 − (ζ )2
py(n) (ζ ) = B √ e 2σx for − A < ζ ≤ A
σx 2π
⎩
0 otherwise.
&∞
Constant B can be calculated from −∞ py(n) (ζ )dζ = 1. Its value is
* +
A
B = 1/ erf √ .
σx 2
Now we have
µy(n) = 0
"A 2
1 1 − (ζ )2
σy2(n) = ζ 2 B C √ e 2σx dζ
A σx 2π
−A
erf √
σx 2
⎛ ⎞
√ − A2
A 2e 2σx2
⎜ C⎟
= σx2 ⎝1 − √ B ⎠.
σx π erf A
√
σx 2
√
By denoting β = A/( 2σx ), the variance σy2(n) can be written as
( 2
)
e− β
σy2(n) = σx2 1 − 2β √ .
π erf ( β)
Solution 7.11. False detection means that we make a wrong decision by

classifying instant n into set N x . The probability is
* +
1 1 T
PF = P{ε(n) > T } = − erf √
2 2 2σε
Now we can find T as

√
T= 2σε erfinv(1 − 2PF ) ≈ 2.33σε
where erfinv(·) is the inverse erf function. Note that the threshold does not
depend on A.
Solution 7.12. The joint probability distribution is
p x(n),y(n) (ξ, ζ ) = p x(n) (ξ ) py(n) (ζ )
since signals are mutually independent. Probability that x (n) > y(n) can be
obtained by integrating p x(n),y(n) (ξ, ζ ) over the region ξ > ζ. It is
"∞ ( ξ −5 )2
"ξ ( ζ −1)2
1 1
P{ x (n) > y(n)} = √ e− 2 √ e− 2 dζdξ ≈ 0.99766.
2π 2π
−∞ −∞
For 1000 instants we expect that x (n) > y(n) is satisfied in about 998
instants.
Solution 7.13. Since the variable
1 M
M n∑
z= x (n)y(n)
=1
is also of zero-mean then its variance is

> ?
2 2 1 M 1 M
M n∑ M m∑
σz = E[z ] = E x (n)y(n) x (m)y(m)
=1 =1
1 M M 1 M M
= ∑ ∑
M 2 n =1 m =1
E [ x ( n ) y ( n ) x ( m ) y ( m )] =
M2 n∑ ∑ E[x(n)x(m)] E[y(n)y(m)]
=1 m =1
1 M 2 2 1 M 2 2 1 2 2
= 2 ∑
M n =1
E [ x ( n )] E [ y ( n )] = 2 ∑
M n =1
σx σy = σ σ .
M x y
Solution 7.14. Probability that the random variable is within −∞ < ξ < ∞
is
"∞ "∞
a ∞
1= pε(n) (ξ )dξ = dξ = a arctan(ξ )|− ∞ = aπ,
1 + ξ2
−∞ −∞
resulting in a = 1/π. The mean value is
"∞
1 ξ
µε = dξ = 0,
π 1 + ξ2
−∞
while the variance

"∞
1 ξ2
σε = dξ → ∞
π 1 + ξ2
−∞
does not exist. This noise belongs to the class of impulsive, heavy tailed,
noises.
Solution 7.15. The transfer function of a causal system is
1
H (z) = .
1 − 0.5z−1
The z-transform of the input signal x (n) is

∞ ∞
X (z) = ∑ x (n )z−n = ∑ aδ(n)z−n = a.
n=−∞ n=−∞
The z-transform of the output signal is
a
Y (z) = H (z) X (z) = , |z| > 1/2.
1 − 0.5z−1
Using the power series expansion of Y (z) we can write
∞
Y (z) = a ∑ (1/2)n z−n .
n =0
The output signal is

y ( n ) = a · 2− n u ( n ).
It has been assumed that the random variable a is uniform within [4, 5]. Its
probability density function is
!
1, ξ ∈ [4, 5]
p a (ξ ) =
0, elsewhere.
The mean value and autocorrelation of the output signal y(n) are
" ∞
µy (n) = E {y(n)} = y(n) p( a)da = 9 · 2−(n+1) u(n)
−∞
61 −(n+m)
ryy (n, m) = E {y(n)y∗ (m)} = 2 u ( n ) u ( m ).
3
The output signal y(n) is not WSS.
Solution 7.16. a) The autocorrelation function of the input signal is
r xx (n) = rεε (n) = δ(n).
Its z-transform and power spectral density are
∞
R xx (z) = ∑ r xx (n)z−n = 1
n=−∞
Sxx (ω ) = 1.

' '2
' '
Syy (ω ) = Ryy (e jω ) = Sxx (ω ) 'H (e jω )' = 1, for ω ̸= 0.
The inverse Fourier transform produces the autocorrelation function

" π
1
ryy (n) = rε h ε h (n) = Syy (ω )e jωn dω = δ(n).
2π −π
b) The z-transform of the cross-correlation of input and output signal y(n) =

ε(n) ∗ h(n) = ε h (n), is
R xy (z) = R xx (z) H (z).
For z = e jω we get
Rεε h (e jω ) = Sεε (ω ) H (e jω ) = H (e jω ),
resulting in % 2
2 sin (nπ/2)
rεε h (n) = h(n) = π n , n ̸= 0
0, n = 0.
It is easy to conclude that the cross-correlation function is antisymmetric
r xy (−n) = −r xy (n).
c) The analytic part of signal x (n) = ε(n) is
∞
x a (n) = ε a (n) = x (n) + jxh (n) = x (n) + j ∑ h ( k ) x ( n − k ).
k=−∞
The Fourier transform of both sides produces
Xa (e jω ) = X (e jω ) + jH (e jω ) X (e jω ).
If we divide both sides by X (e jω ) we get
Xa (e jω )
= Ha (e jω ) = 1 + jH (e jω ) = 1 + sgn(ω )
X (e jω )
⎧
⎨ 2, ω>0
= 1, ω=0
⎩
0, ω < 0.

' '2 ' '2
' ' ' '
Sε a ε a (ω ) = 'Ha (e jω )' Sεε (ω ) = 'Ha (e jω )'
⎧
⎨ 4, ω>0
= 1, ω=0
⎩
0, ω<0
with the autocorrelation function of ε a (n),

⎧ 4
1
" π ⎨ − jnπ , for odd n
rε a ε a (n) = Sε a ε a (ω )e− jωn dω =
2π −π ⎩
0, for even n
Solution 7.17. The power spectral density of the input signal is
∞ ∞
Sxx (ω ) = ∑ r xx (n)e− jωn = ∑ rεε (n)e− jωn = σε2 .
n=−∞ n=−∞

1 z
H (z) = − 1
=
1 − az z−a
with the impulse response h(n) = an u(n) . It is real-valued. The z-transform
of the output signal autocorrelation function is
z
Ryy (z) = H (z) H (1/z) R xx (z) = σ2 .
(z − a)(1 − az) ε
The inverse z-transform results in the autocorrelation function of y(n)
a|n| 2
ryy (n) = σ .
1 − a2 ε
Power spectral density of the output signal is
σε2 σε2
Syy (ω ) = Ryy (e jω ) = −
= .
(1 − ae jω )(1 − ae jω ) 1 − 2a cos ω + a2
Solution 7.18. The mean of y(n) is
% ;
∞ ∞
µy (n) = E ∑ h(k ) x (n − k ) = ∑ ak E{ε(n − k)}u(n − k)
k=−∞ k =0
n
1 − a n +1
= ∑ ak µε = µε u ( n ).
k =0
1−a
The variance is
MP Q2 N
σy2 (n) = E y(n) − µy (n) = E{y2 (n)} − µ2y (n)
n n * +2
k1 k2 1 − a n +1
= ∑ ∑ a a E { ε ( n − k 1 ) ε ( n − k 2 )} u ( n ) − µ ε
1−a
u ( n ).
k =0 k =0
1 2
Since E{ε(n − k1 )ε(n − k2 )} = σε2 δ(k1 − k2 ) + µ2ε , we get
1 − a 2( n +1)
σy2 (n) = σε2 u ( n ).
1 − a2
Solution 7.19. The mean value is
N
µ x = µε + ∑ a k E{ e j ( ωk n + θ k ) } = µ ε ,
k =1
since
"π
j ( ωk n + θk ) 1
E{ e }= e j(ωk n+θk ) dθk = 0.
2π
−π
The autocorrelation is
N
r xx (n) = σε2 δ(n) + µ2ε + ∑ a2k e jωk n ,
k =1
while the power spectral density for −π < ω ≤ π is
N
Sxx (e jω ) = FT{r xx (n)} = σε2 + 2πµ2ε δ(ω ) + 2π ∑ a2k δ(ω − ωk ).
k =1
Solution 7.20. For the optimal filtering d(n) = s(n). The cross-correlation of
the input and desired signal is
rdx (n) = E{d(k ) x (k − n)} = E{s(k )[s∗ (k − n) + ε∗ (k − n)]}

= rss (n) = 0.25|n| .
Its z-transform is
−15z/4
Rdx (z) = Rss (z) = .
(z − 1/4)(z − 4)
The input signal autocorrelation is
r xx (n) = rss (n) + rεε (n) = 0.25|n| + δ(n),
with the z-transform
−15z/4 z2 − 8z + 1
R xx (z) = +1= .
(z − 1/4)(z − 4) (z − 1/4)(z − 4)
The optimal filter transfer function is

Rdx (z) Rss (z) −15z/4
H (z) = = = 2 .
R xx (z) R xx (z) z − 8z + 1
A stable system requires the region of convergence 0.127 < |z| < 7.873. It is
not causal.
Solution 7.21. By direct calculation, in noisy case, we obtain
X (2) = 2001 + j204
and
XR (k ) = 8.
Note that the noise-free DFT value X (2) is 8.
Solution 7.22. With a rectangular window the spectrogram reads
' '2
' N −1 ' N −1 N −1
' − j 2π ' 2π
Sx (n, k ) = ' ∑ x (n + i )e N ' = ∑ ∑ x (n + i1 ) x ∗ (n + i2 )e− j N (i1 −i2 )k .
ik
' i =0 ' i =0 i =0
1 2
a) The mean value of spectrogram is

N −1 N −1 2π
E{Sx (n, k)} = E{ ∑ ∑ x ( n + i 1 ) x ∗ ( n + i 2 ) e − j N ( i1 − i2 ) k }
i1 =0 i2 =0
N −1 N −1 2π
= ∑ ∑ E{x(n + i1 )x∗ (n + i2 )}e− j N (i1 −i2 )k .
i1 =0 i2 =0
Using the fact that the signal s(n) is deterministic and the noise ε(n) is zero-
mean white stationary we get
N −1 N −1 2π
E{Sx (n, k )} = ∑ ∑ s ( n + i 1 ) s ∗ ( n + i 2 ) e − j N ( i1 − i2 ) k
i1 =0 i2 =0
N −1 N −1 2π
+ ∑ ∑ E{ε(n + i1 )ε∗ (n + i2 )}e− j N (i1 −i2 )k
i1 =0 i2 =0
or
N −1 N −1 2π
E{Sx (n, k )} = Ss (n, k ) + σε2 ∑ ∑ δ ( i 1 − i 2 ) e − j N ( i1 − i2 ) k
i1 =0 i2 =0
N −1
= Ss (n, k) + σε2 ∑ 1 = Ss (n, k) + Nσε2
i =0
since for the noise holds

∗
rεε (i ) = E{ε(n + i )ε (i )} = σε2 δ(i ).
b) The variance of Sx (n, k ) is
σ2 = E{Sx (n, k )S∗x (n, k )} − E{Sx (n, k )}E{S∗x (n, k )}.
The first term can be written as

N −1 N −1 N −1 N −1
E{Sx (n, k )S∗x (n, k)} = ∑ ∑ ∑ ∑ E{ x ( n + i1 ) x ∗ ( n + i2 )
i1 =0 i2 =0 i3 =0 i4 =0
2π
× x ∗ (n + i3 ) x (n + i4 )}e− j N (i1 −i2 −i3 +i4 )k
where
E{ x (n + i1 ) x ∗ (n + i2 ) x ∗ (n + i3 ) x (n + i4 )}
= s ( n + i1 ) s ∗ ( n + i2 ) s ∗ ( n + i3 ) s ( n + i4 )
+ s(n + i1 )s∗ (n + i2 )rεε (i4 − i3 ) + s(n + i1 )s∗ (n + i3 )rεε (i4 − i2 )
+ s∗ (n + i2 )s(n + i4 )rεε (i1 − i3 ) + s∗ (n + i3 )s(n + i4 )rεε (i1 − i2 )
+ E{ε(n + i1 )ε∗ (n + i2 )ε∗ (n + i3 )ε(n + i4 )}.
The facts that odd order moments of a Gaussian zero-mean noise are zero
and rεε∗ (k ) = rε∗ ε (k ) = 0 for a complex-valued noise with i.i.d. are used.
According to the relation from the note it holds
E{ε(n + i1 )ε∗ (n + i2 )ε∗ (n + i3 )ε(n + i4 )}

= rεε (i1 − i2 )rεε (i4 − i3 ) + rεε (i1 − i3 )rεε (i4 − i2 ).
After few straightforward transformations, we get
E{Sx (n, k )S∗x (n, k )} = Ss2 (n, k )

N −1 N −1 N −1 N −1
+σε2 ∑ ∑ ∑ ∑ [ s ( n + i1 ) s ∗ ( n + i2 ) δ (i4 − i3 )
i1 =0 i2 =0 i3 =0 i4 =0
+ s ( n + i1 ) s ∗ ( n + i3 ) δ (i4 − i2 ) + s ( n + i4 ) s ∗ ( n + i2 ) δ (i1 − i3 )
∗ − j 2π
N ( i1 − i2 − i3 + i4 ) k
+s(n + i4 )s (n + i3 )δ(i1 − i2 )]e
N −1 N −1 N −1 N −1
+σε4 ∑ ∑ ∑ ∑ [ δ (i1 − i2 ) δ (i4 − i3 )
i1 =0 i2 =0 i3 =0 i4 =0
2π
+δ(i1 − i3 )δ(i4 − i2 )]e− j N (i1 −i2 −i3 +i4 )k .
The final form of the variance is
σ2 = Ss2 (n, k ) + 4Nσε2 Ss (n, k ) + 2N 2 σε4 − (Ss (n, k ) + Nσε2 )2

= 2Nσε2 Ss (n, k) + N 2 σε4 .
The variance is proportional to the signal spectrogram values Ss (n, k ).

Solution 7.23. a) Mean value of the Wigner distribution of x (n) is
L
E{Wx (n, ω )} = ∑ E{ x (n + k ) x ∗ (n − k )}e− j2ωk .
k=− L
Signal is deterministic and not correlated with the white noise ε(n),
E{ x (n + k ) x ∗ (n − k )} = s(n + k )s∗ (n − k ) + rεε (2k ),
where rεε (2k ) is the autocorrelation function of the additive noise ε(n). The
noise variance is σε2 . Then
E{ x (n + k ) x ∗ (n − k )} = s(n + k )s∗ (n − k ) + σε2 δ(2k ).
Final form of the mean value of Wx (n, ω ) is
E{Wx (n, ω )} = Ws (n, ω ) + σε2 .
b) Variance of Wxx (n, ω ) follows from
σ2 = E{Wx (n, ω )Wx∗ (n, ω )} − E{Wx (n, ω )}E{Wx∗ (n, ω )}.
The first term can be written as
E{Wx (n, ω )Wx∗ (n, ω )} =

L L
= ∑ ∑ E{ x (n + k1 ) x ∗ (n − k1 ) x ∗ (n + k2 ) x (n − k2 )}e− j2ω (k1 −k2 ) .
k1 =− L k2 =− L
In the case of a Gaussian zero-mean white stationary noise complex-valued

noise with i.i.d. real and imaginary parts rεε∗ (k ) = rε∗ ε (k ) = 0 we can write
E{ x (n + k1 ) x ∗ (n − k1 ) x ∗ (n + k2 ) x (n − k2 )}
= s(n + k1 )s∗ (n − k1 )s∗ (n + k2 )s(n − k2 ) + s(n + k1 )s∗ (n − k1 )rεε (−2k2 )
+ s(n + k1 )s∗ (n + k2 )rεε (k2 − k1 ) + s∗ (n − k1 )s(n − k2 )rεε (k1 − k2 )
+ s∗ (n + k2 )s(n − k2 )rεε (2k1 ) + rεε (2k1 )rεε (−2k2 ) + rεε
2
( k 1 − k 2 ).
The note from the previous problem is used. Since
E{Wx (n, ω )}E{Wx∗ (n, ω )} = Ws2 (n, ω ) + 2σε2 Ws (n, ω ) + σε4
the Wigner distribution variance is
L L
σ2 = ∑ ∑ [s(n + k1 )s∗ (n + k2 )rεε (k2 − k1 ) + rεε
2
(k1 − k2 )
k1 =− L k1 =− L
L
+s∗ (n − k1 )s(n − k2 )rεε (k1 − k2 )]e− jω (k1 −k2 ) = σε2 ∑ (2|s(n + k )|2 + σε2 ).
k=− L
For an FM signal σ2 = σε2 (2L + 1)(2A2 + σε2 ). This variance is constant.

Solution 7.24. Signal s(n) and noise ε(n) are not correlated. Then
r xx (n) = rss (n) + rεε (n) = 4(0.5)|n| + 2δ(n)

1 0.5z 3z
R xx (z) = 4 − 1
+4 +1= + 1. (7.67)
1 − 0.5z 1 − 0.5z (2z − 1)(2 − z)
a) For the optimal filtering d(n) = s(n). The cross-correlation of the desired
and input signal is
rdx (n) = E{d(k ) x (n − k )} = E{s(k )[s(k − n) + ε(k − n)]} = rss (n) = 4(0.5)|n| .
The optimal filter transfer function is
3z
R (z) (2z−1)(2−z) 3z
H (z) = dx = 3z
= .
R xx (z)
(2z−1)(2−z)
+1 −2z2 + 8z − 2
b) For optimal smoothing d(n) = s(n − 1) with
rdx (n) = E{d(k ) x (n − k )} = E{s(k − 1)[s(k − n) + ε(k − n)]} = rss (n − 1)
and
∞
−n 3z2
Rdx (z) = ∑ 4(0.5)|n−1|z = zRss (z) =
n=−∞ (2z − 1)(2 − z)
follows
3z2
H (z) = .
−2z2 + 8z − 2
c) In the case of prediction d(n) = s(n + 1) and
rdx (n) = E{d(k ) x (n − k )} = E{s(k + 1)[s(k − n) + ε(k − n)]} = rss (n + 1),
∞
−n 3
Rdx (z) = ∑ 4(0.5)|n+1|z = z−1 Rss (z) =
n=−∞ (2z − 1)(2 − z)
with
3
H (z) = .
−2z2 + 8z − 2
Solution 7.25. For the optimal filter d(n) = s(n). The correlation functions
are
r xx (n) = E{ x (k ) x ∗ (k − n)} = E{(s(k ) + ε(k ))(s∗ (k − n) + ε∗ (k − n))}

= rss (n) + 2rsε (n) + rεε (n) = 3(0.9)|n| + 8δ(n)
and
rdx (n) = E{s(k )[s∗ (k − n) + ε∗ (k − n)]} = rss (n) + rsε (n) = 3(0.9)|n| + 2δ(n).
Calculation of the z-transforms and filter transfer function is left to the

reader.
Solution 7.26. The power spectral densities of the signal and the input noise
are !
jω 1 − |ω/2| for |ω/2| < 1
Sdd (e ) =
0 elsewhere
and !
jω 1 − ||ω | − 2| for | ω − 2| < 1
Sεε (e ) = .
0 elsewhere
The optimal filter frequency response is
Sdd (e jω )
H (e jω ) = .
Sdd (e ) + Sεε (e jω )
jω
For 0 ≤ ω < π we get

⎧
⎪
⎪ 1 for ω≤1
⎨ 2− ω
jω ω for 1 < ω ≤ 2
H (e ) =
⎪
⎪ 0 for 2 < ω ≤ 3
⎩
1 for 3 < ω < π
since for 1 < ω ≤ 2 holds
1 − ω/2
H (e jω ) =
1 − ω/2 + (1 − |ω − 2|)
1 − ω/2 2−ω
= = .
1 − ω/2 + (1 + (ω − 2)) ω
The result for −π ≤ ω < 0 is symmetric. It is shown in Fig.7.18(bottom). The

input SNR is
Es 2
SNRi = = =1
Eε 2
or 0 [dB]. The output SNR is
1
&π '
jω '
'
jω '2
&2 ' '2
2π −π Sdd (e ) H ( e ) dω 3/2 + 2 1 (1 − ω2 ) ' 2−ωω ' dω
SNRo = &π ' ' = &2 ' '2
1 jω 2 ' jω '2 2 1 (1 + (ω − 2)) ' 2−ωω ' dω
2π −π Sεε ( e ) H ( e ) dω
10 − 12 ln 2
= = 18.6181
16 ln 2 − 11
or 12.7 [dB].
Solution 7.27. For this calculation the model is

N −1 D E
Vx (n, k ) =
W ∑ xQ (n + m) xQ (n − m) + e(n + m, n − m) e− j2πmk/N
m =0
N −1
= ∑ {[x(n + m) + e(n + m)] [(x(n − m) + e(n − m)] +
m =0
+ e(n + m, n − m)}e− j2πmk/N .
The mean value is

N −1
Vx (n, k )} =
E {W ∑ x (n + m) x (n − m)e− j2πmk/N
m =0
N −1
1 2
+E{ ∑ e(n + m)e(n − m)e− j2πmk/N } = Wx (n, k) + ∆ .
m =0 12
It has been assumed that the errors in two different signal samples are not
correlated E{e(n + m)e(n − m)} = 0 for m ̸= 0 and that the signal and error
are not correlated, E{ x (n + m)e(n − m)} = 0 for any m and n.
7.11 EXERCISE
Exercise 7.1. Signal x20i (n) is equal to the monthly average of maximal daily
temperatures in a city measured from year 2001 to 2015. If we can assume
that the signal for an individual month is Gaussian find the probability that
the average of maximal temperatures: (a) in July is lower than 25, (b) in
August is higher than 39.
Exercise 7.2. Random signal x (n) is such that x (n) = x1 (n) with probability
p. In all other cases x (n) is x2 (n). If the mean and variance of x1 (n) and
x2 (n) are µ x1 , σx21 and µ x2 , σx22 , respectively, find the mean and the variance
of x (n).
Result: µ x = pµ x1 + (1 − p)µ x2 and
@ A @ A
σx2 = p E{ x12 (n)} − µ2x + (1 − p) E{ x22 (n)} − µ2x
= p[σx21 + µ2x1 − µ2x ] + (1 − p)[σx22 + µ2x2 − µ2x ]
= pσx21 + (1 − p)σx22 + p(1 − p)(µ x1 − µ x2 )2 .
Exercise 7.3. Find the mean and variance of a white uniform noise whose
values are within the interval − a ≤ x (n) ≤ a. If that signal is an input to
the FIR system with impulse response h(n) = 1 for 1 ≤ n ≤ N and h(n) = 0
elsewhere, find the mean and variance of the output signal.
Exercise 7.4. Consider a signal x (n) equal to the Gaussian zero-mean noise
with variance σε2 . A new noise y(n) is formed by using the values of x (n)
lower than median value. Find the mean and variance of this new noise
y(n). Result: σy2 = 0.1426σε2 .
Exercise 7.5. A causal system is defined by
1
y ( n ) − y ( n − 1 ) = x ( n ).
2
The input signal is the causal part of a white noise ε(n)
x (n) = ε(n)u(n)
where µε = 0 and rεε (n) = σε2 δ(n). Find the mean value and the autocorrela-
tion ryy (n, m) of the output signal. What is the cross-correlation between the
input and output signal ryx (n, m). Show that for n → ∞ the output signal
tends to a WSS signal.
Exercise 7.6. (a) Calculate the DFT value X (4) for x (n) = exp( j4πn/N ) with
N = 16.
(b) Calculate the DFT of a noisy signal x (n) + ε(n), where the noise is
ε(n) = 1001δ(n) − 899δ(n − 3) + 561δ(n − 11) − 32δ(n − 14).
(c) Estimate the DFT using noisy signal x (n) + ε(n) and
M N
XR (k ) = N median Re ( x (n) + ε(n))e− j2πkn/N
n=0,1,..,N −1
M N
+ jN median Im ( x (n) + ε(n))e− j2πkn/N .
n=0,1,..,N −1
Discuss the results.
Exercise 7.7. The power spectral densities of the signal Sdd (e jω ) and input
noise Sεε (e jω ) are given in Fig.7.19 for two cases. One on the left subplots
and the other on the right subplots. Show that the frequency response of the
optimal filter H (e jω ) is presented in Fig.7.19(bottom subplot for both cases
of signal and noise). Find the SNR at the input and output of the optimal
filter in both cases.
Exercise 7.8. Find the transfer function of an optimal filter for the signal
x (n) = s(n) + ε(n), where ε(n) is a white noise with the autocorrelation
rεε (n) = Nδ(n), and s(n) is a random signal obtained as the output of the
first-order linear system to a white noise with the autocorrelation rss (n) =
a|n| , 0 < a < 1. Signal and noise are not correlated.
Exercise 7.9. A random signal s(n) carries an information. Its autocorre-

lation function is rss (n) = |1n| . A noise with the autocorrelation rεε (n) =
4
0.5δ(n) is added to the signal. Find the optimal filter for:
a) d(n) = s(n) - optimal filtering,
b) d(n) = s(n − 1) - optimal smoothing,
c) d(n) = s(n + 1) - optimal prediction.
Exercise 7.10. Find the power spectral densities of signals whose autocor-
relation functions are:
a) r xx (n) = δ(n) + 2 cos(0.πn) ,
b) r xx (n) = −4δ(n + 1) + 7δ(n) − 4δ(n − 1) .
∞
(c) r xx (n) = 2a cos(ω0 n) + ∑ σ2 (1/2)k δ(n − k )
k =0
1.5 1.5
Sdd (ejω) Sdd (ejω)
1 1
0.5 0.5
0 0
-3 -2 -1 0 1 2 3 -3 -2 -1 0 1 2 3
1.5 1.5
Sεε (ejω) Sεε (ejω)
1 1
0.5 0.5
0 0
-3 -2 -1 0 1 2 3 -3 -2 -1 0 1 2 3
1.5 1.5
jω jω
H(e ) H(e )
1 1
0.5 0.5
0 0
-3 -2 -1 0 1 2 3 -3 -2 -1 0 1 2 3
' '2
Figure 7.19 Power spectral densities of the signal 'S(e jω )' and input noise Sεε (e jω ) along
jω
with the frequency response of an optimal filter H (e ). Two cases are presented, one on the
left subplots and the other on the right subplots.
Part III
Selected Topics
421
Chapter 8
Adaptive Systems
8.1 INTRODUCTION
Classic systems for signal processing are designed to satisfy properties de-
fined in advance. Their parameters are time-invariant. Adaptive systems
change their parameters or form, in order to achieve the best possible per-
formance. These systems are characterized by ability to observe variations
in the input signal behavior and to react to these changes by adapting their
parameters in order to improve the desired performance of the output sig-
nal. Adaptive systems have the ability to "learn" so that they can appropri-
ately adapt the performance when the system environment is changed. By
definition the adaptive systems are time-variant. These systems are often
nonlinear as well. These two facts make the design and analysis of adaptive
systems more difficult than in the case of classical time-invariant systems.
Adaptive systems are the topic of this chapter.
Consider an adaptive system with one input and one output signal,
as in Figure 8.1. In addition to the algorithm that transforms the input
signal to the output signal, the adaptive system have a part that tracks
the system performance and implements appropriate system changes. This
control system takes into account the input signal, the output signal, and
some additional information that can help in making a decision on how the
system parameters should change.
Architecture of an adaptive system whose task is to transform the
input signal in such a way that it is as close to a reference (desired) signal
________________________________________
Authors: Ljubiša Stanković, Miloš Daković
423
424 Adaptive Systems
input adaptive output

signal system signal
adaptation
rule other
data
Figure 8.1 General adaptive system
input x(n) adaptive y(n) output

signal system signal
e(n) - desired
+
error signal d(n) output
Figure 8.2 Adaptive system
as possible is presented in Figure 8.2. Definition of the reference signal

will be crucial and specific for each application. It will be described later
in this chapter. Difference between the output and the reference signal is
used as a measure of the system performance quality. This difference is
called the error signal. It plays an important role in adaptation of the system
parameters.
As in the case of classic systems, the adaptive systems may be imple-
mented with a finite or infinite impulse response. They can be linear and
nonlinear. One of the crucial problems in the adaptive systems with infinite
impulse response is to provide their stability. The control part must take
into account that the system is configured in such a way that its parameters
stay all time within the region that guarantees system stability. In the case
of recursive systems this task is not simple, as it will be illustrated by the
next example.
Example 8.1. Consider an adaptive system defined by the difference relation
y ( n ) = a x ( n ) + b x ( n − 1) + c y ( n − 1) + d y ( n − 2)
where a, b, c and d are real-valued adaptive (changing) parameters of the

system. What is the range for the parameters a, b, c and d so that the system is
stable?
⋆For a given set of parameters, during the time interval when these
parameters do not change, the system can be considered as a time-invariant
system. Its transfer function is
a + bz−1
H (z) = .
1 − cz−1 − dz−2
The stability condition requires that the system poles are within the
unit circle. The pole values are
√
c± c2 + 4d
z1,2 = .
2
Consider two cases:
Case I: Poles are complex-valued, that is c2 + 4d < 0. In this case the parame-
√
ter d must be negative. In addition it has to satisfy the inequality |c| < 2 −d.
Poles of the system are
√
c −c2 − 4d
z1,2 = ±j .
2 2
√
They are within the unit circle if |z1,2 | < 1, meaning −d < 1 or −1 < d < 0.
Parameters a and b do not influence the system stability. They can take
any value. The conditions that the system is stable in this case are
d > −1
√
|c| < 2 −d.
Case II: Poles of the system are real-valued, c2 + 4d ≥ 0. The poles are
√ √
c+ c2 + 4d c− c2 + 4d
z1 = , z2 = .
2 2
The stability condition is
|z1 | < 1 and |z2 | < 1
or ,
2d + c2 ± c 4d + c2 < 2.
real
d
complex
-1
-2
-3 -2 -1 0 1 2 3
c
Figure 8.3 Region of system coefficient values where the system is stable.
In this case the system is stable if
c2 + 4d ≥ 0
,
2d + c2 + c 4d + c2 < 2
,
2d + c2 − c 4d + c2 < 2
The final solution is a set of parameters c and d that satisfies the

conditions defined within Case I or Case II. The region of parameters c and
d values, when the system is stable, is presented in Figure 8.3. Region when
the system has complex-valued poles is presented by dark-gray region, while
the lighter region is the convergence region for two real-valued poles. From
Figure 8.3 we can note that the system stability conditions reduce to
|d| < 1
|c| < 1 − d.
The stability conditions are derived assuming that the coefficients do

not change within the considered time interval. In the case when the param-
eter d assumes absolute value greater than 1 the system will remain stable.
It would be quite complex to establish the stability region if time-varying
nature of the parameters (within the considered interval) would be taken into
account. It is outside of the scope of this textbook.
Example 8.2. Consider an adaptive system defined by
y(n) = h0 (n) x (n) + h1 (n) x (n − 1) + . . . + h N −1 (n) x (n − ( N − 1))

N −1
= ∑ hi ( n ) x ( n − i )
i =0
where hi (n) for i = 0, 1, . . . , N − 1 are real-valued time-varying parameters

of the system. What are the allowed values of parameters hi (n) for a stable
system?
⋆A system is stable if for an arbitrary input signal x (n), with a
bounded amplitude | x (n)| < A x for any n, there is a constant Ay such that
the output signal is also bounded |y(n)| < Ay for any n. In this example it
means
' '
' N −1 ' N −1 N −1
' '
|y(n)| = ' ∑ hi (n) x (n − k)' ≤ ∑ |hi (n)| | x (n − i )| < A x ∑ |hi (n)|
' i =0 ' i =0 i =0
A constant Ay exist if the system coefficients are limited, |hi (n)| < Ah
for any i and n. Then
|y(n)| < Ay = N A x Ah
and the system is stable.
The proof of system stability in this case is simpler that in the case of
the recursive system from previous example. The stability condition is also
simple here. It is sufficient that the system coefficients are bounded.
8.2 LINEAR ADAPTIVE ADDER
Basic structure in an adaptive system is a linear adder (finite impulse

response system). The output signal y(n) is a linear combination of the
input signal x (n) in the considered instant n and its N − 1 previous values
x ( n − 1), x ( n − 2), . . . , x ( n − N + 1)
y ( n ) = h 0 x ( n ) + h 1 x ( n − 1 ) + · · · + h N −1 x ( n − N + 1 )
N −1
= ∑ hi x ( n − i )
i =0
Description and analysis of this system is quite simple. The system is linear.
In addition, the system with finite impulse response is always stable, for
any finite coefficient values. Finally, the realization of these systems is very
simple. In the case of adaptive systems the coefficients hi change their values
-1 -1 -1
z z z
x(n)
h (n) h (n) h (n) h (n)

0 1 2 N-1
y(n)
+ + +
Figure 8.4 Adaptive linear adder.
in time. This simple system is called linear adaptive adder. Taking into
account time-variant nature of the coefficients the system is described by
y ( n ) = h 0 ( n ) x ( n ) + h 1 ( n ) x ( n − 1 ) + · · · + h N −1 ( n ) x ( n − N + 1 )
N −1
= ∑ h i ( n ) x ( n − i ).
i =0
The adaptation process consists of an appropriately defined algorithm for

change of the coefficients hi (n) values with the aim to achieve the desired
system performance. Based on Example 8.2 we can conclude that this sys-
tem is stable if all coefficient values are bounded. Realization of the linear
adaptive adder is presented in Figure 8.4. Time-variant vectors
⎡ ⎤ ⎡ ⎤
x (n) h0 ( n )
⎢ x ( n − 1) ⎥ ⎢ h1 ( n ) ⎥
⎢ ⎥ ⎢ ⎥
X(n) = ⎢ .. ⎥ H(n) = ⎢ .. ⎥
⎣ . ⎦ ⎣ . ⎦
x ( n − N + 1) N ×1
h N −1 ( n ) N ×1
will be introduced for description and analysis of this system. Vector X(n)
commonly consists of the current value of the input x (n) and its N − 1 past
values, while elements of vector H(n) are the system coefficients hi (n) in
the current instant n. The output signal can be written as a product of these
two vectors
y(n) = X T (n)H(n) = H T (n)X(n) (8.1)
where (·)T denotes the vector transpose operation. The output y(n) is a
scalar.
In general the input vector X(n) may not be formed using the de-
layed samples of the input signal x (n). It can be understood, in gen-
eral case, as a vector whose elements are N independent input signals
h (n)
x (n) 0
0
h (n)
x (n) 1
1
+ y(n)
...
h (n)
x (n) N-1
N-1
Figure 8.5 Adaptive linear combinator.
x 0 ( n ), x 1 ( n ), . . . x N −1 ( n ),
⎡ ⎤
x0 ( n )
⎢ x1 ( n ) ⎥
⎢ ⎥
X(n) = ⎢ .. ⎥ .
⎣ . ⎦
x N −1 ( n ) N ×1
This system has N inputs and one output (multiple input single output
system - MISO system). It is called a linear adaptive combinator, Figure 8.5.
The linear adaptive adder is just a special case of linear adaptive combinator
with xi (n) = x (n − i ) for i = 0, 1, . . . , N − 1.
8.2.1 Error Signal
Block diagram of an adaptive system is presented in Figure 8.2. Input signal

is denoted by x (n), while the output signal is y(n). The reference (desired)
signal is denoted by d(n). The error signal e(n) is equal to the difference
between the reference signal d(n) and the output signal y(n),
e ( n ) = d ( n ) − y ( n ).
In adaptive systems the ultimate goal of the adaptation (learning)

process is to adjust system coefficients so that the output signal is as close as
possible to the reference signal. In an ideal case y(n) ≡ d(n) should hold,
when e(n) = 0. The adaptation process can be described as a process of
system parameters modification in order to achieve this goal.
First step in the definition of an appropriate algorithm for the system
parameters modification is in defining a measure of the output signal and
the reference signal difference (similarity). The expected value of error e(n)
is not a good choice for this measure. We will illustrate this fact on a simple
example. Assume that the following error is obtained in 6 consecutive

measurements: [0, 0, 0, 0, 0, 0] in the first realization, [−20, 20, −20, 20, −20, 20]
in the second realization, and [0.1, 0.1, 0.1, 0.1, 0.1, 0.1] in the third realization.
The average value of the error signal in the first two realizations is 0 while
in the third one is 0.1. It would lead to the conclusion that both the first
and the second realization achieved good system performance. At the same
time it would mean that the third realization produced the worst results.
Obviously this is a wrong conclusion. 1
Commonly used measure of the deviation of output signal from the
reference signal is the mean square error (MSE),
ε = E[e2 (n)],
where E[·] denotes the expected value. For the previous example with 6
values of error we get: ε = 0 for the first case, ε = 400 in the second case,
and ε = 0.01 in the third case. We see that this kind of measure meets our
expectation about the measure behavior.
In general a function J (e) is used to define the deviation of the error
signal e(n) from the ideal case. This is a cost function. It should be nonneg-
ative. It should also have a minimum where the error signal achieves its
lowest possible value (in ideal case 0), while local minima should not exist.
From the previous illustration we can conclude that one possible form of
the cost function is the mean square error function
J MSE (e) = E[e2 (n)].
In practical realizations this function can be estimated using an average of

L most recent values of the error signal
1 L −1 2
L k∑
JLS (e) = e ( n − k ).
=0
This measure corresponds to the least square (LS) criterion in the analysis.
Consider now the square error signal in the linear adaptive adder
B C2
e2 (n) = (d(n) − y(n))2 = d(n) − H T (n)X(n) =
= d2 (n) − 2d(n)H T (n)X(n) + H T (n)X(n)X T (n)H(n)
1 A simple modification of the expected value of error that would produce the correct
conclusion would be the expected absolute value of error |e(n)|. However, the absolute
value is not differentiable function (at e(n) = 0). The algorithms for its minimization would
be complex. Therefore it will not be used here (it will be the main form of minimization
function in the chapter dealing with sparse signals).
In the mean square error ε = E[e2 (n)] calculation we should take into
account that the signals d(n) and x (n) are random, while the coefficients
of the system H(n) are deterministic variables
ε = E[e2 (n)] =
= E[d2 (n) − 2d(n)H T (n)X(n) + H T (n)X(n)X T (n)H(n)] =
= E[d2 (n)] − 2H T (n)E[d(n)X(n)] + H T (n)E[X(n)X T (n)]H(n). (8.2)
The linearity property of the expected value operator E [·] is used, E[ A +

B] = E[ A] + E[ B] and E[k · A] = k · E[ A] where A and B are random variables
and k is a constant.
The mean square error (8.2) will be analyzed, with the assumption that
signals x (n) and d(n) are stationary random processes. The first term in (8.2)
E[d2 (n)] = σd2
is equal to the variance σd2 of reference signal d(n). Next term is
⎡ ⎤ ⎡ ⎤
x (n) E[d(n) x (n)]
⎢ x ( n − 1) ⎥ ⎢ E[d(n) x (n − 1)] ⎥
⎢ ⎥ ⎢ ⎥
E[d(n)X(n)] = E[d(n) ⎢ .. ⎥] = ⎢ .. ⎥.
⎣ . ⎦ ⎣ . ⎦
x ( n − N + 1) E[d(n) x (n − N + 1)]
Its elements E[d(n) x (m)] are the cross-correlations of the reference and in-
put signals. They will be denoted by rdx (n, m) = E[d(n) x (m)]. For station-
ary random signals rdx (n, m) is a function of time index difference only,
E[d(n) x (m)] = rdx (n − m). The previous relation can be rewritten in the form
⎡ ⎤ ⎡ ⎤
rdx (n, n) rdx (0)
⎢ rdx (n, n − 1) ⎥ ⎢ rdx (1) ⎥
⎢ ⎥ ⎢ ⎥
E[d(n)X(n)] = ⎢ .. ⎥=⎢ .. ⎥ = rdx . (8.3)
⎣ . ⎦ ⎣ . ⎦
rdx (n, n − N + 1) rdx ( N − 1)
The elements of cross-correlation vector rdx = E[d(n)X(n)] do not depend

on the considered instant n.
The last term in (8.2) is of the form

⎡ ⎤
x (n)
⎢ x ( n − 1) ⎥D E
⎢ ⎥
X(n)X T (n) = ⎢ .. ⎥ x (n) x ( n − 1) ... x ( n − N + 1) =
⎣ . ⎦
x ( n − N + 1)
⎡ ⎤
x (n) x (n) x ( n ) x ( n − 1) ··· x ( n ) x ( n − N + 1)
⎢ x ( n − 1) x ( n ) x ( n − 1) x ( n − 1) ··· x ( n − 1) x ( n − N + 1) ⎥
⎢ ⎥
⎢ .. .. .. .. ⎥.
⎣ . . . . ⎦
x ( n − N + 1) x ( n ) x ( n − N + 1) x ( n − 1) ··· x ( n − N + 1) x ( n − N + 1)
The expected value of this expression is
E[X(n)X T (n)] = R(n)=

⎡ ⎤
r xx (n, n) r xx (n, n − 1) ··· r xx (n, n − N + 1)
⎢ r xx (n − 1, n) r xx (n − 1, n − 1) ··· r xx (n − 1, n − N + 1) ⎥
⎢ ⎥
⎢ .. .. .. .. ⎥.
⎣ . . . . ⎦
r xx (n − N + 1, n) r xx (n − N + 1, n − 1) ··· r xx (n − N + 1, n − N + 1)
For a stationary random signal x (n)
E[X(n)X T (n)] = R = (8.4)

⎡ ⎤
r xx (0) r xx (1) ··· r xx ( N − 1)
⎢ r xx (1) r xx (0) ··· r xx ( N − 2)⎥
⎢ ⎥
⎢ .. .. .. .. ⎥
⎣ . . . . ⎦
r xx ( N − 1) r xx ( N − 2) · · · r xx (0)
In the derivation of this relation, the autocorrelation function r xx (n, m) =

E[ x (n) x (m)] is used. In the case of stationary signal r xx (n, m) = r xx (n − m).
The autocorrelation function of real-valued signals is even r xx (−n) = r xx (n).
The autocorrelation matrix R does not depend on the current instant n.
The mean square error can now be written in the form
ε = σd2 − 2H T (n)rdx + H T (n)RH(n) (8.5)
The task of an adaptive system is to find the coefficients in vector H(n) that
will produce the minimal mean square error ε. In (8.5) we have a vector
rdx of cross-correlations between the reference and input signal and the
autocorrelation matrix of the input signal R. If the statistical behavior of

these signals are known then we can find rdx and R. If that is not the case the
autocorrelation matrix elements can be estimated by averaging over time of
the input signal values
1 M −1
M k∑
r xx (i ) = E[ x (n) x (n − i )] = x (n − k ) x (n − k − i ) (8.6)
=0
where i = 0, 1, . . . , N − 1. This estimation is derived with the assumption

that the random process x (n) is ergodic and that the mean value over
time is equal to the mean value over different realizations. Increasing the
number of terms M in averaging produces a better estimation. However
the computational complexity is increased, as well the required duration
of signal is increased. Note that in (8.6) the signal samples x (n), x (n −
1), . . . , x (n − ( M − 1) − i ) are used. For 0 ≤ i ≤ N − 1 it means that the signal
has to be available from n − ( M + N − 2) to n.
Consider a simple example of a linear adaptive adder of the second
order (N = 2). Then
- . - .
x (n) h (n)
X(n) = H(n) = 0 .
x ( n − 1) h1 ( n )
The mean square error (according to (8.5)) is
- . - .- .
D E rdx (0) D E r xx (0) r xx (1) h0
ε = σd2 − 2 h0 h1 + h0 h1 =
rdx (1) r xx (1) r xx (0) h1
= σd2 − 2rdx (0)h0 − 2rdx (1)h1 + r xx (0)h20 + 2r xx (1)h0 h1 + r xx (0)h21 . (8.7)
Indices n are omitted from the coefficients h0 (n) and h1 (n) for notation
simplicity. The last relation has a simple geometric interpretation. The mean
square error ε is presented as a function of two variables h0 and h1 in Figure
8.6. From this figure (the same as from relation (8.7)) we may conclude that
the error function is a paraboloid. The paraboloid has a minimum at (h0∗ , h1∗ ).
Our aim is to find these values, meaning that the system is adjusted to the
values corresponding to the minimal measure of error.
The minimization process can be done in a numerical or analytical
way. Analytic way is based on the calculation of stationary points of a multi-
dimensional function using partial derivatives. The most common numeric
way of solving this problem is in using iterative procedures. Starting from
an arbitrary point (representing parameters of the system), the direction of
cost function decrease is determined using finite differences. System param-
eters are corrected and the process is repeated until the minimum is found
with a desired accuracy.
80 ε(h ,h )
0 1
ε(h0,h1) 6
60
4
40
2
(h*,h*)
h1 0 1
20
0
0 -2
6 (h*,h*)
4 0 1
h 2 6
1 0 2 4 -2 0 2 4 6
-2 0 h0
-2 h0
Figure 8.6 Mean square error ε as a function of the system coefficients h0 and h1 . The optimal
coefficient values are denoted by h0∗ and h1∗ .
In the considered case we have a paraboloid (convex function) with a

unique solution for the extreme value of ε. It will be determined by using the
partial derivatives of ε with respect to the variables h0 and h1 . The position
(h0∗ , h1∗ ) of minimal ε will be found by equating these derivatives to zero.
The derivatives are
∂ε
= −2rdx (0) + 2r xx (0)h0 + 2r xx (1)h1
∂h0
∂ε
= −2rdx (1) + 2r xx (1)h0 + 2r xx (0)h1 .
∂h1
From ∂ε/∂h0 = 0 and ∂ε/∂h1 = 0 follows
r xx (0)h0 + r xx (1)h1 = rdx (0)

r xx (1)h0 + r xx (0)h1 = rdx (1),
or in matrix form
RH = rdx .
The solution of this matrix equation
H∗ = R−1 rdx .
produces the optimal system coefficients, denoted by h0∗ and h1∗ . This is the
Wienner optimal filter.
In order to simplify notation, a symbolic „differentiation with respect to

a vector” is introduced as
∂ε
= 0.
∂H T
Partial derivatives of ε with respect to each element of vector H are denoted
by ∂ε/∂H T . Each derivative is independently equated to zero. When the
vector H has N elements then ∂ε/∂H T is a system of N equations ∂ε/∂hi ,
i = 0, 1, 2, ..., M − 1. Since ε is a function of N variables h0 , h1 , . . . , h N −1 it
means that ∂ε/∂H T is a gradient of ε, ∇ε = gradient (ε) = ∂ε/∂H T .
Using this simplified symbolic notation we can analyze an Nth order
system. By differentiating (8.5) with respect to the system coefficients we
get
∂ε
= −2rdx + 2RH
∂H T
or
H∗ = R−1 rdx . (8.8)
This kind of calculation of the optimal system coefficients requires
statistical parameters of the input signal (in R) and its cross-correlation with
the reference signal (in rdx ). In addition, it requires an inversion of a matrix
of order N which is numerically demanding
P Q operation (inversion of a matrix
of order N requires an order of O N 3 operations).
Minimal value of the mean square error is
ε min = σd2 − 2H∗T rdx + H∗T RH∗

= σd2 − 2(H∗ )T rdx + (H∗ )T rdx = σd2 − (H∗ )T rdx . (8.9)
Example 8.3. Find the stationary point of the error function
ε(h0 , h1 ) = 2 + 3h0 + 2h1 + 5h0 h1 + 3h20 + 4h21
and prove that it is a minimum of this function.

⋆The partial derivatives of ε(h0 , h1 ) with respect to h0 and h1 are
∂ε
= 6h0 + 5h1 + 3
∂h0
∂ε
= 8h1 + 5h0 + 2.
∂h1
They are equal to zero for
6h0 + 5h1 = −3
8h1 + 5h0 = −2.
This system produces the solution
14 3
h0∗ = − and h1∗ = .
23 23
It is a stationary point of ε(h0 , h1 ). The stationary point can be minimum,
maximum or neither of this two (just a saddle point). To check what kind of
stationary point is the previous solution (h0∗ , h1∗ ) we have to find the second
order partial derivatives of ε(h0 , h1 ). They are
∂2 ε ∂2 ε ∂2 ε
=6 =5 =8
∂h20 ∂h0 ∂h1 ∂h21
The stationary point is a minimum of the function if for h0 = h0∗ and h1 = h1∗
holds
* 2 +2
∂2 ε ∂2 ε ∂2 ε ∂ ε
> 0 and > .
∂h20 ∂h20 ∂h21 ∂h0 ∂h1
In the considered case these inequalities hold (6 > 0 and 6 · 8 > 52 ). Therefore,
the function ε(h0 , h1 ) has a minimum at h0∗ = − 14 ∗ 3
23 and h1 = 23 . The minimum
value is
28
ε(h0∗ , h1∗ ) = .
23
Example 8.4. The input signal x (n) is a zero-mean white noise with variance 1.
The reference signal is d(n) = 12 x (n − 2). Find the optimal coefficients of the
fourth order system.
⋆The optimal coefficients are the solution of
H∗ = R−1 rdx ,
where R is the autocorrelation function of the input signal, equal to

⎡ ⎤ ⎡ ⎤
r xx (0) r xx (1) r xx (2) r xx (3) 1 0 0 0
⎢r xx (1) r xx (0) r xx (1) r xx (2)⎥ ⎢0 1 0 0⎥
R =⎢
⎣r xx (2)
⎥=⎢ ⎥
r xx (1) r xx (0) r xx (1)⎦ ⎣0 0 1 0⎦
r xx (3) r xx (2) r xx (1) r xx (0) 0 0 0 1
since r xx (n) = δ(n). Cross-correlation function rdx (i ) is defined by
1 1 1
rdx (i ) = E[d(n) x (n − i )] = E[ x (n − 2) x (n − i )] = r xx (i − 2) = δ(i − 2).
2 2 2
Therefore D ET
rdx = 0 1 .
0 2 0
The optimal coefficients are

⎡ ⎤ −1 ⎡ ⎤ ⎡ ⎤
1 0 0 0 0 0
⎢
∗ ⎢0 1 0 0⎥ ⎢0⎥ ⎢0⎥
H =⎣ ⎥ · ⎢ ⎥ = ⎢ ⎥.
0 0 1 0⎦ ⎣1⎦ ⎣1⎦
2 2
0 0 0 1 0 0
Indeed, if we set these optimal coefficients we get the output signal in

the form
1 1
y ( n ) = 0 · x ( n ) + 0 · x ( n − 1) + x ( n − 2) + 0 · x ( n − 3) = x ( n − 2) = d ( n ).
2 2
The output signal is equal to the reference signal, with zero error.
Example 8.5. Signal x (n) is observed. The autocorrelation function values r xx (0) =
1, r xx (1) = 0.8, r xx (2) = 0.4 and r xx (3) = 0.1 are obtained by averaging. Find
the parameters of the optimal system that will predict values of x (n) one-step
ahead. The reference signal is d(n) = x (n + 1). Find the first and second order
system, with N = 1 and N = 2. In both cases calculate the value of minimal
error ε min .
⋆For the first-order system N = 1 we have H = [ h0 ], R = [r xx (0)] = 1
and rdx = [ E[d(n) x (n)]] = [ E[ x (n + 1) x (n)]] = [r xx (1)] = 0.8. The optimal
value of system coefficient h0 is
H∗ = h0∗ = R−1 rdx = 1 · 0.8 = 0.8
For the minimal error value ε min calculation we need the value of
σd2 = E[d(n)d(n)] = E[ x (n + 1) x (n + 1)] = r xx (0) = 1.
Based on (8.9) we get
ε min = σd2 − (H∗ ) T rdx = 1 − 0.8 · 0.8 = 0.36.
This is the mean square error in one-step prediction using a first-order

system.
For the second order system (N = 2) we get
- . - .
r xx (0) r xx (1) 1 0.8
R= =
r xx (1) r xx (0) 0.8 1
- . - .
E[d(n) x (n)] E[ x (n + 1) x (n)]
rdx = =
E[d(n) x (n − 1)] E[ x (n + 1) x (n − 1)]
- . - .
r (1) 0.8
= xx = .
r xx (2) 0.4
Optimal values of the system coefficients are

- ∗. - . −1 - . - 4 .
∗ h0 −1 1 0.8 0.8
H = ∗ = R rdx = = 32 .
h1 0.8 1 0.4 −3
The minimal value of ε is

- .
∗ T D4 E 0.8
ε min = σd2 − (H ) rdx = 1 − 3 − 2
3 = 0.2.
0.4
It is significantly lower than in the first-order system.

Note that the optimal first-order system is described by the relation
y(n) = 0.8x (n),
while the second order system has the form
4 2
y(n) = x ( n ) − x ( n − 1).
3 3
Note that by using data given in the example it was possible to calcu-
late the coefficients of a third-order ( N = 3) one-step ahead prediction system
as well.
8.2.2 Autocorrelation Matrix Eigenvalues and Eigenvectors
Consider a square matrix R of order N. A nontrivial vector q is an eigenvec-

tor of matrix R if there exists a scalar λ such that
Rq = λq. (8.10)
The scalar λ is an eigenvalue of R.

For a zero vector q = 0 relation (8.10) is met for any λ. This trivial
solution is not of our interest. Note that if q1 is an eigenvector of matrix R
then the vector q2 = αq1 is also an eigenvector of the considered matrix R
(with the same eigenvalue λ) for an arbitrary scalar α. Since the definition
of eigenvector has this kind of ambiguity, it is commonly assumed that the
eigenvector is normalized (having unit intensity) and defined as
q
q0 =
||q||
where ||q|| is the intensity of q defined as

F
||q|| = q20 + q21 + · · · + q2N −1 .
Calculation of the eigenvalues and eigenvectors is based on their

definition (8.10),
Rq = λq
Rq − λq= 0
(R − λI)q= 0 (8.11)
where I is an identity matrix of order N (diagonal matrix with 1 on diago-

nal). In this way we have obtained a system of N homogeneous equations
with N unknown elements (coordinates) of vector q. Since we are inter-
ested in nontrivial solutions only, then the determinant of (R − λI) should
be equal to zero,
det(R − λI) = 0.
This determinant is an Nth order polynomial with variable λ. This polyno-

mial is called characteristic polynomial of matrix R. Roots of this polynomial
are the eigenvalues of matrix R. It is known that the Nth order polynomial
has N roots. An eigenvector is the associated to each eigenvalue.
Denote by λi roots of the characteristic polynomial and assume that
all roots are of the first-order. Then the rank of matrix R − λI is N − 1. This
implicates that the space of solutions is of dimension one (space of vectors q
satisfying (8.11)). It means that only one eigenvector q0i corresponds to the
eigenvalue λi . In the case that λi is a root of kth order then the rank of matrix
R − λI is for k lower than the order of matrix. The space of solutions is a k-
dimensional space defined by k linear independent vectors q1i , q2i , . . . , qki .
Then there are k independent vectors corresponding to one λi . Note that
vectors q1i , q2i , . . . , qki are not unique here even with an additional condition
that they are orthonormal.
The eigenvalues of the autocorrelation matrix are real-valued and
positive. This follows from the symmetry property of the autocorrelation
matrix. Function r xx (m) is symmetric with maximum at m = 0. For positive
m holds r xx (m) ≥ r xx (m + 1). It means that the eigenvalues of R are positive.
Example 8.6. Consider matrix R defined by
- .
1 0.9
R= .
0.9 1
Find its eigenvalues and eigenvectors.

⋆The characteristic polynomial of matrix R is

- . - .
1 0.9 1 0
det(R − λI) = det( −λ )=
0.9 1 0 1
- .
1−λ 0.9
= det( )=
0.9 1−λ
= (1 − λ)2 − 0.81 = λ2 − 2λ + 0.19.
The roots of this polynomial are the eigenvalues λ0 = 1.9 and λ1 = 0.1. It is
common to index the eigenvalues starting from the highest absolute value
into descending order.
Next we will find the eigenvectors. The eigenvector q0 corresponding to
λ0 = 1.9 follows from (8.11)
( R − λ0 I ) q0 = 0
- .- .
1 − λ0 0.9 q00
=0
0.9 1 − λ0 q01
- .- .
−0.9 0.9 q00
= 0.
0.9 −0.9 q01
We have q00 = q01 = α where α is an arbitrary scalar. The eigenvector q0 is
- . - .
q α
q0 = 00 = .
q01 α
F √
Scalar α is determined so that the intensity of q0 is 1, q200 + q201 = 2α2 , as
√
α = 1/ 2. The final form of q0 is
> 1 ?
√
q0 = 2
√1
2
The eigenvector corresponding to λ = λ1 is obtained in the same way from

(8.11)
( R − λ1 I ) q1 = 0
- .- .
1 − λ1 0.9 q10
=0
0.9 1 − λ1 q11
- .- .
0.9 0.9 q10
= 0.
0.9 0.9 q11
√
It follows that q10 = −q11 = α with α = 1/ 2. Vector q1 assumes the form
> 1 ?
√
q1 = 2
− √1
2
In the previous example we have calculated eigenvectors q0 , q1 and

eigenvalues λ0 , λ1 from the equations
Rq0 = λ0 q0
Rq1 = λ1 q1 .
These two equations can be written in one matrix equation as

- .
D E D E λ0 0
R q0 q1 = q0 q1 .
0 λ1
Introducing the matrix notation for eigenvectors and eigenvalues as

- .
D E q q10
Q = q0 q1 = 00
q01 q11
- .
λ 0
Λ= 0
0 λ1
we may write
RQ = QΛ
or
R = QΛQ−1
Λ=Q−1 RQ.
Matrix Q contains the eigenvectors as its columns. This matrix is orthonor-

mal matrix, Q−1 = Q T . Then we can write
R = QΛQ T
Λ=Q T RQ
The same matrix relations can be written for any order N of autocorrelation
matrix R.
Example 8.7. For the autocorrelation matrix R defined by
⎡ ⎤
3 1 1
R = ⎣1 3 1⎦
1 1 3
find the eigenvalues and eigenvectors and write matrices Q and Λ.

⋆The characteristic polynomial of matrix R is

⎡ ⎤
3−λ 1 1
det(R − λI) = det(⎣ 1 3−λ 1 ⎦) =
1 1 3−λ
= −λ3 + 9λ2 − 24λ + 20.
Roots of this polynomial are λ0 = 5 and λ1,2 = 2. The root λ = 2 is of order 2.

The eigenvector q0 corresponding to λ0 = 5 follows from (8.11) as
( R − λ0 I ) q0 = 0
⎡ ⎤⎡ ⎤
−2 1 1 q00
⎣ 1 −2 1 ⎦ ⎣q01 ⎦ = 0.
1 1 −2 q02
Since the rank of the system matrix is 2 the system does not have a unique
solution. One equation is omitted. Solving two remaining equations for two
unknowns we get q00 = q01 = q02 = α, where α is an arbitrary scalar. The
solution is ⎡ ⎤ ⎡ ⎤
q00 α
q0 = ⎣q01 ⎦ = ⎣α⎦ .
q02 α
Value of scalar α is√
found in such a way to normalize the intensity of q0 . It
follows that α = 1/ 3, or ⎡ ⎤
√1
⎢ 13 ⎥
q0 = ⎢ √ ⎥
⎣ 3⎦.
√1
3
For the second and third eigenvector we use λ = 2
(R − 2I)q2 =0
⎡ ⎤⎡ ⎤
1 1 1 q10
⎣1 1 1⎦ ⎣q11 ⎦ = 0.
1 1 1 q12
The system reduces to one equation only
q10 + q11 + q12 = 0,
with the solution q12 = −q10 − q11 . Therefore we may take two variables
q10 = α and q11 = β as arbitrary. The solution is
⎡ ⎤
α
q = ⎣ β ⎦.
−α − β
We are interested in orthogonal vectors. For the second vector we will use
α = β. After normalization we get
⎡ ⎤
√1
⎢ 16 ⎥
q1 = ⎢ √ ⎥
⎣ 6 ⎦
− √2
6
The third vector should be orthogonal to q1 , meaning that their scalar prod-
uct is zero,
1 1 2
⟨q, q1 ⟩ = √ α + √ β − √ (−α − β) = 0
6 6 6
−α − β = 0
α = − β.
With α = − β after normalization, the third eigenvector is obtained in the form
⎡ 1 ⎤
−√
⎢ √1 2 ⎥
q2 = ⎣ ⎦.
2
0
Matrices Q and Λ contain the eigenvectors and eigenvalues, respectively,
⎡ ⎤
√1 √1 − √1 ⎡ ⎤
⎢ 1 3 6 2 ⎥ 5 0 0
1 1
Q =⎢ √
⎣ 3
√
6
√ ⎥
2 ⎦
Λ = ⎣0 2 0⎦
√1 − √ 2
0 0 0 2
3 6
It is easy to check that QT Q = I and R = QΛQ T .
8.2.3 Error Signal Analysis
The mean square error of a linear adaptive adder has been defined by (8.5)
as
ε = σd2 − 2H T (n)rdx + H T (n)RH(n). (8.12)
Its minimization produced the optimal coefficients
H∗ = R−1 rdx .
The minimal value of the mean square error is obtained for H(n) = H∗ in
(8.12) as
ε min = σd2 − 2(H∗ )T rdx + (H∗ )T RH∗ =

= σd2 − 2(H∗ )T rdx + (H∗ )T rdx =
= σd2 − (H∗ )T rdx = σd2 − (H∗ )T RH∗ (8.13)
The error (8.12) can be expressed in terms of ε min , the autocorrelation matrix
R, and optimal coefficients H∗ . The value of σd2 is calculated using (8.13) and
replaced in (8.12),
ε = ε min + (H∗ )T RH∗ − 2H T rdx + H T RH.
Time index n is omitted to simplify the notation. Since the cross-correlation

vector rdx is equal to rdx = RH∗ , it follows
ε = ε min + (H∗ ) T RH∗ − 2H T RH∗ + H T RH

= ε min + (H∗ )T RH∗ − H T RH∗ − H T RH∗ + H T RH
B C
= ε min + (H∗ )T − H T RH∗ − H T R (H∗ − H)
= ε min + (H∗ − H)T RH∗ − H T R (H∗ − H) .
Note that (H∗ − H) T RH∗ and H T R (H∗ − H) are scalars, when

B CT
(H∗ − H)T RH∗ = (H∗ − H)T RH∗
= (H∗ ) T R T (H∗ − H) = (H∗ ) T R (H∗ − H) ,
with R T = R. The mean square error can be written as
ε = ε min + (H∗ ) T R (H∗ − H) − H T R (H∗ − H)

B C
= ε min + (H∗ )T − H T R (H∗ − H)
= ε min + (H − H∗ )T R(H − H∗ ).
Using the autocorrelation matrix R expressed in terms of its eigenvalues

and eigenvectors R = QΛQ T we get
ε = ε min + (H − H∗ )T QΛQ T (H − H∗ ).
Let us introduce new coefficients V defined by
V = Q T ( H − H ∗ ).
In the new coordinate system whose coordinates are elements of vector V

the minimum of mean square error is achieved for V = 0 (since V = 0 for
H = H∗ ). The mean square error can be written as
ε = ε min + V T ΛV.
In order to illustrate this relation, consider a two-dimensional case

(N = 2), when the mean square error is
- .- .
D E λ0 0 v0
ε = ε min + v0 v1
0 λ1 v1
ε = ε min + v20 λ0 + v21 λ1 .
This equation can be rewritten in the form
v02 v21
ε−ε min + ε−ε min =1
λ0 λ1
representing (for a constant ε) an ellipse in the coordinate system v0 , v1 . The

center of ellipse is at (0, 0). The main axes of the ellipse coincide with the
coordinate axes. The eigenvalues define the ellipse semi-major and semi-
minor axis. This conclusion is in agreement with the previous findings that
the mean square error function in two-dimensional case is a paraboloid in
the coordinate system h0 , h1 . Graphical representation of the mean square
error in v0 , v1 and h0 , h1 systems is presented in Fig. 8.7.
8.2.4 Orthogonality Principle
Two random signals x1 (n) and x2 (n) are orthogonal if
E[ x1 (n) x2 (n)] = 0
holds. When the system coefficients assume their optimal values H∗ =

R−1 rdx the error signal e(n) is orthogonal to the output signal y(n) and
to all components of the input signal vector X(n). In order to prove this
property consider the orthogonality of e(n) to X(n) for H∗ = R−1 rdx with
y(n) = X T (n)H∗ ,
E[e(n)X(n)] = E[(d(n) − y(n))X(n)] = E[d(n)X(n)] − E[X(n)y(n)] =

= rdx − E[X(n)X T (n)H∗ ] = rdx −E[X(n)X T (n)]R−1 rdx =
= rdx −RR−1 rdx = rdx − rdx = 0
Definitions E[X(n)X T (n)] = R and E[d(n)X(n)] = rdx are used here.

h
1
v1
*
h1
h* h0
0
v0
Figure 8.7 Coordinate system change by translation origin to the optimal point and by using
coordinate axes defined by eigenvectors of the autocorrelation matrix
Orthogonality of e(n) to y(n) directly follows from the orthogonality

of the error signal to the input signal, since
E[e(n)y(n)] = E[(H∗ )T X(n)e(n)] = ( H∗ )T E[e(n)X(n)] = 0.
8.3 STEEPEST DESCEND METHOD
Consider the optimal filter whose coefficients are obtained by minimizing

the mean square error
ε = E[e2 (n)].
This minimization resulted in the matrix equation (system of linear equa-
tions)
RH = rdx
Its solution produces the optimal values of the coefficients of the adaptive
system. In order to avoid inversion of matrix R the solution of this system
of linear equations may be obtained using an iterative method. One of the
iterative methods is the steepest descend method. It will be presented next.
This method consists of an arbitrary initial value H0 of system coef-
ficients vector in the first step (usually H0 = 0). The coefficients are then
modified in an iterative way toward the minimum of mean square error ε.
Direction of the steepest accent of function ε is defined by the gradient of

this function ∇ε. The elements of the gradient vector ∇ε are the derivatives
of ε with respect to the variables (coefficients) Hi , ∂ε/∂Hi . Vector form of
these derivatives is ∇ε = ∂ε/∂H. The steepest descend is in the direction
opposite to the gradient. It is −∇ε. The first iteration step is
'
µ ∂ε ''
H1 = H0 + (− )'
2 ∂H H=H0
where µ/2 defines the step in the steepest descend direction. In general the
iterations are defined by
'
µ ∂ε ''
H n +1 = H n + (− )'
2 ∂H H=Hn
for n = 0, 1, . . . . A common stopping criterion for the iterative procedure is

defined by using the difference of two consecutive iteration Hn+1 − Hn . If
the coefficients in two iterations are sufficiently close, the iteration process
is stopped.
The gradient vector can be written as
∂ε
= −2rdx + 2RH.
∂H
The iterative relation is then
Hn+1 = Hn + µ(rdx − RHn ). (8.14)
The iterative algorithm step µ defines the value of coefficients change

in the direction of the steepest descend. Its choice is crucial for the itera-
tive algorithm performance. Too small values of step µ will guarantee the
convergence of the iterative algorithm, but at the expense of a very large
number of iterations. Larger values of µ will reduce the number of iterations
to reach the optimal solution. Too large values of µ could lead to the iterative
algorithm divergence.
The steepest descend method is illustrated on a second order adaptive
system in Figure 8.8. Contour lines represent the mean square error value in
the space of system coefficients h0 , h1 . The optimal coefficient values h0∗ , h1∗
should be obtained as a result of the iterative algorithm. Optimization using
the steepest descend algorithm, with the different steps µ, is considered. The
smallest step µ is used in Figure 8.8(a). The initial value of the system coef-
ficients is denoted by 0 position in the coordinate system h0 , h1 . Positions of
h h
1 1
* *
h1 h1
3 1
2
2
* 1 h0 * h0
h0 h0
0
0
(a) (b)
h
1
3
1
*
h1
* h
2 h0 0
(c)
Figure 8.8 Steepest descent method illustration. Smallest step µ is used in case (a), larger step
is presented in case (b), and the largest µ corresponds to case (c). The steepest descend method
converges in cases (a) and (b), while it diverges in case (c). Contour plot of error function is
presented in all cases. Iterations are marked with dots and numbers 0, 1, 2, . . . , where 0 is the
starting iteration.
the coefficient values in the next iterations are denoted by 1, 2, 3, . . .We can
see that the iterative procedure converges toward the optimal coefficient
values h0∗ , h1∗ . A larger step µ is used in the case presented in Figure 8.8(b).
The iterative algorithm convergence is faster than in the previous case. In
the third case, presented in Figure 8.8(c), a very large step µ is used. The step
is here too large and the iterative algorithm does not converge to the optimal
coefficient values. Note that the convergence in all of these cases does not
depend on the initial position (initial value of the system coefficients).
The range of step µ values when the steepest descend iterative algo-
rithm converges can be determined in an analytical way. The optimal coeffi-
cient values are obtained as a result of the equation RH∗ = rdx . Consider the
deviation of the system coefficients vector Hn+1 in (n + 1)th iteration from
the optimal value H∗ . The deviation is
Hn+1 − H∗ = Hn + µ(RH∗ − RHn ) − H∗

= (I − µR)(Hn − H∗ ).
Relation (8.14) is used. The deviation (Hn+1 − H∗ ) is equal to the deviation
value in the previous iteration (Hn − H∗ ) multiplied by the matrix (I − µR).
It is easy to relate now the deviation in the nth iteration with the initial
deviation
Hn − H∗ = (I − µR)n (H0 − H∗ ).
The autocorrelation matrix R can be written as R = QΛQ T , where Q is the
matrix of eigenvectors (which are orthonormal, Q−1 = Q T and Q T Q = I)
and Λ is the eigenvalue matrix. The deviation in the nth iteration assumes
the form
Hn − H∗ = Q(I − µΛ)n Q T (H0 − H∗ ). (8.15)
Matrix (I − µΛ)n is a diagonal matrix with elements (1 − µλk )n , k =
0, 1, . . . , N − 1, where N is the order of matrix R, i.e., the order of the system.
All matrices on the right side of equation (8.15) do not depend on the itera-
tion index n. The deviation Hn − H∗ will tend to zero for n → ∞ (meaning
that the iterative algorithm will converge toward H∗ ) if the absolute values
of all elements on the diagonal of matrix (I − µΛ) are smaller than 1. In that
case the matrix (I − µΛ)n tends to zero valued matrix when the number of
iterations increase, for n → ∞. Therefore, the convergence condition is that
the inequalities
|1 − µλk | < 1
hold for all k = 0, 1, . . . , N − 1. These inequalities reduce to −1 < 1 − µλk < 1
or 0 < µ < 2/λk for all k. Finally, the steepest descend method converges if
the step µ satisfies the condition
2
µ< (8.16)
λmax
where λmax is the largest eigenvalue of the autocorrelation matrix R.
Calculation of eigenvalues requires a lengthy numerical procedure.
Note that a rough estimate of the maximal step µ can be obtained using
a simple relation
2
µ<
Tr[R]
since λmax < Tr[R]. Here Tr[R] denotes trace of matrix R. It is equal to the
sum of the matrix R elements
@ on
A the diagonal. It can be easily calculated as
Tr[R] = Nr xx (0) = NE | x (n)|2 = Ex , where Ex is the energy of input signal.
Condition (8.16) guarantees that the coefficients deviation will tend

to zero. However, it does not tell anything about the rate of convergence.
Let the eigenvalues of the autocorrelation matrix R be indexed into a
nonincreasing order λ0 ≥ λ1 ≥ . . . ≥ λ N −1 and µ < 2/λ0 . The total square
deviation of all coefficients can be calculated using (8.15). Its value is
EH = ∥Hn − H∗ ∥22 = (Hn − H∗ )T (Hn − H∗ )

B CB C
= (H0 − H∗ )T Q(I − µΛ)n Q T Q(I − µΛ)n Q T (H0 − H∗ )
= (H0 − H∗ )T Q(I − µΛ)2n Q T (H0 − H∗ ).
We can conclude that the total square deviation is a linear combination of
P Qn
the terms (1 − µλi )2 , i.e.,
B Cn B Cn B Cn
EH = A0 (1 − µλ0 )2 + A1 (1 − µλ1 )2 + . . . + A N −1 (1 − µλ N −1 )2
where the coefficients A0 , A1 , . . . , A N −1 are independent from the iteration

index n. Values of (1 − µλi )2 are within the interval from 0 to 1 (since we
assume that the convergence condition is meet). The terms with smaller
values of (1 − µλi )2 converge faster as the iteration index n increases. Now
we will find a value of step µ that will take into account convergence of
all coefficients. For small values of µ the term with the smallest eigenvalue
P Qn
(1 − µλ N −1 )2 is dominant. For step µ close to its upper bound value the
P Qn
term (1 − µλ0 )2 is dominant. The best choice for µ will be its value when
these two terms are equal,
(1 − µλ0 )2 = (1 − µλ N −1 )2
µ(λ20 − λ2N −1 ) = 2(λ0 − λ N −1 )
2
µ= .
λ 0 + λ N −1
In this case, for k = 0, 1, . . . , N − 1 holds
(1 − µλ0 )2 ≥ (1 − µλk )2
(1 − µλ0 )2 − (1 − µλk )2 ≥ 0
µ(λ0 − λk )(2 − µ(λ0 + λk )) ≥ 0
* +
λ0 + λ k
2µ(λ0 − λk ) −1 ≥0
λ 0 + λ N −1
* +
λ k − λ N −1
2µ(λ0 − λk ) ≥ 0,
λ 0 + λ N −1
since µ(λ0 − λk ) ≥ 0 having in mind that λ0 is the largest eigenvalue

and that λk ≥ λ N −1 , where λ N −1 is the smallest eigenvalue. This relation
reaffirms the approach to consider only the behavior of the first and the
last term in EH . The optimal value of step µ, from the point of view of the
coefficients deviation, is
2
µopt =
λmax + λmin
where λmax = λ0 is the largest and λmin = λ N −1 is the smallest eigenvalue

of the input autocorrelation matrix.
In a special case, if all eigenvalues are equal, λ0 = λ1 = . . . = λ N −1 = λ,
the relation for optimal step µ produces
1
µopt = .
λ
The square deviation of the coefficients (for n > 0) is
EH = A0 0n + A1 0n + . . . + A N −1 0n = 0.
In means that the steepest descend method, in this special case, will reach
the optimal system coefficients H∗ in one iteration step.
Example 8.8. Consider autocorrelation of an input signal, as in 8.6

- .
1 0.9
R= .
0.9 1
Assume that the cross-correlation vector of the input and reference signal is
- .
3.8
rdx = .
1.9
Find the optimal system coefficients using the inverse of autocorrelation

matrix. Find adaptive system coefficients using the steepest descend method.
Use 3 different step values: µ1 = 0.5, µ2 = 1, and µ3 = 1.5. In all cases find the
number of iterations to achieve the square deviation value of the coefficients
∥Hn+1 − H∗ ∥2 lower than 0.01. The initial value of H0 is the zero-vector.
⋆The optimal coefficients are
- . −1 - . - .
1 0.9 3.8 11
H∗ = R−1 rdx = = .
0.9 1 1.9 −8
Using the steepest descend method, starting from H0 = 0, we get

- . - . - .
0 3.8 1.9
H1 = H0 + µ1 (rdx − RH0 ) = + 0.5 =
0 1.9 0.95
- . - . - .
1.9 1.045 2.4225
H2 = H1 + µ1 (rdx − RH1 ) = + 0.5 =
0.95 −0.76 0.47
- . - .
2.855 10.993
H3 = , ..., H141 =
0.145 −7.993
After 141 iterations the norm of the coefficients deviation is below 0.01.
Using a larger step, µ2 = 1, we get
- . - . - . - .
3.8 2.09 5.168 10.994
H1 = H2 = H3 = . . . H69 =
1.9 −1.52 0.019 −7.992
The required precision is achieved in 69 iterations.

For the step value µ3 = 1.5 it follows
- . - . - . - .
5.7 −1 39.3 −695
H1 = H2 = . . . H5 = . . . H10 = ...
2.85 −6.27 28.7 −710
The iterative algorithm in this case diverges.

Note that the eigenvalues of the autocorrelation matrix R are λ0 = 1.9
and λ1 = 0.1. The bound for the step is
2 2
µ< = ≈ 1.0526.
λmax 1.9
This condition is satisfied for the first two cases. In the third case the conver-
gence condition is not met. The optimal value of the step is
2
µopt = = 1.
λmin + λmax
This is the reason why the second case produced required precision in a lower
number of iterations than the first case.
Example 8.9. Consider an adaptive system of the second order, described by the
difference equation
y ( n ) = h0 ( n ) x ( n ) + h1 ( n ) x ( n − 1)
where h0 (n) and h1 (n) are real-valued varying system parameters. The in-
put signal x (n) is stationary with the autocorrelation function r xx (m) =
5δ(m) + 3δ(m2 − 1). The reference signal is d(n) with the cross-correlation
between the input and reference signal rdx (m) = δ(m) + 12 δ(m − 1). System
is adapted by using the steepest descend method with step µ. The initial
conditions for the system coefficients are h0 (0) = 0 and h1 (0) = 0. Find the
optimal system coefficients in the sense of minimal mean square error, where
the error is e(n) = d(n) − y(n). Find the coefficient values as a function of
the iteration (time) index n. Find the range for the step µ when the coeffi-
cients converge toward the optimal values. For the cases when the system
coefficients converge find the number of iterations when the mean square
deviation of the coefficients from the optimal values will be lower than 10 −6 .
⋆System is of the second order. Its autocorrelation matrix and cross-
correlation vector are
- . - .
r (0) r xx (1) 5 3
R= xx =
r xx (1) r xx (0) 3 5
- . - .
r (0) 1
rdx = dx = 1 .
rdx (1) 2
The inverse of R is - .
1 5 −3
R −1 = ,
16 −3 5
with the optimal coefficients of the system
H∗ = R−1 rdx
- ∗. - . - . - 7 .
h0 1 5 −3 1
∗ = · 1 = 321 .
h1 16 −3 5 2 − 32
In order to get the coefficients h0 (n) and h1 (n) as a function of the
iteration (time) index n we will use the iteration relation for the steepest
descend method
Hn+1 = Hn + µ(rdx − RHn )
with the initial condition H0 = [0 0] T , or
- . - . *- . - .- .+
h0 ( n + 1) h (n) 1 5 3 h0 ( n )
= 0 +µ 1 − 3 .
h1 ( n + 1) h1 ( n ) 2 5 h1 ( n )
The system of equations is
h0 (n + 1) = h0 (n) + µ (1 − 5h0 (n) − 3h1 (n))
h1 (n + 1) = h1 (n) + µ (1 − 3h0 (n) − 5h1 (n)) .
Expressing h1 (n) from the first equation
1 − 5µ 1 1
h1 ( n ) = h0 ( n ) − h ( n + 1) +
3µ 3µ 0 3
and replacing it into the second equation, to get
1 − 5µ 1 1
h0 ( n + 1) − h ( n + 2) + =
3µ 3µ 0 3
* +
1 − 5µ 1 1
= (1 − 5µ) h0 ( n ) − h ( n + 1) + + µ − 3µh0 (n).
3µ 3µ 0 3
This is a recursive (difference) relation for h0 (n). It can be written as
h0 (n + 2) − 2(1 − 5µ)h0 (n + 1) + (1 − 8µ)(1 − 2µ)h0 (n) = µ(1 − 3µ)
with initial conditions h0 (0) = 0, h0 (1) = h0 (1) + µ (1 − 5h0 (1) − 3h1 (1)) = µ.
The solution of this equation is
3 1 7
h0 ( n ) = − (1 − 8µ)n − (1 − 2µ)n + .
32 8 32
From the relationship between h1 (n) and h0 (n) follows
3 1 1
h1 ( n ) = − (1 − 8µ)n + (1 − 2µ)n − .
32 8 32
Consider limit values
lim h0 (n) and lim h1 (n).

n→∞ n→∞
They are finite if
|1 − 8µ| < 1 and |1 − 2µ| < 1.
Using positive value of the step µ we get µ < 1/4. For this value of step, the
limit values are equal to the optimal system coefficient values. For µ > 1/4
the coefficients tend to infinity. In the limit case µ = 1/4, for a large n (so that
the term with (1 − 2µ)n can be neglected) the coefficients are approximately
equal to
7 3
h0 ( n ) = − (−1)n
32 32
1 3
h1 (n) = − − (−1)n .
32 32
They assume oscillatory form, with oscillations around the optimal values of
the system coefficients.
The number of iterations needed to get the mean square deviation of
the coefficients bellow 10−6 follows from
(h0 (n) − h0∗ )2 + (h1 (n) − h1∗ )2
< 10−6
2
or
9 1
(1 − 8µ)2n + (1 − 2µ)2n < 10−6 .
1024 64
This inequality does not have a closed form solution. For a given step µ
(0 < µ < 1/4) the minimal number of iterations n can be found in a numerical
way. Solutions for some possible values of step µ are given as
µ 0.01 0.1 0.15 0.18 0.19 0.2 0.21 0.22 0.24 0.248
n 239 22 14 11 11 10 12 17 55 282
From this table we can conclude that small values of step µ should not be
used since the convergence is very slow. Based on the values from the table
we can conclude that the optimal step is around µ = 0.2. Next we will find
this value based on the analytical consideration of the coefficients. Assume
an arbitrary value of variable n and use the equality for the mean square error
9 1
(1 − 8µ)2n + (1 − 2µ)2n = 10−6
1024 64
9 B Cn 1 B Cn
(1 − 8µ)2 + (1 − 2µ)2 = 10−6 .
1024 64
This formula provides the relation between n and µ. Finding the value of µ
that produces minimal n is not simple. Note that the left side of the previous
equation consists of two positive terms. Assume that, for a sufficiently large
n, the terms are of the same order. It results in
(1 − 8µ)2 = (1 − 2µ)2
µ(5µ − 1) = 0
or µ = 0.2. For this value of step µ the number of iterations is
log 1024 −6
25 10
n= 9
≈ 9.888 ≈ 10
log 25
These values of µ and n correspond to the numerically obtained ones,
presented in the table. For µ < 0.2 the second term dominates in the mean
square deviation relation. The number of iterations can then be determined
as
1 B Cn
(1 − 2µ)2 = 10−6
64
log(64 · 10−6 )
n= .
log((1 − 2µ)2 )
For µ = 0.15 we get n ≈ 13. 537. This result is in agreement with the numerical
one obtained for n = 14. For µ > 0.2 the first term is dominant and
log(64 · 10−6 )
n= .
log((1 − 8µ)2 )
For µ = 0.22 value n = 17. 594 follows. It corresponds to the numerical result
n = 17.
Example 8.10. Analyze the convergence of the steepest descend method using the
eigenvalues of the autocorrelation matrix from the previous example.
⋆The autocorrelation matrix is
- .
5 3
R= .
3 5
Its eigenvalues follow from det (R − λI) = 0 as

' '
'5 − λ 3 ''
' =0
' 3 5 − λ'
(5 − λ )2 − 9 = 0
λ0 = 8
λ1 = 2
The steepest descend method converges for
2 2 1
µ< = = = 0.25.
λmax 8 4
The optimal rate of convergence is achieved for
2 2 1
µ= = = = 0.2.
λmax + λmin 8+2 5
Example 8.11. The autocorrelation function of an input signal is r xx (m) = 2−|m| .

The cross-correlation of the reference and the input signal is rdx (m) =
2δ(m) + δ(m − 1). Adaptive system is of order N = 3. The coefficients are
adapted using the steepest descend method. Find the optimal coefficient val-
ues, as well as the bound and optimal value for the step µ.
⋆The autocorrelation matrix and cross-correlation vector are
⎡ ⎤ ⎡ ⎤
1 12 14 2
R = ⎣ 12 1 12 ⎦ and rdx = ⎣1⎦ .
1 1 0
4 2 1
The optimal system coefficient values are

⎡ ⎤
2
H∗ = R−1 rdx = ⎣ 1
3
⎦.
− 23
The eigenvalues of the autocorrelation matrix are the roots of characteristic

polynomial
' 1 1'
'1 − λ '
' 1 2 4'
' 1−λ 1'=0
' 2 2'
' 1 1
1 − λ'
4 2
2 9
(1 − λ )3 + − (1 − λ) = 0.
16 16
Their values are

3
λ0 = = 0.75
4
√
9 − 33
λ1 = ≈ 0.407
8
√
9 + 33
λ2 = ≈ 1.843.
8
The steepest descend method converges with a step
2 16
µ< = √ ≈ 1.085.
λmax 9 + 33
Optimal rate of convergence is achieved if
2 2 8
µ= = √ √ = ≈ 0.889.
λmax + λmin 9− 33 9+ 33
+ 8 9
8
8.4 LMS ALGORITHM
Consider first the steepest descend method with iterative adaptation of

system coefficients. Denote the iteration index by k. In general, the input
and reference signals d(n) and x (n) are not stationary. Their statistical
parameters may change with the current instant n. As a consequence, the
coefficients of the adaptive system change in time. The iterative procedure
is then preformed for each instant n according to
Hk+1 (n) = Hk (n) + µ(rdx (n) − R(n)Hk (n))
where rdx (n) and R(n) are defined as the expectations
rdx (n) = E[d(n)X(n)]

R(n) = E[X(n)X T (n)].
Statistical properties of the signals are not fast-varying. For each next instant
n we may use the system coefficients obtained at the previous instant n − 1
(in K iterations) as the initial values
H 0 ( n ) = H K ( n − 1 ).
Assume that only one iteration is done for each time instant n. With K = 1 it
follows
H1 (n) = H0 (n) + µ(rdx (n) − R(n)H0 (n))

H1 (n) = H1 (n − 1) + µ(rdx (n) − R(n)H1 (n − 1))
H1 (n + 1) = H1 (n) + µ(rdx (n + 1) − R(n + 1)H1 (n)).
For notation simplicity, the index denoting the number of iterations will be
omitted (since it has been assumed that it is 1). Then we can write
H(n + 1) = H(n) + µ(rdx (n + 1) − R(n + 1)H(n)).
In the LMS algorithm the autocorrelation matrix R(n + 1) and the cross-
correlation vector rdx (n + 1) are approximated by their instantaneous values
rdx (n + 1) ≈ rdx (n) ≈ d(n)X(n)

R ( n + 1 ) ≈ R ( n ) ≈ X ( n ) X T ( n ).
Using this approximation the iteration formulae are
H(n + 1) = H(n) + µ(d(n)X(n) − X(n)X T (n)H(n)) (8.17)

T
= H(n) + µX(n)(d(n) − X (n)H(n)).
With y(n) = X T (n)H(n) it follows
H(n + 1) = H(n) + µ(d(n) − y(n))X(n). (8.18)
Difference d(n) − y(n) is the error signal e(n). A common LMS algorithm
form reads
H ( n + 1) = H ( n ) + µ e ( n ) X ( n ) (8.19)
In each time instant the coefficients of adaptive system are changed with
respect to their previous values in the direction of input signal vector X(n).
Intensity of the change is determined by the step µ and the error signal at
the previous instant e(n).
For a system of order N the LMS algorithm is numerically very effi-
cient. At each instant n it needs N + 1 multiplication and N additions.
8.4.1 Convergence of the LMS algorithm
Consider a stationary signals when matrix R(n) and vector rdx (n) are time
invariant. Then the LMS algorithm converges „in mean” toward the optimal
system coefficient values H∗
lim E[H(n)] = H∗
n→∞
under the same conditions as in the steepest descend case. The step µ in the
LMS algorithm should be such that
2
µ<
λmax
where λmax denotes the maximal eigenvalue of the autocorrelation matrix

R. It can be easily proven by considering the expected value of the adaptive
system coefficients
E[H(n + 1)] = E[H(n) + µe(n)X(n)] =

= E[H(n)] + µE[X(n)(d(n) − X T (n)H(n))] =
= E[H(n)] + µE[d(n)X(n)] − µE[X(n)X T (n)H(n)] =
= E[H(n)] + µrdx − µE[X(n)X T (n)H(n)].
Assume that the expected value E [H(n)], for a sufficiently large n, does not
depend on n as well as that X(n) and H(n) are mutually independent. Then,
with E[H(n + 1)] = E[H(n)] = H LMS , it follows
H LMS = H LMS + µrdx − µE[X(n)X T (n)] E[H(n)]
or
H LMS = H LMS + µrdx − µRH LMS .
From this relation we get
µRH LMS = µrdx

RH LMS = rdx
H LMS = R−1 rdx
H LMS = H∗ .
This proves the statement that the LMS algorithm coefficients converge „in
mean” to the optimal system coefficient values.
The convergence in mean does not mean that the LMS achieves the
optimal value in the stationary state. If there is a smallest difference between
the reference and the output signal it will cause the coefficients fluctuation.
In addition convergence in mean does not guarantee that the results will
converge to the same values. It can be shown that the LMS algorithm will
converge with finite variations of the coefficients and the error if the step µ
satisfies a more conservative bound
2
µ<
∑kN=1 λk
than the bound µ < 2/λmax requited for the convergence „in mean”. It is
known that the sum of the eigenvalues is equal to the trace of matrix R.
As it has been stated for the steepest
@ descend
A method, the trace can easily
calculated as Tr[R] = Nr xx (0) = NE | x (n)|2 = Ex , where Ex is input signal
energy.
8.5 LMS APPLICATION EXAMPLES
8.5.1 Identification of Unknown System
Consider the problem of unknown system identification. A way to solve this

problem is in using adaptive system with the same input as the input to the
unknown system. The unknown system output is used as a reference signal
d(n) in the adaptive system. If the unknown system can be described by
d(n) = a0 x (n) + a1 x (n − 1) + ... + a M−1 x (n − 1)
then the Nthe order adaptive system, with output signal

⎡ ⎤
x (n)
⎢ x ( n − 1) ⎥
⎢ ⎥
y(n) = H T (n)X(n) = [h0 (n) h1 (n) ...h N −1 (n)] ⎢ .. ⎥,
⎣ . ⎦
x ( n − N + 1)
can adapt its coefficients, through the iterative procedure, in such a way
that y(n) is as close as possible to d(n). In an ideal case, with N ≥ M, it is
possible to obtain limn→∞ H(n) = [ a0 a1 ...a M−1 0 ... 0]. In that case e(n) = 0.
The system is identified when the error is equal to zero. The identification
of an unknown system is illustrated in Figure 8.9.
If the unknown system is an infinite impulse response (recursive)
system or if the order of finite impulse response system is greater than the
adaptive system order, then we will get an approximation of the unknown
system, in the sense of minimal mean square error. The error signal will not
vanish as n increases.
unknown d(n)
system
x(n) adaptive y(n)

system - +
e(n)
Figure 8.9 Identification of unknown system.
Example 8.12. Consider a system with transfer function

H (z) = 3 + 2z−1 − z−2 + z−3 .
It has been assumed that this signal is unknown. Identification of this system
is done using an adaptive system of order N = 3. The identification process
is repeated with an adaptive system of order N = 5. The input to the system
x (n) is Gaussian zero-mean white noise with variance σx2 = 1. The step
µ = 0.05 is used in the adaptive algorithm. Comment the results.
⋆For the input signal x (n) the reference signal is d(n) = 3x (n) +
2x (n − 1) − x (n − 2) + x (n − 3). This reference signal is used in the adaptive
system of order N = 3 implemented as
y ( n ) = H T ( n ) X ( n ) = h0 ( n ) x ( n ) + h1 ( n ) x ( n − 1) + h2 ( n ) x ( n − 1).
The adaptive coefficients are calculated using H(n + 1) = H(n) + µ(d(n) −

y(n))X(n) with H(0) = 0 and µ = 0.05. The results of simulation (error signal
e(n) and coefficients H(n) = [h0 (n) h1 (n) h2 (n)] T ) are presented in Figure
8.10(top).
The error signal does not vanish with N = 3. The adaptive system
cannot identify the system by varying its coefficients. The reason is obvious.
The unknown system is of order 4 and we have tried to identify it with
an adaptive system of order 3. After about 100 iterations the error signal
and the coefficients assume a state with random variations and do not
produce stationary values. At the end of adaptation interval the coefficients
are h0 (200) = 3.16, h1 (200) = 1.79 and h2 (200) = −0.99. The average values
of these coefficients, calculated for n = 100, 101, . . . , 200, are
1 200
101 n=∑
h̄0 = h0 (n) = 2.72 h̄1 = 2.03 h̄2 = −0.92.
100
6 4
h (n)
4 3 0
coefficients h (n)
error signal e(n)
2 2
k
h (n)
1
0 1
-2 0
h (n)
2
-4 -1
(a) (b)
-6 -2
0 50 100 150 200 0 50 100 150 200
time index n time index n
6 4
h (n)
0
4 3
coefficients h (n)
h (n)
error signal e(n)
1
2 2
k
h (n)
3
0 1
h (n)
4
-2 0
h (n)
2
-4 -1
(c) (d)
-6 -2
0 50 100 150 200 0 50 100 150 200
Figure 8.10 Identification of unknown system from Example 8.12. System order is N = 3 (a-
b), and N = 5 (c-d). The error signal is presented on the left and the system coefficients on the
right.
They are close to the true values of first three system coefficients (3, 2, and
−1), meaning that the LMS algorithm in this case follows the true values „in
mean”.
For the fifth order adaptive system (N = 5), after about 100 iterations,
the error signal is almost 0. The adaptive system has identified the unknown
system. The final coefficient values in this case are
h0 (200) = 2.9999 h1 (200) = 1.9999

h2 (200) = −1 h3 (200) = 0.9999 h4 (200) = 0.
The last coefficient h4 (200) = 0, as expected, since the identification of a

fourth order system has been done by a fifth order adaptive system.
3
2 1 0
2
coefficients h (n)
error signal e(n)
4
1
k
0
0 3
-1 -1
1
-2 -2
(a) (b)
-3
0 50 100 150 200 0 50 100 150 200
3
1 0
2 2
coefficients h (n)
error signal e(n)
4
1
k
6
0
5 7 8 9
0 3
-1
-1
1
-2 -2
(c) (d)
-3
0 50 100 150 200 0 50 100 150 200
Figure 8.11 Identification of unknown system from Example 8.13. System order is N = 5
(a) and (b), and N = 10 (c) and (d). The error signal is presented on the left and the system
coefficients on the right. System coefficients hk (n) are labeled with k.
Example 8.13. Repeat the simulation from 8.12 for the case of unknown system
whose transfer function is
11 −1
1− 8 z
H (z) = 1 −1
.
1+ 4z − 15
64 z
−2
Use the step µ = 0.05 and the adaptive systems of order N = 5 and N = 10.
⋆In this case the unknown system is a system with an infinite impulse
response. In theory, we should have an adaptive system with very large
(infinite) order to identify this system. The identification results with the
adaptive systems of order N = 5 and order N = 10 are shown in Figure 8.11.
We can see that the system with order N = 10 reduces the error to a small
value, achieving a good approximation of the unknown system.
signal of interest
with interference
d(n)
interference x(n) adaptive y(n)

- + output
only system e(n)
Figure 8.12 Adaptive system for interference removal.
8.5.2 Noise Cancellation
Consider a setup where the signal of interest s(n) is acquired in a strong

noise environment ε(n). There are many such situations is real life (cockpit
and pilot voice in a plane, hands-free conversation in a car, speaking on
microphone in a loud noise environment are some of the examples). Assume
that in addition to the noisy signal we are in position to record (for example,
by a another microphone) one more signal, far from the desired signal. In
that case the other signal η (n) will contain noise only. This noise is highly
correlated with the noise ε(n) in the desired noisy signal. In such situations
an adaptive system, as in Figure 8.12, will be able to cancel out (significantly
reduce) the noise in the desired signal.
The input signal in this case is the noise only signal η (n), while the
reference signal is s(n) + ε(n). Note that the noises η (n) and ε(n) are highly
correlated, since they have a common source. However the noises are not
the same, since they propagate through different pats, including possibility
of reflections (from the walls or other objects). The desired signal s(n) is
statistically independent from the noise. The adaptive system, in ideal case,
will try make the output signal y(n) as close as possible to the reference
signal s(n) + ε(n). Since its input η (n) is correlated to ε(n) it may achieve its
cancellation when ε(n) = H T (n)X(n). Then the error is equal to the desired
signal, e(n) = d(n) − y(n) = s(n) + (ε(n) − H T (n)X(n)) = s(n).
Example 8.14. Consider a simple setup when we will be in a position to follow the
system behavior in an intuitive way. Assume that the input signal η (n) is a
white zero-mean Gaussian noise with variance ση2 = 1. The desired signal is of
the form s(n) = cos(2πn/512) + 0.5 sin(2πn/256 + π/3), with 0 ≤ n ≤ 5000.
The noise at the position of signal s(n) is ε(n) = 0.5η (n) − 0.7η (n − 1). Find
the optimal coefficients and then the error signal at the output of an LMS
based adaptive system from Figure 8.12. Comment the result with respect to
the LMS step µ.
⋆ A second-order adaptive system with the input
X(n) = [η (n) η (n − 1)] T
will be used. The adaptive system output is y(n) = H T (n)X(n) = h0 (n)η (n) +
h1 (n)η (n − 1). The reference signal is d(n) = s(n)+ ε(n) The input signal
autocorrelation matrix and the cross-correlation vector of the input and
reference signal are
- . - .
rηη (0) rηη (1) 1 0
R= = and
rηη (1) rηη (0) 0 1
- . - .
0.5rηη (0) 0.5
rdx = rεη = .
−0.7rηη (0) −0.7
The optimal coefficient values are
- .
0.5
H∗ = R−1 rdx = ,
−0.7
producing the output y(n) = h0∗ η (n) + h1∗ η (n − 1) = 0.5η (n) − 0.7η (n − 1), as
expected. The error signal is then e(n) = d(n) − y(n) = s(n).
Next the LMS algorithm is used in the adaptation, at each time instant
n, as H(n + 1) = H(n) + µe(n)X(n) with H(0) = 0. For large n the error
value will not vanish since, in an ideal case e(n) = s(n). Therefore the system
coefficients H(n + 1) will fluctuate with µe(n)X(n) ̸= 0. In means that, in
order to reduce these fluctuations, the step µ should be much lower than its
bound µ < 2/λmax = 2 required by the convergence condition. The results
with µ = 0.01 and µ = 0.001 are presented in Figure 8.13.
Example 8.15. Consider a signal s(n) embedded in high noise ε(n). The signal
acquisition is done using two microphones. One close to the source of s(n)
and the other far from this source.
Signal s(n) is modelled as a nonstationary zero-mean Gaussian noise
with variance σs2 (n) = 3 sin(πn/100)4 . Signal ε(n) is a stationary zero-mean
white Gaussian noise with variance σε2 = 300. The noise at the input to the
first and the second microphone is modified by the system transfer functions
H1 (z) = 1 + 0.5 z−1 + 0.2 z−2 − 0.2 z−3 + 0.1 z−4

H2 (z) = 1 − 0.2 z−1 + 0.1 z−2 ,
respectively. Using an adaptive system of order N = 10 reduce (eliminate)

the noise from the signal recorded by the first microphone. Experimentally
find the value of step µ so that the signal-to-noise ratio is about 10dB.
0.5 0.5
coefficients h (n)
coefficients hk(n)
k
0 0
-0.5 -0.5
-1 -1
0 2000 4000 0 2000 4000

3 3
2 2
error signal e(n)
error signal e(n)
1 1
0 0
-1 -1
-2 -2
-3 -3
0 2000 4000 0 2000 4000
Figure 8.13 Simulation results for Example 8.14 – Adaptive system for noise cancelation.
System coefficients are given in upper subplots. Lower subplots present error signal and target
signal (black line).
⋆The simulation results are presented in Figure 8.14. Presented are

the adaptive system coefficients hk (n), error signal e(n) and the signal s(n)
for 0 ≤ n ≤ 500. The step µ = 0.00005 is used. Note that the system cannot
produce the error signal equal to zero since error signal contains signal s(n).
Smaller value of step µ enables lower variations of the coefficients and a
closer approach to the optimal values. In the staring iteration the noise ε(n)
is dominant in the error signal. As the coefficients of the system approach to
their optimal value the error reduces and assumes the values closer to the
desired signal s(n). Experimenting with various step values we concluded
that µ = 0.00005 can achieve the required signal-to-noise ratio
8.5.3 Sinusoidal Disturbance Cancellation
In many application a desired signal is disturbed by a sinusoidal interfer-

ence of unknown frequency. A system for adaptive cancellation of such
1 0
coefficients h (n) 0.8
k
1
0.6
0.4
2
0.2
0 4 5 6 7 8 9
-0.2 3
0 50 100 150 200 250 300 350 400 450 500

time index n
20
10
error signal e(n)
-10
-20
0 50 100 150 200 250 300 350 400 450 500
time index n
Figure 8.14 Simulation results for Example 8.15 – Adaptive system for noise cancelation.
System coefficients are given in upper subplot. Lower subplot presents error signal (gray line)
and target signal (black line).
interference is presented in Figure 8.15. In this case the measured signal

contains a sinusoidal interference A cos (ω0 n + ϕ), in addition to the desired
signal s(n). The adaptive system is set in such a way that the reference signal
d(n) is the measured signal x (n), while the input signal to the adaptive sys-
tem is a delayed version of the measured signal x (n − M ). For a sufficiently
large delay M we may assume that the desired signal s(n) is not correlated,
i.e., that its autocorrelation function is rss (m) = 0 for |m| ≥ M. The reference
signal d(n) will have two components. One component corresponding to
the desired signal s(n), which is not correlated with the input x (n − M ).
The other component is the sinusoidal signal which is correlated for any
delay M. The adaptive system will (in an ideal case) be able to adjust its
parameters to remove correlated component (in this case the sinusoidal in-
terference). Then we will get e(n) = d(n) − y(n) = s(n).
In an ideal case the adaptive system should ' adjust'its coefficients
' ' to be-
have as an inverse notch filter at ω = ±ω0 , i.e., ' H (e jω )' = 'DFT[h∗k ]' = 1 for
d(n)
input -M adaptive y(n) output

x(n) z
system - + e(n)
e(n)
Figure 8.15 Adaptive system for sinusoidal interference removal.
ω = ±ω0 and H (e jω ) = 0 elsewhere. Such a system would produce y(n) ∼ =

A cos (ω0 n + ϕ) if the input signal is x (n) = s(n) + A cos (ω0 n + ϕ), where
s(n) is a wide-band signal. Then d(n) − y(n) ∼ = x (n) − A cos (ω0 n + ϕ) =
s ( n ).
Example 8.16. Consider an adaptive system for the sinusoidal interference cancel-
lation, with input signal
√ B nC
x (n) = s(n) + 200 cos 4π ,
32
where s(n) is a stationary zero-mean Gaussian white noise with autocorre-
lation rss (m) = δ(m) + 0.25δ(m − 1). Simulate the adaptive system of order
N = 32 to eliminate the sinusoidal interference. Use the delay M = 3 and step
µ = 0.00002.
⋆Results of the simulation are presented in Fig. 8.16. The input signal
x (n) is shown in Fig. 8.16(a), and the desired signal s(n) in Fig. 8.16(b). The
adaptive coefficients are presented in Fig. 8.16(c). The output signal error e(n)
is given in Fig. 8.16(d). We can conclude that after about 100 iterations the
output signal is close to s(n), with a delay of M = 3 samples.
8.5.4 Signal Prediction
Adaptive system configuration to predict one step ahead signal value is

presented in Figure 8.17. The input signal x (n) is used as the reference signal
d(n) = x (n). The aim is to obtain this value of signal using its past samples
x (n − 1), x (n − 2), ..., x (n − M ). The input signal vector is of the form
⎡ ⎤
x ( n − 1)
⎢ x ( n − 2) ⎥
⎢ ⎥
X(n) = ⎢ .. ⎥
⎣ . ⎦
x (n − N )
20
10
input signal x(n) 10 5
signal s(n)
0 0
-10 -5
(a) -10 (b)

-20
0 50 100 150 200 0 50 100 150 200
0.1
10
coefficients h (n)
error signal e(n)

0.05 5
k
0 0
-0.05 -5
(c) -10 (d)

-0.1
0 50 100 150 200 0 50 100 150 200
Fourier transfom of hk(200)
1
0.8
0.6
0.4
0.2
(e)
0
-2 0 2
frequency ω
Figure 8.16 Simulation results for Example 8.16. Signal with sinusoidal interference (a), signal
without interference (b), system coefficients (c), output signal (d), and Fourier transform of the
final system coefficients, hk (200), k = 0, 1, 2, ...N − 1, (e).
with y(n − 1) = H T (n)X(n). The error signal is formed as
e ( n ) = d ( n ) − H T ( n ) X ( n ) = x ( n ) − y ( n − 1 ).
If the adaptive system is able to adjust its coefficients so that the error is
small, with y(n − 1) ≈ d(n) = x (n) then its output will predict the next
adaptive y(n) y(n-1)

x(n) z-1 - +
system
output
e(n)
Figure 8.17 Adaptive prediction.
signal value
y ( n ) ≈ x ( n + 1).
Consider a signal described by
x (n) = a1 x (n − 1) + a2 x (n − 2) + ... + a M x (n − M ) + ε(n)
where ε(n) is a zero-mean white noise with variance σε2 . We may expect that
the optimal coefficients for one step ahead prediction should be
H∗ = [ a1 a2 a M 0 ... 0]T
for an adaptive system whose order is N > M with
y(n) = h0∗ x (n − 1) + h1∗ x (n − 2) + ... + h∗N −1 x (n − N ).
The prediction error will depend on the ratio of the recursive part of signal
x (n) and random part ε(n). For large n the error value will not vanish since,
in an ideal case e(n) = ε(n). The system coefficients H(n + 1) will fluctuate
with µe(n)X(n) ̸= 0 causing so called excessive mean square error. In order
to reduce these fluctuations (this kind of error), the step µ should be much
lower than its bound µ < 2/λmax = 2 required by the convergence condition.
The excessive means square error is proportional to the signal energy and
the algorithm step, EMSE = µEx /2.
Example 8.17. Consider a third order adaptive system for signal prediction. As-
sume that the signal x (n) is random signal with autocorrelation function
r xx (m) = σx2 (m)δ(m). Find the output signal, assuming that the adaptive
system has adjusted its coefficients is such a way that they are equal to the
optimal ones.
⋆Samples of the input signal are uncorrelated. Autocorrelation matrix

of the input signal X(n) = [ x (n − 1) x (n − 2) x (n − 3)] T is
⎡ ⎤
σx2 (n) 0 0
R(n) = ⎣ 0 σx2 (n − 1) 0 ⎦.
0 0 2
σx (n − 2)
Cross-correlation between the input and reference signal d(n) = x (n) is

⎡ ⎤ ⎡ ⎤
E[ x (n) x (n − 1)] 0
rdx (n) = ⎣E[ x (n) x (n − 2)]⎦ = ⎣0⎦
E[ x (n) x (n − 3)] 0
Optimal coefficient values are H∗ (n) = R−1 (n)rdx (n) = 0. It means that the
output signal is zero.
Example 8.18. Assume that in the previous example the input signal is stationary
with autocorrelation r xx (m) = 2−|m| . Find the optimal coefficient values and
the form of optimal predictor.
⋆Autocorrelation matrix of the input signal and its cross-correlation
vector with the reference signal are
⎡ 1 1⎤
1 2 4
R = ⎣ 12 1 1⎦
2
1 1
4 2 1
⎡ ⎤ ⎡ ⎤⎡1⎤
E[ x (n) x (n − 1)] r xx (1) 2
rdx = ⎣E[ x (n) x (n − 2)]⎦ = ⎣r xx (2)⎦ ⎣ 14 ⎦
E[ x (n) x (n − 3)] r xx (3) 1
8
The optimal coefficient values are

⎡ 1 1 ⎤ −1 ⎡ 1 ⎤ ⎡1⎤
1 2 4 2 2
H∗ = R−1 rdx = ⎣ 12 1 1⎦
2
⎣1⎦
4 = ⎣0⎦
1 1 1 0
4 2 1 8
The output signal, predicting one step ahead the input signal value is
1
y(n) = x̂ (n + 1) = x ( n ).
2

x (n) = −0.1x (n − 1) + 0.72x (n − 2) + ε(n)
where ε(n) is a zero-mean white noise with variance σε2 = 0.5. Find optimal
values of the system coefficients for one step ahead prediction of a second
order adaptive system. Plot the adaptation coefficients for the second order
LMS algorithm with µ = 0.1 and µ = 0.01. Calculate and plot average of the
prediction square error in 100 realizations in dB for both cases. What is the
convergence bound for µ.
Repeat the calculation for x (n) = 12 x (n − 1) + ε(n) and the first-order
adaptive system.
⋆For the optimal values of the adaptive prediction system we have to
find autocorrelation matrix of the input signal vector, in this case of X(n) =
[ x (n − 1) x (n − 2)]T . Signal x (n) is obtained as the output of a recursive
system whose input is ε(n) and the transfer function is
9 8
1 17 17
H (z) = = + .
1 + 0.1z−1 − 0.72z−2 1 + 0.9z−1 1 − 0.8z−1
Its impulse response is
9 8
h(n) = [ (−0.9)n + (0.8)n ]u(n).
17 17
Therefore the signal x (n) can be written as
x ( n ) = h ( n ) ∗ ε ( n ).
The autocorrelation function of x (n) is
r xx (m) = E[ x (n + m) x (n)]
∞ ∞
= E[ ∑ ∑ ε(n + m − k1 )h(k1 )ε(n − k2 )h(k2 )]
k 1 =0 k 2 =0
∞
= σε2 ∑ h(k )h(k − m)
k =0
since rεε (m) = σε2 δ(m). Thus, we have

∞
r xx (0) = σε2 ∑ h2 (k) = 1.19,
k =0
∞
r xx (1) = σε2 ∑ h(k)h(k − 1) = −0.425,
k =0
and r xx (2) = 0.8993. The optimal coefficient values are
- . −1 - . - .
1.19 −0.425 −0.425 −0.1
H∗ = =
−0.425 1.19 0.8993 0.72
since
@ A @ A
R(n)=E X(n)X T (n) = E [ x (n − 1) x (n − 2)] T [ x (n − 1) x (n − 2)]
- .
r (0) r xx (1)
= xx
r xx (1) r xx (0)
and rdx (n) = E [d(n)X(n)] = [r xx (1) r xx (2)] T .

Adaptive system for prediction is implemented using the LMS al-
gorithm, H(n + 1) = H(n) + µe(n)X(n) with H(0) = 0. Input vector is
X(n) = [ x (n − 1) x (n − 2)] T , while the error signal has the form e(n) =
d(n) − H T (n)X(n) with d(n) = x (n) and H(n) = [h0 (n) h1 (n] T . The results
obtained in each iteration (time index) are presented in Figure 8.18. The re-
sults are averaged over 500 independent realizations. The eigenvalues of ma-
trix R are λ0 = 1.6150 and λ1 = 0.7650. The bound for convergence "in mean"
is µ < 2/λmax = 1.2384. As expected, much lower values of step µ should be
used to reduce random fluctuations of the coefficients.
8.5.5 Adaptive Antenna Arrays
Antenna array (system) is a set of antennas distributed in space to produce

desired performance. Commonly the antennas are located along a line, with
equal consecutive distances, forming a uniform linear antenna array. Signals
are received by individual antennas with different delays, being dependent
on the antennas location. The received signals from each antenna are com-
bined to achieve the antenna array performance. The most common way
to combine signals from different antennas is to add up their appropriately
weighted values. This is done by a linear combinator, Figure 8.5. It produces
a weighted sum of signals obtained from individual antennas.
When the linear combinator changes its weighting coefficients in time,
then it performs a task of an adaptive system. Various tasks can be required
from the adaptive combinator. One possible requirement is that the antenna
array amplifies a signal from a specific direction (being a known direction
of desired signal). A task could be to find the direction of signal arrivals
(DOA) and to track them in time if they change. In some cases, in addition
to the desired signal, the antenna system receives one or more undesired
interferences from different directions. The task of an adaptive system could
be to cancel out or to suppress the interferences.
Consider a linear uniform antenna array with individual antennas
being spaced at l. Assume that a direction of arrival of signal is defined
by angle θ as well that the signal form at the first antenna is
r0 (t) = s(t)e jω0 t ,

0.2 0.2
0 0
h (n), h (n)
h (n), h (n)
-0.2 -0.2
1
1
-0.4 -0.4
0
0
-0.6 -0.6
-0.8 -0.8
0 500 1000 0 500 1000
2 2
0 0
10loge2(n) [dB]
10loge2(n) [dB]
-2 -2
-4 -4
-6 -6
0 500 1000 0 500 1000
Figure 8.18 Coefficients and error in the prediction setup of the second order adaptive LMS
algorithm for µ = 0.1 (left) and µ = 0.01 (right). The results are averaged over 500 realizations.
where s(t) is a narrowband signal and ω0 is its carrier frequency. Signal

arrives to other antennas with a delay. The front of incident wave has to
pass the path l cos(θ ) to arrive at the second antenna. Since the propagation
speed is the speed of light c, time delay from the first to the second antenna
is
td = l cos(θ )/c.
The same delay holds for each next antenna, since the antenna array is
uniform. The signal at (k + 1)th antenna is
* + B
l cos(θ )
C
l cos(θ ) jω0 t−k c
rk (t) = s t − k e .
c
Since the signal s(t) is narrowband, meaning that its amplitude variations
are slow, we may write
* +
kl cos(θ ) ∼
s t− = s ( t ).
c
Including this fact, the signal at the (k + 1)th antenna assumes the form
ω0 l
rk (t) = s (t) e jω0 t e− j c kl cos(θ )
= s (t) e jω0 t e− j2π λ k cos(θ ) ,
where λ is wavelength of the propagating wave λ = 2πc/ω0 .

We can conclude that the signals at different antennas differ in phase
only. It means that it is possible to define an appropriate linear combination
of these signals so that they may add up in phase for a given incident angle
θ, or to cancel out for some other incident angles θ.
The input signals to the linear combinator are demodulated by e− jω0 t
and discretized to form
' '
' l '
xk (n) = rk (t)e− jω0 t ' = s(t)e− j2π λ k cos(θ ) ' .
t=n∆t t=n∆t
This system is presented in Figure 8.19. The output of linear combinator is
N −1 N −1 l
y(n) = ∑ hk (n) xk (n) = s(n∆t) ∑ hk (n)e− j2π λ k cos(θ )
k =0 k =0
N −1
= s(n∆t) ∑ hk (n)e− jωk = s(n∆t)FTk [hk (n)]|ω =2π l cos(θ )
λ
k =0
= s(n∆t)H T (n)a(ω )|ω =2π l cos(θ )
λ
where
H T (n) = [ h0 (n) h1 (n)...h N −1 (n)]

a(ω ) = [1 e− jωk e− jω2k ...e− jω ( N −1) ] T
l
ω = 2π cos(θ )
λ
The output signal y(n) is equal to the input signal s(n∆t) multiplied by
the Fourier transform of the coefficients hk (n), k = 0, 1, ..., N − 1 at ω =
2π λl cos(θ ).
h (n)
x0(n) 0
l h1(n)
x (n) y(n)
1
+
...
...
...
h (n)
x (n) N-1
N-1
θ -
x (n) +
ref
e(n)
incident antenna adaptive
wave array system
Figure 8.19 Uniform antenna array with adaptive system for interference rejection.
Now we will consider an adaptive setup of this system with the aim
to cancel out input interference signals. Assume that several waves with
incident angles θ1 , θ2 ,...,θ P arrives to this antenna array. The input signal to
each antenna is then
P l P
xk (n) = ∑ s p (n∆t)e− j2π λ k cos(θ p ) = ∑ s p (n∆t)e− jω p k
p =1 p =1
with ω p = 2π λl cos(θ p ). The output of linear combinator is

( )
N −1 P N −1
− jω p k
y(n) = ∑ hk (n) xk (n) = ∑ s p (n∆t)
p =1
∑ hk (n)e .
k =0 k =0
It is a sum of the Fourier transforms of FT k [hk (n)] at the frequencies ω p =

2π λl cos(θ p ) multiplied with amplitudes s p (n∆t). If we want to cancel out
all input signals then the weighting coefficients should be adjusted in such
a way that their Fourier transform is zero (notch filter) at the frequencies
corresponding to the directions of arrivals of the considered signals.
Consider now N antennas in the array and use an additional antenna
for the reference signal
P l
d(n) = ∑ s p (n∆t)e− j2π λ N cos(θ p ) .
p =1
The adaptive LMS based system is implemented using
H(n + 1) = H(n) + µe(n)X∗ (n)

with H(0) = 0 (see Subsection 8.6.5). Input vector is
X(n) = [ x0 (n) x1 (n)...x N −1 (n)]T .
The reference signal is the output of reference antenna d(n) = x N (n). With
e(n) = d(n) − H T (n)X(n) and d(n) = x N (n) we can write
@ A@ AT
e(n) = −H T (n) 1 X T (n) x N (n)
All system coefficients, including d(n), are
[−H(n) 1] = − [h0 (n) h1 (n)...h N −1 (n) − 1]T .
They will approach the values such that their Fourier transform is a notch
filter form like function. Then all the input signals will be canceled out
and the error e(n) will be zero-valued (assuming that the order of system
is appropriate for the number of input signal from different directions). It
was assumed that the desired signal was not present (switched off) during
the adaptation process, otherwise it would be canceled out as well. When
the system ends adaptation we can then switch on out desired signal from
a direction that does not correspond to one of interferences. It will pass
through the system, while all interfering signals are canceled out.
This kind of system is simulated using an adaptive system of order
N = 10, with four interfering signals with the directions or arrival θ1 = 30◦ ,
θ2 = 75◦ , θ3 = 90◦ and θ4 = 120◦ . Note that the ability to cancel out a number
of disturbances depends on the system order and the positions of the angles
of arrivals. With for example, 10 coefficients we will not be able to achieve
an arbitrary number of arbitrary positioned zeros in its Fourier transform.
The antenna system gain is
' '
' y(n) ' '' T '
'
'
A(θ ) = ' ' = 'H (n)a(ω )
s(n∆t) ' |ω =2π λ cos(θ ) '
l
' '
' '
= 'FTk [hk (n)]|ω =2π l cos(θ ) '
λ
or in decibels
a(θ ) = 20 log10 A(θ ) [dB].
It is calculated for angles 0◦
≤θ≤ 180◦
and presented in Figure 8.20. The
antenna system is adjusted to cancel out the interference (gain of the system
is here below −25dB). Signals from other directions will pass unattenuated
through this system, with a gain of about 5dB. A radiation plot of this
system is presented in 8.21.
10
0
antenna gain [dB]
-5
-10
-15
-20
-25
-30
0 15 30 45 60 75 90 105 120 135 150 165 180
incident angle [degrees]
Figure 8.20 Antenna system gain for various incident angles. Iterference incident angles are
marked with arrows.
o
θ=90 θ=75 o
o
1 20
θ=
θ=
3 0
o
antenna array
Figure 8.21 Radiation plot of the antenna system.
8.5.6 Acoustic Echo Cancellation
In this case the input to microphone is an acoustic signal. This is the desired
signal in adaptive system. In addition to this signal there are interference
signals coming from speakers. These signals come to the microphone over
direct path and one or more reflected paths. Adaptive system has the task
reflection voice
speaker microphone
direct path
d(n)
input adaptive y(n) output

x(n) system - + e(n)
e(n)
Figure 8.22 Adaptive system for acoustic echo cancelation.
to cancel out the influence of this interference. The system for adaptive
acoustic echo cancellation is presented in Figure 8.22. This kind of adaptive
systems is used in hands-free devices and in the systems for audio commu-
nication over internet.
Example 8.20. Consider a system as in Figure 8.22. Assume that the signal from
microphone is sampled with frequency f s = 11025 Hz. Speed of the acoustic
signal propagation is c = 330 m s . Speaker is at the distance r0 = 27 cm from the
microphone, meaning that the direct component reaches microphone with
a delay of f s r0 /c ≈ 9 samples. The system is in a room whose dimensions
are such that the reflected components passing paths longer than 3m can
be neglected. From this fact we can conclude that the maximal delay is
100 samples. The intensity of reflected components is inversely proportional
to the propagation path. With these assumptions the impulse response of
the system that transfers a signal x (n) from the speaker to the input in
microphone can be modelled as
⎧
⎨ 1 for n = 9
wn
hecho (n) = for 10 ≤ n ≤ 100
⎩ n
0 for other values of n
where wn are uncorrelated zero-mean Gaussian variables of variance 1. Sig-

nal x (n) is modeled as a zero-mean Gaussian white noise with variance
σx2 = 25. The acoustic signal s(n) is modeled as nonstationary
P nπ Q zero-mean
Gaussian random process with variance σs2 (n) = 3 sin4 250 .
⋆Results of simulation of this system are presented in Figure 8.23.
An LMS adaptive system of order N = 100 is used with step µ = 0.00005.
10
signal s(n)
voice
-10
0 250 500 1500 1750 2000 3000 3250 3500

time index n
10
microphone
signal d(n)
-10
0 250 500 1500 1750 2000 3000 3250 3500

time index n
10
signal e(n)
output
-10
0 250 500 1500 1750 2000 3000 3250 3500

time index n
Figure 8.23 Acoustic echo cancelation example.
Acoustic signal is presented. Signal at the output from microphone that

contains components from the speaker is presented as well. The output
signal in this case is equal to the error signal. These signals are shown for
0 ≤ n < 500, for 1500 ≤ n < 2000, and for 3000 ≤ n < 3500. The output signal,
after the adaptation process, clearly contains acoustic signal, what is not case
at the microphone input. The ratio of the acoustic signal and the total echo
of signals at the microphone input is −14dB, while in the error signal the
ratio of the acoustic signal and the remaining echo signals is 12dB. Total
improvement is about 26dB.
System behavior as a function of the step µ is presented in Figure 8.24.
Rejection of the echo signals in dB is used as a parameter for the system
description. We can see that smaller step µ produces higher echo rejection,
but with a slower convergence of the algorithm.
35
30 µ= 0.00001
echo rejection ratio [dB]
25
µ= 0.00005
20
15 µ= 0.00020
10
µ= 0.00050
5
0
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2
time [s]
Figure 8.24 Echo rejection ratio for various steps µ.
8.6 VARIATIONS ON THE LMS ALGORITHM
8.6.1 Sign LMS
The LMS algorithm is numerically efficient. In each iteration only N + 1

multiplications and N additions should be performed. The step parameter
µ can be assumed so that it does not require multiplications, but a shift only
(in decimal system its form 10 −b requires shift only, while in the binary sys-
tem 2−b is a shift operation, where b is an integer). In this way the number
of multiplications will be reduced to N in each iteration. Multiplications (as
computationaly more demanding arithmetic operation than the additions)
may be completely avoided if we use the error value just to find the direction
of change of the adaptive coefficients. If the error is replaced by its sign only,
then we have a sign LMS algorithm,
H(n + 1) = H(n) + µ sign(e(n))X(n).
This form, with appropriate µ, reduces the number of arithmetic operations

to one addition for each coefficient in each iteration.
Another variant of the LMS with reduced number of multiplications is
to apply the sign operator to the input vector X(n). Then we get the signed-
regressor LMS form
H(n + 1) = H(n) + µe(n) sign(X(n)).

The number of arithmetic operations is the same as in the sign LMS algo-
rithm.
For an arbitrary step µ it is still possible to avoid multiplications by
applying sign functions to both the error and the signal vector X(n). The
sign-sign LMS is defined by
H(n + 1) = H(n) + µ sign(e(n)) sign(X(n)).
Note that the change of system coefficients in each iteration is ±µ. It pre-
vents the system from achieving stationary state (coefficients oscillate). To
avoid this effect it is possible to define a sign function with a "dead zone" as
sign(α − D ) + sign(α + D )
signD (α) = .
2
Function sign D (α) is equal to −1 for α < − D, and 1 for α > D, while it is 0
for |α| < D. Value of this function at the discontinuity points is 1/2.
Example 8.21. Consider adaptive system described in 8.12 (page 461). Simulate the
system using signed error LMS, signed regressor LMS, and sign-sign LMS.
Use the adaptive system of order N = 5 with step µ = 0.05.
⋆Simulation results are presented in Figure 8.25. We can conclude that
the convergence is slower when the sign is applied to the error function (in
both cases) than in the case when the sign is applied to the signal vector. If the
error signal is kept in its original form, when the error approaches to zero the
system coefficients approach to their stationary values, without oscillations.
This is not the case for sign error LMS form.
8.6.2 Block LMS
The block LMS differs from the standard in the sense that the coefficients
are not modified at each instant n, but after each K instants. Time index is of
the form n = pK + m, where K is the block length, p is the block index,
and m is the index of a sample within a block 0 ≤ m < K. For adaptive
systems with large N the computation time of standard LMS algorithm can
be reduced by using the block LMS. They can be implemented with the FFT
algorithms (fast block LMS algorithm). Coefficients are adapted in such a
way that all coefficient modifications within a block are added up and the
final coefficients modification is done according to
K −1
H ( p + 1) = H ( p ) + µ ∑ e( pK + m)X( pK + m).
m =0
Signed error LMS Signed regressor LMS Sign-sign LMS

error signals e(n) 5 5 5
0 0 0
-5 (a) -5 (b) -5 (c)

0 100 200 0 100 200 0 100 200
4 4 4
coefficients hk(n)
0 0 0
2 1 2 1 2 1
3 3 3
0 4 0 4 0 4
2 2 2
(d) (e) (f)
-2 -2 -2
0 100 200 0 100 200 0 100 200
time index n time index n time index n
Figure 8.25 Simulation results for Example 8.21.
Example 8.22. Consider the system from Example 8.15 (page 465). Simulations will
be repeated with the block LMS using the block size K = 50.
⋆The results of simulation are presented in Figure 8.26. Note that
the coefficients change at the end of each block. Deviations of the adaptive
system coefficients are lower than in the LMS algorithm. The input signal-to-
noise ratio is −25dB, while this ratio at the output is 11dB.
8.6.3 Normalized LMS Algorithm
Deviations of the adaptive system coefficients in stationary state (when the

error does not vanish) depend on the algorithm step and signal energy.
The idea in normalized LMS algorithms is to reduce the step µ in order
to compensate the influence of signal energy. The step in normalized LMS
algorithm is
µnorm
µ=
1 + X T (n)X(n)
where the normalization is done with the input signal energy ∥X(n)∥22 =
X T (n)X(n) within the considered N signal samples. Value 1 is added in
denominator to avoid the problem with small signal energy. The adaptation
1 0
coefficients h (n)
0.8
k
1
0.6
0.4
2
0.2
4 5
0 6 7 8 9
-0.2 3
0 50 100 150 200 250 300 350 400 450 500

time index n
20
10
error signal e(n)
-10
-20
0 50 100 150 200 250 300 350 400 450 500
time index n
Figure 8.26 Block LMS simulation for Example 8.22.
relation is
µnorm
H ( n + 1) = H ( n ) + e(n)X(n)
1 + X T (n)X(n)
The convergence bound for the step is
µnorm < 2
In applications, the value µnorm = 0.5 is commonly used.

A generalization of the normalized LMS is
µnorm
µ= (8.20)
α + X T (n)X(n)
where α is a small constant, α ≪ X T (n)X(n).

8.6.4 LMS with Variable Step Size
The basic idea of a variable step size (VSS) variant of the LMS algorithm is to
change the step µ during iterations. The step should not be too large to cause
divergence, but also not too small so that it cannot detect possible changes
in the coefficients. Various variable step size algorithms have been derived
in literature, using the error signal, the input signal, the reference signal, and
the output signal in the considered and previous instants. Previous values
of the step µ are also used in the algorithms.
The normalized LMS, in the case that the signal energy changes during
the considered interval, may be considered as a variable step size LMS. A
form of the normalized LMS was used to define an interesting and efficient
variable step size algorithm called generalized normalized gradient descent
(GNGD) algorithm, with appropriate adaptation of coefficient α in (8.20).
It takes into account most of above-mentioned signals. Adaptation formula
for coefficient α is derived in the form
e ( n ) e ( n − 1) X T ( n ) X ( n − 1)
α(n + 1) = α(n) − ρµ 2
(8.21)
(e(n − 1) + X T (n)X(n))
where e(n) = d(n) − H T (n)X(n) is the error signal and ρ is a constant ρ < 1.
A simple form of the variable step size LMS can be obtained using
the standard LMS calculated with two (or several) step values µ. One µmin
should be sufficiently small that the coefficient deviations in the steady state
are small and the other µmax sufficiently large that the convergence is fast
when the change of the coefficients is detected. The crucial decision in this
simple algorithm is when to use the LMS with the small step and when the
LMS with the large step. One of possible criteria is based on the energy of
error signal. If the error increases the algorithm will switch to the larger step,
after few instants, when the energy of error exceeds the threshold α.
The transition period in this case could be much shorter if the variable
step size LMS based on the weighting coefficients bias-variance trade-off is
used. In this algorithm the difference between the coefficients is compared
with the expected standard deviation of the coefficients (with a constant κ)
for considered steps µ,
|hk (n, µmin ) − hk (n, µmax )| ≷ κ (σµmin + σµ max ).
If the difference is small (within confidence interval with few (σµmin +
σµ max )) then the system is assumed to be in the stationary state and small
step µmin should be used. Otherwise the system is in a transition and the
large step µmax should be used. Standard deviations of the coefficients can
be calculated based on the input signal energy and the used steps µ.
Example 8.23. Consider the system from Example 8.15 (page 465). The simulation
will be repeated with the variable step size LMS. In this case it has been
assumed that at the instant n = 200 transfer function H1 (z) changes to
H1 (z)|for n≥200 = 1 + z−1 + 0.2 z−2 + 0.75 z−3 − 0.1 z−4 .
Consider two ways of the step µ changes. In the first case use
αµmin + Ee (n)µmax
µ(n) =
α + Ee (n)
where µmin = 0.00005, µmax = 0.0005 and α = 25, while Ee (n) is an average
energy (power) of the signal in previous instants K = 50
n
1
Ee (n) = ∑ e2 ( n )
K k = n − K +1
In the second case use only two steps µmin and µmax the switching criterion
Ee (n) ≷ α.
⋆The result of simulation are presented in Figure 8.27. At the begin-
ning, the algorithm uses maximal possible step size µmax . Then the step de-
creases. At n = 200 there is an abrupt change in the considered system and
the adaptive system adjusts its step to the new circumstances.
The results using the second way of step size change, using only
two steps µmin and µmax , are presented in Figure 8.28. On the coefficients
plot, a gray shade indicates the region where the system uses larger step
µ(n) = µmax . Within the remaining time intervals the lower value of step
µ(n) = µmin is used.
Example 8.24. A system is defined by

y(n) = x (n) + 0.8x (n − 1)
where x (n) is a zero-mean Gaussian random signal with variance σx2 = 0.6.
Using the constant step LMS with µ = 1 and µ = 0.1 identify the system.
Compare the identification result with the normalized LMS (8.20) using
adaptive α defined by the generalized normalized gradient descent (GNGD)
algorithm, (8.21).
⋆Result of the identification in the form of square error 10 log |e(n)|2

[dB], averaged over 100 realizations, is presented in Figure 8.29. We can
clearly see the difference in the rate of convergence. For the constant step
LMS with µ = 1 we already see an increase in the error due to large step,
meaning that further step increase could lead to algorithm instability, since
2/Tr[R] = 2/(2σx2 ) = 1.6667..
1 0 0
coefficients hk(n) 1 3
0.5
2 1
2
0 4 5 4
5
3
0 50 100 150 200 250 300 350 400 450 500

time index n
-4
x 10
time varying step µ(n)
5 µ
max
4
3
2
1 µ
min
0
0 50 100 150 200 250 300 350 400 450 500
time index n
20
10
e(n) and s(n)
-10
-20
0 50 100 150 200 250 300 350 400 450 500
time index n
Figure 8.27 LMS algorithm with variable step (Example 8.23, first case).
8.6.5 Complex LMS
When the input signal x (n) (or/and the coefficients of an adaptive system
h(n), or its reference signal d(n)) are complex-valued then the complex
LMS algorithm should be used. In this case the square absolute value is
minimized. The error function is
ε = |e(n)|2 = e(n)e∗ (n)
where (·)∗ denoted complex-conjugate value.

1 0 0
coefficients h (n)
k
1 3
0.5
2 1
2
4 5 4
0 5
3
0 50 100 150 200 250 300 350 400 450 500

time index n
20
10
e(n) and s(n)
-10
-20
0 50 100 150 200 250 300 350 400 450 500
time index n
Figure 8.28 LMS algorithm with variable step (Example 8.23, second case).
-50
average error 10loge2(n) [dB]
1 Adaptive GNGD
2 LMS with µ=1
-100 3 LMS with µ=0.1
-150
-200 1 2 3
-250
-300
0 100 200 300 400 500 600 700 800 900 1000
time index n
Figure 8.29 Averaged square error in dB for the constant LMS with µ = 0.1 and µ = 1 and the
variable step size generalized normalized gradient descent (GNGD) algorithm.
It is easy to show that in this case the adaptation formula reads
H ( n + 1 ) = H ( n ) + µ e ( n ) X ∗ ( n ).
In the implementation of the complex LMS a special attention should be

paid to the transpose operation of the complex-valued matrices and vectors.
There are two ways of transposing the complex-valued matrix or vector.
The standard transpose only X T (n) and the Hermitian transpose X H (n) =
P T Q∗
X (n) . In the case of Hermitian transpose, in addition to the standard
transpose, each element is conjugated as well. In the complex LMS the
output signal is calculated using
y(n) = H T (n)X(n) = X T (n)H(n)
where (·) T is standard transpose.

It is important to note that the complex LMS is used in adaptive
antenna arrays in subsection 8.5.5.
8.7 RLS ALGORITHM
The main drawback of the LMS algorithm is in stochastic approximation

of the expected value of error function gradient by its instantaneous value
e(n)X(n). This is the reason why, in some applications, there is a need for
better gradient approximation, resulting in better algorithm convergence.
Of course, this requirement will lead to the increased computational com-
plexity. One such algorithm is the recursive least square (RLS) algorithm.
The basic idea is in better gradient approximation, keeping the number of
arithmetic operation as low as possible.
The error
e (i | n ) = d (i ) − y (i | n ) = d (i ) − X T (i ) H ( n )
is defined as an error that would be obtained in the ith instant if we would

use the coefficients from the nth instant. At the considered instant n we
have n of such errors e(i |n), i = 1, 2, . . . , n. The total error can be calculated
as a sum of squared values of these errors. It also important to include a
forgetting factor which will weight the most recent errors with higher values
that the older errors. An obvious way to define such an error function is
n
e(n) = ∑ λ n −i e2 (i | n )
i =1
where λ is the forgetting factor, a positive number smaller than 1. In practice

values 0.95 < λ < 0.995 are used.
Now we can find the adaptive system coefficients H(n) producing
minimal error function e(n). They follow from
n
∂e(n)
= −2 ∑ λn−i e(i |n)X(i ) = 0.
∂H(n) i =1
By replacing e(i |n) we get
n
∑ λn−i (d(i) − XT (i)H(n))X(i) = 0
i =1
or
n n
∑ λ n −i d (i ) X (i ) = ∑ λ n −i X (i ) X T (i ) H ( n )
i =1 i =1
r̃dx (n) = R̃(n)H(n).
This solution is similar as in the optimal filter case. The difference is that
the cross-correlation vector r̃dx (n) and the autocorrelation matrix R̃(n) are
obtained by a weighted averaging
n
r̃dx (n) = ∑ λ n −i d (i ) X (i )
i =1
n
R̃(n) = ∑ λ n − i X ( i ) X T ( i ).
i =1
The coefficient values are
H(n) = R̃−1 (n)r̃dx (n).
Relation between the coefficients at instant n with the coefficients at instant

n − 1 will be obtained from
H(n) = R̃−1 (n)r̃dx (n)

H(n − 1) = R̃−1 (n − 1)r̃dx (n − 1).
In order to find the relation between H(n) and H(n − 1) we have to find a
relation between R̃−1 (n) and R̃−1 (n − 1) and between r̃dx (n) and r̃dx (n − 1).
By definition
n
R̃(n) = ∑ λ n −i X (i ) X T (i )
i =1
n −1
=λ ∑ λ(n−1)−i X(i)XT (i) + X(n)XT (n) = λR̃(n − 1) + X(n)XT (n).
i =1
The inverse matrix R̃−1 (n) relation is need for a recursion. Using the matrix
inversion formula for
A = B + ab T ,
where A and B are square matrices of order N, and a and b are vector
columns with N elements, we have
A −1 = B −1 − B −1 a (1 + b T B −1 a ) −1 b T B −1 .
Note that (1 + b T B−1 a)−1 is a scalar. Applying this formula on R̃(n) =

λR̃(n − 1) + X(n)X T (n) we get
1 −1
R̃−1 (n) = R̃ (n − 1)
λ
* + −1
1 1 1
− R̃−1 (n − 1)X(n) 1 + X T (n) R̃−1 (n − 1)X(n) X T (n) R̃−1 (n − 1).
λ λ λ
Using the notation
µ(n) = X T (n)R̃−1 (n − 1)X(n)
it follows
1 −1 1
R̃−1 (n) = R̃ (n − 1) − R̃−1 (n − 1)X(n)X T (n)R̃−1 (n − 1).
λ λ(λ + µ(n))
Denoting the inverse matrix R̃−1 (n) by C(n)
C(n) = R̃−1 (n)
and introducing
C ( n − 1) X ( n )
g(n) =
λ + µ(n)
we get
µ ( n ) = X T ( n ) C ( n − 1) X ( n )
1 1
C ( n ) = C ( n − 1 ) − g ( n ) X T ( n ) C ( n − 1 ).
λ λ
The relation between vectors r̃dx (n) and r̃dx (n − 1) is obtained from
n
r̃dx (n) = ∑ λ n −i d (i ) X (i )
i =1
n −1
=λ ∑ λ(n−1)−i d(i)X(i) + X(n)d(n) = λr̃dx (n − 1) + X(n)d(n).
i =1
Now we can write

H(n) = R̃−1 (n)r̃dx (n) = C(n)r̃dx (n) =
1B C
= C(n − 1) − g(n)X T (n)C(n − 1) (λr̃dx (n − 1) + X(n)d(n)) =
λ
= C(n − 1)r̃dx (n − 1) − g(n)X T (n)C(n − 1)r̃dx (n − 1)+
1 1
+ C ( n − 1) X ( n ) d ( n ) − g ( n ) X T ( n ) C ( n − 1) X ( n ) d ( n )
λ λ
or
H ( n ) = H ( n − 1) − g ( n ) X T ( n ) H ( n − 1)
1 1
+ g(n)(λ + µ(n))d(n) − g(n)µ(n)d(n)
λ λ
= H ( n − 1) − g ( n ) X T ( n ) H ( n − 1) + g ( n ) d ( n )
B C
= H ( n − 1) + g ( n ) d ( n ) − X T ( n ) H ( n − 1) .
Finally the RLS formula is obtained as

H ( n ) = H ( n − 1) + g ( n ) e ( n | n − 1 ).
In the initial iteration H(0) = 0 is used, while for the initial matrix C(0) =
R−1 (0) a matrix δI is used, where I is an identity matrix and δ ≫ 1.
Example 8.25. Consider the system from Example 8.15 (page 465). The simulations
will be repeated using the RLS with λ = 0.99.
⋆Results obtained using the RLS algorithm are presented in Figure
8.30. The convergence is faster than in the case of any LMS algorithm variant
in the previous examples. In about 10 iterations the coefficients achieve their
exact values. In the stationary state the variation of coefficients is small.
1 0
coefficients h (n) 1
k
0.5
2
0 4 5 6 7 8 9
0 20 40 60 80 100 120 140 160 180 200

time index n
20
10
error signal e(n)
-10
-20
0 20 40 60 80 100 120 140 160 180 200
time index n
Figure 8.30 RLS algorithm example.
8.8 ADAPTIVE RECURSIVE SYSTEMS
For recursive systems (with infinite impulse response) value of the output
signal at the nth instant depends on the input signal at the nth and previous
N − 1 instants, x (n), x (n − 1), x (n − 2), . . . ,x (n − N + 1). Output signal
depends also on the previous output signal values y(n − 1), y(n − 2),. . . ,
y ( n − L ),
N −1 L
y(n) = ∑ ak (n) x (n − k ) + ∑ bk ( n ) y ( n − k )
k =0 k =1
The vector notations
D ET
X ( n ) = x ( n ) x ( n − 1) . . . x ( n − N + 1)
D ET
Y ( n ) = y ( n − 1) y ( n − 2) . . . y ( n − L )
D ET
A ( n ) = a0 ( n ) a1 ( n ) ... a N −1 ( n )
D ET
B(n) = b1 (n) b2 (n) ... b L (n)
will be used along with their generalized forms

- .
X(n)
U(n) =
Y(n)
- .
A(n)
W(n) = .
B(n)
The output signal can now be written as
y ( n ) = W T ( n ) U ( n ).
An adaptive system should iterate coefficients using

µ
W ( n + 1) = W ( n ) − grad(e2 (n)),
2
where the error signal is
e ( n ) = d ( n ) − y ( n ).
The gradient vector of the error function can be written as
grad(e2 (n)) = −2e(n)G(n),
where G(n) denotes a vector whose elements are

@ A
G(n) = ∂y∂a(n) ∂y∂a(n) . . . ∂a
∂y(n) ∂y(n)
∂b ...
∂y(n)
∂b L
.
0 1 N −1 1
The adaptation rule is now
W ( n + 1 ) = W ( n ) + µ e ( n ) G ( n ).
Derivatives of the output signal with respect to al and bl are
L
∂y(n)
αl (n) = = x ( n − l ) + ∑ bk ( n ) α l ( n − k )
∂al k =1
L
∂y(n)
β l (n) = = y ( n − l ) + ∑ bk ( n ) β l ( n − k ) .
∂bl k =1
In a vector notation
L
G(n) = U(n) + ∑ bk ( n ) G ( n − k ) .
k =1
Generalization of this algorithm is obtained if different steps µ are

used for different coordinates of the error vector. Then instead of step µ
a diagonal matrix of steps is used
M = diag(µ1 , µ2 , . . . , µ N + L ).
Special attention in the adaptive recursive systems has to be paid to

the system stability. It requires additional constraints on coefficients bk (n).
Example 8.26. Consider identification of system from Example 8.13 (page 462).
Here this system will be identified by using an adaptive recursive system
with N = 2 and L = 2. Step µ for the coefficients in the numerator of the
transfer function is 0.025 while its value is 0.005 for the coefficients in de-
nominator. Other parameters are the same as in Example 8.13.
⋆The simulation results are presented in Figure 8.31.
8.9 FROM THE LMS ALGORITHM TO THE KALMAN FILTERS
Consider the problem of unknown system identification, where the

unknown system has the same input signal x (n) as the adaptive system,
and the output of the unknown system is used as a reference signal d(n) =
X T (n)H∗ + ν(n) in the adaptive system. Here ν(n) denotes zero-mean Gaus-
sian measurement noise, with variance σν2 , and it is assumed that it is not
correlated with
D otherE variables. It has been shown that the cost function
J MSE (n) = E e2 (n) can be used to define the deviation of the error signal
from the ideal case. Minimization of this function provides the optimal sys-
tem parameters in form of the Wiener optimal filter. It is also proved that
LMS algorithm converges "in mean" toward the optimal system coefficient
values H∗ . In the unknown system identification framework, the optimal
coefficient values are equal to the coefficients of the uknown system. In each
_____________________________________________________
This Section presents analysis from: D. P. Mandic, S. Kanna and A. G. Constantinides, "On
the Intrinsic Relationship Between the Least Mean Square (LMS) and Kalman Filters", IEEE
Signal Processing Magazine, preprint, Nov. 2015. Adapted for this book by M. Brajović.
1.5
coefficients a (n) and b (n)
a (n)
0
1
k
0.5 b (n)
2
k
0 b (n)
1
-0.5
-1 a (n)
1
-1.5
0 20 40 60 80 100 120 140 160 180 200
time index n
3
2
error signal e(n)
1
0
-1
-2
-3
0 20 40 60 80 100 120 140 160 180 200
time index n
Figure 8.31 Identification of an unknown system (from Example 8.26) using the adaptive
recursive system.
time instant the adaptive system coefficients are changed following the rule
H ( n + 1) = H ( n ) + µ ( n ) e ( n ) X ( n ). (8.22)
by changing previous values of system coefficients in the direction of input

signal vector X(n). Since the LMS algorithm employs stochastic gradient
descent to minimize the cost function J MSE , it performs locally optimal
steps, but not the globally optimal shortest path to the solution, which
especially slows the convergence of algorithm in the case of correlated data.
The step µ(n) which can be in general time-dependent (class of variable
step LMS algorithms) controls the magnitude of adaptation steps, but not
the direction. In order to be able to follow the shortest path to the optimal
solution, i.e. to control the direction among the amplitude of adaptation
steps, the scalar step size µ(n) can be replaced by a positive definite matrix
G(n), which introduces more degrees of freedom in adaptation steps. This
is the first step towards Kalman filters, and (8.22) now becomes
H ( n + 1 ) = H ( n ) + G ( n ) e ( n ) X ( n ) = H ( n ) + g ( n ) e ( n ). (8.23)
Previous recursion is also known as generalized LMS algorithm. Since

the unknown system identification framework is considered, instead of er-
ror e(n) = d(n) − y(n) = d(n) − X T (n)H(n) the weight error vector defined
as the deviation of adaptive coefficients H(n) from the optimal coefficients
H∗
Ĥ(n) = H∗ − H(n) (8.24)
can be introduced, and based on it, we can define a measure of how closely
adaptive system coefficients H(n) approach the optimal solution H∗ . This
measure is the mean square deviation (MSD) and it is given with
MW W2 N
J MSD (n) = E WĤ(n)W (8.25)
Note that it is assumed that the unknown system is deterministic and non-
stationary. Since the weight error vector can be related with the system
output error e(n) with:
e(n) = X T (n)H∗ + ν(n) − X T (n)H(n) = X T (n)Ĥ(n) + ν(n), (8.26)
a relation between J MSE and J MSD can be found indicating that the min-
imization of MSD also corresponds to the minimization of MSE. For the
simplicity of derivation we will assume that X(n) is deterministic which is
a common assumption in Kalman filtering literature, although it is usually
treated as a zero-mean process with autocorrelation matrix R in the con-
text of adaptive
R systems.
S If we introduce the weight error covariance matrix
P(n) = E Ĥ(n)Ĥ T (n) , in order to perform the minimization of J MSD , start-
ing from (8.23) a recursive relation for the matrix P(n) is established
H∗ − H(n + 1) = H∗ − H(n) − G(n)X(n)(X T (n)Ĥ(n) + ν(n))

Ĥ(n + 1) = Ĥ(n) − g(n)X T (n)Ĥ(n) − g(n)ν(n)
B C
Ĥ(n + 1)Ĥ T (n + 1) = Ĥ(n) − g(n)X T (n)Ĥ(n) − g(n)ν(n)
B CT
Ĥ(n) − g(n)X T (n)Ĥ(n) − g(n)ν(n)
B C
P(n + 1)=P(n)− P(n)X(n)g T (n) + g(n)X T (n)P(n)
B C
+ g(n)g T (n) X T (n)P(n)X(n) + ν(n) .
By taking expectations of both sides,

R previously multiplied
S with their
transposes, and having in mind that tr P(n)X(n)g T (n) = g T (n)P(n)X(n)
this leads us to the MSD recursion of the form
J MSD (n + 1) = J MSD (n) − 2g T (n)P(n)X(n) + ∥g(n)∥2 (X T (n)P(n)X(n) + σν2 ).
The optimal learning gain vector g(n) which provides the control over
both direction and amplitude of adaptation steps in (8.23) is obtained by
solving ∂J MSD (n + 1)/∂g(n) = 0 as
P(n)X(n)
g(n) = G(n)e(n) = , (8.27)
X T (n)P(n)X(n) + σν2
which is known as the Kalman gain. Besides the calculation of (8.27), the
Kalman filter which estimates the optimal time-invariant and deterministic
coefficients for each time instant also includes the coefficients adjustment
H(n + 1) = H(n) + g(n)(d(n) − X T (n)H(n)),
as well as the weight error covariance matrix update
P ( n + 1 ) = P ( n ) − g ( n ) X T ( n ) P ( n ). (8.28)
Note that previous algorithm steps for σν2 = 1 can be related with the
RLS algorithm equations.
A generalization of the previous approach assumes time-varying and
stochastic weight vector H∗ (n)
H ∗ ( n + 1 ) = F ( n ) H ∗ ( n ) + q ( n ), (8.29)
T ∗
d(n) = X (n)H (n) + ν(n) (8.30)
with q(n) being

R a zero-mean
S Gaussian stochastic process with covariance
matrix Q = E q(n)q T (n) , and F(n) being a known matrix which describes
the system changes over time (state-transition matrix). It is assumed that the
measurement noise ν(n) is also uncorrelated with q(n). In the framework of
the general Kalman filter, the coefficient vector is updated using the current
state estimate denoted with H(n|n), while the prediction of its next state
is denoted with H(n + 1|n). The prediction step is needed for tracking the
time-varying error surface. The coefficients are updated by
H(n|n) = H(n|n − 1) + g(n)(d(n) − X T (n)H(n|n)) (8.31)

while the coefficients prediction is obtained with
H ( n + 1 | n ) = F ( n ) H ( n | n ). (8.32)
Note that the same definition of the weight error vector Ĥ(n|n) =
H∗ − H(n|n) holds, as well as for weight error covariance matrix
M N
P(n|n) = E Ĥ(n|n)Ĥ T (n|n) .
The weight error covariance matrix is updated in the same manner as for
the time-invariant deterministic case
P ( n | n ) = P ( n | n − 1 ) − g ( n ) X T ( n ) P ( n | n − 1 ), (8.33)
with the respect to the new index notation. The general Kalman filter also
includes the prediction step of weight error matrix which easily follows
from its definition
M N
P(n + 1|n) = E Ĥ(n + 1|n)Ĥ T (n + 1|n) = F(n)P(n|n)F T (n) + Q. (8.34)
Similarly to the time-invariant deterministic case, the Kalman gain which

minimizes MSD is obtained in the following form
P ( n | n − 1) X ( n )
g(n) = G(n)e(n) = . (8.35)
X T (n)P(n|n − 1)X(n) + σν2
Example 8.27. Consider the problem of identification of unknown time-invariant

deterministic system with two coefficients h0 = 3 and h1 = 4 using the stan-
dard LMS algorithm and Kalman filter (for stationary system identification),
with N = 2. The input signal is colored noise x (n) = 5w(n) + 3.4w(n − 1) +
w(n − 2), where w(n) is a zero mean white noise with variance σw2 = 1. The
step µ = 0.0005 is used for the LMS algorithm. Show the convergence paths
on the MSE contour plot. After how many iterations the Kalman filter ap-
proaches the optimal solution?
⋆The convergence paths on the MSE contour are shown in Fig. 8.32.
Numbers on the Kalman filter path indicate that the optimal solution is
obtained after only two iterations.
ε(h0,h1)
6
LMS convergence path
Kalman filter convergence path
1 (h*,h*)
4 0 1
h1
00
0 1 2 3 4 5 6
h0
Figure 8.32 Convergence paths of the LMS algorithm and Kalman filter in the problem of
identification of unknown time-invariant deterministic system. Contour lines are the projec-
tions of the MSE surface on the coefficients plane.
8.10 NEURAL NETWORKS
Artificial neural networks, or just neural networks, represent a simplified

mathematical model of biological systems. In such systems a distributed
parallel data processing is performed, in contrast to the common engineer-
ing systems that are designed for a sequential data processing. Common
computer systems are based on well defined algorithms that are executed
in a sequential order, while in the neural networks a learning period is re-
quired to achieve their satisfactory response to the input data. Correspon-
dence with biological systems, that also require learning, is evident. Com-
monly, a trained network continues to learn and adapt to the new situations
during the exploitation. It means that the process of learning does not end
inputs neural outputs

network
Figure 8.33 Neural network illustration.
with the training period. It continues through the whole functioning of neu-
ral network.
Neural network can be defined as an artificial cell system capable of
accepting, memorizing and applying empirical knowledge. The knowledge
here means that the neural network can respond to an input from the en-
vironment in an appropriate way. Neural network is connected to the envi-
ronment in two ways: through the inputs where the environment influences
the network and through the outputs where the network responses to envi-
ronment, as it is illustrated in Figure 8.33.
The basic element in a neural network is neuron. It is the elementary
unit for a distributed signal processing in a neural network. A full function-
ality of neural networks is achieved using large number of interconnected
neurons. Connections among neurons are one-directional (the outputs from
one neuron can be used as inputs to the other neuron). They are called
synapses, in analogy with the biological systems.
Possible applications of neural networks include almost all aspects of
modern life, text and speech recognition, optimization of a communication
channel, financial forecasts, detection of a fraud credit card usage, are just a
few examples.
Of course, there are many situations when a usage of neural networks
is not justified. In many cases our knowledge about the system, that we
want to control or observe, is sufficient and complete so the problem can be
solved using classical algorithms, with sequential processing on common
computers.
An ideal system for neural networks realization would use indepen-
dent systems for hardware realization of each neuron. Then the distributed
processing would be most efficient. In the cases of monoprocessor comput-
ers, high efficiency is achieved by using very fast sequential data processing.
Typical examples are computer programs for recognition of a scanned text.
x (n) x (n)
1 1 network
function
x (n) x (n)
2 2
y(n) u(n) y(n)
f
...
...
inputs output
activation
function
x (n) x (n)
N N
(a) (b)
Figure 8.34 Neuron schematic symbol (a) and the model based on network and activation
functions (b).
8.10.1 Neuron
The first step in a neuron design is to define its inputs and outputs. In
biological systems the input and output signals to a neuron are electric
potential that can be modelled by real numbers. The same principle is used
in artificial neurons. Illustration of a neuron is given in Figure 8.34(a) for
the case when it has N inputs ( x1 (n), x2 (n), . . . , x N (n)) and one output y(n).
Index n may be a time index, but it can also be understood as a cardinal
number that identifies the input and output index of a neuron.
Neuron represents and algorithm that transforms N input data into
one output signal. It is common to split this algorithm into two parts: 1)
combinatorial process that transforms N input data to one output value
u(n) and 2) the process that produces output signals y(n) based on the
value of u(n). This two-phase model of a neuron is presented in Figure
8.34(b). The algorithm/rule to produce u(n) is called the network function,
while the second part which determines the output value is the activation
function.
Neuron knowledge is accumulated and contained in the way how the
input data are combined, i.e., in the network function.
8.10.2 Network Function
The basic task of the network function is to combine the input data. The
simplest way of combining N input signals is in their linear weighed com-
bination with coefficients wi , i = 1, 2, ..., N. This is a linear network function.
Because of it simplicity, this type of function is commonly used in neurons.
Examples of network function are given in the table.
Name Network function

N
Linear form u(n) = ∑ wi x i ( n ) + θ
i =1
N N
Linear form (II order) u(n) = ∑∑ wik xi (n) xk (n) + θ
i =1k =1
N
w
Product form u(n) = ∏ xi i ( n )
i =1
The values of network function commonly depends not only on the

input data, but also on the current state of the neuron. This state is modeled
by a real variable θ, called bias or threshold. Neuron model with a nonzero
bias and a linear activation function may be simplified if we introduce an
additional input x0 in addition to the existing N inputs. It will be assumed
that this additional input is always set to x0 = 1. Then the bias can be
modelled with a coefficient w0 and the neuron considered as a zero-bias
neuron. This kind of simplification will be used in the presentation that
follows.
The knowledge (as a way to transform input data to output signal) is
contained in the values of the coefficients wi (or wik ) of the network function.
8.10.3 Activation Function
The activation function transform the output value from the network func-
tion to an acceptable output value. A common requirement is that the out-
put values have limited range. Thus, most of the activation functions have
a bounded interval of real numbers as its codomain, like for example, [0, 1]
or [−1, 1] or a set of binary digits. Forms of commonly used activation func-
tions are presented in table. The most important functions from this set are
the unipolar threshold function and the unipolar sigmoid. Some of the acti-
vation functions are presented in Figure 8.35 as well.
f(u) f(u) f(u)
u u u
unipolar sigmoid unipolar threshold function Gaussian function
f(u) f(u) f(u)
u u u
bipolar sigmoid bipolar threshold function limiter
Figure 8.35 Neuron activation functions.
Function Formula
Linear f (u) = u
⎧
⎨ 1 za u>1
Linear with a limiter f (u) = u za −1 ≤ u ≤ 1
⎩
! −1 za u < −1
1 za u > 0
Threshold function (unipolar) f (u) =
! 0 za u < 0
1 za u > 0
Threshold function (bipolar) f (u) =
−1 za u < 0
1
Sigmoid (unipolar) f (u) =
1 + exp(−u)
2
Sigmoid (bipolar) f (u) = −1
1 + exp(−2u)
Inverse tangent function f (u) = π2 arctan
B (u) C
( u − m )2
Gauss function f (u) = exp σ2
In literature hyperbole tangent function is used as well. It is equal the

bipolar sigmoid.
x1(n)
x1(n) y(n)
x2(n) y(n)
x (n)
2
(a) (b)
Figure 8.36 Neural network topology: acyclic (a) and cyclic (b).
8.10.4 Neural Network Topology
A number of neurons is connected to each other and to input signal within a

neural network in order to get a network output. Configuration of the signal
flow can be presented by a graph where the neurons are nodes of the graph
and the signal flow is represented by the directed edges. A special category
are the input edges (its start is not a neuron but a node that represents
acquisition of data outside of the neural network - input node) and the
output edges (its end is not a neuron but a node that represents output
information forwarded to the environment - output node).
From the point of view of topology the neural networks can be classi-
fied in various ways. If a neural network contains a closed loop in the graph
then it is a cyclic (recurrent) neural network. The network containing no
closed loop is an acyclic (feed-forward) neural network. Examples of cyclic
and acyclic networks are presented in Figure 8.36. Cyclic or recursive neu-
ral networks are dynamic nonlinear systems (with memory) whose design,
learning, and implementation is complex, mainly because of the nonlinear
nature of individual neuron activation functions . In acyclic networks there
are no closed loops so that the input information pass to the network output
through a finite number of neurons.
A special category of the acyclic neural networks are layer networks,
where the neurons can be divided into disjunctive subsets (layers). The
output data from one layer are the input data to other layer. Neurons
from the first layer get the information from the network input, while the
neurons from the last layer produce output information from the network.
The simplest cases are neural networks with one or two layers of neurons. In
multilayer configurations it is assumed that the inputs to the mth layer are
the outputs from the (m − 1)th layer of neurons. This approach simplifies
y1(n)
x (n)
1
y (n)
2
x2(n)
input y (n)
layer 3
output
layer
Figure 8.37 Single layer network example.
x1(n)
y1(n)
x2(n) output
layer
input hidden
layer layer II
hidden
layer I
hidden
layer III
Figure 8.38 Four layer neural network example.
mathematical analysis of the neural networks. In situation when the state of

one neuron is not influenced by all input values to that layer, the inputs
without influence are modelled with zero weighting coefficients in the
network function of this neuron.
In layer networks it is common to introduce the zeroth (input) layer
of neurons where there is no data processing. Its function is to forward the
input data to the first layer. The last layer is the output layer. An example of
one-layer neural network is presented in Figure 8.37. An example of neural
network with four layers is shown in Figure 8.38. This network consists of
three hidden layers (I, II, and III) with 3, 2 and 4 neurons, respectively, and
the output layer with one neuron.
8.10.5 Network with Supervised Learning
The network adapts for desired functioning trough a process of training.

The training is implemented using the following algorithm:
1. Data for the network training are acquired. This data consists of
the input-output pairs. The output data are assumed, estimated or
obtained trough experiments. This set of training data pairs if finite.
Denote the number of available input-output pairs by K.
2. The network is initiated, commonly by using random parameters of
neurons (if an a priori information about the range of their values does
not exist). After the initialization, the iterative training procedure is
implemented as follows:
(a) One input-output pair of data is considered. The output of the

neural network is calculated. The output value is compared with
the desired (given) output value. If the output from the neural
network is equal to the desired output value (or sufficiently
close to this value) then no correction in the network is done.
Otherwise when the result of comparison is not satisfactory, the
parameters of neural network are corrected to get a better result
in the considered case.
(b) Next pair of the input-output data is considered. The calculation,
comparison, and correction process is repeated.
(c) This cycle of training ends when all data available for training are
used. This whole cycle is called one epoch.
(d) The result achieved by the neural network in the previous epoch
of training is analyzed. If the output of the neural network has
been satisfactory for each individual pair of the data then the
training is finished. If the result is not satisfactory, then another
epoch of training is needed. Go back to 2a and repeat all previous
steps.
3. Neural network is ready for use. A testing of the trained neural

network can now be done. A common way of testing is in omitting
some pairs of the available input-output data in the training procedure
and in using them after the training process is completed, to test the
neural network accuracy.
The iterative algorithm for training of neural network (steps 2a–2d)

does not necessary converges. Algorithm is usually implemented by im-
posing a maximal number of epochs. In the case that the result is not sat-
isfactory (neural network does not produce accurate results after training)
the training process may be repeated with new initialization of the network
parameters. This is one of the reasons why the random initial values of the
parameters are used.
One of the main problems in the neural networks training is the
way how to modify the parameters when we are not satisfied with the
results. Several networks will be analyzed next, where this problem will
be addressed. In some cases just a small random correction of parameters
can improve results, when the process of parameter changes is locked in a
local minimum.
8.10.6 One-Layer Network with Binary Output - Perceptron
Perceptron as the oldest simple form of neural networks. Perceptron has

to establish if the input data have a desired property or not. It produces a
binary output. For example, the input data may be scanned pixels of one
letter in the text. From neural network we expect a simple answer is the
scanned letter, for example letter „A” or not. Since the output is a logical
variable, an obvious choice for the neuron activation function is a function
with binary output (bipolar or unipolar threshold function). Assume that
unipolar function is used as the activation function
!
1 za u > 0
f (u) = .
0 za u < 0
The neuron network function, in the case of perceptron, is a linear form

N
u(n) = ∑ wk xk (n ) (8.36)
k =1
where it has been assumed that the neuron has N input data. The weighting
coefficients wk represent „knowledge” that the network should get through
the training procedure. This knowledge will be then used in real situations.
The vector notation is
⎡ ⎤ ⎡ ⎤
x1 ( n ) w1
⎢ x2 ( n ) ⎥ ⎢ w2 ⎥
⎢ ⎥ ⎢ ⎥
X(n) = ⎢ . ⎥ W=⎢ . ⎥ .
⎣ .. ⎦ ⎣ .. ⎦
x N ( n ) N ×1 w N N ×1
The network function can be written as
u(n) = W T X(n)
= X T (n) W.
The neuron output is

B C
y(n) = f (u(n)) = f W T X(n) .
Perceptron basic form consists of one neuron only. In topological sense

it is acyclic network. It is one layer neural network. In a similar way it
is possible to define perceptron with more than one neuron. In that case
we have a neural network with M output signals. Network can decide if
the input data contain some out of M properties that we are interested in.
An example of such a network would be a perceptron which, based on the
pixels of a scanned letter, decides which of the letters „A”, „B” or „C” was
scanned. Note than in this case we may obtain more than one 1 at the output
(network, for example has recognized that the input data correspond to the
letter „A” and letter „B”). In the considered case it means that the network
was not able to recognize the letter. However, in some cases the output data
can be defined in such a way that more than one property are satisfied for
one set of input data. For example, in letter recognition, the results may be:
„scanned letter contains horizontal line”, „scanned letter contains vertical
line” and „scanned letter contains oblique” and to train the neural network
to recognize these properties of a scanned letter.
The network output can be written as a column vector with M ele-
ments, ⎡ ⎤
y1 ( n )
⎢ y2 ( n ) ⎥
⎢ ⎥
Y(n) = ⎢ . ⎥ .
⎣ .. ⎦
y M (n) M ×1
The weighting coefficients of individual neurons Wl , l = 1, 2, . . . , M can be
written in a matrix form
D E
W = W1 W2 · · · W M N × M
⎡ ⎤
w11 w21 · · · w M1
⎢ w12 w22 · · · w M2 ⎥
⎢ ⎥
=⎢ . .. .. .. ⎥ .
⎣ .. . . . ⎦
w1N w2N ··· w MN N×M
The relation between output and input can be written as

B C
Y(n) = f W T X(n) .
Consider again simple case of perceptron with one neuron. In the

training process, when for a given input X(n) the network produces output
y(n), its has to be compared with desired value d(n). Possible cases are
y(n) = d(n), when no correction of the network coefficients W is done, and
y(n) ̸= d(n) when the coefficients are modified as
Wnew = Wold + ∆W.
The correction ∆W should be done in such a way that it increases the

possibility to get a desired output for the considered input data. In this case
the output may take only one of two binary values: 0 for u(n) = W T X(n) < 0
or 1 for u(n) = W T X(n) > 0. Assume that the desired value of the output
is 1 and that we obtained 0. It means W T X(n) < 0 holds, while it should
be W T X(n) > 0. The value of W T X(n) should be increased to increase the
possibility of getting the desired output. In the case that the desired output
is 0 and the output from the network is 1, using a similar reasoning we can
conclude that W T X(n) has to be decreased. A way to implement the desired
coefficients modification is
Wnew = Wold + µ(d(n) − y(n))X(n).
In the case when d(n) = 1 and y(n) = 0 it means that Wnew = Wold + µX(n)
or
T T
Wnew X(n) = Wold X(n) + µX T (n)X(n)
T
= Wold X(n) + µ||X(n)||22 ,
where ||X(n)||22 is the squared norm two of vector X(n) (sum of its squared
elements). The value of W T X(n) is increased for µ||X(n)||22 , what was the
aim. If d(n) = 0 and y(n) = 1 then Wnew = Wold − µX(n) holds, meaning
that W T X(n) is reduced for µ||X(n)||22 .
The coefficient µ is the learning coefficient. It is positive. The choice
of parameter µ value is of great importance for the rate of convergence
and learning process of the network. Larger values may reduce the learning
period, but also may influence the convergence of the training process.
Example 8.28. Consider a one-neuron neural network. Assume that the activation
function of the neuron is unipolar threshold function and that the neuron
is biased. The network has three inputs and one output. Set of data for the
neural network training is
⎡ ⎤
1 1 0 0 0 1
X = ⎣1 0 1 1 0 0⎦
0 1 1 0 1 0
D E
D= 1 1 0 0 0 0
where matrix X contains the input data and vector D consists of desired
outputs from the neural network for the considered input data values. Train
the neural network with µ = 0.5.
⋆Since the neuron is biased one more input will be introduced. Its
input value is always 1. After this modification the matrix of input data is
⎡ ⎤
1 1 1 1 1 1
⎢1 1 0 0 0 1⎥
X=⎢ ⎣1 0 1 1 0 0⎦ .
⎥
0 1 1 0 1 0
Initial values of weighting coefficients are random, for example,
⎡ ⎤ ⎡ ⎤
w0 −1
⎢ w1 ⎥ ⎢ 1 ⎥
W=⎢ ⎥ ⎢ ⎥
⎣ w2 ⎦ = ⎣ 1 ⎦ .
w3 0
Now we can start the first epoch of training process. We will use
all input-output data pairs and calculate the output y(n) from the neural
network. The output y(n) will be compared with the desired value d(n) and
the coefficients W will be appropriately modified for each pair of data.
For the first par of data we have
⎡ ⎤
1
B C D E ⎢1⎥
T
y (1) = f W X (1) = f ( −1 1 1 0 ⎣ ⎥ ⎢ ) = 1.
1⎦
0
Since d(1) = 1 the error d(n) − y(n) is 0 and the coefficients are not modified.
For the second pair of data
B C
y(2) = f W T X(2) = 0.
The desired value is d(2) = 1. Since the error is not zero, the coefficients
should be modified as
⎡ ⎤ ⎡ ⎤ ⎡ ⎤
−1 1 −0.5
⎢ 1 ⎥ ⎢1⎥ ⎢ 1.5 ⎥
Wnew = Wold + µ(d(2) − y(2))X(2) = ⎢ ⎥ ⎢ ⎥ ⎢
⎣ 1 ⎦ + 0.5 ⎣0⎦ = ⎣ 1 ⎦ .
⎥
0 1 0.5
Next pair of input-output data is used. After all data pairs are used, the first
epoch of training is finished. Nonzero error appeared in three out of six data
pairs. The final value of the coefficients, after the first training epoch, is
D ET
Wepoch 1 = −1.5 1 0.5 0 .
With this initial value, the second epoch of training is completed, using
the same input-output pairs of data. After the second epoch nonzero error
appeared two times. The final values of the coefficients, after the second
epoch, are
D ET
Wepoch 2 = −1.5 1 1 0 .
The process is continued in the third epoch. In the fifth epoch we came
to the situation that the neural network has made no error. It means that the
training is completed and that more epochs are not needed. The final values
of the coefficients are
D ET
W = −1.5 1.5 0.5 0.5 .
8.10.7 One-Layer Neural Network with Continuous Output
In this kind of neural networks the output signal is not binary, but a real
number (usually within the interval from 0 to 1). It may be interpreted as
a probability that the input data contain or do not contain certain property.
In general any interval of real numbers can be a codomain of the output
function. The main difference from the perceptron is that we do not require
that the neural network achieves an exact precision y(n) − d(n) = 0. In this
case the aim to get a small error in the processing of input results.
Since the output variable is continuous, the activation function should
have such a property as well. Consider, for example, the unipolar sigmoid
activation function
1
f (u) = .
1 + e−u
A simple way to quantify the difference of the output signal from the
desired signal is to use the square error
1
ε(n) = (d(n) − y(n))2
2
where constant 1/2 is introduced to simplify the notation in the period of

the neural network training process. The goal is to minimize the square er-
ror. This minimization can be done in various ways: using steepest descent
method, conjugate gradient method, Newton method, are some of these

methods. We will use the steepest descend method in the correction of the
neural network coefficients.
Consider a network with N inputs. The input data vector is X(n) and
the desired output is d(n). The network output signal is obtained as
( )
B C n
T
y(n) = f (u(n)) = f W X(n) = f ∑ wk xk (n)
k =1
with the square error

( ( ))2
n
1 1
ε(n) = (d(n) − y(n))2 = d(n) − f ∑ wk x k ( n ) . (8.37)
2 2 k =1
This error is a function of the network coefficients wk . Using the steepest

descend method the coefficients modification is done as
∂ε(n)
wk,(new) = wk,(old) − µ
∂wk
or
∂ε(n)
Wnew = Wold − µ
∂W
where ∂ε(n)/∂W is the gradient of error function.
Derivatives can be calculated from (8.37) as
( )
n
∂ε(n) ∂y(n)
= −(d(n) − y(n)) = −(d(n) − y(n)) f ′ ∑ wk x k (n ) x k ( n ).
∂wk ∂wk k =1
For the unipolar sigmoid activation functions we have
d 1 −e−u 1 e−u
f ′ (u) = = − = =
du 1 + e−u (1 + e − u )2 1 + e−u 1 + e−u
* +
1
= f (u) 1 − = f (u) (1 − f (u)) .
1 + e−u
Therefore
∂ε(n)
= −(d(n) − y(n)) y(n) (1 − y(n)) xk (n),
∂wk
where ( )
n
f ∑ wk x k (n ) = y(n)
k =1
is used. The training rule (correction of the coefficients) is
wk,(new) = wk,(old) + µ(d(n) − y(n)) y(n) (1 − y(n)) xk (n)
or in vector form
Wnew = Wold + µ (d(n) − y(n)) y(n) (1 − y(n)) X(n).
In is common to denote (d(n) − y(n)) y(n) (1 − y(n)) by δn so that the

training rule can be written as
Wnew = Wold + µδn X(n).
This rule is called delta-rule. Note that the letter δ is also used for Dirac delta
pulse in some chapters of the book. These two values do not have anything
in common.
For the activation function in the form of bipolar sigmoid
2 1 − e−2u
f (u) = − 1 =
1 + e−2u 1 + e−2u
we would have
4e−2u (1 + e−2u )2 + 4e−2u − (1 + e−2u )2

f ′ (u) = =
(1 + e−2u )2 (1 + e−2u )2
(1 + 2e−2u + e−4u ) − 4e−u
=1−
(1 + e−2u )2
(1 − e−2u )2
=1− = 1 − f 2 (u)
(1 + e − u )2
and the value of δn would be
δn = (d(n) − y(n))(1 − y2 (n)).
Example 8.29. Neural network consists of one unbiased neuron with two input
signals and a sigmoid activation function. Input values are random numbers
from the interval [0, 1]. Available are K = 30 input-output pairs of data.
Training of of the neural network should be done in 30 epochs with µ = 2.
Data for network training are obtained as a set of 30 input values of x1
and x2 . They are assumed as random numbers from the interval from 0 to 1
with a uniform probability density function. For each training pair of random
numbers x1 and x2 the desired output data is calculated using the formula
1 x1 − 2x2
d= + .
2 3 + x12 + 3x22
Find the total square error after the first, second, fifth and thirtieth
epoch. What are the coefficient values at the end of training process? If the
input values x1 = 0.1 and x2 = 0.8 are applied to the network after the training
process is completed find the output value y and compare it with the desired
result d calculated using the formula.
⋆Coefficients of the neuron are w1 and w2 . With the sigmoid activation
function the coefficient corrections are
- . - . - .
w1 w x (n)
= 1 + µ (d(n) − y(n)) y(n) (1 − y(n)) 1 ,
w2 new w2 old x2 ( n )
where index n assumes values from 1 to 30 within one epoch. It denotes the
index of the input-output pair of data. The output y is calculated using
B C
y(n) = f W T X(n) = f (w1 x1 (n) + w2 x2 (n)) .
Initial coefficient values are randomly chosen.

The training process is implemented on a computer and the following
results are obtained: Total square error after the first epoch of training is
0.4266. After the second training epoch the total error is reduced to 0.1062.
The total error after the fifth epoch is 0.0336, while its value at the end of
the training process (after 30 epochs) is 0.0170. The final values of the neuron
coefficients are
w1 = 1.0455 w2 = −1.9401.
For the input data x1 = 0.1 and x2 = 0.8 we get
y = f (w1 x1 + w2 x2 ) = 0.1904
1 x1 − 2x2
d= + = 0.1957.
2 3 + x12 + 3x22
The error is small. The task for neural network in this example was to find a
complex, nonlinear relation between the input and output data.
8.10.8 Multilayer Neural Networks
The multilayer neural networks are characterized by at least one hidden

layer, the layer whose values do not appear as the network output. During
the training process available are the input-output pairs of data, while
the data for hidden layers are not known. On of the approaches to train
this kind of neural network is so called „error backpropagation learning”

method. In this method the known error at the output layer of neurons is
transformed into the error at the output of the previous neuron layer.
This algorithm will be illustrated on an example of a neural network
with two layers and one output. The considered neural network has N
inputs, M neurons in the hidden layer and one neuron in output layer. The
unipolar sigmoid is assumed as the activation function, while the network
function is a linear combination of the inputs. Consider one element from
the training data set pairs (X(n), d(n)). The outputs of the hidden layer are
denoted by U(n) and calculated as
B C
U(n) = f W T X(n) ,
where W N × M is the matrix of neuron weighting coefficients in the hidden

layer, and U(n) is a vector column with M output values of the hidden layer.
The neural network output is
B C B B CC
y(n) = f V T U(n) = f V T f W T X(n)
where V M×1 is the vector of weighting coefficients of the output neuron.

The square error is
1
ε(n) = (d(n) − y(n))2 .
2
The desired output value is d(n). Coefficients V are modified in the same
way as in the previous case with the one-layer neural network with contin-
uous output,
∂ε(n)
vm,(new) = vm,(old) − µ
∂vm
where
∂ε(n) B C
= −(d(n) − y(n)) f ′ V T U(n) um (n)
∂vm
= −(d(n) − y(n)) y(n) (1 − y(n)) um (n).
The mth element of vector U(n) is denoted by um (n).

The final modification relations are
Vnew = Vold + µ (d(n) − y(n)) y(n) (1 − y(n)) U(n)

Vnew = Vold + µδn U(n)
where δn = (d(n) − y(n)) y(n) (1 − y(n)) is the training rule.

Consider now kth neuron in the hidden layer. Coefficients of this neu-
ron are the elements of kth column of matrix W, denoted by Wk . Coefficients
of this neuron are modified as
∂ε(n)
w pk,(new) = w pk,(old) − µ
∂w pk
where
∂ε(n) B B CC B C
= −(d(n) − y(n)) f ′ V T f W T X(n) vk f ′ WkT X(n) x p (n).
∂w pk
The pth element of vector X(n) is denoted by x p (n), while the kth element
P Q
of vector V is vk . Taking into account that uk (n) = f WkT X(n) we get
∂ε(n)
= −(d(n) − y(n)) y(n) (1 − y(n)) vk [uk (n)(1 − uk (n))] x p (n).
∂w pk
Coefficients modification rule for this neuron is
w pk,(new) = w pk,(old) + µ (d(n) − y(n)) y(n) (1 − y(n)) vk [uk (n)(1 − uk (n))]

× x p (n) = w pk,(old) + µ δn2 vk [uk (n)(1 − uk (n))] x p (n)
where δn2 denotes the learning rule for the considered layer of neurons. In
vector form we can write
Wk,(new) = Wk,(old) + µ δn2 vk [uk (n)(1 − uk (n))] X(n).
This is the modification formula for all coefficients of one neuron in the
hidden layer. The modification can be generalized to all neurons in the
hidden layer
W(new) = W(old) + µ δn2 X(n) [V. ∗ U(n). ∗ (1 − U(n))] T ,
where .∗ denotes the element-by-element multiplication, while 1 is the

vector of the same dimension as U(n) whose elements are equal to 1.
The described procedure can be generalized for neural networks with
more than two layers. The basic principle is that based on the error in
one layer, the coefficients are modified in this layer and then in all layers
before the considered layer. It means that the influence of the output error
is transferred in an inverse way (backpropagated) to the correction of the
coefficients of the layers of neurons.
Example 8.30. Consider a two-layer neural network with two neurons in the
hidden layer and one neuron in the output layer. The activation function for
all neurons is the unipolar sigmoid. The task for this neural network is to find
unknown relation between the input and output data. Step µ = 5 is used in
the training process. The data for the training are formed as in Example 8.29,
i.e., as a set of K = 30 input data x1 and x2 that are uniformly distributed
random numbers from the interval 0 ≤ x1 ≤ 1, 0 ≤ x2 ≤ 1. For each training
input value of x1 and x2 the desired signal is calculated as
1 x1 − 2x2
d= + .
2 3 + x12 + 3x22
Find the total square error after 10th, 100th, and 300th epoch. What
are the coefficients of the neurons after the training process? If the values of
x1 = 0.1 and x2 = 0.8 input the trained neural network find the output y and
compare it with the desired result d.
⋆The training process is implement on a computer and the following
results are obtained: Total square error after 10 epochs of training is 0.1503.
After 100 epochs the total square error is reduced to 0.0036, while the squared
error after 300 epochs is 0.0003. The final coefficient values in the hidden and
output layers, W and V, are
- . - .
−0.2911 1.8297 −2.6173
W= V= .
3.4435 −0.6945 2.5889
For the input data x1 = 0.1 and x2 = 0.8 we get

* * - .++
T T 0.1
y= f V f W = 0.1978
0.8
1 x1 − 2x2
d= + = 0.1957.
2 3 + x12 + 3x22
The error is very small. As expected this result is better than in the case of
one-layer neural network (Example 8.29). However, the calculation process
is significantly more demanding.
8.10.9 Neural Networks with Unsupervised Learning
Consider an example of one-layer neural network with N neurons and two

input data. The input data will be here interpreted as the coordinates of
points in plane. If the input data (coordinates of points) exhibit property
of being grouped in certain regions of the two-dimensional plane (regions
are defined by straight lines passing through the origin), then we can ask
the neural network to find the group to which an arbitrary input data pair
(point) belongs. In an ideal case the number of groups (categories) is known

and equal to the number of neurons N. The training process for a neural
network reduces to the selection of the neuron with the highest output
(assume that it is the neuron with index k) and to the modification of its
coefficients using
B C
Wk,(new) = Wk,(old) + µ X(n) − Wk,(old)
After the training process we may expect that each of the neurons
recognizes one category (belonging to one group) of the input signals. If
an uncategorized input signal appears it means that the estimation of the
number of neurons is not good. It should be increased and the training
process should be continued. When two neuron adjust to the same category,
then they produce the same result and one of them can be eliminated. In this
way, we may avoid the assumption that the number of categories (groups)
or neurons N is known in advance.
Example 8.31. Consider a neural network with two input data and 3 neurons. The
task of neural network is to classify the input data in one of three categories.
Each neuron corresponds to one category. The classification decision is made
by choosing the neuron with the highest output. Activation function is a
bipolar sigmoid.
Simulate the neural network in the case when the input data belongs to
one of thee categories with equal probability. Data from the first category are
pairs of Gaussian random variables with probability density function whose
means are x̄1 = 0 and x̄2 = 4 and variances are σx21 = 4 and σx22 = 0.25. For the
data from the second category the mean values and variances of Gaussian
variables are x̄1 = 4, x̄2 = −2, σx21 = 1 and σx22 = 4. In the third category are
the input data with x̄1 = −4 and x̄2 = −2, σx21 = 1 and σx22 = 1. during the
training process the step µ = 0.5 is used.
⋆Results achieved by neural network after 10 and 100 pairs of input
data are presented in Figure 8.39. The categories are indicated with different
colors. Learning process of the neural network in the input classification of
data is fast.
8.10.10 Voting Machines
Voting machines are special forms of the application of neural networks.

Two basic forms of the voting machines are used: neural network ensemble
and mixture of experts.
Voting machines of the neural network ensemble type consists of
several neural networks which are independently designed and trained
10 input data points 100 input data points

8 8
6 6
4 4
2 2
x2
2
0 0
x
-2 -2
-4 -4
-6 -6
-8 -8
-8 -6 -4 -2 0 2 4 6 8 -8 -6 -4 -2 0 2 4 6 8
x1 x
1
Figure 8.39 Example of unsupervised training of a neural network. Input data are classified
into three categories. Regions obtained by the neural network after 10 and 100 input data, are
presented in different colors in the plane of input data.
and a control network that interprets the output data from these neural
networks. All networks are trained to solve the same kind of problem,
meaning that the same data are used as input in all of them, while the control
network decides about the final result, for example using the principle of
majority of votes.
The mixture of experts is a set of neural networks, where each of them
is trained to process one type of the input data. Control network, in this
case, has to choose one or more experts (neural networks) which are trained
for the type of data that appears the input.
Chapter 9
Time-Frequency Analysis
The Fourier transform provides a unique mapping of a signal from the

time domain to the frequency domain. The frequency domain representa-
tion provides the signal’s spectral content. Although the phase characteristic
of the Fourier transform contains information about the time distribution of
the spectral content, it is very difficult to use this information. Therefore, one
may say that the Fourier transform is practically useless for this purpose,
i.e., that the Fourier transform does not provide a time distribution of the
spectral components.
Depending on problems encountered in practice, various representa-
tions have been proposed to analyze non-stationary signals in order to pro-
vide time-varying spectral description. The field of the time-frequency sig-
nal analysis deals with these representations of non-stationary signals and
their properties. Time-frequency representations may roughly be classified
as linear, quadratic, and higher order representations.
Linear time-frequency representations exhibit linearity, i.e., the repre-
sentation of a linear combination of signals equals the linear combination
of the individual representations. From this class, the most important one
is the short-time Fourier transform (STFT) and its variations. The energetic
version of the STFT is called spectrogram. It is the most frequently used tool
in time-frequency signal analysis.
The second class of time-frequency representations are the quadratic
ones. The most interesting representations of this class are those which pro-
vide a distribution of signal energy in the time-frequency plane. They will be
referred to as distributions. The concept of a distribution is borrowed from
_________________________________________________
Authors: Ljubiša Stanković, Miloš Daković, Thayaparan Thayananthan
521
522 Time-Frequency Analysis
w(τ)
x(t)
τ
t
t
x(t+τ)w(τ)
Figure 9.1 Illustration of the signal localization in the STFT calculation.
the probability theory, although there is a fundamental difference. For ex-

ample, in time-frequency analysis, distributions may take negative values.
Other possible domains for quadratic signal representations are the ambi-
guity domain, the time-lag domain and the frequency-Doppler frequency
domain. In order to improve time-frequency representation various higher-
order distributions have been defined as well.
9.1 SHORT-TIME FOURIER TRANSFORM
The idea behind the short-time Fourier transform (STFT) is to apply the
Fourier transform to a portion of the original signal, obtained by introduc-
ing a sliding window function w(t) to localize the analyzed signal x (t). The
Fourier transform is calculated for the localized part of the signal. It pro-
duces the spectral content of the portion of the analyzed signal within the
time interval defined by the width of the window function. The STFT (a
time-frequency representation of the signal) is then obtained by sliding the
window along the signal. Illustration of the STFT calculation is presented in
Fig.9.1.
Analytic formulation of the STFT is
"∞
STFT (t, Ω) = x (t + τ ) w(τ ) e− jΩτ dτ. (9.1)
−∞
From (9.1) it is apparent that the STFT actually represents the Fourier
transform of a signal x (t), truncated by the window w(τ ) centered at
instant t (see Fig. 9.1). From the definition, it is clear that the STFT satisfies
properties inherited from the Fourier transform (e.g., linearity).
By denoting xt (τ ) = x (t + τ ) we can conclude that the STFT is the
Fourier transform of the signal xt (τ )w(τ ),
STFT (t, Ω) = FTτ { xt (τ )w(τ )}.
Another form of the STFT, with the same time-frequency performance,

is
"∞
STFTI I (t, Ω) = x (τ )w∗ (τ − t)e− jΩτ dτ (9.2)
−∞
where w∗ (t) denotes the conjugated window function.

It is obvious that definitions (9.1) and (9.2) differ only in phase, i.e.,
STFTI I (t, Ω) = e− jΩt STFT (t, Ω) for real valued windows w(τ ). We will
mainly use the first STFT form.
Example 9.1. To illustrate the STFT application, let us perform the time-frequency
analysis of the following signal
x (t) = δ(t − t1 ) + δ(t − t2 ) + e jΩ1 t + e jΩ2 t . (9.3)
The STFT of this signal equals
STFT (t, Ω) = w(t1 − t)e− jΩ(t1 −t) + w(t2 − t)e− jΩ(t2 −t)
+ W (Ω − Ω1 )e jΩ1 t + W (Ω − Ω2 )e jΩ2 t , (9.4)
where W (Ω) is the Fourier transform of the used window. The STFT is de-
picted in Fig. 9.2 for various window lengths, along with the ideal represen-
tation. A wide window w(t) in the time domain is characterized by a narrow
Fourier transform W (Ω) and vice versa. Influence of the window to the re-
sults will be studied later.
Example 9.2. The STFT of signal

2
x (t) = e jat (9.5)
can be approximately calculated for a large a, by using the method of station-

ary phase. Find its form and the relation for the optimal window w(τ ) width,
assuming that the window is nonzero for |τ | < T .
STFT (t,Ω) STFT (t,Ω)

wide narrow
Ω Ω
Ω2 Ω2
Ω1 Ω1
(a) (b)
t t t t t t
1 2 1 2
STFToptimal(t,Ω) Ideal TFR(t,Ω)

Ω Ω
Ω2 Ω2
Ω1 Ω1
(c) (d)
t1 t2 t t1 t2 t
Figure 9.2 Time-frequency representation of the sum of two delta pulses and two sinusoids
obtained by using (a) wide window, (b) narrow window (c) medium width window and (d)
ideal time-frequency representation.
⋆Applying the stationary phase method (1.62), we get
"∞
2
STFT (t, Ω) = e ja(t+τ ) w(τ )e− jΩτ dτ
−∞
"T =
2 2 2 2πj
e ja(t+τ ) w(τ )e− jΩτ dτ ≃ e jat e j(2at−Ω)τ0 e jaτ0 w(τ0 )
2a
−T
* +=
jat2 − j(2at−Ω)2 /4a Ω − 2at πj
=e e w (9.6)
2a a
since
2a(t + τ0 ) = Ω.
Note that the STFT absolute value reduces to

' * +' =
' Ω − 2at '' π
'
|STFT (t, Ω)| ≃ 'w
2a ' a. (9.7)
In this case, the width of |STFT (t, Ω)| along frequency does not decrease with
the increase of the window w(τ ) width. The width of |STFT (Ω, t)| around the
central frequency Ω = 2at is
D = 4aT,
where 2T is the window width in the time domain. Note that this relation
holds for a wide window w(τ ), such that the stationary phase method may
be applied. If the window is narrow with respect to the phase variations of
the signal, the STFT width is defined by the width of the Fourier transform
of window. It is proportional to 1/T. Thus, the overall STFT width could
be approximated by a sum of the frequency variation caused width and the
window’s Fourier transform width, that is,
2c
Do = 4aT + , (9.8)
T
where c is a constant defined by the window shape (by using the main lobe
as the window width, it will be shown later that c = 2π for a rectangular
window or c = 4π for a Hann(ing) window). This relation corresponds to
the STFT calculated as a convolution of an appropriately scaled time domain
window whose width is |τ | < 2aT and the frequency domain form of window
W (Ω). The approximation is checked against the exact STFT calculated by
definition. The agreement is almost complete, Fig.9.3. Therefore, there is a
window width T producing the narrowest possible STFT for this signal. It is
obtained by equating the derivative of the overall width to zero,
2c
4a − = 0,
T2
which results in =
c
To = . (9.9)
2a
As expected, for a sinusoid, a → 0, To → ∞. This is just an approximation
of the optimal window, since for narrow windows we may not apply the
stationary phase method (the term 4aT is then much smaller than 2c/T and
may be neglected anyway).
Note that for a = 1/2, when the instantaneous frequency is a symmetry
line for the time and the frequency axis
2c 2c
2− = 0 or 2T = ,
T2 T
meaning that the optimal window should have the widths equal in the time-
domain 2T and in the frequency domain 2c/T (main lobe width).
Exact absolute STFT value STFT approximation

10 10
log of the window width log2(T)
log of the window width log2(T)

9 9
8 8
7 7
6 6
5 5
4 4
3 3
2 2
1 1
-1 0 1 -1 0 1
frequency Ω frequency Ω
Figure 9.3 Exact absolute STFT value of a linear FM signal at t = 0 for various window
widths T = 2, 4, 8, 16, .., 1024 (left) and its approximation calculated as an appropriately scaled
convolution of the time and frequency domain window w(τ ) (right).
The STFT can be expressed in terms of the signal’s Fourier transform
"∞ "∞
1
STFT (t, Ω) = X (θ ) e j(t+τ )θ w(τ ) e− jΩτ dθ dτ
2π
−∞ −∞
"∞ @ A
1
= X (θ )W (Ω − θ ) e jtθ dθ = X (Ω)e jtΩ ∗Ω W (Ω). (9.10)
2π
−∞
where ∗Ω denotes convolution in Ω. It may be interpreted as an inverse

Fourier transform of the frequency localized version of X (Ω), with localiza-
tion window W (Ω) = FT{w(τ )}.
The energetic version of the STFT, called the spectrogram, is defined
by
SPEC (t, Ω) =| STFT (t, Ω) |2

' '2 ' '2
' "∞ ' ' "∞ '
' ∗ − jΩτ
' ' ∗ − jΩτ
'
'
= ' x (τ )w (τ − t)e ' '
dτ ' = ' x (t + τ )w (τ )e dτ '' .
'−∞ ' '−∞ '
Obviously, linearity property is lost in the spectrogram.

x (t)
x (t)
1
2
(a) (b)
t t
|X (Ω)|
|X (Ω)|
1
2
(c) (d)
Ω Ω
Figure 9.4 Two different signals x1 (t) ̸= x2 (t) with the same amplitudes of their Fourier
transforms, i.e., | X1 (Ω)| = | X2 (Ω)|.
Example 9.3. For illustration consider two different signals x1 (t) and x2 (t) pro-
ducing the same amplitude of the Fourier transform, Fig. 9.4,
* + ( * + )
t t 16 t − 128 2
x1 (t) = sin 122π − cos 42π − π
128 128 11 64
( * +2 * + )
t t − 128 t − 120 3 −( t−140 )2
− 1.2 cos 94π − 2π −π e 75
128 64 64
( * + )
t t − 50 2 −( t−50 )2
− 1.6 cos 15π − 2π e 16 (9.11)
128 64
x2 (t) = x1 (255 − t).
Their spectrograms are presented in Fig.9.5. From the spectrograms we can
follow time variations of the spectral content. The signals obviously consist
of one constant high frequency component, one linear frequency component
(in the first signal with increasing frequency as time progresses, and in the
second signal with decreasing frequency), and two chirps (one appearing at
different time instants and the other having different frequency variations).
The signal can be obtained from the STFT calculated at an instant t0 as

"∞
1
x ( t0 + τ ) = STFT (t0 , Ω)e− jΩτ dτ.
w(τ )
−∞
SPEC1(t,Ω)
250
200
150
100
t
50
(a)
0 2.5 3
0.5 1 1.5 2
0
Ω
SPEC2(t,Ω)
250
200
150
100
t
50
(b)
0 2.5 3
0.5 1 1.5 2
0
Figure 9.5 Spectrograms of the signals presented in Fig.9.4.
This relation can be theoretically used for the signal within the region
w(τ ) ̸= 0. In practice it is used within the region of significant window w(τ )
values.
If the window is shifted for R, for each next STFT calculation, then a
set of values
"∞
x (t0 + iR + τ )w(τ ) = STFT (t0 + iR, Ω)e− jΩτ dτ
−∞
is obtained. If the value of step R is smaller than the window duration then
the same signal value is used within two (several) windows. Using a change
of variables iR + τ = λ and summing over all overlapping windows we get
"∞
x (t0 + λ) ∑ w(λ − iR) = ∑ STFT (t0 + iR, Ω)e− jΩλ e jΩiR dλ.
i i −∞
Values of i in the summation are such that for a given λ and R the value of
iR − λ = τ is within the window w(τ ).
If the sum of shifted versions of the windows is constant (without loss
of generality assume equal to 1), ∑ w(τ − iR) = 1, then
i
"∞
x ( t0 + λ ) = ∑ STFT (t0 + iR, Ω)e− jΩλ e jΩiR dλ
i −∞
for any λ. Condition ∑ w(τ − iR) = 1 means that a periodic extension of

i
the window, with period R, is constant. Periodic extension of a continuous
signal corresponds to the sampling of the window Fourier transform at
Ω = 2π 2π
R n in the Fourier domain, (1.59). It means that W ( R n ) = 0 when
n ̸= 0 for ∑ w(λ − iR) = 1.
i
9.2 WINDOWS
The window function plays a crucial role in the localization of the signal
in the time-frequency plane. The most commonly used windows will be
presented next.
9.2.1 Rectangular Window
The simplest window is the rectangular one, defined by

!
1 for |τ | < T
w(τ ) = (9.12)
0 elsewhere
whose Fourier transform is

"T
2 sin(ΩT )
WR (Ω) = e− jΩτ dτ = . (9.13)
Ω
−T
The rectangular window function has very strong and oscillatory

sidelobes in the frequency domain, since the function sin (ΩT )/Ω converges
very slowly, toward zero, in Ω as Ω → ±∞. Slow convergence in the Fourier
domain is caused by a significant discontinuity in time domain, at t = ± T.
The mainlobe width of WR (Ω) is dΩ = 2π/T. In order to enhance signal
localization in the frequency domain, other window functions have been
introduced.
The discrete-time form of the rectangular window is
w(n) = u(n + N/2) − u(n − N/2)

N/2−1
sin(ωN/2)
W (e jω ) = ∑ e− jωn = .
n=− N/2
sin(ω/2)
9.2.2 Triangular (Bartlett) Window
It is defined by
!
1 − |τ/T | for |τ | < T
w(τ ) = (9.14)
0 elsewhere.
It could be considered as a convolution of the rectangular window of

duration T with itself, since
[u(t + T/2) − u(t − T/2)] ∗t [u(t + T/2) − u(t − T/2)]

= (1 − |τ/T |) [u(t + T ) − u(t − T )].
The Fourier transform of the triangular window is a product of two Fourier

transforms of the rectangular window of the width T,
4 sin2 (ΩT/2)
WT (Ω) = . (9.15)
Ω2
Convergence of this function toward zero as Ω → ±∞ is of the 1/Ω2 order.
It is a continuous function of time, with discontinuities in the first derivative
at t = 0 and t = ± T. The mainlobe of this window function is twice wider
in the frequency domain than in the rectangular window case. Its width
follows from ΩT/2 = π as dΩ = 4π/T.
The discrete-time form is
- .
2 |n|
w(n) = 1 − [u(n + N/2) − u(n − N/2)].
N
In the frequency domain its form is
N/2−1 - .
2 |n| − jωn sin2 (ωN/4)
W (e jω ) = ∑ 1− e = .
n=− N/2
N sin2 (ω/2)
9.2.3 Hann(ing) Window
This window is of the form

!
0.5(1 + cos (πτ/T )) for |τ | < T
w(τ ) = (9.16)
0 elsewhere.
Since cos (πτ/T ) = [exp ( jπτ/T ) + exp (− jπτ/T )]/2, the Fourier
transform of this window is related to the Fourier transform of the rect-
angular window of the same width as
1 1 1
WH (Ω) = WR (Ω) + WR (Ω − π/T ) + WR (Ω + π/T )
2 4 4
π 2 sin(ΩT )
= . (9.17)
Ω ( π 2 − Ω2 T 2 )
The function WH (Ω) decays in frequency as Ω3 , much faster than WR (Ω).

The discrete-time domain form is
- * +.
2πn
w(n) = 0.5 1 + cos [u(n + N/2) − u(n − N/2)]
N
with the DFT of the form
N N N
W (k ) = δ ( k ) + δ ( k + 1 ) + δ ( k − 1 ).
2 4 4
If the window is used on the data set from 0 to N − 1 then

- * +.
2πn
w(n) = 0.5 1 − cos [u(n) − u(n − N )]
N
N N N
W ( k ) = δ ( k ) − δ ( k + 1 ) − δ ( k − 1 ).
2 4 4
If a signal is multiplied by the Hann(ing) window the previous relation

also implies the relationship between the DFTs of the signal x (n) calculated
using the rectangular and Hann(ing) windows. The DFT of windowed
signal is moving average (smoothed) form of the original signal,
1
DFT{ x (n)w(n)} = DFT{ x (n)} ∗k DFT{w(n)}
N
1 1 1
= X ( k + 1) + X ( k ) + X ( k − 1)
4 2 4
Example 9.4. Find the window that will correspond to the frequency smoothing
( X (k + 1) + X (k) + X (k − 1))/3, i.e., to
1
DFT{ x (n)w(n)} = DFT{ x (n)} ∗k DFT{w(n)}
N
1 1 1
= X ( k + 1) + X ( k ) + X ( k − 1).
3 3 3
⋆The DFT of this window is

N N N
W (k ) = δ ( k ) + δ ( k + 1) + δ ( k − 1).
3 3 3
In the discrete-time domain the window form is
- * +.
1 2πn
w(n) = 1 + 2 cos [u(n) − u(n − N )].
3 N
Example 9.5. Find the formula to calculate the STFT with a Hann(ing) window, if
the STFT calculated with a rectangular window is known.
⋆From the frequency domain STFT definition
"∞
1
STFT (t, Ω) = X (θ )W (Ω − θ )e jtθ dθ
2π
−∞
easily follows that, if we use the window,
1 1 1
WH (Ω) = WR (Ω) + WR (Ω − π/T ) + WR (Ω + π/T ),
2 4 4
then
1
STFTH (t, Ω) = STFTR (t, Ω) (9.18)
2
1 B πC 1 B πC
+ STFTR t, Ω − + STFTR t, Ω + . (9.19)
4 T 4 T
For the Hann(ing) window w(τ ) of the width 2T, we may roughly
assume that its Fourier transform WH (Ω) is nonzero within the main lattice
| Ω |< 2π/T only, since the sidelobes decay very fast. Then we may write
dΩ = 4π/T. It means that the STFT is nonzero valued in the shaded regions
in Fig. 9.2.
We see that the duration in time of the STFT of a delta pulse is equal
to the widow width dt = 2T. The STFTs of two delta pulses (very short du-
ration signals) do not overlap in time-frequency domain if their distance
is greater than the window duration |t1 − t2 | > dt . Then, these two pulses
can be resolved. Thus, the window width is here a measure of time res-
olution. Since the Fourier transform of the Hann(ing) window converges
fast, we can roughly assume that a measure of duration in frequency is
the width of its mainlobe, dΩ = 4π/T. Then we may say that the Fourier
transforms of two sinusoidal signals do not overlap in frequency if the con-
dition |Ω1 − Ω2 | > dΩ holds. It is important to observe that the product of
the window durations in time and frequency is a constant. In this example,
considering time domain duration of the Hann(ing) window and the width
of its mainlobe in the frequency domain, this product is dt dΩ = 8π. There-
fore, if we improve the resolution in the time domain dt , by decreasing T, we
inherently increase the value of dΩ in the frequency domain. This essentially
prevents us from achieving the ideal resolution (dt = 0 and dΩ = 0) in both
domains. A general formulation of this principle, stating that the product of
effective window durations in time and in frequency cannot be arbitrarily
small, will be presented later.
9.2.4 Hamming Window
This window has the form

!
0.54 + 0.46 cos (πτ/T )) for |τ | < T
w(τ ) = (9.20)
0 elsewhere.
A similar relation between the Hamming and the rectangular window

transforms holds, as in the case of Hann(ing) window.
The Hamming window was derived starting from
w(τ ) = a + (1 − a) cos (πτ/T ))
within |τ | < T, with

* +
2 sin(ΩT ) sin((Ω − π/T ) T ) sin((Ω + π/T ) T )
W (Ω) = a + (1 − a ) + .
Ω Ω − π/T Ω + π/T
If we choose such a value of a to cancel out the second sidelobe at its

maximum (i.e., at ΩT ∼
= 2.5π) then we get
* +
2aT T T
0= − (1 − a ) +
2.5π 1.5π 3.5π
resulting in
a = 25/46 ∼
= 0.54. (9.21)
This window has several sidelobes, next to the mainlobe, lower than the
previous two windows. However, since it is not continuous at t = ± T, its
decay in frequency, as Ω → ±∞, is not fast. Note that we let the mainlobe
to be twice wider than in the rectangular window case, so we cancel out not
the first but the second sidelobe, at its maximum.
The discrete-time domain form is
- * +.
2πn
w(n) = 0.54 + 0.46 cos [u(n + N/2) − u(n − N/2)]
N
with
W (k ) = 0.54Nδ(k ) + 0.23Nδ(k + 1) + 0.23Nδ(k − 1).
9.2.5 Blackman and Kaiser Windows
In some applications it is crucial that the sidelobes are suppressed, as much

as possible. This is achieved by using windows of more complicated forms,
like the Blackman window. It is defined by
!
0.42 + 0.5 cos (πτ/T ) + 0.08 cos (2πτ/T ) for |τ | < T
w(τ ) = (9.22)
0 elsewhere.
This window is derived from
w(τ ) = a0 + a1 cos (πτ/T ) + a2 cos (2πτ/T )
with a0 + a1 + a2 = 1 and canceling out the Fourier transform values W (Ω)

at the positions of the third and the fourth sidelobe maxima (i.e., at ΩT ∼ =
3.5π and ΩT ∼ = 4.5π). Here, we let the mainlobe to be three times wider than
in the rectangular window case, so we cancel out not the first nor the second
but the third and fourth sidelobes, at their maxima.
The discrete-time and frequency domain forms are

- * + * +.
2πn 4πn N N
w(n) = 0.42 + 0.5 cos + 0.08 cos [u(n + ) − u(n − )]
N N 2 2
W (k ) = [0.42δ(k ) + 0.25(δ(k + 1) + δ(k − 1)) + 0.04(δ(k + 2) + δ(k − 2))] N.
Further reduction of the sidelobes can be achieved by, for example,

the Kaiser (Kaiser-Bessel) window. It is an approximation to a restricted
time duration function with minimum energy outside the mainlobe. This
window is defined by using the zero-order Bessel functions, with a localiza-
tion parameter. It has the ability to keep the maximum energy within the
mainlobe, while minimizing the sidelobe energy. The sidelobe level can be
as low −70 dB, as compared to the mainlobe, and even lower. This kind of
window is used in the analysis of signals with significantly different ampli-
tudes, when the sidelobe of one component can be much higher than the
amplitude of the mainlobe of other components.
These are just a few of the windows used in signal processing. Some
windows, along with the corresponding Fourier transforms, are presented
in Fig. 9.6.
Example 9.6. Calculate the STFT at t = 0 with a Hamming and Blackman

window of the signals x1 (t) = 2 cos(4πt/T ) + 2 cos(12πt/T ) and x2 (t) =
2 cos(4πt/T ) + 0.001 cos(64πt/T ) with T = 128 using in numerical calcula-
tion ∆t = 1. Comment the results.
⋆The STFT at t = 0 is shown in Fig.9.7. The resolution of close compo-

nents in x1 (t) is better with the Hann(ing) than with the Blackman window,
since the main lobe of the Blackman window is wider. Small signal in x2 (t) is
visible in the STFT with the Blackman window since its side-lobes are much
lower than in the Hamming window.
9.2.6 Discrete Form and Realizations of the STFT
In numerical calculations the integral form of the STFT should be dis-

cretized. By sampling the signal with sampling interval ∆t we get
"∞
STFT (t, Ω) = x (t + τ )w(τ )e− jΩτ dτ
−∞
∞
≃ ∑ x ((n + m)∆t)w(m∆t)e− jm∆tΩ ∆t.
m=−∞
10 log|W(Ω)|
W(Ω)
w(τ)
τ Ω Ω
10 log|W(Ω)|
W(Ω)
w(τ)
τ Ω 10 log|W(Ω)| Ω
W(Ω)
w(τ)
τ Ω Ω
10 log|W(Ω)|
W(Ω)
w(τ)
τ Ω Ω
10 log|W(Ω)|
W(Ω)
w(τ)
τ Ω Ω
Figure 9.6 Windows in the time and frequency domains: rectangular window (first row),
triangular (Bartlett) window (second row), Hann(ing) window (third row), Hamming window
(fourth row), and Blackman window (fifth row).
0 0
|STFT(0,Ω)|
|STFT(0,Ω)|
10 10
-5 -5
10 10
-1 -0.5 0 0.5 1 -1 -0.5 0 0.5 1

0 0
|STFT(0,Ω)|
|STFT(0,Ω)|
10 10
-5 -5
10 10
-1 -0.5 0 0.5 1 -1 -0.5 0 0.5 1

Figure 9.7 The STFT at n = 0 calculated using the Hamming window (left) and the Blackman
window (right) of signals x1 (n) (top) and signal x2 (n) (bottom).
By denoting
x (n) = x (n∆t)∆t
and normalizing the frequency Ω by ∆t, ω = ∆tΩ, we get the time-discrete

form of the STFT as
∞
STFT (n, ω ) = ∑ w(m) x (n + m)e− jmω . (9.23)
m=−∞
We will use the same notation for continuous-time and discrete-time signals,
x (t) and x (n). However, we hope that this will not cause any confusion since
we will use different sets of variables, for example t and τ for continuous
time and n and m for discrete time. Also, we hope that the context will be
always clear, so that there is no doubt what kind of signal is considered.
It is important to note that STFT (n, ω ) is periodic in frequency with

period 2π. The relation between the analog and the discrete-time form is
∞
STFT (n, ω ) = ∑ STFT (n∆t, Ω + 2kΩ0 ) with ω = ∆tΩ.
k=−∞
The sampling interval ∆t is related to the period in frequency as ∆t = π/Ω0 .

According to the sampling theorem, in order to avoid the overlapping of the
STFT periods (aliasing), we should take
π π
∆t = ≤
Ω0 Ωm
where Ωm is the maximal frequency in the STFT. Strictly speaking, the

windowed signal x (t + τ )w(τ ) is time limited, thus it is not frequency
limited. Theoretically, there is no maximal frequency since the width of
the window’s Fourier transform is infinite. However, in practice we can
always assume that the value of spectral content of x (t + τ )w(τ ) above
frequency Ωm , i.e., for |Ω| > Ωm , can be neglected, and that overlapping
of the frequency content above Ωm does not degrade the basic frequency
period.
The discretization in frequency should be done by a number of sam-
ples greater than or equal to the window length N. If we assume that the
number of discrete frequency points is equal to the window length, then
N/2−1
STFT (n, k ) = STFT (n, ω )|ω = 2π k = ∑ w(m) x (n + m)e− j2πmk/N (9.24)
N
m=− N/2
and it can be efficiently calculated using the fast DFT routines
STFT (n, k ) = DFTm {w(m) x (n + m)},
for a given instant n. When the DFT routines with indices from 0 to N − 1
are used, then a shifted version of w(m) x (n + m) should be formed for the
calculation for N/2 ≤ m ≤ N − 1. It is obtained as w(m − N ) x (n + m − N ),
since in the DFT calculation periodicity of the signal w(m) x (n + m), with
period N, is inherently assumed.
Example 9.7. Consider a signal with M = 16 samples, x (0), x (1),...., x (15), write
a matrix form for the calculation of a four-sample STFT. Present nonoverlap-
ping and overlapping cases of the STFT calculation.
⋆For the calculation of (9.24) with N = 4, when k = −2, −1, 0, 1, for

given instant n, the following matrix notation can be used
⎡ ⎤ ⎡ ⎤⎡ ⎤
STFT (n, −2) W44 W42 1 W4−2 x ( n − 2)
⎢ STFT (n, −1) ⎥ ⎢ W 2
⎢ ⎥ ⎢ 4 W41 1 W4−1 ⎥ ⎢
⎥ ⎢ x ( n − 1)
⎥
⎥
⎣ STFT (n, 0) ⎦ = ⎣ 1 1 1 1 ⎦⎣ x (n) ⎦
STFT (n, 1) W4−2 W4−1 1 W41 x ( n + 1)
or
STFT(n) = W4 x(n)
with STFT (n) = [STFT (n, −2) STFT (n, −1) STFT (n, 0) STFT (n, 1)] T , x(n) =
[ x (n − 2) x (n − 1) x (n) x (n + 1)] T , and W4 is the DFT matrix of order
four with elements W4mk = exp(− j2πmk/N ). Here a rectangular window
is assumed. Including the window function, the previous relation can be
written as
STFT (n)= W4 H4 x(n),
with ⎡ ⎤
w(−2) 0 0 0
⎢ 0 w(−1) 0 0 ⎥
H4 = ⎢
⎣
⎥
0 0 w (0) 0 ⎦
0 0 0 w (1)
being a diagonal matrix whose elements are the window values w(m),
H4 =diag(w(m)), m = −2, −1, 0, 1 and
⎡ ⎤
w(−2)W44 w(−1)W42 w (0) w(1)W4−2
⎢ w(−2)W 2 w(−1)W41 w (0) w(1)W4−1 ⎥
W4 H 4 = ⎢
⎣
4 ⎥.
⎦
w(−2) w(−1) w (0) w (1)
w(−2)W4−2 w(−1)W4−1 w (0) w(1)W4 1
All STFT values for the nonoverlapping case are obtained as

⎡ ⎤
x (0) x (4) x (8) x (12)
⎢ x (1) x (5) x (9) x (13) ⎥
STFT = W4 H4 ⎢ ⎥
⎣ x (2) x (6) x (10) x (14) ⎦ = W4 H4 X4,4
x (3) x (7) x (11) x (15)
where STFT is a matrix of the STFT values with columns corresponding to

the calculation instants and the rows to the frequencies. This matrix is of the
form
D E
STFT = STFT M (0) STFT M ( M ) ... STFT M ( N − M )
⎡ ⎤
STFT (2, −2) STFT (6, −2) STFT (10, −2) STFT (14, −2)
⎢ STFT (2, −1) STFT (6, −1) STFT (10, −1) STFT (14, −1) ⎥
=⎢⎣ STFT (2, 0)
⎥.
STFT (6, 0) STFT (10, 0) STFT (14, 0) ⎦
STFT (2, 1) STFT (6, 1) STFT (10, 1) STFT (14, 1)
Matrix X4,4 is formed of by using four successive signal values in each

column. Notation X N,R will be used to denote the signal matrix with columns
containing N signal values and the difference of the first signal value indices
in the successive columns is R. For R = N the nonoverlapping calculation is
performed.
For a STFT calculation with overlapping R < N, for example with the
time step in the STFT calculation R = 1, we get
⎡ ⎤
x (0) x (1) x (2) ... x (10) x (11) x (12)
⎢ x (1) x (2) x (3) ... x (11) x (12) x (13) ⎥
STFT = H4 W4 ⎢
⎣ x (2)
⎥
x (3) x (4) ... x (12) x (13) x (14) ⎦
x (3) x (4) x (5) ... x (13) x (14) x (15)
STFT =W4 H4 X4,1 .
The step R defines the difference of arguments in two neighboring columns.

In the first case the difference of arguments in two neighboring columns was
4 (time step in the STFT calculation was R = 4 equal to the window width,
meaning nonoverlapped calculation). In the second example difference is
R = 1 < 4, meaning the overlapped STFT calculation. Note that the window
function HN and the DFT matrix WN remain the same for both cases.

2 2 2 2
x (t) = e−t e− j6πt − j32πt + e−4(t−1) e j16πt + j160πt .
Assuming that the values of the signal with amplitudes bellow 1/e4 could be
neglected, find the sampling rate for the STFT-based analysis of this signal.
Write the approximate spectrogram expression for the Hann(ing) window of
N = 32 samples in the analysis. What signal will be presented in the time-
frequency plane, within the basic frequency period, if the signal is sampled
at ∆t = 1/128?
⋆The time interval, with significant signal content, for the first signal
component is −2 ≤ t ≤ 2, with the frequency content within −56π ≤ Ω ≤
−8π, since the instantaneous frequency is Ω(t) = −12πt − 32π. For the
second component these intervals are 0 ≤ t ≤ 2 and 160π ≤ Ω ≤ 224π. The
maximal frequency in the signal is Ωm = 224π. Here we have to take into
account possible spreading of the spectrum caused by the lag window. Its
width in the time domain is dt = 2T = N∆t = 32∆t. Width of the mainlobe
in frequency domain dw is defined by 32dw ∆t = 4π, or Ωw = π/(8∆t).
Thus, taking the sampling interval ∆t = 1/256, we will satisfy the sampling
theorem condition in the worst instant case, since π/(Ωm + dw ) = 1/256.
In the case of the Hann(ing) window with N = 32 and ∆t = 1/256,
the lag interval is N∆t = 1/8. We will assume that the amplitude variations
within the window are small, that is, w(τ )e−(t+τ ) ∼
2 2
= w(τ )e−t for −1/16 <
τ ≤ 1/16. Then, according to the stationary phase method, we can write the
STFT approximation,
2
B C B C
1 −8( t −1)2 2
|STFT (t, Ω)|2 = 16 e−2t w2 Ω+12πt+32π
12π + 32 e w Ω−32πt−160π
32π
with t = n/256 and Ω = 256ω within −π ≤ ω < π.

In the case of ∆t = 1/128 the signal will be periodically extended with
period 2Ω0 = 256π. The basic period will be for −128π ≤ Ω < 128π. It means
that the first component will remain unchanged within the basic period,
while the second component is outside the basic period. However, its replica
shifted for one period to the left, that is, for −256π, will be within the basic
period. It will be located within 160π − 256π ≤ Ω ≤ 224π − 256π, that is,
within −96π ≤ Ω ≤ −32π. Thus, the signal represented by the STFT in this
case will correspond to
2 2 2 2
xr (t) = e−t e− j6πt − j32πt + e−4(t−1) e j16πt + j(160−256)πt ,
with approximation,
2
B C B C
1 −8( t −1)2 2
|STFT (t, Ω)|2 = 16 e−2t w2 Ω+12πt+32π
12π + 32 e , w Ω−32πt−96π
32π
(9.25)
with t = n/128 and Ω = 128ω within −π ≤ ω < π or −128π ≤ Ω < 128π.
9.2.7 Recursive STFT Realization
For the rectangular window, the STFT values at an instant n can be calcu-
lated recursively from the STFT values at n − 1, as
STFTR (n, k ) = [ x (n + N/2 − 1) − x (n − N/2 − 1)](−1)k e j2πk/N

+ STFTR (n − 1, k )e j2πk/N .
This recursive formula follows easily from the STFT definition (9.24).
For other window forms, the STFT can be obtained from the STFT
obtained by using the rectangular window. For example, according to (9.18)
the STFT with Hann(ing) window STFTH (n, k ) is related to the STFT with
rectangular window STFTR (n, k ) as
1 1 1
STFTH (n, k ) = STFTR (n, k ) + STFTR (n, k − 1) + STFTR (n, k + 1).
2 4 4
This recursive calculation is important for hardware implementation of

the STFT and other related time-frequency representations (e.g., the higher
order representations implementations based on the STFT).
k
(-1) e j2kπ/N
x(n+N/2-1) + + STFT (n,k)
R
-N
z z-1
-1
a1
STFT (n,k+1)
R
a
STFTR(n,k) 0 STFTH(n,k)
+
a
STFTR(n,k-1) -1
Figure 9.8 Recursive implementation of the STFT for the rectangular and other windows.
A system for the recursive implementation of the STFT is shown in

Fig. 9.8. The STFT obtained by using the rectangular window is denoted by
STFTR (n, k ), Fig.9.8, while the values of coefficients are
1 1 1
( a −1 , a 0 , a 1 ) = ( , , ),
4 2 4
( a−1 , a0 , a1 ) = (0.23, 0.54, 0.23),
( a−2 , a−1 , a0 , a1 , a2 ) = (0.04, 0.25, 0.42, 0.25, 0.04)
for the Hann(ing), Hamming and Blackman windows, respectively.

Note that in general instead of multiplying the signal by the previous
window functions, for each calculation instant n, the STFT matrix STFT can
be calculated without window multiplication (using a rectangular window).
The STFT matrix for the Hann(ing) window, for example, is obtained as
STFT H = 0.5STFT +0.25STFT↓ + 0.25STFT↑ , where STFT↓ and STFT↑
are the STFT matrices with circularly shifted rows down and up for one
position, respectively.
9.2.8 Filter Bank STFT Implementation
According to (9.1), the STFT can be written as a convolution

STFT(n,0)
w(n) ↓R
STFT(n,1)
w(n) e
j2πn/N ↓R
x(n)
...
STFT(n,N-1)
w(n) e
j2πn(N-1)/ N ↓R
Figure 9.9 Filter bank realization of the STFT
"∞
STFT (t, Ω) = x (t + τ ) w(τ ) e− jΩτ dτ
−∞
"∞ @ A
= x (t − τ )w(τ )e jΩτ dτ = x (t) ∗t w(t)e jΩt
−∞
where an even, real valued, window function is assumed, w(τ ) = w(−τ ).

For a discrete set of frequencies Ωk = k∆Ω = 2πk/( N∆t), k = 0, 1, 2, ..., N − 1,
and discrete values of signal, we get that the discrete STFT, (9.24), is an
output of the filter bank with impulse responses
@ A
STFT (n, k ) = x (n) ∗n w(n)e j2πkn/N = x (n) ∗n hk (n)
hk (n) = w(n)e j2πkn/N

k = 0, 1, ..., N − 1
what is illustrated in Fig.9.9. The next STFT can be calculated with time step
R∆t, meaning downsampling in time with factor 1 ≤ R ≤ N. Two special
cases are: no downsampling, R = 1, and nonoverlapping calculation, R = N.
Influence of R to the signal reconstruction will be discussed later.
9.2.8.1 Overlapping windows
Nonoverlapping cases are important and easy for analysis. They also keep
the number of the STFT coefficients equal to the number of the signal sam-
ples. However, the STFT is commonly calculated using overlapping win-
dows. There are several reasons for introducing overlapped STFT repre-
sentations. Rectangular windows have poor localization in the frequency
domain. The localization is improved by other window forms. In the case of
nonrectangular windows some of the signal samples are weighted in such
a way that their contribution to the final representation is small. Then we
want to use additional STFT with a window positioned in such a way that
these samples contribute more to the STFT calculation. Also, in the param-
eters estimation and detection the task is to achieve the best possible esti-
mation or detection for each time instant instead of using interpolations for
the skipped instants when the STFT with a big step (equal to the window
width) is calculated. Commonly, the overlapped STFTs are calculated using,
for example, rectangular, Hann(ing), Hamming, Bartlett, Kaiser, or Black-
man window of a constant window width N with steps N/2, N/4, N/8, ...
in time. Computational cost is increased in the overlapped STFTs since more
STFTs are calculated. A way of composing STFTs calculated with a rectan-
gular window into a STFT with, for example, the Hann(ing), Hamming, or
Blackman window, is presented in Fig.9.8.
If a signal x (n) is of duration M, in some cases in addition to the
overlapping in time, an interpolation in frequency is done, for example up
to the DFT grid with M samples. The overlapped and interpolated STFT of
this signal is calculated, using a window w(m) whose width is N ≤ M, as
N/2−1
STFTN (n, k) = ∑ w(m) x (n + m)e− j2πmk/M
m=− N/2
n = N/2 + 1, N/2 + 2, .., M − N/2
k = − M/2, − M/2 + 1, ..., −1, 0, 1, ..., M/2 − 1.
Example 9.9. The STFT calculation of a signal whose frequency changes linearly is
done by using a rectangular window. Signal samples within 0 ≤ n ≤ M − 1
with M = 64 were available. The nonoverlapping STFT of this signal is
calculated with a rectangular window of the width N = 8 and presented in
Fig.9.10. The nonoverlapping STFT values obtained by using the rectangular
window are shifted in frequency, scaled, and added up, Fig. 9.11, to produce
the STFT with a Hamming window, Fig. 9.12.
The STFT calculation for the same linear FM signal will be repeated for
the overlapping STFT with step R = 1. Results for the rectangular and Ham-
ming window (obtained by a simple matrix calculation from the rectangular
STFT with rectangular window

31
30
29
28 S8(4,3) S8(12,3) S8(20,3) S8(28,3) S8(36,3) S8(44,3) S8(52,3) S8(60,3)
27
26
25
24
23
22
21
20
19
S8(4,2) S8(12,2) S8(20,2) S8(28,2) S8(36,2) S8(44,2) S8(52,2) S8(60,2)
18
17
16
15
14
13
12 S8(4,1) S8(12,1) S8(20,1) S8(28,1) S8(36,1) S8(44,1) S8(52,1) S8(60,1)
11
10
9
8
7
6
5
4 S8(4,0) S8(12,0) S8(20,0) S8(28,0) S8(36,0) S8(44,0) S8(52,0) S8(60,0)
3
2
1
0
-1
-2
-3
-4 S8(4,-1) S8(12,-1) S8(20,-1) S8(28,-1) S8(36,-1) S8(44,-1) S8(52,-1) S8(60,-1)
-5
-6
-7
-8
-9
-10
-11
-12 S8(4,-2) S8(12,-2) S8(20,-2) S8(28,-2) S8(36,-2) S8(44,-2) S8(52,-2) S8(60,-2)
-13
-14
-15
-16
-17
-18
-19
-20 S8(4,-3) S8(12,-3) S8(20,-3) S8(28,-3) S8(36,-3) S8(44,-3) S8(52,-3) S8(60,-3)
-21
-22
-23
-24
-25
-26
-27
-28 S8(4,-4) S8(12,-4) S8(20,-4) S8(28,-4) S8(36,-4) S8(44,-4) S8(52,-4) S8(60,-4)
-29
-30
-31
-32
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63
Figure 9.10 The STFT of a linear FM signal x (n) calculates using a rectangular window of the
width N = 8.
window case) are presented in Fig.9.13. Three window widths are used here.
The same procedure is repeated with the windows zero padded up to the
widest used window (interpolation in frequency). The results are presented
in Fig.9.14. Note that regarding to the amount of information all these fig-
ures do not differ from the basic time-frequency representation presented in
Fig.9.10.
STFT (n,k-1) STFT (n,k) STFT (n,k+1)

R R R
0.23 STFTR(n,k-1) + 0.54 STFTR(n,k) + 0.23 STFTR(n,k+1)

= STFTH(n,k)
Figure 9.11 The STFT of a linear FM signal calculated using a rectangular window (from the
previous figure), along with its frequency shifted versions STFTR (n, k − 1) and STFTR (n, k −
1). Their weighted sum produces the STFT of the same signal with a Hamming window
STFTH (n, k).
9.2.9 Signal Reconstruction from the Discrete STFT
Signal reconstruction from non-overlapping STFT values is obvious for a

rectangular window. A simple illustration is presented in Fig.9.15. Win-
dowed signal values are reconstructed from the STFTs by a simple inversion
of each STFT
STFT (n) = W N Hw x(n)

Hw x(n) = IDFT{STFT (n)} = W− 1
N STFT (n )
where Hw is a diagonal matrix with the window values as its elements,

Hw = diag(w(m)).
Example 9.10. Consider a signal with M = 16 samples, x (0), x (1),...., x (16). Write
a matrix form for the signal inversion using a four-sample STFT (N = 16)
calculated with the rectangular and a Hann(ing) window: (a) Without over-
lapping, R = 16. (b) With a time step in the STFT calculation of R = 2.
⋆ (a) For the nonoverlapping case the STFT calculation is done accord-
ing to: ⎡ ⎤
x (0) x (4) x (8) x (12)
⎢ x (1) x (5) x (9) x (13) ⎥
STFT = W4 H4 ⎢ ⎣ x (2) x (6) x (10) x (14) ⎦ .
⎥
x (3) x (7) x (11) x (15)

with H4 =diag([w(−2) w(−1) w(0) w(1)]) and W4 is the corresponding four
sample DFT matrix.
STFT with Hamming window

31
30
29
28 S8(4,3) S8(12,3) S8(20,3) S8(28,3) S8(36,3) S8(44,3) S8(52,3) S8(60,3)
27
26
25
24
23
22
21
20
19
S8(4,2) S8(12,2) S8(20,2) S8(28,2) S8(36,2) S8(44,2) S8(52,2) S8(60,2)
18
17
16
15
14
13
12 S8(4,1) S8(12,1) S8(20,1) S8(28,1) S8(36,1) S8(44,1) S8(52,1) S8(60,1)
11
10
9
8
7
6
5
4 S8(4,0) S8(12,0) S8(20,0) S8(28,0) S8(36,0) S8(44,0) S8(52,0) S8(60,0)
3
2
1
0
-1
-2
-3
-4 S8(4,-1) S8(12,-1) S8(20,-1) S8(28,-1) S8(36,-1) S8(44,-1) S8(52,-1) S8(60,-1)
-5
-6
-7
-8
-9
-10
-11
-12 S8(4,-2) S8(12,-2) S8(20,-2) S8(28,-2) S8(36,-2) S8(44,-2) S8(52,-2) S8(60,-2)
-13
-14
-15
-16
-17
-18
-19
-20 S8(4,-3) S8(12,-3) S8(20,-3) S8(28,-3) S8(36,-3) S8(44,-3) S8(52,-3) S8(60,-3)
-21
-22
-23
-24
-25
-26
-27
-28 S8(4,-4) S8(12,-4) S8(20,-4) S8(28,-4) S8(36,-4) S8(44,-4) S8(52,-4) S8(60,-4)
-29
-30
-31
-32
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63
Figure 9.12 The STFT of a linear FM signal x (n) calculated using the Hamming window with
N = 8. Calculation is illustrated in the previous figure.
The inversion relation is

⎡ ⎤
x (0) x (4) x (8) x (12)
⎢ x (1) x (5) x (9) x (13) ⎥
⎢ ⎥ = H−1 W−1 STFT
⎣ x (2) x (6) x (10) x (14) ⎦ 4 4
x (3) x (7) x (11) x (15)
where the elements of diagonal matrix H− 1

4 are proportional to 1/w (m ),
−1
H4 =diag([1/w(−2) 1/w(−1) 1/w(0) 1/w(1)]). If a rectangular window
is used in the STFT calculation then H4−1 = I4 is unity matrix and this kind of
STFT with rectangular window, N=48 STFT with Hamming window, N=48
Figure 9.13 Time-frequency analysis of a linear frequency modulated signal with overlapping
windows of various widths. Time step in the STFT calculation is R = 1.
Figure 9.14 Time-frequency analysis of a linear frequency modulated signal with overlapping
windows of various widths. Time step in the STFT calculation is R = 1. For each window
width the frequency axis is interpolated (signal in time is zero padded) up to the total number
of available signal samples M = 64.
STFT(2,k) STFT(6,k) STFT(10,k) STFT(14,k)
6
S (2,1) S (6,1) S (10,1) S (14,1)
4 4 4 4
5
2
S4(2,0) S4(6,0) S4(10,0) S4(14,0)
1
-1
-2
S4(2,-1) S4(6,-1) S4(10,-1) S4(14,-1)
-3
-4
-5
-6
S4(2,-2) S4(6,-2) S4(10,-2) S4(14,-2)
-7
-8
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
x(0), x(1), x(2), x(3) x(4), x(5), x(6), x(7) x(8), x(9),x(10),x(11) x(12),x(13),x(14),x(15)
x(2+m)w(m)= x(6+m)w(m)= x(10+m)w(m)= x(14+m)w(m)=

IDFT{STFT(2,k)} IDFT{STFT(6,k)} IDFT{STFT(10,k)} IDFT{STFT(14,k)}
m=-2,-1,0,1 m=-2,-1,0,1 m=-2,-1,0,1 m=-2,-1,0,1
Figure 9.15 Illustration of the signal reconstruction from the STFT with nonoverlapping
windows.
calculation can be used. However if a nonrectangular window is used then

some of the window values are quite small. The signal value is then obtained
by multiplying the inverse DFT with large values 1/w(m). This kind of
division with small values is very imprecise, if any noise in the reconstructed
signal is expected. In the Hann(ing) window case the ending point is even
zero-valued, so 1/w(m) does not exist.
(b) The STFT calculation is done with overlapping with step R = 2,
Fig.9.16. For N = 4 and calculation step R = 2 the STFT calculation corre-
sponds to
⎡ ⎤
0 x (0) x (2) x (4) x (6) x (8) x (10) x (12) x (14)
⎢ 0 x (1) x (3) x (5) x (7) x (9) x (11) x (13) x (15) ⎥
STFT = W4 H4 ⎢ ⎣ x (0) x (2) x (4) x (6) x (8) x (10) x (12) x (14) 0
⎥
⎦
x (1) x (3) x (5) x (7) x (9) x (11) x (13) x (15) 0
The inversion is
W4−1 STFT = H4 X =
⎡ ⎤
0 x (0)w(−2) x (2)w(−2) x (4)w(−2) ... x (14)w(−2)
⎢ 0 x (1)w(−1) x (3)w(−1) x (5)w(−1) ... x (15)w(−1) ⎥
⎢ ⎥
⎣ x (0) w (0) x (2) w (0) x (4) w (0) x (6) w (0) ... 0 ⎦
x (1) w (1) x (3) w (1) x (5) w (1) x (7) w (1) ... 0
where X is the matrix with signal elements. The window matrix is left
on the right side, since in general it may be not invertible. By calculating
W4−1 STFT we can then recombine the signal values. For example, the
element producing x (0)w(0) in the first column is combined with the element
producing x (0)w(−2) in the second column to get x (0)w(0) + x (0)w(−2) =
x (0), since for the Hann(ing) window of the width N holds w(n) + w(n −
N/2) = 1. The same is done for other signal values in the matrix obtained
after inversion,
x (0)w(0) + x (0)w(−2) = x (0)
x (1)w(1) + x (1)w(−1) = x (1)
x (2)w(0) + x (1)w(−2) = x (2)
...
x (15)w(1) + x (15)w(−1) = x (15)
Note that the same relation would hold for a triangular window, while for a
Hamming window a similar relation would hold, with w(n) + w(n − N/2) =
1.08. The results should be corrected in that case, by a constant factor of 1.08.
Illustration of the STFT calculation for an arbitrary window width N
at n = n0 is presented in Fig.9.16. Its inversion produces x (n0 + m)w(m) =
IDFT{STFTN (n0 , k )}. Consider the pervious STFT value in the case of
nonoverlapping windows. It would be STFTN (n0 − N, k ). Its inverse
IDFT{STFTN (n0 − N, k )} = x (n0 − N + m)w(m)
is also presented in Fig.9.16. As it can be seen, by combining these two inverse

transforms we would get signal with very low values around n = n0 − N/2̇.
If one more STFT is calculated at n = n0 − N/2 and its inverse combined with
previous two it will improve the signal presentation within the overlapping
region n0 − N ≤ n < n0 . In addition for the most of common windows
w(m − N ) + w(m − N/2) + w(m) = 1 (or a constant) within 0 ≤ m < N
meaning that the sum of overlapped inverse STFTs, as in Fig.9.16, will give
the original signal within n0 − N ≤ n < n0 .
In general, let us consider the STFT calculation with overlapping

windows. Assume that the STFTs are calculated with a step 1 ≤ R ≤ N
in time. Available STFT values are
...
STFT (n0 − 2R),
STFT (n0 − R), (9.26)
STFT (n0 ),
STFT (n0 + R),
STFT (n0 + 2R),
...
Based on the available STFT values (9.26), the windowed signal values can
be reconstructed as
Hw x(n0 + iR) = W− 1
N STFT (n0 + iR ), i = ... − 2, −1, 0, 1, 2, ...
For m = − N/2, − N/2 + 1, ..., N/2 − 1 we get signal values x (n0 + iR + m)
1 N/2−1
w(m) x (n0 + iR + m) = ∑ STFT (n0 + iR, k)e j2πmk/N .
N k=−
(9.27)
N/2
Since R < N we we will get the same signal value within different STFT, for
different i. For example, for N = 8, R = 2 and n0 = 0 we will get the value
x (0) for m = 0 and i = 0, but also for m = −2 and i = 1 or m = 2 and i = −1,
and son on. Then in the reconstruction we should use all these values to get
the most reliable reconstruction.
Let us reindex the reconstructed signal values (9.27) by substitution
m = l − iR
1 N/2−1
w(l − iR) x (n0 + l ) = ∑ STFT (n0 + iR, k)e j2πlk/N e− j2πiRk/N
N k=− N/2
− N/2 ≤ l − iR ≤ N/2 − 1.
x(n)
n
n -N n -N/2 n
0 0 0
w(m) w(m)
m
x(n - N+m)w(m) x(n +m)w(m)
0 0
w(m)
x(n)w(n-n +N/2) x(n0- N/2+m)w(m)

0
m
n
x(n)w(n-n +N)+x(n)w(n-n +N/2)+x(n)w(n-n )

0 0 0
Figure 9.16 Illustration of the STFT calculation with windows overlapping in order to pro-
duce an inverse STFT whose sum will give the original signal within n0 − N ≤ n < n0 .
If R < N then a value of signal x (n0 + l ) will be obtained by inverting
1 N/2−1
w ( l ) x ( n0 + l ) = ∑ STFT (n0 , k)e j2πlk/N
N k=− N/2
but also it will be obtained within the inversions
...
1 N/2−1
w(l − 2R) x (n0 + l ) = ∑ STFT (n0 + 2R, k)e j2πlk/N e− j2π2Rk/N
N k=− N/2
1 N/2−1
w ( l − R ) x ( n0 + l ) = ∑ STFT (n0 + R, k)e j2πlk/N e− j2πRk/N
N k=− N/2
1 N/2−1
w ( l + R ) x ( n0 + l ) = ∑ STFT (n0 − R, k)e j2πlk/N e j2πRk/N
N k=− N/2
1 N/2−1
w(l + 2R) x (n0 + l ) = ∑ STFT (n0 − 2R, k)e j2πlk/N e j2π2Rk/N
N k=− N/2
...
as far as w(l − 2iR), for i = 0, ±1, ±2, ... is within
− N/2 ≤ l − 2iR < N/2.
By summing all reconstructions over i satisfying − N/2 ≤ l − iR ≤ N/2 − 1

we get the reconstructed signal x (n0 + l ). It is undistorted (up to a constant)
if
c(l ) = ∑ w(l − iR) = const. = C (9.28)
i
since
∑ w(l − iR)x(n0 + l ) = Cx(n0 + l )
i
for any n0 and l. Note that ∑i w(l − iR) is a periodic extension of w(l ) with
a period R. If W (e jω ) is the Fourier transform of w(l ) then the Fourier
transform of its periodic extension is equal to the samples of W (e jω ) at
ω = 2πk/R. The condition (9.28) is equivalent to
W (e j2πk/R ) = CNδ(k ) for k = 0, 1, ..., R − 1.
Special cases:
x(n) ..., STFT(n-7,k), STFT(n-3,k), STFT(n+1,k), ...
N/2
w(3) STFT(n-3,0) w(3) x(n-0)
x(n-0) ↓ -4
z
-1
z
w(2) STFT(n-3,1) w(2) x(n-1)
x(n-1) ↓ -4
z
-1
z
w(1) STFT(n-3,2) w(1) x(n-2)
x(n-2) ↓ -4
z
-1
z
w(0) STFT(n-3,3) w(0) x(n-3)
x(n-3) ↓ -4
z
-1 STFT IDFT
z
w(-1) STFT(n-3,4) w(-1) x(n-4) x(n-4)
x(n-4) ↓ +
-1
(DFT)
z
w(-2) STFT(n-3,5) w(-2) x(n-5) x(n-5)
x(n-5) ↓ +
z-1
w(-3) STFT(n-3,6) w(-3) x(n-6) x(n-6)
x(n-6) ↓ +
-1
z
w(-4) STFT(n-3,7) w(-4) x(n-7) x(n-7)
x(n-7) ↓ +
R=N/2=4
Figure 9.17 Signal reconstruction from the STFT for the case N = 8, when the STFT is
calculated with step R = N/2 = 4 and the window satisfies w(m) + w(m − N/2) = 1. This
is the case for the rectangular, Hann(ing), Blackman and triangular windows. The same holds
for the Hamming window up to a constant scaling factor of 1.08.
1. For R = N (nonoverlapping), relation (9.28) is satisfied for the rectan-

gular window, only.
2. For a half of the overlapping period, R = N/2, condition (9.28) is met
for the rectangular, Hann(ing), Hamming, and triangular window.
Realization in this case for N = 8 and R = N/2 = 4 is presented in
Fig.9.17. Signal values with a delay of N/2 = 4 samples are obtained at
the exit. The STFT calculation process is repeated after each 4 samples,
producing blocks of 4 signal samples at the output.
3. The same holds for R = N/2, N/4, N/8, if the values of R are integers.
4. For R = 1, (the STFT calculation in each available time instant), any
window satisfies the inversion relation. In this case we may also use a
simple reconstruction formula, Fig.9.18

( )
1 N/2−1 1 N/2−1 N/2−1
− j2πmk/N
∑ STFT (n, k) = N ∑
N k=−
w(m) x (n + m) ∑ e
N/2 m=− N/2 k =− N/2
= w (0 ) x ( n ).
Very efficient realizations, for this case, are the recursive ones, instead
of the direct DFT calculation, Fig.9.8.
In analysis of non-stationary signals our primary interest is not in
signal reconstruction with the fewest number of calculation points. Rather,
we are interested in tracking signals’ non-stationary parameters, like for
example, instantaneous frequency. These parameters may significantly vary
between neighboring time instants n and n + 1. Quasi-stationarity of signal
within R samples (implicitly assumed when down-sampling by factor of R
is done) in this case is not a good starting point for the analysis. Here, we
have to use the time-frequency analysis of signal at each instant n, without
any down-sampling.
9.2.10 Time-Varying Windows
In general, varying window widths could be used for different time-

frequency points. When Ni changes with ni we have the case of a time-
varying window. Assuming a rectangular window we can write,
Ni /2−1 − j 2π
N mk
STFTNi (ni , k ) = ∑ x ( ni + m ) e i (9.29)
m=− Ni /2
Notation STFTNi (n, k ) means that the STFT is calculated using signal sam-
ples within the window [ni − Ni /2, ni + Ni /2 − 1] for − Ni /2 ≤ k ≤ Ni /2 − 1,
corresponding to an even number of Ni discrete frequencies from −π to π.
For an odd Ni , the summation limits are ±( Ni − 1)/2. Let us restate that
a wide window includes signal samples over a wide time interval, losing
the possibility to detect fast changes in time, but achieving high frequency
resolution. A narrow window in the STFT will track time changes, but with
a low resolution in frequency. Two extreme cases are Ni = 1 when
STFT1 (n, k ) = x (n)
and Ni = M when
STFTM (n, k ) = X (k ),
x(n)
w(3) STFT(n-3,0)
x(n-0)
z-1
w(2) STFT(n-3,1)
x(n-1)
-1
z
w(1) STFT(n-3,2)
x(n-2)
-1
z
w(0) STFT(n-3,3) 1/(Nw(0)) x(n-3)
x(n-3) +
z-1 STFT
w(-1) STFT(n-3,4)
x(n-4)
(DFT)
z-1
w(-2) STFT(n-3,5)
x(n-5)
z-1
w(-3) STFT(n-3,6)
x(n-6)
-1
z
w(-4) STFT(n-3,7)
x(n-7)
Figure 9.18 Signal reconstruction when the STFT is calculated with step R = 1.
where M is the total number of all available signal samples and X (k ) =

DFT{ x (n)}.
In vector notation
STFT Ni (ni ) = W Ni x Ni (ni ),
where STFT Ni (ni ) and x Ni (ni ) are column vectors. Their elements are
STFTNi (ni , k ), k = − Ni /2,..., Ni /2 − 1 and x (ni + m), m = − Ni /2,..., Ni /2 −
1, respectively
STFT Ni (ni ) = [STFTNi (ni , − Ni /2) ...STFTNi (ni , Ni /2 − 1)] T

x Ni (ni ) = [ x (ni − Ni /2) ...x (ni + Ni /2 − 1)] T .
Matrix W Ni is an Ni × Ni DFT matrix with elements
WNi (m, k ) = exp(− j2πmk/Ni ),

where m is the column index and k is the row index of the matrix. The STFT
value STFTNi (ni , k ) is presented as a block in the time-frequency plane of
the width Ni in the time direction, covering all time instants [ni − Ni /2, ni +
Ni /2 − 1] used in its calculation. The frequency axis can be labeled with the
DFT indices p = − M/2, ..., M/2 − 1 corresponding to the DFT frequencies
2π p/M (dots in Fig.9.19). With respect to this axis labeling, the block
STFTNi (ni , k) will be positioned at the frequency 2πk/Ni = 2π (kM/Ni )/M,
i.e., at p = kM/Ni . The block width in frequency is M/Ni DFT samples.
Therefore the block area in time and DFT frequency is always equal to
the number of all available signal samples M as shown in Fig.9.19 where
M = 16.
Example 9.11. Consider a signal x (n) with M = 16 samples. Write the expression
for calculation of the STFT value STFT4 (2, 1) with a rectangular window.
Indicate graphically the region of time instants used in the calculation and
the frequency range in terms of the DFT frequency values included in the
calculation of STFT4 (2, 1)?
⋆The STFT value STFT4 (2, 1) is:
1 2π
STFT4 (2, 1) = ∑ x (2 + m ) e − j 4 m .
m=−2
It uses discrete-time samples of x (n) within

−2 ≤ 2 + m < 1
0 ≤ n ≤ 3.
The frequency term is exp(− j2πm/4). For the DFT of a signal with M = 16
15 2π
X (k ) = ∑ x (n)e− j 16 mk
n =0
k = −8, −7, ... − 1, 0, 1, ..., 6, 7
this frequency would correspond to the term exp(− j2π4m/16). Therefore
k = 1 corresponds to the frequency p = 4 in the DFT. Since the whole fre-
quency range −π ≤ ω < π in the case of Ni = 4 is covered with 4 STFT val-
ues STFT4 (2, −2), STFT4 (2, −1), STFT4 (2, 0), and STFT4 (2, 1) and the same
frequency range in the DFT has 16 frequency samples, it means that each
STFT value calculated with Ni = 4 corresponds to a range of frequencies cor-
responding to 4 DFT values,
k = −2, corresponds to p = −8, −7, −6, −5
k = −1, corresponds to p = −4, −3, −2, −1
k = 0, corresponds to p = 0, 1, 2, 3
k = 1, corresponds to p = 4, 5, 6, 7.
7 7
6 6
S4(2,1) S4(6,1) S4(10,1) S4(14,1)
5 5
S (11,0)
S2(13,0)
S (15,0)
S2(1,0)
S (3,0)
S (5,0)
S (7,0)
S (9,0)
4 4
3 3
2
2 2
S4(2,0) S4(6,0) S4(10,0) S4(14,0)
1 1
0 0
-1 -1
-2 -2
S4(2,-1) S4(6,-1) S4(10,-1) S4(14,-1)
-3 -3
S (11,-1)
S2(13,-1)
S (15,-1)
S2(1,-1)
S (3,-1)
S (5,-1)
S (7,-1)
S (9,-1)
-4 -4
-5 -5
2
-6 -6
S4(2,-2) S (6,-2) S (10,-2) S (14,-2)
4 4 4
-7 -7
-8 -8
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
7 7
S8(12,3)
6 6
S4(4,1) S4(8,1) S4(6,1)
5 5
S8(12,2)
S (11,0)
S2(13,0)
S2(1,0)
S2(1,0)
S (3,0)
4 4
3 3
2
2
S8(12,1)
2 2
S (4,0) S (8,0) S (6,0)
4 4 4
1 1
S8(12,0)
S (14,0)
S (15,0)
0 0
-1 -1
1
S8(12,-1)
-2 -2
S4(4,-1) S4(8,-1) S4(6,-1)
-3 -3
S8(12,-2)
S (11,-1)
S (13,-1)
S2(1,-1)
S2(1,-1)
S (3,-1)
-4 -4
-5 -5
2
S8(12,-3)
2
-6 -6
S4(4,-2) S4(8,-2) S4(6,-2)
-7 -7
S8(12,-4)
-8 -8
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Figure 9.19 The nonoverlapping STFTs with: (a) constant window of the width N = 4, (b)
constant window of the width N = 2, (c)-(d) time-varying windows. Time index is presented
on the horizontal axis, while the DFT frequency index is shown on the vertical axis (the STFT
is denoted by S for notation simplicity).
This discrete-time and the DFT frequency region, 0 ≤ n ≤ 3 and 4 ≤ p ≤ 7, is

represented by a square denoted by S4 (2, 1) in Fig.9.19(a).
In a nonoverlapping STFT, covering all signal samples
x =[ x (0), x (1), ..., x ( M − 1)] T
with STFT Ni (ni ), the STFT should be calculated at n0 = N0 /2, n1 = N0 +

N1 /2, n2 = N0 + N1 + N2 /2,..., nK = M − NK /2. A matrix form for all STFT
values is
⎡ ⎤
W N0 0 ··· 0
⎢ 0 W N1 ··· 0 ⎥
⎢ ⎥
STFT = ⎢ .. .. .. .. ⎥x
⎣ . . . . ⎦
0 0 ··· W NK
−1
STFT = W̃x = W̃W M X, (9.30)
where STFT is a column vector containing all STFT vectors STFT Ni (ni ),
i = 0, 1,..., K, X = W M x is a DFT of the whole signal x (n), while W̃ is a block
matrix (M × M) formed from the smaller DFT matrices W N0 , W N1 , ...,W NK ,
as in (9.29). Since the time-varying nonoverlapping STFT corresponds to a
decimation-in-time DFT scheme, its calculation is more efficient than the
DFT calculation of the whole signal. Illustration of time-varying window
STFTs is shown in Fig.9.19(c), (d). For a signal with M samples, there
is a large number of possible nonoverlapping STFTs with a time-varying
window Ni ∈ {1, 2, 3, ..., M }. The exact number will be derived later.
Example 9.12. Consider a signal x (n) with M = 16 samples, whose values are
x = [0.5, 0.5, −0.25, j0.25, 0.25, − j0.25, −0.25, 0.25, −0.25, 0.25, 0.5, 0.5,
− j0.5, j0.5, 0, −1]. Some of its nonoverlapping STFTs are calculated according
to (9.29) and shown in Fig.9.19. Different representations can be compared
based on the concentration measures, for example,
µ[STFTN (n, k )] = ∑∑ |STFTN (n, k )| = ∥STFT∥1 .

n k
The best STFT representation, in this sense, would be the one with the small-
est µ[STFTN (n, k)]. For the considered signal and its four representations
shown in Fig.9.19 the best representation, according to this criterion, is the
one shown in Fig.9.19(b).
Example 9.13. Consider a signal x (n) with M = 8 samples. Its values are x (0) = 0,
x (1) = 1, x (2) = 1/2, x (3) = −1/2, x (4) = 1/4, x (5) = − j/4, x (6) = −1/4,
and x (7) = j/4.
(a) Calculate the STFTs of this signal with rectangular window of the
widths N = 1, N = 2, N = 4. Use the following STFT definition
N/2−1
STFTN (n, k ) = ∑ x (n + m)e− j2πmk/N .
m=− N/2
For an odd N, the summation limits are ±( N − 1)/2. Calculate STFT1 (n, k)
for n = 0, 1, 2, 3, 4, 5, 6, 7, then STFT2 (n, k ) for n = 1, 3, 5, 7, then STFT4 (n, k)
for n = 2, 6 and STFT8 (n, k) for n = 4. For frequency axis use notation
k = 0, 1, 2, 3, 4, 5, 6, 7.
(b) Assuming that time-varying approach is used in the nonoverlap-

ping STFT calculation, find the total number of possible representations.
(c) Calculate the concentration measure for each of the cases in (b)
and find the representation (nonoverlapping combination of previous STFTs)
when the signal is represented with the smallest number of coefficients. Does
it correspond to the minimum of µ[STFT (n, k)]?
⋆(a) The STFT values are:
– for N = 1
STFT1 (n, 0) = x (n), for all n = 0, 1, 2, 3, 4, 5, 6, 7;
– for N = 2
STFT2 (n, 0) = x (n) + x (n − 1)
STFT2 (1, 0) = 1,
STFT2 (3, 0) = 0,
STFT2 (5, 0) = (1 − j)/4,
STFT2 (7, 0) = (−1 + j)/4
STFT2 (n, 1) = x (n) − x (n − 1)

STFT2 (1, 1) = 1,
STFT2 (3, 1) = −1,
STFT2 (5, 1) = (−1 − j)/4,
STFT2 (7, 1) = (1 + j)/4
– for N = 4 and n = 2, 6
STFT4 (n, 0) = x (n − 2) + x (n − 1) + x (n) + x (n + 1)
STFT4 (2, 0) = 1,
STFT4 (6, 0) = 0
STFT4 (n, 1) = − x (n − 2) + jx (n − 1) + x (n) − jx (n + 1)

STFT4 (2, 1) = (1 + 3j)/2,
STFT4 (6, 1) = 0
STFT4 (n, 2) = x (n − 2) − x (n − 1) + x (n) − x (n + 1)

STFT4 (2, 2) = 0,
STFT4 (6, 2) = 0,
STFT4 (n, 3) = − x (n − 2) − jx (n − 1) + x (n) + jx (n + 1)

STFT4 (2, 3) = (1 − 3j)/2,
STFT4 (6, 3) = −1
M=4.41 M=4.60 M=4.60 M=4.79 M=3.41
M=4.00 M=4.19 M=4.19 M=4.38 M=3.00, Optimal
M=5.41 M=5.60 M=5.60 M=5.79 M=4.41
M=5.00 M=5.19 M=5.19 M=5.38 M=4.00
M=5.51 M=5.70 M=5.70 M=5.89 M=4.51
Figure 9.20 Time-frequency representation in various lattices (grid-lines are shown), with
concentration measure M = µ[SPEC (n, k )] value. The optimal representation, with respect
to this measure, is presented with thicker gridlines. Time axis is n = 0, 1, 2, 3, 4, 5, 6, 7 and the
frequency axis is k = 0, 1, 2, 3, 4, 5, 6, 7.
(b) Now we have to make all possible nonoverlapping combinations of these

transforms and to calculate the concentration measure for each of them. Total
number of combinations is 25. The absolute STFT values are shown in Fig.
9.20, along with measure
µ[STFT (n, k)] = ∑n ∑k |STFT (n, k )|
for each case. (c) By measuring the concentration for all of them, we will get
3π/4
frequency
π/2
π/4
0
0 1 2 3 4 5
time
Figure 9.21 Areas in the time-frequency plane.
that the optimal combination, to cover the time-frequency plane, is
STFT1 (0, 0) = x (0) = 0

STFT1 (1, 0) = x (1) = 1
STFT2 (3, 1) = x (3) − x (2) = −1
STFT2 (3, 0) = x (3) + x (2) = 0
STFT4 (6, 0) = x (4) + x (5) + x (6) + x (7) = 0
STFT4 (6, 1) = − x (4) + jx (5) + x (6) − jx (7) = 0
STFT4 (6, 2) = x (4) − x (5) + x (6) − x (7) = 0
STFT4 (6, 3) = − x (4) − jx (5) + x (6) + jx (7) = −1
with just three nonzero transformation coefficients. It corresponds to the

minimum of µ[SPEC (n, k)].
In this case there is an algorithm for efficient optimal lattice determina-
tion, based on two regions consideration, starting from lattices 1, 19, and 25
from the Fig. 9.20, corresponding to the constant window widths of N = 1,
N = 2, and N = 4 samples.
Example 9.14. Discrete signal x (n) for n = 0, 1, 2, 3, 4, 5 is considered. Time-

frequency plane is divided as presented in Fig. 9.21.
(a) Denote each region in the figure by appropriate coefficient STFTNi (n, k ), where
N is window length, n is the time index, and k is the frequency index.
(b) Write relations for coefficients calculation and write transformation matrix T.
(c) By using the transformation matrix, find √ STFT values√ if signal samples are
x (0) = 2, x (1) = −2, x (2) = 4, x (3) = 3, x (4) = − 3, x (5) = 0.
STFT2(1,1)
STFT (4,2)
3
3π/4
STFT1(2,0)
frequency
π/2 STFT3(4,1)
STFT2(1,0)
π/4
STFT3(4,0)
0
0 1 2 3 4 5
time
Figure 9.22 Denoted areas in the time-frequency plane.
(d) If the STFT coefficients for signal y(n) are
STFT2 (1, 0) = 4, STFT2 (1, 1) = 0

STFT1 (2, 0) = 1, STFT3 (4, 0) = 0
STFT3 (4, 1) = 3, STFT3 (4, 2) = 3
find the signal samples y(n).
⋆(a) Denoted areas are presented in Fig. 9.22. (b) The STFT values are
obtained using
( N −1)/2−1
STFTN (n, k) = ∑ x (n + m)e− j2πmk/N or
m=−( N −1)/2
N/2−1
STFTN (n, k) = ∑ x (n + m)e− j2πmk/N
m=− N/2
for and odd and even number of samples N, respectively. It follows
STFT2 (1, 0) = x (0) + x (1)

STFT2 (1, 1) = − x (0) + x (1)
STFT1 (2, 0) = x (2)
STFT3 (4, 0) = x (3) + x (4) + x (5)

√ √
−1 + j 3 −1 − j 3
STFT3 (4, 1) = x (3) + x (4) + x (5)
2 2
√ √
−1 − j 3 −1 + j 3
STFT3 (4, 2) = x (3) + x (4) + x (5).
2 2
The transformation matrix (where the STFT coefficients are arranged into
column vector S) is
⎡ ⎤
1 1 0 0 0 0
⎢ −1 1 0 0 0 0 ⎥
⎢ ⎥
⎢ 0 0 1 0 0 0 ⎥
⎢ ⎥
T=⎢ 0 0 0 1√ 1 1√ ⎥.
⎢ ⎥
⎢ −1+ j 3 −1− j 3 ⎥
⎣ 0 0 0 2√ 1 2√ ⎦
−1− j 3 −1+ j 3
0 0 0 2 1 2
(c) The STFT coefficients are

⎡ ⎤⎡ ⎤ ⎡ ⎤
1 1 0 0 0 0 2 0
⎢ −1 1 0 0 0 0 ⎥⎢ −2 ⎥ ⎢ −4 ⎥
⎢ ⎥⎢ ⎥ ⎢ ⎥
⎢ 0 0 1 0 0 0 ⎥⎢ ⎥ ⎢ 4 ⎥
⎢
S=⎢ 0
⎥⎢
⎥⎢ √4 ⎥=⎢ ⎥
⎥.
0 0 1√ 1 1√ ⎥ ⎢ 0 √
⎢
⎢
⎥⎢
⎥⎣ √3 ⎥ ⎢ ⎥
⎣ 0 0 0
−1+ j 3
1
−1− j 3
⎦ − 3 ⎦ ⎢ 3+ j3
⎣ − 2√
3 ⎥
⎦
2√ 2√
−1− j 3 −1+ j 3 0 3− j3 3
0 0 0 2 1 2 2
(d) The signal samples y(n) are obtained as T−1 S resulting in

D ET
y (5) y (4) y (3) y (2) y (1) y (0)
D ET
= 2 2 1 −1 2 −1 .
Example 9.15. A discrete signal x (n) is considered for 0 ≤ n < M. Find the number
of the STFTs of this signal with time-varying windows.
(a) Consider arbitrary window widths from 1 to M.
(b) Consider dyadic windows, that is, windows whose width is 2 m ,
where m is an integer, such that 2m ≤ M. In this case find the number of time-
varying window STFTs for M = 1, 2, 3, ..., 15, 16.
⋆(a) Let us analyze the problem recursively. Denote by F ( M) the
number of STFTs for a signal with M samples. It is obvious that F (1) = 1,
that is, for one-sample signal there is only one STFT (signal sample itself).
If M > 1, we can use window with widths k = 1, 2, . . . M, as the first analysis
window. Now let us analyze remaining ( M − k) samples in all possible ways,
so we can write a recursive relation for the total number of the STFTs. If
the first window is one-sample window, then the number of the STFTs is
F ( M − 1). When the first window is a two-sample window, then the total
number of the STFTs is F ( M − 2), and so on, until the first window is the M-
sample window, when F ( M − M) = 1. Thus, the total number of the STFTs
for all cases is
F ( M ) = F ( M − 1) + F ( M − 2) + . . . + F (1) + 1
We can introduce F (0) = 1 (meaning that if there are no signal samples we
have only one way to calculate time-varying window STFT) and obtain
M
F ( M ) = F ( M − 1) + F ( M − 2) + . . . F (1) + F (0) = ∑ F( M − k)
k =1
Now, for M > 1 we can write

M −1 M
F ( M − 1) = ∑ F( M − 1 − k) = ∑ F( M − k)
k =1 k =2
and
M M
F ( M ) − F ( M − 1) = ∑ F ( M − k ) − ∑ F ( M − k ) = F ( M − 1)
k =1 k =2
F ( M) = 2F ( M − 1).
resulting in F ( M) = 2 M−1 .
(b) In a similar way, following the previous analysis, we can write
F ( M ) = F ( M − 20 ) + F ( M − 21 ) + F ( M − 22 ) + · · · + F ( M − 2m )
⌊log2 M⌋
= ∑ F ( M − 2m )
m =0
where ⌊log2 M ⌋ is an integer part of log2 M. Here we cannot write a simple

recurrent relation as in the previous case. It is obvious that F (1) = 1. We can
also assume that F (0) = 1. By unfolding recurrence we will get
F (2) = F (1) + F (0) = 2
F (3) = F (2) + F (1) = 3
F (4) = F (3) + F (2) + F (0) = 6
..
.
The results are presented in the table
M 1 2 3 4 5 6 7 8
F( M) 1 2 3 6 10 18 31 56
M 9 10 11 12 13 14 15 16
.
F( M) 98 174 306 542 956 1690 2983 5272
Note that the approximative formula

@ A
F ( M) ≈ 1.0366 · (1.7664) M−1
where [·] is an integer part of the argument, holds, with relative error smaller
then 0.4% for 1 ≤ M ≤ 1024. For example, for M = 16 we have 5272 different
ways to split time-frequency plane into non-overlapping time-frequency
regions.
9.2.11 Frequency-Varying Window
The STFT may use frequency-varying window as well. For a given DFT
frequency pi the window width in time is constant, Fig.9.23
Ni /2−1 − j 2π
N mk i
STFTNi (n, k i ) = ∑ w(m) x (n + m)e i .
m=− Ni /2
For example, value of STFT4 (2, −1) is
2−1
STFT4 (2, −1) = ∑ x (2 + m)e− j2πm(−1)/4 .
m=−2
It position in the time-frequency plane is shown in 9.23(left).

For the signal used to illustrate the frequency-varying STFT in 9.23,
the best concentration (out of the presented four) is the one shown in the
last subplot. Optimization can be done in the same way as in the case of
time-varying windows.
The STFT can be calculated by using the signal’s DFT instead of the
signal. There is a direct relation between the time and the frequency domain
STFT via coefficients of the form exp ( j2πnk/M ). A dual form of the STFT
is:
1 M −1
P(i ) X (k + i )e j2πin/M ,
M i∑
STFT (n, k ) = (9.31)
=0
STFT M (k ) = W− 1 −1
M P M X ( k ).
Frequency domain window P(i ) may be of frequency varying width. This

form is dual to the time-varying form. Forms corresponding to frequency
varying windows, dual to the ones for the time-varying windows, can be
7 7
6 6
S4(2,1) S4(6,1) S4(10,1) S4(14,1)
5 5
S (11,0)
S2(13,0)
S (15,0)
S2(1,0)
S2(3,0)
S (5,0)
S2(7,0)
S2(9,0)
4 4
3 3
2
2
S8(4,1) S8(12,1)
2 2
1 1 S16(8,1)
0 0 S16(8,0)
-1 -1
-2 -2
S4(2,-1) S4(6,-1) S4(10,-1) S4(14,-1)
-3 -3
S (11,-1)
S (13,-1)
S (15,-1)
S (1,-1)
S (3,-1)
S (5,-1)
S2(7,-1)
S (9,-1)
-4 -4
-5 -5
2
-6 -6
S4(2,-2) S4(6,-2) S4(10,-2) S4(14,-2)
-7 -7
-8 -8
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
7 7 S16(8,7)
6 6 S (8,6)
16
5 5
S8(4,2) S8(12,2)
S (11,0)
S2(13,0)
S (15,0)
S2(1,0)
S2(3,0)
S (5,0)
S2(7,0)
S2(9,0)
4 4
3 3
2
S (4,1) S8(12,1)
8
2 2
1 1
0 0
S4(2,0) S (6,0)
4
S (10,0)
4
S (14,0)
4
-1 -1
-2 -2
S4(2,-1) S4(6,-1) S4(10,-1) S4(14,-1)
-3 -3
-4 -4
S4(2,-1) S4(6,-1) S4(10,-1) S4(14,-1)
-5 -5
S8(4,-3) S8(12,-3)
-6 -6
-7 S16(8,-7) -7
S8(4,-4) S8(12,-4)
-8 S16(8,-8) -8
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Figure 9.23 Time-frequency analysis with the STFT using frequency-varying windows.
easily defined, for example, for a rectangular frequency domain window, as
⎡ ⎤
W− 1
N0 0 ··· 0
⎢ 0 W− 1
··· 0 ⎥
⎢ N1 ⎥
STFT = ⎢
⎢ .. .. .. .. ⎥ X,
⎥ (9.32)
⎣ . . . . ⎦
0 0 ··· W−NK
1
where X = [ X (0), X (1), ..., X ( M − 1)] T is the DFT vector.

9.2.12 Hybrid Time-Frequency-Varying Windows
In general, spectral content of signal changes in time and frequency in an ar-

bitrary manner. Combining time-varying and frequency-varying windows
we get hybrid time–frequency-varying windows with STFTN(i,l ) (ni , k l ),
N(i,l ) /2−1
− j N2π mk l
STFTN(i,l ) (ni , k l ) = ∑ w(i,l ) (m) x (ni + m)e (i,l ) (9.33)
m=− N(i,l ) /2
For a graphical representation of the STFT with varying windows, the cor-
responding STFT value should be assigned to each instant n = 0, 1, ..., M − 1
and each DFT frequency p = − M/2, − M/2 + 1, ..., M/2 − 1 within a block.
In the case of a hybrid time–frequency-varying window the matrix form is
obtained from the definition for each STFT value. For example, for the STFT
calculated as in Fig.9.24, for each STFT value an expression based on (9.33)
should be written. Then the resulting matrix STFT can be formed.
There are several methods in the literature that adapt windows or
basis functions to the signal form for each time instant or even for every
considered time and frequency point in the time-frequency plane. Selection
of the most appropriate form of the basis functions (windows) for each time-
frequency point includes a criterion for selecting the optimal window width
(basis function scale) for each point.
9.3 WAVELET TRANSFORM
The first form of functions having the basic property of wavelets was used
by Haar at the beginning of the twentieth century. At the beginning of
1980’s, Morlet introduced a form of basis functions for analysis of seismic
signals, naming them “wavelets”. Theory of wavelets was linked to the
image processing by Mallat in the following years. In late 1980s Daubechies
presented a whole new class of wavelets that can be implemented in a
simple way, by using digital filtering ideas. The most important applications
of the wavelets are found in image processing and compression, pattern
recognition and signal denoising. Here, we will only link the basics of the
wavelet transform to the time-frequency analysis.
Common STFT is characterized by a constant window and constant
time and frequency resolutions for both low and high frequencies. The ba-
sic idea behind the wavelet transform, as it was originally introduced by
Morlet, was to vary the resolution with scale (being related to frequency)
7
STFT8(12,3)
6
STFT4(2,1) STFT4(6,1)
5
STFT8(12,2)
4
3
STFT8(4,1) STFT8(12,1)
2
1 STFT16(8,1)
frequency
0 STFT16(8,0)
-1
STFT8(4,-1)
-2
STFT (10,-1)
4
-3
STFT (13,-1)
STFT (15,-1)
STFT8(4,-2)
-4
2
-5
-6
STFT4(2,-2) STFT4(6,-2) STFT4(10,-2)
-7
-8
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
time
Figure 9.24 A time-frequency varying grid in the STFT calculation.
in such a way that a high frequency resolution is obtained for signal com-
ponents at low frequencies, whereas a high time resolution is obtained for
signal at high frequency components. This kind of resolution change could
be relevant for some practical applications, like for example seismic signals.
It is achieved by introducing a frequency variable window width. Window
width is decreased as frequency increases.
The basis functions in the STFT are
"∞
STFTI I (t, Ω0 ) = x (τ )w(τ − t)e− jΩ0 τ dτ
−∞
# $ "∞
− jΩ0 τ ∗
= x ( τ ), w ( τ − t ) e = ⟨ x (τ ), h (τ − t)⟩ = x (τ )h∗ (τ − t)dτ
−∞
where h(τ − t) = w(τ − t)e jΩ0 τ is a a band-pass signal. It is obtained when

a real-valued window w(τ − t) is modulated by e jΩ0 τ .
When the above idea about wavelet transform is translated into the
mathematical form and related to the STFT, one gets the definition of a
continuous wavelet transform
"∞
1 τ−t
WT (t, a) = , x (τ ) h∗ ( )dτ (9.34)
| a| a
−∞
where h(t) is a band-pass signal, and the parameter a is the scale. This
transform produces a time-scale, rather than the time-frequency signal rep-
resentation. For the Morlet wavelet the relation between the scale and the
frequency is a = Ω0 /Ω. In order to establish a strong formal relationship
between the wavelet transform and the STFT, we will choose the basic Mor-
let wavelet h(t) in the form
h(t) = w(t)e jΩ0 t (9.35)
where w(t) is a window function and Ω0 is a constant frequency. For the

Morlet wavelet we have a modulated Gaussian function
=
1 −αt2 jΩ0 t
h(t) = e e
2π
where the values of α and Ω0 are chosen

√ such that the ratio of h(0) and
the first maximum is 1/2, Ω0 = 2π α/ ln 2. From the definition of h(t) it
is obvious that small Ω (i.e., large a) corresponds to a wide wavelet, i.e., a
wide window, and vice versa. The basic idea of the wavelet transform and
its comparison with the STFT is illustrated in Fig. 9.25.
Substitution of (9.35) into (9.34) leads to a continuous wavelet trans-
form form suitable for a direct comparison with the STFT:
"∞
1 τ − t − jΩ0 τ −t
WT (t, a) = , x (τ )w∗ ( )e a dτ. (9.36)
| a| a
−∞
From the filter theory point of view the wavelet transform, for a given
scale a, could
, be considered as the output of system with impulse response
∗
h (−t/a) | a|, i.e.,
F
WT (t, a) = x (t) ∗t h∗ (−t/a) | a |,
where ∗t denotes a convolution in time. Similarly the STFT, for a given Ω,

may be considered as STFTI I (t, Ω) = x (t) ∗t [w∗ (−t)e jΩt ]. If we consider
Wavelet expansion functions STFT expansion functions
Ω=Ω0/2
a=2
(a) (b)
t t
Ω=Ω0
a=1
(c) (d)
t t
Ω=2Ω0
a=1/2
(e) (f)
t t
Figure 9.25 Expansion functions for the wavelet transform (left) and the short-time Fourier
transform (right). Top row presents high scale (low frequency), middle row is for medium scale
(medium frequency) and bottom row is for low scale (high frequency).
these two band-pass filters from the bandwidth point of view we can see
that, in the case of STFT, the filtering is done by a system whose impulse
response w∗ (−t)e jΩt has a constant bandwidth, being equal to the width of
the Fourier transform of w(t).
Constant Q-Factor Transform: The quality factor Q for a band-pass filter,
as measure of the filter selectivity, is defined as
Central Frequency
Q=
Bandwidth
In the STFT the bandwidth is constant, equal to the window Fourier trans-
form width, Bw . Thus, factor Q is proportional to the considered frequency,
Ω
Q= .
Bw
In the case of the wavelet transform the bandwidth of impulse response is

the width of the Fourier transform of w(t/a). It is equal to B0 /a, where B0
is the constant bandwidth corresponding to the mother wavelet (wavelet in
WT(t,Ω) STFT(t,Ω)
Ω Ω
Ω2 Ω2
Ω1 Ω1
(a) (b)
t1 t2 t t1 t2 t
Figure 9.26 Illustration of the wavelet transform (a) of a sum of two delta pulses and two
sinusiods compared with STFT (b)
scale a = 1). It follows

Ω Ω
Q= = 0 = const.
B0 /a B0
Therefore, the continuous wavelet transform corresponds to the passing a
signal through a series of band-pass filters centered at Ω, with constant
factor Q. Again we can conclude that the filtering, that produces Wavelet
transform, results in a small bandwidth (high frequency resolution and low
time resolution) at low frequencies and wide bandwidth (low frequency and
high time resolution) at high frequencies.
Example 9.16. Find the wavelet transform of signal (9.3)
x (t) = δ(t − t1 ) + δ(t − t2 ) + e jΩ1 t + e jΩ2 t . (9.37)
⋆Its continuous wavelet transform is

1 @ A
WT (t, a) = , w((t1 − t)/a)e− jΩ0 (t1 −t)/a + w((t2 − t)/a)e− jΩ0 (t2 −t)/a
| a|
F @ A
+ | a| e jΩ1 t W [ a(Ω0 /a − Ω1 )] + e jΩ2 t W [ a(Ω0 /a − Ω2 )] . (9.38)
where w(t) is a real-valued function. The transform (9.38) has nonzero values
in the region depicted in Fig. 9.26(a).
In analogy with spectrogram, the scalogram is defined as the squared

magnitude of a wavelet transform:
SCAL (t, a) =| WT (t, a) |2 . (9.39)

The scalogram obviously loses the linearity property, and fits into the cate-
gory of quadratic transforms.
9.3.1 Filter Bank and Discrete Wavelet
This analysis will start by splitting the signal’s spectral content into its high
frequency and low frequency part. Within the STFT framework, this can be
achieved by a two sample rectangular window
w ( n ) = δ ( n ) + δ ( n + 1 ),
with N = 2. A two-sample window STFT is
1 1
STFT (n, 0) = √ ∑ x (n + m)e− j0
2 m =0
1
= √ ( x (n) + x (n + 1)) = x L (n), (9.40)
2
for k = 0, corresponding to low frequency ω = 0 and
1
x H (n) = √ ( x (n) − x (n + 1)) (9.41)
2
for k = 1 corresponding to high frequency ω = π. A time-shifted (anticausal)

version of the STFT
N −1
1
STFT (n, k ) = √ ∑ x (n + m)e− j2πkm/N
N m =0
N/2−1 − j2πkm/N in order to

is used, instead of STFT (n, k ) = ∑m =− N/2 x (n + m )e
remain within the common √ wavelet literature notation. For the same reason
the STFT√ is scaled by N (a form when the DFT and IDFT have the same
factor 1/ N).
This kind of signal analysis leads to the Haar (wavelet) transform. In
the Haar wavelet transform the high-frequency part, x H (n) is not processed
any more. It is kept with this (high) two-samples resolution in time. The res-
olution in time of x H (n, 1) is just slightly (two-times) lower than the original
√
signal sampling interval. The lowpass part x L (n) = ( x (n) + x (n + 1)) / 2
will be further processed. After the signal samples x (n) and x (n + 1) are
processed using (9.40) and (9.41), then next two samples x (n + 2) and
x (n + 3) are analyzed. The highpass part is again calculated x H (n + 2) =
√
( x (n + 2) − x (n + 3)) /√2 and kept as it is. Lowpass part x L (n + 2) =
( x (n + 2) + x (n + 3)) / 2 is considered as a new signal, along with its cor-
responding previous sample x L (n).
Spectral content of the lowpass part of signal is divided, in the same
way, into its low and high frequency part,
1
x LL (n) = √ ( x L (n) + x L (n + 2))
2
1
= [ x (n) + x (n + 1) + x (n + 2) + x (n + 3)]
2
1
x LH (n) = √ ( x L (n) − x L (n + 2))
2
1
= [ x (n) + x (n + 1) − [ x (n + 2) + x (n + 3)]] .
2
The highpass part x LH (n) is left with resolution four in time, while the
lowpass part is further processed in the same way, by dividing spectral
content of x LL (n) and x LL (n + 4) into its low and high frequency part. This
process is continued until the full length of signal is achieved. The Haar
wavelet transformation matrix in the case of signal with 8 samples is
⎡ √ ⎤ ⎡ ⎤⎡ ⎤
2W1 (0, H ) 1 −1 0 0 0 0 0 0 x (0)
⎢ √2W (2, H ) ⎥ ⎢ 0 0 1 −1 0 0 0 0 ⎥ ⎢ x (1) ⎥
⎢ √ 1 ⎥ ⎢ ⎥⎢ ⎥
⎢ ⎥ ⎢
⎢ √2W1 (4, H ) ⎥ ⎢ 0 0 0 0 1 −1 0 0 ⎥ ⎥⎢
⎢ x (2) ⎥
⎥
⎢ ⎥ ⎢
⎢ 2W1 (6, H ) ⎥ ⎢ 0 0 0 0 0 0 1 −1 ⎥ ⎢
⎥
⎢ x (3) ⎥
⎥.
⎢ ⎥=⎢ (9.42)
⎢ 2W2 (0, H ) ⎥ ⎢ 1 1 −1 −1 0 0 0 0 ⎥⎥
⎢
⎢ x (4) ⎥
⎥
⎢ ⎥ ⎢
⎢ 2W2 (4, H ) ⎥ ⎢ 0 0 0 0 1 1 −1 −1 ⎥⎥⎢
⎢ x (5) ⎥
⎥
⎢ √ ⎥ ⎣
⎣ 2 2W4 (0, H ) ⎦ 1 1 1 1 −1 −1 −1 −1 ⎣
⎦ x (6) ⎦
√
2 2W4 (0, L) 1 1 1 1 1 1 1 1 x (7)
This kind of signal transformation was introduced by Haar more than a century
ago . In this notation scale a = 1 values of the wavelet coefficients W1 (2n, H )
are equal to the highpass part of signal calculated using two samples,
W1 (2n, H ) = x H (2n). The scale a = 2 wavelet coefficients are W2 (4n, H ) =
x LH (4n). In scale a = 4 there is only one highpass and one lowpass coef-
ficient at n = 0, W4 (8n, H ) = x LLH (8n) and W4 (8n, L) = x LLL (8n). In this
way any length of signal N = 2m can be decomposed into Haar wavelet
coefficients.
The Haar wavelet transform has a property that its highpass coeffi-
cients are equal to zero if the analyzed signal is constant within the analyzed
time interval, for considered scale. If signal has large number of constant
value samples within the analyzed time intervals, then many Haar wavelet
transform coefficients are zero valued. They can be omitted in signal storage
or transmission. In recovery their values are assumed as zeros and the orig-
inal signal is obtained. The same can be done in the case of noisy signals,
when all coefficients bellow an assumed level of noise can be zero-valued
and the signal-to-noise ratio in the reconstructed signal improved.
9.3.1.1 Lowpass and Highpass Filtering and Downsampling
Although the presented Haar wavelet analysis is quite simple we will use
it as an example to introduce the filter bank framework of the wavelet
transform. Obvious results from the Haar wavelet will be used to introduce
other wavelet forms. For the Haar wavelet calculation two signals x L (n) and
x H (n) are formed according to (9.40) and (9.41), based on the input signal
x (n). Transfer functions of the discrete-time systems producing these two
signals are
1
H L ( z ) = √ (1 + z ) (9.43)
2
1
H H ( z ) = √ (1 − z ) .
2
Frequency responses of these systems assume the form
1 B C
HL (e jω ) = √ 1 + e jω
2
1 B C
jω
HH (e ) = √ 1 − e jω
2
' ' √ ' '
with amplitude characteristics ' HL (e jω )' = 2 |cos(ω/2)|, and ' HH (e jω )' =
√
2 |sin(ω/2)|, presented in Fig.9.27. As expected, they represent a quite
rough forms of lowpass and highpass filters. In general, this principle
is kept for all wavelet transforms. The basic goal for all of them is to
split the frequency content of a signal into its lowpass part and highpass
part providing, in addition, a possibility of simple and efficient signal
reconstruction.
After the values representing lowpass and highpass part of signal √
are obtained, next values of the √ signals x L ( n ) = [ x ( n ) + x ( n + 1 )] / 2
and x H (n) = [ x (n) − x (n + 1)] / 2 are calculated after one time instant is
skipped. Therefore the output signal is downsampled by factor of two. The
2
|HL(ejω)|2+|HH(ejω)|2=2
1.8
1.6 jω
|H (e )|=|DFT{φ (n)}|
L 1
1.4
1.2
0.8
|HH(ejω)|=|DFT{ψ1(n)}|
0.6
0.4
0.2
0
-3 -2 -1 0 1 2 3
Figure 9.27√ Amplitude of the Fourier transform of basic Haar wavelet and scale function
divided by 2.
new downsampled signals will be denoted by
s L (n) = x L (2n)
s H (n) = x H (2n). (9.44)
Downsampling of a signal x (n) to get the signal y(n) = x (2n) is

described in the z-transform domain by the function
1 1
Y (z) = X (z1/2 ) + X (−z1/2 ).
2 2
This relation can easily be verified using the z-transform definition
∞
X (z) = ∑ x (n )z−n
n=−∞
∞ ∞
X (z1/2 ) + X (−z1/2 ) = ∑ x (n)[(z−1/2 )n + (−z−1/2 )n ] = ∑ 2x (2n)z− n
n=−∞ n=−∞
1 1
Z{ x (2n))} = Y (z) = X (z1/2 ) + X (−z1/2 ). (9.45)
2 2
For the signals s L (n) = x L (2n) and s H (n) = x H (2n) the system implementa-
tion is presented in Fig.9.28.
H (z)
H ↓ [X(z1/2)H (z1/2+X(z-1/2 )H (z-1/2 )]/2
H H
2
x(n)
X(z)
H (z)
L ↓ [X(z1/2)H (z1/2+X(z-1/2 )H (z-1/2 )]/2
H H
Figure 9.28 Signal filtering by a low pass and a high pass filter followed by downsaampling
by 2.
If the signals s L (n)and s H (n) are passed through the lowpass and
highpass filters HL (z) and HH (z) and then downsampled,
1 1
S L (z) = HL (z1/2 ) X (z1/2 ) + HL (−z1/2 ) X (−z1/2 )
2 2
1 1
S H (z) = HH (z ) X (z ) + HH (−z1/2 ) X (−z1/2 )
1/2 1/2
2 2
hold.
9.3.1.2 Upsampling
Let us assume that we are not going to transform the signals s L (n) and
s H (n) any more. The only goal is to reconstruct the signal x (n) based on its
downsampled lowpass and highpass part signals s L (n) and s H (n). The first
step in the signal reconstruction is to restore the original sampling interval
of the discrete-time signal. It is done by upsampling the signals s L (n) and
s H ( n ).
Upsampling of a signal x (n) is described by
y(n) = [...x (−2), 0, x (−1), 0, x (0), 0, x (1), 0, x (2), 0, ...] .
Its z-transform domain form is
Y ( z ) = X ( z2 ),
since
∞
X ( z2 ) = ∑ x (n)z−2n = ...x (−1)z2 + 0 · z1 + x (0) + 0 · z−1 + x (1)z−2 + ....
n=−∞
(9.46)
Upsampling of a signal x (n) is defined by
!
x (n/2) for even n
y(n) = = Z −1 { X (z2 ))}.
0 for odd n
If a signal x (n) is downsampled first and then upsampled, the result-

ing signal transform is
1 B 1/2 C2 1 B C2
Y (z) = X( z ) + X (− z1/2 )
2 2
1 1
Y (z) = X (z) + X (−z). (9.47)
2 2
P Q B C
In the Fourier domain it means Y (e jω ) = ( X e jω + X e j(ω +π ) . This form
B C
indicates that an aliasing component X e j(ω +π ) appeared in this process.
9.3.1.3 Reconstruction Condition
In general, when the signal is downsampled and upsampled the aliasing

appears since the component X (−z) exists in addition to the original signal
X (z) in (9.47). The upsampled versions of signals s L (n) and s H (n) should
be appropriately filtered and combined in order to eliminate aliasing. The
conditions to avoid the aliasing in the reconstructed signal will be studied
next.
In the reconstruction process the signals are upsampled (S L (z) →
S L (z2 ) and S H (z) → S H (z2 )) and passed through the reconstruction filters
GL (z) and GL (z) before being added up to form the output signal, Fig.9.29.
2
SH(z)
H (z) ↓ ↑ G (z)
H H
2
x(n) y(n)
+
X(z) Y(z)
2
H (z) ↓ ↑ G (z)
L L
SL(z)
2
Figure 9.29 One stage of the filter bank with reconstruction, corresponding to the one stage
of the wavelet transform realization.
The output signal transforms are
1 1
YL (z) = S L (z2 ) GL (z) = [ HL (z) X (z) + HL (−z) X (−z)] GL (z)
2 2
1 1
YH (z) = S H (z2 ) GH (z) = [ HH (z) X (z) + HH (−z) X (−z)] GH (z)
2 2
Y (z) = YL (z) + YH (z)
1 1
= [ HL (z) GL (z) + HH (z) GH (z)] X (z)
2 2
1 1
+[ HL (−z) GL (z) + HH (−z) GH (z)] X (−z).
2 2
Condition for alias-free reconstruction is
Y ( z ) = X ( z ).
It means that
HL (z) GL (z ) + H H (z) G H (z) = 2 (9.48)

HL (−z) GL (z) + HH (−z) GH (z) = 0. (9.49)
These are general conditions for a correct (alias-free) signal reconstruction.

Based on the reconstruction conditions we can show that the lowpass

filters satisfy
HL (z) GL (z) + HL (−z) GL (−z) = 2 (9.50)

P(z) + P(−z) = 2, (9.51)
where P(z) = HL (z) GL (z).
From (9.49) we may write
HL (−z) GL (z)
GH (z) =
HH (−z)
HL (z) GL (−z)
HH (z) = .
GH (−z)
Second expression is obtained from (9.49) with z being replaced by −z,

when HL (z) GL (−z) + HH (z) GH (−z) = 0. Substituting these values into
(9.48) we get
HL (−z) GL (z) HL (z) GL (−z)

HL (z) GL (z) + =2
HH (−z) GH (−z)
or
HL (z ) GL (z)
[ HH (−z) GH (−z) + HL (−z) GL (−z)] = 2.
HH (−z) GH (−z)
Since the expression within the brackets is equal to 2 (reconstruction condi-
tion (9.48) with z being replaced by −z) then
HL (z ) GL (z)
=1 (9.52)
HH (−z) GH (−z)
and (9.50) follows with
HH (z) GH (z) = HL (−z) GL (−z).
In the Fourier transform domain the reconstruction conditions are
HL (e jω ) GL (e jω ) + HH (e jω ) GH (e jω ) = 2 (9.53)
HL (−e jω ) GL (e jω ) + HH (−e jω ) GH (e jω ) = 0.
9.3.1.4 Orthogonality Conditions
The wavelet transform is calculated using downsampling by a factor 2. One

of the basic requirements that will be imposed to the filter impulse response
for an efficient signal reconstruction is that it is orthogonal to its shifted
version with step 2 (and its multiples). In addition the wavelet functions in
different scales should be orthogonal. Orthogonality of wavelet function in
different scales will be discussed later. The orthogonality condition for the
impulse response is
⟨h L (m), h L (m − 2n)⟩ = δ(n) (9.54)

∑ h L (m)h L (m − 2n) = δ(n).
m
For the Haar wavelet transform this condition is obviously satisfied. In

general, for wavelet transforms when the duration of impulse response
h L (n) is greater than two, the previous relation can be understood as a
downsampled convolution of h L (n) and h L (−n)
r (n) = h L (n) ∗ h L (−n) = ∑ h L (m)h L (m − n),

m
−1
Z{r (n))} = HL (z) HL (z )
' '2
' '
FT{r (n))} = 'HL (e jω )' .
The Fourier transform of the downsampled convolution, for real-valued

h L (n) is, (9.45)
1 '' '2 1 '

' '
'2
'
FT{r (2n))} = 'HL (e jω/2 )' + 'HL (−e jω/2 )' .
2 2
From r (2n) = δ(n) follows

' '2 ' '2
' ' ' '
'HL (e jω )' + 'HL (−e jω )' = 2.
The impulse response is orthogonal, in the sense of (9.54), if the frequency

response satisfies
' '2 ' '2

' ' ' '
'HL (e jω )' + 'HL (e j(ω +π ) )' = 2.
Time domain form of relation (9.50) is
h L (n) ∗ g L (n) + [(−1)n h L (n)] ∗ [(−1)n g L (n)] = 2δ(n)

∑ h L (m) gL (n − m) + ∑(−1)n h L (m) gL (n − m) = 2δ(n)
m m
∑ h L (m) gL (2n − m) = δ(n).

m
If the impulse response h L (n) is orthogonal, as in (9.54), then the last relation
is satisfied for
g L (n) = h L (−n).
In the z-domain it holds

G L ( z ) = H L ( z −1 )
and we may write (9.48) in the form
GL (z) GL (z−1 ) + GL (−z) GL (−z−1 ) = 2 (9.55)
or P(z) + P(−z) = 2 with P(z) = GL (z) GL (z−1 ). Relation (9.48) may also
written for HL (z) as well
HL (z) HL (z−1 ) + HL (−z) HL (−z−1 ) = 2.
9.3.1.5 FIR Filter and Orthogonality Condition
Consider a lowpass anticausal FIR filter of the form
K −1
h L (n) = ∑ hk δ(n + k )
k =0
and the corresponding causal reconstruction filter
K −1
g L (n) = h L (−n) = ∑ hk δ(n − k )
k =0
GL (e jω ) = HL (e− jω )
If the highpass filters are obtained from corresponding lowpass filters by

reversal, in addition to common multiplication by (−1)n , then
g H (n) = (−1)n g L (K − n)
K K
GH (e jω ) = ∑ gH (n)e− jωn = ∑ (−1)n gL (K − n)e− jωn
n =0 n =0
K K
= ∑ (−1)K−m gL (m)e− jω(K−m) = (−1)K e− jωK ∑ e jπm gL (m)e− j(−ω)m
m =0 m =0
− jωK − j(ω −π ) − jωK
= −e GL (e ) = −e GL (−e− jω )
or
GH (e jω ) = −e− jωK GL (−e− jω ) = −e− jωK HL (−e jω )
for GL (e jω ) = HL (e− jω ). Similar relation holds for the anticausal h H (n)
impulse response
h H (n) = (−1)n h L (−K − n).

0 0
HH (e jω ) = ∑ h H (n)e− jωn = ∑ (−1)n h L (−n − K )e− jωn
n=−K n=−K
0
= ∑ (−1)−K−m h L (m)e jω (m+K) = −e jωK HL (−e− jω )
m=−K
The reconstruction conditions are satisfied since, according to (9.48) and

(9.52), a relation corresponding to
HH (z) GH (z) = HL (−z) GL (−z)
holds in the Fourier domain

@ A@ A
HH (e jω ) GH (e jω ) = −e jωK HL (−e− jω ) −e− jωK HL (−e jω )
= HL (−e− jω ) HL (−e jω ) = GL (−e jω ) HL (−e jω ).
In this way all filters are expressed in terms of GL (e jω ) or HL (e jω ).

For example, if GL (e jω ) is obtained using (9.55), with appropriate
design conditions, then
HL (e jω ) = GL (e− jω )
GH (e jω ) = −e− jωK GL (−e− jω )
HH (e jω ) = −e jωK GL (−e jω ). (9.56)
Note that the following symmetry of the frequency response amplitude

functions holds
' ' ' ' ' ' ' '
' ' ' ' ' ' ' '
'HL (e jω )' = 'GL (e− jω )' = 'HH (e j(ω +π ) )' = 'HH (e− j(ω +π ) )' .
The highpass and lowpass response orthogonality
∑ h L (m)h H (m − 2n) = 0
m
∑ gL (m) gH (m − 2n) = 0 (9.57)

m
is also satisfied with these forms of transfer functions for any n. Since
Z{h L (n) ∗ h H (−n)} = HL (z) HH (z−1 )
and Z{ h L (2n) ∗ h H (−2n)} = 0, in the Fourier domain this relation assumes

the form
HL (e jω ) HH (e− jω ) + HL (−e jω ) HH (−e− jω ) = 0.
This identity follows from the second relation in (9.53)
HL (−e jω ) GL (e jω ) + HH (−e jω ) GH (e jω ) = 0
with HH (−e jω ) = e jωK HL (e− jω ), GH (e jω ) = −e− jωK GL (−e− jω ),

and HL (e jω ) = GL (e− jω ) as
GL (−e− jω ) GL (e jω ) − e jωK GL (e jω )e− jωK GL (−e− jω ) = 0.
9.3.1.6 Haar Wavelet Implementation
The condition that the reconstruction filter GL (z) has zero value at z = e jπ =
−1 means that its form is GL (z) = a(1 √+ z−1 ). This form without additional
requirements would produce a = 1/ 2 from the reconstruction relation
GL (z) GL (z−1 ) + GL (−z) GL (−z−1 ) = 2. The time domain filter form is
1
g L (n) = √ [δ(n) + δ(n − 1)] .
2
It corresponds to the Haar wavelet. All other filter functions can be defined
using g L (n) or GL (e jω ).
The same result would be obtained starting from the filter transfer
functions for the Haar wavelet already introduced as
1
H L ( z ) = √ (1 + z )
2
1
H H ( z ) = √ (1 − z ) .
2
The reconstruction filters are obtained from (9.48)-(9.49)
1 1
√ (1 + z ) G L ( z ) + √ (1 − z ) G H ( z ) = 2
2 2
1 1
√ (1 − z ) G L ( z ) + √ (1 + z ) G H ( z ) = 0
2 2
as
1 B C
G L ( z ) = √ 1 + z −1 (9.58)
2
1 B C
G H ( z ) = √ 1 − z −1
2
with
1 1
g L (n) = √ δ(n) + √ δ ( n − 1) (9.59)
2 2
1 1
g H (n) = √ δ(n) − √ δ ( n − 1 ).
2 2
The values impulse responses in the Haar wavelet transform (relations
(9.43) and (9.59)) are:
√ √ √ √
n 2h L (n) 2h H (n) n 2g L (n) 2g H (n)
0 1 1 0 1 1
−1 1 −1 1 1 −1
A detailed time domain filter bank implementation of the reconstruc-
tion process in the Haar wavelet case is described. The reconstruction is
implemented in two steps:
1) The signals s L (n) and s H (n) from (9.44) are upsampled, according
to (9.46), as
r L (n) = [s L (0) 0 s L (1) 0 s L (2) 0 ...s L ( N − 1) 0]

r H (n) = [s H (0) 0 s H (1) 0 s H (2) 0 ...s H ( N − 1) 0]
These signals are then passed trough the reconstruction filters. A sum of the
outputs from these filters is
y(n) = r L (n) ∗ g L (n) + r H (n) ∗ g H (n)

1 1 1 1
= √ r L ( n ) + √ r L ( n − 1) + √ r H ( n ) − √ r H ( n − 1)
2 2 2 2
1
= √ [ x L (0) 0 x L (2) 0 x L (4)....0 x L (2N − 2) 0] +
2
1
+ √ [0 x L (0) 0 x L (2)....0 x L (2N − 2)]
2
1
+ √ [ x H (0) 0 x H (2) 0 x H (4)....0 x H (2N − 2) 0]
2
1
− √ [0 x H (0) 0 x H (2)....0 x H (2N − 2)] .
2
where s L (n) = x L (2n) and s H (n) = x H (2n). From the previous relation
follows
1
y(0) = √ [ x L (0) + x H (0)] = x (0)
2
1
y(1) = √ [ x L (0) − x H (0)] = x (1)
2
...
1
y(2n) = √ [ x L (2n) + x H (2n)] = x (2n)
2
1
y(2n + 1) = √ [ x L (2n) − x H (2n)] = x (2n + 1).
2
A system for implementation of the Haar wavelet transform of a signal
with eight samples is presented in Fig.9.30. It corresponds to the matrix form
realization (9.42).
Example 9.17. For a signal x (n) = [1, 1, 2, 0, 2, 2, 0, 0, 2, 2, 2, 2, 0, 0, 0, 0] calculate
the Haar wavelet transform coefficients, with their appropriate placement in
the time-frequency plane corresponding to a signal with M = 16 samples.
⋆The wavelet transform of a signal with M = 16 samples after the
stage a = 1 is shown in Fig.9.31(a). The whole frequency range is divided
into two subregions, √ denoted by L and H within the coefficients √ W1 (n, L) =
[ x (n) + x (n + 1)] / 2 and W1 (n, H ) = [ x (n) − x (n − 1)] / 2 calculated at
instants n = 0, 2, 3, 6, 8, 10, 12, 14. In the second stage ( a = 2) the highpass re-
gion is not transformed, while the lowpass part s2 (n) = W1 (2n, L) is divided
discrete-time n 0 1 2 3 4 5 6 7
W1(0,H) W1(2,H) W1(4,H) W1(6,H)

H (z) ↓
H scale a=1
2
x(n)
W (0,H) W (4,H)
2 2
H (z) ↓
H scale a=2
2
W (0,H)
4
H (z) ↓ H (z) ↓
L H scale a=3
2 2
H (z) ↓
L
2
W (0,L)
4
H (z) ↓
L
first stage second stage third stage
Figure 9.30 Filter bank for the wavelet transform realization
√
into its lowpass and highpass region√ W2 (n, L) = [s2 (n) + s2 (n + 1)] / 2 and
W2 (n, H ) = [s2 (n) − s2 (n + 1)] / 2, respectively, Fig.9.31(b). The same calcu-
lation is performed in the third and fourth stage, Fig.9.31(c) - (d).
9.3.1.7 Daubechies D4 Wavelet Transform
The Haar wavelet has the duration of impulse response equal to two. In
one stage, it corresponds to a two-sample STFT calculated using a rectan-
gular window. Its Fourier transform presented in Fig.9.27 is quite rough
approximation of a lowpass and highpass filter. In order to improve filter
performance, an increase of the number of filter coefficients should be done.
A fourth order FIR system will be considered. The impulse response of an-
ticausal fourth order FIR filter is h L (n) = [ h L (0), h L (−1), h L (−2), h L (−3)] =
[ h0 , h1 , h2 , h3 ].
15 15
14 14
13 13
W (10,H)
W1(12,H)
W1(14,H)
W (10,H)
W (12,H)
W1(14,H)
W (0,H)
W (2,H)
W (4,H)
W1(6,H)
W1(8,H)
W1(0,H)
W1(2,H)
W (4,H)
W1(6,H)
W (8,H)
12 12
11 11
1
1
1
1
10 10
9 9
8 8
7 7
6 6
W (0,H) W2(4,H) W2(8,H) W2(12,H)
2
5 5
W (10,L)
W1(12,L)
W1(14,L)
W1(0,L)
W (2,L)
W (4,L)
W1(6,L)
W1(8,L)
4 4
1
3 3
1
2 2
W (0,L) W (4,L) W (8,L) W (12,L)
2 2 2 2
1 1
0 0
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
(a) (b)
15 15
14 14
13 13
W (10,H)
W1(12,H)
W1(14,H)
W (10,H)
W (12,H)
W1(14,H)
W1(0,H)
W (2,H)
W (4,H)
W1(6,H)
W1(8,H)
W1(0,H)
W1(2,H)
W (4,H)
W1(6,H)
W (8,H)
12 12
11 11
1
1
1
1
10 10
9 9
8 8
7 7
6 6
W (0,H) W (4,H) W (8,H) W (12,H) W (0,H) W (4,H) W (8,H) W (12,H)
2 2 2 2 2 2 2 2
5 5
4 4
3 3
W (0,H) W (8,H) W (0,H) W (8,H)
3 3 3 3
2 2
1 1 W (0,H)
W (0,L) W (8,L) 4
0 3 3
0 W (0,L)
4
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
(c) (d)
Figure 9.31 Wavelet transform of a signal with M = 16 samples at the output of stages 1, 2, 3
and 4, respectively. Notation Wa (n, H ) is used for the highpass value of coefficient after stage
(scale of) a at an instant n. Notation Wa (n, L) is used for the lowpass value of coefficient after
stage (scale of) a at an instant n.
If the highpass and reconstruction filter coefficients are chosen such

that
n h L (n) h H (n) n g L (n) g H (n)
0 h0 h3 0 h0 h3
−1 h1 − h2 1 h1 − h2 . (9.60)
−2 h2 h1 2 h2 h1
−3 h3 − h0 3 h3 − h0
then relation (9.56) is satisfied with K = 3, since h L (n) = g L (−n), g H (n) =
(−1)n g L (3 − n), and h H (n) = (−1)n g L (n + 3).
The reconstruction conditions
HL (z) GL (z ) + H H (z ) G H (z) = 2
HL (−z) GL (z) + HH (−z) GH (z) = 0
are satisfied if
h20 + h21 + h22 + h23 = 1.
Using the z-transform of the corresponding filters, it follows
HL (z ) GL (z ) + H H (z ) G H (z )
B CB C
= h 0 + h 1 z + h 2 z 2 + h 3 z 3 h 0 + h 1 z −1 + h 2 z −2 + h 3 z −3
B CB C
+ − h 0 z 3 + h 1 z 2 − h 2 z + h 3 − h 0 z −3 + h 1 z −2 − h 2 z −1 + h 3
= 2(h20 + h21 + h22 + h23 ) = 2
and
HL (−z) GL (z) + HH (−z) GH (z)

B CB C
= h 0 − h 1 z + h 2 z 2 − h 3 z 3 h 0 + h 1 z −1 + h 2 z −2 + h 3 z −3
B CB C
+ h0 z3 + h1 z2 + h2 z + h3 −h0 z−3 + h1 z−2 − h2 z−1 + h3 = 0.
For the calculation of impulse response values h0 , h1 , h2 , h3 of a fourth

order system (9.60) four independent equations (conditions) are needed.
We already have three √ conditions. The filter has to satisfyjπzero-frequency
j0
condition HL (e ) = 2, high-frequency condition HL (e ) = 0 and the
reconstruction condition h20 + h21 + h22 + h23 = 1. Therefore one more condition
is needed. In the Daubechies D4 wavelet derivation the fourth condition is
imposed so that the derivative of the filter transfer function at ω = π is equal
to zero
'
dHL (e jω ) ''
' = 0.
dω ω =π
This condition, meaning a smooth approach to zero-value at ω = π, also

guarantees that the output of high-pass filter HH (−z) to the linear input
signal, x (n) = an + b, will be zero. This will be illustrated later. Now we
have a system of four equations,

√ √
h0 + h1 + h2 + h3 = 2 from HL (e j0 ) = 2
h20 + h21 + h22 + h23 = 1 reconstruction condition
h0 − h1 + h2 − h3 = 0 from HL (e jπ ) = 0
'
dHL (e jω ) ''
−h1 + 2h2 − 3h3 = 0 from ' = 0.
dω ω =π
Its solution produces the fourth order Daubechies wavelet coefficients (D4)
n h L (√n) h H (√n) n g L (√n) g H (√n)

1+√ 3 1−√ 3 1+√ 3 1−√ 3
0 0
4 √2 4 2√ 4 √2 4 2√
3+√ 3
−1 − 3−√ 3 1 3+√ 3
− 3−√ 3
4 √2 4√ 2 4 √2 4√ 2
3−√ 3 3+√ 3 3−√ 3 3+√ 3
−2 2
4 √2 4 2√ 4 √2 4 2√
1−√ 3
−3 − 1+√ 3 3 1−√ 3
− 1+√ 3
4 2 4 2 4 2 4 2
Note that this is just one of possible symmetric solutions of the previous
system of equations, Fig.9.32.
The reconstruction conditions for the fourth order FIR filter
HL (e jω ) = h0 + h1 e jω + h2 e j2ω + h3 e j3ω
with Daubechies wavelet coefficients (D4) can also be checked in a graphical

way by calculating
' '2 ' '2
' ' ' '
'HL (e jω )' + 'HL (e j(ω +π ) )' = 2
HL (e j(ω +π ) ) HL∗ (e jω ) + HL (e jω ) HL∗ (e j(ω +π ) ) = 0.
From Fig.9.33, we can see that it is much better approximation of low and
high pass filters than in the Haar wavelet case, Fig.9.27.
Another way to derive Daubechies wavelet coefficients (D4) is in using
relation (9.55)
P(z) + P(−z) = 2
with
P ( z ) = G L ( z ) H L ( z ) = G L ( z ) G L ( z −1 )
Condition imposed on the transfer function GL (z) in D4 wavelet is that its
value and the value of its first derivative at z = −1 are zero-valued (smooth
1 g (n) 1 g (n)
L H
0.5 0.5
0 0
-0.5 -0.5
-4 -3 -2 -1 0 1 2 3 4 -4 -3 -2 -1 0 1 2 3 4
time n time n
1 h (n) 1 h (n)
L H
0.5 0.5
0 0
-0.5 -0.5
-4 -3 -2 -1 0 1 2 3 4 -4 -3 -2 -1 0 1 2 3 4
time n time n
Figure 9.32 Impulse responses of the D4 filters.
2 jω 2 jω 2
|HL(e )| +|HH(e )| =2
1.8
1.6
|H (ejω)|=|DFT{φ (n)}|
L 1
1.4
1.2
0.8 jω
|H (e )|=|DFT{ψ (n)}|
H 1
0.6
0.4
0.2
0
-3 -2 -1 0 1 2 3
Figure 9.33 Amplitude of the Fourier transform of basic Daubechies D4 wavelet and scale
function.
approach to the highpass zero value)

'
'
GL (e jω )' =0
ω =π
'
dGL (e jω ) ''
= 0.
dω 'ω =π
P Q2
Then GL (z) must contain a factor of the form 1 + z−1 . Since the filter
P Q2
order must be even (K must be odd), taking into account that 1 + z−1
would produce a FIR system with 3 nonzero coefficients, then we have to
add at least one factor of the form a(1 + z1 z−1 ) to GL (z). Thus, the lowest
order FIR filter with an even number of (nonzero) impulse response values
is B C2
G L ( z ) = 1 + z −1 a (1 + z 1 z −1 )
with B C2 B C2
P ( z ) = 1 + z −1 1 + z1 R ( z )
where
@ A@ A
R(z) = a(1 + z1 z−1 ) a(1 + z1 z1 ) = z0 z−1 + b + z0 z.
Using
P(z) + P(−z) = 2
only the terms with even exponents of z will remain in P(z) + P(−z)
producing
(4z0 + b)z2 + 8z0 + 6b + (4z0 + b)z−1 = 1

8z0 + 6b = 1
4z0 + b = 0
The solution is z0 = −1/16 and b = 1/4. It produces az1 = z0 = −1/16 and

a2 + z21 = b = 1/4 with
√
1 B √ C 1− 3
a = √ 1 + 3 and z1 = √
4 2 1+ 3
and
* +2 B B
1 √ √ C CB √ B √ C C
R(z) = √ 1+ 3 + 1 − 3 z −1 1 + 3 + 1 − 3 z 1 .
4 2
The reconstruction filter transfer function is

1 B √ B √ C C
G L ( z ) = √ (1 + z −1 )2 1 + 3 + 1 − 3 z −1
4 2
with
1 B √ C B √ C
g L ( n ) = √ [ 1 + 3 δ ( n ) + 3 + 3 δ ( n − 1)
4 2
B √ C B √ C
+ 3 − 3 δ(n − 2) + 1 − 3 δ(n − 3)].
All other impulse responses follow from this one (as in the presented table).
Example 9.18. Consider a signal that is a linear function of time
x (n) = an + b.
Show that the condition
'
dHL (e jω ) ''
−h L (−1) + 2h L (−2) − 3h L (−3) = 0 following from ' =0
dω '
ω =π
is equivalent to the condition that highpass coefficients (output from HH (e jω ))

are zero-valued, Fig.9.33. Show that the lowpass coefficients remain a linear
function of time.
⋆The highpass coefficients after the first stage W1 (2n, H ) are obtained
by downsampling W1 (n, H ) whose form is
W1 (n, H ) = x (n) ∗ h H (n)
= x (n)h H (0) + x (n + 1)h H (−1) + x (n + 2)h H (−2) + x (n + 3)h H (−3)
= x ( n ) h3 − x ( n + 1) h2 + x ( n + 2) h1 − x ( n + 3) h0
= ( an + b)h3 − ((n + 1) a + b)h2 + ((n + 2) a + b)h1 − ((n + 3) a + b)h0
= ( a(n + 3) + b) (−h0 + h1 − h2 + h3 ) − a (h1 − 2h2 + 3h3 ) = 0
if
−h0 + h1 − h2 + h3 = 0 and
h1 − 2h2 + 3h3 = 0.
The lowpass coefficients are obtained by downsampling
W1 (n, L) = x (n) ∗ h L (n)
= x ( n ) h0 + x ( n + 1) h1 + x ( n + 2) h2 + x ( n + 3) h3
= ( an + b)h0 + ((n + 1) a + b)h1 + ((n + 2) a + b)h2 + ((n + 3) a + b)h3
= ( an + b) (h0 + h1 + h2 + h3 ) + a (h1 + 2h2 + 3h3 )
= a1 n + b1
√ √
where a1 = 2a and b1 = 2b + 0.8966a.
Thus we may consider that the highpass D4 coefficients will indicate

the deviation of the signal from a linear function x (n) = an + b. In the first
stage the coefficients will indicate the deviation from the linear function
within four samples. In the next stage the equivalent length of wavelet is
doubled. The highpass coefficient in this stage will indicate the deviation
of the signal from the linear function within doubled number of signal
samples, and so on. This a significant difference from the STFT nature that
is derived based on the Fourier transform and the signal decomposition and
tracking its frequency content.
Example 9.19. Show that with the conditions

√ √
h0 + h1 + h2 + h3 = 2 from HL (e j0 ) = 2
jπ
−h0 + h1 − h2 + h3 = 0 from HL (e ) = 0
the reconstruction condition
h20 + h21 + h22 + h23 = 1
is equivalent to the orthogonality property of the impulse response and its

shifted version for step 2
h0 h1 h2 h3 0 0 0 0
0 0 h0 h1 h2 h3 0 0
given by
h2 h0 + h3 h1 = 0.
⋆If we write the sum of squares of the first two equations follows
2(h20 + h21 + h22 + h23 ) + 4h0 h2 + 4h1 h3 = 2.
Therefore, the conditions
h20 + h21 + h22 + h23 = 1
and
h0 h2 + h1 h3 = 0
√
follow from each other if h0 + h1 + h2 + h3 = 2 and −h0 + h1 − h2 + h3 = 0
are assumed.
The matrix for the D4 wavelet transform calculation in the first stage
is of the form
⎡ ⎤ ⎡ ⎤⎡
⎤
W1 (0, L) h0 h1 h2 h3 0 0 0 x (0 )
0
⎢ W1 (0, H ) ⎥ ⎢ h3 − h2 h1 − h0 0 0 0 ⎥⎢ x (1) ⎥
0
⎢ ⎥ ⎢ ⎥⎢ ⎥
⎢ W1 (2, L) ⎥ ⎢ 0 0 h0 h1 h2 h3 0 ⎥⎢ x (2) ⎥
0
⎢ ⎥ ⎢ ⎥⎢ ⎥
⎢ W1 (2, H ) ⎥ ⎢ 0 0 h3 − h2 h1 − h0 0 ⎥⎢ x (3) ⎥
0
⎢ ⎥=⎢ ⎥⎢ ⎥
⎢ W1 (4, L) ⎥ ⎢ 0 0 0 0 h0 h1 h2 ⎥⎢ x (4) ⎥.
h3
⎢ ⎥ ⎢ ⎥⎢ ⎥
⎢ W1 (4, H ) ⎥ ⎢ 0 0 0 0 h3 − h2 h1 ⎥⎢ x (5) ⎥
− h0
⎢ ⎥ ⎢ ⎥⎢ ⎥
⎣ W1 (6, L) ⎦ ⎣ h2 h3 0 0 0 0 h0 ⎦⎣ x (6) ⎦
h1
W1 (6, H ) h1 − h0 0 0 0 0 h3 − h2
x (7 )
(9.61)
In the first row of transformation matrix the coefficients corresponds to
h L (n), while the second row corresponds to h H (n). The first row produces
D4 scaling function, while the second row produces D4 wavelet function.
The coefficients are shifted for 2 in next rows. As it has been described
in the Hann(ing) window reconstruction case, the calculation should be
performed in a circular manner, assuming signal periodicity. That is why
the coefficients are circularly shifted in the last two rows.
Example 9.20. Consider a signal x (n) = 64 − |n − 64| within 0 ≤ n ≤ 128. How

many nonzero coefficients will be in the first stage of the wavelet transform
calculation using D4 wavelet functions. Assume that the signal can appropri-
ately be extended so that the boundary effects can be neglected.
⋆In the first stage all highpass coefficients corresponding to linear

four-sample intervals will be zero. It means that out of 64 high pass coef-
ficients (calculated with step two in time) only one nonzero coefficient will
exist, calculated for n = 62, including nonlinear interval 62 ≤ n ≤ 65. It means
that almost a half of the coefficients can be omitted in transmission or stor-
age, corresponding to 50% compression ratio. In the DFT analysis this would
correspond to a signal with a half of (the high frequency) spectrum being
equal to zero. In the wavelet analysis this process would be continued with
additional savings in next stages of the wavelet transform coefficients calcu-
lation. It also means that if there is some noise in the signal, we can filter
out all zero-valued coefficients using an appropriate threshold. For this kind
of signal (piecewise linear function of time) we will be able to improve the
signal-to-noise ratio for about 3 dB in just one wavelet stage.
Example 9.21. For the signal x (n) = δ(n − 7) defined within 0 ≤ n ≤ 15 calcu-
late the wavelet transform coefficients using the D4 wavelet/scale function.
Repeat the same calculation for the signal x (n) = 2 cos(16πn/N ) + 1 with
0 ≤ n ≤ N − 1 with N = 16.
⋆The wavelet coefficients in the first stage (scale a = 1, see also

Fig.9.30) are
W1 (2n, H ) = x (2n)h H (0) + x (2n + 1)h H (−1)

+ x (2n + 2)h H (−2) + x (2n + 3)h H (−3)
= x (2n)h3 − x (2n + 1)h2 + x (2n + 2)h1 − x (2n + 3)h0
with √ √ √ √
1− 3 3− 3 3+ 3 1+ 3
[ h3 , h2 , h1 , h0 ] = [ √ , √ , √ , √ ].
4 2 4 2 4 2 4 2
In specific, W1 (0, H ) = 0, W1 (2, H ) = 0, W1 (4, H ) = −0.4830, W1 (6, H ) =
−0.2241, W1 (8, H ) = 0, W1 (10, H ) = 0, W1 (12, H ) = 0, and W1 (14, H ) = 0.
The lowpass part of the first stage values
s2 (n) = W1 (2n, L) = x (2n)h0 + x (2n + 1)h1 + x (2n + 2)h2 + x (2n + 3)h3
are W1 (0, L) = 0, W1 (2, L) = 0, W1 (4, L) = −0.1294, W1 (6, L) = 0.8365, W1 (8, L) =

0, W1 (10, L) = 0, W1 (12, L) = 0, and W1 (14, L) = 0. Values of s2 (n) are defined
for 0 ≤ n ≤ 7 as s2 (n) = −0.1294δ(n − 2) + 0.8365δ(n − 3). This signal is the
input to the next stage (scale a = 2). The highpass output of the stage two is
W2 (4n, H ) = s2 (n)h3 − s2 (n + 1)h2 + s2 (n + 2)h1 − s2 (n + 3)h0 .
The values of W2 (4n, H ) are: W2 (0, H ) = −0.5123, W2 (4, H ) = −0.1708,

W2 (8, H ) = 0, and W2 (12, H ) = 0. The lowpass values at this stage at the input
to the next stage (a = 3) calculation
s3 (n) = W2 (4n, L) = s2 (n)h0 + s2 (n + 1)h1 + s2 (n + 2)h2 + s2 (n + 3)h3 .
They are W2 (0, L) = −0.1373, W2 (4, L) = 0.6373, W2 (8, L) = 0, and W2 (12, L) =

0.
Since there is only 4 samples in s3 (n) this is the last calculation. The
coefficients in this stage are W3 (0, H ) = −0.1251, W3 (8, H ) = −0.4226 and
W3 (0, L) = 0.4668, W3 (8, L) = −0.1132. The absolute value of the wavelet
transform of x (n) with D4 wavelet function is shown in Fig.9.34.
For the signal x (n) = 2 cos(2π8n/N ) + 1 with 0 ≤ n ≤ N − 1 with
N = 16 the same calculation is done. Here it is important to point out that
the circular convolutions should be used. The wavelet transform coefficients
are W1 (2n, L) = 1.4142 and W1 (2n, H ) = 2.8284. Values in the next stage are
W2 (2n, H ) = 0 and W2 (2n, L) = 2. The third stage values are W3 (2n, H ) = 0
and W3 (2n, L) = 2.8284. Compare these results with Fig. 9.26(a). Since the
impulse response duration is 4 and the step is 2 this could be considered as a
kind of signal analysis with overlapping.
15 15
14 14
13 13
W (10,H)
W (12,H)
W1(14,H)
W1(10,H)
W1(12,H)
W (14,H)
W1(0,H)
W1(2,H)
W1(4,H)
W (6,H)
W (8,H)
W1(0,H)
W1(2,H)
W (4,H)
W1(6,H)
W (8,H)
12 12
11 11
1
1
1
1
10 10
9 9
8 8
7 7
6 6
W2(0,H) W2(4,H) W2(8,H) W2(12,H) W2(0,H) W2(4,H) W2(8,H) W2(12,H)
5 5
4 4
3 3
W (0,H) W (8,H) W (0,H) W (8,H)
3 3 3 3
2 2
1 1
W (0,L) W (8,L) W (0,L) W (8,L)
3 3 3 3
0 0
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Figure 9.34 Daubechies D4 wavelet transform (absolute value) of the signal x (n) = δ(n − 7)
using N = 16 signal samples, 0 ≤ n ≤ N − 1 (left). The Daubechies D4 wavelet transform
(absolute value) of the signal x (n) = 2 cos(2π8n/N ) + 1, 0 ≤ n ≤ N − 1, with N = 16 (right).
The inverse matrix for the D4 wavelet transform for a signal with
N = 8 samples would be calculated from the lowest level in this case
for a = 2 with coefficients W2 (0, L), W2 (0, H ), W2 (4, L), and W2 (4, H ). The
lowpass part of signal at level a = 1 would be reconstructed using
⎡ ⎤ ⎡ ⎤⎡ ⎤
W1 (0, L) h0 h3 h2 h1 W2 (0, L)
⎢ W1 (2, L) ⎥ ⎢ h1 − h2 h3 − h0 ⎥ ⎢ W2 (0, H ) ⎥
⎢ ⎥ ⎢ ⎥⎢ ⎥.
⎣ W1 (4, L) ⎦ = ⎣ h2 h1 h0 h3 ⎦ ⎣ W2 (4, L) ⎦
W1 (6, L) h3 − h0 h1 − h2 W2 (4, H )
After the lowpass part W1 (0, L), W1 (2, L), W1 (4, L), and W1 (6, L) are recon-
structed, they are used with wavelet coefficients from this stage W1 (0, H ),
W1 (2, H ), W1 (4, H ), and W1 (6, H ) to reconstruct the signal as
⎡ ⎤ ⎡ ⎤⎡ ⎤
x (0) h0 h3 0 0 0 0 h2 h1 W1 (0, L)
⎢ x (1) ⎥ ⎢ h1 − h2 0 h3 − h0 ⎥ ⎢ ⎥
⎢ ⎥ ⎢ 0 0 0 ⎥⎢ W1 (0, H ) ⎥
⎢ x (2) ⎥ ⎢ h2 h1 h0 h3 0 0 0 ⎥ ⎢
0 ⎥⎢ W1 (2, L) ⎥
⎢ ⎥ ⎢ ⎥
⎢ x (3) ⎥ ⎢ h3 − h0 h1 − h2 0 ⎥ ⎢ ⎥
⎢ ⎥=⎢ 0 0 0 ⎥⎢ W1 (2, H ) ⎥ .
⎢ x (4) ⎥ ⎢ 0 0 h2 h1 h0 h3 0 ⎥ ⎢
0 ⎥⎢ W1 (4, L) ⎥
⎢ ⎥ ⎢ ⎥
⎢ x (5) ⎥ ⎢ 0 0 h3 − h0 h1 − h2 0 ⎥ ⎢ ⎥
⎢ ⎥ ⎢ 0 ⎥⎢ W1 (4, H ) ⎥
⎣ x (6) ⎦ ⎣ 0 0 0 0 h2 h1 h0 h3 ⎦ ⎣ W1 (6, L) ⎦
x (7) 0 0 0 0 h3 − h0 h1 − h2 W1 (6, H )
(9.62)
This procedure can be continued for signal of length N = 16 with one more
stage. Additional stage would be added for N = 32 and so on.
Example 9.22. For the Wavelet transform from the previous example find its
inverse (reconstruct the signal).
⋆The inversion is done backwards. From W3 (0, H ), W3 (0, L), W3 (8, H ),
W3 (8, L) we get signal s3 (n) or W2 (2n, L) as
⎡ ⎤ ⎡ ⎤⎡ ⎤
W2 (0, L) h0 h3 h2 h1 W3 (0, L)
⎢ W2 (4, L) ⎥ ⎢ h1 −h2 h3 −h0 ⎥ ⎢ W3 (0, H ) ⎥
⎢ ⎥ ⎢ ⎥⎢ ⎥
⎣ W2 (8, L) ⎦ = ⎣ h2 h1 h0 h3 ⎦ ⎣ W3 (8, L) ⎦
W2 (12, L) h3 − h0 h1 − h2 W3 (8, H )
⎡ ⎤⎡ ⎤ ⎡ ⎤
h0 h3 h2 h1 0.4668 −0.1373
⎢ h1 −h2 h3 −h0 ⎥ ⎢ −0.1251 ⎥ ⎢ 0.6373 ⎥
=⎢ ⎣ h2
⎥⎢ ⎥=⎢ ⎥.
h1 h0 h3 ⎦ ⎣ −0.1132 ⎦ ⎣ 0 ⎦
h3 − h0 h1 − h2 −0.4226 0
Then W2 (4n, L) = s3 (n) are used with the wavelet coefficients W2 (4n, H ) to
reconstruct W1 (2n, L) or s2 (n) using
⎡ ⎤ ⎡ ⎤⎡ ⎤
W1 (0, L) h0 h3 0 0 0 0 h2 h1 W2 (0, L)
⎢ W1 (2, L) ⎥ ⎢ 0 h3 − h0 ⎥ ⎢ ⎥
⎢ ⎥ ⎢ h1 − h2 0 0 0 ⎥⎢ W2 (0, H ) ⎥
⎢ ⎥ ⎢
W1 (4, L) ⎥ ⎢ h2 h1 h0 h3 0 0 0 0 ⎥⎢
⎥ W2 (4, L) ⎥
⎢ ⎢ ⎥
⎢ W1 (6, L) ⎥ ⎢ h3 − h0 h1 − h2 0 0 0 0 ⎥⎢ W2 (4, H ) ⎥
⎢ ⎥=⎢ ⎥⎢ ⎥.
⎢ W1 (8, L) ⎥ ⎢ 0 h2 h1 h0 h3 0 ⎥⎢ W2 (8, L) ⎥
⎢ ⎥ ⎢ 0 0 ⎥⎢ ⎥
⎢ ⎥ ⎢
W1 (10, L) ⎥ ⎢ 0 0 h3 − h0 h1 − h2 0 0 ⎥⎢
⎥ W2 (8, H ) ⎥
⎢ ⎢ ⎥
⎣ W1 (12, L) ⎦ ⎣ 0 0 0 0 h2 h1 h0 h3 ⎦⎣ W2 (12, L) ⎦
W1 (14, L) 0 0 0 0 h3 − h0 h1 − h2 W2 (12, H )
The obtained values W1 (n, L) with the wavelet coefficients W1 (n, H ) are used
to reconstruct the original signal x (n). The transformation matrix in this case
is of 16 × 16 order and it is formed using the same structure as the previous
transformation matrix.
9.3.1.8 Daubechies D4 Wavelet Functions in Different Scales
Although the wavelet realization can be performed using the same basic
function presented in the previous section, here we will consider the equiva-
lent wavelet function h H (n) and equivalent scale function h L (n) in different
scales. To this aim we will analyze the reconstruction part of the system. As-
sume that in the wavelet analysis of a signal only one coefficient is nonzero.
Also assume that this nonzero coefficient is at the exit of all lowpass fil-
ters structure. It means that the signal is equal to the basic scale function in
δ (n) GL(z)
φ0(n)=hL(n)
↑ GL(z)
φ1(n)
2
0 GH(z) ↑ GL(z)
2
φ2(n)
↑ GH(z)
↑ GH(z)
Figure 9.35 Calculation of the upsampled scale function.
the wavelet analysis. The scale function can be found in an inverse way, by
reconstructing signal corresponding to this delta pulse like transform. The
system of reconstruction filters is shown in Fig.9.35. Note that this case and
coefficient in the Haar transform would correspond to W4 (0, L) = 1 in (9.42)
or in Fig.9.30. The reconstruction process consists of signal upsampling and
passing it trough the reconstruction stages. For example, the output of the
third reconstruction stage has the z-transform
Φ2 ( z ) = G L ( z ) G L ( z2 ) G L ( z4 ).
In the time domain the reconstruction is performed as
φ0 (n) = δ(n) ∗ g L (n) = g L (n)

φ1 (n) = [φ0 (0) 0 φ0 (1) 0 φ0 (2) 0 φ0 (3)] ∗ g L (n)
φ2 (n) = [φ1 (0) 0 φ1 (1) 0 ... φ1 (8) 0 φ1 (9)] ∗ g L (n)
....
φa+1 (n) = ∑ φa ( p) g L (n − 2p)
p
where g L (n) is the four sample impulse response (Daubechies D4 coeffi-

cients). Duration of the scale function φ1 (n) is (4 + 3) + 4 − 1 = 10 samples,
while the duration of φ2 (n) is 19 + 4 − 1 = 22 samples. The scale function for
0 G (z)
L
↑ G (z)
L
ψ (n)
1
2
δ (n) G (z)
H
ψ (n)=h (n) ↑ G (z)
L
0 H
2
ψ (n)
2
↑ G (z)
H
↑ G (z)
H
Figure 9.36 Calculation of the upsampled wavelet function
different scales a (exists of different reconstruction stages) are is presented

in Fig.9.37. Normalized values φa (n)2( a+1)/2 are presented. The amplitudes
are scaled by 2(a+1)/2 in order to keep their values within the same range
for various a.
In a similar way the wavelet function ψ(n) is calculated. The mother
wavelet is obtained in the wavelet analysis of a signal when only one
nonzero coefficient exists at the highpass of the lowest level of the signal
analysis. To reconstruct the mother wavelet the reconstruction system as
in Fig.9.36 is used. The values of ψ(n) are calculated: using the values of
g H (n) at the first input, upsampling it and passing trough the reconstruction
system with g L (n), to obtain ψ1 (n) and repeating this procedure for the next
steps. The resulting z-transform is:
Ψ ( z ) = G H ( z ) G L ( z 2 ) G L ( z4 ).
In the Haar transform (9.42) and Fig.9.30 this case would correspond to
W4 (0, H ) = 1.
Calculation in the time of the wavelet function in different scales is

done using
ψ0 (n) = δ(n) ∗ g H (n) = g H (n)

ψ1 (n) = [ψ1 (0) 0 ψ1 (1) 0 ψ1 (2) 0 ψ1 (3)] ∗ g L (n)
ψ2 (n) = [ψ2 (0) 0 ψ2 (1) 0 ... ψ2 (8) 0 ψ2 (9)] ∗ g L (n)
....
ψa+1 (n) = ∑ ψa ( p) g L (n − 2p)
p
Different scales of the wavelet function, are presented in Fig.9.37.

Values are normalized using ψa (n)2( a+1)/2 .
Wavelet function are orthogonal in different scales, with correspond-
ing steps, as well. For example, it is easy to show that
⟨ψ0 (n − 2m), ψ1 (n)⟩ = 0
since
( )
⟨ψ0 (n − 2m), ψ1 (n)⟩ = ∑ g H ( p) ∑ gH (n − 2m) gL (n − 2p) =0
p n
for any p and m according to (9.57).

Note that the wavelet and scale function in the last row are plotted
as the continuous functions. The continuous wavelet transform (CWT) is
calculated by using the discretized versions of the continuous functions.
However in contrast to the discrete wavelet transform whose step in time
and scale change is strictly defined, the continuous wavelet transform can
be used with various steps and scale functions.
Example 9.23. In order to illustrate the procedure it has been repeated for
the Haar wavelet when g L (n) = [1 1] and g H (n) = [1 −1]. The results are
9.3.1.9 Daubechies D6 Wavelet Transform
The results derived for Daubechies D4 wavelet transform can be extended

to higher order polynomial functions. Consider a sixth order FIR system
h L (n) = [ h L (0), h L (−1), h L (−2), h L (−3), h L (−4), h L (−5)]

= [ h0 , h1 , h2 , h3 , h4 , h5 ].
Daubechies scaling function D4 Daubechies wavelet D4
1 1
0 0
-1 -1
0 10 20 30 40 0 10 20 30 40
1 1
0 0
-1 -1
0 10 20 30 40 0 10 20 30 40
1 1
0 0
-1 -1
0 10 20 30 40 0 10 20 30 40
1 1
0 0
-1 -1
0 10 20 30 40 0 10 20 30 40
1 1
0 0
-1 -1
0 1 2 3 0 1 2 3
Figure 9.37 The Daubechies D4 wavelet scale function and wavelet calculated using the filter
bank relation in different scales: a = 0 (first row), a = 1 (second row), a = 2 (third row), a = 3
(fourth row), a = 10 (fourth row-approximation of a continuous domain). The amplitudes are
scaled by 2(a+1)/2 to keep them within the same range. Values ψa (n)2(a+1)/2 and φa (n)2(a+1)/2
are presented.
Haar scaling function Haar wavelet
1 1
0 0
-1 -1
0 5 10 15 0 5 10 15
1 1
0 0
-1 -1
0 5 10 15 0 5 10 15
1 1
0 0
-1 -1
0 5 10 15 0 5 10 15
1 1
0 0
-1 -1
0 5 10 15 0 5 10 15
Figure 9.38 The Haar wavelet scale function and wavelet calculated using the filter bank
relation in different scales. Values are normalized 2 (a+1)/2 .
√
In addition to the conditions HL (e j0 ) = 2 and HL (e jπ ) = 0, written as
√
h0 + h1 + h2 + h3 + h4 + h5 = 2
h0 − h1 + h2 − h3 + h4 − h5 = 0,
the orthogonality conditions
h0 h2 + h1 h3 + h2 h4 + h3 h5 = 0
h0 h4 + h1 h5 = 0,
are added. Since the filter order is 6 then two orthogonality conditions must
be used. One for shift 2 and the other for shift 4.
The linear signal cancellation condition is again used as
−h1 + 2h2 − 3h3 + 4h4 − 5h5 = 0.
The final condition in the Daubechies D6 wavelet transform is that the

quadratic signal cancellation is achieved for highpass filter, meaning
B C' '
' d2 ∑5n=0 hn e jωn '' 5 '
d2 HL (e jω ) '' ' 2 jωn '
' = ' = − ∑ n hn e ' = 0.
dω 2 ω =π dω 2 ' n =0
'
ω =π ω =π
This condition is of the form
−h1 + 22 h2 − 32 h3 + 42 h4 − 52 h5 = 0
From the set of five equations the Daubechies D6 wavelet transform

coefficients are obtained as
h L (n) = [1.1411, 0.4705, 0.6504, 0.0498, −0.1208, −0.1909].
This is one of possible symmetric solutions of the previous system. From

the definition it is obvious that the highpass coefficients will be zero as far
as the signal is of quadratic nature within the considered interval. These
coefficients can be used as a measure of the signal deviation from the
quadratic form in each scale.
Implementation is the same as in the case of Haar or D4 wavelet
transform. Only difference is in the filter coefficients form.
This form can be also derived from the reconstruction conditions
and the fact that the transfer function GL (z) contains a factor of the form
(1 + z−1 )3 since z = −1 is its third order zero, according to the assumptions.
9.3.1.10 Coifflet Transform
In the Daubechies D6 wavelet transform the last condition is introduced so

that the output of high-pass filter is zero when the input signal is quadratic.
Another way to form filter coefficients for a six sample wavelet is to intro-
duce the condition that the first moment of the scale function is zero, instead
of the second order moment of the wavelet function. In this case symmetric
form of coefficients should be used in the definition
√
h L (−2) + h L (−1) + h L (0) + h L (1) + h L (2) + h L (3) = 2
h2L (−2) + h2L (−1) + h2L (0) + h2L (1) + h2L (2) + h2L (3) = 1
−2h L (−2) + h L (−1) − h L (1) + 2h L (2) − 3h L (3) = 0
h L (−2)h L (0) + h L (−1)h L (1) + h L (0)h L (2) + h L (1)h L (3) = 0
h L (−2)h L (2) + h L (−1)h L (3) = 0.
The first-order moment of h L (n) is
−2h L (−2) − h L (−1) + h L (1) + 2h L (2) + 3h L (3) = 0
This is so called sixth order coifflet transform. Its coefficients are

√ √
h(−2) = ( 2 − 14)/32,
√ √
h(−1) = (−11 2 + 14)/32,
√ √
h(0) = (7 2 + 14)/16,
√ √
h(1) = (− 2 − 14)/16,
√ √
h(2) = ( 2 − 14)/32,
√ √
h(3) = (−3 2 + 14)/32.
9.3.1.11 Discrete Wavelet Transform - STFT
Originally the wavelet transform was introduced by Morlet as a frequency

varying STFT. Its aim was to analyze spectrum of the signal with varying
resolution in time and frequency. Higher resolution in frequency was re-
quired at low frequencies, while at high frequencies high resolution in time
was the aim, for specific analyzed seismic signals.
The Daubechies D4 wavelet/scale function is derived from the con-
dition that the highpass coefficients of a signal with linear change in time
(x (n) = an + b) are zero-valued. Higher order Daubechies wavelet/scale
functions are derived by increasing the order of the signal polynomial
changes. Frequency of a signal does not play any direct role in the discrete-
wavelet transform definition using Daubechies functions. In this sense it
would be easier to relate the wavelet transform to the linear (D4) and higher
order interpolations of functions (signals), within the intervals of various
lengths (corresponding to various wavelet transform scales), than to the
spectral analysis where the harmonic basis functions play the central role.
Example 9.24. Consider a signal x (n) with M = 16 samples, 0 ≤ n ≤ M − 1. Write

the Daubechies D4 wavelet transform based decomposition of this signal that
will divide the frequency axis into four equal regions.
⋆In the STFT a 4−point (N −point) signal would be used to calculate

4 (or N) coefficients of the frequency plane. The wavelet transform divides
the time-frequency plane into two regions (high and low) regardless of the
number of the signal values (wavelet transform coefficients) being used.
If the Haar wavelet is used in Fig.9.39 then by dividing both highpass
bands and lowpass bands in the same way the short-time Walsh-Hadamard
transform with 4-sample nonoverlapping calculation would be obtained. In
the cases of Daubechies 4D wavelet transform, a kind of short time analysis
with the Daubechies functions would be obtained. For the Daubechies D4
function the scale 2 functions:
φ1 (n) = h LL (n) = [ h L (0) 0 h L (1) 0 h L (2) 0 h L (3)] ∗ h L (n) (9.63)

ϕ1 (n) = h LH (n) = [ h H (0) 0 h H (1) 0 h H (2) 0 h H (3)] ∗ h L (n)
ψ1 (n) = h HL (n) = [ h L (0) 0 h L (1) 0 h L (2) 0 h H (3)] ∗ h H (n)
κ1 (n) = h HH (n) = [ h H (0) 0 h H (1) 0 h H (2) 0 h H (3)] ∗ h H (n) (9.64)
would be used to calculate W (4n, 0), W (4n, 1), W (4n, 2), and W (4n, 3),
Fig.9.40. The asymmetry of the frequency regions is visible.
Note that the STFT analysis of this case, with a Hann(ing) window
of N = 8 and calculation step R = 4 will result in the same number of
instants, however the frequency range will be divided in 8 regions, having
a finer grid. This grid is redundant with respect to the signal and to the
wavelet transform. Both, the signal and the wavelet transform have 16 values
(coefficients).
9.3.2 S-Transform
The S-transform (the Stockwell transform) is conceptually a combination of

the STFT analysis and wavelet analysis. It employs a common window, as
in the STFT, with a frequency variable length as in the wavelet transform.
The frequency-dependent window function produces a higher frequency
resolution at lower frequencies, while at higher frequencies sharper time
localization can be achieved, the same as in the continuous wavelet case.
For a signal x (t) it reads as
discrete-time n 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
HL(z) ↓ W(0,3) W(4,3) W(8,3) W(12,3)

2
H (z) ↓
H
2
H (z) ↓ W(0,2) W(4,2) W(8,2) W(12,2)
H
2
x(n)
H (z) ↓ W(0,2) W(4,2) W(8,2) W(12,2)

H
2
H (z) ↓
L
2
H (z) ↓ W(0,0) W(4,0) W(8,0) W(12,0)
L
2
Figure 9.39 Full coverage of the time-frequency plane using the filter bank calculation and
systems with impulse responses corresponding to the wavelet transformation.
+∞
" 2 2
|Ω| − (τ −t) 2 Ω
Sc (t, Ω) = x (τ )e 8π e− jΩτ dτ, (9.65)
(2π )3/2
−∞
with substitutions τ − t → τ, the above equation can be rewritten as follows

+∞
"
|Ω| e− jΩt −τ
2 Ω2
Sc (t, Ω) = x (t + τ )e 8π 2 e− jΩτ dτ. (9.66)
(2π )3/2
−∞
For the window function of form
|Ω| − τ2 Ω22
w(τ, Ω) = e 8π , (9.67)
(2π )3/2
the definition of the continuous S-transform can be rewritten as follows
+∞
"
− jΩt
Sc (t, Ω) = e x (t + τ )w(τ, Ω)e− jΩτ dτ. (9.68)
−∞
Daubechies functions D4 Spectral form of Daubechies functions D4
1 4
0 2
-1
0
0 2 4 6 8 10 -1 -0.5 0 0.5 1
1 4
0 2
-1
0
0 2 4 6 8 10 -1 -0.5 0 0.5 1
1 4
0 2
-1
0
0 2 4 6 8 10 -1 -0.5 0 0.5 1
1 4
0 2
-1
0
0 2 4 6 8 10 -1 -0.5 0 0.5 1
Figure 9.40 Daubechies functions: Scaling function (first row), Mother wavelet function
(second row), Function producing the low-frequency part in the second stage of the high
frequency part in the first stage (third), Function producing the high-frequency part in the
second stage of the high frequency part in the first stage (fourth). Time domain forms of the
functions are left while its spectral content is shown on the right.
A discretization over τ of (9.68) results in the discrete form of S-transform
Sd (t, Ω) = e− jΩt ∑ x (t + n∆t)w(n∆t, Ω)e− jΩn∆t ∆t. (9.69)

n
It may be considered as a STFT with frequency-varying window.

9.4 LOCAL POLYNOMIAL FOURIER TRANSFORM
After the presentation of the wavelet transform we will shift back our
attention to the frequency of the signal, rather than to its amplitude values.
There are signals whose instantaneous frequency variations are known up
to an unknown set of parameters. For example, many signals could be
expressed as polynomial-phase signals
2 +a 3 +···+ a N +1 )
x (t) = Ae j(Ω0 t+a1 t 2t Nt
where the parameters Ω0 , a1 , a2 , . . . , a N are unknown. For nonstationary

signals, this approach may be used if the nonstationary signal could be
considered as a polynomial phase signal within the analysis window. In
that case, the local polynomial Fourier transform (LPFT) may be used. It is
defined as
"∞
2 +Ω 3 +···+ Ω N +1 )
LPFTΩ1 ,Ω2 ,...,Ω N (t, Ω) = x (t + τ )w(τ )e− j(Ωτ +Ω1 τ 2τ Nτ dτ.
−∞
(9.70)
In general, parameters Ω1 , Ω2 , . . . , Ω N could be time dependent, that is, for
each time instant t, the set of optimal parameters could be different.
Realization of the LPFT reduces to the local signal x (t + τ ) demodula-
2 3 N +1
tion by e− j(Ω1 τ +Ω2 τ +···+Ω N τ ) followed by the STFT calculation.
Example 9.25. Consider the second-order polynomial-phase signal
2
x ( t ) = e j ( Ω0 t + a1 t ) .
Show that its LPFT could be completely concentrated along the instantaneous
frequency.
⋆Its LPFT has the form
"∞
2
LPFTΩ1 (t, Ω) = x (t + τ )w(τ )e− j(Ωτ +Ω1 τ ) dτ
−∞
"∞
2 2
= e j ( Ω0 t + a1 t )
w(τ )e− j(Ω−Ω0 −2a1 t)τ e− j(Ω1 − a1 )τ dτ. (9.71)
−∞
For Ω1 = a1 , the second-order phase term does not introduce any distortion
to the local polynomial spectrogram,
' '
' LPFTΩ = a (t, Ω)'2 = |W (Ω − Ω0 − 2a1 t)|2 ,
1 1
with respect to the spectrogram of a sinusoid with constant frequency. For a

wide window w(τ ), like in the case of the STFT of a pure sinusoid, we achieve
high concentration.
The LPFT could be considered as the Fourier transform of win-

dowed signal demodulated with exp( − j(Ω1 τ 2 + Ω2 τ 3 + · · · + Ω N τ N +1 )).
Thus, if we are interested in signal filtering, we can find the coefficients
Ω1 , Ω2 , . . . , Ω N , demodulate the signal by multiplying it with exp (− j(Ω1 τ 2 +
Ω2 τ 3 + · · · + Ω N τ N +1 )) and use a standard filter for almost a pure sinu-
soid. In general, we can extend this approach to any signal x (t) = e jφ(t)
by estimating its phase φ(t) with φ X(t) (using the instantaneous frequency
estimation that will be discussed later) and filtering demodulated signal
x (t) exp(− jφ X(t)) by a lowpass filter. The resulting signal is obtained when
the filtered signal is returned back to the original frequencies, by modula-
tion with exp( jφ X(t)).
Example 9.26. Consider the first-order LPFT of a signal x (t). Show that the second-
order moments of the LPFT could be calculated based on the windowed
signal moment, windowed signal’s Fourier transform moment and one more
LPFT moment for any Ω1 in (9.70), for example for Ω1 = 1.
⋆The second-order moment of the first-order LPFT,
"∞
2
LPFTΩ1 (t, Ω) = xt (τ )e− j(Ωτ +Ω1 τ ) dτ,
−∞
defined by
"∞ ' '2
1
MΩ1 = Ω2 ' LPFTΩ1 (t, Ω)' dΩ (9.72)
2π
−∞
is equal to
' B C' 2
"∞ '' d xt (τ )e− jΩ1 τ 2 ''
MΩ1 = ' ' dτ,
' dτ '
−∞
' '
2
since the LPFT could be considered as the Fourier transform of xt (τ )e− jΩ1 τ ,
2
that is, LPFTΩ1 (t, Ω) = FT{ xt (τ )e− jΩ1 τ }, and the Parseval’s theorem is used.
After the derivative calculation
"∞ '' '2
'
MΩ 1 = ' dxt (τ ) − j2Ω1 τxt (τ )' dτ =
' dτ '
−∞
"∞ '' '
dx (τ ) ''2 dx (τ ) dx ∗ (τ )
('' t ' + j2Ω1 τxt∗ (τ ) t − j2Ω1 τxt (τ ) t + |2Ω1 τxt (τ )|2 )dτ.
dτ dτ dτ
−∞
We can recognize some of the terms in the last line, as

"∞ '' '2
' "∞ ' '2
M0 = ' dxt (τ ) ' dτ = 1 Ω2 ' LPFTΩ1 =0 (t, Ω)' dΩ.
' dτ ' 2π
−∞ −∞
This is the moment of Xt (Ω) = FT{ xt (τ )}, since the integral of |dxt (τ )/dτ |2
over τ is equal to the integral of | jΩXt (Ω)|2 over Ω, according to Parseval’s
theorem. Also, we can see that the last term in MΩ1 contains the signal
moment,
"∞
mx = τ 2 | xt (τ )|2 dτ, (9.73)
−∞
multiplied by 4Ω21 . Then, it is easy to conclude that
"∞ * +
d[ xt (τ )] d[ x ∗ (τ )]
MΩ1 − M0 − 4m x Ω21 = Ω1 j2τxt∗ (τ ) − j2τxt (τ ) t dτ.
dτ dτ
−∞
Note that the last integral does not depend on parameter Ω1 . Thus, the
relation among the LPFT moments at any two Ω1 , for example, Ω1 = a and
an arbitrary Ω1 , easily follows as the ratio
MΩ1 = a − M0 − 4a2 m x a
= . (9.74)
MΩ1 − M0 − 4Ω21 m x Ω1
With a = 1, by leaving the notation for an arbitrary Ω1 unchanged, we get
M1 − M0 − 4m x 1
= , (9.75)
MΩ1 − M0 − 4Ω21 m x Ω1
with M1 = MΩ1 =1 .
Obviously, the second-order moment, for any Ω1 , can be expressed as
a function of other three moments. In this case the relation reads
MΩ1 = 4Ω21 m x + Ω1 ( M1 − M0 − 4m x ) + M0 .
Example 9.27. Find the position and the value of the second-order moment min-
imum of the LPFT, based on the windowed signal moment, the windowed
signal’s Fourier transform moment, and the LPFT moment for Ω1 = 1.
⋆The minimal value of the second-order moment (meaning the best
concentrated LPFT in the sense of the duration measures) could be calculated
from
dMΩ1
=0
dΩ1
as
M1 − M0 − 4m x
Ω1 = − .
8m x
Since m x > 0 this is a minimum of the function MΩ1 . Thus, in general, there
is no need for a direct search for the best concentrated LPFT over all possible
values of Ω1 . It can be found based on three moments.
The value of MΩ1 is
( M1 − M0 − 4m x )2
MΩ1 = M0 − . (9.76)
16m x
Note that any two moments, instead of M0 and M1 , could be used in
the derivation.
The fractional Fourier transform easily reduces to the first-order LPFT.
9.4.1 Fractional Fourier Transform with Relation to the LPFT
The fractional Fourier transform (FRFT) for an angle α (α ̸= kπ) is defined

as
"∞
Xα ( u ) = x (τ )Kα (u, τ )dτ, (9.77)
−∞
where =
1 − j cot α j(u2 /2) cot α j(τ2 /2) cot α − juτ csc α
Kα (u, τ ) = e e e . (9.78)
2π
It can be considered as a rotation of signal in the time-frequency plane for
an angle α. Its inverse can be considered as a rotation for angle −α
"∞
x (t) = Xα (u)K−α (u, t)du.
−∞
√
Special cases of the FRFT reduce to: X0 (u) = x (u) and Xπ/2 (u) = X (u)/ 2π,
that is, the signal and its Fourier transform.
The windowed FRFT is
F "∞
1− j cot α j(u2 /2) cot α 2 /2) cot α
Xw,α (t, u) = 2π e x (t + τ )w (τ )e j(τ e− juτ csc α dτ.
−∞
(9.79)
Relation between the windowed FRFT and the first-order LPFT is
=
1 − j cot α j(u2 /2) cot α
Xw,α (t, u) = e LPFTΩ1 (t, Ω) (9.80)
2π
where Ω1 = cot(α)/2 and Ω = u csc(α). Thus, all results can be easily

converted from the first-order LPFT to the windowed FRFT, and vice versa.
That is the reason why we will not present a detailed analysis for this
transform after the LPFT has been presented.
By using a window, local forms of the FRFT are introduced as:
"∞
STFTα (u, v) = Xα (u + τ )w(τ )e− jvτ dτ (9.81)
−∞
"∞
STFTα (u, v) = x (t + τ )w(τ )Kα (u, τ )dτ (9.82)
−∞
meaning that the lag truncation could be applied after signal rotation or
prior to the rotation. Results are similar. A similar relation for the moments,
like (9.75) in the case of LPFT, could be derived here. It states that any FRFT
moment can be calculated if we know just any three of its moments.
9.5 HIGH-RESOLUTION STFT
High-resolution techniques are developed for efficient processing and sepa-

ration of very close sinusoidal signals (in array signal processing, separation
of sources with very close DOAs). Among these techniques the most widely
used are Capon’s method, MUSIC, and ESPRIT. The formulation of high-
resolution techniques could be extended to the time-frequency representa-
tions. Here we will present a simple formulation of the STFT and the LPFT
within Capon’s method framework.
9.5.1 Capon’s STFT
Here we will present the STFT formulation in a common array signal-

processing notation. The STFT of a discrete time signal x (n) in (causal)
notation
N −1
1
STFT (ω, n) = ∑ x (n + m)e− jωm
N n =0
can be written as
1 H
STFT (ω, n) = ŝω (n) = h H x(n) = a (ω )x(n)
N
a H (ω ) = [1 e−iω e−iω2 ...e−iω ( N −1) ] (9.83)
x(n) = [ x (n) x (n + 1) x (n + 2)... x (n + N − 1)] T ,
where T denotes the transpose operation, and H denotes the conjugate and
transpose (Hermitian) operation. Normalization of the STFT with N is done,
as in the robust signal analysis.
The average power of the output signal ŝω (n), over M samples (ergod-
icity over M samples around n is assumed), for a frequency ω, is
1
|ŝω (n)|2
M∑
P(ω ) = (9.84)
n
1 H 1 1
= 2
a (ω ) ∑[x(n)x H (n)]a(ω ) = 2 a H (ω )R̂ x a(ω ),
N M n N
where R̂ x is the matrix defined by
1
x ( n ) x H ( n ).
M∑
R̂ x =
n
The standard STFT (9.83) can be derived based on the following con-
sideration. Find h as a solution of the problem
min{h H h} subject to h H a(ω ) = 1. (9.85)

h
This minimization problem will be explained through the next example.

Example 9.28. Show that the output power of the filter producing s(n) = h H x(n)
is minimized for the input x(n) = Aa(ω ) + ε(n), with respect the input white
noise ε(n), whose autocorrelation function is R̂ε = ρI if h H h is minimal
subject to h H a(ω ) = 1.
⋆The output for the noise only is sε (n) = h H ε(n), while its average
power is
1 1
|h H ε(n)|2 = ∑ h H ε(n)ε H (n)h
M∑n M n
( )
H 1 H
ε(n)ε (n) h =ρ h H h.
M∑
=h
n
Minimization of h H h is therefore equivalent to the output white noise power

minimization.
The condition h H a(ω ) = 1 means that the input in form of a sinusoid
Aa(ω ), at frequency ω, should not be changed, that is, if x(n) = Aa(ω ), then
h H x(n) = h H Aa(ω ) = A.
Thus, the condition h H a(ω ) = 1 means that the estimate is unbiased with
respect to input sinusoidal signal with amplitude A.
The solution of minimization problem (9.85) is
∂
{h H h + λ(h H a(ω ) − 1)} = 0 subject to h H a(ω ) = 1,
∂h H
2h = −λa(ω ) subject to h H a(ω ) = 1
resulting in
a(ω ) 1
h= = a(ω ) (9.86)
a H (ω )a(ω ) N
and the estimate (9.83), which is the standard STFT, follows.
Consider now a different optimization problem, defined by
1
|h H x(n)|2 } subject to h H a(ω ) = 1.
M∑
min{ (9.87)
h n
Two points are emphasized in this optimization problem. First, the

1
weights are selected to minimize the average power M ∑n |h H x(n)|2 of
the output signal of the filter. It means that the filter should give the best
possible suppression of all components of signals-plus-noise components of
the observations as well as a suppression of the components of the desired
signal for all time-instants (minimization of the power of y(n)). Second, by
setting the condition h H a(ω ) = 1, in the considered time instant n the signal
amplitude is preserved at the output.
The optimization problem can be rewritten in the form
1
h H x(n)x H (n)h} subject h H a(ω ) = 1.
M∑
min{
h n
By denoting
1
x ( n ) x H ( n ),
M∑
R̂x =
n
we get
min{h H R̂x h} subject to h H a(ω ) = 1.

h
The constrained minimization
∂
{h H R̂x h + λ(h H a(ω ) − 1)} = 0 subject to h H a(ω ) = 1.
∂h H
gives the solution
−1 λa (ω )
h = −R̂x subject to h H a(ω ) = 1. (9.88)
2
The solution can be written in the form
R̂− 1
x a(ω )
ĥ = , (9.89)
a H (ω )R̂x−1 a(ω )
where
1
x ( n ) x H ( n ).
M∑
R̂x = (9.90)
n
The output signal power, in these cases, corresponds to Capon’s form
of the STFT, defined by
1
|h H x(n)|2 = h H R̂x h
M∑
SCapon (ω ) = (9.91)
n
( )H
R̂−
x
1 a(ω ) R̂− 1
x a(ω )
= R̂ x (9.92)
a H (ω )R̂− 1
x a(ω ) a H (ω )R̂− 1
x a(ω )
1
= . (9.93)
a H (ω )R̂x−1 a(ω )
Note that a H (ω )R̂− 1

x a ( ω ) is a real valued scalar. Along with (9.90), we
can use a sliding window estimate of the autocorrelation matrix in the form
n+K/2
1
x ( p ) x H ( p ),
K + 1 p=n∑
R̂x (n) = (9.94)
−K/2
where K is a parameter defining the width of a symmetric sliding window.

Inserting R̂x (n, K ) instead of R̂x in (9.91) gives the STFT with weights mini-
mizing the output power in (9.87), for the observations in the neighborhood
of the time instant of interest n.
The mean value of this power function, calculated in the neighborhood

of the time n over the window used in (9.94), gives an averaged Capon’s
STFT as follows
1
SCapon (n, ω ) = . (9.95)
a H (ω )R̂− 1
x (n)a(ω )
where n indicates the time instant of the interest and the mean is calculated
over the observations y(n) in the corresponding window.
In the realization the autocorrelation function is regularized by a unity
matrix I thus, we use
n+K/2
1
x( p)x H ( p) + ρI.
K + 1 p=n∑
R̂(n) = (9.96)
−K/2
instead of R̂x (n) for the inverse calculation in (9.95) and (9.91).
9.5.2 MUSIC STFT
In the MUSIC formulation of the high resolution STFT the eigenvalue

decomposition of the autocorrelation matrix (9.96) is used as
n+K/2
1
x( p)x H ( p) + ρI = V H (n)Λ(n)V(n),
K + 1 p=n∑
R̂(n) =
−K/2
R̂−1 (n) = V H (n)Λ−1 (n)V(n).
Note that the Capon spectrogram, using eigenvalues and eigenvectors of

the autocorrelation matrix, can be written as
1
SCapon (n, ω ) =
a H ( ω ) V H ( n ) Λ −1 ( n ) V ( n ) a ( ω )
1
= N
∑ 1
λk |STFTk (n, ω )|2
k =1
where
STFTk (n, ω ) = a H (ω )vk (n)
is the STFT of the kth eigenvector (column) of the autocorrelation matrix
R̂(n), corresponding to the eigenvalue λk . If the signal has N − M com-
ponents then the first N − M largest eigenvalues λk (corresponding to the
smallest values 1/λk ) will represent the signal space (components), and the
remaining M eigenvalues will correspond to the noise space (represented
by ρI in the definition of autocorelation matrix R̂(n)).
If a frequency ω corresponds to a signal component, then all eigenvec-
tors corresponding to the noise space will be orthogonal to that harmonic,
being represented by a H (ω ). It means that the spectrograms of all noise
space only components will be very small at the frequencies corresponding
to the signal frequencies.
The MUSIC STFT is defined based on this fact. It is calculated using
the eigenvectors corresponding to noise space, as
1 1
SMUSIC (n, ω ) = = , (9.97)
a H (ω )V H
M V M a(ω )
N
2
∑ |STFTk (n, ω )|
k = N − M +1
where V M is the eigenvector matrix containing only M eigenvectors corre-

sponding to the M lowest eigenvalues in Λ, representing the space of noise.
In this case the signal has N − M components corresponding to the largest
eigenvalues. A special case with M = 1 is the Pisarenko method.
Example 9.29. Calculate high resolution forms of the spectrogram for two-
component signal whose frequencies ω0 + ∆ω and ω0 − ∆ω may be con-
sidered as constants around the instant of interest n = 128,
x (n) = exp( jn(ω0 + ∆ω )) + exp( jn(ω0 − ∆ω )),
ω0 = 1 and ∆ω = 0.05.
In the STFT calculation use a rectangular window of the width N = 16.
Use 15 samples for averaging (estimation) of the autocorrelation matrix,
as well as its regularization by a 0.0001 · I (corresponding to noise signal
x (n) + ε(n), where ε(n) is complex white noise with variance σε2 = 0.0001).
Assume that signal samples needed for autocorrelation function estimation
are also available.
⋆ Signal values around n = 128 are considered. The STFT is calculated
using N = 16 signal samples
x(128) = [ x (128) x (129) x (130)... x (143)] T
and a rectangular window. The mainlobe with of this window is D = 4π/N =
π/4 = 0.7854. Its will not be able to resolve two components closer than
2∆ω ∼ D/2 = 0.3927. Considered ∆ω = 0.05 is well below this limit. The
STFT is interpolated in frequency up to 2048 samples. The result is shown in
Fig. 9.41(a). Next the autocorrelation matrix
1 128+7
R̂(128) = ∑ x( p)x H ( p) + 0.00001 · I
15 p=128 −7
is estimated using the signal vectors x( p) = [ x ( p) x ( p + 1) x ( p + 2)... x ( p +

15)]. Note that values of signal from x (128 − 7) for p = 128 − 7 up to p =
128 + 7 + 15 are needed for this calculation. Values of vector
a(ω ) = [1 eiω eiω2 ...eiω ( N −1) ] T
are calculated at the frequencies of interest ω = 2πk/2048, for k = 0, 1, 2, ..., 1023.

The Capon’s STFT is then
1 1
SCapon (128, ω ) = = .
a H (ω )R̂−1 (128)a(ω ) 16
∑ 1
λk |STFTk (n, ω )|2
k =1
Its value is presented in Fig. 9.41(b),(d).

The MUSIC spectrogram is obtained by calculating the eigenvectors of
R̂(128) and using only N − 2 eigenvectors corresponding to the noise space
eigenvalues of this matrix (there are 2 signal components)
1 1
SMUSIC (n, ω ) = H V a(ω )
=
a H (ω )V14 14
16
2
∑ |STFTk (n, ω )|
k =3
where V14 is a 14 × 16 matrix containing 14 eigenvectors vk (n), k = 3, 4, ...16,

corresponding to the noise space (2 eigenvectors corresponding to two largest
eigenvalues, being the signal space, are omitted). The STFT of eigenvector
vk (n) is denoted by STFTk (n, ω ). The MUSIC spectrogram is presented in
Fig. 9.41(c),(e).
The case corresponding to one eigenvector being used in the spectro-
gram |STFT16 (n, ω )|2 (a form of Pisarenko spectrogram, when only the low-
est eigenvector is considered as the noise space) is presented in Fig. 9.41(f).
Note that in the case of Pisarenko spectrogram it is sufficient (and required
by its definition) to use only N = 3 window width (number of components
plus one).
Normalized values of all spectrograms are presented in Fig. 9.41.
9.5.3 Capon’s LPFT
With varying coefficients or appropriate signal multiplication, before the

STFT calculation, a local polynomial version of Capon’s transform could be
defined. For example, for a linear frequency-modulated signal of the form
2 +ω
x (n) = Ae j(α0 n 0 n + ϕ0 )
Spectrogram Capon spectrogram (normalized)
1 1
0.5 0.5
0 0
(a) (b)
0 1 2 3 0 1 2 3
Ω Ω
MUSIC spectrogram (normalized) Capon spectrogram (zoomed log scale)
1
-1
10
-2
0.5 10
-3
10
0 -4
(d) 10 (d)
0 1 2 3 0.95 1 1.05
Ω Ω
MUSIC spectrogram (zoomed log scale) Pisarenko spectrogram (zoomed log scale)
-1 -1
10 10
-2 -2
10 10
-3 -3
10 10
-4 -4
10 (e) 10 (f)
0.95 1 1.05 0.95 1 1.05
Ω Ω
Figure 9.41 (a) The standard STFT using a rectangular window N = 16. The STFT is interpo-
lated in frequency up to 2048 samples. (b) Capon’s spectrogram calculated in 2048 frequency
points. (c) MUSIC spectrogram calculated in 2048 frequency points. (d) Capon’s spectrogram
zoomed to the signal components. (e) MUSIC spectrogram zoomed to the signal components.
(f) Pisarenko spectrogram zoomed to the signal components.
we should use (9.95) or (9.91) with a signal of the form

n+K/2
1
xα ( p)x aH ( p)
K + 1 p=n∑
R̂x (n, K, α) =
−K/2
2
with xα ( p) = x( p)e− jαp ,
with α as a parameter. The high-resolution form of the LPFT can be used for
efficient processing of close linear frequency-modulated signals, with the
same rate within the considered interval.
0.5 0.5
0 0
t
t
-0.5 -0.5
(a) (b)
-500 0 500 -500 0 500
Ω Ω
0.5 0.5
0 0
t
-0.5 -0.5
(c) (d)
-500 0 500 -500 0 500
Ω Ω
Figure 9.42 (a) The standard STFT, (b) the LPFT, (c) Capon’s STFT, and (d) Capon’s LPFT-
based representations of two close almost linear frequency-modulated signals.
Example 9.30. The Capon LPFT form is illustrated on an example with a signal
with two close components
x (t) = exp( j128πt(0.55 − t/2) + j5πt3 ) + exp( j128πt(0.45 − t/2) + j5πt3 ),
that in addition to the linear frequency-modulated contained a small disturb-

ing cubic phase term. The considered time interval was −1 ≤ t ≤ 1 − ∆t with
∆t = 2/512, ρ = 0.5, K = 30, and the frequency domain is interpolated eight
times. The standard STFT, LPFT, Capon’s STFT, and Capon’s LPFT-based rep-
resentations are presented in Fig. 9.42.
In general, higher-order polynomial or any other nonstationary signal,

with appropriate parametrization, can be analyzed in the same way.
9.6 WIGNER DISTRIBUTION
The dimensions of the STFT blocks (resolutions) are determined by the

window width. The best STFT for a signal would be the one whose win-
dow form fits the best to the signal’s time-frequency content. Consider,
for example, an important and simple signal such as a linear frequency
Optimal STFT with a Hann window Wigner distribution with a Hann window
Figure 9.43 Optimal STFT (absolute value, calculated with optimal window width) and the
Wigner distribution of a linear frequency modulated signal.
modulated (LFM) chirp. For simplicity of analysis assume that its instan-
taneous frequency (IF) coincides with the time-frequency plane diagonal.
It is obvious that, due to symmetry, both time and frequency resolution
are equally important. Therefore, the best STFT would be the one calcu-
lated by using a constant window whose (equivalent) widths are equal
in time and frequency domain. With such a window both resolutions will
be the same. However, these resolutions could be unacceptably low for
many applications. It means that the STFT, including all of its possible time
and/or frequency-varying window forms, would be unacceptable as a time-
frequency representation of this signal. The overlapping STFT could be used
for better signal tracking, without any effect on the resolution.
A way to improve time-frequency representation of this signal is in
transforming the signal into a sinusoid whose constant frequency is equal to
the instantaneous frequency value of the linear frequency modulated signal
at the considered instant. Then, a wide window can be used, with a high
frequency resolution. The obtained result is valid for the considered instant
only and the signal transformation procedure should be repeated for each
instant of interest.
A simple way to introduce this kind of signal representation is pre-
sented. Consider an LFM signal,
x (t) = A exp( jφ(t)) = A exp( j( at2 /2 + bt + c)).

Its instantaneous frequency changes in time as
Ωi (t) = dφ(t)/dt = at + b.
One of the goals of time-frequency analysis is to obtain a function that will

(in an ideal case) fully concentrate the signal power along its instantaneous
frequency. The ideal representation would be
I (t, Ω) = 2πA2 δ(Ω − Ωi (t)).
For a quadratic function φ(t), it is known that
dφ(t) τ τ
τ = φ(t + ) − φ(t − )
dt 2 2
= τ ( at + b) = τΩi (t).
This property can easily be converted into an ideal time-frequency repre-

sentation for the linear frequency modulated signal by using
FTτ { x (t + τ/2) x ∗ (t − τ/2)} =

FTτ { A2 e jΩi (t)τ } = 2π A2 δ(Ω − Ωi (t)).
The Fourier transform of x (t + τ/2) x ∗ (t − τ/2) over τ, for a given t, is called

the Wigner distribution. It is defined as
"∞
WD (t, Ω) = x (t + τ/2) x ∗ (t − τ/2)e− jΩτ dτ. (9.98)
−∞
The Wigner distribution is originally introduced in quantum mechanics.

The illustration of the Wigner distribution calculation is presented in Fig.
9.44.
Expressing x (t) in terms of X (Ω) and substituting it into (9.98) we get
"∞
1
WD (t, Ω) = X (Ω + θ/2) X ∗ (Ω − θ/2)e jθt dθ (9.99)
2π
−∞
what represents a definition of the Wigner distribution in the frequency

domain.
x(t) considered instant t t

WD(t,Ω)
x(t+τ/2)
τ
x(t- τ/2)
τ
x(t+τ/2)x*(t- τ/2)
τ
FT{x(t+τ/2)x*(t- τ/2)}
Ω Ω
Figure 9.44 Illustration of the Wigner distribution calculation, for a considered time instant t.
Real values of a linear frequency modulated signal (linear chirp) are presented.
It is easy to show that the Wigner distribution satisfies the marginal

properties. From the Wigner distribution definition, it follows
"∞
∗ 1
x (t + τ/2) x (t − τ/2) = IFT{WD (t, Ω)} = WD (t, Ω)e jΩτ dΩ (9.100)
2π
−∞
which, for τ = 0, produces (9.126)
"∞
2 1
| x (t)| = WD (t, Ω)dΩ. (9.101)
2π
−∞
Based on the definition of the Wigner distribution in the frequency domain,

(9.99), one may easily prove the fulfillment of the frequency marginal.
Example 9.31. Find the Wigner distribution of signals: (a) x (t) = δ(t − t1 ) and (b)
x (t) = exp( jΩ1 t).
⋆ The Wigner distribution of signal x (t) = δ(t − t1 ) is

"∞
WD (t, Ω) = δ(t − t1 + τ/2)δ(t − t1 − τ/2)e− jΩτ dτ
−∞
= 2δ(2(t − t1 ))e− j2Ω(t−t1 ) = δ(t − t1 ),
since | a| δ( at) x (t) = δ(t) x (0). From the Wigner distribution definition in
terms of the Fourier transform, for x (t) = exp( jΩ1 t) with X (Ω) = 2πδ(Ω −
Ω1 ), follows
WD (t, Ω) = 2πδ(Ω − Ω1 ).
A high concentration of time-frequency representation for both of these
signals is achieved. Note that this fact does not mean that we will be able
to achieve an arbitrary high concentration simultaneously, in a point, in the
time-frequency domain.
2
Example 9.32. Consider a linear frequency modulated signal, x (t) = Ae jbt /2 . Find
its Wigner distribution.

⋆ In this case we have
x (t + τ/2) x ∗ (t − τ/2) = | A|2 e jbtτ
with
WD (t, Ω) = 2π | A|2 δ(Ω − bt).
Again, a high concentration along the instantaneous frequency in the time-
frequency plane may be achieved for the linear frequency modulated signals.
These two examples demonstrate that the Wigner distribution can

provide superior time-frequency representation of one-component signal,
in comparison to the STFT.
Example 9.33. Calculate the Wigner distribution for a linear frequency modulated
signal, with Gaussian amplitude (Gaussian chirp signal)
2
/2 j(bt2 /2+ct)
x (t) = Ae− at e .
⋆ For the chirp signal, the local autocorrelation function reads as

2 2
R(t, τ ) = x (t + τ/2) x ∗ (t − τ/2) = | A|2 e− at e− aτ /4 e jbtτ + jcτ .
The Wigner distribution is obtained as the Fourier transform of R(t, τ ),

=
2 π − (Ω−bt−c)2
WD (t, Ω) = 2| A|2 e− at e a . (9.102)
a
The Wigner distribution from the previous example is obtained with c = 0

√ 2
and a → 0, since 2 π/ae−Ω /a → 2πδ(Ω) as a → 0.
The Wigner distribution of the Gaussian chirp signal is always positive,
as it could be expected from a distribution introduced with the aim to
represent local density of signal energy. Unfortunately, this is the only signal
when the Wigner distribution is always positive, for any point in the time-
frequency plane (t, Ω). This drawback is not the only reason why the study of
time-frequency distributions does not end with the Wigner distribution.
9.6.1 Auto-Terms and Cross-Terms in the Wigner Distribution
For the multi-component signal
M
x (t) = ∑ xm (t)
m =1
the Wigner distribution has the form
M "∞ B
M
τC ∗B τ C − jΩτ
WD (t, Ω) = ∑ ∑ xm t +
2
xn t −
2
e dτ.
m=1 n=1−∞
Besides the auto-terms

M "∞
τ ∗ τ
WDat (t, Ω) = ∑ xm (t + ) x (t − )e− jΩτ dτ,
m=1−∞
2 m 2
the Wigner distribution contains a significant number of cross-terms,
M M "∞
τ ∗ τ
WDct (t, Ω) = ∑ ∑ xm (t + ) x (t − )e− jΩτ dτ.
m =1 n =1 − ∞
2 n 2
n̸=m
Usually, they are not desirable in the time-frequency signal analysis. Cross-
terms can mask the presence of auto-terms, which makes the Wigner distri-
bution unsuitable for the time-frequency analysis of signals.
For a two-component signal with auto-terms located around (t1 , Ω1 )
and (t2 , Ω2 ) (see Fig.9.45) the oscillatory cross-terms are located around
((t1 + t2 )/2, (Ω1 + Ω2 )/2).
Example 9.34. Analyze auto-terms and cross-terms for two-component signal of
the form
1 2 jΩ1 t 1 2 − jΩ1 t
x ( t ) = e − 2 ( t − t1 ) e
+ e − 2 ( t + t1 ) e
Auto-term
Ω
2
Ω Oscillatory
1
cross-term
Auto-term
0 t t t
1 2
Figure 9.45 Wigner distribution of two component signal.
⋆ In this case we have

√ 2 2 √ 2 2
WD (t, Ω) = 2 πe−(t−t1 ) −(Ω−Ω1 ) + 2 πe−(t+t1 ) −(Ω+Ω1 )
√ 2 2
+ 4 πe−t −Ω cos(2t1 Ω − 2Ω1 t)
where the first and second terms represent auto-terms while the third term
is a cross-term. Note that the cross-term is oscillatory in both directions. The
oscillation rate along the time axis is proportional to the frequency distance
between components 2Ω1 , while the oscillation rate along frequency axis is
proportional to the distance in time of components, 2t1 . The oscillatory nature
of cross-terms will be used for their suppression.
To analyze auto-terms and cross-terms, the well-known ambiguity

function can be used as well. It is defined as:
"∞ B τC ∗B τ C − jθt
AF (θ, τ ) = x t+ x t− e dt. (9.103)
2 2
−∞
It is already a classical tool in optics as well as in radar and sonar signal

analysis.
The ambiguity function and the Wigner distribution form a two-

dimensional Fourier transform pair
AF (θ, τ ) = FT2D
t,Ω {WD (t, Ω )},
⎡ ⎤
"∞ "∞ "∞
WD (t, Ω) =
1 ⎣ x (u + τ ) x ∗ (u − τ )e− jθu du⎦ e jθt− jΩτ dτdθ,
2π 2 2
−∞ −∞ −∞
where the integration over frequency related variable θ assumes factor

1/(2π ) and the positive sign in the exponent exp ( jθt).
Consider a signal whose components are limited in time to
xm (t) ̸= 0 only for |t − tm | < Tm .

∗ (t − τ/2) ̸ = 0 only
In the ambiguity (θ, τ ) domain we have xm (t + τ/2) xm
for
− Tm < t − tm + τ/2 < Tm

− Tm < t − tm − τ/2 < Tm .
It means that xm (t + τ/2) xm ∗ (t − τ/2) is located within |τ | < 2T , i.e.,

m
around the θ-axis independently of the signal’s position tm . Cross-term
between signal’s m-th and n-th component is located within |τ + tn − tm | <
Tm + Tn . It is dislocated from τ = 0 for two components that do not occur
simultaneously, i.e., when tm ̸= tn .
From the frequency domain definition of the Wigner distribution a
corresponding ambiguity function form follows
"∞ * + * +
1 θ θ jΩτ
AF (θ, τ ) = X Ω+ X∗ Ω − e dΩ. (9.104)
2π 2 2
−∞
From this form we can conclude that the auto-terms of the components,
limited in frequency to Xm (Ω) ̸= 0 only for |Ω − Ωm | < Wm , are located
in the ambiguity domain around τ-axis within the region |θ/2| < Wm . The
cross-terms are within
|θ + Ωn − Ωm | < Wm + Wn ,
where Ωm and Ωn are the frequencies around which the Fourier transform
of each component lies.
| AF (θ,τ) |
τ
Cross-term
τ
2
Auto-terms
0
τ
1
Cross-term
θ 0 θ θ
1 2
Figure 9.46 Auto and cross-terms for two-component signal in the ambiguity domain.
Therefore, all auto-terms are located along and around the ambiguity
domain axis. The cross-terms, for the components which do not overlap in
the time and frequency, simultaneously, are dislocated from the ambiguity
axes, Fig. 9.46. This property will be used in the definition of the reduced
interference time-frequency distributions.
The ambiguity function of a four-component signal consisting of two
Gaussian pulses, one sinusoidal and one linear frequency modulated com-
ponent is presented in 9.47.
Example 9.35. Let us consider signals of the form
1 2
x1 ( t ) = e − 2 t
1 2 jΩ1 t 1 2 − jΩ1 t
x 2 ( t ) = e − 2 ( t − t1 ) e
+ e − 2 ( t + t1 ) e
The ambiguity function of x1 (t) is

√ 1 2 1 2
AFx1 (θ, τ ) = πe− 4 τ − 4 θ
while the ambiguity function of two-component signal x2 (t) is

√ 1 2 1 2 √ 1 2 1 2
AFx2 (θ, τ ) = πe− 4 τ − 4 θ e jΩ1 τ e− jt1 θ + πe− 4 τ − 4 θ e− jΩ1 τ e jt1 θ +
√ − 1 (τ −2t )2 − 1 (θ −2Ω )2 √ − 1 (τ +2t )2 − 1 (θ +2Ω )2
πe 4 1 4 1
+ πe 4 1 4 1
AF(θ,τ)
100
50
τ 0
-50
-100
0 1 2 3
-3 -2 -1
θ
Figure 9.47 Ambiguity function of signal from Fig.9.4
In the ambiguity domain (θ, τ ) auto-terms are located around (0, 0) while
cross-terms are located around (2Ω1 , 2t1 ) and (−2Ω1 , −2t1 ) as presented in
Fig. 9.46.
9.6.2 Wigner Distribution Properties
A list of the properties satisfied by the Wigner distribution follows. The

obvious ones will be just stated, while the proofs will be given for more
complex ones. In the case when the Wigner distributions of more than one
signal are considered, the signal will be added as an index in the Wigner
distribution notation. Otherwise signal x (t) is assumed, as a default signal
in the notation.
P1 – Realness
For any signal holds,
WD ∗ (t, Ω) = WD (t, Ω).
P2 – Time-shift property
The Wigner distribution of a signal shifted in time
y ( t ) = x ( t − t0 ),
is
WDy (t, Ω) = WDx (t − t0 , Ω).
P3 – Frequency shift property

For a modulated signal
y(t) = x (t)e jΩ0 t ,
we have
WDy (t, Ω) = WDx (t, Ω − Ω0 ).
P4 – Time marginal property
"∞
1
WD (t, Ω)dΩ = | x (t)|2 .
2π
−∞
P5 – Frequency marginal property
"∞
WD (t, Ω)dt = | X (Ω)|2 .
−∞
P6 – Time moments property
"∞ "∞ "∞

1
tn WD (t, Ω)dtdΩ = tn | x (t)|2 dt.
2π
−∞ −∞ −∞
1
&∞
⋆ This property follows from 2π −∞ WD (t, Ω )dΩ = | x (t)|2 .
P7 -Frequency moments property
"∞ "∞ "∞

n
Ω WD (t, Ω)dΩdt = Ωn | X (Ω)|2 dΩ.
−∞ −∞ −∞
P8 – Scaling
For a scaled version of the signal
F
y(t) = | a| x ( at), a ̸= 0,
the Wigner distribution reads
WDy (t, Ω) = WDx ( at, Ω/a).

P9 – Instantaneous frequency property

For x (t) = A(t)e jφ(t)
&∞
Ω WD (t, Ω) dΩ d
&−∞
∞ = Ωi (t) = arg[ x (t)] = φ′ (t). (9.105)
−∞ WD (t, Ω ) dΩ
dt
⋆ In order to prove this property, we will use the derivative of the

inverse Fourier transform of the Wigner distribution
"∞
d[ x (t + τ/2) x ∗ (t − τ/2)] 1
= jΩ WD (t, Ω)e jΩτ dΩ
dτ 2π
−∞
with x (t) = A(t)e jφ(t) , calculated at τ = 0. It results in
"∞
j 1
Ω WD (t, Ω) dΩ = [ x ′ (t) x ∗ (t) − x (t) x ∗′ (t)] = jφ′ (t) A2 (t).
2π 2
−∞
&∞
With the frequency marginal property −∞ WD (t, Ω ) dΩ = 2πA2 (t), this
property follows.
P10 – Group delay
For signal whose Fourier transform is of the form X (Ω) = | X (Ω)| e jΦ(Ω) ,
the group delay t g (Ω) = −Φ′ (Ω) is
&∞
t WD (t, Ω) dt d
&−∞∞ = t g (Ω) = − arg[ X (Ω)] = −Φ′ (Ω).
−∞ WD (t, Ω ) dt
dΩ
The proof is the same as in the instantaneous frequency case, using the
frequency domain relations.
P11 – Time constraint
If x (t) = 0 for t outside [t1 , t2 ], then WD (t, Ω) = 0 for t outside [t1 , t2 ].
⋆ The Wigner distribution is a function of x (t + τ/2) x ∗ (t − τ/2). If

x (t) = 0 for t outside [t1 , t2 ] then x (t + τ/2) x ∗ (t − τ/2) is different from
zero within
t1 ≤ t + τ/2 ≤ t2 and t1 ≤ t − τ/2 ≤ t2 .
The range of values of t defined by the previous inequalities is t1 ≤ t ≤
t2 .
P12 – Frequency constraint
If X (Ω) = 0 for Ω outside [Ω1 , Ω2 ], then, also WD (t, Ω) = 0 for Ω outside

[ Ω1 , Ω2 ].
P13 – Convolution
"∞
WDy (t, Ω) = WDh (t − τ, Ω)WDx (τ, Ω)dτ.
−∞
for
"∞
y(t) = h(t − τ ) x (τ )dτ,
−∞
P14 – Product
"∞
1
WDy (t, Ω) = WDh (t, Ω − v)WDx (t, v)dv
2π
−∞
for
y ( t ) = h ( t ) x ( t ).
⋆ The local autocorrelation of y(t) is h(t + τ/2)h∗ (t − τ/2) x (t +
τ/2) x ∗ (t − τ/2). Thus, the Wigner distribution of y(t) is the Fourier trans-
form of the product of local autocorrelations h(t + τ/2)h∗ (t − τ/2) and
x (t + τ/2) x ∗ (t − τ/2). It is a convolution in frequency of the corresponding
Wigner distributions of h(t) and x (t). Property P13 could be proven in the
same way using the Fourier transforms of signals h(t) and x (t).
P15 – Fourier transform property
WDy (t, Ω) = WDx (−Ω/c, ct) (9.106)
for F
y(t) = |c|/(2π ) X (ct), c ̸= 0.
⋆ Here the signal y(t) is equal to the scaled version of the Fourier
transform of signal x (t),
"∞ B
|c| cτ C ∗ B cτ C − jΩτ
WDy (t, Ω) = X ct + X ct − e dτ
2π 2 2
−∞
"∞ * + * +
1 θ ∗ θ
= X ct + X ct − e j(−Ω/c)θ dθ. (9.107)
2π 2 2
−∞
Comparing (9.99) to (9.98), with ct → Ω and (−Ω/c) → t, we get
"∞ * + * + * +
Ω τ ∗ Ω τ − jctτ Ω
WDy (t, Ω) = x − + x − − e dτ = WDx − , ct .
c 2 c 2 c
−∞
P16 – Chirp convolution

* +
Ω
WDy (t, Ω) = WDx t − ,Ω (9.108)
c
for F
2
y(t) = x (t) ∗ |c|e jct /2 .
, 2 , 2
⋆ With Y (Ω) = FT{ x (t) ∗t |c|e jct /2 } = 2πjX (Ω)e− jΩ /(2c) and the
signal’s Fourier transform-based definition of the Wigner distribution, proof
of this property reduces to the next one.
P17 – Chirp product
WDy (t, Ω) = WDx (t, Ω − ct)
for
2 /2
y(t) = x (t)e jct .
⋆ The Wigner distribution of y(t) is
"∞ B τ C jc(t+τ/2)2 /2 ∗ B τ C − jc(t−τ/2)2 /2 − jΩτ

x t+ e x t− e e dτ
2 2
−∞
"∞ B τC ∗B τ C jctτ − jΩτ
= x t+ x t− e e dτ = WDx (t, Ω − ct). (9.109)
2 2
−∞
P18 – Moyal property

' '2
"∞ "∞ ' "∞ '
1 ' '
'
WDx (t, Ω)WDy (t, Ω)dtdΩ = ' x (t)y(t)dt'' . (9.110)
2π '−∞ '
−∞ −∞
⋆ This property follows from
"∞ "∞ "∞ B

1 τ C B τ C B τ2 C ∗ B τ2 C
x t + 1 x∗ t − 1 y t + y t−
2π 2 2 2 2
−∞ −∞ −∞
"∞
× e− jΩτ1 e− jΩτ2 dΩ dτ1 dτ2 dt
−∞
"∞ "∞ B τC ∗B τC B τC ∗B τC
= x t+ x t− y t− y t+ dτdt.
2 2 2 2
−∞ −∞
With t + τ/2 = u and t − τ/2 = v, we get

' '2
"∞ "∞ ' "∞ '
' '
= x (u) y∗ (u)du ∗
x (v) y (v) dv = ' x (t)y(t)dt'' .
'
−∞ −∞
'−∞ '
9.6.3 Pseudo and Smoothed Wigner Distribution
In practical realizations of the Wigner distribution, we are constrained with

a finite time lag τ. A pseudo form of the Wigner distribution is then used. It
is defined as
"∞
PWD (t, Ω) = w(τ/2)w∗ (−τ/2) x (t + τ/2) x ∗ (t − τ/2)e− jΩτ dτ (9.111)
−∞
where window w(τ ) localizes the considered lag interval. If w(0) = 1, the
pseudo Wigner distribution satisfies the time marginal property. Note that
the pseudo Wigner distribution is smoothed in the frequency direction with
respect to the Wigner distribution
"∞
1
PWD (t, Ω) = WD (t, θ )We (Ω − θ )dθ
2π
−∞
where We (Ω) is a Fourier transform of w(τ/2)w∗ (−τ/2).

The pseudo Wigner distribution example for multi-component signals
is presented in Fig.9.48. The pseudo Wigner distribution example for multi-
component signals is presented in Fig.9.48. Mono-component case with
PWD (t,Ω)
1
250
200
150
100
t
50
(a)
0 2.5 3
0.5 1 1.5 2
0
Ω
PWD (t,Ω)
2
250
200
150
100
t
50
(b)
0 2.5 3
0.5 1 1.5 2
0
Figure 9.48 Pseudo Wigner distribution of a signals from Fig.9.4
sinusoidally frequency modulated signal is presented in Fig.9.49. Note that

significant inner interferences are present.
Monocomponent case with sinusoidally frequency modulated signal
is presented in Fig.9.49. Note that significant inner interferences are present.
Example 9.36. For a sinusoidally frequency modulated signal
x (t) = exp(− j32 cos(πt/64))
calculate an approximate value of the pseudo Wigner distribution with a

window w(τ ) of the width T = 2.
PWD(t,Ω) PWD(t,Ω)
100 100
50 50
t 0 t 0
-50 -50
-100 (a) -100 (b)
1 2 3 1 2 3
0 0
Ω Ω
Figure 9.49 Pseudo Wigner distribution for sinusoidally frequency modulated signal. Narrow
window (left) and wide window (right).
⋆ The pseudo Wigner distribution of this signal is
"2
PWD (Ω, t) = e j32 cos(π (t−τ/2)/64) e− j32 cos(π (t−τ/2)/64) w(τ )e− jΩτ dτ.
−2
By using the Taylor expansion

B πτ C π
cos πt/64 ± = cos(πt/64) ∓ sin(πt/64)τ
128 128
B π C2 τ 2 B π C3
3
τ1,2
− cos(πt/64) + sin(πt/64) ,
128 2 128 6
with |τ1,2 | ≤ 2 in the Taylor series reminder, we get
"2 π3 τ 3 +τ23
j32 sin(πt/64) 1
WD (Ω, t) = e jπ/2 sin(πt/64)τ e 1283 6 w(τ )e− jΩτ dτ.
−2
' '
' π3 τ 3 +τ 3 '
Obviously, '256 128 sin(πt/64) 1 6 2 ' ≤ 0.081, since |τ1,2 | ≤ 2. Thus, we may
write
PWD (Ω, t) ∼
= W (Ω − π/2 sin(πt/64)),
where W (Ω) is the Fourier transform of window w8 (τ ). For a Hann(ing)
window this approximation holds for wider windows as well, since its values
toward the ending points are small, meaning that the effective window width
is lower than the window width itself.
9.6.4 Discrete Pseudo Wigner Distribution
If the signal in (9.111) is discretized in τ with a sampling interval ∆t, then a

sum instead of an integral is formed. The pseudo Wigner distribution of a
discrete-lag signal, for a given time instant t, is given by
∞ B C B C B C B C
∗ ∗ − jmΩ∆t
PWD (t, Ω) = ∑ w m ∆t
2 w − m ∆t
2 x t + m ∆t
2 x t − m ∆t
2 e ∆t.
m=−∞
(9.112)
Sampling in τ with ∆t = π/Ω0 , Ω0 > Ωm corresponds to the sampling of
signal x (t + τ/2) in τ/2 with ∆t/2 = π/(2Ω0 ).
The discrete-lag pseudo Wigner distribution is the Fourier transform
of signal
* + * + * + * +
∆t ∗ ∆t ∆t ∗ ∆t
R(t, m) = w m w −m x t+m x t−m ∆t.
2 2 2 2
For a given instant t, it can be written as
∞
PWD (t, ω ) = ∑ R(t, m)e− jmω
m=−∞
with ω = Ω∆t. If the sampling interval satisfies the sampling theorem, then
the sum in (9.112) is equal to the integral form (9.111).
A discrete form of the pseudo Wigner distribution, with N + 1 samples
and ω = 2πk/( N + 1), for a given time instant t, is
N/2
PWD (t, k ) = ∑ R(t, m)e− j2πmk/( N +1) .
m=− N/2
Here, N/2 is an integer. This distribution could be calculated by using the

standard DFT routines.
For discrete-time instants t = n∆t, introducing the notation
R(n∆t, m∆t)
* + * + * + * +
∆t ∗ ∆t ∆t ∗ ∆t
=w m w −m x n∆t + m x n∆t − m ∆t
2 2 2 2
BmC B mC B C
m ∗ B m C
R(n, m) = w w∗ − x n+ x n− ,
2 2 2 2
the discrete-time and discrete-lag pseudo Wigner distribution can be written

as
∞ BmC B mC B mC ∗B m C − jmω
∗
PWD (n, ω ) = ∑ w
2
w −
2
x n +
2
x n −
2
e .
m=−∞
(9.113)
Notation x (n + m/2), for given n and m, should be understood as the
signal value at the instant x ((n + m/2)∆t). In this notation, the discrete-time
pseudo Wigner distribution is periodic in ω with period 2π.
Since various discretization steps are used (here and in open litera-
ture), we will provide a relation of discrete indexes to the continuous time
and frequency, for each definition, as
* +
2πk
PWD (t, Ω)|t=n∆t, Ω= 2πk = PWD n∆t, → PWD (n, k).
( N +1)∆t ( N + 1)∆t
The sign → could be understood as the equality sign in the sense of sam-
pling theorem (Example 2.13). Otherwise it should be considered as a corre-
spondence sign. The discrete form of (9.111), with N + 1 samples, is
* +
2πk
PWD n∆t, → PWD (n, k )
( N + 1)∆t
N/2 BmC B mC B mC ∗B m C − j2πkm/( N +1)
PWD (n, k ) = ∑ w w∗ − x n+ x n− e ,
m=− N/2
2 2 2 2
where N/2 is an integer, − N/2 ≤ k ≤ N/2 and ω = Ω∆t = 2πk/( N + 1) or

Ω = 2πk/(( N + 1)∆t).
In order to avoid different sampling intervals in time and lag in the
discrete Wigner distribution definition, the discrete Wigner distribution can
be oversampled in time, as it has been done in lag. It means that the same
sampling interval ∆t/2, for both time and lag axes, can be used. Then, we
can write
* +
∆t
R n , m∆t → R(n, m)
2
* + * + * + * + * +
∆t ∆t ∆t ∆t ∆t ∆t ∆t
R n , m∆t = w m w∗ −m x n +m x∗ n − m ∆t
2 2 2 2 2 2 2
R(n, m) = w(m)w∗ (−m) x (n + m) x ∗ (n − m)
The discrete-time and discrete-lag pseudo Wigner distribution, in this case,

is of the form
∞
PWD (n, ω ) = 2 ∑ w(m)w∗ (−m) x (n + m) x ∗ (n − m)e− j2mω . (9.114)
m=−∞
It corresponds to the continuous-time pseudo Wigner distribution (9.111)

with substitution τ/2 → τ
"∞
PWD (t, Ω) = 2 w(τ )w∗ (−τ ) x (t + τ ) x ∗ (t − τ )e− j2Ωτ dτ.
−∞
The discrete pseudo Wigner distribution is given here by

* +
n∆t 4πk
PWD , → PWD (n, k )
2 ( N + 1)∆t
N/2
PWD (n, k ) = ∑ w(m)w∗ (−m) x (n + m) x ∗ (n − m)e− j4πmk/( N +1)
m=− N/2
(9.115)
for − N/2 ≤ 2k ≤ N/2. Since, the standard DFT routines are commonly
used for the pseudo Wigner distribution calculation, we may use every
other (2k) sample in (9.115) or oversample the pseudo Wigner distribution
in frequency (as it has been done in time). Then,
* +
n∆t 2πk
2 ( N + 1)∆t
N/2
PWD (n, k ) = ∑ w(m)w∗ (−m) x (n + m) x ∗ (n − m)e− j2πmk/( N +1) .
m=− N/2
(9.116)
This discrete pseudo Wigner distribution, oversampled in both time and in

frequency by factor of 2, has finer time-frequency grid, producing smaller
time-frequency estimation errors at the expense of the calculation complex-
ity.
Example 9.37. Signal x (t) = exp( j31πt2 ) is considered within −1 ≤ t ≤ 1. Find the
sampling interval of signal for discrete pseudo Wigner distribution calcula-
tion. If the rectangular window of the width N + 1 = 31 is used in analysis,
find the pseudo Wigner distribution values and estimate the instantaneous
frequency at t = 0.5 based on the discrete pseudo Wigner distribution.
⋆ For this signal the instantaneous frequency is Ωi (t) = 62πt. It is

within the range −62π ≤ Ωi (t) ≤ 62π. Thus, we may approximately as-
sume that the maximal frequency is Ωm = 62π.The sampling interval for the
Fourier transform would be ∆t ≤ 1/62. For the direct pseudo Wigner distri-
bution calculation, it should be twice smaller, ∆t/2 ≤ 1/124. Therefore,√the
discrete version of the pseudo Wigner distribution, normalized with 2 ∆t,
at t = 0.5 or n = 62, is (9.115)
15 2 2
PWD (n, k) = ∑ e j31π ((n+m)/124) e− j31π ((n−m)/124) e− j4πmk/31
m=−15
15 sin( π8 (n − 16k))
= ∑ e jπmn/124 e− j4πmk/31 = π .
m=−15
sin( 248 (n − 16k))
The argument k, when the pseudo Wigner distribution reaches maximum for
n = 62, follows from 62 − 16k = 0 as
! 6 - .
62
k̂ = arg max PWD (n, k ) = = 4,
k 16
where [·] stands for the nearest integer. Obviously, the exact instantaneous
frequency is not on the discrete frequency grid. The estimated value of the in-
stantaneous frequency at t = 1/2 is Ω̂ = 4π k̂/(( N + 1)∆t) = 16π/(31/62) =
32π. The true value is Ωi (1/2) = 31π. When the true frequency is not on
the grid, the estimation can be improved by using the interpolation or dis-
placement bin, as explained in Chapter 1. The frequency sampling inter-
val is ∆Ω = 4π/(( N + 1)∆t) = 8π, with maximal estimation absolute error
∆Ω/2 = 4π.
If we used the standard DFT routine (9.116) with N + 1 = 31 and all
available frequency samples, we would get
M 2 2
N
PWD (n, k ) = DFT31 e j31π ((n+m)/124) e− j31π ((n−m)/124)
15 2 2 sin( π8 (n − 8k))
= ∑ e j31π ((n+m)/124) e− j31π ((n−m)/124) e− j2πmk/31 = π .
m=−15
sin( 248 (n − 8k))
The maximum would be at k̂ = 8, with the estimated frequency Ω̂ =

2π k̂/(( N + 1)∆t). Thus, Ω̂ = 32π, as expected. By this calculation, the fre-
quency sampling interval is ∆Ω = 2π/(( N + 1)∆t) = 4π, with the maximal
estimation absolute error ∆Ω/2 = 2π.
By using an odd number of samples N + 1 in the previous definitions,

the symmetry of the product x (n + m) x ∗ (n − m) is preserved in the summa-
tion. However, when an even number of samples is used, that is not the case.
To illustrate this effect, consider a simple example of signal, for n = 0, with
N = 4 samples. Then, four values of the signal x (m), used in calculation, are
x (m) x (−2) x (−1) x (0 ) x (1 )

.
x (−m) x (1) x (0 ) x (−1) x (−2)
So, in forming the local autocorrelation function, there are several possibili-
ties. One is to omit sample x (−2) and to use an odd number of samples, in
this case as well. Also, it is possible to periodically extend the signal and to
form the product based on
x (m) · · · x (1) x (−2) x (−1) x (0 ) x (1 ) x (−2) x (−1)

x (−m) · · · x (−1) x (−2) x (1 ) x (0 ) x (−1) x (−2) x (1 )
we ( m ) ··· 0 0 w e (1 ) w0 (0 ) w e (1 ) 0 0
Here we can use four product terms, but with the first one formed as
x (−2) x ∗ (−2), that is, as x (− N/2) x ∗ (− N/2). When a lag window with zero
ending value is used (for example, a Hann(ing) window), this term does not
make any influence to the result. The used lag window must also follow the
symmetry, for example we (m) = cos2 (πm/N ), when,
* +
n∆t 2πk
2 N∆t
N/2−1
PWD (n, k ) = ∑ we (m) x (n + m) x ∗ (n − m)e− j2πmk/N
m=− N/2
N/2−1
= ∑ we (m) x (n + m) x ∗ (n − m)e− j2πmk/N ,
m=− N/2+1
since we (− N/2) = 0. However, if the window is nonzero at the ending point

m = − N/2, this term will result in a kind of aliased distribution.
In order to introduce another way of the discrete Wigner distribution
calculation, with an even number of samples, consider again the continuous
form of the Wigner distribution of a signal with a limited duration. Assume
that the signal is sampled in such a way that the sampling theorem can be
applied and the equality sign used (Example 2.13). Then, the integral may
be replaced by a sum
N
∗ ∆t − jmΩ∆t
WD (t, Ω) = ∑ x (t + m ∆t
2 ) x (t − m 2 )e ∆t
m=− N
N/2
∗ ∆t − j2mΩ∆t
= ∑ x (t + 2m ∆t
2 ) x (t − 2m 2 )e ∆t
m=− N/2
N/2−1
∗ ∆t − j(2m+1)Ω∆t
+ ∑ x (t + (2m + 1) ∆t
2 ) x ( t − (2m + 1) 2 )e ∆t. (9.117)
m=− N/2
The initial sum is split into its even and odd terms part. Now, let us assume
that the signal is sampled in such a way that twice wider sampling interval
∆t is also sufficient to obtain the Wigner distribution (by using every other
signal sample). Then, for the first sum (with an odd number of samples)
holds,
N/2
1
∑ x (t + m∆t) x ∗ (t − m∆t)e− j2mΩ∆t ∆t = WD (t, Ω).
m=− N/2
2
The factor 1/2 comes from the sampling interval. Now, from (9.117) follows
N/2−1
∗ ∆t − j(2m+1)Ω∆t 1
∑ x (t + (2m + 1) ∆t
2 ) x ( t − (2m + 1) 2 )e ∆t = WD (t, Ω).
m=− N/2
2
(9.118)
This is just the discrete Wigner distribution with an even number of sam-
ples. If we denote
x (t + (2m + 1) ∆t ∆t
2 ) = x ( t + m∆t + 2 ) = xe ( t + m∆t )
√
x (n∆t + m∆t + ∆t 2 ) 2∆t = xe ( n + m )
then
∆t ∆t
x (t − m∆t − 2 ) = x ( t − m∆t + 2 − ∆t )
√
∆t
x (n∆t − m∆t + 2 − ∆t ) 2∆t = xe (n − m − 1).
The summation terms, for example for n = 0, are of the form
xe (m) ... xe (−2) xe (−1) x e (0 ) x e (1 ) ...

.
xe (−m − 1) ... x e (1) x e (0) xe (−1) xe (−2) ...
They would produce a modulated version of the pseudo Wigner distribu-

tion, due to the shift of a half of the sampling interval. However, this shift
can be corrected as (9.118)
N/2−1
WD (t, Ω) = e− jΩ∆t ∑ xe (t + m∆t) xe∗ (t − m∆t − ∆t)e− j2mΩ∆t (2∆t)
m=− N/2
for any t and Ω (having in mind the sampling theorem). Thus, we may also
write
* +
πk
WD n∆t, → WD (n, k )
N∆t
N/2−1
WD (n, k ) = e− jπk/N ∑ xe (n + m) xe∗ (n − m − 1)e− j2πmk/N . (9.119)
m=− N/2
In MATLAB notation, relation (9.6.4) can be implemented, as follows.

The signal values are
xn+ = [ xe (n − N/2), xe (n − N/2 + 1), ..., xe (n + N/2 − 1)],
x− ∗ ∗ ∗
n = [ xe ( n + N/2 − 1), xe (n + N/2 − 2), ..., xe (n − N/2)].
The vector of Wigner distribution values, for a given n and k, is
! B CT 6
WD (n, k )=e− jπk/N x+
n ∗ x −
n . ∗ e − jπkm/N
,
where e− jπkm/N is the vector with elements e− jπkm/N , for − N/2 ≤ m ≤

N/2 − 1, ∗ is the matrix multiplication and . ∗ denotes the vector multiplica-
tion term by term.
Thus, in the case of an even number of samples, the discrete Wigner
distribution of a signal xe (n), calculated according to (9.6.4), corresponds to
the original signal x (t) related to xe (n) as
√
xe (n) ↔ x (n∆t + ∆t/2) 2∆t.
To check this statement, consider the time marginal property of this distri-
bution. It is
1 N/2−1
∑ WD (n, k)
N k=− N/2
( )
N/2−1
1 N/2−1 − j(2m+1)πk/N
= ∑ xe (n + m) xe∗ (n − m − 1) ∑ e
m=− N/2
N k=− N/2
( )
N/2−1
1 j(2m+1)π/2 1 − e− j(2m+1)π
= ∑ xe (n + m) xe∗ (n − m − 1)
e
m=− N/2
N 1 − e− j(2m+1)π/N
' '
N/2−1
∗
'
' 1 ''2
= ∑ ( xe (n + m) xe (n − m − 1)δ(2m + 1)) = ' xe (n − )'
m=− N/2
2
= | x (n∆t)|2 (2∆t),
for |2m + 1| < N.

Since for any signal y(n) and its DFT holds
DFT N/2 {y(n) + y(n + N/2)} = Y (2k ),
where
Y (k ) = DFT N {y(n)},
the pseudo Wigner distribution (9.6.4), without frequency ovesampling, in
the case of an even N, can be calculated as
* +
2πk
WD n∆t, → WD (n, k)
N∆t
N/4−1
WD (n, k ) = e− jπk/( N/2) ∑ ( R(n, m) + R(n, m + N/2)) e− j2πmk/( N/2)
m=− N/4
where
R(n, m) = xe (n + m) xe∗ (n − m − 1).
Periodicity in m, for a given n, with period N is assumed in R(n, m), that
is, R(n, m + N ) = R(n, m) = R(n, m − N ). It is needed to calculate R(n, m +
N/2) for − N/4 ≤ m ≤ N/4 − 1 using R(n, m) for − N/2 ≤ m ≤ N/2 − 1
only.
In the case of real-valued signals, in order to avoid the need for
oversampling, as well as to eliminate cross-terms (that will be discussed
later) between positive and negative frequency components, their analytic
part is used in calculations.
9.6.5 From the STFT to the Wigner Distribution via S-Method
The pseudo Wigner distribution can be calculated as
"∞
1
PWD (t, Ω) = STFT (t, Ω + θ )STFT ∗ (t, Ω − θ )dθ. (9.120)
π
−∞
Where STFT is defined as

"∞
STFT (t, Ω) = x (t + τ )w(τ )e− jΩτ dτ. (9.121)
−∞
This can be proven by substituting (9.121) into (9.120).

Relation (9.120) has led to the definition of a time-frequency distribu-
tion
"L P
1
SM (t, Ω) = P(θ )STFT (t, Ω + θ )STFT ∗ (t, Ω − θ )dθ, (9.122)
π
− LP
where P(θ ) is a finite frequency domain window (we also assume rectangu-
lar form), P(θ ) = 0 for |θ | > L P . Distribution obtained in this way is referred
to as the S-method. Two special cases are: the spectrogram P(θ ) = πδ(θ ) and
the pseudo Wigner distribution P(θ ) = 1.
The S-method can produce a representation of a multi-component sig-
nal such that the distribution of each component is its Wigner distribution,
avoiding cross-terms, if the STFTs of the components do not overlap in time-
frequency plane.
Consider a signal
M
x (t) = ∑ xm (t)
m =1
where xm (t) are monocomponent signals. Assume that the STFT of each
component lies inside the region Dm (t, Ω), m = 1, 2, ..., M and assume that
regions Dm (t, Ω) do not overlap. Denote the length of the m-th region along
Ω, for a given t, by 2Bm (t), and its central frequency by Ω0m (t). Under this
assumptions the S-method of x (t) produces the sum of the pseudo Wigner
distributions of each signal component
M
SMx (t, Ω) = ∑ PWDxm (t, Ω), (9.123)
m =1
if the width of the rectangular window P(θ ), for a point (t, Ω), is defined by
!
Bm (t) − |Ω − Ω0m (t)| for (t, Ω) ∈ Dm (t, Ω)
L P (t, Ω) =
0 elsewhere.
To prove this consider a point (t, Ω) inside a region Dm (t, Ω). The integra-
tion interval in (9.122), for the m-th signal component is symmetrical with
respect to θ = 0. It is defined by the smallest absolute value of θ for which
Ω + θ or Ω − θ falls outside Dm (t, Ω), i.e.,
|Ω ± θ − Ω0m (t)| ≥ Bm (t).

For Ω > Ω0m (t) and positive θ, the integration limit is reached for θ =
Bm (t) − (Ω − Ω0m (t)). For Ω < Ω0m (t) and positive θ, the limit is reached
for θ = Bm (t) + (Ω − Ω0m (t)). Thus, having in mind the interval symmetry,
an integration limit which produces the same value of integral (9.122) as the
value of (9.120), over the region Dm (t, Ω), is given by L P (t, Ω). Therefore,
for (t, Ω) ∈ Dm (t, Ω) we have SMx (t, Ω) = PWDxm (t, Ω). Since regions
Dm (t, Ω) do not overlap we have
M
SMx (t, Ω) = ∑ PWDxm (t, Ω).
m =1
Note that any window P(θ ) with constant width

L P ≥ max{ L P (t, Ω)}
(t,Ω)
M
produces SMx (t, f ) = ∑m =1 PWDxm (t, Ω ), if the regions
' Dm (t, Ω) for m' =
1, 2, .., M, are at least 2L P apart along the frequency axis, 'Ω0p (t) − Ω0q (t)' >
B p (t) + Bq (t) + 2L P , for each p, q and t. This is the S-method with constant
window width. The best choice of L P is the value when P(θ ) is wide
enough to enable complete integration over the auto-terms, but narrower
than the distance between the auto-terms, in order to avoid the cross-terms.
If two components overlap for some time instants t, then the cross-term will
appear, but only between these two components and for that time instants.
A discrete form of the S-method (9.122) reads
L
SM L (n, k ) = ∑ S N (n, k + i )S∗N (n, k − i )
i =− L
for P(i ) = 1, − L ≤ i ≤ L (a weighted form P(i ) = 1/(2L + 1) could be used).

A recursive relation for the S-method calculation is
SM L (n, k ) = SM L−1 (n, k ) + 2 Re[S N (n, k + L)S∗N (n, k − L)], (9.124)
The spectrogram is the initial distribution SM0 (n, k ) = |S N (n, k )|2 and
2 Re[S N (n, k + i )S∗N (n, k − i )], i = 1, 2,..., L are the correction terms. Changing
parameter L we can start from the spectrogram ( L = 0) and gradually make
the transition toward the pseudo Wigner distribution by increasing L.
For the S-method realization we have to implement the STFT first,
based either on the FFT routines or recursive approaches suitable for hard-
ware realizations. After we get the STFT we have to “correct” the ob-
tained values, according to (9.124), by adding few “correction” terms to the
spectrogram values. Note that S-method is one of the rare quadratic time-
frequency distributions allowing easy hardware realization, based on the
hardware realization of the STFT, presented in the first part, and its “correc-
tion” according to (9.124). There is no need for analytic signal since the cross-
terms between negative and positive frequency components are removed in
the same way as are the other cross-terms. If we take that STFT (n, k ) = 0
outside the basic period, i.e., when k < − N/2 or k > N/2 − 1, then there
is no aliasing when the STFT is alias-free (in this way we can calculate the
alias-free Wigner distribution by taking L = N/2 in (9.124)). The calcula-
tion in (9.124) can be performed for the whole matrix of the S-method and
the STFT. This can significantly save time in some matrix based calculation
tools.
There are two ways to implement summation in the S-method. The
first one is with a constant L. Theoretically, in order to get the Wigner
distribution for each individual component, the number of correcting terms
L should be such that 2L is equal to the width of the widest auto-term. This
will guarantee cross-terms free distribution for all components which are at
least 2L frequency samples apart.
The second way to implement the S-method is with a time-frequency
dependent L = L(n,k) . The summation, for each point (n, k ), is performed
as long as the absolute values of S N (n, k + i ) and S∗N (n, k − i ) for that (n, k )
are above an assumed reference level (established, for example, as a few
percents of the STFT maximal value). Here, we start with the spectrogram,
L = 0. Consider the correction term S N (n, k + i )S∗N (n, k − i ) with i = 1. If the
STFT values are above the reference level then it is included in summation.
The next term, with i = 2 is considered in the same way, and so on. The
summation is stopped when a STFT in a correcting term is below the
reference level. This procedure will guarantee cross-terms free distribution
for components that do not overlap in the STFT.
Example 9.38. A signal consisting of three LFM components,

3
x (n) = ∑ Ai exp( jai πn/32 + jbi πn2 /1024),
i =1
with
( a1 , a2 , a3 ) = (−21, −1, 20)
and
(b1 , b2 , b3 ) = (2, −0.75, −2.8),
is considered at the instant n = 0. The IFs of the signal components are k i = ai ,
while the normalized squared amplitudes of the components are indicated by
dotted lines in Fig.9.50. An ideal time-frequency representation of this signal,
at n = 0, would be
I (0, k) = A21 δ(k − k1 ) + A22 δ(k − k2 ) + A23 δ(k − k3 ).
The starting STFT, with the corresponding spectrogram, obtained by using

the cosine window of the width N = 64 is shown in Fig.9.50(a),(b). The first
correction term is presented in Fig.9.50(c). The result of summing the spec-
trogram with the first correction term is the S-method with L = 1, Fig.9.50(d).
The second correction term (Fig.9.50(e)) when added to SM1 (0, k ), produces
the S-method with L = 2, Fig.9.50(f). The S-methods for L = 3, 5, and 8, end-
ing with the Wigner distribution (L = 31) are presented in Fig.9.50(g)-(j). Just
a few correction terms are sufficient in this case to achieve a high concen-
tration. The cross-terms start appearing at L = 8 and increase as L increases
toward the Wigner distribution. They make the Wigner distribution almost
useless, since they cover a great part of the frequency range, including some
signal components (Fig.9.50(j)). The optimal number of correction terms L is
the one that produces the best S-method concentration (sparsity), using the
ℓ1/2 -norm of the spectrogram and the S-method (corresponding to the ℓ1 -
norm of the STFT). In this case the best concentrated S-method is detected for
L = 5. The spectrogram is the initial distribution SM0 (n, k ) = |S N (n, k)|2 and
2 Re[S N (n, k + i )S∗N (n, k − i )], i = 1, 2,..., L are the correction terms. Consider-
ing the parameter L as a frame index, we can make a video of the transition
from the spectrogram to the Wigner distribution.
Example 9.39. The adaptive S-method realization will be illustrated on a five-

component signal x (t) defined for 0 ≤ t < 1 and sampled with ∆t = 1/256.
The Hamming window of the width Tw = 1/2 (128 samples) is used for STFT
calculation. The spectrogram is presented in Fig.9.51(a), while the S-method
with the constant Ld = 3 is shown in Fig.9.51(b). The concentration improve-
ment with respect to the case Ld = 0, Fig.9.51(a), is evident. Further increasing
of Ld would improve the concentration, but the cross-terms would also ap-
pear. Small changes are noticeable between the components with constant
instantaneous frequency and between quadratic and constant instantaneous
frequency component. An improved concentration, without cross-terms, can
be achieved by using the variable window width Ld . The regions Di (n, k),
determining the summation limit Ld (n, k ) for each point (n, k ), are obtained
by imposing the reference level corresponding to 0.14% of its maximal value
STFT |SN(0,k)|
first correction term second correction term
* *
2Re[SN(0,k+1) SN (0,k-1)] 2Re[SN(0,k+2) SN (0,k-2)]
(a) (c) (e)

+ +
|S (0,k)|2=SM (0,k) (b)+(c)=(d) SM (0,k) (d)+(e)=(f) SM (0,k)

N 0 1 2
k k k
-32 -16 0 16 31 -32 -16 0 16 31 -32 -16 0 16 31
(b) (d) (f)
SM3(0,k) SM5(0,k) SM6(0,k)
k
(g) -32 -16 0 16 31 (i)
(h)
SM8(0,k) SM9(0,k) SM31(0,k)=WD(0,k)
k
(j) (k) -32 -16 0 16 31
(l)
Figure 9.50 Analysis of a signal consisting of three LFM components (at the instant n = 0).
(a) The STFT with a cosine window of the width N = 64. (b) The spectrogram. (c) The first
correction term. (d) The S-method (SM) with one correction term. (e) The second correction
term. (f) The S-method with two correction terms. (g) The S-method with three correction
terms. (h) The S-method with five correction terms. (i) The S-method with six correction terms.
(j) The S-method with eight correction terms.(k) The S-method with nine correction terms. (l)
The Wigner distribution (the S-method with L = 31 correction term).
1 1
0.5 0.5
t 0 t 0
-0.5 (a) -0.5 (b)

-1 400 600 800 -1 400 600 800
0 200 0 200
Ω Ω
0.5
1
t 0 0.5
t 0
-0.5
(c) -0.5 (d)

-1
0 200 400 600 800 -1 600 800
0 200 400
Ω Ω
Figure 9.51 Time-frequency analysis of a multi-component signal: a) Spectrogram, b) The S-

method with a constant window, with L P = 3, c) Regions of support for the S-method with
a variable window width calculation, corresponding to Q2 = 725, d) The S-method with the
variable window width calculated using regions in c).
at that time instant n. They are defined as:
!
1 when |STFTxi (n, k)|2 ≥ Rn
Di (n, k ) =
0 elsewhere
and presented in Fig.9.51(c). White regions mean that the value of spectro-
gram is below 0.14% of its maximal value at that time instant n, meaning
that the concentration improvement is not performed at these points. The
signal dependent S-method is given in Fig.9.51(d). The method sensitivity,
with respect to the reference level is low.
9.7 GENERAL QUADRATIC TIME-FREQUENCY DISTRIBUTIONS
In order to provide additional insight into the field of joint time-frequency

analysis, as well as to improve concentration of time-frequency represen-
tation, energy distributions of signals were introduced. We have already
mentioned the spectrogram which belongs to this class of representations
and is a straightforward extension of the STFT. Here, we will discuss other
distributions and their generalizations.
The basics condition for the definition of time-frequency energy dis-
tributions is that a two-dimensional function of time and frequency P(t, Ω)
represents the energy density of a signal in the time-frequency plane. Thus,
the signal energy associated with the small time and frequency intervals ∆t
and ∆Ω, respectively, would be
Signal energy within [Ω + ∆Ω, t + ∆t] = P(t, Ω)∆Ω∆t.
However, point by point definition of time-frequency energy densities in

the time-frequency plane is not possible, since the uncertainty principle pre-
vents us from defining concept of energy at a specific instant and frequency.
This is the reason why some more general conditions are being considered
to derive time-frequency distributions of a signal. Namely, one requires that
the integral of P(t, Ω) over Ω, for a particular instant of time should be equal
to the instantaneous power of the signal | x (t)|2 , while the integral over time
for a particular frequency should be equal to the spectral energy density
| X (Ω)|2 . These conditions are known as marginal conditions or marginal
properties of time-frequency distributions.
Therefore, it is desirable that an energetic time-frequency distribution
of a signal x (t) satisfies:
– Energy property
"∞ "∞
1
P(t, Ω) dΩ dt = Ex , (9.125)
2π
−∞ −∞
– Time marginal properties
"∞
1
P(t, Ω) dΩ = | x (t)|2 , and (9.126)
2π
−∞
|x(t)|2
Integration over Ω
Ω Ω
Integration over t
t 2
|X(Ω)|
P(t,Ω)
Figure 9.52 Illustration of the marginal properties
– Frequency marginal property
"∞
P(t, Ω) dt = | X (Ω)|2 , (9.127)
−∞
where Ex denotes the energy of x (t). It is obvious that if either one of

marginal properties (9.126), (9.127) is fulfilled, so is the energy property.
Note that relations (9.125), (9.126) and (9.127), do not reveal any informa-
tion about the local distribution of energy at a point (t, Ω). The marginal
properties are illustrated in Fig. 9.52.
Next we will introduce some distributions satisfying these properties.
Time and frequency marginal properties (9.126) and (9.127) may be
considered as the projections of the distribution P(t, Ω) along the time
and frequency axes, i.e., as the Radon transform of P(t, Ω) along these
two directions. It is known that the Fourier transform of the projection
of a two-dimensional function on a given line is equal to the value of
the two-dimensional Fourier transform of P(t, Ω), denoted by AF (θ, τ ),
along the same direction (inverse Radon transform property). Therefore, if
P(t, Ω) satisfies marginal properties then any other function having two-
dimensional Fourier transform equals to AF (θ, τ ) along the axes lines θ = 0
and τ = 0, and arbitrary values elsewhere, will satisfy marginal properties,
Fig. 9.53.
Assuming that the Wigner distribution is a basic distribution which
satisfies the marginal properties (any other distribution satisfying marginal
properties can be used as the basic one), then any other distribution with
θ AF(τ,θ)
|x(t)|2 FT [ |x(t)|2 ]
t τ
Integration over Ω
FT [ | X(Ω)| ]
2
2D FT
Ω Ω
Integration over t
t 2
|X(Ω)|
P(t,Ω)
Figure 9.53 Marginal properties and their relation to the ambiguity function.
two-dimensional Fourier transform
AFg (θ, τ ) = c(θ, τ )FT2D

t,Ω {WD (t, Ω )} = c (θ, τ ) AF (θ, τ ) (9.128)
where c(0, τ ) = 1 and c(θ, 0) = 1, satisfies marginal properties as well.

The inverse two-dimensional Fourier transform of AFg (θ, τ ) produces
the Cohen class of distributions, introduced from quantum mechanics into
the time-frequency analysis by Claasen and Mecklenbäuker, in the form
"∞ "∞ "∞

1
CD (t, Ω) = c(θ, τ ) x (u + τ/2) x ∗ (u − τ/2)e jθt− jΩτ − jθu dudτdθ
2π
−∞ −∞ −∞
(9.129)
where c(θ, τ ) is called the kernel in the ambiguity domain.
Alternatively, the frequency domain definition of the Cohen class of
distributions is
"∞ "∞ "∞

1
CD (t, Ω) = X (u − θ/2) X ∗ (u + θ/2)c(θ, τ )e jθt− jτΩ+ jτu dudτdθ.
(2π )2
−∞ −∞ −∞
(9.130)
Various distributions can be obtained by altering the kernel function
c(θ, τ ). For example, c(θ, τ ) = 1 produces the Wigner distribution, while for
c(θ, τ ) = e jθτ/2 the Rihaczek distribution follows.
The Cohen class of distributions, defined in the ambiguity domain:

"∞ "∞
1
CD (t, Ω) = c(θ, τ ) AF (θ, τ )e jθt− jΩτ dτ dθ (9.131)
2π
−∞ −∞
can be written in other domains, as well. The time-lag domain form is

obtained from (9.129), after integration on θ, as:
"∞ "∞
CD (t, Ω) = c T (t − u, τ ) x (u + τ/2) x ∗ (u − τ/2)e− jΩτ dτ du. (9.132)
−∞ −∞
The frequency-Doppler frequency domain form follows from (9.130), after

integration on τ, as:
"∞ "∞
1
CD (t, Ω) = CΩ (θ, Ω − u) X (u + θ/2) X ∗ (u − θ/2)e jθt dθ du.
(2π )2
−∞ −∞
(9.133)
Finally, the time-frequency domain form is obtained as a two-dimensional
convolution of the two-dimensional Fourier transforms, from (9.131), as:
"∞ "∞
1
CD (t, Ω) = Π(t − u, Ω − ξ )WD (u, ξ ) du dξ. (9.134)
2π
−∞ −∞
Kernel functions in the respective time-lag, Doppler frequency-frequency

and time-frequency domains are related to the ambiguity domain kernel
c(θ, τ ) as:
"∞
1
c T (t, τ ) = c(θ, τ )e jθt dθ (9.135)
2π
−∞
"∞
CΩ (θ, Ω) = c(θ, τ )e− jΩτ dτ (9.136)
−∞
"∞ "∞
1
Π(t, Ω) = c(θ, τ )e jθt− jΩτ dτ dθ. (9.137)
2π
−∞ −∞
According to (9.134) all distributions from the Cohen class may be consid-
ered as 2D filtered versions of the Wigner distribution. Although any dis-
tribution could be taken as a basis for the Cohen class derivation, the form
with the Wigner distribution is used because it is the best concentrated dis-
tribution from the Cohen class with the signal independent kernels.
9.7.1 Reduced Interference Distributions
The analysis performed on ambiguity function and Cohen class of time-

frequency distributions leads to the conclusion that the cross-terms may
be suppressed or eliminated, if a kernel c(θ, τ ) is a function of a two-
dimensional low-pass type. In order to preserve the marginal properties
c(θ, τ ) values along the axis should be c(θ, 0) = 1 and c(0, τ ) = 1.
Choi and Williams exploited one of the possibilities defining the dis-
tribution with the kernel of the form
2 τ 2 /σ2
c(θ, τ ) = e−θ .
The parameter σ controls the slope of the kernel function which affects the
influence of cross-terms. Small σ causes the elimination of cross-terms but
it should not be too small because, for the finite width of the auto-terms
around θ and τ coordinates, the kernel will cause their distortion, as well.
Thus, there should be a trade-off in the selection of σ.
Here we will mention some other interesting kernel functions, produc-
ing corresponding distributions, Fig. 9.54.
Born-Jordan distribution
sin( θτ
2 )
c(θ, τ ) = θτ
,
2
Zhao-Atlas-Marks distribution
sin( θτ
2 )
c(θ, τ ) = w(τ ) |τ | θτ
,
2
Sinc distribution
!
θτ 1 for |θτ/α| < 1/2
c(θ, τ ) = rect( ) =
α 0 otherwise
Butterworth distribution
1
c(θ, τ ) = ,
1 + ( θθτ
c τc
)2N
where w(τ ) is a function corresponding to a lag window and α, N, θc and τc

are constants in the above kernel definitions.
c(θ,τ) c(θ,τ)
100 100
50 50
τ 0 τ 0
-50 -50
(a) (b)
-100 -100
0 2 0 2
-2 -2
θ θ
c(θ,τ) c(θ,τ)
100 100
50 50
τ 0 τ 0
-50 -50
(c) (d)
-100 -100
0 2 0 2
-2 -2
θ θ
Figure 9.54 Kernel functions for: Choi-Williams distribution, Born-Jordan distribution, Sinc
distribution and Zhao-Atlas-Marks distribution.
The spectrogram belongs to this class of distributions. Its kernel in

(θ, τ ) domain is the ambiguity function of the window
"∞ B τC B τ C − jθt
c(θ, τ ) = w t− w t+ e dt = AFw (θ, τ ).
2 2
−∞
Since the Cohen class is linear with respect to the kernel, it is easy to
conclude that a distribution from the Cohen class is positive if its kernel
can be written as
M
c(θ, τ ) = ∑ ai AFwi (θ, τ ),
i =1
where ai ≥ 0, i = 1, 2, ..., M.
There are several ways for calculation of the reduced interference dis-
tributions from the Cohen class. The first method is based on the ambiguity
function (9.131):
1. Calculation of the ambiguity function,
2. Multiplication with the kernel,
3. Calculation of the inverse two-dimensional Fourier transform of this
product.
The reduced interference distribution may also be calculated by using
(9.132) or (9.134) with appropriate kernel transformations defined by (9.135)
and (9.137). All these methods assume signal oversampling in order to avoid
aliasing effects. Figure 9.55 presents the ambiguity function along with ker-
nel (Choi-Williams). Figure 9.56(a) presents Choi-Williams distribution cal-
culated according to the presented procedure. In order to reduce high side
lobes of the rectangular window, the Choi-Williams distribution is also cal-
culated with the Hann(ing) window in the kernel definition c(θ, τ )w(τ ) and
presented in Fig. 9.56(b). The pseudo Wigner distribution with Hann(ing)
window is shown in Fig. 9.48.
For the discrete-time signals. there are several ways to calculate a
reduced interference distributions from the Cohen class, based on (9.131),
(9.132), (9.133), or (9.134).
The kernel functions are usually defined in the Doppler-lag domain
(θ, τ ). Thus, here we should use (9.131) with the ambiguity function of a
discrete-time signal
∞ * + * +
∆t ∗ ∆t
AF (θ, m∆t) = ∑ x p∆t + m x p∆t − m e− jpθ∆t ∆t.
p=−∞ 2 2
The signal should be sampled as in the Wigner distribution case. For a given
lag instant m, the ambiguity function can be calculated by using the stan-
dard DFT routines. Another way to calculate the ambiguity function is just
to take the inverse two-dimensional transform of the Wigner distribution.
Note that the corresponding transformation pairs are time ↔ Doppler and
lag ↔ f requency, that is, t ↔ θ and τ ↔ Ω. The relation between discretiza-
tion values in the Fourier transform pairs (considered interval, sampling
AF(θ,τ) and CW kernel
100
50
τ 0
-50
-100
-3 -2 -1 0 1 2 3
θ
Figure 9.55 Ambiguity function for signal from Fig.9.4 with the Choi-Williams kernel
interval in time ∆t, number of samples N, sampling interval in frequency

∆Ω = 2π/( N∆t)) is discussed in Chapter 1.
The generalized ambiguity function is obtained as
AFg (l∆θ, m∆t) = c(l∆θ, m∆t) AF (l∆θ, m∆t) (9.138)

∞ * + * +
∆t ∆t − j l∆θ p∆t
= c(l∆θ, m∆t) ∑ x p∆t + m x ∗ p∆t − m e ∆t,
p=−∞ 2 2
while a distribution, with kernel c(θ, τ ) is the two-dimensional inverse

Fourier transform in the form
1 ∞ ∞
CD (n∆t, k∆Ω) = ∑ ∑ AFg (l∆θ, m∆t)e− jkm∆t∆Ω e jnl∆θ∆t ∆t∆θ.
2π l =−∞ m=−∞
In this notation we can calculate

R S
CD (n, k ) = IDFT2D
l,m AFg ( l, m )
where the values of AFg (l, m) are calculated according to (9.138).

CWD(t,Ω)
250
200
150
t 100
50 (a)
0 2.5 3
0.5 1 1.5 2
0
Ω
CWD(t,Ω)
250
200
150
t 100
50 (b)
0 2.5 3
0.5 1 1.5 2
0
Ω
Figure 9.56 Choi-Williams distribution: (a) direct calculation, (b) calculation with the kernel
multiplied by a Hann(ing) lag window.
In the time-lag domain, the discrete-time form reads
∞ ∞
CD (n∆t, k∆Ω) = ∑ ∑ c T (n∆t − p∆t, m∆t)
p=−∞m=−∞
* + * +
∆t ∆t
× x p∆t + m x ∗ p∆t − m e− jkm∆t∆Ω (∆t)2 (9.139)
2 2
with
1 ∞
c T (n∆t − p∆t, m∆t) = ∑ c(l∆θ, m∆t)e jnl∆θ∆t e− jl p∆θ∆t ∆θ.
2π l =− ∞
For the discrete-time signals, it is common to write and use the Cohen
class of distributions in the form
∞ ∞
CD (n, ω ) = ∑ ∑ c T (n − p, m) x ( p + m) x ∗ ( p − m)e− j2mω , (9.140)
p=−∞ m=−∞
where
* + * +
∗ ∆t ∗ ∆t
x ( p + m) x ( p − m) = x ( p + m) x ( p − m) ∆t
2 2
* +
∆t ∆t
c T (n − p, m) = c T (n − p) , m∆t
2 2
* +
∆t
CD (n, ω ) → CD n , Ω∆t .
2
Here we should mention that the presented kernel functions are of
infinite duration along the coordinate axis in (θ, τ ) thus, they should be
limited in calculations. Their transforms exist in a generalized sense only.
9.7.2 Kernel Decomposition Method
Distributions from the Cohen class can be calculated by using decomposi-

tion of the kernel function in the time-lag domain. Starting from
"∞ "∞
CD (t, Ω) = c T (t − u, τ ) x (u + τ/2) x ∗ (u − τ/2)e− jΩτ dτdu
−∞ −∞
with substitutions u + τ/2 = t + v1 and u − τ/2 = t + v2 we get t − u =

−(v1 + v2 )/2 and τ = v1 − v2 , resulting in
"∞ "∞
v1 + v2
CD (t, Ω) = c T (− , v1 − v2 ) x (t + v1 ) x ∗ (t + v2 )e− jΩ(v1 −v2 ) dv1 dv2
2
−∞ −∞
The discrete-time version of the Cohen class of distribution can be written,

as
* +
n1 + n2
CD (n, ω ) = ∑ ∑ c T − , n1 − n2 [ x (n + n1 )e− jωn1 ][ x (n + n2 )e− jωn2 ]∗ .
n1 n2 2
Assuming that C is a square matrix of finite dimension, with elements:

* +
n + n2
C ( n1 , n2 ) = c T − 1 , n1 − n2
2
we can write
CD (n, ω ) = xn CxnH
where xn is a vector with elements x (n + n1 )e− jωn1 . We can now perform
the eigenvalue decomposition, finding solutions of det (C − λI) = 0 and
determining eigenvectors matrix Q that satisfies QQ H = I and
C = QΛQ H ,
where Λ is a diagonal matrix containing the eigenvalues. It results in
CD (n, ω ) = (xn Q)Λ(xn Q) H
Then, it is easy to conclude that the Cohen class of distribution can be

written as a sum of spectrograms:
CD (n, ω ) = ∑ λi |STFTqi (n, ω )|2

i
where λi represents eigenvalues, while qi are corresponding eigenvectors

of C, i.e., columns of Q, used as windows in the STFT calculations.
Example 9.40. A four-component real-valued signal with M = 384 samples is
considered. Its STFT is calculated with a Hann(ing) window of the width
N = 128 with a step of 4 samples. The spectrogram (L = 0) is shown in
Fig.9.57(a). The alias-free Wigner distribution (L = N/2) is presented in Fig.
9.57(b). The Choi-Williams distribution of analytic signal is shown in Fig.
9.57(c). Its cross-terms are smoothed by the kernel, that also spreads the auto-
term of the LFM signal and chirps. The S-method with L = 10 is shown in
Fig. 9.57(d). For graphical presentation, the distributions are interpolated by
a factor of 2. In all cases the pure sinusoidal signal is well concentrated. In
the Wigner distribution and the SM the same concentration is achieved for
the LFM signal.
SPEC(t,Ω) WD(t,Ω)
250 250
200 200
150 150
t 100 t 100
50 50
0 2 2.5 3 0 2 2.5 3
a) 0.5 1 1.5 b) 0.5 1 1.5
0 0
Ω Ω
CWD(t,Ω) SM(t,Ω)
250 250
200 200
150 150
t 100 t 100
50 50
0 2 2.5 3 0 2 2.5 3
c) 0.5 1 1.5 d) 0.5 1 1.5
0 0
Ω Ω
Figure 9.57 Time-frequency representation of a four component signal: (a) the spectrogram,
(b) the Wigner distribution, (c) the Choi-Williams distribution, and (d) the S-method.
Chapter 10
Sparse Signal Processing
A discrete-time signal can be transformed into other domains using

different signal transformations. Some signals that cover the whole consid-
ered interval in one domain could be sparse in a transformation domain, i.e.,
could be located within a few nonzero coefficients. Compressive sensing is
a field dealing with a model for data acquisition including the problem of
sparse signal recovery from a reduced set of observations. A reduced set
of observations can be a result of a desire to sense a sparse signal with
the lowest possible number of measurements/observations (compressive
sensing). It can also be a result of a physical or measurement unavailability
to take a complete set of observations. Since the signal samples are linear
combinations of the signal transformation coefficients they could be consid-
ered as the observations of a sparse signal in the transformation domain. In
applications it could also happen that some arbitrarily positioned samples
of a signal are so heavily corrupted by disturbances that it is better to omit
them and consider as unavailable in the analysis and to try to reconstruct
the signal with a reduced set of samples. Although the reduced set of ob-
servations/samples appears in the first case as a result of user strategy to
compress the information, while in the next two cases the reduced set of
samples is not a result of user intention, all of them can be considered within
the unified framework. Under some conditions, a full reconstruction of a
sparse signal can be performed with a reduced set of observations/samples,
as in the case if a complete set of samples/observations were available. A
priori information about the nature of the analyzed signal, i.e., its sparsity
in a known transformation domain, must be used in this analysis. Sparsity
_________________________________________________
Authors: Ljubiša Stanković, Miloš Daković, Srdjan Stanković, Irena Orović
665
666 Sparse Signal Processing
is the main requirement that should be satisfied in order to efficiently apply

the compressive sensing methods for sparse signal reconstruction.
The topic of this chapter is to analyze the signals that are sparse in
one of the common transformations domains. The DFT is used as a study
case. The compressive sensing results and algorithms are presented and
used only as a tool to solve engineering problems, involving sparse signals.
10.1 ILLUSTRATIVE EXAMPLES
Before we start the analysis we will describe few widely known examples
that can be interpreted and solved within the context of sparse signal
processing and compressive sensing.
Consider a large set of real numbers X (0), X (1),...,X ( N − 1). Assume
that only one of them is nonzero (or different from a common and known
expected value). We do not know either its position or its value. The aim is to
find the position and the value of this number. This case can easily be related
to many real life examples when we have to find one sample which differs
from other N − 1 samples. The nonzero value (or the difference from the
expected value) will be denoted by X (i ). A direct way to find the position
of nonzero (different) sample would be to perform up to N measurements
and compare each of them with zero (the expected) value. However, if N
is very large and there is only one nonzero (different than expected) sample
we can get the result in just a few observations/measurements. A procedure
for the reduced number of observations/measurements is described next.
Take random numbers as weighting coefficients ai , i = 0, 1, 2, ..., N − 1,
for each sample. Measure the total value of all N weighted samples, with
weights ai , from the set. Since only one is different from the common and
known expected value m (or from zero) we will get the total measured value
M = a1 m + a2 m + ... + ai (m + X (i )) + ... + a N m.
From this measured value M subtract the expected value MT = ( a1 + a2 +

... + a N )m. The obtained value of this observation/measurement, denoted
by y(0), is
N −1
y (0) = M − M T = ∑ a k X ( k ) = a i X ( i ),
k =0
since nonzero value in the space of X (0), X (1),...,X ( N − 1) is at one position

only, X (k ) = X (i )δ(k − i ).
As an illustration consider a set of N bags. Assume that only one bag
contains all false coins of a weight m + X (i ). It is different from the known
1 2 3 N
a a a a
1 2 3 N
One bag with false coins Two bags with false coins
M-M = a m+a m+...+a (m+X(i))+...+a m M-M = a m+...+a (m+X(i))+...+a (m+X(k))+...+a m
T 1 2 i N T 1 i k N
-(a 1 +a2 +...+ai +...+aN )m=aiX(i) -(a 1 +a2 +...+ai +...+aN )m=aiX(i)+akX(k)
i=?, X(i)=? i=?, k=? X(i)=?, X(k)=?
Figure 10.1 There are N bags with coins. One of them, at an unknown position, contains false
coins. False coins differ from the true ones in mass for unknown X (i ) = ∆m. The mass of the
true coins is m. Set of coins for measurement is formed using a1 coins from the first bag, a2 coins
from the second bag, an so on. The total measured value is M = a1 m + ... + ai (m + X (i )) + ... +
a N m. The difference of this value from the case if all coins were true is M − MT . Equation for
the case with one and two bags with false coins are presented (left and right).
weight m of true coins. The goal is to find the position and the difference
in weight of false coins. From each of N bags we will take ai , i = 1, 2, ...N,
coins. Number of coins from the ith bag is denoted by ai . The total measured
weight of all coins from N bags is M, Fig.10.1.
After the expected value is subtracted the observation/measurement
y(0) is obtained
N −1
y (0 ) = ∑ X (k )ψk (0), (10.1)
k =0
where the weighting coefficients for this measurement are denoted by

ψk (0) = ak , k = 0, 1, ..., N − 1. In the space of unknowns (variables) X (0),
X (1),...,X ( N − 1) this equation represents an N-dimensional hyperplane.
We know that only one unknown X (k ) is nonzero at the unknown position
k = i. Any cross-section of hyperplane (10.1) with any of coordinate axes

could be a solution of our problem. Assuming that a single X (k ) is nonzero
a solution will exist for any k. Thus, one measurement would produce N
possible single nonzero values equal to
X (k ) = y(0)/ψk (0), ψk (0) ̸= 0, k = 0, 1, 2, ..., N − 1.
As expected, from one measurement we are not able to solve the problem
and to find the position and the value of nonzero sample.
If we perform one more measurement y(1) with another set of weight-
ing coefficients ψk (1), k = 0, 1, ..., N − 1, and get measured value y(1) =
X (i )ψi (1) the result will be a hyperplane
N −1
y (1) = ∑ X (k )ψk (1).
k =0
This measurement will produce a new set of possible solutions for each X (k )
as
X (k) = y(1)/ψk (0), k = 0, 1, 2, ..., N − 1.
If these two hyperplanes (sets of solutions) produce only one common value
X (i ) = y(0)/ψi (0) = y(1)/ψi (1).
then it is the solution of our problem.

As an example consider N = 5 sets of coins. The common weight of
true coins is 2. In the first measurement we use ψi (0) = ai = i coins from
each set. The total weight of coins is M = 31. It is obtained by measuring (1 +
2 + 3 + 4 + 5)2 + iX (i ) = M, where X (i ) is the unknown weight difference
of false coins. It means that iX (i ) = 1, since all true coins would produce
(1 + 2 + 3 + 4 + 5)2 = 30. If the false coins were in the first set the weight
difference would be X (1) = 1/1 = 1, if they were in the second set then
X (2) = 1/2, and so on, X (3) = 1/3, X (4) = 1/4, X (5) = 1/5. False coins can
be in any of five sets. Perform one more measurement with ψi (1) = ai = i2
coins from each set. Total measured weight is now M = 113. It is obtained as
M = 2(12 + 22 + 32 + 42 + 52 ) + i2 X (i ) = 113. Obviously i2 X (i ) = 3. Again
if the false coins were in the first set then X (1) = 3/1, the second would
produce X (2) = 3/22 = 3/4, and so on, X (3) = 3/32 = 1/3, X (4) = 3/16,
X (5) = 3/25. The solution satisfying both equations is X (3) = 1/3. Thus,
false coins are in the third set. Their weight is 2 + 1/3 = 7/3. Note that we
would not be able to solve the problem with two measurements if we got
two values X (i ) and X (k ) for i ̸= k satisfying both equations.
In a matrix form these two measurements are

⎡ ⎤
- . - . X (0 )
y (0) ψ0 (0) ψ1 (0) ... ψ N −1 (0 ) ⎢ X (1 ) ⎥
= ⎢ ⎥
y (1) ψ0 (1) ψ1 (1) ... ψ N −1 (1 ) ⎣ ... ⎦
X ( N − 1)
y = AX
where A is the matrix of coefficients (measurement matrix)

- .
ψ0 (0) ψ1 (0) ... ψN −1 (0)
A=
ψ0 (1) ψ1 (1) ... ψN −1 (1)
and y are observations/measurements of sparse variable X.

Common value for two measurements X (i ) = y(0)/ψi (0) and X (i ) =
y(1)/ψi (1) is unique if
ψi (0)ψk (1) − ψi (1)ψk (0) ̸= 0
for any i ̸= k.
In order to prove this statement assume that two different solutions
X (i ) and X (k ), for the case of one nonzero coefficient, satisfy the same
measurement hyperplane equations
ψi (0) X (i ) = y(0), ψi (1) X (i ) = y(1)
and
ψk (0) X (k ) = y(0), ψk (1) X (k ) = y(1).
Then
ψi (0) X (i ) = ψk (0) X (k )
and
ψi (1) X (i ) = ψk (1) X (k ).
If we divide these two equations we get
ψi (0)/ψi (1) = ψk (0)/ψk (1)
or ψi (0)ψk (1) − ψi (1)ψk (0) = 0. This is contrary to the assumption that

ψi (0)ψk (1) − ψi (1)ψk (0) ̸= 0.
The same conclusion can be made considering matrix form relations
for X (i ) and X (k ). If both of them may satisfy the same two measurements
then - . - .- .
y (0 ) ψi (0) ψk (0) X (i )
=
y (1 ) ψi (1) ψk (1) 0
- . - .- .
y (0) ψi (0) ψk (0) 0
= . (10.2)
y (1) ψi (1) ψk (1) X (k )
Subtraction of the previous matrix equations results in
- .- .
ψi (0) ψk (0) X (i )
= 0.
ψi (1) ψk (1) − X (k )
For ψi (0)ψk (1) − ψi (1)ψk (0) ̸= 0 follows X (i ) = X (k ) = 0. Therefore two

different nonzero solutions X (i ) and X (k ) in this case cannot exist. This
concludes the proof that the solution is unique if
- .
ψi (0) ψk (0)
ψi (0)ψk (1) − ψi (1)ψk (0) = det ̸= 0
ψi (1) ψk (1)
for any i ̸= k. It also means that rank (A2 ) = 2 for any A2 being a 2 × 2 sub-
matrix of the matrix of coefficients (measurement matrix) A. For additional
illustration of this simple problem see Section 10.5.2.
In numerical and practical applications we would not be satisfied,
if for example ψi (0)ψk (1) − ψi (1)ψk (0) ̸= 0 but ψi (0)ψk (1) − ψi (1)ψk (0) =
ε close to zero. In this case the theoretical condition for a unique solution
would be satisfied, however the analysis and possible inversion would be
highly sensitive to any kind of noise, including quantization noise. Thus,
a practical requirement is that the determinant is not just different from
zero, but that it sufficiently differs from zero so that an inversion stability
and robustness to a noise is achieved. Inversion stability for a matrix B is
commonly described by the condition number of matrix
λmax
cond {B} =
λmin
where λmax and λmin are the largest and the smallest eigenvalue of matrix
B (when B H B = BB H )1 . The inversion stability worsens as λmin approaches
to zero (when λmin is small as compared to λmax ). For stable and robust
1 The value of determinant of matrix B is equal to the product of its eigenvalues, det {B} =
λ1 λ2 ...λ N , where N is the order of square matrix B. Note that the condition number can
be interpreted as a ratio of the norms-two (square roots of energies) of noise ε and signal x
after and before inversion y + yε = B−1 (x+ε). This number is always greater or equal to 1.
The best value for this ratio is achieved when λmin is close to λmax .
calculations a requirement
λmax
≤1+δ
λmin
is imposed, with a nonnegative constant δ being sufficiently small. In our

example this condition should hold for any submatrix A2 = B.
The previous experiment can be repeated assuming two nonzero val-
ues X (i ) and X (k ), Fig.10.1(right). In the case of two nonzero coefficients,
two measurements
N −1
y (0) = ∑ X (l )ψl (0) = X (i )ψi (0) + X (k )ψk (0) (10.3)
l =0
N −1
y (1) = ∑ X (l )ψl (1) = X (i )ψi (1) + X (k )ψk (1)
l =0
will result in X (i ) and X (k ) for any i and k. They are the solution of a system
with two equations and two unknowns. Therefore, with two measurements
we cannot get a result of the problem and find the positions and the values
of nonzero coefficients. If two more measurements are performed then an
additional system with two equations
y(2) = X (i )ψi (2) + X (k )ψk (2) (10.4)

y(3) = X (i )ψi (3) + X (k )ψk (3)
is formed. Two systems of two equations (10.3) and (10.4) could be solved
for X (i ) and X (k ) for each combination of i and k. If these two systems
produce only one common solution pair X (i ) and X (k ) then this pair is the
solution of our problem. As in the case of one nonzero coefficient, we may
show that the sufficient condition for a unique solution is
⎡ ⎤
ψk1 (0) ψk2 (0) ψk3 (0) ψk4 (0)
⎢ ψk (1) ψk2 (1) ψk3 (1) ψk4 (1) ⎥
det ⎢ 1
⎣ ψk (2)
⎥ ̸= 0 (10.5)
1
ψk2 (2) ψk3 (2) ψk4 (2) ⎦
ψk1 (3) ψk2 (3) ψk3 (3) ψk4 (3)
for any k1 , k2 , k3 and k4 or rank (A4 ) = 4 for any A4 , where A4 is a

4 × 4 submatrix of the matrix of coefficients A. In numeric realizations, the
condition is cond {A4 } ≤ 1 + δ with sufficiently small δ for all A4 . Suppose
that (10.5) holds and that two pairs of solutions of the problem X (k1 ), X (k2 )
and X (k3 ), X (k4 ) exist. Then

⎡ ⎤ ⎡ ⎤⎡ ⎤
y (0 ) ψk1 (0) ψk2 (0) ψk3 (0) ψk4 (0) X (k1 )
⎢ y(1) ⎥ ⎢ ψk (1) ψk2 (1) ψk3 (1) ψk4 (1) ⎥ ⎢ X (k2 ) ⎥
⎢ ⎥ ⎢ ⎥⎢ ⎥
⎣ y(2) ⎦ = ⎣ ψk (2)
1
1
ψk2 (2) ψk3 (2) ψk4 (2) ⎦ ⎣ 0 ⎦
y (3 ) ψk1 (3) ψk2 (3) ψk3 (3) ψk4 (3) 0
and
⎡ ⎤ ⎡ ⎤⎡ ⎤
y (0) ψk1 (0) ψk2 (0) ψk3 (0) ψk4 (0) 0
⎢ y(1) ⎥ ⎢ ψk (1) ψk2 (1) ψk3 (1) ψk4 (1) ⎥ ⎢ 0 ⎥
⎢ ⎥ ⎢ ⎥⎢ ⎥.
⎣ y(2) ⎦ = ⎣ ψk (2)
1
1
ψk2 (2) ψk3 (2) ψk4 (2) ⎦ ⎣ X (k3 ) ⎦
y (3) ψk1 (3) ψk2 (3) ψk3 (3) ψk4 (3) X (k4 )
By subtracting of these two systems we get

⎡ ⎤⎡ ⎤
ψk1 (0) ψk2 (0) ψk3 (0) ψk4 (0) X (k1 )
⎢ ψk (1) ψk2 (1) ψk3 (1) ψk4 (1) ⎥ ⎢ X (k2 ) ⎥
0=⎢ 1
⎣ ψk (2)
⎥⎢ ⎥.
1
ψk2 (2) ψk3 (2) ψk4 (2) ⎦ ⎣ − X (k3 ) ⎦
ψk1 (3) ψk2 (3) ψk3 (3) ψk4 (3) − X (k4 )
Since (10.5) holds then X (k1 ) = X (k2 ) = X (k3 ) = X (k4 ) = 0, meaning that
the assumption about two independent pairs of solutions with two nonzero
coefficients is not possible.
This approach to solve a problem (and to check the solution unique-
ness) is illustrative, however not computationally feasible. For example, for
a simple case with N = 1024 and just two nonzero coefficients, in order to
find a solution we have to solve two times systems of equations (10.3) and
(10.4) for each possible combination of i and k and to compare their solu-
tions. Total number of combinations of two indices out of the total number
of N indices is * +
N
∼ 106 .
2
In order to check the solution uniqueness we should calculate a determinant
value for all combinations of four indices k1 , k2 , k3 and k4 out the set of N
values. The number of determinants is ( N4 ) ∼ 1012 . If one determinant of the
forth order is calculated in 10 −5 [sec], then more than 5 days are needed
to calculate all determinants for this quite simple case of two nonzero
coefficients.
As a next example consider a signal described by a weighted sum of
K harmonics from a set of possible oscillatory functions e j2πkn/N , k = 0, 1, 2,
..., N − 1,
x (n) = A1 e j2πk1 n/N + A2 e j2πk2 n/N + ... + AK e j2πkK n/N ,
with K ≪ N. In the DFT domain this signal will be sparse with X (k ) =

DFT { x (n)} having only few nonzero values at k = k i , i = 1, 2, ..., K. Accord-
ing to the sampling theorem the sampling of this kind of signals should be
adjusted to the maximal expected signal frequency k = max{k1 , k2 , ..., k K }.
For an arbitrary set of frequencies, it means that we should adjust sampling
in time to the maximal possible frequency k = N − 1 and to use the full set of
N signal values/measurements at n = 0, 1, 2, ..., N in order to avoid aliasing.
However, if we know that the signal consists of only K ≪ N functions
with unknown amplitudes, then regardless of their frequencies, the signal
can be fully reconstructed from a reduced set of samples. Samples can be
considered as weighted measurements of the sparse function X (k ),
N −1
y (0 ) = x ( n1 ) = ∑ X (k )ψk (n1 )
k =0
with the weighting coefficients ψk (n1 ) = exp( j2πn1 k/N )/N. The previous
relation is the IDFT. Now a similar analysis like in the previous illustrative
example can be performed, assuming for example K = 1 or K = 2. We can
find position and value of nonzero X (k ) using just a few of signal samples
y(i ). This model corresponds to many signals in real life. For example, in
the Doppler-radar systems the speed of a radar target is transformed into
a frequency of a sinusoidal signal. Since the returned signal contains only
one or just a few of targets, the signal representing target velocity is a sparse
signal in the DFT domain. It can be reconstructed from fewer number of
samples than the total number of radar return signal samples N, Fig.10.2.
The signal model with complex-valued sinusoids is specific and very
important in engineering applications. We will focus most of our presen-
tation to this model. To illustrate complexity of the problem we will dis-
cus the simplest possible case consisting of one complex sinusoid at a fre-
quency k0 . Within the previous framework it means that we consider a
case with only one nonzero DFT coefficient at an unknown frequency in-
dex k0 . Assume that two samples/observations x (n1 ) = A exp( j2πk0 n1 /N )
and x (n2 ) = A exp( j2πk0 n2 /N ) of this signal are available. Note that signal
amplitude A is complex-valued and includes the initial phase. In order to
find the unknown position (frequency index) form the ratio
x ( n1 )
= exp( j2πk0 (n1 − n2 )/N ).
x ( n2 )
60 X(k) Two target velocities 2 x(n)

within 64 bins range
1
40
0
20
-1
Target velocities transformed
0 -2 into a dense signal
0 20 40 60 (a) 0 20 40 60 (b)
2 y(n) 60 Y(k) DFT of y(n)

before reconstruction
1
40
0
20
-1
-2 Mesurements of x(n) 0
0 20 40 60 (c) 0 20 40 60 (d)
Figure 10.2 (a) Signal in the frequency domain, where it is sparse (velocities of two targets
in Doppler radar signal). (b) Signal in the time domain, where it is dense. (c) Reduced set of
measurements (samples) and (d) its DFT before reconstruction, calculated using the available
samples only. Real parts of signals are presented.
From this relation follows
2πk0 (n1 − n2 )/N = arg{ x (n1 )} − arg{ x (n2 )} + 2kπ,
where k is an arbitrary integer. Then
arg{ x (n1 )} − arg{ x (n2 )} k

k0 = N+ N. (10.6)
2π (n1 − n2 ) n1 − n2
Let us analyze the ambiguous term kN/(n1 − n2 ) role in the determination

of k0 . For n1 − n2 = 1, this term is kN, meaning that any frequency k0
would be ambiguous with kN. Any value k0 + kN for k ̸= 0, in this case,
will be outside the basic period 0 ≤ k ≤ N − 1. Thus, we may find k0
in a unique way, within 0 ≤ k0 ≤ N − 1. However, for |n1 − n2 | = L > 1,
the terms kN/(n1 − n2 ) = kN/L produce shifts within the frequency basic
period. Then several possible solutions for the frequency k0 are obtained.
For example, for a signal with N = 16 and k0 = 5 if we use n1 = 1 and n2 = 5,
a possible solution of (10.6) is k0 = 5, but also
k0 = 5 + 16k/4,
or k0 = 9, k0 = 13, and k0 = 1 are possible solutions for frequency within

0 ≤ k0 ≤ 15. Therefore, for |n1 − n2 | = L > 1 more than two samples are
needed to resolve this ambiguity. An interesting case with nonuniform
(random) sampling positions ti , x (ti ) = A exp( jω0 ti ), will be discussed at
the end of this chapter.
The final illustrative example is based on Schepp-Logan phantom in
computed tomography. This example was the one that attracted significant
engineering community attention to the compressive sensing field and ini-
tiated many other applications. Consider a simplified, one-dimensional ver-
sion of this model, along the line p(k ) = P(200, k ), Fig.10.3(white horizontal
line). Its derivative X (k ) = p(k ) − p(k − 1) is sparse, Fig.10.3(left, middle).
From the computed tomography it is known that just some and nonun-
formly positioned values of the Fourier transforms of the image are avail-
able. In one dimension we may say that the available values correspond to
the samples of the Fourier transform of X (k ), i.e. to x (ξ ) = FT{ X (k )}, at
some nonuniform positions of ξ. Thus, the problem here is to reconstruct
the sparse image p(k ) (producing X (k )) from a reduced number of arbitrary
positioned samples of x (ξ ). Note that in reality the DFT values are available
at the polar coordinate system, with their number and distance changing
with the projection angle. Also in two dimensions the difference opera-
tor p(k ) − p(k − 1) would be replaced by corresponding two-dimensional
difference relation (approximation of the gradient F vector intensity, or total
variation, corresponding to one-dimension form | p(k ) − p(k − 1)|2 ). In
traditional computed tomography reconstruction algorithms, based on the
Fourier transform inversion, the main idea is to reconstruct fast changing
x (ξ ) = FT { X (k )} on a fine rectangular grid, using nearest or linear or spline
interpolations. The sparsity of p(k ) (producing X (k )) has not been taken into
account in these approaches.
In some applications there could be an indirect linear relation between
the sample/measurements and the sparsity domain. These cases can be
solved in a similar way as in the case of a direct relation, as it will be shown
at the end of this chapter.
In this chapter we will mainly use the DFT as the domain of signal
sparsity, since it plays the central role in engineering applications. Note
that in the compressive sensing theory random measurement matrices are
mainly used. The topic of this chapter is to analyze the signals that are
sparse in one of common transformations domains (DFT is used as a study
case). The compressive sensing results and algorithms are used as a tool to
solve this kind of problems, involving sparse signals.
1 p(k)=P(200,k)
0.5
0
0 100 200 300 400 500
1 X(k)=p(k)-p(k-1)
-1
0 100 200 300 400 500
1 x(ξ)=FT[X(k)]
-1
-2 0 2
Figure 10.3 Shepp-Logan model for the computed tomography reconstruction (left), along
with its slice along indicated line (right-top), its derivative (right-middle) and its Fourier
transform.
10.2 SPARSITY AND REDUCED SET OF SAMPLES/OBSERVATIONS
Consider a signal x (n) and its transformation domain coefficients X (k ),
N −1
x (n) = ∑ X (k )ψk (n)
k =0
or
x= ΨX,
where Ψ is the transformation matrix with elements ψk (n), x is the signal
vector column, and X is the transformation coefficients vector column. A
signal is sparse in the transformation domain if the number of nonzero
transform coefficients K is much lower than the number of the original
signal samples N, i.e., if
X (k ) = 0
for
/ {k1 , k2 , ..., k K } = K,
k∈
The number of nonzero samples is
∥X∥0 = card {X} = K,
where
N −1
∥ X ∥0 = ∑ | X (k )|0
k =0
and card {X} is the notation for the number of nonzero transformation
coefficients in X. Counting the nonzero coefficients in a signal representation
can be achieved by using the so called ℓ0 -norm denoted by ∥X∥0 . This form
is referred to as the ℓ0 -norm (norm-zero) although it does not satisfy norm
properties. By definition | X (k )|0 = 0 for | X (k )| = 0 and | X (k )|0 = 1 for
| X (k)| ̸= 0.
A signal x (n), whose transformation coefficients are X (k ), is sparse in
this transformation domain if
card {X} = K ≪ N.
For linear signal transforms the signal can be written as a linear combination
of the sparse domain coefficients X (k )
x (n) = ∑ X (k )ψk (n). (10.7)

k∈{k1 ,k2 ,...,k K }
A signal sample can be considered as a measurement/observation of linear

combination of values X (k ). Topic of this chapter is to show that a signal of
sparsity K in a transformation domain can be reconstructed from a reduced
set of M samples/observations. In the reconstruction, two approaches are
possible:
-In the first approach the reconstruction process is done by reconstruct-
ing sparse transform coefficients X (k ). By reconstructing all sparse coeffi-
cients X (k ) we would be able to reconstruct all signal values x (n) and have
a complete signal x (n) for all 0 ≤ n ≤ N − 1.
-We may also reconstruct x (n) samples/measurements up to the com-
plete set of data, using the available samples/measurements and the spar-
sity of coefficients X (k ). Then all N coefficients X (k ) can easily be calculated
from the full set of signal samples/measurements. This approach, involving
complete set of samples, can be used with common signal transformation
matrices when a well defined complete set of samples exists.
2 x(n)
-1
-2
0 20 40 60 80 100 120
2 y(n)
-1
-2
0 20 40 60 80 100 120
Figure 10.4 Signal x (n) and available samples y(n).
Assume that samples of x (n) are available at some random positions
ni ∈ M ={n1 , n2 , ..., n M }⊂ N = {0, 1, 2, 3, ..., N − 1}.
Here N = {0, 1, 2, 3, ..., N − 1} is the set of all samples of a signal x (n)

and M ={n1 , n2 , ..., n M } is its random subset with M elements, M ≤ N.
Case when the samples/measurements are taken at random instants ti of a
continuous signal x (t), will be considered later as well. The available signal
values are denoted by vector y, Fig.10.4,
y = [ x (n1 ), x (n2 ), ..., x (n M )] T .
The available samples (measurements of a linear combination of X (k ))

defined by (10.7), for ni ∈ M ={n1 , n2 , ..., n M }, can be written as a system of
M equations
⎡ ⎤ ⎡ ⎤⎡ ⎤
x ( n1 ) ψ0 (n1 ) ψ1 (n1 ) ψ N −1 ( n 1 ) X (0 )
⎢ x (n2 ) ⎥ ⎢ ψ0 (n2 ) ψ1 (n2 ) ψ N −1 ( n 2 ) ⎥ ⎢ X (0 ) ⎥
⎢ ⎥=⎢ ⎥⎢ ⎥
⎣ ... ⎦ ⎣ ... ... ... ⎦⎣ ... ⎦
x (n M ) ψ0 (n M ) ψ1 (n M ) ψ N −1 ( n M ) X ( N − 1)
or
y = AX
where A is the M × N matrix of measurements/observations/available
signal samples.
The fact that the signal is sparse with X (k ) = 0 for k ∈
/ {k1 , k2 , ..., k K } =
K is not included in the measurement matrix A since the positions of
the nonzero values are unknown. If the knowledge that X (k ) = 0 for k ∈ /
{k1 , k2 , ..., k K } = K were included then a reduced observation matrix would
be obtained as
⎡ ⎤ ⎡ ⎤⎡ ⎤
x ( n1 ) ψk1 (n1 ) ψk2 (n1 ) ψkK (n1 ) X (k1 )
⎢ x (n2 ) ⎥ ⎢ ψk (n2 ) ψk (n2 ) ψkK (n2 ) ⎥ ⎢ ⎥
⎢ ⎥=⎢ 1 2 ⎥ ⎢ X (k2 ) ⎥
⎣ ... ⎦ ⎣ ... ... ... ⎦ ⎣ ... ⎦
x (n M ) ψk1 (n M ) ψk2 (n M ) ψkK (n M ) X (k K )
or
y = AK XK .
Matrix AK would be formed if we knew the positions of nonzero samples
k ∈ {k1 , k2 , ..., k K } = K. It would follow from the measurement matrix A by
omitting the columns corresponding to the zero-valued coefficients X (k ).
Assuming that there are K nonzero coefficients X (k ), out of the total
number of N values, the total number of possible different matrices AK is
equal to the number of combinations with K out of N. It is equal to ( N K ).
In the common signal transform cases (like the DFT) the set of miss-
ing/unavailable samples can be defined as well
yc = [ x (n M+1 ), x (n M+2 ), ..., x (n N )] T .
The union of sets y and yc is a set containing all signal samples (complete
set of samples/measurements). If x is the complete set of samples then
x = y ∪ yc .
The signal of interest x (n) can also be measured in an indirect way.

It is common to assume that each indirect measurement f (n) is a linear
combination of signal samples
f (i ) = bi1 x (0) + bi2 x (1) + ... + biN x ( N − 1), (10.8)
where i = 0, 1, ..., M − 1. In matrix form the relation between signal samples

x (n) and M indirect measurements is given by
f = B M x.
Since the signal is related to its sparsity domain by x = ΨX, the measure-
ments are related to the sparsity domain form of signal as
f = B M ΨX = AX,
where
A = B M Ψ.
An example of indirect measurements is a linear signal transform
N −1
f (i ) = ∑ h (i − m ) x ( m )
m =0
with bim = h(i − m − 1). In this case samples of the output signal (a trans-
form of original signal) are the measurements, while the sparsity domain
is the transformation domain of the input (original) signal. All linear signal
transforms can be considered within this framework.
10.3 TRANSFORMATION MATRIX PARAMETERS
Consider a discrete-time signal x (n) of duration N, for 0 ≤ n ≤ N − 1

within a transformation domain defined by the basis functions set ψk (n),
k = 0, 1, ..., N
N −1
x (n) = ∑ X (k )ψk (n)
k =0
with
N −1
X (k) = ∑ x ( n ) ϕ k ( n ).
n =0
In a matrix form
x= ΨX and X = Φx.
For the DFT matrix the elements of matrices Ψ and Φ are

1 −k 1
ψk (n) = WN = e j2πk/N
N N
ϕk (n) = WNk
= e− j2πk/N ,
i.e., Φ = W N and Ψ = W− 1 1 H
N = N W N . The elements of matrix W N are WN
k
and W N H is the Hermitian transpose of W .

N
Other signal transformation matrices can be considered in the same
way as the DFT. In the case of common signal transforms the measurement
matrix A is formed from the complete N × N transformation matrix Ψ by
omitting the rows corresponding to the unavailable samples. When matrix
Ψ is the inverse DFT matrix then the measurement matrix A is a partial IDFT
matrix.
The normalized form of the partial DFT matrix would have functions
1 −k 1
ψk (n) = √ WN = √ e j2πk/N
M M
so that its energy over M measurements (energy of a column of measure-
ments matrix A) is
M
⟨ψk , ψk∗ ⟩ = ∑ |ψk (ni )|2 = 1.
i =1
−k
For the common DFT matrix ψk (n) = WN /N with
⟨ψk , ψk∗ ⟩ = N.
In the introductory example we have mentioned a multiplication of

measurements with random numbers. This kind of matrices play central
role in compressive sensing theory. For example, zero-mean independent
random Gaussian variables
1
ψk (n) = N (0, )
N
can be used as basis functions. Variance of the Gaussian random variable
is σε2 = 1/N so that a sum of N variables (corresponding to the number of
measurements) has unity variance. This kind of basis functions is orthonor-
mal in mean since
E{ψk2 (n)} = 1
and
E{ψi (n)ψk (n)} = 0, for i ̸= k.
√
√Bernoulli random matrix, whose elements take the value 1/ N and
−1/ N, is also used in compressive sensing. An interesting class of mea-
surement matrices is called structured random matrices. One type of such
matrices is obtained by random sampling of functions that have a sparse
expansion in terms of an orthonormal system. The partial DFT matrix is
one of such examples. The randomness is a result of the random sampling
positions. Another more complex example of such sampling and structured
random matrix will be presented on the case of nonuniform sampling of
signal x (t) and the DFT transform as its sparsity domain, by the end of this
chapter.
10.3.1 Unitary Matrix
A linear transformation matrix Ψ is unitary matrix if it satisfies the property
⟨ΨX, ΨY⟩ = EΨ ⟨X, Y⟩
where ⟨X, Y⟩ denotes scalar product of two N-dimensional vectors
⟨X, Y⟩ = X (0)Y ∗ (0) + X (1)Y ∗ (1) + ... + X ( N − 1)Y ∗ ( N − 1)
and EΨ is the energy of ψ
EΨ = ⟨ψk , ψk∗ ⟩ = |ψk (0)|2 + |ψk (1)|2 + ... + |ψk ( N − 1)|2 .
It is assumed that EΨ is the same for any k. For normal basis functions,
EΨ = 1 by definition, and
⟨ΨX, ΨY⟩ = ⟨X, Y⟩ .
In the DFT case the basis would be normal if we used

1 −k 1
ψk (n) = √ WN = √ e j2πk/N
N N
1 1
ϕk (n) = √ WNk
= √ e− j2πk/N .
N N
For the notation common in signal processing
1 −k 1
ψk (n) = W = e j2πk/N .
N N N
Then EΨ = 1/N in the DFT case. Note that the unitary property in the DFT
is just Parseval’s theorem, since ΨX = x and ΨY = y. With EΨ = 1/N the
relation
results in
N −1 N −1
1
∑ x (n)y∗ (n) = ∑ X ( k )Y ∗ ( k ) .
n =0 N k =0
10.3.2 Isometry and Restricted Isometry Property
A transformation matrix Ψ satisfies the isometry property if it preserves the

vector intensity in the N-dimensional space, i.e., if
∥ΨX∥22 = EΨ ∥X∥22 (10.9)
where ∥X∥22 is defined as
∥X∥22 = | X (0)|2 + | X (1)|2 + ... + | X ( N − 1)|2 .
For a transformation matrix Ψ which satisfies the isometry property the

following relation holds
1
EΨ ∥ΨX∥22 − ∥X∥22
= 0.
∥X∥22
A transformation matrix Ψ satisfies the restricted isometry property

(RIP) with a constant δ if
' 1 '
' ∥ΨX∥2 − ∥X∥2 '
' EΨ 2 2'
' ' ≤ δ. (10.10)
' ∥X∥22 '
Here the isometry condition that 1

EΨ ∥ΨX∥22 − ∥X∥22 = 0 is relaxed to the
condition that the relative absolute value of E1Ψ ∥ΨX∥22 − ∥X∥22 is sufficiently
small as compared to the signal energy, i.e., that it is within 0 ≤ δ < 1 range.
The restricted isometry constant is a measure of how much the transform
matrix Ψ differ from an isometry transform (10.9).
If δ = 1 the isometry does not hold since for ∥X∥22 ̸= 0 we have
1 2
EΨ ∥ ΨX ∥2 = 0. It means that some nonzero coordinates of X are projected
to zero-values in 1
EΨ ∥ΨX∥22 .
For an M × N matrix A the restricted isometry property is satisfied

with δ if ' 1
' 2 2 ''
' EA ∥AX∥2 − ∥X∥2 '
' ' ≤ δ. (10.11)
' ∥ X ∥2 '
2
For a K −sparse signal the restricted isometry is satisfied with δK if
' 1 2 2 ''
'
' E A ∥ A K X K ∥2 − ∥ X K ∥2 '
' ' ≤ δK
' ∥ X K ∥2 2
'
for all possible (earlier described) submatrices AK of matrix A. The re-

stricted isometry property must hold for any sparsity lower than K as well.
The restricted isometry property can be written as
1
− ∥XK ∥22 δK ≤ ∥AK XK ∥22 − ∥XK ∥22 ≤ ∥XK ∥22 δK
EA
or
1
EA ∥AK XK ∥22
1 − δK ≤ ≤ 1 + δK .
∥XK ∥22
for 0 ≤ δK < 1. For δK = 0 the isometry property holds for AK .
10.3.3 Coherence
The coherence index of a matrix A is defined as a maximal absolute value

of normalized scalar product of its two columns
µ = max |µ(m, k )| , for m ̸= k
where
M
1 1
µ(m, k) =
ψ (n )|2
∑ ψm (ni )ψk∗ (ni ) = EA ⟨ψm , ψk∗ ⟩ (10.12)
∑iM
=1 | k i i =1
Y Z
and ψk is the kth column of matrix A with E A = ψk , ψk∗ . This index plays
an important role in the analysis of measurement matrices.
The coherence index cannot be arbitrary small for an M × N matrix A
(M < N). The Welch upper bound relation holds
<
N−M
µ≥ . (10.13)
M ( N − 1)
The Welch limit for matrix A, whose columns have energy E A , will be
proven next.
Denote the elements of matrix E1 A H A by b(m, k ). By definition, the
A
trace of this matrix is a sum of its diagonal elements,
N
1 H
Trace{ A A} = ∑ b(m, m) = N.
EA m =1
1 H
Trace and energy are related to the eigenvalues λi of EA A A as
M
1 H
Trace{ A A } = ∑ λi
EA i =1
W W
W 1 H W2 N N M
W W
W E A AW =
A
∑ ∑ |b(m, k)|2 = ∑ λ2i .
2 m =1 k =1 i =1
We may write
( )2 W W
B C2 M M W 1 H W2
2 H
N = Trace{A A} = ∑ λi ≤ M∑ λ2i = M W A AW
W
W . (10.14)
i =1 i =1
E A 2
Schwartz’s inequality
(λ1 + λ2 + ... + λ M )2
≤ M. (10.15)
λ21 + λ22 + ... + λ2M
is used. Since the elements b(m, k ) are equal to the scalar products (10.12) of
columns ψm ∗ (n ) and ψ (n ) then
i k i
W W ' '2
W 1 H W2 M M M M ' 1 '
W
WE
A
A A W =
W ∑ ∑ |b(m, k)|2 = ∑ '
A
ψ ∗
∑ ' E m k ''
⟨ , ψ ⟩
2 m =1 k =1 m =1 k =1
M M
= ∑ ∑ |µ(m, k)|2 ≤ ( N + N ( M − 1)µ2 ). (10.16)
m =1 k =1
with µ(m, m) = 1 and |µ(m, k )| ≤ µ for m ̸= k. Relation (10.13) follows, from
N 2 ≤ M ( N + N ( N − 1 ) µ2 ).
The equality holds for matrices that form an equiangular tight frame.
From the presented proof for the Welch bound we can see that two inequal-
ities in (10.14) and (10.16) become equalities if
λ1 = λ2 = ... = λ M
∗
|⟨ψm , ψk ⟩| = µ for any m, k.
A matrix satisfying these properties is an equiangular tight frame. The par-

tial DFT matrix with a large number of columns for some specific combi-
nations of rows can be treated as a rough approximation of an equiangular
tight frame.
As it will be seen later by using a measurement matrix with a smallest
possible µ we will be able to reconstruct the signal of a largest possible
sparsity K, with a given number of measurements M. It is interesting to note
that a number of optimization procedures, in order to find the best possible
measurement matrix, uses the minimization of this parameter.
Example 10.1. Since Schwartz’s inequality for discrete-time signals
( )2
M M M
∑ x (n)y(n) ≤ ∑ x 2 ( n ) ∑ y2 ( n ).
n =1 n =1 n =1
will be used few more times (in various forms), within this chapter, we will
present its proof here. Note that with y(n) = 1 and x (n) = λn it produces
(10.15). Previous inequality easily follows from
M M
0≤ ∑ ∑ (x(n)y(m) − x(m)y(n))2
n =1 m =1
M M M M M M
= ∑ ∑ x 2 ( n ) y2 ( m ) − 2 ∑ ∑ x (n)y(n) x (m)y(m) + ∑ ∑ x 2 ( m ) y2 ( n ).
n =1 m =1 n =1 m =1 n =1 m =1
Since the first and last sums are equal, Schwartz’s inequality follows from
( )2
M M M
2 2
2 ∑x (n) ∑y (m) − 2 ∑ x (n)y(n) ≥ 0.
n =1 m =1 n =1
With y(n) = 1 and x (n) = | x (n)| Schwartz’s inequality can also be written as
( )2
M M
∑ |x(n)| ≤M ∑ |x(n)|2 (10.17)
n =1 n =1
or
√ 1
∥ x ∥1 ≤ M ∥x∥2 or ∥ x ∥2 ≥ √ ∥ x ∥1
M
with [
\ M
M \
∥ x ∥1 = ∑ | x (n)| and ∥x∥2 = ] ∑ | x (n)|2 .
n =1 n =1
Equality in this relation holds when | x (n)| = Cy(n) = C, i.e., for | x (1)| =
| x (2)| = ... = | x ( M)|.
For a K sparse vector X holds
( )2
K K
∑ |X (ki )| ≤ K ∑ | X (k i )|2
i =1 k =1
1
∥ X ∥2 ≥ √ ∥ X ∥1 .
K
10.3.4 Restricted Isometry and Coherence
For a measurement matrix A we may write
∥AX∥22 = ∥y∥22 = |y(0)|2 + |y(1)|2 + ... + |y( M − 1)|2

= | x (n1 )|2 + | x (n2 )|2 + ... + | x (n M )|2
' '2 ' '2 ' '2
' N −1 ' ' N −1 ' ' N −1 '
' ' ' ' ' '
= ' ∑ ψn1 (k) X (k)' + ' ∑ ψn2 (k) X (k )' + ... + ' ∑ ψn M (k ) X (k )'
' k =0 ' ' k =0 ' ' k =0 '
( ) ( )
M M
= ∑ |ψni (0)|2 |X (0)|2 + ... + ∑ |ψni ( N − 1)|2 | X ( N − 1)|2
i =1 i =1
% ;
N −1 N −1 M
∗
+ ∑ ∑ 2 Re X (k1 ) X (k2 ) ∑ ψni (k1 )ψn∗i (k2 ) . (10.18)
k 1 =0 k 2 = k 1 +1 i =1
2
Using E A = ∑iM
=1 |ψni (k )| and µ (k 1 , k 2 ) =
1
EA ∑iM ∗
=1 ψni (k 1 )ψni (k 2 ) we
get
N −1 N −1 N −1
∥AX∥22 = E A ∑ | X (k )|2 + ∑ ∑ 2 Re { X (k1 ) X ∗ (k2 )µ(k1 , k2 ) EΨ } .
k =0 k 1 =0 k 2 = k 1 +1
(10.19)
Since the restricted isometry property reads
' '
' 1 2 2 '' 2
'
' E ∥AX∥2 − ∥X∥2 ' ≤ δK ∥X∥2 , (10.20)
A
the value of restricted isometry constant δK , for an arbitrary signal X (k ), is

' N −1 N −1 ∗
'
'∑ '
' k1 =0 ∑k2 =k1 +1 2 Re { X (k1 ) X (k2 )µ(k1 , k2 )} '
δK = max ' 2 '. (10.21)
' ∑ N −1 | X (k )|
k =0
'
Value on the right side of inequality is highly signal dependent. We will find
an estimate of its bound. Since
|Re { X (k1 ) X ∗ (k2 )µ(k1 , k2 )}| ≤ | X (k1 ) X ∗ (k2 )| |µ(k1 , k2 )|

≤ µ | X (k1 ) X ∗ (k2 )|
we can write
% ;
∑kN1− 1 N −1 ∗
=0 ∑i =k2 +1 2 | X (k 1 ) X (k 2 )|
δK ≤ µ max 2
.
∑kN=−01 | X (k )|
For example, for sparsity K = 2 only two specific values of X (k ) are

nonzero. Assume that their positions are k = k1 and k = k2
N −1
∥A2 X∥22 = E A ∑ | X (k)|2 + 2 Re { X (k1 ) X ∗ (k2 )µ(k1 , k2 ) EΨ } .
k =0
Then we get ' '

' X (k1 ) X ∗ (k 2 ) '
' '
δ2 ≤ 2µ max ' '=µ (10.22)
' | X (k1 )|2 + | X (k2 )|2 '
since
| X (k1 )|2 + | X (k2 )|2
≥ 2. (10.23)
| X (k1 ) X ∗ (k2 )|
The maximal value in Schwartz’s inequality (10.22) is achieved for | X (k1 )| =
| X (k2 )| and µ = max |µ(k1 , k2 )|. Inequality (10.23) easily reduces to the well
known inequality
1
(a + ) ≥ 2
a
for a > 0 and ( a + 1a ) = 2 for a = 1. Since the limit value may be achieved for
a specific signal, if our aim is that (10.20) holds for any signal, we may write
δ2 = µ.
For K = 3 inequality (10.20) and (10.21) assume the form
' 1 2 2 ''
'
' E A ∥ A3 X ∥2 − ∥ X ∥2 ' | X (k1 ) X ∗ (k2 )| + | X (k1 ) X ∗ (k3 )| + | X (k2 ) X ∗ (k3 )|
' ' ≤ 2µ
' ∥X∥22 ' | X (k1 )|2 + | X (k2 )|2 + | X (k3 )|2
( )
(| X (k1 )| + | X (k2 )| + | X (k3 )|)2
= − 1 µ ≤ (3 − 1) µ = 2µ.
| X (k1 )|2 + | X (k2 )|2 + | X (k3 )|2
where Schwartz’s inequality (10.17) is used with M = 3 and x (n) = | X (k n )|.

Therefore
δ3 ≤ (K − 1)δ2 = 2µ.
The maximal value of the signal dependent term is achieved if | X (k1 )| =

| X (k2 )| = | X (k3 )|.
The results can easily be generalized to any K ≥ 2 using
K K ' ' * +2
K
2∑ ∑ ' X (k i ) X ∗ (k j )' ∑ | X (k i )|
i =1 j = i +1 i =1
K
= K
− 1 ≤ ( K − 1 ).
2 2
∑ | X (k i )| ∑ | X (k i )|
i =1 i =1
and Schwartz’s inequality (10.17) with x (n) = | X (k n )|.

The restricted isometry constant inequality is
δK ≤ (K − 1)δ2 = (K − 1)µ.
Equality holds for | X (k1 )| = | X (k2 )| = ... = | X (k K )|.

Since δK ≤ (K − 1)δ2 = (K − 1)µ then the matrix satisfies the restricted
isometry relation with
' '
' 1 2 2 '' 2
' −
'E ∥ AX ∥ 2 ∥ X ∥ 2 ' ≤ ( K − 1) µ ∥ X ∥2 .
A
In general, it does not mean that that there is no lower values of bound
δK such that the restricted isometry inequality is satisfied. This is just an
estimate of the upper bound value of the constant δK . Equality could be
checked by examining the imposed inequality conditions.
' '
' '
For the DFT matrix with |ψni (k )| = 'e j2πk/N /N ' = 1/N and E A =
M/N 2 from (10.19) we get
N −1
M
∥AX∥22 = ∑ | X (k)|2 (10.24)
N2 k =0
% ;
1 N −1 N −1
∗
M
j2πni (k1 −k2 )/N
+ 2 ∑ ∑ 2 Re X (k1 ) X (k2 ) ∑ e .
N k =0 k i =1
1 2 = k 1 +1
Introducing notation
% ;
N −1 N −1 M
1 ∗ j2πni (k1 −k2 )/N
α= ∑ ∑
M ∥X∥22 k1 =0 k2 =k1 +1
2 Re X (k1 ) X (k2 ) ∑ e (10.25)
i =1
we can write
N2
∥AX∥22 = ∥X∥22 + α ∥X∥22 . (10.26)
M
For M = N it is easy to check that the isometry property
N ∥AX∥22 = ∥X∥22
holds since ∑iM

=1 e
j2πni (k1 −k2 )/N = 0 for M = N and n ∈ M = N = {0, 1, ..., N −
i
1}.
The restricted isometry property is satisfied with δ = max |α|.
Example 10.2. The value of α, for signals with sparsity K ≥ 2 in the DFT domain,
can be related to the average power of available samples/measurements y(n)
2 2
defined by Py = ∑iM =1 | x ( ni )| /M = ∥ AX ∥2 /M and the average power of
signal Px = ∑nN=−11 | x (n)|2 /N = ∥X∥22 /N 2 . The constant α from (10.26) can
be written in the form
N2
M ∥AX∥22 − ∥X∥22 1
M ∥AX∥22 − 1
N2
∥X∥22 Py
α= = = − 1.
∥X∥22 1
N2
∥X∥22 Px
If all samples are used (M = N) then Py = Px and α = 0. The isometry

property holds for any signal sparsity, as expected. For an arbitrary M < N
the restricted isometry property will not hold if Py = 0 can be obtained for
any combination of M out of N signal samples. Then |α| = 1.
The uncertainty principle for a discrete signal x (n) and its DFT states
that a product of the number of nonzero samples in the time domain ( N −
Nz ) and the number of nonzero values in the frequency domain K is always
greater or equal to the total number of signal samples N. It can be written as

K ( N − Nz ) ≥ N,
where Nz is the number of zero-valued signal samples in the time domain.
The number of zero values in a signal is such that N − Nz ≥ N/K or Nz ≤
N − N/K. For a unique reconstruction of a signal whose sparsity is K we
should be able to reconstruct a signal of sparsity 2K using any combination
of M available signal samples. Then Nz ≤ N − N/(2K ) should hold. It means
that the number of available samples should be greater than the maximal
number of zero signal values (to avoid the event Py = 0 with probability 1),
that is
2K − 1
M > Nz = N
2K
samples are needed so that Py ̸= 0 for any combination of M available signal
samples/measurements. For example, for K = 1 the unique solution can be
obtained with M > N/2 signal samples, for K = 2 with M > 3N/4, for K = 4
using M > 7N/8, and so on. If the number of missing samples is denoted by
Q = N − M then the condition for a unique reconstruction of signal with
sparsity K using any M < N signal samples requires that the number of
unavailable samples Q satisfies N − Q > N (1 − 1/(2K )) or
N N
K< = .
2Q 2( N − M )
This is an interesting and simple, but pessimistic bound. It will be discussed
again at the end of this chapter.
10.3.5 Restricted Isometry and Eigenvalues
The restricted isometry property is satisfied for a K sparse signal, with a

restricted isometry constant δK , if the inequality
' '
' 1 2 2 '' 2
'
' E ∥AK XK ∥2 − ∥XK ∥2 ' ≤ δK ∥XK ∥2
A
holds for any combination of K out of N columns AK of measurement matrix

A whose order is M × N. Note that (for real-valued A)
1
EA ∥AK XK ∥22 1 T T
EA XK AK AK XK
= .
∥XK ∥22 XKT XK
According to the standard matrix norm relation
∥BX∥22 X T B T BX
dmin ≤ = ≤ dmax ,
∥X∥22 ∥B∥22
where dmin and dmax denote the minimal and the maximal eigenvalue of
Gram matrix B T B. Eigenvalues of Gram matrix are real and nonnegative.
In our case
1 T
BT B = A AK .
EA K
Using this inequality we can write
1
EA ∥AK XK ∥22
dmin ≤ ≤ dmax
∥XK ∥22
1
EA ∥AK XK ∥22
1 − δmin ≤ ≤ 1 + δmax
∥XK ∥22
where constants δmin and δmax are defined by δmin = 1 − dmin , δmax =
dmax − 1. A symmetric form of the restricted isometry property is commonly
used with
δK = max{δmin , δmax }.
A symmetric restricted isometry property inequality
1
EA ∥AK XK ∥22
1 − δK ≤ ≤ 1 + δK
∥XK ∥22
1 T
is obtained. It can be related to the condition number of matrix EA AK AK
defined by
! 6
1 T dmax
cond AK AK =
EA dmin
Since
1 − δK ≤ dmin ≤ dmax ≤ 1 + δK
it means ! 6
1 T 1 + δK
cond A AK ≤ .
EA K 1 − δK
Small values of δK , close to 0, mean robust and stable invertibility of Gram
matrix. In theory 0 ≤ δK < 1 is sufficient.
If the eigenvalues of matrix E1 AKT AK are denoted by di then, by
A
definition,
1 T
det( A A K − di I ) = 0
EA K
The eigenvalues λi of a matrix E1 AKT AK −I by definition satisfy

A
- .
1 T
det( A A K − I − λi I ) = 0
EA K
1 T
det( A AK −(λi + 1)I) = 0.
EA K
1 T 1 T
Relation between the eigenvalues of EA AK AK and EA AK AK −I is
λi = di − 1.
In a symmetric case the restricted isometry property bounds δmin , δmax
are symmetric for small sparsity, while for large sparsity the value δmax
dominates. It is common to calculate δK = δmax or
! * +6
1 T
δK = dmax − 1 = λmax = max eig AK AK −I , (10.27)
EA
where λmax is the maximal eigenvalue of E1 AKT AK −I for all ( N K ) combina-

A
tion of {k1 , k2 , ..., k K } ⊂ {0, 1, 2, ..., N − 1}. The restricted isometry property is
satisfied with δK = λmax . Then δK is calculated as the maximal value over all
possible realizations of matrix AK from matrix A (including all matrices of
order lower than K). This calculation is an NP hard problem.
Example 10.3. Gaussian zero-mean random signals are used as the measurement
basis functions
1
ψk (n) = N (0, ).
M
These functions are used to weight measurements of a sparse signal X (k )
of sparsity K. The total number of samples is N = 2048. The number of
measurements is M = 1024. Form the measurements matrix A whose di-
mension is M × N. Then form 10, 000 random realizations of AK for K =
8, 16, 32, 256, 1024 (using K randomly positioned columns of A). Calculate all
eigenvalues of AK T A (Wishart matrix) and of A T A − I. Estimate the largest
K K K
and the lowest value for each case and estimate the restricted isometry con-
stant.
⋆Fig.10.5 shows the histograms (normalized) of the eigenvalues for
N = 2048, M = 1024 and K = 8, 16, 32, 256, 1024. The limits fit well with the
expected mean values of the bounds
( = )2
K
E{dmax ( M, K )} = 1 + (10.28)
M
( = )2
K
E{dmin ( M, K )} = 1 − ,
M
derived√in literature for√a large M. Dashed thick vertical lines indicate the
values 2 − 1 and −( 2 − 1) for λ. Later it will be shown that these
limits play an important role in the definition of a sufficiently small δK . The
absolute reconstruction limit δK = 1 is achieved first with E {dmax ( M, K )} =
* F +2 F
K K
√
1+ M = 2 or M ≤ 2 − 1 for K ≤ 0.1716M. We can see than the
case K = 16 is the last one whose eigenvalues in 10, 000 realizations are
within limits, meaning that M = 1024 observations are sufficient for unique
reconstruction (in the sense of these limits) of K = 8 sparse signal (for a K
sparse signal the reconstruction requires that all limits and constants are
satisfied for a 2K sparse signal). Note that the presented values are only
the mean values. Values dmax ( M, K ) and dmin ( M, K ) are random variables.
Minimal and maximal values obtained in 10, 000 realizations are given in the
table. √ √
Limit 2 − 1 in λ or 2 in d is achieved using (10.28) for K = 0.0358M.
For M = 1024 its value is K = 36.6. Therefore this kind of bounds estimate
is optimistic. The value of the bound determined by the mean value is
lower than the maximal value based bound of a random variable, as we can
see from the table. Calculation of the bounds with satisfactory probability,
taking into account stochastic nature of eigenvalue limits, may be found in
literature.
K = 8, λmin = −0.24, λmax = 0.27,
K = 16, λmin = −0.30, λmax = 0.35,
K = 24, λmin = −0.34, λmax = 0.41,
K = 32, λmin = −0.37, λmax = 0.48,
K = 64, λmin = −0.47, λmax = 0.65,
K = 128, λmin = −0.60, λmax = 0.91,
K = 256, λmin = −0.76, λmax = 1.32,
K = 1024, λmin = −0.98, λmax = 3.08.
Limit cases for K/M ≪ 1 and for the case K = M easily follow.
Example 10.4. Write the full DFT transformation matrix for a signal of N = 8
samples.
(a) Show that it satisfies the unitary and isometry property (restricted
isometry property with δ = 0).
(b) Write the measurement matrix A if the number of available signal
samples/measurements in time domain is M = 6.
(c) If the sparsity in the DFT domain is K = 2 what is the form of the
submatrix A2 and the isometry constant δ2 .
(d) Write δ2 in terms of coherence index µ.
(e) Consider cases with K = 3 and K = 4. Comment the results.
pλ (ξ) pd (ξ)
2 K=8 2 K=8
1 1
0 0
-1 -0.5 0 0.5 1 0 0.5 1 1.5 2
pλ (ξ) pd (ξ)
2 K=16 2 K=16
1 1
0 0
-1 -0.5 0 0.5 1 0 0.5 1 1.5 2
pλ (ξ) K=32 pd (ξ)

2 2 K=32
1 1
0 0
-1 -0.5 0 0.5 1 0 0.5 1 1.5 2
p (ξ) K=64 pd (ξ)

2 λ 2 K=64
1 1
0 0
-1 -0.5 0 0.5 1 0 0.5 1 1.5 2
3 pλ (ξ) K=256 3 p (ξ) K=256

d
2 2
1 1
0 0
-1 0 1 2 0 1 2 3
15 p (ξ) K=1024 15 p (ξ) K=1024
λ d
10 10
5 5
0 0
-1 0 1 2 3 0 1 2 3 4
Figure 10.5 Histograms (normalized) of the eigenvalues of AKT AK (Wishart matrix) and
AKT AK −I matrix for N = 2048, M = 1024 and K = 8, 16, 32, 256, 1024. Dashed thick vertical lines
√ √
show the limits 2 − 1 and −( 2 − 1) sufficient for unique K/2 signal reconstruction.
⋆(a) For the DFT the transformation matrix is
x= ΨX
x = [ x (0), x (1), x (2), x (3), x (4), x (5), x (6), x (7)] T
X = [ X (0), X (1), X (2), X (3), X (4), X (5), X (6), X (7)] T
⎡ ⎤∗
1 1 1 1 1 1 1 1
⎢ 1 W81 W82 W83 W84 W85 W86 W87 ⎥
⎢ ⎥
⎢ 1 W82 W84 W86 W88 W810 W812 W814 ⎥
⎢ 3 6
⎥
1⎢ 1 W8 W8 W8 W812 W815 W818 W821
9 ⎥
Ψ= ⎢ ⎥
8⎢⎢ 1 W 4
8 W 8
8 W 8
12 W 16 W 20 W 24 W 28
8 8 8 8
⎥
⎥
⎢ 1 W 5 W 10 W 15 W 20 W 25 W 30 W 35 ⎥
⎢ 8 8 8 8 8 8 8 ⎥
⎣ 1 W 6 W 12 W 18 W 24 W 30 W 36 W 42 ⎦
8 8 8 8 8 8 8
1 W87 W814 W821 W828 W835 W842 W849
where ∗ denotes complex conjugate and W8nk = exp(− j2πnk/8). The trans-
formation matrix W is unitary matrix according Parseval’s theorem

N −1
1 N −1
∑ x (n)y∗ (n) = X ( k )Y ∗ ( k )
n =0
N k∑
=0
The isometry property follows from the unitary property of Ψ as
⟨ΨX, ΨX⟩ = EΨ ⟨X, X⟩

1
∥ΨX∥22 = ∥X∥22
N
N −1
1 N −1
∑ |x(n)|2 = N ∑ |X (k)|2 .
n =0 k =0
As expected for the full DFT matrix, the isometry property is satisfied, since
' '
' N ∥ΨX∥2 − ∥X∥2 '
' 2 2'
' ' ≤ δ with δ = 0.
' ∥X∥22 '
(b) For M = 6 random samples/measurements at
n i ∈ { n1 , n2 , n3 , n4 , n5 , n6 } = M
⊂N = {0, 1, 2, 3, 4, 5, 6, 7}
the available signal values are
y = [ x (n1 ), x (n2 ), x (n3 ), x (n4 ), x (n5 ), x (n6 )] T

with
y = AX
⎡ ⎤∗
1 W8n1 W82n1 W83n1 W84n1 W85n1 W86n1 W87n1
⎢ 1 W8n2 W82n2 W83n2 W84n2 W85n2 W86n2 W87n2 ⎥
⎢ ⎥
1⎢
⎢ 1 W8n3 W82n3 W83n3 W84n3 W85n3 W86n3 W87n3
⎥
⎥
A= ⎢ ⎥
8⎢
⎢ 1 W8n4 W82n4 W83n4 W84n4 W85n4 W86n4 W87n4 ⎥
⎥
⎣ 1 W8n5 W82n5 W83n5 W84n5 W85n5 W86n5 W87n5 ⎦
1 W8n6 W82n6 W83n6 W84n6 W85n6 W86n6 W87n6
where * denotes complex-conjugate.

(c) A submatrix of A of the order K = 2, for two arbitrary nonzero
coefficients at k1 and k2 , is
⎡ ⎤∗
W n1 k 1 W8n1 k2
⎢ 8n2 k1 ⎥
⎢ W8 W8n2 k2 ⎥
⎢ ⎥
1⎢ W8n3 k1 W8n3 k2 ⎥
A2 = ⎢⎢
⎥
⎥
8 ⎢ W8n4 k1 W8n4 k2 ⎥
⎢ ⎥
⎣ W8n5 k1 W8n5 k2 ⎦
W8n6 k1 W8n6 k2
For the isometry property this matrix is a special case of (10.24) using only
k = k1 and k = k2 ,
6 B C 2 M N
∥A2 X∥22 = | X (k1 )|2 + | X (k2 )|2 + Re X (k1 ) X ∗ (k2 )∑6i=1 W8−ni k1 W8ni k2
64 64
! 6
B C M
2 ∗ (k ) − ni k 1 ni k 2
64 2 2 2 Re X ( k ) X ∑ W W
6 ∥ AX ∥2 − | X ( k 2 )| + | X (k 2 )| 6 1 2 8 8
i =1
2 2
= 2 2
| X (k1 )| + | X (k2 )| | X (k1 )| + | X (k2 )|
(d) Using the coherence definition
# $
ψk1 , ψk∗2
µ(k1 , k2 ) = # $
ψk1 , ψk∗
1
for the DFT we get
1 6 1 6
µ(k1 , k2 ) = ∑ W8−ni k1 W8ni k2 = ∑ e j2πni (k1 −k2 )/8 .
6 i =1 6 i =1
Maximal value of |µ(k1 , k2 )| is the coherence index
µ = max |µ(k1 , k2 )| . (10.29)

k1 ,k2,
k1 ̸=k2
The restricted isometry constant is

' B C'
' 64 ∥AX∥2 − | X (k )|2 + | X (k )|2 '
' 6 2 2 2 '
δ2 = '' 2 2
'
'
' | X (k2 )| + | X (k2 )| '
' '
' 2 Re { X (k ) X ∗ (k )µ(k , k )} '
' 1 2 1 2 '
= max ' 2 2 '
' | X (k1 )| + | X (k2 )| '
' '
' µ(k1 , k2 ) '
' '
≤ 2 | X (k1 ) X ∗ (k2 )| ' 2 2 '≤µ
' | X (k1 )| + | X (k2 )| '
The maximal value in this inequality is achieved for | X (k1 )| = | X (k2 )| and
max |µ(k1 , k2 )| = µ. Having in mind inequality for Re { X (k1 ) X ∗ (k2 )µ(k1 , k2 )}
the overall maximum is achieved for | X (k1 )| = | X (k2 )| with
arg{ X (k1 ) X ∗ (k2 )µ(k1 , k2 )} = rπ/2
where r is an integer.
To comment the results consider the least mean square solution of
system
A2 X =y
H
= A2H y
A2 A2 X
B C −1
X = A2H A2 A2H y
⎛⎡ ⎤T ⎡ ⎤ ∗ ⎞ −1
W8n1 k1 W8n1 k2 W8n1 k1 W8n1 k2
⎜⎢ ⎥ ⎢ ⎥ ⎟
⎜⎢ W8n2 k1 W8n2 k2 ⎥ ⎢ W8n2 k1 W8n2 k2 ⎥ ⎟
⎜⎢ ⎥ ⎢ ⎥ ⎟
⎜⎢ W8n3 k1 W8n3 k2 ⎥ ⎢ W8n3 k1 W8n3 k2 ⎥ ⎟
=⎜
⎜⎢
⎢ ⎥
⎥
⎢
⎢
⎥
⎥
⎟
⎟ X0
⎜⎢ W8n4 k1 W8n4 k2 ⎥ ⎢ W8n4 k1 W8n4 k2 ⎥ ⎟
⎜⎢ ⎥ ⎢ ⎥ ⎟
⎝⎣ W8n5 k1 W8n5 k2 ⎦ ⎣ W8n5 k1 W8n5 k2 ⎦ ⎠
W8n6 k1 W8n6 k2 W8n6 k1 W8n6 k2
where
X0 = NA2H y (10.30)
P QT
and A2H = A2∗ . Then by multiplying A2H A2 we get
- . −1
M Mµ∗ (k1 , k2 )
X= X0 , (10.31)
Mµ(k1 , k2 ) M
with
1 M j2πni (k1 −k2 )/N
M i∑
µ(k1 , k2 ) = e .
=1
The determinant of the system is

- .
M Mµ∗ (k1 , k2 )
det = M2 (1 − |µ(k1 , k2 )|2 ).
Mµ(k1 , k2 ) M
Obviously if
1 M j2πni (k1 −k2 )/N
M i∑
µ(k1 , k2 ) = e = ±1
=1
when
ρ2 = µ = max |µ(i, k )| = 1
the system does not have a (unique) solution. It means that measurements
y(n) are not independent and that during the projection of the N dimensional
space of the sparse vector X to the space of dimension M < N by the
linear transformation AX = y the information about one of the two nonzero
coordinates is lost, i.e. it is projected to zero and can not be recovered.
The inversion robustness in (10.31) is the highest when µ(k1 , k2 ) = 0.
The reconstruction is done in this case using the identity matrix. For values
of µ(k1 , k2 ) increasing toward 1 the determinant value M2 (1 − µ2 (k1 , k2 ))
reduces. It means the results in the reconstruction are multiplied by 1/ ( M2 −
M2 µ2 (k1 , k2 )). If there is noise in the measurements y, i.e., in the initial
estimate X0 = A H y, then the noise in the reconstruction will be increased,
meaning degradation of the signal-to-noise ratio. Therefore the values of
ρ2 = max |µ(i, k )| close to 1 are not desirable in the reconstruction, although
in theory, the reconstruction is possible. Reduction of the value of isometry
constant ρK toward zero will be of crucial importance in the application of
some reconstruction algorithms that will be presented later.
The values of
1 M j2πni (k1 −k2 )/N
M i∑
µ(k1 , k2 ) = e (10.32)
=1
for the DFT matrix are calculated for all possible (k1 , k2 ) and presented in
Fig.10.6. The coherence index value is equal to the maximal absolute value
of µ(k1 , k2 ). Signals of sparsity K = 2 (top), K = 3 (middle), and K = 4
(bottom) are considered for all possible positions of the available samples
ni and nonzero coefficients k i . The restricted isometry constant for this signal
with N = 8 samples and M = 6 observations (available samples) at ni for
i = 1, 2, 3, 4, 5, 6 is also calculated. The restricted isometry property
B constant
C
1 T
δK is calculated by using eigenvalues of the matrix Λ = eig 6 AK AK −I for
all possible nonzero positions of X (k ), as in (10.27). Then, for example for
K = 2, δ(k1 , k2 ) = λmax = max {Λ} is calculated for each possible AK . Finally
δ2 = maxk1 ,k2 δ(k1 , k2 ). Note that in this case equality in δK ≤ (K − 1)µ holds
for all K, where µ = maxk1 ,k2 |µ(k1 , k2 )|, Fig.10.6.
(e) The calculation is done for K = 3 and K = 4 as well. The restricted
isometry property is not satisfied for matrix AK in the case K = 4. The
Histogram of max|µ(k ,k )| Histogram of δ(k ,k )

i j 1 2
for all k , k , and n , i=1,2,3,4,5,6 for all k , k , and n , i=1,2,3,4,5,6
1 2 i 1 2 i
300 300
N=8, K=2, M=6 N=8, K=2, M=6
200 200
δ =0.333
2
100 100
0 0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
Histogram of max|µ(k ,k )| Histogram of δ(k ,k ,k )

i j 1 2 3
for all k , k , k , and n , i=1,2,3,4,5,6 for all k , k , k , and n , i=1,2,3,4,5,6
1 2 3 i 1 2 3 i
800 1200
N=8, K=3, M=6 N=8, K=3, M=6
1000
600
800
400 600
δ3=0.667
400
200
200
0 0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
Histogram of max|µ(k ,k )|
i j
Histogram of δ(k1,k2,k3,k4)
for all k , k , k , k , and n , i=1,2,3,4,5,6 for all k , k , k , k , and n , i=1,2,3,4,5,6
1 2 3 4 i 1 2 3 4 i
600
N=8, K=4, M=6 N=8, K=4, M=6
1000
400
500
200
δ4=1
0 0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
Figure 10.6 The coherence index value and the restricted isometry constant for signal with
N = 8 samples and M = 6 observations (available samples) at ni for i = 1, 2, 3, 4, 5, 6. Signals of
sparsity K = 2 (top), K = 3 (middle), and K = 4 (bottom) are considered for all possible positions
of the available samples ni and nonzero coefficients k i . The DFT is the transformation matrix.
maximal value of isometry constant is ρ4 = 1. It is interesting to note that

this value of isometry constant (when the isometry property does not hold)
is obtained for missing samples n7 = 1 and n8 = 5 (available samples x (ni ) are
at positions ni = 0, 2, 3, 4, 6, 7 for the nonzero positions of X (k) at k1 = 1, k2 = 3,
k3 = 5, and k4 = 7 or at k1 = 0, k2 = 2, k3 = 4, and k4 = 6. The same result
is obtained for missing samples n7 = 3 and n8 = 7 and the same nonzero
positions of the DFT coefficients X (k ). Having in mind the symmetry the
total number of realizations when the restricted isometry property does not
hold is 8 out of 1960 realizations. The probability that the restricted isometry
property is lost with an arbitrary signal of length N = 8, whose sparsity is
K = 4, with M = 6 observations is 0.0041.
10.3.6 Unique Reconstruction Condition and RIP
From the introductory examples we have seen that for a signal of sparsity
K = 1, two samples/measurements may produce full reconstruction. We
have also shown that any two samples/measurements may not be suffi-
cient. The solution is unique if the determinant of any second order linear
system of equations, for these measurements, is nonzero
- .
ψi (0) ϕ k (0 )
det ̸= 0
ψi (1) ϕ k (1 )
for all i ̸= k. In the case with K = 1 if there is more than M = 2 measurements

then at least any two of them should satisfy this condition. In the same way
we have concluded in the introductory analysis that for the case of sparsity
K = 2 at least four samples/measurements. The solution is unique if the
determinant of system for any signal of sparsity K = 4 is nonzero, (10.5). In
this way we can make a generalization for a K sparse signal. Obviously we
should have at least M ≥ 2K samples/measurements
y = AX (10.33)
They are sufficient for a unique reconstruction of a K sparse signal if M sam-

ples/measurements are independent in such a way that any 2K dimensional
sparse signal can be reconstructed from them, i.e., that all determinants of
2K order are nonzero
⎡ ⎤
ψk1 (n1 ) ψk2 (n1 ) ... ψk2K (n1 )
⎢ ψk (n2 ) ψk2 (n2 ) ... ψk2K (n2 ) ⎥
det(A2K ) = det ⎢
⎣
1 ⎥ ̸= 0
⎦ (10.34)
... ... ... ....
ψk1 (n2K ) ψk2 (n2K ) ... ψk2K (n2K )
for at least one combination of available measurements {n1 , n2 , ..., n2K } ⊂

{n1 , n2 , ..., n M }, with M ≥ 2K, and any combination of {k1 , k2 , ..., k2K } ⊂
{0, 1, 2, ..., N − 1}. Note that the number of 2K class combinations of N
N
elements is very large. It is (2K ).
Since our task is to check if there is 2K independent measurements, it
means that we want to check if the rank of matrix A2K is rank(A2K ) = 2K.
There is no need for combinations over ni (to form a quadratic matrix
2K × 2K from M × 2K matrix) if M > 2K since the rank of M × 2K matrix
A2K can be checked by checking the rank of a 2K × 2K matrix A2K T A
2K using
T
rank(A2K ) = rank(A2K A2K ).
T A
Matrix A2K 2K is the Gram matrix of A2K . For matrices A2K with complex
elements the conjugate transpose (Hermitian matrix) is used A2K H A . A
2K
T
way to check if the rank of A2K A2K is 2K is to calculate and check
T
det(A2K A2K ) = d1 d2 ...d2K ̸= 0
T A . All eigenvalues d of symmetric

where d1 d2 ...d2K are eigenvalues of A2K 2K i
T
matrix A2K A2K
T
di = eig(A2K A2K )
T A
are nonnegative. Rank of A2K T
2K is 2K if the minimal eigenvalue of A2K A2K
is
dmin > 0.
This should be satisfied for all combinations of {k1 , k2 , ..., k2K } ⊂ {0, 1, ..., N −
1}. It means that the reconstruction will be achieved if
% ;
1
EA ∥A2K X2K ∥22
min = dmin > 0.
∥X2K ∥22
For a practical matrix invertibility and robustness, commonly the condition

number ! 6
1 T dmax
cond A2K A2K =
EA dmin
is used. The restricted isometry in this sense means
1
EA ∥A2K X2K ∥22
1 − δ2K ≤ ≤ 1 + δ2K
∥X2K ∥22
with δ2K = max{1 − dmin , dmax − 1} and 1 − δ2K ≤ dmin ≤ dmax ≤ 1 + δ2K ,
! 6
1 T 1 + δ2K
cond A A2K ≤ .
E A 2K 1 − δ2K
It means that the invertibility is possible, in theory, if 0 ≤ δ2K < 1.

The previous analysis can be summarized by the next statement. If X
is a K sparse vector of dimension N then it can uniquely be reconstructed
from a reduced set of M samples/measurements y = AX if the measurement
matrix A is such that its submatrices A2K satisfy 2K restricted isometry
property with constant 0 ≤ δ2K < 1 for all combinations of 2K out of N
columns.
This statement will be proven by contradiction. Assume that X is a K
sparse and the system of samples/measurements y = AX does not produce
a unique solution. It means that two different signals X and H of sparsity K
may satisfy the same measurements equation
y = AX and y = AH.
Then
(AX − AH)=0
A(X − H)=0.
The vector X − H is in general 2K sparse since it contains K nonzero ele-

ments of X and K different nonzero elements of H. Within the restricted
isometry property context it means
' 1 2 2 ''
'
' EA ∥A(X − H)∥2 − ∥X − H∥2 '
' ' ≤ δ2K .
' ∥ X − H ∥2 2
'
Since A(X − H)= 0 it follows δ2K = 1. If
0 ≤ δ2K < 1
then ∥A(X − H)∥22 ̸= 0 meaning AX = y and AH = y is not possible for two

different vectors of sparsity K if δ2K < 1. This condition should be satisfied
for all combinations of {k1 , k2 , ..., k2K } ⊂ {0, 1, 2, ..., N − 1}.
Reconstruction with δ2K close to 1 should be avoided due to high influ-
ence of possible noise in data. Small values of δ2K , close to 0, means robust
and stable invertibility. Imposing any specific limit for condition number
is equivalent to imposing the restricted isometry property

M with a Nspecific
1 T A
constant δ2K . For example, the requirement that cond E A2K 2K < 2 is
A
the same as
0 ≤ δ2K < 1/3.
10.3.7 Rank and Spark of a Matrix
Consider a matrix A with M rows containing N ≥ M elements (N columns).

The rank of matrix A is equal to the largest number of independent columns
(rows). Obviously for a nonzero matrix
1 ≤ rank {A} ≤ M.
The spark of matrix A is the smallest number of dependent columns (rows).

By definition if one column contains all zero elements then spark {A} = 1.
In general,
2 ≤ spark {A} ≤ M + 1.
Example 10.5. Find the rank and spark of matrix
⎡ ⎤
1 0 1 3 2 2
A=⎣ 0 2 3 2 1 1 ⎦
−1 4 2 1 −1 3
The rank of matrix A is rank {A} = 3 since we may easily check that the
determinant of a matrix formed using first three columns of A is nonzero. If
that determinant was zero, then before concluding that rank of A is lower
than 3 we should try with all possible combinations of columns. If all com-
binations
M ofN3 columns were dependent, then we should check if rank {A} =
rank AA T = 2 by forming all possible 2 × 2 submatrices. If any of them has
a nonzero determinant then the rank would be 2, otherwise the rank would
be one when only one nonzero element of matrix A exists.
There are several methods for calculation of rank of matrix without
combinatorial search.MThe rank
N calculation can be simplified using the fact
that rank {A} = rank AA T . The only one matrix
⎡ ⎤
19 13 8
AA T = ⎣ 13 19 18 ⎦
8 18 32
M N
should be checked for the possible rank 3. Note also that det AA T =
M N M N
λ1 λ2 λ3 where {λ1 , λ2 , λ3 } = eig AA T . Therefore for rank AA T = 3 all
eigenvalues should be different from zero. In addition, the rank is equal to

the number of nonzero elements of {λ1 , λ2 , λ3 }.
For spark we have to find minimal number of dependent columns.
-Obviously there is no all zero column, thus
spark {A} > 1.
-The first possible solution is spark {A} = 2. The spark of matrix A is 2 if

there are two dependent columns. Two columns are dependent if there is
linear relation between their elements. It means that spark {A} = 2 if there is
any par of two columns such that its rank is equal to one, i.e., that they are
proportional to each other. Here we have to check rank of all combinations
of two columns. the total number of combinations is (62) = 15. We have
calculated all combinations and found that for all of them rank was 2. It
means that spark {A} > 2.
-Next we have to check all possible combinations of three columns.
There are (63) = 20 combinations of 3 columns. We have calculated rank of all
20 combinations of columns and found that there is a dependent combination
of columns. Namely, the fourth column of A is obtained by multiplying the
first column by 3 and adding it to the second column. Since we have found
at least one dependent combination of columns, further search is not needed.
Therefore the lowest number of dependent columns is 3. It means that
spark {A} = 3.
If there were not dependent columns in this calculation using combinations

of 3 columns, then the spark would be spark {A} = M + 1 = 4 by definition.
Relation between the rank and spark can be established based on

following consideration. If a matrix has spark {A} = 3 it means that all
combinations of two columns are independent and that there is at least
one dependent combination of three columns. Since the rank of matrix
rank {A} = 2 if at least one combination of two columns are independent
then obviously if spark {A} = 3 then rank {A} ≥ 2. In general if
spark {A} = p
then it means that all combinations of p − 1 columns are independent

(including all combination of lower than p − 1 number of columns). Since
the rank requires that at least one combination of p columns is independent
then
rank {A} ≥ p − 1.
It means that
rank {A} ≥ spark {A} − 1

spark {A} ≤ rank {A} + 1.
If a matrix A has M rows and N ≥ M columns then 0 ≤ rank {A} ≤ M

and 1 ≤ spark {A} ≤ M + 1. In a special case of orthogonal square matrix
N = M, by definition spark {A} → ∞. The sense of this definition will be
clarified later.
10.3.8 Spark and the Solution Uniqueness
The spark of measurement matrix is used for very simple definition of the
existence of the sparsest solution of a minimization problem
min ∥X∥0 subject to y = AX.
If the vector X is of sparsity K, with
∥ X ∥0 = K
then if
1
K < spark {A}
2
the solution X is unique.
In order to prove this statement consider a matrix A whose spark is
spark {A}. Then for a sparse vector X of sparsity K = spark {A} obviously
there exists such a combination of nonzero elements in X so that they
coincide with the dependent columns. Then we can obtain
AX = 0.
This property is used for the spark definition as well
min ∥X∥0 such that AX = 0.
Note that for any X of sparsity K < spark {A} the relation AX = 0 will
not hold, since nonzero signal elements of X cannot produce a zero result
when multiplied by columns which are independent. Since K < spark {A}
it means that in all cases K columns are independent.
The proof that K < 12 spark {A} means that X, being solution of AX = y,
is unique, will be based on contradiction.
Assume that X is a solution and that it satisfies K < 12 spark {A} but
that there is another solution H such that AH = y which is also sparse with
sparsity lower than the sparsity of X, i.e., lower than 12 spark {A}.
Since
AH = AX = y
A(H − X) = 0
then
spark {A} = min ∥H − X∥0 such that A(X − H) = 0.
or
spark {A} = ∥H − X∥0 ≤ ∥H∥0 + ∥X∥0

spark {A} − ∥H∥0 ≤ ∥X∥0 .
If there is another solution H such that ∥H∥0 < 12 spark {A} then from
the last inequality follows ∥X∥0 > 12 spark {A} . This is a contradiction to
the assumption that both solutions H and X have sparsity lower than
1
2 spark { A }.
The spark of matrix can be related to the coherence of matrix. The
relation is
1
spark {A} = 1 +
µ(A)
where µ(A) (or just µ) is the coherence index of matrix A. The proof is based
on the quadratic norm positivity of the matrix A T A.
The coherence index value is (10.29)
'Y '
' ψ , ψ∗ Z '
'Y i kZ'
µ(A) = max |µ(i, k )| = max ' '.
i ̸=k ' ψi , ψi∗ '
where ψi are
Y columns
Z of matrix A. It is assumed that all columns are of equal
1 ∗
energy M ψi , ψi = 1.
The maximal possible value of spark is spark {A} = M + 1 when there
is no dependent columns. Then
1
K< ( M + 1) .
2
For K sparse signal we must have at least
M ≥ 2K.
Note that for random matrices we have
spark {A} = M + 1
with a very high probability. However, in the cases of noisy signals or ap-
proximately sparse signals, more robust calculations are required increasing
the number of required observations. For a quadratic and orthogonal matrix
A the coherence index is µ(A) = 0 and for that matrix spark {A} → ∞, by
definition.
For the illustrative example from the beginning of this chapter we
had a condition that one false bag can be discovered if we performed two
measurements
⎡ ⎤
- . - . X (0 )
y (0) ψ0 (0) ψ1 (0) ... ψN −1 (0) ⎢ ⎢ X (1 ) ⎥
⎥
= ⎣ ⎦
y (1) ψ0 (1) ψ1 (1) ... ψN −1 (1) ...
X ( N − 1)
y = AX
such that ψi (0) ϕk (1) − ψi (1) ϕk (0) ̸= 0 for any combination of columns i and
k. It means that two columns are not dependent, i.e., that
ψi (0) ψ (1 )
= i does not hold for any i ̸= k.
ψk (0) ψk (1)
Assuming that there is no an all zero column then spark {A} = 3 meaning
that a signal X of sparsity K < 12 (2 + 1) can be recovered.
Within this framework we can now consider the case with three mea-
surements
⎡ ⎤
⎡ ⎤ ⎡ ⎤ X (0 )
y (0) ψ0 (0) ψ1 (0) ... ψN −1 (0) ⎢ ⎥
⎣ y(1) ⎦ = ⎣ ψ0 (1) ψ1 (1) ... ψN −1 (1) ⎦ ⎢ X (1 ) ⎥
⎣ ... ⎦
y (2) ψ0 (2) ψ1 (2) ... ψN −1 (2)
X ( N − 1)
y = AX.
Then one nonzero value of X can be recovered if the spark of A is

spark {A} ≥ 3. Since the spark in this case can assume a value up to 4, the
value of spark is equal or greater than 3 if any combination of two columns
are not dependent. The condition that spark {A} ̸= 2 is
ψi (0) ψ (1 ) ψ (2 )
= i = i does not hold for any i ̸= k.
ψk (0) ψk (1) ψk (2)
- .
ψi (0) ϕ k (0 )
In the notation of the determinants it means that det ̸= 0
- . ψi (1) ϕ k (1 )
ψi (1) ϕk (1)
or det ̸= 0. In the terminology of a matrix rank it means
ψi (2) ϕk (2)
that rank {A2 } = 2 for any submatrix A2 of two columns of A. The matrix
A2 has two columns and M rows. For the rank calculation there is no need
for combinations over rows since
M N
rank {A2 } = rank A2T A2 ,
where A2T A2 is 2 × 2 matrix. Still all combinations over different columns

should be checked. There are ( N2 ) = N ( N − 1)/2 of them.
Checking a spark of order p is an NP hard problem since all combina-
tion of p out of N elements should be checked.
Example 10.6. Gaussian zero-mean random signals are used as the measurement
basis functions
1
ψk (n) = N (0, ).
M
These functions are used to weight measurements of a sparse signal X (k )
whose sparsity is K and the total number of samples is N = 2048. The number
of measurements is M = 1024. Using the coherence relations estimate the
largest value of K so that the measurements matrix satisfies the restricted
isometry property with δ2K < 0.41 with a probability of 0.9999.
⋆The coherence index for columns k1 and k2 is
M
µ(k1 , k2 ) = ∑ ψk 1
(ni )ψk∗2 (ni ).
i =1
Since the variance of ψk1 (ni ) is 1/M then E A = 1. For Gaussian variables the
variance of random variable µ(k1 , k2 ) is
1 1
σ2 = Mσψ2 σψ2 = M = 1/M
MM
(see Problem 7.13). As a sum of large number of random variables the
resulting variable µ(k1 , k2 ) can be considered as Gaussian with variance
σ2 = 1/M. Since δ2K ≤ (2K − 1)µ where µ = max |µ(k1 , k2 )| then using the
equality δ2K = (2K − 1)µ in the estimation for a given δ2K then all absolute
values of µ(k1 , k2 ) should satisfy
δ2K
|µ(k1 , k2 )| ≤ µ =
( − 1)
2K
√
with a high probability P = erf(S/ 2), following
√
µ Mδ2K
S= =
σ (2K − 1)
sigma rule. In order to find P (and corresponding S) note that there are ( N2 )
different values of µ(k1 , k2 ). Assuming that they are independent
√ B
Mδ2K √ C( 2 )
N
Pr{max |µ(k1 , k2 )| ≤ } = erf(S/ 2)

(2K − 1)
for ( N2 ) = 2038 · 1024 ∼ 106 the value S = 6.5 will produce the above proba-
bility of order 0.9999. It means
√
Mδ2K
(2K − 1) = = 2.02.
S
The largest value of K according to this analysis is K = 1. This is a
very pessimistic estimation, as compared to the analysis in Fig.10.5. There
we could expect a unique reconstruction, with the same probability, for
K = 16/2 = 8. Note that here M = δS2K (2K − 1)2 holds. Calculations closer
to the expected results are derived in literature.
Welsh bound <
N−M
µ≥
M ( N − 1)
and the restricted isometry property with
<
N−M
δK = (K − 1)
M ( N − 1)
for M ≪ N lead to (K − 1)2 = δK2 M. This leads to K = O( M1/2 ) what is much

lower than the theoretically known reconstruction limit being of order of
M/ ln( N/M).
10.4 NORM-ZERO BASED RECONSTRUCTION
Although the ℓ0 -norm cannot be used in the direct minimization, the algo-
rithms based on the assumption that some coefficients X (k ) are equal to
zero, and the minimization of the number of remaining nonzero coefficients
that can reconstruct sparse signal, may efficiently be used.
10.4.1 Direct Combinatorial Search
The reconstruction process can be formulated as finding the positions and

the values of K nonzero coefficients X (k ) of a sparse signal (or all signal x (n)
values) using a reduced set of signal values x (ni ),
ni ∈ M = {n1 , n2 , ..., n M } ⊂ {0, 1, 2, ..., N − 1}
such that
min ∥X∥0 subject to y = AX
where ∥X∥0 = card{X} = K. Consider a discrete-time signal x (n). Signal is
sparse in a transformation domain defined by the basis functions set ψk (n),
k = 0, 1, ..., N − 1. The number of nonzero transform coefficients K is much
lower than the number of the original signal samples N, i.e., X (k ) = 0 for
/ {k1 , k2 , ..., k K } = K,
k∈
K ≪ N. A signal
x (n) = ∑ X (k )ψk (n). (10.35)
k∈{k1 ,k2 ,...,k K }
of sparsity K can be reconstructed from M samples, where M ≤ N. In

the case of signal x (n) which is sparse in the transformation domain there
are K nonzero unknown values X (k1 ), X (k2 ),...,X (k K ). Other transform
coefficients X (k ), for k ∈ / {k1 , k2 , ..., k K } = K, are zero-valued.
Just for the beginning assume that the transformation coefficient posi-
tions {k1 , k2 , ..., k K } are known. Then the minimal number of equations to
find the unknown coefficients (and to calculate signal x (n) for any n) is K.
The equations are written for at least K time instants ni , i = 1, 2, ..., M ≥ K,
where the signal is available/measured,
∑ X (k)ψk (ni ) = x(ni ), for i = 1, 2, ..., M ≥ K. (10.36)

k∈K
In a matrix form this system of equations is
AK XK = y, (10.37)
where XK is the vector of unknown nonzero coefficients values (at the

known positions) and y is the vector of available signal samples,
XK = [ X (k1 ) X (k2 ) ... X (k K )] T (10.38)

y = [ x ( n1 ) x (n2 ) ... x (n M )] T
⎡ ⎤
ψk1 (n1 ) ψk2 (n1 ) ... ψkK (n1 )
⎢ ψk (n2 ) ψk2 (n2 ) ... ψkK (n2 ) ⎥
AK = ⎢
⎣
1 ⎥.
⎦ (10.39)
... ... ... ....
ψk1 (nK ) ψk2 (nK ) ... ψkK (nK )
Matrix AK is the measurements matrix A with the columns corresponding

to the zero-valued transform coefficients k ∈ / {k1 , k2 , ..., k K } being excluded.
For a given set {k1 , k2 , ..., k K } = K the coefficients reconstruction condition
can be easily formulated as the condition that system (10.37) has a (unique)
solution, i.e., that there are K independent equations,
rank (AK ) = K.
Note that this condition does not guarantee that another set {k1 , k2 , ..., k K } =
K can also have a (unique) solution, for the same set of available samples.
The uniqueness of solution is considered within the previous subsections.
It requires rank (A2K ) = 2K for any submatrix A2K of the measurements
matrix A. It will be addressed for the DFT case again later in this chapter.
System (10.36) is used with K ≪ M ≤ N. Its solution, in the mean
squared sense, follows from the minimization of difference of the available
signal values and the values Rproduced
S by inverse transform of the recon-
structed coefficients, min X (k) e2 where
' '2
' '
2 ' '
e = ∑ 'y(n) − ∑ X (k )ψk (n)' =
n∈M
' k∈K
'
= (y − AK XK ) H (y − AK XK ) = ∥y∥22 − 2XKH AKH y + XKH AKH AK XK (10.40)
or M N
min (y − AK XK ) H (y − AK XK )
where exponent H denotes the Hermitian conjugate. The derivative over

X ∗ ( p) is (Chapter I, equation (1.13))
∂e2
= 2 ∑ (y(n) − ∑ X (k )ψk (n))ψ∗p (n).
∂X ∗ ( p) n∈M k∈K
The minimum of quadratic form error is reached for
∑ ψ∗p (n)y(n) = ∑ ∑ ψk (n)ψ∗p (n)X (k)

n∈M n∈M k∈K
for p = 0, 1, ..., N − 1.
In matrix form this system of equations reads
AKH y = AKH AK XK .
Its solution is B C −1
XK = AKH AK AKH y. (10.41)
It can be obtained by a symbolic vector derivation of (10.40) as
∂e2
= −2AKH y + 2AKH AK XK = 0.
∂XKH
If we do not know the positions of the nonzero values X (k ) for k ∈ {k1 ,

k2 , ..., k K } = K then all possible combinations of {k1 , k2 , ..., k K } ⊂ N should
be tested. There are ( N K ) of them. It is not computationally feasible problem.
Thus we must try to find a method to estimate {k1 , k2 , ..., k K } in order to
recover values of X (k).
10.4.2 Pseudoinverse matrix
In (10.41) we have used a form of the pseudoinverse of a matrix. In general,

if A H A is invertible then the pseudoinverse of matrix A is defined by
B C −1
pinv(A) = A H A AH .
This is the left pseudoinverse since
pinv(A)A = I
In the case that AA H is invertible the pseudoinverse is defined by

B C −1
pinv(A) = A H AA H .
It is the right pseudoinverse since
Apinv(A) = I.
For the considered matrices, for an M × N matrix A with M < N, the

matrix AA H of dimension M × M can be invertible (the highest possible
rank of an M × N matrix, with M < N, is M). For a matrix AK whose
dimension is M × K, with K < M, the matrix A H A of dimension K × K can
be invertible.
For a system of equations AX = y if there is one solution
X0 = pinv(A)y
then all solutions are
X =pinv(A)y + [I−pinv(A)A]z (10.42)
where z is an arbitrary vector.

For invertible A H A holds pinv(A)A = I and the solution
B C −1
X =pinv(A)y = A H A AH y
is unique.
For invertible AA H we have an indeterminate system. All solutions
can be written in form (10.42) with arbitrary z. It can be easily shown that,
in this case, by using the norm-two ( ℓ2 -norm) minimization
min ∥X∥2 subject to AX = y

B C −1
the solution is X =pinv(A)y = A H AA H y with z = 0.
B C −1
For the DFT analysis, the signal corresponding to X = A H AA H y,
would be
B C −1 - . B C −1
H H A
x R = ΨX = ΨA AA y= A H AA H y
Ac
⎡ B C −1 ⎤
H H - .
⎢ AA AA y ⎥ y
=⎣ B C −1 ⎦ = , (10.43)
0
Ac A H AA H y
- A is.the measurement matrix and Ac is its complement to Ψ so that

where
A
Ψ= . This is the result expected from Parseval’s theorem.
Ac
10.4.3 Estimation of Unknown Positions
Solution of the minimization problem, assuming that the positions of the

nonzero signal coefficients in the sparse domain are known, is presented
in the previous two subsections. The next step is to estimate the coefficient
positions, using the available samples. A simple way is to try to estimate the
positions based on signal samples that are available, ignoring unavailable
samples. This kind of transform estimate is
X̂ (k ) = ∑ x ( n ) ϕ k ( n ), (10.44)
n∈M
where for the DFT ϕk (n) = exp(− j2πnk/N ) and n ∈ M = {n1 , n2 , ..., n M }.
Since ϕk (n) = Nψk∗ (n) this relation can be written as (10.30)
X̂ = NA H y
where A is the measurement matrix. With K ≪ M ≪ N the coefficients

X̂ (k ), calculated with M samples, are random variables. Note that using
(10.44) in calculation is the same as assuming that the values of unavailable
samples x (n), n ∈ / M, is zero. This kind of calculation corresponds to the
result (10.43) that would be achieved for the signal transform if ℓ2 -norm is
used in minimization.
Algorithm
A simple and computationally efficient algorithm, for signal recovery,
can now be implemented as follows:
(i) Calculate the initial transform estimate X̂ (k ) by using the avail-
able/remaining signal values
X̂ (k ) = ∑ x (n) ϕk (n) (10.45)

n∈M
H
or X̂= NA y.
(ii) Set the transform values X (k ) to zero at all positions k except the
highest ones. Alternative:
(ii) Set the transform values X (k ) to zero at all positions k where this
initial estimate X̂ (k ) is below a threshold Tr ,
X (k ) = 0 for k ̸= k i , i = 1, 2, ..., K̂
' '
k i = arg{' X̂ (k )' > Tr }.
This criterion is not sensitive to Tr as far as all nonzero positions of the

original transform are detected ( X̂ (k ) is above the threshold) and the total
number K̂ of transform values in X̂ (k ) above the threshold is lower than the
number of available samples, i.e., K ≤ K̂ ≤ M.
All K̂ − K transform values that are zero in the original signal will be
found as zero-valued.
(iii) The unknown nonzero (including K̂ − K zero valued) transform
coefficients could be then easily calculated by solving the set of M equations
for available instants n ∈ M, at the detected nonzero candidate positions k i ,
i = 1, 2, ..., K̂,
K̂
∑ X (k i )ψki (n) = x (n), for n ∈ M.
i =1
This system of the form
AK XK = y
is now reduced to the problem with known positions of non zero coefficients
(considered in the previous section). It is solved in the least square sense as
(10.41)
B C −1
XK = AKH AK AKH y. (10.46)
The reconstructed coefficients X (k i ), i = 1, 2, ..., K̂, (denoted by vector XK ) are

exact, for all frequencies. If some transform coefficients, whose true value
should be zero, are included (when K < K̂) the resulting system will produce
their correct (zero) values.
Comments: In general, a simple strategy can be used by assuming that
K̂ = M and by setting to zero value only the smallest N − M transform
coefficients in X̂ (k ). System (10.36) is then a system of M linear equations
with K̂ = M unknown transform values X (k i ). If the algorithm fails to detect
a component the procedure can be repeated after the detected components
are reconstructed and removed. This simple strategy is very efficient and
if there is no input noise. Large K̂, close or equal to M, will increase the
probability that full signal recovery is achieved in one step. It will be shown
later that in the case of an additive (even small) input noise in all signal
samples a reduction of the number K̂ as close to the true signal sparsity K as
possible will improve the signal-to-noise ratio.
Example 10.7. Consider a discrete signal
x (n) = 1.2e j2πn/16+ jπ/4 + 1.5e j14πn/16− jπ/3 + 1.7e j12πn/16 ,
for 0 ≤ n ≤ 15, sparse in the DFT domain since only three DFT values are
different than zero. Assume now that its samples x (2), x (4), x (11), and x (14)
are not available. Show that, in this case, the exact DFT reconstruction may
be achieved by: (1) Calculating the initial DFT estimate by setting unavailable
sample values to zero
X̂ (k) = ∑ x(n)e j2πkn/16 =16A H y,

n∈M
where
n ∈ M = {0, 1, 3, 5, 6, 7, 8, 9, 10, 12, 13, 15}.
(2) Detecting, for example K = 3 positions of maximal DFT values, k1 , k2 , and
k3 , and (3) calculating the reconstructed DFT values at k1 , k2 , and k3 from
system
3
∑ X (ki )e j2πk n/16 = x(n),
i
i =1
where n ∈ M = {0, 1, 3, 5, 6, 7, 8, 9, 10, 12, 13, 15} are the instants where the
signal is available.
⋆The discrete-time signal x (n), with 0 ≤ n ≤ 15 is shown in Fig. 10.7.
The signal is sparse in the DFT domain since only three DFT values are
different than zero (Fig. 10.7(second row)). The CS signal, with missing
samples x (2), x (4), x (11), and x (14), being set to 0 for the initial DFT
estimation, is shown in Fig. 10.7 (third row). The DFT of the signal, with
missing values being set to 0, is calculated and presented in Fig. 10.7 (fourth
row). There are three DFT values, at k1 = 1, k2 = 6, and k3 = 7
K = {1, 6, 7}
above the assumed threshold, for example, at level of 11. The rest of the DFT
values is set to 0. This is justified by using the assumption that the signal is
sparse. Now, we form a set of equations, for these frequencies k1 = 1, k2 = 6,
and k3 = 7 as
3
∑ X (ki )e j2πk n/16 = x(n),
i
i =1
where n ∈ M = {0, 1, 3, 5, 6, 7, 8, 9, 10, 12, 13, 15} are the instants where the
signal is available. Since there are more equations than unknowns, the system
P Q −1 H
AK XK = y is solved using XK = AKH AK AK y. The obtained reconstructed
values are exact, for all frequencies k, as in Fig. 10.7(second row). They are
shown in Fig. 10.7 (fifth row).
If the threshold was lower, for example at 7, then six DFT values at
positions
K = {1, 6, 7, 12, 14, 15}
are above the assumed threshold. The system with six unknowns
6
∑ X (ki )e j2πk n/16 = x(n),
i
i =1
where n ∈ M = {0, 1, 3, 5, 6, 7, 8, 9, 10, 12, 13, 15} will produce the same values
for X (1), X (6), and X (7) while the values X (12) = X (14) = X (15) = 0 will be
obtained.
If the threshold is high to include the strongest signal component only,
then the solution is obtained through an iterative procedure described later,
after noise analysis.
10.4.4 Unavailable/Missing Samples Noise in Initial Estimation
The initial DFT calculation (10.44) is done assuming zero-valued missing

samples. The initial calculation quality has a crucial importance for success-
ful signal recovery. With a large number of randomly positioned missing
4
Original signal
2
0
-2
-4
0 5 10 15
30
DFT of original signal
20
10
0
0 5 10 15
4
Signal with 4 missing samples
2
0
-2
-4
0 5 10 15
30
DFT of signal with
20 4 missing samples set to 0
threshold for reconstruction
10
0
0 5 10 15
30
Reconstructed DFT
20 on detected frequencies
10
0
0 5 10 15
Figure 10.7 Original signal in the discrete-time domain (first row); the DFT of the original
signal (second row); signal with four missing samples at n = 2, 4, 11, and 14 set to zero (third
row); the DFT of signal with missing values being set to 0 (fourth row). The reconstructed
signal assuming that the DFT contains components only at frequencies where the initial DFT is
above threshold (fifth row). Absolute values of the DFT and real part of signal are shown.
samples, the missing samples manifest themselves as a noise in this initial

transform. For a sparse signal of the form
K
x (n) = ∑ A p e j2πnk p /N ,
p =1
the initial DFT is calculated using n ∈ M = {n1 , n2 , ..., n M }, as NA H y, or
K
X (k ) = ∑ x (n)e− j2πnk/N = ∑ ∑ A p e− j2πn(k−k p )/N . (10.47)
n∈M n ∈ M p =1
We can distinguish two cases: (1) For k = k i ∈ {k1 , k2 , ..., k K } then, with
M = card(M),
K
X ( k i ) = Ai M + ∑ ∑ A p e− j2πn(ki −k p )/N .
n∈M p=1,p̸=i
The value of
K
Ξ= ∑ ∑ A p e− j2πn(ki −k p )/N (10.48)
n∈M p=1,p̸=i
with a random set M = {n1 , n2 , ..., n M }, for 1 ≪ M ≪ N, can be considered

as a random variable. Its mean value over different realizations of available
samples (different realizations of sets M) is E{Ξ} = 0. The mean value of
X (k i ) is
E{ X (k i )} = Ai M.
/ {k1 , k2 , ..., k K } the mean value of (10.47) is
(2) For k ∈
E{ X (k )} = 0.
The mean value of (10.47) for any k is of the form
K
E{ X (k )} = M ∑ A p δ ( k − k p ).
p =1
The variance of signal transform is

K
2 N−MD E
σN (k) = var( X (k)) = ∑ A2p M N−1
1 − δ(k − k p ) . (10.49)
p =1
This relation will be derived next. To simplify notation the variance of X (k )

will be calculated for K = 1 with k ̸= k1 . The variance is defined by
% ;
var{ X (k )} = E ∑ ∑ | A1 |2 e− j2πm(k−k1 )/N e j2πn(k−k1 )/N (10.50)
n∈M m∈M
% ;
2 2 − j2πm(k−k1 )/N j2πn(k−k1 )/N
= ∑ E | A1 | + ∑ | A1 | e e
m∈M n∈M,n̸=m
Obviously, % ;
2
E ∑ | A1 | = | A1 |2 M.
n∈M
Full set of signal samples would produce the DFT of original signal. It
means that all variables e j2πn(k−k1 )/N are not statistically independent for
(k − k1 ) ̸= 0. They satisfy
N −1
e− j2πm(k−k1 )/N ∑ e j2πn(k−k1 )/N = 0
n =0
since the sum over all discrete-time instants is deterministic and X (k ) = 0

for k ̸= k1 . Its expectation is
N −1
∑ E{e− j2πm(k−k1 )/N e j2πn(k−k1 )/N } = 0 (10.51)
n =0
Since all values e j2πn(k−k1 )/N (with random n) are equally distributed we
may write their expected value over many realizations of different sets M
as
E{e− j2πm(k−k1 )/N e j2πn(k−k1 )/N } = B, for n ̸= m, (10.52)

− j2πn(k−k1 )/N j2πn(k−k1 )/N
E{ e e } = 1 for n = m.
From (10.51) and (10.52) follows
( N − 1) B + 1 = 0.
Now we can easily calculate terms in (10.50)

% ;
2 − j2πm(k−k1 )/N j2πn(k−k1 )/N
E ∑ | A1 | e e
n∈M,n̸=m
* +
1
= | A1 |2 ( M − 1) B = | A1 |2 ( M − 1) − .
N−1
Finally, the variance of X (k ), for k ̸= k1 , is

* * ++
1 N−M
2
σN (k) = var( X (k)) = | A1 |2 M 1 + ( M − 1) − = | A1 |2 M .
N−1 N−1
Of course, for k = k1 we get

2
σN (k1 ) = 0
since all terms in X (k ) are summed in phase with no random variation. The
ratio of signal amplitude X (k1 ) and standard deviation σN (k ) for k ̸= k1 is
the crucial parameter for a correct signal detection. Its value is
<
σN (k ) N−M
= .
| X (k1 )| M ( N − 1)
√
For small M, when ( N − M )/( N − 1) ∼ 1 we have σN (k )/ | X (k1 )| = M.
For M ≪ N a rough approximation var( X (k )) = | A1 |2 M follows. It corre-
sponds to the assumption of statistically independent variables.
Note that the variance in a multicomponent signal with K > 1 is sum
of the variance of individual components at all frequencies k
K ' '2
2 N−M
σN (k ) = M
N−1 ∑ 'Ap' , (10.53)
p =1
except at k i ∈ {k1 , k2 , ..., k K } when the values are lower for | Ai |2 M NN−−M
1
K ' '2
2 N−M
σN (k i ) = M
N−1 ∑ 'Ap' ,
p =1
p ̸ =i
since all ith component values are then added up in phase at k = k i , without
random variations.
According to the central limit theorem, for 1 ≪ M ≪ N the real and
imaginary parts of the DFT values for noise only positions k ∈ / {k1 , k2 , ..., k K }
can be described by Gaussian distribution, N (0, σN 2 /2) with zero-mean and
variance σN 2 = σ2 ( k ). Real and imaginary parts of the DFT value, at the

N
pth signal component position k p ∈ {k1 , k2 , ..., k K }, can be described by the
Gaussian distributions
N ( M Re{ A p }, σS2p /2), and

N ( M Im{ A p }, σS2p /2), (10.54)
respectively, where
N−M
2
σS2p = σN − A2p M ,
N−1
according to (10.49).
Example 10.8. Consider a three-component signal
x (t) = A1 exp( j2πk1 t/N ) + A2 exp( j2πk2 t/N ) + A3 exp( j2πk3 t/N ) (10.55)
with A1 = 1, A2 = 0.75, A3 = 0.25, {k1 , k2 , ..., k K } = {58, 117, 21}, within 0 ≤ t ≤

256. With t = n∆t, ∆t = 1 and N = 257 the signal is sparse in the DFT domain.
Random realizations of the initial DFT (10.45) are given in Fig.10.8, for several
values of the number of available samples M. We can see that a low value of
M does not provide possibility to detect the signal component positions. All
three components are visible for larger values of M. When signal frequencies
are detected then the signal is recovered using (10.41) and the available
samples in the discrete-time domain at ni ∈ {n1 , n2 , ..., n M }, with detected
frequencies {k1 , k2 , ..., k K }. Obviously from a noisy observation of the DFT
16 64 128
M=16 M=64 M=128
12 48 96
8 32 64
4 16 32
0 0 0
1 128 257 1 128 257 1 128 257
signal transform
192 224 256
M=192 M=224 M=257
144 168 192
96 112 128
48 56 64
0 0 0
1 128 257 1 128 257 1 128 257
frequency
Figure 10.8 Initial DFT of a signal with various number of available samples M. Available M
samples are a random subset of N samples taken according to the sampling theorem interval.
Dots represent the original signal DFT values, scaled with M/N to match the mean value of
the DFT calculated using a reduced set of signal samples. The DFT values are presented as a
function of the frequency index.
we can distinguish two cases: 1) When the number of available samples is

large and all components are above a threshold that can be calculated based
on (10.49). Then all signal frequencies will be distinguishable as peaks in the
DFT. 2) If the number of available samples is low or there are components
with much lower amplitudes then the largest component is detected and
estimated first. It is subtracted from the signal. The next one is detected and
the signal is estimated using the frequency from this and the previous step(s).
The estimated two components are subtracted from the original signal. The
frequency of next components is detected, and the process with estimation
and subtraction is continued until the energy is negligible. This iterative
procedure will be the topic of next subsection.
Example 10.9. For a discrete-time signal

1 1
x (n) = e j2πk1 n/N + e j2πk2 n/N + e j2πk3 n/N ,
2 4
with N = 64 the DFT is calculated using a random set of M = 16 samples.
Calculation is performed with 105 random realizations with randomly po-
sitioned M samples and random values of k1 , k2 , and k3 . Histogram of the
DFT values, at a noise only position k ∈ / {k1 , k2 , k3 } and at the signal compo-
nent k = k1 position, is presented in Fig.10.9 (left). Histogram of the initial
DFT real part is shown, along with the corresponding Gaussian functions
N−M 5 N−M
N (0, 21
16 N −1 ) and N ( M, 16 N −1 ), shown by dots confirming (10.54). The
same calculation is repeated with M = 64, Fig.10.9(right). Note that the vari-
ance factor 21/16 = 1 + (1/2)2 + (1/4)2 follows from (10.53).
We can see that the mean value of the Gaussian variable X (k ) can
be used for the signal component position detection. Also the variance is
different for noise only and the signal component positions. It can also be
used for the signal position detection.
In the case with M = 16 the histograms are close to each other, meaning
that there is a small probability that a signal component is missdetected.
Histograms are well separated in the case with M = 64. It means that the
signal components will be detected with an extremely high probability in
this case. Calculation of the detection probability is straightforward with the
assumed probability density functions.
The spark based relation can be obtained within the framework of

previous analysis if we assume that the noises in (10.48) due to missing
samples, coming from different components of the same (unity) amplitude
Ai , are added up with the same phase to produce
K
X (k ) = ∑ ∑ e− j2πn(k−k p )/N = K ∑ e− j2πn(k−k p )/N
n ∈ M p =1 n∈M
at some frequency k ∈/ {k1 , k2 , ..., k K }. Random variable ∑n∈M e− j2πn(k−k p )/N

(since n ∈ M is random) should also assume its maximal possible value
(calculated over all possible k p and all possible positions k, k ̸= k p ). The
maximal possible value of this variable is related to the coherence index
Distribution of noise only DFT, ℜ{X(k)} Distribution of noise only DFT, ℜ{X(k)}
0.2
p (ξ) M=16 pℜ{X(k)}(ξ) M=64
ℜ{X(k)}
0.2 0.15
0.1
0.1
0.05
0 0
-5 0 5 10 15 20 -20 0 20 40 60 80
Distribution of signal DFT, ℜ{X(k1)} Distribution of signal DFT, ℜ{X(k1)}
0.2
p (ξ) M=16 p (ξ) M=64
ℜ{X(k )} ℜ{X(k )}
1 1
0.2 0.15
0.1
0.1
0.05
0 0
-5 0 5 10 15 20 -20 0 20 40 60 80
Figure 10.9 Histograms and Gaussian probability density functions for the signal and noise
only positions in the initial DFT for a three-component signal with N = 128 and M = 16 (left)
and M = 64 (right). The histograms are calculated in 10 5 random realizations of M available
samples and random signal frequency positions.
of the partial DFT matrix as (10.32)

' '
' ' 1 ' '
' − j2πn ( k − k ) /N '
µ = max 'µ(k, k p )' = max ' ∑ e p
'.
M k,k p 'n∈M '
It means that maximal possible value of this variable is µM. It should

also be assumed that (K − 1) remaining noise components (due to missing
samples) at the component position k = k p assume the same maximal value
µM and that all of them are subtracted in phase from the signal mean
value M at k = k p . Condition for the correct detection of a component
position at k = k p is then such that the minimal possible amplitude of the
component M − Mµ(K − 1) is greater than the maximal possible noise MµK
at k ∈
/ {k1 , k2 , ..., k K }, i.e.,
M − Mµ(K − 1) > MµK

or
1
K < (1 + 1/µ).
2
According to several very unlikely assumptions that have been made, we
can state that this is a very pessimistic bound for K. Therefore, for a high
degree of randomness, a probabilistic approach may be more suitable for
the analysis than the spark based relation.
This kind analysis will be repeated on the case of Gaussian real-valued
random matrix. In this case there is no complete set of measurements. This
analysis then can be considered as a reduced set of measurements analysis.
In this case B C −1
X = AT A AT y
can be again considered as

B C −1
X = AT A X0
where
X0 = A T y
is the initial estimation. It uses available reduced set of M measurements y
to calculate N values of X0 . Its value is the same as if a complete transfor-
mation matrix existed and all values of the missing measurements (to com-
plete set of measurements) were considered as zero. If the initial estimation
X0 = A T y can produce correct positions of nonzero values in a K-sparse X
then the solution will be straightforward using only nonzero values of X
denoted by XK and corresponding measurements submatrix AK as
B C −1
XK = AKT AK AKT y.
Assume that the measured signal is
K K
x (n) = ∑ X (k i )ψki (n) = ∑ Ai ψki (n)
i =1 i =1
with the elements of y being x (n) for n ∈ M and k i ∈ {k1 , k2 , ..., k K }. Then the
elements of X0 = A T y are
K
X0 ( k ) = ∑ A i
i =1
∑ ψk (n)ψki (n).
n∈M
Obviously
E { X0 (k )} = 0 for k ̸= k i
E { X0 (k )} = Ai for k = k i
R S
since E ∑n∈M ψk2 (n) = 1. For k ̸= k i
∑ ψk (n)ψki (n) ≤ µ
n∈M
by definition, where µ is the coherence.

In the worst case, assuming ∑n∈M ψk2 (n) = 1 and Ai = 1 for all k i , the
smallest possible value at k = k i would be obtained if all (K − 1) components
assume the lowest possible value −µ
X0 ( k i ) = 1 − ( K − 1 ) µ
The signal components should assume this lowest possible case and it
should be greater than the highest possible value at a k ̸= k i
X0 (k ) = Kµ.
It should hold
1 − (K − 1)µ > Kµ
* +
1 1
K< 1+ .
2 µ
We can easily see why the coherence index based limit in Example 10.6,
produced very conservative estimate. It calculates the sparsity limit as-
suming that an order of K Gaussian variables ∑n∈M ψk (n)ψki (n) assume,
at the same time, maximal upper limit and that (K − 1) variables assume
at the same time lower limit −µ. The eigenvalue based calculation does
not make such an assumption. Therefore it is closer to the expected be-
havior, although it also assumes a specific, the worst case, signal form.
(Note: Show that any other A1 ≥ A2 ≥ ... ≥ AK ≥ 0 will produce more re-
laxed condition than when all amplitudes are equal A1 − µ( A2 + ... + AK ) >
µ( A1 + A2 + ... + AK )).
A realistic and very simplified probabilistic approach would be based
on:
(1) Variance of K random variables ∑n∈M ψk (n)ψki (n) corresponding
to signal components k i in the worst case is K/M.
(2) Variance of N − K random variables ∑n∈M ψk (n)ψki (n) for the

positions k i not corresponding to signal components is (K − 1)/M.
(3) Distance between the mean values of signal components and noise
only components is 1.
(4) Probability density function of these two
√ classes of random vari-
ables should be well separated. For example, if K/M ≤ 1/6 then
= =
1 K
3 ( K − 1) + 3 <1
M M
with 3-sigma rule Gaussian distributions guarantees probabilities of 1 −

10−4 order that these distributions will not overlap (any of N − K signal
free samples with at least one of K signal samples). It means
M
K≤
36
For M = 1024 we get
K ≤ 28
or 2K < 28 for the unique solution, corresponding to Fig.10.5.
10.4.5 Iterative Procedure
If components with very different amplitudes exist and the number of

available samples is not large, then the iterative procedure should be used.
This procedure could be implemented as follows. The largest component is
detected and estimated first. It is subtracted from the signal. The next one
is detected and the signal is estimated using the frequency from this and
the previous step(s). The estimated two components are subtracted from
the original signal. The frequency of next components is detected, and the
process of estimations and subtractions is continued until the energy of the
remaining signal is negligible or bellow an expected additive noise level.
Algorithm
(i) Calculate the initial transform estimate X̂1 (k ) by using the avail-
able/remaining signal values x1 (n) = x (n)
X̂1 (k ) = ∑ x (n) ϕk (n)

n∈M
Set the transform values X̂ (k ) to zero at all positions k except the

highest one at k = k1 ,
K̂1 = {k1 }. Set the counter to r = 1.

Form the matrix A1 using the available samples in time n ∈ N A and
detected index k ∈ K̂1 , with one nonzero component. Calculate the estimate
of the transformation coefficient at k = k1
B C −1
X̂1 = A1H A1 A1H y.
Calculate the signal estimation (as the inverse DFT)
x̂1 (n) = X̂1 (k1 )ψk1 (n), for n ∈ M
and check
2
∑n∈M | x (n) − x̂1 (n)|
ϵ= 2
.
∑n∈M | x (n)|
If, for example ϵ < 10−5 , stop the calculation and use x (n) = x̂1 (n). If not
then go to the next step.
(ii) Set the counter to r = r + 1. Form a signal
er (n) = x (n) − x̂r−1 (n),
at the available sample positions and calculate the transform
Êr (k ) = ∑ er ( n ) ϕ k ( n ).
n∈M
Set the transform values Êr (k ) to zero at all positions k except the highest
one at k = k r . Form the set of r indices, using union of the previous maxima
positions and the detected position, as
K̂r = {K̂r−1 , kr }.
Form matrix Ar using the available samples in time n ∈ M and detected K̂r
indices k ∈ K̂r . Calculate the estimate of Kr transformation coefficients
B C −1
X̂Kr = ArH Ar ArH y.
Calculate the signal
x̂r (n) = ∑iK̂=r 1 X̂r (k i )ψki (n), for n ∈ M

and check
2
∑n∈M | x (n) − x̂r (n)|
ϵ= 2
.
∑n∈M | x (n)|
If, for example ϵ < 10−5 , stop the calculation and use
x (n) = x̂r (n).
Else repeat step (ii).

Example 10.10. Signal
n π n π
x (n) = sin(12π + ) + 0.7 cos(40π + ) − 0.4
N 4 N 3
with N = 64 is shown in Fig.10.10. Small number of samples is available
M = 16 with different signal amplitudes, making one-step recovery impos-
sible. The available signal samples y(n) are shown in Fig.10.10(second row,
left). The iterative procedure is used and for the detected DFT positions dur-
ing the iterations the recovered signal is calculated according to the presented
algorithm. The recovered DFT values in the rth iteration are denoted as Xr (k )
and presented in Fig.10.10. After first iteration the strongest component is
detected and its amplitude estimated. At this stage other components be-
have as noise (this will be analyzed later in this section) and make amplitude
value inaccurate. Accuracy improves as the number of detected components
increases in next iterations. After five steps the agreement between the re-
constructed signal and the available signal samples was complete. Then the
algorithm is stopped. The DFT of the recovered signal is presented as X5 (k )
in the last subplot of Fig.10.10. Its agreement with the DFT of the original
signal, Fig.10.10 (first row, right) is complete.
10.4.6 Influence of Additive Input Noise
Assume now that an input additive noise ε(n) exists in the available signal
samples. Note that the noise due to missing samples influences the results
in the sense of the possibility to recover the signal. When the recovery is
achieved the result accuracy is related to the input additive noise in signal
samples and the number of available samples as it will be shown next.
The reconstruction equations (10.36) for noisy samples are
K
x (n) + ε(n) = ∑ X (k i )ψki (n), for n ∈ M.
i =1
for the detected indices k = {k1 , k2 , ..., k K }. Matrix form of these equations is
y+ε = AK XK .
50
2 x(n) X(k)
40
1
30
0
20
-1
10
-2
0
0 20 40 60 -20 0 20
50
2 y(n) X1(k)
40
1
30
0
20
-1
10
-2
0
0 20 40 60 -20 0 20
50 50
X (k) X (k)
40 2 40 3
30 30
20 20
10 10
0 0
-20 0 20 -20 0 20
50 50
X4(k) X5(k)
40 40
30 30
20 20
10 10
0 0
-20 0 20 -20 0 20
Figure 10.10 Iterative signal recovery
It is a system of M linear equations with K unknowns. The solution is

obtained from
AKH (y+ε) = AKH AK XK

B C −1
XK = AKH AK AKH (y+ε)
XK = XKS + XKN . (10.56)
Here the true signal transform coefficients are

B C −1
XKS = AKH AK AKH y,
and B C −1
XKN = AKH AK AKH ε
is the noise influence to the reconstructed signal coefficients.

In terms of matrix norms the bound for ratio of noise in the recon-
structed and the original signal follows from
WB W
W H C −1 W W W
W
∥XKN ∥2 ≤ W AK AK W W HW
W WA K W ∥ ε ∥2
2 2
√
∥XKN ∥2 dmax 1 ,
≤ ≤ 1 + δK .
∥ ε ∥2 dmin 1 − δK
WP Q −1 W
W W
The fact that W AKH AK W ≤ 1/dmin ≤ 1/(1 − δK ) is used, where dmin
2 W W W W
is the smallest eigenvalue of WAKH AK W2 . The norm of WAKH W2 is equal to
FW W W W √ √
WA H AK W , meaning WA H W ≤ dmax ≤ 1 + δK , where dmax is the largest
K 2 W K 2
W
eigenvalue of WAKH AK W2 .
For a small noise, a simplified analysis can be performed. If all signal
samples were available, the input signal-to-noise (SNR) ratio, would be
2
∑nN=−01 | x (n)| Ex
SNRi = 10 log 2
= 10 log .
∑nN=−01 |ε(n)| Eε
Assume that the noise energy in the available samples is
EεA = ∑ |ε(n)|2 . (10.57)

n∈M
The true amplitude in the signal transform at the index k p , in the case if
all signal samples were used, would be N A p , where A p is the amplitude
of the signal component corresponding to the index k p . To compensate
the resulting transform for the known bias in amplitude when only M
available samples are used the coefficient should be multiplied by N/M.
In a full recovery, a signal transform coefficient is equal to the coefficient
of the original signal with all signal samples being used. The noise in the
transform coefficients are multiplied by the same factor. The energy of noise
2 N/M2 . The SNR in the

in the reconstruction algorithm is increased to EεA
recovered signal is
2
∑nN=−01 | x (n)|
SNR = 10 log . (10.58)
N2 2
M2 ∑n∈M |ε(n)|
Since only K out of N coefficients are used in the reconstruction the

energy of the reconstruction error is reduced for the factor of K/N as well.
The energy of noise in the recovered signal is
K N2
|ε(n)|2 .
N M2 n∑
EεR =
∈M
The SNR in the recovered signal is
2
∑nN=−01 | x (n)|
SNR = 10 log 2
. (10.59)
K̂N
M2 ∑n∈M |ε(n)|
Since the variances in all samples and the available samples are the same
then
1 1 N −1
∑ |ε(n)|2 = |ε(n)|2
N n∑
(10.60)
M n∈M =0
Thus, the SNR in the recovered signal is
* +
K
SNR = SNRi − 10 log . (10.61)
M
Using the number K in reconstruction as small as possible (in ideal case

equal to the signal sparsity) improves the results when a small additional
input noise exists in all signal samples
Example 10.11. This simple theoretical result is tested on the DFT and signal
x (n) = A1 exp( j2πk1 n/N ) + A2 exp( j2πk2 n/N ) + A3 exp( j2πk3 n/N )
(10.62)
with A1 = 1, A2 = 0.75, A3 = 0.25, {k1 , k2 , k3 } = {58, 117, 21}, within 0 ≤ n ≤
N − 1 = 255, with additive noise ε(n) of variance σε2 = 1. For a random set
of M available samples the initial DFT is calculated using (10.45). Since a
large number of available samples M is used in these simulations the signal
components {k1 , k2 , k3 } are easily detected in one step. The signal is recovered
by (10.46) for the set of available signal samples y = [ x (n1 ) x (n2 ) ... x (n M )] T
and the detected frequencies are {k1 , k2 , k3 }. The input SNR was SNRi =
2.6383 [dB]. The theoretical result for the output SNR, for example, M =
N/2 = 128 and K = 3, according to (10.61) is SNR = 18.88 [dB]. For statistical
check of the results, 100 random realizations of the available sample positions
are used. The statistical SNR was obtained as SNR = 18.87 [dB]. Its agreement
with the theory is high.
10.4.7 Nonsparse Signal Reconstruction
According to the results in Section 10.4.4 the missing samples can be repre-
sented by a noise influence. Assume that we use a reconstruction algorithm
for a signal of sparsity K on a signal whose DFT coefficients X are not sparse
(or not sufficiently sparse). Denote by XK the sparse signal with K nonzero
coefficients equal to the largest K coefficients of X. Suppose that the number
of components K and the measurements matrix satisfy the reconstruction
conditions so that a reconstruction algorithm can detect (one by one or at
once) largest K components (A1 , A2 ,...,AK ) and perform signal reconstruc-
tion to get X R . The remaining N − K components (AK +1 ,AK +2 ,...,A N ) will
be treated as a noise in these K largest components. Variance from a signal
component is | Ai |2 M ( N − M )/( N − 1). After reconstruction this variance
is multiplied by ( N/M )2 , according to the analysis in previous subsection,
producing
N 2 M( N − M) ∼ N−M
| A i |2 2 2
= | Ai | N .
M N−1 M
The total energy of noise in the reconstructed K largest components X R will
be
N−M N
∥X R −XK ∥22 = KN | A i |2
M i=∑ K +1
Denoting the energy of remaining signal, when the K largest are removed
from the original signal, by
N
∥X − XK ∥22 = N ∑ | A i |2
i = K +1
we get
N−M
∥X R −XK ∥22 = K ∥X − XK ∥22 .
M
If the signal is sparse, i.e., X = XK , then
∥X R −XK ∥22 = 0.
The same result follows if N = M. The error will be zero if a complete DFT
matrix is used in the calculation of any signal component.
Finally using Schwartz’s inequality for X − XK having N − K nonzero
elements,
1
∥ X − X K ∥2 ≤ √ ∥ X − X K ∥1 ,
N−K
follows =
N−M K
∥ X K − X R ∥2 ≤ ∥ X − X K ∥1 .
M N−K
In the case of additive input noise with variance σε2 , a general expres-
sion is obtained in the form
N−M K
∥X R −XK ∥22 = K ∥X − XK ∥22 + Nσε2 .
M M
Example 10.12. Consider a nonsparse signal
x (n) = e j2πk1 n/N + 0.8e j2πk2 n/N + 0.77e j2πk3 n/N + 0.75e j2πk4 n/N
255 * +[1+(i −5)/50]
1
+∑ e j2πki n/N
i =5
3
where k i , i = 1, 2, ..., 255 are random frequency indices from 0 to N − 1. Using

N = 257 and M = 192 the first K = 4 components of signal are reconstructed.
The remaining 251 signal components are considered as disturbance. Recon-
struction of K = 4 largest components is done in 100 independent realizations
with different frequencies and positions of available samples. The result for
noise-free case is
( )
∥XK ∥22
SNRstat = 10 log = 23.1476
∥X R −XK ∥22
( )
∥XK ∥22
SNRtheor = 10 log 2
= 23.1235.
K N− M
M ∥ X − X K ∥2
Note that a closed form expression for ∥X − XK ∥22 in SNRtheor can be obtained
since we assumed that the amplitudes of disturbing components are coeffi-
cients of a geometric series. One realization is presented in Fig.10.11.
In the case of additive complex-valued noise of variance σε2 = 2 the
results are
( )
∥XK ∥22
SNRstat = 10 log = 17.0593
∥X R −XK ∥22
( )
∥XK ∥22
SNRtheor = 10 log 2
= 17.0384.
K N− M K
M ∥ X − XK ∥2 + M Nσε
2
350
300
250
200
150
100
50
0
50 100 150 200 250
Figure 10.11 Single realization reconstruction of K = 4 largest signal components of a non-

sparse noisy signal.
A decrease in the SNR due to noise is

( 2
)
K N− M
M ∥ X − X K ∥2
∆SNRtheor = 10 log 2
= −6.0851.
K N− M K 2
M ∥ X − X K ∥2 + M Nσε
The simulation is repeated with M = 128 and the same noise. The SNR values
are SNRtheor = 14.3345 and SNRstat = 14.4980.
10.5 NORM-ONE BASED RECONSTRUCTION
A direct way to solve the ℓ0 -norm based minimization problem
M0 = ∥X∥0 = card {X} = K,
subject to y = AX is a search over all possible combinations of nonzero

samples, starting from the lowest K = 1, until a smallest set of K nonzero
coefficients satisfying y = AK XK is found. In general, this is not computa-
tionally feasible problem. Indirect methods, based on an initial estimation
of the nonzero positions, are presented in the previous section.
Note that the ℓ0 -norm sparsity measure can be considered as a limit
case of the concentration measure
N −1
Mp = ∑ | X (k)| p = ∥ X (k)∥ pp ,
k =0
for p → 0. We can expect that the behavior of this measure will not signif-
icantly change if p is slightly increased from 0. This kind of concentration
measure with 0 ≤ p ≤ 1 has been used for decades in optimization of time-
frequency representations, as an alternative to measure based on the ratio
of higher order norms.
In compressive sensing the most commonly used sparsity measure
is the norm with p = 1 since it is the only convex function for p within
the interval 0 ≤ p ≤ 1. Convex form of the measure enables application of
linear programming in the solution of the minimization problem. Thus the
minimization problem formulation with p = 1 is
min ∥X∥1 subject to y = AX,
where
N −1
M1 = ∥ X ∥1 = ∑ | X (k)| .
k =0
Under some conditions minimization of the ℓ1 -norm sparsity measure can

produce the same result as the ℓ0 -norm minimization. These conditions will
be considered in this section.
Note that norms with p > 1 cannot be used. For p = 2 this measure
is equivalent with the well-known ℓ2 -norm used in definitions of standard
signal transforms. In the standard signal transforms the measure with ℓ2 -
norm has a minimum when the missing signal samples/measurements are
set to zero. Parserval’s theorem states that the energy of a signal in the time
domain is the same as the energy of the Fourier transform in the frequency
domain. A signal has the lowest energy when its missing samples are zero-
valued. Associating any nonzero value to the missing samples will increase
the signal energy. The same holds in the frequency domain since the energy
in the frequency domain equals to the energy in the time domain. The
minimization solution with the ℓ2 -norm is therefore trivial. With this norm,
we attempt to minimize
N −1
∥X∥22 = ∑ | X (k)|2 .
k =0
According to Parseval’s theorem we have ∥X∥22 = N ∑nN=−01 | x (n)|2 . Since

any value other than x (n) = 0 for the unavailable/missing signal samples,
would increase ∥X∥22 , then the solution for the non-available samples, with
respect to the ℓ2 -norm, are all zero values (a proof of this fact has been
presented within the section dealing with pseudoinverse matrix as well).
Resulting transform X (k ) is then not sparse. It was the reason why this norm
was not used as a concentration measure as well.
Example 10.13. Minimization in a space with two variables x, y will be illustrated
on the cases with p = 1, p = 1/2, p = 1/4 and p = 2 using the condition
y = ax + ḃ. Note that in the case of p = 1 the result of function z = | x | + |y|
minimization subject to y = ax + ḃ is a point with minimal value of z =
| x | + |y| on the line where the surface z = | x | + |y| intersects with the plane
y = ax + ḃ (the plane y = ax + ḃ in x, y, z space is z independent). Constant
values of | x | + |y| are presented by isolines on the first subplot of Fig.10.12.
The minimal value of z is the one where projection of y = ax + ḃ on z = 0
touches the isoline of z = | x | + |y|. All points on isolines crossing this line
correspond to larger values of z = | x | + |y| while all isolines corresponding
to lower values of z = | x | + |y| do not have a common point with the plane
y = ax + ḃ. The minimization of z = | x | + |y| with y = ax + ḃ can also be
written as P ' 'Q
min (| x | + |y|) = min | x | + ' ax + ḃ' .
Since we have a sum of two piecewise linear functions | x | and | ax + b| its
minimum is either at x = 0 or at ax + b = 0 for | a| < 1 or | a| > 1, respectively.
Therefore the function z = | x | + | ax + b| will have a minimum at one of
these two points. For y = 0.5x + 1 the solution is (0, 1) and for y = 3x − 3
the solution is (1, 0), Fig.10.12. The solution is the same for p = 1, p = 1/2
(when z = | x |1/2 + | ax + b|1/2 ), and p = 1/4. For p = 2 the solution follows
as a minimum of z = x2 + ( ax + b)2 . It is (−0.4, 0.8) and (0.9, −0.3) for the
considered functions, respectively. This is just a mathematical illustration of
a constraint minimization. Due to its low dimensionality it cannot be defined
within the measurements and sparsity framework (when for sparsity K = 1
at least two measurements are required).
10.5.1 Illustrations in the Signal Domain
Minimization of the sparsity measures will be illustrated on the signal and

its DFT in recovering missing samples.
Example 10.14. For signals x (n) given by:
(a)
4
x (n) = e j10πn/N + e j14πn/N ,
5
(b)
1 1
x (n) = e j10πn/N + e j14πn/N + e j4πn/N ,
4 5
for 0 ≤ n ≤ N − 1, with N = 64 find the DFT and the number of nonzero
coefficients (sparsity) K in the DFT domain. Calculate the measure values
N −1
Mp = ∑ | X (k)| p
k =0
2 2
1 1
y=0.5x+1 y=0.5x+1
0 0
1/2 1/2
|y|+|x|=z |y| +|x| =z
-1 -1
y=3x-3 y=3x-3
-2 -2
-2 -1 0 1 2 -2 -1 0 1 2
2 2
1 1
y=0.5x+1 y=0.5x+1
0 0
1/4 1/4 2 2
|y| +|x| =z y +x =z
-1 -1
y=3x-3 y=3x-3
-2 -2
-2 -1 0 1 2 -2 -1 0 1 2
Figure 10.12 Illustration of constrained minimization with various norms.
for p = 0, p = 1, and p = 2. Could the measure for p = 1 be used for comparing

sparsity of two different signals?
(c) If the signal sample x (2) in signal
!
A1 e j10πn/N + A2 e j14πn/N , for n ̸= 2, 0 ≤ n ≤ N − 1
x (n) =
z for n = 2
may assume an arbitrary value x (2) = z, find the value of z minimizing

each considered sparsity measure. Comment on this example why M2 = Ex
(energy) based signal measure cannot be used as a signal sparsity measure.
⋆(a) The DFT of signal x (n) is
N −1
4
X (k) = ∑ x (n)e− j2πnk/N = Nδ(k − 5) + Nδ(k − 7).
n =0
5
The sparsity measures for this signal are
N −1
M0 = ∥ X ∥0 = ∑ | X (k)|0 = 2
k =0
N −1
4 9N
M1 = ∥ X ∥1 = ∑ | X (k)|1 = N (1 + ) = .
k =0
5 5
The measure for p = 2 is equal to the signal energy. Its value is
N −1
16 41N 2
M2 = ∑ | X (k)|2 = N 2 (1 + )= .
k =0
25 25
(b) For this signal x (n) the DFT is
N −1
1 1
X (k) = ∑ x (n)e− j2πnk/N = Nδ(k − 5) + Nδ(k − 7) + Nδ(k − 2).
n =0
4 5
The measures are

N −1
M0 = ∥ X ∥0 = ∑ | X (k)|0 = 3
k =0
N −1
1 1 29N
M1 = ∥ X ∥1 = ∑ | X (k)|1 = N (1 + + )= .
k =0
4 5 20
The signal energy is
N −1
1 1 441N 2
M2 = ∑ | X (k)|2 = N 2 (1 + + )= .
k =0
16 25 400
We can see that M0 counts the number of nonzero coefficients. The

measure M1 cannot be used to compare sparsity of different signals since its
value in the second case is lower than in the first case.
(c) For an arbitrary x (2) = z we can write
N −1 @ A
X (k ) = ∑ A1 e j10πn/N + A2 e j14πn/N e− j2πnk/N
n =0
@ A
+ z − A1 e j10π2/N − A2 e j14π2/N e− j2π2k/N
= A1 Nδ(k − 5) + A2 Nδ(k − 7) + Z0 (k), (10.63)
with
@ A
Z0 (k) = z − A1 e j10π2/N − A2 e j14π2/N e− j2π2k/N = z0 e− j2π2k/N .
It is obvious that
⎧
⎪
⎪ N for Z0 (k ) ̸= 0 and Z0 (5) ̸= − A1 N and Z0 (7) ̸= − A2 N
⎨ N−1 for Z0 (k ) ̸= 0 and (Z0 (5) = − A1 N or Z0 (7) = − A2 N)
M0 =
⎪
⎪ N−2 for Z0 (k ) ̸= 0 and (Z0 (5) = − A1 N and Z0 (7) = − A2 N)
⎩
2 for Z0 (k ) = 0, i.e., for z = A1 e j10π2/N + A2 e j14π2/N .
Minimal value of M0 is achieved for | Z0 (k )| = 0 when
z = x (2) = A1 e j10π2/N + A2 e j14π2/N .
Therefore the ℓ0 -norm based measure minimization recovers missing

signal sample in such a way to complete the form of a signal with smallest
number of complex sinusoids, producing the minimal count of nonzero DFT
coefficients. The smallest value of N when measure M0 can be used to
produce z = x (2) must satisfy N − 2 > 2. Then the value of M0 in the fourth
line will be smaller than the value of M0 in the third line. In this case it means
N ≥ 5.
For the ℓ1 -norm based sparsity measure, from
X (k ) = A1 Nδ(k − 5) + A2 Nδ(k − 7) + Z0 (k ),
follows
N −2
M1 = | A1 N + Z0 (5)| + | A2 N + Z0 (7)| + ∑ | Z0 (k)|
k =0
k̸=5,k̸=7
' ' ' '
' ' ' '
= 'A1 N + z0 e− j2π10/N ' + 'A2 N + z0 e− j2π14/N ' + ( N − 2) |z0 |
We know that in the case of correct solution for |z0 | = 0
M1 = | A1 | N + | A2 | N.
Therefore in any other case when |z0 | ̸= 0

' ' ' '
' ' ' '
'A1 N + z0 e− j2π10/N ' + 'A2 N + z0 e− j2π14/N ' + ( N − 2) |z0 | > | A1 | N + | A2 | N
(10.64)
should hold, including the case when the phases of A1 N and z0 e− j2π10/N and
the phases'of A2 N and z0 e− j2π14/N ' are
' opposite. This is the' worst case since
' − j2π10/N ' ' − j2π14/N '
the terms 'A1 N + z0 e ' and 'A2 N + z0 e ', corresponding to
signal components are minimal. In the worst case
' ' ' '
' ' ' '
| A1 N | − 'z0 e− j2π10/N ' + | A2 N | − 'z0 e− j2π14/N ' + ( N − 2) |z0 | > | A1 | N + | A2 | N
should also hold. It reduces to
( N − 4) | z0 | > 0
for any |z0 | ̸= 0. 2 The minimization result |z0 | = 0 is the same as in the ℓ0 -
norm based measure if N ≥ 5. The minimal requirement for this reconstruc-
tion is N = 5. The number of available samples is M = 4 and the signal spar-
sity is K = 2.
Note that the condition for ℓ0 -norm to fail for N = 4 was Z0 (5) =
z0 e− j2π10/N = − A1 N and Z0 (7) = z0 e− j2π14/N = − A2 N. It means that A1 =
A2 e− j2π4/N should hold. In the ℓ1 -norm the phases of A1 N and z0 e− j2π10/N
and the phases of A2 N and z0 e− j2π14/N should only be opposite, in the worst
case. The condition for the ℓ0 -norm to fail is just a special case of the ℓ1 -norm
condition with | A1 | = | A2 | = |z0 | /N. If the condition for the ℓ0 -norm to fail
is satisfied then the condition for the ℓ1 -norm to fail is satisfied as well. This
conclusion, drown from a very specific example, will be generalized later.
For the energy
N −1 N −1 N −1
M2 = NEx = ∑ | X (k)|2 = N ∑ | x (n)|2 = N ( ∑ | x (n)|2 + | x (2)|2 ).
k =0 n =0 n =0
n ̸ =2
2
Since the value of ∑nN=−0,n
1
̸=2 | x (n )| is constant (the available samples are exact
and that they should not be changed) then the value of M2 is minimal if
| x (2)| = |z| = 0.
Therefore in the ℓ2 -norm (or energy) based minimization the missing sample
will be set in such a way to produce the minimal energy. That is zero value
of the missing samples/measurements. The reconstructed DFT using M2
minimization is
@ A
X (k ) = A1 Nδ(k − 5) + A2 Nδ(k − 7) + − A1 e j10π2/N − A2 e j14π2/N e− j2π2k/N .
In general X (k) ̸= 0 for all k, (10.63).
The simplest illustrative reconstruction can be performed based on a

direct search over all unavailable/missing sample values, by minimizing
the sparsity measure. This method is not computationally feasible except
for very small number of missing samples. We will use it just two illustrate
the sparsity measures behavior.
Consider a complete set of signal samples { x (1), x (2), ..., x ( N − 1)}. If
M of them y= [ x (n1 ), x (n2 ), ..., x (n M )] are available, the missing N − M
samples are yc =[ x (n M+1 ), x (n M+2 ), ..., x (n N )] with x= y ∪ yc . The values
2 Note that
' in any other than ' the' worst case the condition
' would' be ( N − 4)'|z0 | + α > 0
where ' A1 N + z0 e− j2π10/N ' + ' A2 N + z0 e− j2π14/N ' = | A1 N | − 'z0 e− j2π10/N ' + | A2 N | −
' − j2π14/N '
' z0 e ' + α, where α is a positive constant. This condition is less restrictive for |z0 |
than the worst case when α = 0.
of missing samples are the solution of the sparsity measure minimization

N −1
min ∑ | X (k)| p subject to y= AX
yc
k =0
Since this is a direct search method then any valid sparsity measure can
be used. From the available samples we can estimate the range limits for
the missing samples A. For example, A = max | x (ni )|, i = 1, 2, ..., M. In the
direct search approach we can vary each missing sample value from − A to
A with a step ∆x = 2A/( L − 1), where L is the number of considered values
within the selected range. It is obvious that the reconstruction error in each
sample is limited by the step 2A/( L − 1) used in the direct search. Number
of the analyzed values for N − M coefficients (variables) is L( N − M) . For any
reasonable accuracy the value of L is large and the number of calculations
L( N − M) is extremely large. One possible approach to reduce the number
of calculations in the direct search is to use a large step (small L) for the
first (rough) estimation, then to reduce the step around the rough estimate
of unavailable/missing values x (n M+1 ), x (n M+2 ),..., x (n N ). This procedure
can be repeated several times, until the desired accuracy is achieved.
Example 10.15. Consider a discrete signal
x (n) = cos(2πn/N ) + 0.5 sin(8πn/N ) + 0.4 cos(30πn/N + π/3) − 0.8
(10.65)
for n = 0, 1, . . . , N − 1, and N = 256 is the number of signal samples. The case
of two missing samples x (n N −1 ) and x (n N ) is presented. The direct search
is performed over a wide range [−3, 3] with a step of 0.01. Sparsity measure
M p is calculated for p = 0, p = 1/2, p = 1, and p = 2. Results for M p /N
are shown in Fig. 10.13. The measure minimum is located on the true sample
values for p ≤ 1 (norms ℓ1 and lower). The measure minimum for p > 1 (ℓ2
norm, for p = 2) is not located at the true signal values, as expected.
Note that p ≤ 1 produces accurate position of the sparsity measure at
the missing sample positions. For ℓ0 -norm the value of measure is constant
and equal to N everywhere, except at the exact values of the missing samples.
For p = 2 the measure with ℓ2 -norm has a minimum when the missing signal
samples are set to zero, which is not the solution of this problem.
10.5.2 Illustration in the Sparsity Domain
In general we have a reduced set of M measurements/samples of a sparse

signal with N samples y = AX. In the minimization process the values of
X (k ) can be considered as variables. One measurement/sample defined by
ψ0 (ni ) X (0) + ψ1 (ni ) X (1) + ... + ψN −1 (ni ) X ( N − 1) = y(i )

5 5
4.5 4.5
4 4
3.5 3.5
3 3
2.5 2.5
2 2
1 1
0 0
-1 -1
x(n ) x(n )
N -2 N -2
-3 -1 -2 -3 -3 -1 -2 -3
1 0 x(nN-1 ) 1 0 x(n )
N-1
5 5
4.5 4.5
4 4
3.5 3.5
3 3
2.5 2.5
2 2
1 1
0 0
-1 -1
x(n ) x(n )
N -2 N -2
-3 -1 -2 -3 -3 -1 -2 -3
1 0 x(n ) 1 0 x(n )
N-1 N-1
Figure 10.13 Measure as a function of two missing sample values yc (0) = x (n N −1 ) and
yc (1) = x (n N ) corresponding to various norms. True values of missing samples are presented
with lines. For the presentation all measures are normalized to the interval from 2.5 to 4.9.
represents an N −dimensional hyperplane with variables X (k ) and con-

stants ψk (ni ) and y(i ). For graphical illustrations we will assume real-valued
functions and signals.
Consider a signal in the transformation domain X (k ) with three pos-
sible values, N = 3 and k = 0, 1, 2. Assume that X (k ) is sparse with sparsity
K = 1. It means that only one coefficient X (k ) is nonzero. Using measure-

ments with functions ψk (ni ), k = 0, 1, 2, and ni ∈ {0, 1, 2} we get a plane in
three-dimensional space
ψ0 (ni ) X (0) + ψ1 (ni ) X (1) + ψ2 (ni ) X (2) = y(i ).
The solution will be illustrated in the space of variables X (0), X (1), and
X (2).
Consider one measurement denoted by
ψ0 (0) X (0) + ψ1 (0) X (1) + ψ2 (0) X (2) = y(0). (10.66)
To simplify notation it has been assumed that ni = i. This is a plane in the

three-dimensional space of variables X (0), X (1), and X (2). Since the signal
is sparse with K = 1 it means that only one X (k ) value is nonzero. The
solution of problem is an intersection of plane (10.66) with a coordinate axis.
In general plane (10.66) has three intersections with coordinate axes. Thus
there are three possible solutions. The problem cannot be solved using only
one measurement, Fig.10.14(a).
If we add one more measurement then two planes of variables X (0),
X (1), and X (2) are obtained
ψ0 (0) X (0) + ψ1 (0) X (1) + ψ2 (0) X (2) = y(0) (10.67)

ψ0 (1) X (0) + ψ1 (1) X (1) + ψ2 (1) X (2) = y(1).
In general, these two planes intersect along a line in the three-dimensional

space. If there is only one common intersection point with coordinate axes,
for both planes, then it is the solution of our problem, Fig.10.14(b).
However, in special cases two planes (measurements/samples) may
not be sufficient to get a unique solution:
-First special case is when two planes (10.67) intersect along a line
passing through two of the possible solutions (intersecting with two coor-
dinate axes at the common points). It means that the intersection line lies
in one of the coordinate planes, Fig.10.14(c). Then these two planes are not
sufficient to find a unique solution.
-Second special case is when the planes intersect along the axis, con-
taining the solution. In that case the origin (with trivial solution with spar-
sity 0) will be a possible (undesired) solution as well.
-Finally if planes (10.67) coincide then all coefficients of one plane are
just scaled versions of the coefficients in the other equation, Fig.10.14(d).
In this case the second measurement does not introduce any additional
information with respect to the already existing measurement.
Figure 10.14 Illustration of solution for N = 3 and K = 1 for various possible cases.
Consider the direction vector p of the measurements line defined by

the system of planes (10.67). It is normal to vectors of the planes defined
by (ψ0 (0), ψ1 (0), ψ2 (0)) and (ψ0 (1), ψ1 (1), ψ2 (1)). Vector p coordinates are
equal to the vector product
' '
' i X (0) i X (1) i X (2) '
' '
p = '' ψ0 (0) ψ1 (0) ψ2 (0) ',
'
' ψ0 (1) ψ1 (1) ψ2 (1) '
where i X (k) are unity vectors along coordinate axes representing X (k ). For
sparsity K = 1 the solution is unique if the measurements line is not within
any of coordinate planes. The components of vector p
p X (0) = ψ1 (0)ψ2 (1) − ψ2 (0)ψ1 (1)

p X (1) = ψ0 (0)ψ2 (1) − ψ2 (0)ψ0 (1)
p X (2) = ψ0 (0)ψ1 (1) − ψ1 (0)ψ0 (1)
must be such that

M' ' ' ' ' 'N
' ' ' ' ' '
min 'p X (0) ' , 'p X (1) ' , 'p X (2) ' > 0.
Then the measurements line of system will (10.67) not lie in one of the
coordinate planes, meaning that the solution is unique, Fig.10.14. Note that
the values of vector p components are equal to the determinants of the
system presented and discussed in the first illustrative example, (10.2).
In the ℓ0 -norm based minimization, the task is to solve
N −1
min ∥X∥0 = ∑ | X (k)|0 subject to y = AX
k =0
Therefore in the original N dimensional space the solution is on the inter-

section of the maximal possible number of N −dimensional hyperplanes of
the form X (k K +1 ) = 0, X (k K +2 ) = 0,..., X (k N ) = 0 satisfying all available
equations/conditions
ψl1 (ni ) X1 + ψl2 (ni ) X2 + ... + ψlK (ni ) XK = y(i ),

i = 1, 2, , ..., M.
When the number of zero values of X (k ) is maximal then the number of its
nonzero values (the sparsity) is minimal.
Example 10.16. Find the minimal sparsity solution for measurements
0.3617X (0) − 0.4942X (1) + 0.3611X (2) = −0.4550
−0.2991X (0) − 0.4967X (1) + 0.4052X (2) = −0.5105
using combinatorial approach and ℓ0 sparsity measure.
⋆Start with possible sparsity K = 1. Then we find solutions of these
equations with all possible combinations with one nonzero coefficient: { X (0),
X (1) = 0, X (2) = 0}, { X (0) = 0, X (1), X (2) = 0}, and { X (0) = 0, X (1) = 0,
X (2)}. For each of these combinations we get a solution of the first and the
second equation. The solution which is the same for the first and second
equation is { X (0) = 0, X (1) = 0, X (2) = −1.2600}. It is the solution of the
problem. Signal is of sparsity card { X (k )} = 1.
In general, direct optimization using the ℓ0 -norm is combinatorial NP-

hard problem. For a signal with N samples, whose sparsity is K, the number
of combinations is ( N
K ). This is the reason why convex formulation of the
problem is done using the ℓ1 -norm
N −1
min ∥X∥1 = ∑ | X (k )| subject to y = AX.
k =0
The problem is solved by minimizing
z = ∥X∥1 = | X (0)| + | X (1)| + | X (2)| + ... + | X ( N − 1)|
subject to the set of given hyper-planes (measurements)
ψl1 (ni ) X1 + ψl2 (ni ) X2 + ... + ψlK (ni ) XK = y(i ),
for i = 1, 2, ..., M.
For the graphical illustration we use the three-dimensional signal with
transformation coefficients X (k ), k = 0, 1, 2. We will also assume that the
sparsity is K = 1 and that M = 2 measurements/samples are available. In
this case we minimize
z = | X (0)| + | X (1)| + | X (2)|
subject to two available measurements (assuming ni = i)
ψ0 (0) X (0) + ψ1 (0) X (1) + ψ2 (0) X (2) = y(0) (10.68)

ψ0 (1) X (0) + ψ1 (1) X (1) + ψ2 (1) X (2) = y(1).
The available measurements in this case represent a line (measurements

line) in the considered space. The solution is obtained by finding the mini-
mal value of z when the three-dimensional ℓ1 -norm "ball"
z = | X (0)| + | X (1)| + | X (2)|
has a common point with line (10.68), Fig.10.15 (left). Since the sparsity
K = 1 is assumed, intersection of the measurements line is at the corner
of the ℓ1 -norm "ball". Considering the values of minimization function z =
| X (0)| + | X (1)| + | X (2)| along the line (10.68) its minimum will be achieved
at the corner, which is a sparse solution of the problem. It is important to
note that, in this case, the solution is the same as if we used minimization
Figure 10.15 Illustration of solution with norm-one and norm-1/4 (close to norm-zero) for a
three dimensional case. In lower graphics a view from the direction where the measurements
line and norm-1/4 ball are touching is presented.
of the norm close to the ℓ0 -norm, for example, z = | X (0)|1/4 + | X (1)|1/4 +

| X (2)|1/4 , presented in Fig.10.15 (right).
The ℓ0 -norm and ℓ1 -norm based minimizations can produce the same
result. Illustration of the conditions that have to be satisfied by measure-
ments line to get the same result with these two norms will be presented on
the three-dimensional case. For the illustration assume that the solution of
the problem is X (0) = z0 ̸= 0. In addition to the condition required by the
ℓ0 -norm that the measurements line does not lie within the planes X (1) = 0
or X (2) = 0,
|ψ0 (0)ψ2 (1) − ψ2 (0)ψ0 (1)| > 0 and |ψ0 (0)ψ1 (1) − ψ1 (0)ψ0 (1)| > 0
the measurements line in the ℓ1 -norm case should not have such a direction
to intersect with (go thought) the ℓ1 -norm "ball" | X (0)| + | X (1)| + | X (3)| =
z0 . Therefore, in the worst case the measurements line should intersect the
plane X (0) = 0 just outside the thick line | X (1)| + | X (2)| = z0 . If a part
of line is in the first octant then it means that it should pass above the
line | X (1)| + | X (2)| = z0 , Fig.10.16. Several possible measurements lines are
presented in Fig.10.16 (top-left). Their intersections with X (0) = 0 plane are
denoted by numbers from 1 to 7. For the measurements lines presented by
2, 3 or 4, the ℓ1 -norm minimization will produce the correct result for X (k ).
It is ( X (0), 0, 0). Line 1 is the critical case when z = | X (0)| + | X (1)| + | X (3)|
is constant along whole line within the first octant (any value within this
interval can be the minimization solution). Value of z = | X (0)| + | X (1)| +
| X (3)| will not be minimal at ( X (0), 0, 0) for lines 5, 6 and 7. The ℓ1 -norm
function assumes lower values along these lines than at ( X (0), 0, 0) point, as
the line penetrate into the ℓ1 -norm "ball".
A unified condition for all possible nonzero values of X (k ), is that the
direction of the measurements line has such direction vectors p X (0) , p X (1) ,
and p X (2) that its minimal coordinate along any of axes X (k ) is such that it
passes above the minimization ℓ1 -norm "ball". It means
' ' ' ' ' ' ' ' ' ' ' '
' ' ' ' ' ' ' ' ' ' ' '
'p X (0) ' + 'p X (1) ' + 'p X (2) ' − max{'p X (0) ' , 'p X (1) ' , 'p X (2) '}
' ' ' ' ' ' > 1.
' ' ' ' ' '
max{'p X (0) ' , 'p X (1) ' , 'p X (2) '}
' ' ' ' ' ' ' '

' ' ' ' ' ' ' '
For example, for 'p X (0) ' = max{'p X (0) ' , 'p X (1) ' , 'p X (2) '} we get
' ' ' '

' ' ' ' ' ' ' '
'p X (1) ' 'p X (2) '
' '+' ' > 1, and ''p X (1) '' ̸= 0, ''p X (2) '' ̸= 0.
' ' ' '
'p X (0) ' 'p X (0) '
If this relation is satisfied for the worst case, it means that it holds for other
directions as well. Then the line should pass trough X (0) = 0 outside the
region indicated by thick lines in Fig.10.16. It includes lines 2,3, and 4. The
imposed condition is still very close to line 1. If the measurements line is
close to line 1 it would be sensitive to even a small noise.
In the case of the ℓ1/2 -norm based measure
z = | X (0)|1/2 + | X (1)|1/2 + | X (3)|1/2
minimization we can see from illustration in Fig.10.16 that some measure-

ments which did not produce the correct result with ℓ1 -norm will be able to
reconstruct the original sparse signal values (line 5). However lines 6 and 7
will not produce the correct sparse result even with the ℓ1/2 -norm. In this
case the line direction will be considered with respect to the thick line in
Fig.10.16 (middle) defined by
[' ' [
\' \' '
\ 'p X (1) '' \ ''p X (2) ''
\ \
]' ' + ]' ' > 1.
' ' ' '
'p X (0) ' 'p X (0) '
The same analysis with direction lines is repeated with a function

being closer to ℓ0 -norm
z = | X (0)|1/5 + | X (1)|1/5 + | X (3)|1/5 .
For a norm close to the ℓ0 -norm the condition reduces to the discussed case
when all direction coordinates should be slightly greater than zero
[' ' [
\' \' '
\ 'p X (1) '' \ ''p X (2) ''
\
5 \
5
] ' ' +] ' ' > 1.
' ' ' '
'p X (0) ' 'p X (0) '
In this case all measurements corresponding to lines 1-6 will produce correct
result. The measurement 7 is the only one which will not produce the correct
sparse solution, Fig.10.16 (bottom).
An ideal measurements line would correspond to the case when full
isometry is preserved, i.e., when
∥A2 X2 ∥22
1 − δ2 ≤ ≤ 1 + δ2
∥X2 ∥22
with δ2 = 0. It has been assumed that columns of A are normalized with

E A = 1. Then
% ; % ;
∥A2 X2 ∥22 ∥A2 X2 ∥22
dmax = max eig = dmin = min eig = 1.
∥X2 ∥22 ∥X2 ∥22
X(2) X(2)
5 3
7
1 2
6
4
X(1) X(1)
p=1 p=1
X(0) X(2) X(0) X(2)
5 3
7
1 2
6
4
X(1) X(1)
p = 0.5 p = 0.5
X(0) X(2) X(0) X(2)
5 3
7
1 2
6
4
X(1) X(1)
p = 0.2 p = 0.2
X(0) X(0)
Figure 10.16 Minimization function | X (0)| + | X (1)| + | X (3)| = z0 in the first coordinate
system octant (X (0), X (1), X (3) > 0) thick lines. A dot at (0, 1, 1) surrounded with a gray
rectangular region belongs to the ideal measurement line.
It means that all eigenvalues of A2T A2 , for any combination

√ of two columns,
are 1 ≤ di ≤ 1. All eigenvalues of A2 are then ± di = ±1. Since the determi-
nant of A2 (direction vector coordinates for three-dimensional ' case) ' is equal
' '
to the product of eigenvalues of A2 , it means p X (i) = ±1 or 'p X (i) ' = 1. The
ideal case, corresponding to (0, 1, 1) point in the first octant, is presented by
the dot in Fig.10.16 and will be used as a reference in examples.
Figure 10.17 Minimization using the l1 -norm with the solution illustration for the case when
the measurements line crosses through the l1 -norm "ball".
Example 10.17. The previous relations are tested on K = 1 sparse signal with N = 3
possible values of X (k ) using two measurements with random Gaussian
coefficients ψk (n) = N (0, 1/2). Reconstruction mean square error for each
of 1000 realizations, classified using the measurements line directions, is
presented.
-In 791 random realizations we had the case that the measurements
line direction is outside the ℓ1 -norm "ball". The error in reconstruction using
the ℓ1 -norm minimization for the measurements line directions outside the
ℓ1 -norm "ball" is shown in Fig.10.18 (top). We see that for all cases with the
measurements line directions outside the ℓ1 -norm "ball" the reconstruction is
successful, with a small (computer precision) error.
Reconstruction square error with l and directions outside l "ball"

1 1
0
10
-10
10
-20
10
-30
10
0 100 200 300 400 500 600 700
Reconstruction square error with l and directions through l "ball"

1 1
0
10
-10
10
-20
10
-30
10
0 50 100 150 200

1/2 1
0
10
-10
10
-20
10
-30
10
0 50 100 150 200

1/4 1
0
10
-10
10
-20
10
-30
10
0 50 100 150 200
Figure 10.18 Reconstruction square error in 1000 realizations, classified using the data line
direction: Error using l1 minimization for directions outside the l1 "ball" (top). Error using l1
minimization for directions through the l1 "ball" (second). Error using l1/2 minimization for
directions through the l1 "ball" (third). Error using l1/4 minimization for directions through the
l1 "ball" (bottom).
-In 209 random realization we had the case that the measurements line
direction is crossing the ℓ1 -norm "ball", Fig, 10.17. In all these cases the ℓ1 -
norm based reconstruction was not successful. Error using ℓ1 minimization
for directions through the ℓ1 -norm "ball" is presented in Fig.10.18 (second).
-All 209 random realizations (when the measurements line direction
is crossing the ℓ1 -norm "ball") are also considered by using the l1/2 -norm
minimization. Then many of the measurements lines crossing the ℓ1 -norm
"ball" will not be crossing the ℓ1/2 -norm "ball". Recovery results for the
directions crossing the ℓ1 -norm "ball" by using the ℓ1/2 -norm minimization
are presented in Fig.10.18 (third). As expected many full recovery realizations
are achieved.
-Finally all 209 random realizations when the measurements line di-
rection is crossing the ℓ1 -norm "ball" are considered by using the ℓ1/4 -norm
minimization. Error using the ℓ1/4 minimization for directions through the
ℓ1 -norm "ball" is given in Fig.10.18 (bottom). All cases are successfully re-
covered since the ℓ1/4 -norm is close to the ℓ0 -norm. It would fail in a low
probable case when the measurements line would pass trough (or would be
very close to) one of the coordinate planes.
-Two specific examples of measurements (illustrating the reconstruc-
tion calculation) with directions inside and outside "ball" will be given in
detail next. For the direction outside the ℓ1 -norm "ball" the measurement
1.3681X (0) − 1.1171X (1) − 1.9446X (2) = 2.4502

−0.3370X (0) − 1.2624X (1) − 0.0207X (2) = 0.0261
is considered. Minimization is done by expressing X (0) and X (1) from

the measurement equations in terms of X (2) and then by minimizing z =
| X (0)| + | X (1)| + | X (2)|. It is now a function of one variable X (2) only.
Minimization is done in a numeric way. The result is X (0) = 0, X (1) = 0,
and X (2) = −1.2600.
An example of a measurement that produces a direction through the
ℓ1 -norm "ball" is
0.3617X (0) − 0.4942X (1) + 0.3611X (2) = −0.4550

−0.2991X (0) − 0.4967X (1) + 0.4052X (2) = −0.5105.
Minimization of z = | X (0)| + | X (1)| + | X (2)| produces X (0) = 0.0802, X (1) =

0.9800, and X (2) = 0.0007. If the measure z = | X (0)|1/2 + | X (1)|1/2 + | X (2)|1/2
is used in the minimization (in a numeric way) it will produce the correct
result (the same X (k ) as in the previous measurement), when the measure-
ments line was outside the ℓ1 -norm ball.
In some applications a random Gaussian measurement matrix

⎡ ⎤
ψ0 (n1 ) ψ1 (n1 ) ψ N −1 ( n1 )
⎢ ψ0 (n2 ) ψ1 (n2 ) ψ N −1 ( n2 ) ⎥
A =⎢
⎣
⎥
⎦
... ... ...
ψ0 (n M ) ψ1 (n M ) ψ N −1 ( n M )
is normalized so that the energy of each column is ∥ψi ∥22 = 1. In that case
randomness is reduced and ψi (nm ) can be considered as coordinates of an
M-dimensional vector ψi whose ending points are on the M-dimensional
unity sphere. This condition can change behavior of the measurement ma-
trix.
Example 10.18. For the normal set of measurement coefficients (when the column
energies are normalized)
ψk2 (0) + ψk2 (1) = 1, k = 0, 1, 2
the transformation can be written as
∥A2 X∥22 = |ψi (0) X (i ) + ψk (0) X (k)|2 + |ψi (1) X (i ) + ψk (1) X (k)|2
B C B C
= |ψi (0)|2 + |ψi (1)|2 | X (i )|2 + |ψk (0)|2 + |ψk (1)|2 | X (k)|2
+2 [ψi (0)ψk (0) + ψi (1)ψk (1)] X (i ) X (k)
and
∥A2 X∥22 − ∥X∥22 X (i ) X ( k )
= 2 [ψi (0)ψk (0) + ψi (1)ψk (1)]
∥X∥22 ∥X∥22
≤ ψi (0)ψk (0) + ψi (1)ψk (1) = µ(i, k)
since 2X (i ) X (k )/ ∥X∥22 ≤ 1, as shown in (10.23). For the normal set of coeffi-

cients we have
µ2 (i, k ) + p2X (l ) = 1
since
[ψi (0)ψk (0) + ψi (1)ψk (1)]2 + [ψi (0)ψk (1) − ψk (0)ψi (1)]2
@ A @ A
ψi2 (0) ψk2 (0) + ψk2 (1) + ψi2 (1) ψk2 (0) + ψk2 (1) = 1
for i ̸= k ̸= l and i, k, l ∈ {0, 1, 2}. Therefore the condition
δ2 = max {|µ(i, k )|} < 1
is equivalent to M' ' ' ' ' 'N

' ' ' ' ' '
min 'p X (0) ' , 'p X (1) ' , 'p X (2) ' > 0.
In this case, it can be shown that

' ' ' ' ' '
' ' ' ' ' '
'p X (i) ' + 'p X (k) ' ≥ 'p X (l ) '
for any i, k, l. It means that the normalized matrix (for the three-dimensional
case) will always satisfy the condition that the ℓ1 -norm and the ℓ0 -norm
solutions are the same (measurements lines are always outside the ℓ1 -norm
"ball").
10.5.3 Equivalence of the Norm-Zero and Norm-One Based Minimiza-

tion Solutions
Consider an N-dimensional vector X whose sparsity is K and its M mea-

surements y = AX. The measurements matrix A is an M × N matrix, with
K < M ≤ N. A reconstruction of vector X can be achieved from a reduced set
of samples/measurements using the sparsity measures minimization. The
ℓ0 -norm based solution of sparsity measure minimization
recovers K sparse vector X from a reduced set of measurements if the

measurements matrix A satisfies the restricted isometry property for a 2K
sparse vector
1 2
E A ∥ A2K X2K ∥2
1 − δ2K ≤ ≤ 1 + δ2K
∥X2K ∥22
with
0 ≤ δ2K < 1.
The ℓ1 -norm based minimization
min ∥X∥1 subject to y = AX (10.69)
produces the same result as the ℓ0 -norm based minimization if the restricted
isometry property is satisfied with the constant
√
0 ≤ δ2K < 2 − 1.
Note that other possible upper bounds on the isometry constant have been
derived in literature. Illustration of the reason why the restricted isometry
condition has to be more strict in the ℓ1 -norm based minimization than in
the ℓ0 -norm is presented in the previous section. Proof is outside of the
mathematical tools used in this book.
If the signal X is not sparse then the solution of minimization problem

(10.69) denoted by X R will satisfy
∥ X K − X ∥1
∥X R −X∥2 ≤ C0 √ (10.70)
K
where XK is K sparse signal whose nonzero values are equal to K largest

values of X. If the signal X is of sparsity K then ∥XK −X∥2 = 0 and X R = X.
∥ X K − X ∥1
Note that according to Schwartz’s inequality √ ≤ ∥ X K − X ∥2 .
K
Example 10.19. Consider a signal with coefficients X = [ X0 a b] where |b| < | a| <
X0 . Consider M = 2 measurements with idealized measurements line when
δ2K = 0 (in real cases δ2K can be small but nor zero), defined by
X ( 0 ) − X0 X (1) − a X (2) − b
= = =t
−1 1 1
Find the result of minimization problem (10.69) as a function of a and b.
⋆Replacing X (0) = X0 − t, X (1) = a + t and X (2) = b + t, where t is
the line parameter, we get the value of minimization function z = ∥X∥1 along
the measurements line in the form
z = | X0 − t | + | a + t | + | b + t | .
Minimum of this function is at
t0 = median{ X0 , − a, −b}
since the function z increases both right and left from t0 . It increases with
rate 1 until the first of X0 , − a, −b is reached left and right, and then increases
toward +∞ as t tends toward ±∞. More details about median based mini-
mization will be given in the next subsection.
Illustration is presented in Fig.10.19 with
X0 = 2/3, a = 2/9, and b = −1/9
when
t0 = median{2/3, −2/9, 1/9} = 1/9 = −b
with
XR (0) = X0 + b, XR (1) = a − b and XR (2) = 0

XR (0) = 5/9, XR (1) = 1/3 and XR (2) = 0
It means that the solution is a signal X R with reduced sparsity as compared

to the original X. The data line will touch the ℓ1 -norm "ball" somewhere on
the edges. Since all edges are within the coordinate planes, it means that
minimization (10.69) will reduce the sparsity to at least K = 2 for any X = [ X0

a b]. We can see that value X0 = 2/3 = 6/9 is reduced to XR (0) = 5/9 and
XR (1) = 1/3 = 3/9, while XR (2) = 0.
The absolute error in X (0) and X (1) is equal to b. Energy of error is
proportional to the energy of the reduced coordinates
F
∥X R −X∥2 = ( X0 − ( X0 + b))2 + ( a − ( a − b))2 + (b − 0)2
√ ∥ X − X ∥1
= |b| 3 ≤ C0 K√
K
| X0 + b − X0 | + | a − ( a − b)| + |0 − 0| |b|
= C0 √ = C0 2 √ .
2 2
√
The equality holds here with C0 = 3/2. For a = 0 and b = 0 the solution is
X = [ X0 0 0] as expected.
In the case of noisy measurements when
∥y − AX∥2 ≤ ϵ
then
∥ X K − X ∥1
∥X R −X∥2 ≤ C0 √ + C1 ϵ
K
where C0 and C1 are constants depending on δ2K .
Example 10.20. For Examples 10.3 and 10.4 estimate the maximal signal sparsity
when the solutions using the ℓ1 -norm based minimization and the ℓ0 -norm
based minimization are the same.
√
⋆The restricted isometry property is satisfied with ρK = λmax < 2 − 1
for K = 24 in Example 10.3. It means that the uniqueness is guarantied for
signals of sparsity K/2 = 12. Note that this is a statistical estimate in 10000
realizations. The true bound is slightly lower.
In the case of the DFT matrix√ in Example 10.4 the restricted isometry
property was satisfied with ρK < 2 − 1 for K = 2 only, meaning that in the
recovery we can guarantee the same solution for sparsity K = 1 only, with
M = 6 out of N = 8 samples.
The order of signal sparsity K such that the signal can be recovered
using M measurements/samples has been derived in literature as
M
K<C for Gaussian measurement matrix
log( N/M )
M
K<C for partial DFT matrix
log6 N
Figure 10.19 Minimization using the ℓ1 -norm and the solution illustration for the case when
the measurements line corresponds to noisy data.
where C are (different) constants.

More conservative bounds would be obtained if we used the spark
definition. Then two minimization formulations produce the same solution
if
1 1 1
K < spark(A) = (1 + )
2 2 µ
where µ is the coherence index of the measurement matrix A. It is easy to
show that this condition is the same as the requirements that the measure-
ment matrix A (its A2K submatrices) with coherence µ satisfies the restricted
isometry property with δ2K = (2K − 1)µ
' '
' 1 2 2 '' 2
'
' E ∥AX∥2 − ∥X∥2 ' ≤ (2K − 1)µ ∥X∥2 .
A
The condition that
δ2K = (2K − 1)µ < 1
is the same as K < 12 (1 + 1/µ). Note that δ2K = (2K − 1)µ is just a bound
of δ2K . For a matrix A there could be a lower and less restrictive constant
satisfying the restricted isometry property.
P Q −1
In an ideal case the matrix AKT AK should be identity matrix for any
combination of K columns. It means that the lines are with vector coordinate
1 in each direction. Reconstruction condition would be always satisfied. The
transformation y = AX would correspond to a rotation on a sphere with all
axis 1. Each X (0), X (1), X (2) would be transformed as y = AX keeping its
amplitude. Since this is not the case then the transform y = AX will change
amplitudes in addition to the rotation. For matrix A (not square matrix) the
maximal gain of vector X is obtained in a direction defined by the maximal
eigenvector. In reality
B C −1 B C−1
XK = AKT AK AKT y = AT
K AK X
0
1
∥XK ∥22 ≤ ∥X0 ∥22
d2min
√
with d2min = (1 − δk )2 . The condition δ2 < 2 − 1 would here mean that
1/d2min > 0.343. It has been assumed that E A = 1.
10.6 MEDIAN BASED FORMULATION
Illustrative explanation of the ℓ1 -norm based minimization can be presented

on a slightly more general case with N-dimensional signal. Consider the
case with M = N − 1 measurements. All coefficients X (k ) can be expressed
as a function of one coefficient, for example. X (0). For common signal
transforms, when a complete set of measurements exists, there is only one
degree of freedom in the minimization and the measurements are on a line
in the N dimensional space. First assume ideal directions of measurement
line
p X (0) = p X (1) = ... = p X ( N −1) = 1.

Then
X (1) = X (0) − b1
X (2) = X (0) − b2
...
X ( N − 1 ) = X (0 ) − b N −1 (10.71)
where bi are unknown coefficients. Cost function for minimization is
z = ∥X∥1 = | X (0)| + | X (0) − b1 | + ... + | X (0) − b N −1 | (10.72)
The solution of this minimization problem is the median
| X (0)| = arg{min{z}} = median{0, b1 , ..., b N −1 }.
Then for any value of coefficients bi at least one X (k ) will be equal to zero,
since at least one of the elements in z is zero. It means that the solution will
be of sparsity K = N − 1 at least.
In order to prove that the median produces position of (10.72) mini-
mum assume that the total number of terms N is an odd number. Function
z in (10.72) is a sum of the functions of form | x − a|. The rate (derivative) of
these functions is +1 for x > a and −1 for x < a. If there are N terms, as in
(10.72), then the rate of function z will be + N for x → ∞. Going now back
from x → ∞ toward the term with largest shift, the rate will remain + N.
At the position of the largest shift, the rate of this term will change from
+1 to −1 meaning that the overall rate of z will be reduced to +( N − 2).
By passing each term, the rate will be reduced for additional factor of 2. It
means that after the kth term the rate will be ( N − 2k ). The rate of z will
change its sign when ( N − 2k ) = −1. This will be the position of function z
minimum. It is k = ( N + 1)/2 and it corresponds to the middle coefficient
positions, i.e., to the median of coefficients (shifts).
Example 10.21. As an example consider the case with N = 7 and M = 6 measure-
ments AX = y producing an ideal line in a seven-dimensional space of the
form (10.71). with b1 = 0.7, b2 = 0.2, b3 = −0.5, b4 = 1, b5 = 0.8, and b6 = −0.9.
For the data presented in Fig.10.20 the solution is | X (0)| = arg{min{z}} =
median{0, 0.7, 0.2, −0.5, 1, 0.8, −0.9} = 0.2 with the coefficient corresponding
to X (2) = X (0) − 0.2 = 0 being equal to zero.
If the signal sparsity is K < N/2 then there will exist more than
N/2 values bi = b such that | X (0) − bi | = 0. The solution of minimization
problem then will not depend on other bk ̸= bi = b and will be unique
Functions |x-x 1|, |x-x 2|, ..., |x-x 7|

2
1.8
1.6
1.4
1.2
0.8
0.6
0.4
0.2
0
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1
z=|x-x 1|+|x-x 2|+...+|x-x 7|

9
dz/dx=-N=-7
8
7
dz/dx=-5
dz/dx=N=7
6
dz/dx=5
5 dz/dx=-3
dz/dx=3
dz/dx=1
dz/dx=-1
4
arg{min{z}}= median{x1,x2,x3,x4,x5,x6,x7}
3
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1
Figure 10.20 Median as the solution of minimization problem.

arg{min{z}} = median{0, b1 , ..., bi , bi , ..., bi , ..., b N −1 } = bi = b. Therefore for

one missing sample M = N − 1 the solution is unique for signals whose
sparsity is K < N/2.
If the directions are not ideal but p X (0) = a0 , p X (1) = a1 , ..., p X ( N −1) =
a N −1 then a form corresponding to the weighted median appears. For
N = 2P + 1 a weighted median produces the same result as the unweighted
median if a sum of the smallest P + 1 coefficient values is greater than a sum
of its P largest values
P N −1
∑ | ai | > ∑ | ai | .
i =0 i = P +1
For P = 1 we get
| a0 | + | a1 | + | a2 | − max{| a0 | , | a1 | , | a2 |}
> 1.
max{| a0 | , | a1 | , | a2 |}
This relation' corresponds

' to the thick line for the ℓ1 -norm in Fig.10.16(top)
' '
with | a0 | = 'p X (0) ' = max{| a0 | , | a1 | , | a2 |} and
' ' ' '

' ' ' '
'p X (1) ' + 'p X (2) '
' ' > 1.
' '
'p X (0) '
Consider next the case when two degrees of freedom exist (with M =
N − 2 measurements). All coefficients X (k ) can be expressed as a function
of, for example, X (0) and X (1) as
X (2) = a2,0 X (0) + a2,1 X (1) − b2 ,

...
X ( N − 1) = a N −1,0 X (0) + a N −1,1 X (1) − b N −1 .
Then
z = ∥ X (k )∥1 = | X (0)| + | X (1)| + | a20 X (0) + a21 X (1) − b2 |

+ ... + | a N −1,0 X (0) + a N −1,1 X (1) − b N −1 |
The solution of the minimization problem is a two-dimensional median. It

is a point in X (0), X (1) plane such that a sum of absolute distances from the
lines
X (0 ) = 0
X (1 ) = 0
a2,0 X (0) + a2,1 X (1) − b2 = 0
...
a N −1,0 X (0) + a N −1,1 X (1) − b N −1 = 0
is minimal3 . Median here is not so simple as in the one-dimensional case.

Various algorithms have been proposed for multidimensional (multivariate
or spatial) median form. Note that by crossing a line ai,0 X (0) + ai,1 X (1) −
xi = 0 we will always either increase or reduce the rate of the function z, as
in one dimensional case.
An illustration of signal with N = 6 is presented in Fig.10.21. Value of
z is presented, along with measurements lines, for the case of two degrees
of freedom (two dimensional variable space). From this figure we can see
that the number of measurements is M = 4 and the sparsity of signal is
K = 2 since the distance of the function z minimum point from four planes
is 0. There are two nonzero distances (to the thick black lines) meaning that
there are two nonzero coefficients X (k ). It is interesting that in this case the
marginal median (minimization along axes X (0) and X (1) independently
would produce the same result, since one of the zero values is on the axis).
For any value of variables at least two X (k ) will be equal to zero,
since at least two of the elements in z are zero. It means that, in general,
the solution will be of sparsity K = N − 2 at least.
In the case of M measurements the system
AX = y
contains M equations with N unknowns. It means that there are N − M

free variables, while M can be calculated based on the free variables. Let us
3 Distance of a plane ax + by + c = 0 from a point x0 , y0 is

| ax0 + by0 + c|
d0 = √ = | ax0 + by0 + c|
a2 + b2
√
if a2 + b2 = 1.
z(x,y)=|x|+|y|+|y+2x-2 |+|0.7y+0.5x-0.5 |+|y+1.6x+1.4|+|0.3y-0.5x+0.5 |

-1.5
-1
-0.5
median (z(x,y))=(1,0)
2
0
0.5
1.5
-1.5 -1 -0.5 0 0.5 1 1.5
Figure 10.21 Illustration of a two-dimensional median.
denote M unknowns X (k ) by vector X M . Then it can be written
⎡ ⎤ ⎡ ⎤⎡ ⎤
x ( n1 ) ψ0 (n1 ) ψ1 (n1 ) ψ N − M −1 ( n 1 ) X (0 )
⎢ x (n2 ) ⎥ ⎢ ψ0 (n2 ) ψ1 (n2 ) ψ N − M −1 ( n 2 ) ⎥ ⎢ X (1 ) ⎥
⎢ ⎥=⎢ ⎥⎢ ⎥
⎣ ... ⎦ ⎣ ... ... ... ⎦⎣ ... ⎦
x (n M ) ψ0 (n M ) ψ1 (n M ) ψ N − M −1 ( n M ) X ( N − M − 1)
⎡ ⎤⎡ ⎤
ψ N − M ( n 1 ) ψ N − M +1 ( n 1 ) ψ N −1 ( n1 ) X ( N − M)
⎢ ψ N − M ( n 2 ) ψ N − M +1 ( n 2 ) ψ N −1 ( n2 ) ⎥ ⎢ ⎥
+⎢ ⎥ ⎢ X ( N − M + 1) ⎥
⎣ ... ... ... ⎦ ⎣ ... ⎦
ψ N − M ( n M ) ψ N − M +1 ( n M ) ψ M −1 ( n M ) X ( N − 1)
y= B N − M X0,N − M−1 +C M X N − M,N −1

X N − M,N −1 = C− 1 −1
M y − C M B N − M X0,N − M−1
where X0,N − M−1 is the vector of free variables X (0), X (1),...,X ( N − M −

1), with corresponding measurement coefficients C− 1
N − M B N − M . Then the
minimization problem can be written as
z = ∥ X (k )∥1 = | X (0)| + | X (1)| + ... + | X ( N − M − 1)|

W W
W 1 W
+ WC−M y − C −1
M B N − M X 0,N − M−1 W .
1
It is reduced to (N − M)-dimensional median formulation over variables

X (0), X (1),..., X ( N − M − 1). Note that the multidimensional median cal-
culation is not simple as in the case of one-dimensional problem.
10.7 NORM-ONE BASED RECONSTRUCTION ALGORITHMS
In the ℓ1 -norm based reconstructions the problem is formulated as
where ∥X∥1 = ∑kN=−01 | X (k )|. The problem can be formulated in Lagrangian

form
F (X) = ∥y − AX∥22 + λ ∥X∥1
where F (X) is the function to be minimized.

Reformulation of the problem in a constrained form reads
min ∥y − AX∥22 subject to ∥X∥1 < ε
or
min ∥X∥1 subject to ∥y − AX∥22 < ε,
where ε is sufficiently small parameter.

There are many ways to solve the stated problem, based on the con-
strained or Lagrangian form. Many of them are developed within the regres-
sion theory. Here we will present just one of them, based on the least abso-
lute selection and shrinkage operator (LASSO) formulation and Lagrangian
minimization form.
10.7.1 LASSO- Minimization
The ℓ1 -norm based minimization can be formulated as the minimization of

y − AX with a condition imposed on X.
The standard ridge formulation within the regression framework
would minimize the error
∥y − AX∥22 = (y − AX)T (y − AX)

= ∥y∥22 − X T A T y − y T AX + X T A T AX
subject to the minimal energy values of X, i.e. subject to ∥X∥22 . The mini-
mization of the ridge constraint problem can be reformulated in Lagrangian
form using a parameter λ as
M N
X = arg min ∥y − AX∥22 + λ ∥X∥22 .
X
Minimization of
F (X) = ∥y − AX∥22 + λ ∥X∥22

= ∥y∥22 − X T A T y − y T AX + X T A T AX+λX T X
can be obtained in a closed form using the symbolic derivative operator
∂F (X)
= −2A T y + 2A T AX + 2λX = 0
∂X T
as B C −1
Xridge = A T A + Iλ A T y.
Parameter λ balances the error and constraint. Its inclusion makes that the
inversion is nonsingular even if A T A is singular. Real valued matrix A is
assumed, otherwise Hermitian conjugate and transpose A H would be used.
The standard ridge regression minimizes the energy of solution X (k )
and not its sparsity, Fig.10.22. That is the reason while the ℓ1 -norm con-
straint is introduced in the cost function
F (X) = ∥y − AX∥22 + λ ∥X∥1

= ∥y∥22 − X T A T y − y T AX + X T A T AX+λX T sign{X}
with the LASSO minimization problem formulation

M N
X = arg min ∥y − AX∥22 + λ ∥X∥1 .
X
|X(0)|2+|X(1)|2 |X(0)|+|X(1)| |X(0)|1/4+|X(1)|1/4
1 1 1
0.5 0.5 0.5
0 0 0
-0.5 -0.5 -0.5
-1 -1 -1
-1 0 1 -1 0 1 -1 0 1
Figure 10.22 Minimization with constraint: in ridge regression (left), LASSO regression (mid-
dle), and the ℓ1/4 -norm being a function closer to the ℓ0 -norm .
Function ∥X∥1 promotes sparsity. It produces the same results (under cer-
tain conditions) as if ∥X∥ p , with p close to 0, is used, Fig.10.22.
10.7.1.1 Iterative Calculation
The minimization problem with the ℓ1 -norm constraint does not have a
close form solution. It is solved in iterative ways. In order to define an
iterative procedure we will add a nonnegative term, having zero value at
the solution Xs of the problem,
G (X) = (X − Xs )T (αI − A T A)(X − Xs ),
to the function F (X). This term will not change the minimization solution.
New function is
H (X) = F (X) + (X − Xs )T (αI − A T A)(X − Xs ).
where α is such that the added term is always nonnegative. It means α >
λmax , where λmax is the largest eigenvector of A T A. Gradient of H (X) is
∂H (X)
∇ H (X)= = −2A T y + 2A T AX+λsign{X} + 2(αI − A T A)(X − Xs ).
∂X T
Solution of ∇ H (X) = 0 is
λ
−A T y+ sign{X}−(αI − A T A)Xs + αX = 0
2
λ 1
X+ sign{X} = A T (y − AXs ) + Xs .
2α α
Corresponding iterative relation is of the form
λ 1
X s +1 + sign{Xs+1 } = A T (y − AXs ) + Xs .
2α 2α
Note that the solution of scalar equation
x + λsign( x ) = y
is obtained using soft-thresholding rule defined by a function soft (y, λ) as

⎧
⎨ y+λ for y < −λ
x = soft(y, λ) = 0 for |y| ≤ λ .
⎩
y−λ for y>λ
or
soft(y, λ) = sign(y)max{0, |y| − λ}.
The same rule can be applied to each coordinate of vector Xs+1 ,
1 λ
Xs+1 =soft( A T (y − AXs ) + Xs , ) (10.73)
α 2α
or
1 λ
X (k )s+1 =soft( ( a(k ) − b(k )) + X (k )s , )
α 2α
where a(k ) and b(k ) are coordinates of vectors a and b defined by a = A T y
and b = A T AXs .
This is the iterative soft-thresholding algorithm (ISTA) for LASSO
minimization. It can be easily modified to improve convergence to fast ISTA
(FISTA). Note that this is just one of possible solutions of the minimization
problem with the ℓ1 -norm.
The Lagrangian constant λRis a balance S between the error and the
ℓ1 -norm value, while α = 2 max eig{A T A} is commonly used. The al-
gorithms that solve this kind of problem are implemented as functions
X =lasso(A, y).
Example 10.22. Measurement matrix A is formed as a Gaussian random matrix
of the size 40 × 60. Since there are 40 measurements the random variable
N (0, σ2 ) with σ2 = 1/40 is used. The original sparse signal of the total
length N = 60 is X (k ) = δ(k − 5) + 0.5δ(k − 12) + 0.9δ(k − 31) − 0.75δ(k −
45) in the transformation domain. It is measured with a matrix A with
40 measurements stored in vector y. All 60 signal values are reconstructed
using these 40 measurements y and the matrix A, in 1000 iterations. In
1 1
λ=0.01 λ=0.0001
0.75 0.75
sparse signal X(k)
sparse signal X(k)

0.5 0.5
0.25 0.25
0 0
-0.25 -0.25
-0.5 -0.5
-0.75 -0.75
-1 -1
0 10 20 30 40 50 60 0 10 20 30 40 50 60
index k index k
Figure 10.23 A sparse signal with N = 60 and K = 4 reconstructed using a reduced set of
M = 40 observations and LASSO iterative algorithm. The results for λ = 0.01 and λ = 0.0001
are presented.
the initial iteration X0 = 0 is used. Then for each next s the new values
of X are
R calculated S using (10.73), given data y and matrix A. Value of α =
2 max eig{A T A} is used. The results for λ = 0.01 and λ = 0.0001 are
presented in Fig.10.23. For very small λ = 0.0001 the result is not sparse, since
the constraint is too weak.
10.7.2 Signal Domain Reconstruction with a Gradient Algorithm
It is shown that the sparse signal reconstruction can be formulated as a con-

strained minimization problem. The sparsity measure is minimized having
in mind constraints defined by available samples/measurements). If a com-
plete set of samples/measurements can be defined then the signal recon-
struction can be formulated as a minimization problem where the missing
samples/measurements yc are considered as minimization variables, while
available samples/measurements y remain unchanged. The simplest way
to solve this problem is in changing all missing samples within the range
of their possible values and then to select the combination of their values
which produced the minimal sparsity measure. This method has been illus-
trated in Example 10.15 on a signal with two missing samples. However,
when the number of missing samples is large, then a direct search over all
missing sample values cannot be used due to its high calculation complex-
ity.
Minimization of the sparsity measure M can be implemented with
gradient descent (or steepest descent) method instead of using a direct
search over missing sample values. Minimum sparsity measure position is

determined through an iterative procedure
'
( m +1) (m) ∂M ''
yc = yc −α
∂yc 'yc =yc(m)
(m)
where yc is the vector of missing samples in the mth iteration and M is
(m)
the sparsity measure. Gradient of sparsity measure calculated at yc = yc is
denoted by ∂M/∂yc | (m) , while α is the iteration step. For the algorithm
yc =yc
convergence a convex measure function is required.
A signal x (n) that is sparse in a transformation domain X (k ) =
T { x (n)} is used for illustration. As in Example 10.15 it has been assumed
that two samples x (n N −1 ) and x (n N ) are not available, yc = ( x (n N −1 ),
x (n N )). Signal x a (n) is formed. Its values at the available sample positions
y = ( x (n1 ), x (n2 ), ..., x (n M )), M = N − 2, are considered as constants. Sam-
ples x (n N −1 ) and x (n N ) at the positions q1 = n N −1 and q2 = n N are con-
sidered as variables. For various values of x (n N −1 ) and x (n N ) the sparsity
measure of x a (n) is calculated as M = ∥T [ x a (n)]∥1 = ∥X a ∥1 and presented
in Fig. 10.24, along with illustration of the gradient ∂M/∂yc |yc =0 coordi-
nates at x (n N −1 ) = 0, x (n N ) = 0 .
Consider a signal x (n) with available samples at n ∈ M. Signal is
sparse in a transformation domain X (k ) = T { x (n)}. The DFT will be used
as a study case, X (k ) = DFT[ x (n)].
(0)
As the initial estimate of reconstructed signal x a we will use values
that would follow as a result of the ℓ2 -norm based minimization of the signal
(0)
transform. Values of x a are
!
(0) 0 for missing samples, n ∈ NQ
x a (n) =
x (n) for available samples, n ∈ M
where NQ is the set of missing sample positions. The available samples are
considered as constants, while the missing samples are changed through
(m)
iterations. Denote by x a the values of the signal reconstructed after m
iterations. The minimization process can be described as
(m)
min ∥X a ∥1 subject to x a (n)= x (n) for n ∈ M
(m)
where Xa (k ) = DFT[ Xa (n)]. Since the task is to find the position of func-
tion z = ∥X a ∥1 minimum, trough an iterative procedure, the relation for
5
|| X ||
a 1
4.5
3.5
2.5
2
1
-1
x(n )
N
-2
x(n ) -3
-3 -1 -2 N-1
1 0
Figure 10.24 Sparsity measure function in the case of two unavailable signal sam-
ples yc = ( x (n N −1 ), x (n N )) with corresponding gradient. Available samples are y =
( x (n1 ), x (n2 ), ..., x (n N −2 )).
missing samples calculation can be defined by using the gradient of sparsity

measure '
( m +1) (m) 1 ∂ ∥X a ∥1 ''
yc = yc − α (10.74)
N ∂yc 'yc =yc(m)
(m)
where yc is the vector of variables (missing signal sample values) in the
mth iteration, Fig. 10.13. Factor 1/N is introduced for the DFT analysis
so that coefficients X (k ) are equal to the signal amplitudes in time. The
coordinates of gradient vector g(ni ) = ∂ ∥X a ∥1 / (∂yc N ) in the mth iteration
can be estimated using finite differences of the sparsity measure calculated
for each variable (missing sample) ni ∈ NQ
∥X+ −
a ∥1 − ∥ X a ∥1
g ( ni ) =
2∆N
where
Xa+ (k ) = T { x +
a (n )}
Xa− (k ) = T { x −
a (n )}
and
(m)
x+
a ( n ) = x a ( n ) + ∆δ ( n − ni )
(m)
x−
a ( n ) = x a ( n ) − ∆δ ( n − ni ).
For ni ∈ M there are no changes of the signal values, g(ni ) = 0. A parameter

for finite difference calculation is denoted by ∆. All g(n) values form vector
denoted by Gm with elements Gm (n). The minimum of sparsity measure
is obtained when all unavailable samples are equal to the values of the
original signal values, i.e., when the signal is reconstructed (assuming that
the recovery conditions are satisfied).
10.7.2.1 Finite Difference Step
Before presenting the algorithm, the basic idea and parameters in (10.74)
will be discussed. Assume first a simple case when a single signal sample at
n0 ∈ NQ is not available, with card {M} = N − 1. This sample is considered
as variable. It may assume an arbitrary signal value x a (n0 ) = x (n0 ) + z(n0 ),
where z(n0 ) is a variable representing shift from the true signal value at n0 .
In order to estimate the finite difference of the sparsity measure
N −1
∥ X a ∥1 = ∑ | Xa (k)| ,
k =0
due to the change of variable z(n0 ), form the signals
x+
a ( n ) = x ( n ) + ( z ( n ) + ∆ ) δ ( n − n0 )
x−
a ( n ) = x ( n ) + ( z ( n ) − ∆ ) δ ( n − n0 ),
where ∆ is a parameter. The finite difference of the sparsity measure is
∥X+ −
a ∥1 − ∥ X a ∥1
g ( n0 ) = .
2N∆
The pulses δ(n − n0 ) are uniformly spread over all frequencies in the DFT
domain. Then
Xa+ (k ) = X (k ) + (z(n0 ) + ∆)e j2πn0 k/N

Xa− (k ) = X (k ) + (z(n0 ) − ∆)e j2πn0 k/N
holds. Since the signal is sparse (K ≪ N) in a rough analysis we may neglect

changes in a few nonzero values of X (k ). We may approximately write
W +W N −1 ' '
WX a W = ∑ ' Xa+ (k )' ∼
= µ + | z ( n0 ) + ∆ | N
1
k =0
W −W N −1 ' '
WX a W = ∑ ' Xa− (k )' ∼
= µ + |z(n0 ) − ∆| N,
1
k =0
where µ = ∥X∥1 is the sparsity measure of the original signal x (n). There-
fore the gradient approximation of the sparsity measure ∥X a ∥1 along the
direction of variable z(n0 ) is
∥X+ −
a ∥1 − ∥ X a ∥1 ∼ | z ( n0 ) + ∆ | − | z ( n0 ) − ∆ |
g ( n0 ) = = .
2N∆ 2∆
For deviations from the true signal value smaller than the step |z(n0 )| < ∆
we get
z ( n0 )
g ( n0 ) ∼
= ∼ z ( n0 ). (10.75)
∆
It means that the gradient value can be used as an indicator of the signal
value deviation from the correct value (this property will be later used
for detection of impulsive noise in signal samples as well). For a large
|z(n0 )| > ∆
1
g ( n1 ) ∼
= sign(z(n0 )). (10.76)
2
In that case the gradient assumes correct direction toward minimum posi-
tions, with a deviation independent intensity.
In order to analyze the influence of ∆ to the solution precision, when
z(n0 ) is very small, assume that we have obtained the exact solution and
that the change of sparsity measure is tested on the change of sample x (n0 )
for ±∆. Then for a signal x (n) = ∑iK=1 Ai e j2πn0 ki /N of sparsity K the DFTs of
x+ −
a (n ) = x ( n ) + ∆δ (n − n0 ) and x a ( n ) = x (n ) − ∆δ ( n − n0 ) are
W +W '
K '
WX a W = ' − j2πn0 k i /N '
1 ∑ i
'A + ∆e ' + ( N − K )∆
i =1
W −W K ' '
WX a W = ' − j2πn0 k i /N '
1 ∑ i
'A − ∆e ' + ( N − K )∆.
i =1
For the worst case analysis, assume that Ai are in phase with e− j2πn0 ki /N
and ∆ ≤ | Ai | when
W +W K
WX a W = ∑ | Ai | + K∆ + ( N − K)∆ = µ + N∆
1
i =1
W −W K
WX a W = ∑ | Ai | − K∆ + ( N − K)∆ = µ + ( N − 2K )∆.
1
i =1
where µ = ∥X∥1 . Therefore g(n0 ) = (∥X+ −

a ∥1 − ∥ X a ∥1 ) / (2N∆ ) ̸ = 0. The
correct signal value will not be a stationary state. The algorithm will move
the solution from x (n0 ) to x (n0 ) + b in order to produce g(n0 ) = 0 in the
stationary point. Then ∥X+ −
a ∥1 = µ + N (∆ − b ) is equal to ∥ X a ∥1 = µ0 + ( N −
2K )(∆ + b). It means that the stationary point will be biased. The worst case
bias b follows from
N (∆ − b) = ( N − 2K )(∆ + b) (10.77)
K K
b= ∆∼= ∆ for K ≪ N. (10.78)
N−K N
The bias upper limit can be reduced by using very small ∆. However,
calculation with a small ∆ would be time consuming (with many iterations).
Efficient implementation can be done by using ∆ of an order of signal
amplitude in the initial iteration. When the algorithm reaches a stationary
point, with a given ∆, the value of mean squared error will assume its almost
constant value. The error will be changing the gradient direction around
correct point only, for almost π. This fact may be used as an indicator
to reduce the step ∆, in order to approach the signal true value with a
given precision. For example, if the signal amplitudes are of order of 1 and
K/N = 0.1 taking ∆ = 1 in the first iteration will produce the solution with
a precision better than 20 [dB]. Then, the step ∆ should be reduced, for
example to ∆ = 0.1. A precision better than 40 [dB] would be obtained, and
so on.
Through simulation study it has been concluded that appropriate step
parameter value in (10.74) is related to the finite difference step as α = 2∆.
10.7.2.2 Algorithm
The presented analysis is used as a basic idea for the algorithm summarized
as follows:
(0)
Step 0:) Set m = 0 and form the initial signal estimate x a (n) defined for n ∈
N as
!
(0) 0 for missing samples, n ∈ NQ
x a (n) = , (10.79)
x (n) for available samples, n ∈ M
where N = {0, 1, . . . , N − 1} and NQ = N\M is the complement of M with

respect to N. The initial value for an algorithm parameter ∆ is estimated as
∆ = max | x (n)|. (10.80)
n∈M
(m)
Step 1: Set x p (n) = x a (n). This signal is used in Step 3 in order to estimate
reconstruction precision.
Step 2.1: Set m = m + 1. For each missing sample at ni ∈ NQ form signals
x+ −
a ( n ) and x a (n ):
(m)
x+
a ( n ) = x a (n ) + ∆δ (n − ni )
(m)
x−
a ( n ) = x a (n ) − ∆δ (n − ni ). (10.81)
Step 2.2: Estimate differential of the signal transform measure
∑kN=−01 | Xa+ (k )| − ∑kN=−01 | Xa− (k )|

g ( ni ) = (10.82)
2N∆
where Xa+ (k ) = T { x + − − +
a ( n )} and X a ( k ) = T { x a (n )} are transforms of x a (n )
−
and x a (n).
Step 2.3: Form a gradient vector Gm with the same length as the signal. At
the positions of available samples n ∈ M, this vector has value Gm (n) = 0.
At the positions of missing samples n ∈ NQ its values are Gm (n) = g(n),
calculated by (10.82).
Step 2.4: Correct the values of estimated signal y a (n) iteratively by
(m) ( m −1)
x a (n) = x a (n) − αGm (n), (10.83)
where the step parameter α = 2∆ is commonly used.
Step 2.5: Calculate angle β m between successive gradients as
∑kN=−01 Gm−1 (k ) Gm (k )
β m = arccos F F
N −1 2
G
∑ k =0 m −1 ( k ) ∑kN=−01 Gm
2 (k )
If angle β m is lower than 170◦ and the maximal allowed number of iterations
is not reached (m < mmax ) go to Step 2.1.
Step 3: If the maximal allowed number of iterations is reached stop the
algorithm. Otherwise calculate
(m)
∑n∈NQ | x p (n) − x a (n)|2
Tr = 10 log10 (m)
.
∑n∈NQ | x a (n)|2
Value of Tr is an estimate of the reconstruction error to signal ratio, calcu-

lated for missing samples only. If Tr is above the required precision thresh-
old (for example, if Tr > −100dB), the calculation procedure √ should be re-
peated with smaller ∆. For example, set new ∆ value as ∆/ 10 or ∆/10 and
go to Step 1.
Step 4: Reconstruction with the required precision is obtained in m itera-
tions or the maximal allowed number of iterations is reached. The recon-
(m)
structed signal is x R (n) = x a (n).
By performing presented iterative procedure, the missing values will
converge to the true signal values, producing the minimal concentration
measure in the transformation domain.
- The inputs to the algorithm are the signal length N, the set of
available samples M, the available signal values x (ni ), ni ∈ M, the required
precision Tmax , and maximal number of iterations.
- Instead of calculating signals (10.81) and their transforms for each
ni ∈ N Q we can calculate
' + ' '' (m) '
' Xa (k )' = 'Xa (k ) + ∆Dn (k )''
i
' − ' '' (m) '
' Xa (k )' = 'Xa (k ) − ∆Dn (k )''
i
(m) (m)
with Xa (k) = T { x a (n)} and Dni (k ) = T {δ(n − ni )} = exp(− j2πni k/N ),
for the DFT and each ni ∈ M. Since Dni (k ) are independent of the iteration
number m they can be calculated independently from the DFT of the signal.
n
x (n) = 3 sin(20π )
N
with N = 8. Missing samples are n ∈ N Q = {1, 6}. The signal is reconstructed
using a simplified gradient based algorithm using Step 0 to Step 2.4, from
(10.79) to (10.83), in 60 iterations. The initial algorithm parameter ∆ = 1 and
4
4
3.5 5.5
3.5
4.5
3
2.5
2
5
3.5
4
1.5
4.5
4
1
5.5
0.5
-0.5 4.5
5
6
-1
-4 -3 -2 -1 0 1
Figure 10.25 Illustration of a signal reconstruction using adaptive gradient algorithm.
the initial value of missing samples x (1) = 0 and x (6) = 0 are used. The values
of missing samples in the first 20 iterations are presented by dots (connected
by a line) in Fig.10.25. After about 6 iterations the algorithm with ∆ = 1
does not significantly change the missing sample values (zoomed changes
are shown in lower subplot within the figure). Close to the stationary point
obtained for ∆ = 1 the gradient coordinates are almost zero-valued (with
direction changes for almost π), since the measures are on the contour with
almost the same measure (circles). After the step is reduced to ∆ = 0.1 in
the 20th iteration, the algorithm resumes its fast approach toward the exact
value, until a new stationary state. With a new change of ∆ to ∆ = 0.01 the
approach is again continued.
K
The stationary state bias for ∆ = 1 is lower than N ∆ = 1/4 (it corre-
sponds to the bias caused MSE lower than 15.5 [dB]). By each reduction of ∆
to ∆/10 the bias caused MSE will be lower for 20 [dB]. The reconstruction re-
sult and the MSE for the estimated missing values x (1) and x (6) is presented
in Fig.10.26.
The calculation is repeated with the signal
n n n
x (n) = 3 sin(20π ) + 2 cos(60π ) + 0.5 sin(46π )
N N N
and N = 32. Missing samples are n ∈ N Q = {2, 4, 5, 7, 9, 13, 17, 19, 24, 26, 28, 31}.
The result for this case is shown in Fig.10.27.
Original signal x(n) Available samples

3 3
0 0
-3 -3
0 2 4 6 8 0 2 4 6 8
Reconstructed signal in 60 iteratons Reconstruted MSE in [dB]
0
3
-20
0 -40
-60
-3
-80
0 2 4 6 8 0 20 40 60
time n iteration
Figure 10.26 Gradient-based reconstruction of a sparse signal.
10.7.2.3 Comments on the Algorithm
- In a gradient-based algorithm, a possible divergence is related to the

algorithm behavior for large steps ∆. Small steps influence the rate of the
algorithm approach to the solution only (assuming that it exists). Here, we
will examine the algorithm behavior for a large value of step ∆. We can write
' + ' ' − ' '' (m) ' ' '
' Xa (k )' − ' Xa (k )' = 'Xa (k ) + ∆Dn (k )'' − ''Xa(m) (k ) − ∆Dn (k )''
i i
(' ' ' ')
' (m) '
Xa (k ) ' ' ' Xa (k ) ''
(m)
'
= ∆ | Dni (k)| '1 + ' − '1 − ' .
' ∆Dni (k ) ' ' ∆Dni (k ) '
(m)
Considering the complex number a = Xa (k )/(∆Dni (k )), with | a| ≪ 1 for
a large ∆, from the problem geometry it is easy to show that the following
bounds hold 0 ≤ ||1 + a| − |1 − a|| ≤ 2 | a| . Exact value of this expression
depends on the phase of a. Therefore,
'' ' ' '' ' '
' (m) '
0 ≤ '' Xa+ (k )' − ' Xa− (k )'' ≤ 2 'Xa (k )' .
Lower
' ' 0 is obtained if a is imaginary-valued, while the upper limit
limit
' (m) '
2 'Xa (k )' follows if a is real-valued.
It means that the value of the finite difference | Xa+ (k )| − | Xa− (k )| , that
is used to correct the missing signal samples, does not depend on the value
of the step ∆ if ∆ is large. The missing signal values will be adapted for
Original signal x(n) Available samples
4 4
2 2
0 0
-2 -2
-4 -4
0 10 20 30 0 10 20 30
Reconstructed signal in 5 iteratons Reconstructed signal in 15 iteratons
4 4
2 2
0 0
-2 -2
-4 -4
0 10 20 30 0 10 20 30
Reconstructed signal in 60 iteratons Reconstruted MSE in [dB]
0
4
-20
2
0 -40
-2
-60
-4
-80
0 10 20 30 0 20 40 60
time n iteration
Figure 10.27 Gradient-based reconstruction of a sparse signal.
a value independent on ∆ in that case. The values of missing'samples' will

' (m) '
oscillate within the range of the original signal values of order 'Xa (k )' /N,
until ∆ is reduced in the iterations below the signal magnitude. Then the
missing samples will start approaching to the position of the sparsity mea-
sure minimum. The initial values will be arbitrary changed within the signal
amplitude order as far as ∆ is too large. It will not influence further conver-
gence of the algorithm, when the step ∆ assumes appropriate values.
- Since two successive gradient vectors are required to calculate the
gradient angle β m , it is calculated starting from the second iteration for each
∆.
- Algorithm output is the reconstructed signal x R (n), n = 0, 1, ..., N − 1.
- Other signal transforms can be used instead of the DFT. The only re-
quirement is that signal is sparse in that transform domain (two-dimensional
DCT will be presented later).

K/2
x (t) = ∑ Ai cos(2πtki /T + ϕi ), (10.84)
i =1
with t = n∆t, ∆t = 1, and the total number of samples N = T/∆t. The

sparsity parameter K is changed from K = 2 to K = N/2. The amplitudes
Ai , frequencies k i , and phases ϕi are taken randomly. Amplitude values
are modeled as Gaussian random variables with variance 1, the frequency
indices assume random numbers within 1 ≤ k i ≤ N − 1, and the phases
assume uniform random values within 0 ≤ φi ≤ 2π, in each realization. The
reconstruction is performed by using 100 realizations for each K with random
sets of missing Q = N − M samples in each realization. The reconstructed
signals x R (n) are obtained. The results are presented in Fig.10.28 in a form of
the signal-to-reconstruction-error ratio (SRR) in [dB]
2
∑nN=−01 | x (n)|
SRR = 10 log 2
. (10.85)
∑nN=−01 | x (n) − x R (n)|
Bright colors indicate the region where the algorithm had fully recovered
missing samples in all realizations, while dark colors indicate the region
where the algorithm could not recover missing samples in any realization.
In the transition region for M slightly greater than 2K we have cases when
the signal recovery is not achieved and the cases of full signal recovery. The
simulations are done for N = 128 and for N = 64, Fig.10.28(a),(b). A stopping
criterion for the accuracy of 120 [dB] is used. It corresponds to a precision
in the recovered signal of an input samples precision if they are acquired by
a 20-bit A/D converter. The case with N = 64 is repeated with an additive
input Gaussian noise such that the input signal-to-noise ratio is 20 [dB] in
each realization Fig.10.28(c). The reconstruction error in this case is limited
by the input signal-to-noise value. The number of iterations to achieve the
required precision is presented in Fig.10.28(d). We can see that the number
of iterations is well bellow 100 for the most important region where the
reconstruction was achieved in all realizations (high values of M and small
value of K, M ≫ K).The number of iterations is quite small in the region
where the reconstruction can be achieved.
An illustration of the algorithm performance regarding to the SRR and
the gradient angle β m in one realization, with K = 6, is presented in Fig.10.29.
The algorithm reached 120 [dB] accuracy in 47 iterations. From√ the gradient
angle graph we see that the algorithm step is reduced to ∆/ 10 → ∆ in about
each 4 iterations. According √ to (10.77) the expected MSE improvement by
each reduction of ∆ is 20 log( 10) = 10 [dB].
Signal-to-reconstruction error [dB] Signal-to-reconstruction error [dB]
60 30
120 120
50 25
100 100
sparsity K
sparsity K
40 20 80
80
30 60 15 60
20 40 10 40
10 20 5 20
0 0
20 40 60 80 100 120 20 40 60
(a) available samples M (b) available samples M
Signal-to-noise ratio [dB] Number of iterations [n]
30 30 400
15
25 25
300
sparsity K
sparsity K
20 20
10
15 15 200
10 5 10
100
5 5
0
20 40 60 20 40 60
(c) available samples M (d) available samples M
Figure 10.28 Signal-to-reconstruction-error (SRR) averaged over 100 realizations for various
sparsity K and number of available samples M: (a) The total number of samples is N = 128. (b)
The total number of samples is N = 64. (c) With a Gaussian noise in the input signal, SNR = 20
[dB] and N = 64. (d) Number of iterations to reach the solution with the defined precision.
10.8 ON THE UNIQUENESS OF THE DFT OF SPARSE SIGNALS
In general, the reconstructed signal uniqueness is guarantied if the restricted

isometry property is used and checked with appropriate isometry constants.
However, two problems exist in the implementation of this approach. For
a specific measurement matrix it produces quite conservative bounds. In
practice it would produce a large number of false alarms for nonuniqueness.
In addition, uniqueness check with the restricted isometry property requires
a combinatorial approach, which is an NP hard problem.
successive gradient angles

200
βm
150
100
50
0
0 10 20 30 40 50
iteration
100
SSR [dB]
50
0 10 20 30 40 50
iteration
Figure 10.29 Angle between successive gradient estimations β m and the signal-to-
reconstruction-error ratio (SRR) as a function of the number of iterations in the algorithm for
one signal realization with 6 nonzero DFT coefficients and M = 64.
In the adaptive gradient-based method the missing samples (measure-

ments) are considered as the minimization variables. The available samples
values are known and fixed. The number of variables in the minimization
process is equal to the number of missing samples/measurements in the ob-
servation domain. This approach is possible when the common signal trans-
forms are the domains of signal sparsity. Then the missing and available
samples/measurements form a complete set of samples/measurements.
The DFT will be considered here as the signal sparsity domain. The
solution uniqueness is defined in the sense that the variation of the missing
sample values cannot produce another signal of the same sparsity. In the
case when the signal is already reconstructed then the uniqueness is checked
in the sense that there is no other signal of the same or lower sparsity with
the same set of available samples.
Consider a signal x (n) with n ∈ N={0, 1, 2, ...., N − 1}. Assume that
Q of its samples at the positions qm ∈ N Q = {q1 , q2 , ...., q Q } are miss-
ing/omitted. The signal is sparse in the DFT domain, with sparsity K. The
reconstruction goal is to get x (n), for all n ∈ N using available samples at
n ∈ M = N\NQ . A new signal of the form
x a (n) = x (n) + z(n)
will be analyzed here. For the available signal positions n ∈ M the value of
z(n) is fixed z(n) = 0, while z(n) may take arbitrary value at the positions of
missing samples n = qm ∈ NQ = {q1 , q2 , ...., q Q }. If x (n) is a K sparse signal
then the DFT of x a (n) is
Xa (k ) = X (k ) + Z (k )
K Q
= N ∑ Ai δ(k − k0i ) + ∑ z(qm )e− j2πqm k/N .
i =1 m =1
Positions of nonzero values in X (k ) are k0i ∈ K = {k01 , k02 , ...., k0K } with
amplitudes X (k0i ) = N Ai . The values of missing samples of x a (n) = x (n) +
z(n) for n ∈ NQ are considered as variables. The goal of reconstruction
process is to get x a (n) = x (n), or z(n) = 0 for all n ∈ N. This goal should be
achieved by minimizing a sparsity measure of the signal transform Xa (k ).
Existence of the unique solution of this problem depends on the number of
missing samples, their positions, and the signal form.
If a signal with the transform X (k ) of sparsity K is obtained using a re-
construction method, with a set of missing samples, then the reconstruction
X (k ) is unique if there is no other signal of the same or lower sparsity that
satisfies the same set of available samples (using the same set of missing
samples as variables).
Example 10.25. Consider the simplest case of one missing sample at position
n = q. The signal sparsity is K. Signal reconstruction is based on x a (n) =
x (n) + zδ(n − q) where z indicates an arbitrary deviation from the true signal
value, since the missing sample x (q) is considered as variable. The DFT of
x a (n) is
K
Xa (k ) = N ∑ Ai δ(k − k0i ) + ze− j2πkq/N .
i =1
The number of nonzero DFT coefficients is
' K '0 N
' − j2πk0i q/N '
card{X a } = ∥X a ∥0 = ∑ 'N A i + ze ' + ∑ | z |0
i =1 i = K +1
Possible sparsity of Xa (k ) is
⎧
⎪
⎪ N for |z| ̸= 0 and z ̸= − N Ai e j2πk0i q/N for any i
⎪
⎪
⎨ N − 1 for |z| ̸= 0 and z = − N Ai e j2πk0i q/N for one i only
∥ X a ∥0 = ... ... ...
⎪
⎪ j2πk0i q/N for i = 1, .., K
⎪
⎪ N − K for | z | ̸ = 0 and z = − N A ie
⎩
K for |z| = 0.
(10.86)
With just one missing value and arbitrary signal, the minimum of ∥X a ∥0 is
achieved at |z| = 0 only if the signal sparsity is lower than the lowest possible
sparsity with |z| ̸= 0,
K < N − K.
It means K < N/2. For K = N/2 the last two rows of (10.86) will produce the
same result N − K = N/2 and K = N/2. In that case the minimum of ∥X a ∥0
is not unique. Note that this is true only if the considered signal x (n) has a
very specific form
A1 e j2πk01 q/N = A2 e j2πk02 q/N = A3 e j2πk03 q/N = ... = AK e j2πk0K q/N = C. (10.87)
In reality the case that all components have equal amplitudes | A1 | = | A2 | =

| A3 | = ... = | AK | and that the missing sample position q is such that
arg { A1 } + 2πk01 q/N = arg { A2 } + 2πk02 q/N = ... = arg { AK } + 2πk0K q/N
(10.88)
is a zero probability event.
It is interesting to note that if the last two conditions are satisfied by a
signal x (n) then the DFT coefficients from (10.87) are the frequency domain
samples of a harmonic signal B exp(e j2πkq/N ), at k ∈ {k01 , k02 , ...., k0s }. Its IDFT
is a delta pulse with the group delay at the position of missing sample
IDFT{ B exp(e j2πkq/N )} = Bδ(n − q).
Example 10.26. Consider a signal x (n) with N = 32 and two missing samples at
qm ∈ N Q = {3, 19}.
Signal sparsity is K. In order to simplify the notation assume that one DFT
value of the reconstructed signal is X (5) = 2.
(a) Show that the limit for sparsity K (when we can claim that the
reconstructed sparse signal is unique, assuming that all signal amplitudes
may assume arbitrary values) is K < 8.
(b) What are the properties that a signal must satisfy in the limit case
K = 8 so that the solution is not unique?
(c) What is the sparsity limit if the missing samples are at qm ∈ N Q =
{5, 9}?
(d) Repeat the analysis for qm ∈ N Q = {4, 7}.

(e) What is the sparsity limit if the missing samples are qm ∈ N Q =
{3, 4, 19}?
⋆(a) Consider Xa (k) = X (k) + Z (k) with z(n) assuming nonzero val-
ues z3 and z19 at n = 3 and n = 19, respectively. The DFT Z (k ) of z(n) is equal
to
B C
Z (k) = z3 e− j2π3k/32 + z19 e− j2π19k/32 = e− j2π3k/32 z3 + (−1)k z19
k = 0, 1, ..., 31.
In the worst case for the minimization Z (k ) should have maximal possible
number of zeros and they should remain in Xa (k) = X (k ) + Z (k ). We con-
clude that either z3 = z19 or z3 = −z19 should hold (when sparsity of Z (k) is
16), otherwise the sparsity of Z (k ) would be 32. In addition, in the worst case
nonzero values of Z (k ) could cancel out all K components including assumed
X (5) = 2. Therefore the maximal number of zeros in Xa (k ) with nonzero z(n)
is 16 + K. The sparsity of Xa (k ) is then 32 − (16 + K ). It should be greater than
the sparsity K of the correct solution when all z(n) = 0 and Xa (k) = X (k ). It
means
32 − (16 + K ) > K
should hold. This completes the proof that K < 8 should hold.
(b) Since z3 = z19 would produce Z (2k + 1) = 0 it would not be able
to cancel X (5). Therefore for the worst case analysis we must use z3 = −z19
with
Z (5) = e− j2π15/32 (z3 − z19 ) = − X (5) = −2.
It means z3 = −z19 = −e j2π15/32 and
%
−2e− j2π (3k−15)/32 for odd k
Z (k) =
0 for even k.
In order to cancel all nonzero values of X (k ), all of them must be located at

odd positions (where Z (k ) is nonzero)
X (k) ̸= 0 for k ∈ {5, k02 , k03 , k04 , k05 , k06 , k07 , k08 }.
The values of X (k) must be of opposite sign and equal amplitude to the
corresponding (determined) values of Z (k )
X (k0i ) = − Z (k0i ) = 2e− j2π (3k0i −15)/32 for i = 2, 3, ..., 8
resulting in
%
2e− j2π (3k−15)/32 for k ∈ {5, k02 , k03 , k04 , k05 , k06 , k07 , k08 }
X (k ) = (10.89)
0 elsewhere.
In this case sparsity of X (k) + Z (k ) is 8, the same as the sparsity of X (k ).

Two solutions of our minimization problem are signal x (n) = IDFT[ X (k )] and
x (n) + z(n) where
z(n) = IDFT[ Z (k )] = δ(n − 3) − δ(n − 19).
Both of these signals have the same sparsity K = 8 and satisfy the same set
of available samples. However, if the sampled signal x (n) is not the signal of
very specific from (10.89) then the solution of sparsity K = 8 will be unique
for a given set of available samples. Then z(n) = δ(n − 3) − δ(n − 19) will not
be able to cancel all 8 DFT values of signal and the sparsity of X (k ) + Z (k )
will be 8 only for z(n) = 0, producing correct unique solution. Signal Y (k ) =
− Z (2k − 1) is Y (k) = 2e− j2π (3(2k−1)−15)/32 = 2e− j2π (3k−9)/16 . It is periodic
with period N/Q = 16. Group delay of this signal is n0 = 3 with period 16.
Therefore within n = 0, 1, ..., 31 group delays n0 = 3 and n0 + 16 = 19 of Y (k )
correspond to the missing sample positions. The signal must have the form
X (k0m ) ∈ {2e− j2π (3k−9)/16 | k = 0, 1, . . . , N
Q − 1}, with k = 3 corresponding to
k0m = 2k − 1 = 5 producing X (5) = 2.
(c) Influence of missing samples highly depends on their positions. If
the missing samples are at qm ∈ N Q = {5, 9} then
B C
Z (k) = z5 e− j2π5k/32 + z9 e− j2π9k/32 = e− j2π5k/32 z5 + e− j2πk/8 z9 .
Maximal number of zero values in Z (k ) is now 4. Assuming that nonzero

values can cancel out all signal nonzero samples, maximal number of zeros
in X (k ) + Z (k ) is 4 + K with sparsity ( N − K − 4). It should be greater than
the signal sparsity K. Thus in this case uniqueness condition is K > 28 − K,
or K > 18.
(d) In the case of qm ∈ NQ = {4, 7} two nonzero variables z(4) and z(7)
can produce only one zero value in Z (k) since
Z (k) = z4 e− j2π4k/32 + z7 e− j2π7k/32 = 0
for z4 = −z7 e− j2π3k/32 . In addition, all K signal nonzero values X (k) can be
cancel out. Then the uniqueness relation is N − 1 − K > K.
(e) If the missing samples are qm ∈ N Q = {3, 4, 19} then this case may
be considered as a case with three variables producing two nonzero values
in Z (k ), but also it can be considered as {3, 19} ∪ {4}, when z(4) = 0 and two
variables z(3) and z(19) define sparsity as in (a). The second case is worse,
meaning that it defines the resulting sparsity K < 8.
The analysis presented in the previous two examples can be general-

ized, taking into account the positions and number of missing samples, to a
simple uniqueness test as follows:
Test 1: Consider a signal x (n) that is sparse in the DFT domain with
unknown sparsity. Assume that the signal length is N = 2r samples and that Q
samples are missing at the instants q m ∈ N Q . Assume that the reconstruction

is performed and that the DFT of reconstructed signal is of sparsity K. The
reconstruction result is unique if the inequality
M N
K < N − max 2h ( Q2h − 1 ) − K
h=0,1,...,r −1
holds. Integers Q2h are calculated as
Q2h = max {card{q : q ∈ NQ and mod(q, 2h ) = b}} (10.90)

b=0,1,...,2h −1
Example 10.27. Consider a signal with N = 25 = 32 and Q = 9 missing samples at

qm ∈ N Q = {2, 3, 8, 13, 19, 22, 23, 28, 30}.
Using Test 1 we will find the sparsity limit K when we are able to claim that
the reconstructed sparse signal is unique for any signal form.
-For h = 0 we use Q20 = Q and get 20 ( Q20 − 1) − 1 = ( Q − 1) − 1 = 9.
-For h = 1, the number Q21 is the greater value of
card{q : q ∈ N Q and mod(q, 2) = 0} = card{2, 8, 22, 28, 30} = 5

R S
card q : q ∈ N Q and mod(q, 2) = 1 = card{3, 13, 19, 23} = 4,
i.e., the maximal number of even or odd positions of missing samples. Thus
Q21 = max {5, 4} = 5 with 21 ( Q21 − 1) = 8.
-Next Q22 is calculated as the maximal number of missing samples
whose distance is multiple of 4. For various initial counting positions b =
0, 1, 2, 3 the numbers of missing samples with distance being multiple of 4 are
2, 1, 3, and 3, respectively. Then Q22 = max {2, 1, 3, 3} = 3 with 22 ( Q2h − 1) = 8.
-For Q23 the number of missing samples at distances being multiple
of 8 are found for various b = 0, 1, 2, 3, 4, 5, 6, 7. The value of Q23 is 2 with
23 ( Q23 − 1) = 8.
-Finally we have two samples at distance 16 (samples at the positions
q2 = 3 and q5 = q2 + N/2) producing Q24 = Q16 = 2 with 24 (2 − 1) = 16.
The reconstructed signal of sparsity K is unique if
M N
K < N − max 2 h ( Q2h − 1) − K
h=0,1,2,3,4
K < 32 − max {9, 8, 8, 8, 16} − K
K < 32 − 16 − K
or
K < 8.
Test 1 considers general signal form. It includes the case when the
amplitudes of signal components are related to each other and related to
the missing sample positions. The specific signal form required by Test 1, to
reach its bound, is analyzed in the example. Since this kind of relation is a
zero-probability event, the condition obtained by neglecting the probability
that the signal values are dependent to each other and related to missing
sample positions at the same time is presented next.
C1: Assume that the amplitudes of signal components in Test 1 are arbitrary
with arbitrary phases so that the case when all of them can be related to the values
defined by using the missing sample positions is a zero-probability event. The
reconstruction result is not unique if the inequality
M N
K ≥ N − max 2 h ( Q2h − 1 ) − 1
h=0,1,...,r −1
holds. Integers Q2h are calculated in the same way as in the Test1.
Example 10.28. Consider a signal with N = 25 = 32 and Q = 9 missing samples at
qm ∈ N Q = {2, 3, 8, 13, 19, 22, 23, 28, 30}.
The sparsity limit K when we are able to claim that the reconstructed sparse
signal is not unique is
M N
K ≥ N − max 2 h ( Q 2h − 1 ) − 1
h=0,1,2,3,4
K ≥ 32 − max {9, 8, 8, 8, 16} − 1
K ≥ 15.
Corollary C1 provides the uniqueness test for the given positions

of unavailable samples. In the cases with h > 0 it exploits the periodic
structure of the transformation matrix of missing samples. The periodical
form assumes that the positions of possible zero values in Z (k ) do not
interfere with the signal nonzero value positions. This is possible in the
worst case analysis.
Test 2: Consider a signal x (n) that is sparse in the DFT domain with
unknown sparsity. Assume that the signal length is N = 2r samples and that Q
samples are missing at the instants q m ∈ NQ . Also assume that the reconstruction
is performed and that the DFT of reconstructed signal is of sparsity K. Assume
that the positions of the reconstructed nonzero values in the DFT are k 0i ∈ K =
{k01 , k02 , ...., k0s } Reconstruction result is unique if the inequality
M N
K < N − max 2h ( Q2h − 1) − K + 2S2r−h
h=0,1,...,r −1
holds. Integers Q2h and S2r−h are calculated as
Q2h = max {card{q : q ∈ NQ and mod(q, 2h ) = b}}

b=0,1,...,2h −1
Q2h −1
S 2r − h = ∑ Ph (l )
l =1
Ph (l ) = sort {card{k : k ∈ K and mod(k, 2r−h ) = b}}
b=0,1,...,2r−h −1
where Ph (1) ≤ Ph (2) ≤ ... ≤ Ph (2r−h ).

Note: For S2r−h = 0 this Test reduces to the Test 1. For the DFT values
equally distributed over all positions this Test produces result close to
K ≥ N − Q.
C2: Assume that the positions of the reconstructed nonzero values in the DFT
are k0i ∈ K = {k01 , k02 , ...., k0s }. Assume that the amplitudes of signal components
Test 2 are arbitrary with arbitrary phases so that the case when all of them can
be related to the values defined by using the missing sample positions is a zero-
probability event. Reconstruction result is not unique if the inequality
M N
K≥N− max 2 h ( Q 2 h − 1 ) − 1 + S 2r − h
h=0,1,...,r −1
holds. Integers Q2h and S2r−h are calculated as in the Test 2. The case when all of
signal components can be related to the values defined by using the missing sample
positions is considered here.
Example 10.29. Consider a signal with N = 32 and Q = 9 missing samples at
qm ∈ N Q = {2, 3, 8, 13, 19, 22, 23, 28, 30}.
Assume that with these missing samples we have reconstructed signals with
nonzero DFT values at the positions
a) K = {1, 3, 5, 7, 9, 11, 13, 15, 17, 21, 23, 25, 27, 29, 31},
b) K = {1, 3, 5, 9, 13, 17, 21, 29, 31, 2, 4, 8, 12, 16, 20, 24, 30}.
By testing these two signals we get the following decisions. According to

Test 1 we cannot claim uniqueness in either of these cases since K = 15
in the first case and K = 17 in the second case. Both are greater than the
Test1 bound K < 8. The same holds for Corollary C1 since both are K ≥ 15.
By testing these results with Test 2 we get that in case a) the solution is
nonunique. It is due to very specific form of the reconstructed signal with all
components being found at the odd frequency positions. Since the sparsity
was defined by periodicity 16 in qm ∈ NQ , then variations of two signal
samples z(q2 = 3) and z(q5 = 19) can produce a signal X (k ) + Z (k ) with

lower sparsity than the reconstructed signal. These two samples, as variables,
are able to produce many ( N/2) zero values in Z (k) either at odd or even
positions in frequency. In this case they are at even positions of X (k ) + Z (k ).
However, in signal b) that is not the case. Nonzero values are distributed
over both even and odd frequency positions. Although sparsity of this signal
is K = 17 the reconstruction is unique. The distribution of nonzero values in
the reconstructed X (k ) is such that by varying two samples z(q2 = 3) and
z(q5 = 19) we cannot produce a signal X (k ) + Z (k) of lower sparsity with
nonzero z(q2 = 3) and z(q5 = 19). The limit in this case is defined by the
lower periodicity in z(q) than N/2. Thus, if we obtain this signal using a
reconstruction algorithm the solution is unique.
Example 10.30. Consider a signal with N = 1024 and Q = 512 missing samples at
qm ∈ N Q = {0, 2, 4, ...1022}. The reconstructed signal is at the frequencies: a)
K = {3}, b) K = {3, 515}. We can easily check that in all cases with Test 1,
Corollary C1 and Test 2, the reconstruction is nonunique although K = 1 or
K = 2 is much smaller than the available number of samples N − Q = 512. The
answer is obtained almost immediately, since the computational complexity
of Test 1, Corollary C1 and Test 2, is of order O( N ).
10.9 INDIRECT MEASUREMENTS/SAMPLING
In some applications a signal x (n) is measured in an indirect way. Consider

the case when each measurement f (n) is a linear combination of all signal
samples
f (i ) = bi1 x (0) + bi2 x (1) + ... + biN x ( N − 1) (10.91)
with i = 0, 1, ..., M − 1. In this case the reconstructed signal is x =[ x (0) x (1) ...
x ( N − 1)] T or its sparse transform X. Matrix relation between signal samples
x (n) and M indirect measurements f (i ) is
f = BM x
where elements of vector f are f (i ) and the elements of M × N matrix B M

are bin . For the transformation x = ΨX the sparsity domain to measurements
domain relation is
f = B M ΨX.
Minimization problem is defined by
min ∥X∥1 subject to f = B M ΨX = AX,

where
A = B M Ψ.
As a simple study case for this kind of measurements consider a
discrete-time signal x (n) obtained by sampling a continuous-time signal
x (t) at nonuniform (or random) positions. Using the results presented in
this chapter we can state that if the signal x (t) satisfies the sampling theorem
and its DFT is sparse, then the signal can be reconstructed from a reduced set
of samples x (ti ) at {t1 , t2 , ..., t M } not corresponding to the sampling theorem
positions.
Since the DFT is used in the analysis, we can assume that the con-
tinuous time signal is periodically extended with a period T. According to
the sampling theorem, the period T is related to the number of samples N,
the sampling interval ∆t, and the maximal frequency Ωm as Ωm = π/∆t =
πN/T. The continuous-time signal can be written as an inverse Fourier se-
ries
N/2−1
x (t) = ∑ Xk e j2πkt/T , (10.92)
k=− N/2
with the Fourier series coefficients being related to the DFT as Xk N = X (k ) =

DFT[ x (n)] and x (n) = x (n∆t). The discrete-time index n corresponds to
the continuous-time instant t = n∆t. Discrete-frequency indices are k ∈
{− N/2, ..., −1, 0, 1, ..., N/2 − 1}. Any signal value can be reconstructed from
the samples taken according to the sampling theorem, 3.6
N −1 t
sin[(n − ∆t )π ]
x (t) = ∑ x (n)e j(n−t/∆t)π/N t
. (10.93)
n =0 N sin[(n − ∆t )π/N ]
This relation holds for an even N. Similar relation can be written for an odd
N, Section 3.6.
For a sparse x (n) in the DFT domain, the number K of nonzero
transform coefficients X (k ) is much lower than the number of the original
signal samples N within T, K ≪ N, i.e., X (k ) = NXk = 0 for k ∈ / {k1 , k2 , ...,
k K }. A signal
K
x (t) = ∑ Xki e j2πki t/T . (10.94)
i =1
of sparsity K can be reconstructed from a reduced set of M samples if the

recovery conditions are met.
Consider a random set of possible nonuniform sampling instants
{t1 , t2 , ..., t N },
ti = i∆t + νi , (10.95)
where, for example, νi is a uniform random variable −∆t/2 ≤ νi ≤ ∆t/2.

Here tni denotes a time instant, while in the uniform sampling the discrete-
time index ni has been used to indicate instant corresponding to ni ∆t.
Assume that a set of M signal samples are available
f =[ x (tn1 ), x (tn2 ), ..., x (tn M )] T
at instants
tni ∈ T A = {tn1 , tn2 , ..., tn M }.
being a random subset of {t1 , t2 , ..., t N }, with tni = ni ∆t + νni . The measure-
ments matrix relation is, (10.92)
⎡ ⎤ ⎡ − j2πNtn /(2T ) ⎤⎡ ⎤
x ( t n1 ) e 1 ... e j2π ( N −2)tn1 /(2T ) X− N/2
⎢ x (tn ) ⎥ ⎢ e− j2πNtn2 /(2T ) ... e j2π ( N −2)tn2 /(2T ) ⎥ ⎢
⎥
X− N/2+1 ⎥
⎢ 2 ⎥=⎢
⎢ ⎥⎢ ⎥
⎣ ... ⎦ ⎣ ... ... ... ⎦ ⎣ ... ⎦
x (tn M ) e − j2πNtn M /(2T )
... e j2π ( N −2)tn M /(2T ) X N/2−1
(10.96)
f= AX
The analysis presented in this chapter can be used to solve this problem and
calculate sparse coefficients Xk from the reduced set of observations f. The
measurements matrix in this case is a structured random matrix.
The nonzero positions of the Fourier transform coefficients can be
estimated using the available measurements only
X0 = A H f or
X0 (k ) = NX0,k = ∑ x (tni )e− j2πktni /T (10.97)
t ni ∈T A
Note that a sparse signal X (k ) with components at k ∈ / {k1 , k2 , ..., k K } can

be written in form (10.94). For a frequency k = k p and the signal component
Xk p exp( j2πk p t/T ) all terms in (10.97) will be the same
'
'
Xk p e j2πk p tni /T e− j2πktni /T ' = Xk p .
k=k p
Therefore, the mean value of estimator (10.97), using M instants tni , is
K
E{ X (k )} = M ∑ X k p δ ( k − k p ).
p =1
The variance of this estimator is different from the case when the avail-
able signal samples were at the sampling theorem positions. The condition
that a value of the DFT coefficient at k ̸= k p is zero (with zero variance) if
M = N samples are used, does not hold any more. The total variance can be
estimated as a simple sum of variances
K D E
var{ X (k )} = ∑ Xk2p M 1 − δ(k − k p ) . (10.98)
p =1
For small M we have ( N − M )/( N − 1) ∼ = 1 and expressions (10.49) and

(10.98) produce similar results.
In reconstruction we may use the estimated maxima of X (k ) obtained
from (10.97). If K positions of nonzero coefficients are correctly estimated
then their values (elements of vector XK ) follow from (10.96) as
f= AK XK
B C −1 B C −1
XK = AKH AK AKH f = AKH AK X0K .
Example 10.31. Some of the random realizations of the initial DFT (10.97) for
signal (10.55) are given in Fig.10.30. In contrast to the partial DFT matrix case,
the variance of the estimator (10.97) does not tend to zero as M approaches
to N. However, we can see that the signal frequencies can be detected and
used to recover the signal using (10.37) and (10.39) with known time instants
ti ∈ {tn1 , tn2 , ..., tn M } and detected frequencies {k1 , k2 , ..., k K }.
The results for several random realization and nonuniform sampling of
signal (10.55), with recalculated signal values at the sampling theorem posi-
tions, are shown in Fig.10.31. As the number of available samples approaches
to the total number of samples N the reconstructed DFT is again noise-free,
Fig.10.31. For the signal defined by (10.55) the variance of initial DFT is cal-
culated in 100 random realizations of the sets of available samples for the
cases of when the signal is sampled according to the sampling theorem and
for nonuniform sampling without and with recalculation. The results for the
variance is presented in Fig. 10.32. From Fig.10.32 we can conclude that the
recalculation is not efficient for a small number of available samples, when
M ≪ N. In that case even slightly worse results are obtained than without re-
calculation, what could be expected, since the recalculated signal with many
inserted zeros is not sparse any more. For a large number of available sam-
ples (in Fig.10.32 for M > 5N/8) the recalculation produces better results,
approaching to the sparse signal without any deviation, for N = M.
The problem with indirect measurements can also be reformulated

using the DFT framework results. If the signal values were available at
16 64 128
M=16 M=64 M=128
12 48 96
8 32 64
4 16 32
0 0 0
1 128 257 1 128 257 1 128 257
signal transform
192 224 256
M=192 M=224 M=257
144 168 192
96 112 128
48 56 64
0 0 0
1 128 257 1 128 257 1 128 257
frequency
Figure 10.30 DFT of a signal with various number of available samples M. Available M
samples are taken at random positions within 0 ≤ ti ≤ T. Dots represent the original signal
DFT values, scaled with M/N to match the mean value of the DFT calculated using a reduced
set of samples signal.
ti ∈ T A for M = N the signal values at the sampling theorem positions

could be recovered from this set of available samples, denoted by f N . The
transformation matrix relating samples taken at ti with the signal values at
the sampling theorem positions, according to (10.93), is
⎡ ⎤ ⎡ ⎤⎡ ⎤
x ( t1 ) b11 b12 ... b1N x (0 )
⎢ x (t2 ) ⎥ ⎢ b21 b22 ... b2N ⎥ ⎢ x (1 ) ⎥
⎢ ⎥ ⎢ ⎥⎢ ⎥
⎣ ... ⎦ = ⎣ ... ... ... ... ⎦ ⎣ ... ⎦
x (t N ) b N1 b N2 ... b NN x ( N − 1)
fN = BN x
and
x = B− 1
N fN . (10.99)
with
sin[( j − tni /∆t)π ]
bij = e j( j−tni /∆t)π/N
N sin[( j − tni /∆t)π/N ]
If a reduced set of available samples is used we know just M < N
of signal samples/measurements (10.91). Each available sample is a linear
16 64 128
M=16 M=64 M=128
12 48 96
8 32 64
4 16 32
0 0 0
1 128 257 1 128 257 1 128 257
signal transform
192 224 256
M=192 M=224 M=257
144 168 192
96 112 128
48 56 64
0 0 0
1 128 257 1 128 257 1 128 257
frequency
Figure 10.31 DFT of a signal with various number of available samples M. Available M
samples are a random subset of N nonuniform samples taken at random positions within the
sampling theorem interval. Dots represent the original signal DFT values, scaled with M/N to
match the mean value of the DFT calculated using a reduced set of samples signal.
combination of all signal samples taken at the sampling theorem rate
f (i ) = x (tni ) = bi1 x (0) + bi2 x (1) + ... + biN x ( N − 1)

tni ∈ T A = {tn1 , tn2 , ..., tn M }.
For the initial calculation the values at unavailable positions ti ∈

/ T A are
assumed to be zero
yc =[ x (tn M+1 ), x (tn M+2 ), ..., x (tn N )].
Their positions are assumed at the sampling theorem instants, tni = ni ∆t for
t ni ∈
/ T A , since they are not known anyway,
ni ∆t = tni ∈
/ TA.
An illustration for N = 8 is presented in Fig.10.33. The available samples are

f =[ x (t0 ) x (t2 ) x (t3 ) x (t4 ) x (t6 ) x (t7 )] T with assumed unavailable samples
yc = [ x (1) x (5)]. They form a complete observation vector for the initial
450
400
350
300
variance
250
200
150
100
50
0
64 128 192 256
number of available samples
Figure 10.32 Variance of the DFT for three methods of sampling and various number of
available samples M. (1)-line with marks "x": Available samples a subset of samples taken
at the sampling theorem grid (solid line-theory, marks "x"-statistics). (2)-line with marks
"o": Randomly positioned M samples taken within 0 ≤ ti ≤ T (solid line-theory, marks "o"-
statistics). (3)-marks "+": Nonuniform randomly shifted samples from the sampling theorem
grid. (4)-marks "*": Nonuniform randomly shifted available samples being recalculated on the
sampling theorem grid.
iteration
(0)
x a = [ x (t0 ) x (1) x (t2 ) x (t3 ) x (t4 ) x (5) x (t6 ) x (t7 )]T
= [ x (t0 ) 0 x (t2 ) x (t3 ) x (t4 ) 0 x (t6 ) x (t7 )]T .
Using the recalculation procedure, the problem can be solved using

the gradient algorithm as well.
The missing samples yc =[ x (n M+1 ), x (n M+2 ), ..., x (n N )]
ni = qi ∈ NQ = {q1 , q2 , ...., q Q } = {n M+1 , n M+2 , ...., n N },
are considered as variables.

The adaptive gradient algorithm is used as follows:
4
x(t0) x(t3)
2 x(t )
ni
6
x(t ), for t=t
x(5)
0
x(1)
x(t4)
-2 x(t2)
x(t )
7
-4
0 1 2 3 4 5 6 7 8
time
Figure 10.33 Illustration of indirect (nonunform) sampling with N = 8 and M = 6 available

samples x (t0 ), x (t2 ), x (t3 ), x (t4 ), x (t6 ), x (t7 ), and two unavailable samples whose positions
are assumed at x (∆t) = x (1) and x (5∆t) = x (5).
-For each missing signal sample position qi ∈ NQ the gradient of the

sparsity measure is estimated forming the signal (in the (m + 1)th iteration)
(m)
x+
a ( qi ) = x a ( qi ) + ∆
(m)
x−
a (qi ) = x a ( qi ) − ∆.
The available samples x (tni ), tni ∈ T A = {tn1 , tn2 , ..., tn M } are unchanged.
Since the sparsity domain is the DFT of signal x =[ x (0), x (1), ..., x ( N − 1)]
then the signals x+ −
a and x a are used to recalculate corresponding signals at
the sampling theorem positions x1 and x2 according to (10.99)
x1 = B − 1 +
N xa
and
x2 = B − 1 +
N xa
Sparsity minimization using the DFT of these signals X1 (k ) = DFT[ x1 (n)]
and X2 (k ) = DFT[ x2 (n)], with the estimation of the sparsity measure gradi-
ent
∑ N −1 | X1 (k )| − ∑kN=−01 | X2 (k )|
g ( q i ) = k =0 (10.100)
2N∆
reduces this problem to the problem with the sampling at the sampling
theorem rate. The reconstruction is then based on the same procedure using
the steps (10.82)-(10.83) from the presented algorithm.
Example 10.32. Consider the signal defined by (10.84) with M samples at instants
tni ∈ T A = {tn1 , tn2 , ..., tn M }
where tni = ni ∆t + νni and νni is a uniform random variable −∆t/2 ≤ νni ≤
∆t/2. Similar results for the SRR and the average number of iterations, for
various M and sparsity K, are obtained as in Fig.10.28. They will not be re-
peated. A particular realization with K = 6 nonzero DFT coefficients, out of
N = 128, and a number of available samples M = 16 within the transition
region, when the recovery is not always obtained, is considered. The realiza-
tions, when the recovery conditions, for a given signal and for some of the
considered sets of available samples, are met, can be detected. The criterion
for detection of a sparse signal after reconstruction is the measure of signal
sparsity. In this case measures closer to the ℓ0 -norm should be used. For ex-
ample, with ℓ1/4 -form in the case of a nonsparse reconstruction all transform
coefficients are nonzero with ∑kN=−01 | X (k)/N |1/4 ∼ N. For a full recovery of a
sparse signal the number of nonzero coefficients (the measure value) is much
lower since K ≪ N.
Among 100 performed realizations a possible sparse recovery event is
detected when the described sparsity measure of the result is much lower
than N. The set of DFT coefficient positions for the detected sparse signal
is K = {22, 35, 59, 69, 93, 106}. This sparse reconstruction is checked for
uniqueness using the Test 1. The missing samples are from the set qm ∈ N Q .
It is a set difference of all samples N= {n |0 ≤ n ≤ 127 } and
M = {7, 14, 18, 21, 34, 37, 51, 69, 79, 82, 89, 90, 99, 100, 113, 117} .
For h = 0, 1, ..., r − 1 = 6 corresponding values of Q2h and S2r−h , defined in Test

1, are calculated. Their values are:
h 0 1 2 3 4 5 6
Q2h 112 58 31 16 8 4 2 .
S27−h 0 0 4 5 4 4 2
Note that Q20 = 112 is the total number of missing samples, while Q21 is
obtained by counting odd and even samples in N Q and taking higher number
of these two. Since there are 54 samples at odd positions and 58 samples at
even positions, it means that Q21 = 58.
For h = 2 there are 31 missing sample qm ∈ N Q with mod(qm , 4) = 0, 26
missing samples with mod (qm , 4) = 1, 27 missing samples with mod (qm , 4) =
2, and 28 missing samples with mod (qm , 4) = 3, resulting in
Q22 = max{31, 26, 27, 28} = 31,
and so on. We can easily conclude that samples x (1) and x (65) are missing,
meaning that Q64 assumes its maximal possible value Q64 = 2.
Similar counting is done to get S27−h . For example,
Q64 −1 1
S27−6 = S21 = ∑ P6 (l ) = ∑ P6 (l ) = P6 (1),
l =1 l =1
where array P6 (l ) is obtained by sorting number of even and odd elements in

K. Since there are 2 even and 4 odd elements P6 (1) = 2 and P6 (2) = 4 resulting
in S21 = 2.
As expected this set of 112 missing samples N Q does not guarantee
a unique solution for an arbitrary signal of sparsity K = 6. By using the
Test 1 with S2r−h = 0 and Q2h presented in the previous table we easily get
that the solution uniqueness for this set N Q and arbitrary signal requires
K < 4. However, for the specific available signal values, a sparse signal
is reconstructed in this case, with nonzero coefficients at K = {22, 35, 59,
69, 93, 106}. The uniqueness then means that starting from this signal we
cannot find another signal of the same sparsity by varying the missing signal
samples positioned at n ∈ N Q . Test 1 then gives the answer that this specific
recovered signal x R (n), with specific missing sample values and positions
N Q , is unique. It means that starting from x R (n) we cannot get another
signal of the same or lower sparsity by varying the missing samples only. The
reconstructed signal is presented in Fig.10.34. The signal-to-reconstruction-
error ratio defined by (10.85), calculated for all signal samples, is SRR =
111.08 dB. It corresponds to the defined reconstruction algorithm precision
of about 100 dB.
In addition to the considered case two obvious cases in the uniqueness
analysis may appear: 1) when both, the reconstructed signal and the worst
case analysis produce a unique solution using the set of missing samples NQ ,
and 2) when both of them produce a result stating that a signal with certain
sparsity cannot be reconstructed in a unique way with N Q . 3) Finally, it is
interesting to mention that there exists a third case when the set of missing
samples can provide a unique reconstruction of sparse signal (satisfying
unique reconstruction condition if it were possible to use ℓ0 -norm in the
minimization process), however the ℓ1 -norm based minimization does not
satisfy the additional restricted isometry property constraints to produce this
solution (the same solution as the one which would be produced by the ℓ0 -
norm). This case will be detected in a correct way using the presented Test
1. It will indicate that a unique solution is possible using N Q , while if the
ℓ1 -norm based minimization did not produce this solution as a result of the
reconstruction algorithm, the specific reconstructed signal will not satisfy the
uniqueness condition.
In the considered complex-valued signal case a sample variation is

done in four directions ±∆ ± j∆. The estimated gradient vector is complex-
valued.
available randomly positioned signal samples

x(ti )
7.5
-7.5
0 16 32 48 64 80 96 112
time
original and reconstructed signal on the sampling interval grid
x(t), xR(n)
7.5
-7.5
0 16 32 48 64 80 96 112
time
Figure 10.34 Available randomly positioned samples x (ti ) (dots) of a sparse signal x (t)
(top). Reconstructed signal x R (n) at the sampling theorem positions (crosses) along with the
available samples (dots) (bottom). Continuous-time signal x (t) is presented by solid line.
10.10 PROCESSING OF SPARSE SIGNALS WITH IMPULSIVE NOISE
Processing of signals corrupted with impulsive noise is common situa-

tion in practical applications. Consider a discrete signal x (n), 0 ≤ n ≤ N − 1
which is sparse with sparsity K in, for example DFT domain. Assume that
I samples of the signal x (n) at unknown positions n ∈ NQ are corrupted
with impulsive noise ε(n), while the other samples at positions n ∈ M are
uncorrupted. The noise ε(n) can be then modeled as: ε(n) = 0 for n ∈ M and
ε(n) assumes arbitrary values for n ∈ NQ .
If the corrupted samples are considered as unavailable, it is obvious
that the original signal can be reconstructed if a sufficient number of uncor-
rupted samples exists. In this formulation, uncorrupted signal samples are
considered as available observations/measurements.
_____________________________________________________
This Section presents results from: L. Stankovic, M. Dakovic and S. Vujovic, "Reconstruc-
tion of Sparse Signals in Impulsive Disturbance Environments", preprint, 2014. Adapted for
this book by S. Vujović.
10.10.1 Direct Search Procedure
Very simple and intuitive idea is used first to address the problem of this
kind of noise elimination. A random set of M signal samples is used and
considered as available samples/measurements. The number of available
samples should be sufficiently large so that the signal of assumed sparsity
K can be reconstructed. Signal is then reconstructed. If nonnoisy samples are
selected then a sparse signal will be obtained. Detection of a sparse signal
reconstruction event is done by measuring sparsity of the obtained signal.
By using a sparsity measure close to l0 -norm the reconstruction realizations
containing disturbed samples, will produce nonsparse signal with the value
of sparsity measure close to the total number of samples N. In the case when
only the uncorrupted samples are used in the reconstruction, the sparsity
measure value is of order K, which is much lower than the total number of
samples N. The measure of form
N −1
M{ X (k)} = ∑ | X (k)/N | p , (10.101)
k =0
can be used with a small p so that its behavior is similar to the l0 -norm. In the
calculation with a finite precision, a sparse recovery will produce very small
(but nonzero) transformation coefficients values X (k ) at the positions where
they should be zero. Value of p should be such that | X (k )| p at these positions
is much lower than the value of | X (k )| p at the original nonzero signal
positions. Robustness to small but non-zero values in X (k ) is achieved using
p slightly greater than zero, for example p = 1/4. A threshold Tµ within
K < Tµ < N can be used in order to detect a sparse reconstruction event.
Now we will estimate the probability that all samples from a ran-
domly chosen subset are uncorrupted. The total number of samples in this
randomly chosen subset is M, at the positions n ∈ M. Probability that the
first randomly chosen sample is not affected by the described disturbance
is ( N − I )/N since there are N samples in total and N − I of them are un-
corrupted. Similarly, the probability that both the first and second chosen
samples are not affected by disturbance is NN− I NN−−I −1 1 . In general, probabil-
ity that all of M randomly chosen samples at the positions n ∈ M are not
affected by a disturbance is
M −1
N−I−i
P( M, N ) = ∏ N−i
. (10.102)
i =0
The probability P( M, N ) decreases as the number of terms in the product
increases, since NN−−I −i i < 1. In order to improve probability of a sparse
recovery event, it is important to keep the number of samples M in the

observation set M as low as possible, while satisfying the reconstruction
condition. For a fixed number of pulses I, the expected number of random
realizations to achieve at least one sparse recovery event using a subset of
M samples is 1/P( M, N ).
Example 10.33. Consider a N = 128 samples of signal
l
x (n) = ∑ Ai cos(2πki n/N + φi ) (10.103)
i =1
which is sparse in DFT domain with sparsity K = 2l = 6. A 1000 realizations

of reconstructions are performed, and for each realization, the amplitudes,
frequencies, and phases of the signal were taken randomly within 1 ≤ Ai ≤ 2,
1 ≤ k i ≤ 63 and 0 ≤ φi ≤ 2π. Signal x (n) is corrupted by an impulsive noise
ε(n), which is expected in about 12% of the signal values, corresponding
to I = 15 corrupted signal samples, Fig.10.35(a). Since the signal sparsity is
K = 6, the full recovery will be possible, with a high probability, if we use
M = 32 samples in reconstruction, assuming that all of M = 32 samples are
not affected by disturbance (Fig.10.28). Using (10.102) we can calculate proba-
bility that none of M = 32 randomly chosen samples will be affected by noise.
It is P(32, 128) = 0.0099. It means that we can expect an order of 10 full recov-
ery realizations in 1000 trials. Impulsive noise used in this example is of the
form ε(n) = ε 1 (n)/ε 2 (n) + ε 3 (n)/ε 4 (n) + 10ε 5 (n) where ε i (n), i = 1, 2, 3, 4, 5,
for n ∈ NQ are the unit variance Gaussian noises. It is important to note that
the results do not depend on the disturbance amplitude values or their distri-
bution. The l1 -norm is used as sparsity measure in the reconstruction process.
As we can see from Fig.10.35(b) there are some realization of sparse
signal recovery corresponding to the algorithm precision value of SRR. The
measure of reconstructed signal sparsity (10.101) is a criterion for sparse sig-
nal recovery detection, Fig.10.35(d). The sparsity measure values for the cases
when a nonsparse signal is recovered is much higher than K. Low values of
sparsity measure correspond to high SRR, Fig.10.35(b). In Fig.10.35(c), the
realization with the smallest sparsity measure is used to reconstruct the sig-
nal x (n). From Figs.10.35(b) and (d) we can conclude that there are 3 full
sparse signal recoveries in 200 random realizations. The reconstructed sig-
nal is x R (n). In reality we need only one full recovery realization. Calcula-
tion should be stopped when the sparsity measure threshold is reached first
time.
10.10.2 Criteria for Selecting Samples
The presented direct search procedure can be used on signal with a small
number of corrupted samples since a number of the random realizations
20
x(n)+ε(n) SRR [dB]
10
-10
(a) (b)
-20
0 50 100 0 200
time realization index
60
M{X(k)}
10 x(n), x (n)
R
5 40
0
20
-5
(c) (d)
-10 0
0 50 100 0 200
time realization index
Figure 10.35 Reconstruction of a signal with I = 15 out of N = 128 samples being affected
by an impulsive disturbance. In each realization 96 randomly chosen samples are removed.
Total number of realizations is 200. a) The available corrupted signal; b) The SRR for each of
200 realizations; c) The original (black line) and the reconstructed (dots) signal for the best
realization; d) The sparsity measure for each of 200 realizations.
required to have an uncorrupted subset of signal samples increases with

the number of corrupted samples.
10.10.2.1 L-statistics Based Elimination
In some applications the impulsive noise is much stronger than the signal.
The trimmed L-statistics can be used to eliminate the corrupted signal
samples, without any search procedure. The values of signal samples x (n)
are ordered into a nonincreasing sequence
| x (n1 )| ≥ ... ≥ | x (ni )| ≥ | x (ni+1 )| ≥ ... ≥ | x (n N )| . (10.104)
If strong impulsive noise components exist, well above the signal level, then
very large absolute values of signal samples should be omitted as corrupted.
After these samples are removed then the remaining M < N samples
y = { x (n1 ), ..., x (n M )} .
are used as the available observations in the signal reconstruction. The

number of omitted samples or the threshold for signal samples elimination
are studied within the robust analysis.
The L-statistics and the direct search methods can be combined. Some
of the corrupted samples may be eliminated based on their values using
the L-statistics, while a small number of corrupted samples at unknown
positions can be found by a random selection of subsets. This approach is
based on a priori knowledge of impulsive disturbance.
10.10.2.2 Iterative procedure
A criterion that will mark some signal samples as probably more corrupted
than the others is presented next. In this process, no particular distribution
or number of corrupted samples is assumed.
Consider a corrupted signal xε (n) = x (n) + ε(n). For each time instant
we will form two signals x + −
a ( n ) = xε (n ) + ∆δ (n − m ) and x a (n ) = x ε (n ) −
∆δ(n − m), where m = 0, ..., N − 1. Then, a difference of measure values is
calculated as
N −1 ' ' N −1 ' '
g(m) = ∑ ' Xa+ (k )' − ∑ ' Xa− (k )' , (10.105)
k =0 k =0
where Xa+ (k )= DFT[ x +

a ( n )]
and Xa− (k )
= DFT[ x −a (n )]. For a large step size
∆, according to (10.75) g(m) ∼ ε(m). The signal samples at the positions
where the value | g(m)| is highest are eliminated and considered as un-
available in the reconstruction. However, in the remaining samples there
could also exist some samples corrupted with noise. In order to remove
the remaining noisy samples, two procedures can be used. First one is to
apply direct search over the remaining samples. The second one is based
on repeating the previous difference of measures based elimination in an
iterative way.
Algorithm:
Step 0: At the beginning, denote with NQ set of signal sample po-
sitions which are selected with criterion (large | g(m)|). Most of them are
heavily corrupted by noise. The goal is to locate remaining corrupted sam-
ples. The set of remaining sample positions M is a set complement of N Q .
Step 1: For each sample in M perform reconstruction under the as-
sumption that the considered sample is also unavailable, i.e. that it belongs
to NQ . Comparing concentration measure of reconstructed signals before
and after adding each considered samples to NQ , we can find candidates

for corrupted samples. This analysis is a consequence of the fact that the re-
moval of a sample with a high noise will significantly improve sparsity mea-
sure of the recovered signal. Sparsity measure will almost not be changed
by removing a sample with low/no noise.
Step 2: Choose few r = 1, 2, 3, or 4 samples from M after whose
removal the best improvement in measure value was produced in Step 1.
Move them from M to NQ and repeat Steps 1-2 with the new NQ and M.
Comment: For r = 1 only the sample causing maximal measure change
will be moved from one to another set. In order to make procedure more
efficient we may remove r = 2 or r = 3 samples since the procedure detects
few the largest changes with a high reliability. At the end of It iterations, in
total It r samples with remaining noise could be removed.
Example 10.34. Consider N = 128 samples of a signal defined by (10.84) with

sparsity K = 10. The disturbance of form ε(n) = 40(ε 1 (n) − 0.5) + 40(ε 2 (n) −
0.5) is used, where ε 1 (n) and ε 2 (n) are white uniform noises. The number
of samples affected by a disturbance is I = 64. This kind of disturbance is
chosen since a large number of its values are within the signal amplitude
range at the positions where the disturbance exits. The iterative removal
procedure is used for reconstruction, with r = 4 samples being added to
the set of unavailable/corrupted samples NQ in each iteration. In an ideal
case if there is no miss-detection, all corrupted samples from this example
will be removed in 64/4 = 16 iterations. The more realistic scenario is that
there exist few missdetections. Results of reconstruction are presented in Fig.
10.36. The numbers of omitted samples in the realizations were Q = 72 for the
considered signals. It means that just a few miss-detections existed. In Fig.
10.37 are presented disturbance values, in the order as they were detected
and omitted by the algorithm. Note that the algorithm followed quite well
the significance order of the disturbance in their omission. In order to more
accurately check this procedure, the same process is repeated 100 times with
arbitrary signal amplitudes and frequencies. In all realizations, all corrupted
samples where among 72 samples selected by the criterion.
10.10.3 Uniqueness of the Obtained Solution
After a sparse signal is reconstructed from a reduced set of samples its

uniqueness should be confirmed. In theory, even if the reconstructed signal
corresponds to the original signal at the instants of available samples, it still
does not mean that there does not exist another signal satisfying the same
set of available samples.
150 150
SRR [dB] M{X(k)}
K=10 K=10
100
100
50
50
0
-50 (a) 0 (b)

0 16 32 48 64 80 0 16 32 48 64 80
removed samples removed samples
Figure 10.36 Reconstruction of a sparse signal when corrupted samples are removed by using
the criterion in iterative way. In each iteration r = 4 samples are removed. a) The SRR during
the iterations. b) The sparsity measure during the iterations for a signal of sparsity K = 10.
40
disturbance samples
30 K=10
20 2A
10
0
20 40 60 80 100 120
sorting index
Figure 10.37 Disturbance values in the signal, sorted according to the introduced significance
criterion, with signal range in amplitude 2A.
Here we will illustrate the presented uniqueness test on the signal

from the last example, whose sparsity is K = 10 and the algorithm has
removed Q = 72 out of N = 128 samples. Using the theorem for the specific
set of removed samples NQ we obtained the sparsity limit K < 16. It means
that the reconstruction is unique.
For the same number of missing samples the theorem is run 100, 000
times with arbitrary possible distribution of Q = 72 removed sample posi-
tions. The probability that a signal with sparsity K is unique, with randomly
removed Q = 72 samples is presented in Fig.10.38. Probability that the worst
0
10
uniqueness probability, Q=72 out of N=128
-1
10
-2
10
-3
10
-4
10
0 5 10 15 20 25 30 35
sparsity K
Figure 10.38 Sparsity limit probability distribution for the worst possible case of signal with
Q = 72 out of N = 128 samples in 100,000 random realizations.
case signal with sparsity K = 10 is unique for Q = 72 is
Probability [K = 10 is unique, with Q = 72] = 0.8723.
10.11 IMAGE RECONSTRUCTION
The gradient based algorithm is applied on the image x (n, m). As the
transformation domain the two-dimensional DCT (in symmetric form) will
be used
N −1 N −1 * + * +
2π (2m + 1)k 2π (2n + 1)l
C (k, l ) = vk vl ∑ ∑ x (m, n) cos cos ,
m =0 n =0 4N 4N
√ √
where v0 = 1/N and vk = 1/N for k ̸= 0. Assume that random set of
pixels is available (not corrupted) at (n, m) ∈ M. The goal is to reconstruct
_________________________________________
This section is written by Isidora Stanković.
unavailable pixels. In order to apply a CS reconstruction algorithm, the

image sparsity is assumed in the DCT domain. The DCT of an image
is usually calculated by using 8x8 blocks. Most of the common images
could be considered as sparse in the DCT domain without any additional
processing. If we want to be sure that the original image, which will be
processed, is sparse we can pre-process it by calculating the DCT of its
8x8 blocks and set the lowest amplitude coefficients to zero. By making
the image sparse in the DCT domain we will not make a notable visual
difference with respect to the original image.
Using the available pixels (measurements), an initial image is formed.
It assumes the original image values at the positions of available pixels,
while the missing pixels are set to zero (or arbitrary) value. This new image
is defined as
!
(0) x (m, n) for (n, m) ∈ M
x a (m, n) =
0 for (n, m) ∈ NQ
Note that for the missing pixels any value within the possible image
values range can be assumed in the initial step. The algorithm will recon-
struct the true image values at these positions. For graphical representation
of missing pixels the value 255 corresponding to a white pixel or 0 will be
used. Then the corrupted pixels are black or white pixels, Fig. 10.39.
For each missing sample signals x + +
a ( m, n ) and x a (m, n ) are formed:
x+
a (m, n ) = x
( p)
(m, n) + ∆δ(m − mi , n − ni )
x−
a (m, n ) = x
( p)
(m, n) − ∆δ(m − mi , n − ni ). (10.106)
The finite difference of the signal transform measure is calculated
∥Ca+ (k, l )∥1 − ∥Ca− (k, l )∥1
g ( mi , ni ) = (10.107)
2∆
where Ca+ (k, l ) = DCT[ x + − −
a (m, n )] and Ca (k, l ) = DCT[ x a (m, n )].
A gradient matrix Gm,n is of the same size as the image. At the
positions of available samples (n, m) ∈ M, this matrix has zero value, Gm,n =
0. At the missing sample positions n ∈ N Q its values are Gm,n = g(m, n),
calculated using (10.107).
The image values are corrected iteratively as
( p) ( p −1)
x a (m, n) = x a (m, n) − 2∆Gm,n . (10.108)
The change of step ∆ and the stopping criterion are the same as in one-
dimensional case. The results in 50 iterations are shown in Fig. 10.39. Re-
constructed image after 1, 3, and 50 iterations are presented.
Noisy image Reconstructed image. Iteration: 1
Reconstructed image. Iteration: 3 Reconstructed image. Iteration: 50
Figure 10.39 Reconstruction of image using the gradient-based algorithm.

Cohen class of distributions, 655
discrete form, 660
Index kernel decomposition, 662
Coherence, 684, 687
Complex sinusoidal signal, 22
Adaptive reconstruction, 771 Compressive sensing, 666
Adaptive systems, 423 Continuous signals, 21
Allpass system, 246 Convolution
Ambiguity function, 629 circular, 115
Analog signals, 16 continuous, 33
Analytic part, 40 discrete-time, 65
Antenna array, 473 in frequency domain, 39, 73
Anticausal systems, 186 Cosine series, 29
Attenuation, 56
Auto-regressive (AR), 185 Derivative
Autocorrelation function, 336 complex function, 26
Autocovariance function, 336 Difference equation, 183, 186
Differential equation, 51
Backward difference, 226 Differentiator, 73
Bandpass filter, 244 Digital signals, 16
Bilinear transform, 230 Direct realization I, 268
Binomial random variable, 342 Direct realization II, 268
Blackman window, 534 Dirichlet conditions, 24
Block LMS algorithm, 482 Discrete Cosine transform (DCT), 140
Born-Jordan distribution, 657 Discrete Fourier transform (DFT), 107, 124
Butterworth filter, 53, 240 Discrete Hartley transform (DCT), 165
discrete-time, 237 Discrete pseudo Wigner distribution, 639
Discrete Sine transform (DST), 143
Capon’s method, 614 Discrete system, 64
local polynomial Fourier transform (LPFT), Discrete-time signals (discrete signals), 59
620 Displacement, 137
short-time Fourier transform (STFT), Downsampling, 576
618 Duality property, 36
Cascade realization, 276
Causal system, 34, 67 Eigenvalues, 438
Causal systems, 181 Eigenvectors, 438
Central limit theorem, 344 Energy, 24, 63
Characteristic polynomial, 186 Equiangular Tight Frame, 686
Choi-Willimas distribution, 663 Ergodicity, 337
811
812 Index
Error function, 345 Interpolation, 120

Error signal Inverse system, 247
adaptive system, 429 Isometry, 683
ISTA algorithm, 769
Fast Fourier transform, 126
decimaton-in-frequency, 126 Kalman filter, 495
decimaton-in-time, 128 Kronecker delta function, 60
Finite impulse response (FIR), 185
frequency domain design, 291 L-statistics, 331
realization, 284 Lagrangian, 766
First-order statistics, 319 Laplace transform, 48
Fixed point arithmetic, 382 LASSO minimization, 767
Floating point arithmetic, 390 Leakage effect, 135
IEEE standard, 392 Linear adaptive adder, 427
mu-law and A-law, 392 Linear phase systems, 285
Fourier series, 27, 41, 122 Linear system, 33
Fourier transform, 35, 41, 122 LMS algorithm, 457
matrix, 112 antenna systems, 473
of discrete-time signals, 67, 124 block, 482
properties, 37 complex, 487
signum function, 37 convergence, 459
Fractional Fourier transform, 613 echo cacellation, 479
relation to the LPFT, 614 identification, 460
windowed, 614 noise cancelation, 464
Frequency estimation, 136 prediction, 470
sign, 481
Goertzel algorithm, 188 sinusoidal disturbance, 467
Gradient, 768 variable step, 485
Gramm matrix, 692 Local polynomial Fourier transform, 610
Group delay, 46, 250, 285, 633 moments, 611
relation to fractional Fourier transform,
Haar transform, 152, 575 613
Hadamard transform, 151 Lowpass filter, 236
Hamming window, 533
Hann(ing) window, 70, 531 Magnitude, 24
Hartley series, 29 Marginal properties, 653
Highpass filter, 242 Matched filter, 369
Hilbert transform, 40 Matched z-transform method, 223
Homogeneous equation, 187 Measurement Matrix, 679
Bernoulli Random, 682
Image reconstruction, 809 Gaussian Random, 681
Impulse invariance method, 218 Indirect, 680
Impulse signal Partial DFT, 681
continuous (delta function), 22 Structured Random, 682, 793
discrete-time, 60 Median, 329, 356
Indirect measurements, 792 Minimum phase system, 247
Infinite impulse response (IIR), 185 Moments
Initial condition LPFT, 612
continuous, 50 Morlet wavelet, 571
Instantaneous frequency, 42, 633 Moving average (MA), 185
Moyal property, 635 Quantization, 376

MUSIC, 618
Random signals, 319
Narrowband signals Rank of matrix, 704
spectral estimation, 366 Rayleigh distribution
Neural networks, 501 unform, 349
activation function, 503 Reconstruction uniqueness, 783
acyclic, 506 Reconstruction uniquness, 787
continuous output, 512 Rectangular window, 70
cyclic, 506 Recursive systems
error backpropagation, 516 adaptive, 493
layer, 506 Reduced interference distributions
multilayer, 516 discrete form, 657
network function, 503 Region of convergence, 170
perceptron, 508 Resolution, 533
supervised, 507 Restricted isometry, 683, 687
unsupervised, 519 constant, 683, 688
voting mashines, 519 eigenvalues, 691
Neuron, 502 uniqueness, 701
Noise, 340 Ridge regression, 767
binary, 341 RLS algorithm
compex Gaussian, 349 variable step, 489
Gaussian, 344
S-method, 647
Laplacian, 350
S-transform (the Stockwell transform), 607
missing samples, 718
Sampling
reconstruction, 729
nonunform, 792
unform, 340
Sampling theorem, 101
Noisy signal
for periodic signals, 130
Fourier transform, 352
in the frequency domain, 41
Norm zero, 710
in the time domain, 77
Norm-one, 735
Schwartz’s inequality, 686
ball, 748
Second-order statistics, 336
Norm-zero, 677
Sensitivity of system, 271
Notch filter, 182, 212
Short-time Fourier transform (STFT), 522
discrete, 538
Optimal filter, 372, 434 discrete-time, 535
Orthogonality principle, 445 filter bank, 542
Overflow, 384 frequency-varying, 567
hybrid, 569
Parallel realization, 280 inversion, 529, 546
Parseval’s theorem, 74, 114, 682 optimal window, 525
Perceptron, 508 optimisation, 560
Period of a discrete signal, 62 overlapping, 544
Pisarenko method, 619 recursive, 541
Power, 24, 63 time-varying, 556
Power spectral density, 338, 362 Sign LMS algorithm, 481
Probability, 326 Sinc distribution
density function, 326 discrete form, 657
Probability density function, 328 Soft-thresholding, 769
814 Index
Spark of matrix, 704 Hamming, 291, 533

coherence, 760 Hann(ing), 291
uniqueness, 706 Hann(ing) (Hann), 531
Sparse signals, 666 Kaiser, 535
Sparsity, 676 rectangular, 529
Stable system, 34, 67 Windows, 287
Stable systems, 181
Standard deviation, 331 Yule-Walk equation, 367
Stationary phase method, 42
Stationary signals, 337 z-transform, 169, 191
Steepest descend method, 446 inverse, 174
Zero-padding, 120
Taylor series, 42 Zhao-Atlas-Marks distribution
Trace of matrix, 449 discrete form, 657
Unit step signal

continuous (Heaviside function), 21
discrete-time, 60
Unitary matrix, 682
Upsampling, 578
Variance, 331
Voting mashines, 519
Walsh-Hadamard transform, 151

Wavelet transform, 569
Coifflet, 606
Daubechies D4, 588
Daubechies D6, 602
filter bank, 574
Haar, 585
orthogonality, 582
reconstruction condition, 579
scale function, 600
wavelet function, 600
Welch bound, 684
Welch periodogram, 368
Wide sense stationary signals, 337
Wiener filter, 372
Wigner distribution, 623, 656
auto-terms, 627
cross-terms, 627
discrete form, 639
properties, 631
pseudo, 636
S-method, 647
smoothed, 637
Window, 529
Bartlett (triangular), 289, 530
Blackman, 534
Bibliography
[1] S. T. Alexander, Adaptive Signal Processing: Theory and Applications,

Springer-Verlag, 1986.
[2] R. Allred, Digital Filters for Everyone, CreateSpace Independent Publish-

ing Platform; 2 edition, 2013.
[3] M. Amin, Compressive Sensing for Urban Radar, CRC Press, 2014.
[4] A. Antoniou, Digital Signal Processing: Signals, Systems, and Filters,

McGraw-Hill Education, 2005.
[5] F. Auger and F. Hlawatsch, Time-Frequency Analysis: Concepts and Meth-

ods, Wiley-ISTE, 2008.
[6] D. Blandford and J. Parr, Introduction to Digital Signal Processing, Pren-

tice Hall, 2012.
[7] B. Boashash, Time-Frequency Signal Analysis and Processing: A Compre-

hensive Reference, Second Edition, Academic Press, 2015.
[8] P. Bremaud, Mathematical Principles of Signal Processing: Fourier and

Wavelet Analysis, Springer, 2002.
[9] S. A. Broughton, Discrete Fourier Analysis and Wavelets: Applications

to Signal and Image Processing, Wiley-Interscience, 2008.
[10] L. F. Chaparro, Signals and Systems using MATLAB, Academic Press,

2011.
[11] C. T. Chen, Signals and Systems: A Fresh Look, CreateSpace Independent

Publishing Platform, 2011.
815
816 Bibliography
[12] V. Chen, D. Tahmoush, and W. J. Miceli, Radar Micro-Doppler Signature-

Processing and Applications, The Institution of Engineering and Technol-
ogy (IET), 2014.
[13] L. Cohen, Time-frequency Analysis, Prentice-Hall, 1995.
[14] A. G. Constantinides, System Function of Discrete-Time Systems, Aca-

demic Press, 2001.
[15] M. Davenport, Digital Signal Processing, Kindle edition, 2011.
[16] P. S. R. Diniz, E. A. B. da Silva, and S. L. Netto, Digital Signal Processing:

System Analysis and Design , Cambridge University Press; 2 edition,
2010.
[17] I. Daubechies, Ten Lectures on Wavelets, SIAM, 1992.
[18] M. Elad, Sparse and Redundant Representations: From Theory to Applica-

tions in Signal and Image Processing, Springer, 2010.
[19] Y. C. Eldar and G. Kutyniok Compressed Sensing: Theory and Applications,

Cambridge University Press, 2012.
[20] M. E. Frerking, Digital Signal Processing in Communication Systems,

Springer, 1994.
[21] P. Flandrin, Time-Frequency/Time-Scale Analysis, Academic Press, 1999.
[22] R. C. Gonzalez and R. E. Woods, Digital Image Processing, Prentice Hall,

2007.
[23] N. Hamdy, Applied Signal Processing: Concepts, Circuits, and Systems,

CRC Press, 2008.
[24] M. H. Hayes, Schaums Outline of Digital Signal Processing, 2nd Edition,

Schaum’s Outline Series, 2011.
[25] S. Haykin, Least-Mean-Square Adaptive Filters, Wiley-Interscience, 2003.
[26] S. Haykin and B. Van Veen, Signals and Systems, Wiley, 2002.
[27] E. Ifeachor and B. Jervis, Digital Signal Processing: A Practical Approach,

Prentice Hall; 2 edition, 2001.
[28] V. K. Ingle and J. G. Proakis, Digital Signal Processing Using MATLAB,

CL Engineering; 3 edition, 2011.
[29] S. Kay, Fundamentals of Statistical Signal Processing, Volume I: Estimation

Theory, Prentice Hall, 1993.
[30] S. Kay, Fundamentals of Statistical Signal Processing, Volume II: Detection

Theory, Prentice Hall, 1998.
[31] E. Kudeki and D. C. Munson Jr., Analog Signals and Systems, Prentice
Hall, 2008.
[32] S. M. Kuo, B. H. Lee, and W. Tian, Real-Time Digital Signal Process-

ing: Fundamentals, Implementations and Applications, Wiley, third edition,
2013.
[33] B. P. Lathi and Z. Ding, Modern Digital and Analog Communication

Systems , Oxford University Press, 2009.
[34] B. P. Lathi and R. A. Green, Essentials of Digital Signal Processing, Cam-

bridge University Press, 2014.
[35] S. R. Laxpati and V. Goncharoff, Practical Signal Processing and its Appli-
cations, Part I, CreateSpace Independent Publishing Platform, 2013.
[36] R. G. Lyons, Understanding Digital Signal Processing, Pearson, 2010.
[37] S. Mallat, A Wavelet Tour of Signal Processing, Academic Press, second

edition, 1999.
[38] D. P. Mandić and S. L. Goh, Complex Valued Nonlinear Adaptive Filters:

Noncircularity, Widely Linear and Neural Models, John Wiley & Sons,
2009.
[39] D. P. Mandić, M. Golz, A. Kuh, D. Obradovic, and T. Tanaka, Signal

Processing Techniques for Knowledge Extraction and Information Fusion,
Springer, 2008.
[40] D. G. Manolakis and V. K. Ingle, Applied Digital Signal Processing: Theory

and Practice, Cambridge University Press, 2011.
[41] D. G. Manolakis, D. Manolakis, V. K. Ingle, S. M. and Kogon, Statisti-

cal and Adaptive Signal Processing: Spectral Estimation, Signal Modeling,
Adaptive Filtering and Array Processing, Artech House, 2005.
[42] J. H. McClellan, R. W. Schafer, and M. A. Yoder, Signal Processing First,

Prentice Hall, 2003.
[43] S. K. Mitra, Digital Signal Processing, McGraw-Hill, 2010.

818 Bibliography
[44] B. Mulgrew, P. Grant, and J. Thompson, Digital Signal Processing: Con-

cepts and Applications, Palgrave Macmillan; 2nd Edition edition, 2002.
[45] R. Newbold, Practical Applications in Digital Signal Processing, Prentice

Hall, 2012.
[46] A. V. Oppenheim and A. S. Willsky, Signals and Systems, Prentice Hall,

2008.
[47] A. V. Oppenheim and R. W. Schafer, Discrete-Time Signal Processing,

[48] A. V. Oppenheim and G. C. Verghese, Signals, Systems and Inference,

[49] A. Papulis, Signal Analysis, McGraw-Hill, 1997
[50] C. L. Phillips, J. Parr, and E. Riskin, Signals, Systems, and Transforms,

[51] B. Porat, A Course in Digital Signal Processing, Wiley, 1996.
[52] J. G. Proakis and D. G. Manolakis, Digital Signal Processing: Principles,

Algorithms, and Applications, Pearson; 4 edition, 2013.
[53] A. Quinquis, Digital Signal Processing Using Matlab, Wiley-ISTE, 2008.
[54] L. R. Rabiner and B. Gold, Theory and Application of Digital Signal

Processing, Prentice Hall, 1975.
[55] M. A. Richards, Fundamentals of Radar Signal Processing, McGraw-Hill

Education, second edition, 2014.
[56] M. J. Roberts, Signals and Systems: Analysis Using Transform Methods &
MATLAB, McGraw-Hill Education, 2011.
[57] A. H. Sayed, Adaptive Filters, Wiley-IEEE, 2008.
[58] L. L. Scharf, Statistical Signal Processing: Detection, Estimation, and Time

Series Analysis, Prentice Hall, 1991.
[59] M. Soumekh, Synthetic Aperture Radar Signal Processing with MATLAB

Algorithms, Wiley-Interscience, 1999.
[60] L. Stanković, Digital Signal Processing, Naucna knjiga, Beograd, Second

edition 1989 (in Montenegrin/Serbian/Croat/Bosnian).
[61] L. Stanković, M. Daković, and T. Thayaparan, Time-frequency Signal

Analysis with Applications, Artech House, Boston, 2013.
[62] L. Stanković and I. Djurović, Solved Problems in Digital Sig-

nal Processing, University of Montenegro, 1998 (in Montene-
grin/Serbian/Croat/Bosnian).
[63] S. Stanković, I. Orović, E. Sejdić, Multimedia Signals and Systems,

Springer, second edition, 2015.
[64] H. Stark and J. W. Woods, Probability and Random Processes with Applica-
tions to Signal Processing, Prentice Hall, third edition, 2001.
[65] L. Tan and J. Jiang, Digital Signal Processing: Fundamentals and Applica-
tions, Academic Press; 2 edition , 2013.
[66] S. Theodoridis and R. Chellappa, eds., Academic Press Library in Signal

Processing, Vols.1-4, Academic Press, 2013.
[67] A. Uncini, Fundamentals of Adaptive Signal Processing (Signals and Com-
munication Technology), Springer, 2014.
[68] M.Vetterli and J.Kovačević, Wavelets and Subband Coding, CreateSpace

Independent Publishing Platform, 2013.
[69] M. Vetterli, J. Kovačević, and V. K. Goyal, Foundations of Signal Process-
ing, Cambridge University Press, 2014.
[70] B. Widrow and D. Stearns, Adaptive Signal Processing, Prentice Hall,

1985.
About the Author
Ljubiša Stanković was born in Montenegro on June 1, 1960. He received

a BSc degree in electrical engineering from the University of Montenegro
in 1982 with the award as the best student at the University. As a student
he won several competitions in mathematics in Montenegro and former
Yugoslavia. He received an MSc degree in communications from the Uni-
versity of Belgrade, and a PhD in theory of electromagnetic waves from the
University of Montenegro in 1988. As a Fulbright grantee, he spent 1984-
1985 academic year at the Worcester Polytechnic Institute, Worcester, MA.
Since 1982, he has been on the faculty at the University of Montenegro,
where he has been a full professor since 1995. In 1997-1999, he was on leave
at the Ruhr University Bochum, Germany, supported by the Alexander von
Humboldt Foundation. At the beginning of 2001, he was at the Technische
Universiteit Eindhoven, The Netherlands, as a visiting professor. During
the period of 2003-2008, he was the rector of the University of Montenegro.
He was ambassador of Montenegro to the United Kingdom, Iceland, and
Ireland 2011-2015. During his stay in United Kingdom he was a visiting aca-
demic at the Imperial College London, 2013-2014. His current interests are in
signal processing. He published about 400 technical papers, more than 140
of them in the leading journals. Stanković received the highest state award
of Montenegro in 1997 for scientific achievements.
Stanković was an associate editor of the IEEE Transactions on Image
Processing, an associate editor of the IEEE Signal Processing Letters, and an
associate editor of the IEEE Transactions on Signal Processing. Stanković is
a member of the Editorial Board of Signal Processing. He is a member of the
National Academy of Sciences and Arts of Montenegro (CANU) since 1996
and a member of the European Academy of Sciences and Arts. Stanković is
a Fellow of the IEEE for contributions to time-frequency signal analysis.
820

DOS Matlab

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

DOS Matlab

Uploaded by

Copyright:

Available Formats

DIGITAL

Library of Congress Cataloging-in-Publication Data

c 2015 Ljubiša Stanković, All Rights Reserved

Printed by CreateSpace Independent Publishing Platform,

Available from Amazon.com and other online and bookstores

No part of this book may be reproduced or utilized in any form or by any

II Discrete Signals and Systems 57

2.3 Sampling Theorem in the Time Domain 75

Chapter 3 Discrete Fourier Transform 107

Chapter 4 z-Transform 169

4.7 Problems 193

Chapter 5 From Continuous to Discrete Systems 217

Chapter 6 Realization of Discrete Systems 267

Chapter 7 Discrete-Time Random Signals 319

7.1.4 Variance 331

III Selected Topics 421

8.5.1 Identiﬁcation of Unknown System 460

Chapter 9 Time-Frequency Analysis 521

9.2.10 Time-Varying Windows 556

Chapter 10 Sparse Signal Processing 665

10.4.4 Unavailable/Missing Samples Noise in Initial

About the Author 820

book is a result of author’s thirty-three years of experience in

explanations for each presented transform, method, algorithm or approach.

is a physical process, mathematical function, or any other

continuous discrete-time digital

of discrete independent variable values. This is a discrete-time signal. The

x(t) x(n) y(n) y(t)

Advantages of processing signals in digital form are in their ﬂexibility

of discrete-time signals are obtained by sampling continuous-

1.1 CONTINUOUS-TIME SIGNALS

One-dimensional signals, represented by a function of time as a continuous

The impulse signal (or delta function) is deﬁned as

The impulse signal is equal to 0 everywhere, except at t = 0, where it

Using the previous relation, it is possible to relate the unit-step signal

x (t) = A sin(Ω0 t + ϕ). (1.5)

This signal is periodic in time, since it satisﬁes the periodicity condition

In this case, the period is T = 2π/Ω0 .

x (t) = Ae j(Ω0 t+ ϕ) = A cos(Ω0 t + ϕ) + jA sin(Ω0 t + ϕ) (1.7)

Example 1.1. Find the period of a signal

⋆This signal consists of N + 1 components. The constant component A0

Some parameters that can be used to describe a signal are:

• Maximum absolute value (magnitude) of a signal

Mx = max | x (t)| , (1.8)

• Signal instantaneous power

Px (t) = | x (t)|2 . (1.10)

The average signal power is deﬁned by

The average power is a time average of energy. Energy signals are

1.2 PERIODIC SIGNALS AND FOURIER SERIES

Consider a periodic signal x (t) with a period T. It can be expressed as a sum

x (t) = · · · + X−1 e− j2πt/T + X0 e− j0 + X1 e j2πt/T + · · · (1.11)

The set of basis functions {e j2πnt/T : −∞ < n < ∞}, is an orthonormal

within the period, is

Note: A derivative of complex function F (z) = u + jv = u( x, y) + jv( x, y) with

where Λ is an arbitrary constant.

⋆The signal x (t) is a periodic extension of x0 (t), with period T = 2.

with X0 = 1/4. Values of Xn are presented in Fig. 1.2.

The reconstructed signal, with M = 1, 2, 6, and 30, is shown in Fig. 1.3.

1.2.1 Fourier Series of Real-Valued Signals

The Fourier series form for real-valued signals is

corresponding to the Fourier cosine series coefﬁcients . Similar expression is

Example 1.6. For a signal

Write the reconstruction formula with M Fourier series coefﬁcients.

with X0 = 1/4. The reconstructed signal with M coefﬁcients is