Novel Least Mean Square Algorithm

B. T. Krishna
Dept. of Electronics and Communication Engineering, Jawaharlal Nehru Technological
University Kakinada, Kakinada, Andhrapradesh, India533003,Email:
tkbattula@gmail.com
Abstract
A Novel Least Mean Square(LMS) algorithm using First order Al-Alaoui
dierentiator is proposed. The condition for stability is also derived. The
performance of the proposed Algorithm is compared with existing algorithms
by taking an example. The results prove the ecacy of the proposed Algo-
rithm.
Keywords: Digital dierentiator; Adaptive Filters; Steepest Descent
Algorithm; LMS algorithm; System Identication
1. Introduction
LMS algorithm is one of the Adaptive ltering Algorithms, which is used
to produce lesser value of Mean Squared error and is invented in 1960s[1-2].
This Algorithm is a Stochastic implementation of the Steepest descent algo-
rithm with Cost function J being Mean Squared Error[6]. For cost function
J the steepest descent is described as the vector dierential equation[2,5,6]
dW
dt
=
J
W(t)
Where W is vector of unknown weights and is a scalar gain.
Although LMS algorithm is simple to implement and robust it suers from
Low convergence speed. Many of the variations in the LMS algorithm are
also proposed.
If the derivative in the above Equation is approximated by Backward
derivative Conventional LMS Algorithm is obtained[5]. By approximating
the derivative term with the Trapezoidal rule, a Novel Trapezoidal LMS(TLMS)
is proposed in [5]. But, higher the closeness of the approximation to the ideal
one, there is a possibility of deriving a novel LMS Algorithm. So, in this
letter, Al-Alaoui approximation is used to derive novel LMS Algorithm. Al-
Alaoui dierentiator is obtained by interpolating Backward and Trapezoidal
Preprint submitted to Computers and Electrical Engineering October 18, 2014
integrators, inversion and Stabilization[7].
Digital dierentiators nd applications in Radar, Sonar, and Bio-medical
Engineering etc. These can be of either Innite Impulse Response (IIR) type
or Finite Impulse Response (FIR) type. But IIR type lters are preferred
because of Low cost, less space etc. In [5], a Trapezoidal rule based LMS al-
gorithm is proposed. In this paper, First order Al-Alaoui dierentiator based
steepest descent algorithm is presented and its performance is compared with
the existing one.
The paper is organized as follows. Derivation of Al-Alaoui digital dier-
entiator based novel steepest descent algorithm and condition for stability is
proposed in Section 2. Results and Conclusions are presented in Section 3.
2. LITERATURE SURVEY
Two of the well-known integration techniques are rectangular and Trape-
zoidal methods. If the derivative term
dW(t)
dt
in Eqn.(1)is approximated by
rectangular Euler integration[2,5],
dW
dt

W
k+1
W
k
T
(1)
Where T is the sampling interval normalized to unity and W
k
is the discrete
weight vector at sample interval k. From Eqns 1 and 2 it can be shown
that[5]
W(n + 1) = W(n)
J
W(n)
(2)
Solving for the gradient of mean square error would result in[6],
W(n + 1) = W(n) + 2e(n)x(n) (3)
This is the equation for LMS algorithm which is used to estimate the weight of
the lter[6]. LMS algorithms have many applications in the areas of adaptive
lters, image processing and signal processing [1, 2, 5]. Similarly, according
to trapezoidal integration[5],
dW
dt

2W
k+1
T
1 z
1
1 + z
1
(4)
where z
1
is the backward shift operator. Using Eqn.5 in Eqn.1 and simpli-
fying,we have,
W
k+1
= W
k

2
J
W
k

2
J
W
k1
(5)
2
Considering a FIR lter with input x(n) and output y(n),
y(n) =
N1

i=0
w(i)x(n i) = W
T
(n)x(n) (6)
d(n) being the desired signal and the error signal can be computed as,
e(n) = d(n) y(n) (7)
and hence the mean square error (MSE) is given by
J = E[e
2
(n)] (8)
By substituting Eqn.9 in Eqn.6, we have,
W(n + 1) = W(n)

2
e
2
(n)
W(n)

2
e
2
(n)
W(n 1)
(9)
J
W(n)
=
e
2
(n)
W(n)
= 2e(n)
e(n)
W(n)
= 2e(n)
(d(n)y(n))
W(n)
= 2e(n)
(W
T
(n)x(n))
W(n)
(10)
Using the results in Eqn.(11) in Eqn.(6) and simplifying,
W(n + 1) = W(n) + e(n)x(n) + e(n 1)x(n 1) (11)
This is the equation for Trapezoidal Steepest-descent algorithm(TLMS)[2].
For stability and convergence the step size satises[2],
0 < <
1

max
(12)
Where
max
is the largest eigenvalue.
3. AL-ALAOUI DIGITAL DIFFERENTIATOR BASED NOVEL
STEEPEST DESCENT ALGORITHM
z-domain representation of Backward and Trapezoidal Integrators is[4,7],
H
Rect
(z) =
zT
z 1
(13)
3
Figure 1: Magnitude Response Comparison Of Digital Integrators
H
Trap
(z) =
T (z + 1)
2 (z 1)
(14)
Fig.1. Compares the Backward and Trapezoidal Integrators with the ideal
integrator which is dened by the Transfer function H(j) =
1
j
.
Al-Alaoui digital dierentiator is designed by interpolating the rectangu-
lar Euler integrator and Trapezoidal integrator and is given by[4],
H
N
(z) = aH
Rect
(z) + (1 a)H
Trap
(z) (15)
It is observed that the value of a =
3
4
provides better approximation to the
ideal integrator[4]
H
N
(z) =
T
8
(z + 7)
(z 1)
(16)
Applying the approach proposed in [4], the new dierentiator is,
G(z) =
8
7T
(z 1)
(z +
1
/
7
)
(17)
Magnitude and Phase response comparison of AlAlaoui digital dieren-
tiator with the ideal one (G(j) = j) are shown in Fig.2 and 3 respectively.
4
Figure 2: Magnitude Response Comparison Of Digital Dierentiators
Figure 3: Phase Response Comparison Of Digital Dierentiators
From the transfer function of minimum phase dierentiator,dened by
5
Eqn.(1), it can be written that,
dW
dt

8W
n+1
7T
1 z
1
1 +
z
1
/
7
(18)
On applying the above dierentiator to steepest descent algorithm,
W(n + 1) = W(n)
7
8
J(n)
W(n)

8
J
W(n 1)
(19)
Using the same approach as TLMS algorithm[5], one obtains
W(n + 1) = W(n) +
7
4
e(n)x(n) +

4
e(n 1)x(n 1) (20)
This is the equation for Novel Steepest-descent algorithm based on Al-Alaoui
digital dierentiator. For m = n+1 weights, it can be seen that the computa-
tional complexity is same as that of the TLMS algorithm and when compared
to LMS algorithm, requires only (1 + m) extra multiples and m extra addi-
tions. To nd the condition for stability of the Proposed Algorithm,let use
dene weight error vectors c(n) and c(n + 1) as,
c(n) = w(n) w
0
(21)
c(n + 1) = w(n + 1) w
0
= c(n) +

4
(7e(n)x(n) + e(n 1)x(n 1))
(22)
The Weiner solution is given by[2],
w
0
= R
1
p (23)
where R and p are Cross correlation and Auto correlation respectively and
are dened as[2,5],
R = E(x(n)x
T
(n)) (24)
p = E(d(n)x
T
(n)) = E(d
T
(n)x(n)) (25)
It can be shown that the error signal e(n) is given by,
e(n) = d(n) y(n) = e
0
(n) x
T
(n)c(n) (26)
where,
e
0
(n) = d(n) x
T
(n)w
0
(27)
6
From the equation for error term e(n) , it can be written easily as,
e(n 1) = e
0
(n 1) x
T
(n 1)c(n 1) (28)
The nal expression for weight error vector is shown to be,
c(n + 1) =
_
I
7
4
x(n)x
T
(n)
_
c(n) +

4
_
7x(n)e
0
(n) + x(n 1)e
0
(n 1) x
T
(n 1)x(n 1)c(n 1)
_
(29)
For the sake of simplicity, the expected values of the individual terms of the
above equation are calculated and dispalyed in Eqns(14-17) for convenience,
E
__
I
7
4
x(n)x
T
(n)
_
c(n)
_
=
_
I
7
4
R
_
E (c(n)) (30)
E (x(n)e
0
(n)) = (p Rw
0
) (31)
E
_
x
T
(n 1)x(n 1)c(n 1)
_
= RE (c(n 1)) (32)
E (x(n 1)e
0
(n 1)) = E
_
x(n 1)
_
d(n 1) x
T
(n 1)w
0
__
= p Rw
0
(33)
By taking the expectations on both sides of the Eqn.(30), and simplifying,
E (c(n + 1)) =
_
I
7
4
_
RE(c(n))

4
RE (c(n 1)) (34)
Dene a Unitary similarity transformation E(v(n)) = Q
T
E(c(n)) where Q
has its columns as eigen values of the matrix R. So, Q
T
Q = I and R = QQ
T
where is a diagonal matrix of eigen values
1
,
2
, ............
n+1
. Multiply-
ing with Q
T
on both sides of the Eqn.(18),and using the unitary similarity
transformations,
E(v(n)) =
_
I
7
4

_
E(v(n 1))

4
E (v(n 2)) (35)
For the k
th
natural mode,the above equation reduces to,
E(v(n)) =
_
I
7
4

k
_
E(v(n 1))

k
E (v(n 2)) (36)
For stability, the roots of the set of polynomials should lie within the unit-
circle.
1
_
1
7
4

k
_
z
1
+

4

k
z
2
= 0 k = 0, 1, 2, ....n + 1 (37)
7
By applying Jurys test it is shown that the condition for stability as[8],
0 < <
4
3
max
(38)
where
max
is the largest eigen value of R.
4. RESULTS AND CONCLUSIONS
To illustrate the ecacy of the proposed AlAlaoui based LMS algorithm,
a system identication example is taken into consideration. Consider the
identication of a 4
th
order FIR system w(z) = 1+2z
1
+3z
2
+4z
3
+5z
4
with additive white noise of variable variance and is chosen as 0.1. The
performance for various values of SNRs is illustrated in the Figures 4, 5 and
6. Table I shows the values of steady-state mean and variance of the weight
estimate for dierent values of SNR.
Figure 4: Comparison of LMS, TLMS and Novel steepest descent algorithms for 40dB
SNR
It can be seen that the Novel steepest descent algorithm have shows supe-
rior performance over TLMS, LMS as the variance of weight estimate is lesser
for varied values of SNR. The performance of Novel steepest descent algo-
rithm when compared with conventional approaches (namely LMS, TLMS)
observed to have faster convergence and lower mean square error.
8
Figure 5: Comparison of LMS, TLMS and Novel steepest descent algorithms for 26dB
SNR
Figure 6: Comparison of LMS, TLMS and Novel steepest descent algorithms for 19dB
SNR
[1] J.G.Proakis, D.G.Manolakis,Digital Signal Processing: Principles, Algo-
rithms and Applications,Prentice-Hall,Upper Saddle River, New Jersey,
2007.
9
Table 1: Comparison of Mean and Variance of Weight Estimates in LMS, TLMS, and
Novel Steepest Descent Algorithms For Dierent SNR Values
SNR 40dB 26dB 19dB
Weight Variance(LMS) 0.4352 0.4736 0.9134
Weight Variance(TLMS) 0.4388 0.5034 0.8345
Weight Variance(Proposed) 0.4347 0.4798 0.8259
Mean Weight estimate (LMS) 5.074 4.866 4.861
Mean Weight estimate (TLMS) 5.003 4.858 4.866
Mean Weight estimate (Proposed) 5.049 4.864 4.864
[2] Bernard Widrow, Samuel D. Stearns, Adaptive Signal Processing, Pear-
son Education Inc, 2007.
[3] Rabiner, L.R., and Steiglitz, K, The design of wide-band recursive
and non-recursive digital dierentiators, IEEE Transactions on Audio
Electro acoust., vol.18, 1970, pp 204209.
[4] M.A.Al-Alaoui, Novel Digital Integrator and Dierentiator, Electron-
ics Letters, vol.29, Feb. 1993, pp.376378..
[5] T.J.Moir, The Trapezoidal Method of Steepest-Descent and its Ap-
plication to Adaptive ltering, The Open Signal Processing Journal,
Vol.3, 2010, pp.15.
[6] Bernard Widrow,Thinking About Thinking: The Discovery of the LMS
Algorithm,IEEE Signal Processing Magazine, 2005,pp.100106.
[7] B.T.Krishna, S.Srinivasa Rao,On design and applications of digital dif-
ferentiators,Fourth International Conference on Advanced Computing,
ICoAC 2012, 1315 Dec. 2012 ,pp.17.
10