ms for a Neural

ally I ~ ~ i b ~ ~ e ~
Qiang Gan*#,Jun Yao' and K.R.Subramanian*

* School of Electrical and ElecfronicEngineering

Nanyang Technological University
Nanyang Avenue, Singapore 639798
E-mail: eqgan@tu. edusg
Abstract
This paper presents a neural network with its output
layer as a classijier and its hidden layer constrained by
laterally inhibited receptive Pel& asfeature extractor, in
which the idea that wavelet transforms are very suitable
for modeling the primary visual informationprocessing is
reflected. Two learning algorithms for designing the
receptive fields are proposed. The problem associated
with local minima caused by the inherent oscillatory
property in laterally inhibited receptive fields is combated
in the algorithm using discrete wavelets. Good
performance is obtained in the experiment of ECG signal
classification using the neural network.

1. Introduction
Neural networks have been established as a general tool
for approximation and classification by fitting
inputloutput data effectively into nonlinear models. The
multilayer perceptron, in which a neuron receives inputs
from all the neurons in the adjacent pre-layer, is widely
used for function approximation and signal classification.
On most occasions it performs quite satisfactorily.
However, when the network input is characterized by
time-frequency localized features, the generally used
multilayer perceptron with unconstrained global
connections between the adjacent layers does not work
well. In the human visual system there exist example
models for dealing with this problem [l]. That is the
conception of receptive field, the shape of which can be
adapted with the visual input under certain constraints. In
the human visual system there are various receptive fields
of different shapes. For example, there are Gauss function
shaped receptive fields for local smoothing, and Gabor

0-7803-4859- 1/98 $ I O.oOO1998 EEE

1156

Department of Biomedical Engineering

Southeast University
Nanjing 2 I0096
P.R.China

function shaped receptive fields for combining local

smoothing and sharpening which provides the function of
lateral inhibition. We find that this kind of receptive fields
can not be formed automatically by learning without any
constraints on the weights in a multilayer perceptron.
Wavelet transform is a good model for the receptive
fields in the human visual system [2]. Because a wavelet
function satisfies the admissibility condition [3], it must be
oscillatory across its zero points. Hence, wavelet hctions
provide natural models for laterally inhibited receptive
fields, which are good at extracting time-frequency
localized features. Actually, Gabor function has been
widely used in theoretical studies of the primary visual
information processing such as lateral inhibition and it can
be regarded as a mother wavelet of good time-frequency
localization properties. Through dilation and translation a
wavelet filter bank can be formed as a group of receptive
fields which approximately perform wavelet transforms.
Hence, in the design of receptivefields we can benefit from
the advanced theory of wavelet transforms.
There have been several pieces of work done on
combining neural networks with wavelet transforms which
perform as the receptive fields of hidden neurons. Szu [4]
developed neural network adaptive wavelets for signal
representation and classification, and tentatively applied
them in phoneme recognition and image compression.
Gan [5] proposed a wavelet neural network architecture
and applied it to ECG signal classification. Dickhaus [6]
and Akay [7]have also studied biomedical signal detection

and classification using different wavelet network

structures. The key issue in the design of this kind of
neural networks is how to obtain optimal sets of dilation
and translation parameters (or wavelet parameters). In all
the networks mentioned above, continuous wavelet
parameters are used and trained by the gradient-descent
learning algorithm, or preset and fixed wavelets are

applied. Because of the inherent oscillatoryproperty of the

wavelet function, learning wavelet parameters is easy to
sink into local minima and it is difficult to get the optimal
result. In this paper, we use Gabor function to constrain the
receptive fields of the hidden neurons so that the lateral
inhibition is introduced into the network. Furthermore,
discrete wavelets are used and a method for calculating
wavelet parameters is proposed to combat the problem of
unconvergence in the learning process.
The remainder of this paper is organized as follows.
The neural network formulation is put forward in
section 2. Two learning algorithmsare proposed in section
3. Simulation studies on ECG signal classification are
carried out in section 4, followed by discussions and
conclusions in section 5 .

2. Neural network formulation

The neural network under consideration, with laterally
inhibited receptive fields, can be described as follows:

i = 1,2,......, NOL.

s( t ) = (1 - e-') /

where, y f L and

X,

(1 + e-.' )

denote the output and the input of the

network respectively,
represent the weights between
output and hidden layers, h ( ( j - b 1 ) / u , ) produce the
weights between hidden and input layers, and NOL, NHL,
NIL are respectively the number of nodes in output,
hidden, and input layers.

h(4 and can be adjusted by dilation and translation

parameters a, and b, . If we regard the Gabor function as
a mother wavelet, the hidden layer is actually composed of
a wavelet filter bank which plays the role of feature
extraction. In connection with the human visual system,
the weights connected to the hidden layer performjust like
receptive fields with lateral inhibition because the Gabor
function shapes like a Mexican hat as shown in Fig.1.
Compared to a general feedforward neural network, the
neural network described by (1)-(3) has a similar structure,
but the weights connected to the hidden layer are
constrained and adapted indirectly by learning dilation
and translation parameters.
As we know, Gabor wavelets are nonorthogonal. Gabor
function is selected here as the mother wavelet based on
the following reasons. First, although there are many
orthogonal wavelet bases, the isotropic (or symmetric),
compactly supported and orthogonal wavelets do not exist.
Second, according to the uncertainty principle about the
time-frequency resolution, a Gaussian or modulated
Gaussian provides the optimal tradeoff between time
localization and frequency resolution. However, Gauss
function does not satisfy the admissibility condition and
can not provide lateral inhibition. Gabor function is a
modulated Gaussian, which not only has optimal joint
time-frequency resolution [ti],but also introduces lateral
inhibition. Third, Gabor function has been proved to be a
very good model for the primary visual information
processing. Furthermore, our purpose of using the
wavelets is to extract useful features, but not to reconstruct
signals.
It is well known that wavelet transforms are good at
representing signals with time-frequency localized
features. The neural network proposed here would be
suitable for the classification of signals or patterns with
time-frequency localized characteristics.

3. Learning algorithms

-0.5
-5

Learning algorithms are developed in order to obtain

optimal weights for classification and wavelets for
extracting features from a particular type of signals. In the
case when continuous parameters are used, a learning
algorithm can be derived out based on the gradient-descent
method. This kind of learning algorithm may not converge
when there exist oscillatory functions in the neural
network formulation. We try to combat this problem by
using discrete wavelets and learning them by a
constructive algorithm.

Figure I.Gabor function and lateral inhibition

Note that the weights connected to the hidden layer are

generated by dilation and translation of a Gabor function

3.1 For receptive fields with continuous

1157

parameters

If the wavelet parameters a, and b, take continuous

values, a back-propagation base algorithm can be used to
train the neural network defined by (1)-(3), although
trivial modification on the rules for updating weights and
parameters should be made. We can derive the updating
rules as follows:

3.2 For receptive fields with discrete parameters

Firstly, the wavelet parameters are constrained to take
discrete values by the following equations:
a, =ao6, = n,b,a,"

(14)
=nlboal

In this way, the receptive fields are reformed as follows:

where m, ,n, E Z . Instead of training a, and b, directly

by error back-propagation, they can be calculated
according to (14) if we can determine the values of
a,,b,,m,, andn,. The formula for determining these
parameters are derived in the following.
h,, (t) will define sets of windows in a wavelet filter
bank. For brevity, we just consider a particular set of
windows defined by wavelet parameters a and b,
corresponding to a,,b,,m, andn. In the discrete domain,
the windows indexed by m and n and defined by h,, (t) in
the time-frequency plane can be written as follows:

[nb,af + aft* - afAh,nb,a," +aft*+ afAh)

a* A, a' A,
x[---+--)
af a f ' a r a,

In the above equations, d, represents the desired output,

y p is the output of the Ith node in the hidden layer, andE
denotes the error used in the back-propagation.
To keep the receptive fields laterally inhibited, the

mother wavelet function has to be oscillatory across its

zero points, as shown in Fig. 1. This will make the error
function E highly nonconvex. Therefore, local minima are
expected in the learning process, or the learning process
may not converge. This is the major shortage to be
overcome in training this kind of neural networks. We will
resolve this problem by using discrete wavelets in the
following subsection.

1158

where, f* and A,, are the centre and the radius of the
mother wavelet h(t), respectively, while "and A, are
those of the Fourier transform i ( w ) , respectively.
Actually, t * , A h , w ' , a d Ah can be determined by
w, and CT which are parameters in the Gabor function.
Selecting suitable values for parameters a, and bo is
important. From the point of view of function
reconstruction, we hope that suitable selection of a, and
bo can make h,(t) constitute a tight frame [3]. On the
other hand, from the point of view of filter banks, we hope
that the windows defined by h,, (t) can properly cover the
interesting areas in the time-frequency plane. Because we
are interested in extracting useful time-frequencylocalized
features, we will select the values of a, and bo from the
point of view of filter banks. If a, and bo are not properly
selected, the windows will be too sparse to cover the whole

time-frequency plane and the information in signals will

be lost; or the windows will be overlapped and the
information extracted is redundant. From (16) we note that
the windows in the frequency domain are not influencedby
the translation parameter. Let us consider two adjacent
windows corresponding to m and m+l. In order that the
windows can cover the whole frequency domain, while
having no window overlapped, the following equality
should be satisfied:

In the time domain, the situation is more complicated,

because the windows are affected by both dilation and
translation parameters and the number of adjacent
windows are more than two. However, we only pay
attention to the selection of bo in the time domain. We
consider two adjacent windows with index m unchanged.
In this case, the windows in time domain should satisfy:

( n + l)boar+ a r t * - arA,
= nboa," + art" + a,"Ah ,
Le., bo = 2Ah

(18)

Although (18) does not give an ideal selection of bo, it

provides a simple and satisfactory solution.
Now let us consider how to determine m and n. As we
know, the centre of the window defined by h,(t) in the
frequency domain is w' / (~zu,"),and that in the time
domain is nb,a,"+a,"t'. If the maximum and the
minimum frequencies of the input signals are f,, and
f,, (f,, f 0) ,respectively, and the maximum length of
input signals is T,, ,then the following inequalitiesshould
be satisfied

w*
fmin

5%

or

t8
xa'*
-- >n>---

bo

a30

t*

bo

(if bo c0)
(23)

Because the values of m and n are limited by (2 1)-(23),

we can even take into account all the integer values of m
and n in the whole ranges given above and calculate the
output values of the corresponding wavelet filters. Those
m and n which result in output values larger than the
given thresholds will be selected. Note that the number of
wavelet filters, or the number of hidden nodes, are
automatically determined by learning.
Given a set of training signals, we can get satisfactory
wavelet parameters using the above algorithm. It is able to
avoid the problem of unconvergence which exists in the
continuous wavelet parameter adjustment as described in
(4)-( 13). It should be noted that the weights connected to
the output layer are still trained using the backpropagation algorithm.

4. Simulation studies
To test the classification performance of the neural
network with laterally inhibited receptive fields and the
corresponding learning algorithms, we apply the network
to ECG signal classification. The hidden layer consists of
three subnetworks, each receiving one period of the ECG
signal respectively. All the outputs of the subnetworks are
connected to an output layer where the final classification
is accomplished. The network size is set as follows:
NOL=4, NIL=460x3, and the size of the hidden layer is
determined by the learning algorithm. Three subnetworks
are needed because generally at least three periods of ECG
signals are required for diagnosing some cardiac disease
[5]. Because the page length of the paper is limited, the
details of the network architecture are not fully discussed
here. The input-output function of the network is given as
follows:

f max

0 5 nb,a," + a r t * 5 T,,
Hence, the ranges of m and
follows:

(20)
should be constrained as

1159

i = 42,. .....,NOL
where, k is the subnetwork index. In the case of applying
continuous wavelets, a, and b, are directly trained
according to (4)-(13). If discrete wavelets are used,
a, and b, are calculated according to (14), with
ao,bo,m, undn, determined by (17)-(18) and (21)-(23).

The ECG signals used in our simulation are from the

MIT/BIH arrhythrma database. We have selected Normal,
Bundle Branch Block, Infarct, and Premature Ventricular
Contraction waves, each consistingof 200x3 periods. They
are divided into a training set and a testing set. Some
typical examples of ECG signals are shown in Fig. 2. We
note that the ECG signals are mostly flat and their high
frequency components are localized to short time intervals.
For this kind of signals, time-frequency localized
representationssuch as wavelet transforms are very useful.

Normal

Bundle Branch Block

Infarct

training phase. The differences between the ECG wave

and the centres can be used as inputs to the network.
With discrete wavelets, in order to make the Gabor
function satisfy the admissibility condition of wavelet
transforms, its parameters are set as follows: o2= 8, and
W , =5.3. According to (17) and (18), uo=1.14 and
bo = 3.28. We select integer values of mkl and nk, from
the ranges given by (2 1)-(23), only those corresponding to
the wavelet filters that obtain output values larger than the
given thresholds are maintained to form the receptive
fields, which extract useful time-frequency localized
features of the input signal for classification. The number
of wavelet filters selected in each subnetwork by the
learning
procedure
is
as
follows:
NHL,= 15, NHL, = 16, M L , = 15.
After learning, the classification performance of the
network trained with discrete wavelets is tested. The
simulation result is given in the row marked as NN-LI2 in
Table 1. In order to make a comparison, the network
trained with continuous wavelets is also investigated in the
simulation studies. The corresponding classification
performance is given in the row marked as NN-LI1 in
Table 1. We note that it often happens for the network with
continuous wavelets to sink into local minima or not to
converge. The performance of a standard BP network with
a similar structure is also given in Table 1. It is shown that
the neural networks with laterally inhibited receptive fields
achieve better recognition rate than the BP network. We
note that the neural network proposed in this paper and the
BP network have a similar structure and operate in the
same way in the testing phase, although their weights are
trained by different algorithms. Therefore, the comparison
here is reasonable. Although the ECG signals in the testing
set are corrupted by noise and fake peaks, and have never
been seen by the network before testing, the recognition
rate is considerably high.

Premature Ventricular Contraction

Figure 2. Typical ECG signals

Before entering the network, the ECG waves are

preprocessed. The preprocessing includes the detection of
peaks of R-waves and the normalization which makes the
amplitudes of ECG signals minus their mean values
change from -1 to +l. Three periods of ECG waves are
directly inputted into the network at the same time. The
peaks of the R-waves are located at the centres of the three
subnetworks respectively. To further improve the
classification performance, a centre for each class of ECG
waves can be obtained by a clustering algorithm in the

1160

Net type I Normal B.B.B I Infarct I P.V.C

BP
I 85% 88.75% I 99% I 98%
NN LI1 I 92.5% 1 95% 198.75% 90.5%
NN L12 I 100% 96.6% I 98.25% 98.25%

5. Discussions and conclusions

In the simulation studies, Gauss function shaped
receptive fields have also been tested. It is illustrated that
Gabor function shaped receptive fields perform better in
the ECG signal classification. The major difference
between Gauss and Gabor functions lies in that Gabor

function is oscillatory across its zero points and thus

provides lateral inhibition. The smoothing effect of Gabor
function as a receptive field plays a role of removing noise,
while its sharpening ability resulting from the lateral
inhibition can enhance the localized features in signals.
Obviously, the lateral inhibition provided by Gabor
function shaped receptive fields improves the feature
extraction ability of the neural network. We note that
lateral inhibition is important , but it can not be learnt
using BP-type algorithms without any constraints on the
weights in advance.
The oscillatory behavior of the Gabor function makes it
satisfy the admissibility condition of wavelet functions.
The Gabor function shaped receptive fields form a wavelet
filter bank and the hidden layer of the neural network plays
a rule similar to a group of wavelet transforms. What is
more, by learning the wavelet parameters it is possible to
achieve the optimal or sub-optimal result.
From the point of view of the generalization ability of
neural networks, we should be able to figure out
theoretically how the lateral inhibition influences the
performance surface of the neural network. This would be
one of our future research directions.
In this paper, Gabor function shaped receptive fields are
successfully introduced into a neural network for signal
classification. By using discrete wavelets a learning
algorithm is derived such that the problem of
unconvergence caused by the oscillatory property of the
receptivefields can be eliminated. Simulationstudies show
that introducing lateral inhibition into the neural network
is useful to improve its classificationperformance.

References

1161

R.B.Pinter and B.Nabet, ed., Nonlinear Vision:

Determination of Neural Receptive Fields, Function,
and Networks, CRC Press, 1992.
L.Gaudart, J.Crebassa, and J.P.Petrakian, Wavelet
transform in human visual channels, Applied
Optics,vo1.32, no.22, 1993,pp.4119-4127.
LDaubechies, Ten Lectures on Wavelets, Society for
Industrial and Applied Mathematics, Philadelphia,
1992.
H.Szu, B.Telfer, and S.Kadambe, Neural network
adaptive wavelets for signal representation and
classification, Optical Engineering, vo1.3 1, 110.9,
1992, pp.1907-1916.
[51 Q.Gan, J.Yao, H.-C.Peng, and J.-Z.Zhou, Wavelet
neural network for ECG signal classification, Proc.
of Int. Conf. on Biomedical Engineering, Hong
Kong, 1996.
H.Dickhaus and H.Heinrich, "Classifying biosignals
with wavelet networks, IEEE Engineering in
Medicine and Biology Magazine, vo1.15, no.5, 1996,
pp. 103-111.
171 Y.M.Akay, M.Akay, W.Welkowitz, and J.Kostis,
Noninvasive detection of coronary artery disease
using wavelet-based fuzzy neural networks, IEEE
Engineering in Medicine and Biology Magazine,
vo1.13, 110.5, 1994, pp.761-764.
J.Ben-Arie and K.R.Rao, Nonorthogonal signal
representation by Gaussians and Gabor functions,
E E E Trans. on Circuits and Systems, 11, vo1.42,
no.6, 1995, pp.402-413.