Professional Documents
Culture Documents
in
Fundamentals of Neural Networks : Soft Computing Course Lecture 7 14, notes, slides
multi
layer
feed-forward
e.
forward
in
recurrent
Hebbian
learning,
competitive
ub
Learning
learning;
Supervised
et
cs
learning
ww
layer
w.
for
training
perceptron,
architecture,
clustering,
and
training.
classification,
Applications
pattern
www.csetube.in
of
recognition,
neural
function
www.csetube.in
8 hours)
Slides
03-12
1. Introduction
in
ub
e.
Recurrent networks.
24-29
Competitive
Unsupervised
et
algorithms:
learning;
Supervised
cs
Learning
Learning
Learning
Hebbian
Learning,
Stochastic
learning,
w.
ww
6. Single-Layer NN System
39
40
References :
02
www.csetube.in
www.csetube.in
to
simulate
its
learning
process.
An
artificial
neural
network
in
e.
ub
et
ww
03
w.
cs
www.csetube.in
www.csetube.in
SC - Neural Network Introduction
1. Introduction
Neural Computers mimic certain processing capabilities of the human brain.
- Neural
in
biological
systems
involves
adjustments
to
the
synaptic
connections that exist between the neurons. This is true of ANNs as well.
ww
w.
cs
et
ub
e.
in
04
www.csetube.in
www.csetube.in
SC - Neural Network Introduction
fault
in
e.
biological
et
ub
cs
brains.
parallel architecture of
w.
ww
www.csetube.in
www.csetube.in
SC - Neural Network Introduction
in
increased.
e.
In the 1950 and 60's, many researchers (Block, Minsky, Papert, and
ub
et
cs
more powerful than the learning rules used by Hebb. The perceptron caused
w.
ww
Minsky & Papert (1969) showed that perceptron could not learn those
functions which are not linearly separable.
The neural networks research declined throughout the 1970 and until mid
80's because the perceptron could not learn certain important functions.
Neural network regained importance in 1985-86. The researchers, Parker
and LeCun discovered a learning algorithm for multi-layer networks called
back propagation that could solve problems that were not linearly
separable.
06
www.csetube.in
www.csetube.in
SC - Neural Network Introduction
are
branching
fibers
that
is
singular
fiber
carries
in
e.
information.
At
any
ub
for
et
cs
w.
ww
Myelin Sheath consists of fat-containing cells that insulate the axon from electrical
activity. This insulation acts to increase the rate of transmission of signals. A gap
exists between each myelin sheath cell along the axon. Since fat inhibits the
propagation of electricity, the signals jump from one gap to the next.
Nodes of Ranvier are the gaps (about 1 m) between myelin sheath cells long axons
are Since fat serves as a good insulator, the myelin sheaths speed the rate of
transmission of an electrical impulse along the axon.
Synapse is the point of connection between two neurons or a neuron and a muscle or
a gland. Electrochemical communication between neurons takes place at these
junctions.
Terminal Buttons of a neuron are the small knobs at the end of an axon that release
chemicals called neurotransmitters.
07
www.csetube.in
www.csetube.in
SC - Neural Network Introduction
in
The input /output and the propagation of information are shown below.
e.
et
ub
output activations.
cs
Synapses
ww
w.
the
junctions
allow
signal
transmission
between
the
neuro-transmitters.
McCulloch-Pitts introduced a simplified model of this real neurons.
08
www.csetube.in
www.csetube.in
SC - Neural Network Introduction
Output
Input n
in
e.
ub
et
cs
In other words ,
w.
ww
www.csetube.in
www.csetube.in
SC - Neural Network Introduction
1.5 Notations
Recaps :
s = x 1 + x2 + x 3 + . . . . + x n =
i=1
xi
(1 x n)
Y = ( y1 , y2 , y3 , . . ., yn )
Z = X + Y = (x1 + y1 , x2 + y2 , . . . . , xn + yn)
Multiply: Two vectors of same length multiplied to give a scalar.
e.
in
p = X . Y = x1 y1 + x2 y2 + . . . . + xnyn =
ww
w.
cs
et
ub
10
www.csetube.in
i=1
xi yi
www.csetube.in
SC - Neural Network Introduction
Matrices : m x n matrix ,
row no = m , column no = n
w11
w11
. .
. .
w1n
w21
w21
. .
. .
w21
. .
. .
. .
. .
. .
. .
wmn
W =
wm1 w11
Add or Subtract :
component by component.
a11 a12
a21 a22
b11
b12
b21
b22
A+B =C,
cij
aij + bij
c11 = a11+b11
c12 = a12+b12
C21 = a21+b21
b11
b12
b21
b22
c11
c12
c21
c22
(a11 x b11)
(a12 x B21)
c12
(a11 x b12)
(a12 x B22)
C21
(a21 x b11)
(a22 x B21)
C22
(a21 x b12)
(a22 x B22)
w.
c11
ww
matrix C.
(m x p)
e.
cs
a21 a22
aik bkj
et
a11 a12
k=1
ub
cij =
elements
in
Multiply : matrix A
11
www.csetube.in
www.csetube.in
SC - Neural Network Introduction
1.6 Functions
The Function y= f(x) describes a relationship, an input-output mapping,
from x to y.
Threshold or Sign function :
sgn(x) defined as
Sign(x)
O/P
.8
1
sgn (x) =
if x 0
0 if x < 0
.6
.4
.2
0
-4
-3
-2
-1
4 I/P
.8
1
-x
12
ww
w.
cs
1+e
et
.6
sigmoid (x) =
O/P
ub
e.
Sign(x)
in
.2
0
-4
-3
-2
www.csetube.in
-1
4 I/P
www.csetube.in
SC - Neural Network Artificial Neuron Model
in
Output
e.
Input n
ub
et
cs
where
If
If
i=1
n
i=1
sgn (
ww
Output =
w.
of 1 to n inputs is written as
n
i=1
Input i
- )
Input i
then Output = 1
Input i
<
then Output = 0
www.csetube.in
www.csetube.in
SC - Neural Network Artificial Neuron Model
x1
W1
x2
W2
Activation
Function
i=1
xn
Wn
Threshold
Synaptic Weights
in
Weighting Factors w
e.
ub
et
cs
i=1
xi wi
ww
Threshold
w.
I = X .W = x1 w1 + x2 w2 + . . . . + xnwn =
= f{
i=1
xi wi - k }
www.csetube.in
www.csetube.in
SC - Neural Network Artificial Neuron Model
to the neuron minus its threshold value. This is then passed through
the sigmoid function. The equation for the transition in a neuron is :
a = 1/(1 + exp(- x))
x =
where
ai wi - Q
ai
wi
is the weight
in
e.
Activation Function
ub
et
cs
- Linear Function,
w.
- Threshold Function,
ww
www.csetube.in
www.csetube.in
SC - Neural Network Artificial Neuron Model
- O/P Vertical axis shows the value the function produces ie output.
- All functions f are designed to produce values between 0 and 1.
Threshold Function
A threshold (hard-limiter) activation function is either a binary type or
a bipolar type as shown below.
I/P
-1
cs
0 if I < 0
w.
O/p
if I 0
et
bipolar threshold
ub
Y = f (I) =
in
O/p
e.
binary threshold
ww
I/P
-1
Y = f (I) =
if I 0
-1 if I < 0
www.csetube.in
www.csetube.in
SC - Neural Network Artificial Neuron Model
Piecewise Linear
O/p
-1
+1
I/P
-1
Y = f (I) =
I 0
if
if -1 I 1
if
I < 0
e.
in
-1
ww
w.
cs
et
ub
17
www.csetube.in
www.csetube.in
SC - Neural Network Artificial Neuron Model
is
called
O/P
2.0
=
=
1.0
Y = f (I) =
1+e
0.5
0.5
This is explained as
0 for large -ve input values,
in
-2
, 0 f(I) 1
-4
- I
e.
ub
et
The sigmoidal
cs
function is
achieved
using
exponential equation.
w.
ww
www.csetube.in
www.csetube.in
SC - Neural Network Artificial Neuron Model
Example :
The neuron shown consists of four inputs with the weights.
x1=1
+1
x2=2
+1
-1
X3=5
xn=8
Activation
Function
Summing
Junction
+2
=0
Threshold
Synaptic
Weights
in
+1
ub
cs
et
I = XT . W =
e.
+1
-1
= 14
+2
(1 x 1) + (2 x 1) + (5 x -1) + (8 x 2) = 14
w.
19
ww
y (threshold) = 1;
www.csetube.in
www.csetube.in
SC - Neural Network Architecture
number
of
simple
highly
interconnected
processing
elements
Example :
e5
V3
V5
e2
e4
in
V1
V4
ub
e3
e.
e5
V2
et
w.
E = { e1 , e2 , e3 , e4, e5 }
ww
20
cs
Vertices V = { v1 , v2 , v3 , v4, v5 }
Edges
as
www.csetube.in
www.csetube.in
SC - Neural Network Architecture
of
a single layer of
series of weights. The synaptic links carrying weights connect every input
to every output , but not other way. This way it is considered a network of
feed-forward type. The sum of the products of the weights and the inputs
is calculated in each neuron node, and if the value is above some threshold
(typically 0) the neuron fires and takes the activated value (typically 1);
otherwise it takes the deactivated value (typically -1).
input xi
output yj
weights wij
w11
x1
y1
w21
w12
y2
e.
in
w22
x2
w2m
ub
w1m
wn1
et
wn2
ym
wnm
Single layer
Neurons
ww
w.
cs
xn
www.csetube.in
www.csetube.in
SC - Neural Network Architecture
Input
hidden layer
weights vij
w11
v11
x1
x2
w12
y1
v21
y1
y2
w11
v1m
y3
v2m
w1m
ym
V m
Hidden Layer
neurons yj
e.
in
vn1
et
ub
Input Layer
neurons xi
yn
Output Layer
neurons zk
cs
w.
ww
- The input layer neurons are linked to the hidden layer neurons;
the
the first hidden layers, m2 neurons in the second hidden layers, and n
output neurons in the output layers is written as ( - m1 - m2 n ).
The Fig. above illustrates a multilayer feed-forward network with a
configuration ( - m n).
22
www.csetube.in
www.csetube.in
SC - Neural Network Architecture
y1
x1
y2
y1
x2
Yn
ub
e.
in
ym
Feedback
links
Hidden Layer
neurons yj
et
Input Layer
neurons xi
Output Layer
neurons zk
cs
w.
ww
www.csetube.in
www.csetube.in
SC - Neural Network Learning methods
and
- Reinforced Learning
in
- Competitive and
- Stochastic learning.
ww
w.
cs
et
ub
e.
24
www.csetube.in
www.csetube.in
SC - Neural Network Learning methods
Supervised Learning
(Error based)
Error Correction
Gradient descent
Least Mean
Square
Unsupervised Learning
Hebbian
ub
e.
Back
Propagation
in
Stochastic
Reinforced Learning
(Output based)
et
ww
w.
cs
25
www.csetube.in
Competitive
www.csetube.in
SC - Neural Network Learning methods
Supervised Learning
- A teacher is present during learning process and presents expected
output.
- Every input pattern is used to train the network.
- Learning process is based on comparison, between network's computed
improved performance.
Unsupervised Learning
- No teacher is present.
- The expected or desired output is not presented to the network.
in
e.
ub
Reinforced learning
et
- A teacher is present but does not present the expected or desired output
cs
ww
answer.
w.
- A reward is given for correct answer computed and a penalty for a wrong
Note : The Supervised and Unsupervised learning methods are most popular
forms of learning compared to Reinforced learning.
26
www.csetube.in
www.csetube.in
SC - Neural Network Learning methods
Hebbian Learning
Hebb proposed a rule based on correlative weight adjustment.
In this rule,
are associated by
where Yi
There
are
i=1
Xi YiT
many
variations
of
this
rule
ww
w.
cs
et
ub
e.
in
27
www.csetube.in
proposed
by
the
other
www.csetube.in
SC - Neural Network Learning methods
differentiable, because
the
updates
of
is required to be
weight is dependent on
th
and the j
th
Wij = (
E /
Wij )
in
gradient
e.
Note : The Hoffs Delta rule and Back-propagation learning rule are
ub
ww
w.
cs
et
28
www.csetube.in
www.csetube.in
SC - Neural Network Learning methods
Competitive Learning
- In this method, those neurons which respond strongly to the input
Stochastic Learning
- In this method the weights are adjusted in a probabilistic fashion.
- Example
: Simulated
ww
w.
cs
et
ub
e.
in
29
www.csetube.in
www.csetube.in
SC - Neural Network Systems
the
previous
sections,
the
Neural
Network
Architectures
and
the
Learning methods have been discussed. Here the popular neural network
systems are listed. The grouping of these systems in terms of architectures
and the learning methods are presented in the next slide.
in
Boltzmann machines
ub
Cauchy machines
e.
BSB ( Brain-State-in-a-Box)
et
Hopfield Network
w.
Perceptron
ww
Neoconition
cs
www.csetube.in
www.csetube.in
SC - Neural Network Systems
based on
Architectural types
Hebbian
Competitive
Stochastic
Single-layer
feed-forward
ADALINE,
Hopfield,
Percepton,
AM,
Hopfield,
LVQ,
SOFM
Multi-layer
feed- forward
CCM,
MLFF,
RBF
Neocognition
RNN
BAM,
BSB,
Hopfield,
ART
Boltzmann and
Cauchy
machines
in
Recurrent
Networks
e.
ub
ww
w.
cs
et
31
www.csetube.in
www.csetube.in
SC - Neural Network Single Layer learning
6. Single-Layer NN Systems
Here, a simple Perceptron Model and an ADALINE Network Model is presented.
6.1 Single layer Perceptron
Definition :
output yj
weights wij
w11
x1
y1
w21
w12
w22
x2
y2
w2m
w1m
in
wn1
e.
wn2
xn
ym
ub
wnm
cs
et
Single layer
Perceptron
ww
y j = f (net j) =
w.
Fig.
if net j
if net j
0
< 0
where net j =
32
www.csetube.in
i=1
xi wij
www.csetube.in
SC - Neural Network Single Layer learning
i.e.
ij
ij
If the output is 1
= W
ij
ij
If the output is 0
. xi
but should have been 1 then the weights are
in
K+1
i.e.
= W
ij
ij
+ . xi
ub
K+1
i.e.
ij
is
is
w.
xi
ww
33
ij
cs
et
Where
K+1
e.
www.csetube.in
www.csetube.in
SC - Neural Network Single Layer learning
separates
the sets.
Example
S1
S1
S2
in
S2
ub
e.
ww
w.
cs
34
et
www.csetube.in
www.csetube.in
SC - Neural Network Single Layer learning
XOR Problem :
Exclusive OR operation
X2
Input x1
0
1
0
1
Input x2
Output
0
1
1
0
(0, 1)
0
0
1
1
Even parity
Odd parity
(0, 0)
(1, 1)
X1
(0, 1)
one side of the line and the dots on the other side.
in
e.
ww
w.
cs
et
ub
35
www.csetube.in
even parity
input
www.csetube.in
SC - Neural Network Single Layer learning
x0 = 1
i=1
xi wi
Step 4 :
if net j
if net j
Step 5 :
yj
net j
i=1
xi wij
yj
cs
et
where
< 0
ub
e.
1
y j = f (net j) =
in
for
w.
If all the input patterns have been classified correctly, then output
Step 6 :
ww
i= 0, 1, 2, . . . . , n
i= 0, 1, 2, . . . . , n
goto step 3
END
36
www.csetube.in
www.csetube.in
SC - Neural Network ADALINE
its
weights
are
determined
by
the
normalized
least
mean
square (LMS) training law. The LMS learning rule is also referred to
delta rule.
as
x1
W1
x2
W2
Output
xn
in
Neuron
e.
Wn
+
Desired Output
w.
cs
et
ub
Error
ww
www.csetube.in
www.csetube.in
SC - Neural Network ADALINE
is
similar
to
a linear neuron
X = [x1 , x2 , . . . , xn]
the
input
vector
to the network.
The weights are adaptively adjusted based on delta rule.
After
network
performs
an
the
function
weights
are
is
not
used
ub
Once
activation
e.
scalar value.
The
dimensional
mapping
to
in
Thus,
properly
during
adjusted,
the
the
training
phase.
response
of
the
the
responses
training
to
set.
cs
in
high
w.
not
et
degree
the
with
the
test
produces
inputs,
consistent
it
is
said
ww
network
www.csetube.in
www.csetube.in
SC - Neural Network Applications
Clustering:
Classification/Pattern recognition:
The
task
of
pattern
(like
handwritten
recognition
symbol)
to
one
is
to
assign
of
many
an
classes.
input
This
pattern
category
Function approximation :
The
tasks
of
in
The
task
is
to
forecast
ub
Prediction Systems:
some
et
function approximation.
e.
disciplines require
future
values
of
time-sequenced
cs
w.
ww
www.csetube.in