Professional Documents
Culture Documents
in
Fundamentals of Neural Networks : Soft Computing Course Lecture 7 14, notes, slides
in
forward network, multi layer feed-forward network, recurrent
e.
networks. Learning methods in neural networks : unsupervised
Learning - Hebbian learning, competitive learning; Supervised
ub
www.csetube.in
www.csetube.in
Soft Computing
Topics
(Lectures 07, 08, 09, 10, 11, 12, 13, 14 8 hours)
Slides
1. Introduction 03-12
in
Single layer Feed-forward network, Multi layer Feed-forward network,
Recurrent networks. e.
ub
4. Learning Methods in Neural Networks 24-29
8. References : 40
02
www.csetube.in
www.csetube.in
in
The artificial neural networks are made of interconnecting artificial
e.
neurons which may share some properties of biological neural networks.
ub
03
ww
www.csetube.in
www.csetube.in
SC - Neural Network Introduction
1. Introduction
04
in
e.
ub
et
cs
w.
ww
www.csetube.in
www.csetube.in
SC - Neural Network Introduction
1.1 Why Neural Network
The conventional computers are good for - fast arithmetic and does
what programmer programs, ask them to do.
in
The von Neumann machines are based on the processing/memory
e.
abstraction of human information processing.
ub
The neural networks are based on the parallel architecture of
et
biological brains.
cs
05
www.csetube.in
www.csetube.in
SC - Neural Network Introduction
1.2 Research History
The history is relevant because for nearly two decades the future of
Neural network remained uncertain.
McCulloch and Pitts (1943) are generally recognized as the designers of the
first neural network. They combined many simple processing units together
that could lead to an overall increase in computational power. They
suggested many ideas like : a neuron has a threshold level and once that
level is reached the neuron fires. It is still the fundamental way in which
ANNs operate. The McCulloch and Pitts's network had a fixed set of weights.
Hebb (1949) developed the first learning rule, that is if two neurons are
active at the same time then the strength between them should be
increased.
in
In the 1950 and 60's, many researchers (Block, Minsky, Papert, and
e.
Rosenblatt worked on perceptron. The neural network model could be
ub
proved to converge to the correct weights, that will solve the problem. The
weight adjustment (learning algorithm) used in the perceptron was found
et
more powerful than the learning rules used by Hebb. The perceptron caused
cs
Minsky & Papert (1969) showed that perceptron could not learn those
ww
The neural networks research declined throughout the 1970 and until mid
80's because the perceptron could not learn certain important functions.
06
www.csetube.in
www.csetube.in
SC - Neural Network Introduction
1.3 Biological Neuron Model
in
and somas), muscles, or glands.
Axon hillock is the site of summation
e.for incoming information. At any
ub
moment, the collective influence of all
neurons that conduct impulses to a given
et
Myelin Sheath consists of fat-containing cells that insulate the axon from electrical
ww
activity. This insulation acts to increase the rate of transmission of signals. A gap
exists between each myelin sheath cell along the axon. Since fat inhibits the
propagation of electricity, the signals jump from one gap to the next.
Nodes of Ranvier are the gaps (about 1 m) between myelin sheath cells long axons
are Since fat serves as a good insulator, the myelin sheaths speed the rate of
transmission of an electrical impulse along the axon.
Synapse is the point of connection between two neurons or a neuron and a muscle or
a gland. Electrochemical communication between neurons takes place at these
junctions.
Terminal Buttons of a neuron are the small knobs at the end of an axon that release
chemicals called neurotransmitters.
07
www.csetube.in
www.csetube.in
SC - Neural Network Introduction
Information flow in a Neural Cell
The input /output and the propagation of information are shown below.
in
Fig. Structure of a neural cell in the human brain
e.
Dendrites receive activation from other neurons.
ub
et
output activations.
cs
www.csetube.in
www.csetube.in
SC - Neural Network Introduction
1.4 Artificial Neuron Model
Input1
Input 2
Output
Input n
in
A processing unit sums the inputs, and then applies a non-linear
e.
activation function (i.e. squashing / transfer / threshold function).
ub
An output line transmits the result to other neurons.
et
In other words ,
cs
09
www.csetube.in
www.csetube.in
SC - Neural Network Introduction
1.5 Notations
Z = X + Y = (x1 + y1 , x2 + y2 , . . . . , xn + yn)
in
i=1
10 e.
ub
et
cs
w.
ww
www.csetube.in
www.csetube.in
SC - Neural Network Introduction
Matrices : m x n matrix , row no = m , column no = n
in
n
elements cij = aik bkj e.
k=1
ub
a11 a12 b11 b12 c11 c12
x =
a21 a22 b21 b22 c21 c22
et
cs
11
www.csetube.in
www.csetube.in
SC - Neural Network Introduction
1.6 Functions
.8
1 if x 0 .6
sgn (x) =
.4
0 if x < 0
.2
0
-4 -3 -2 -1 0 1 2 3 4 I/P
in
Sign(x)
1
O/P e.
ub
.8
1 .6
et
sigmoid (x) =
-x
1+e
cs
.2
w.
0
-4 -3 -2 -1 0 1 2 3 4 I/P
ww
12
www.csetube.in
www.csetube.in
SC - Neural Network Artificial Neuron Model
2. Model of Artificial Neuron
- A processing unit sums the inputs, and then applies a non-linear activation
Input 1
Input 2
Output
in
Input n
e.
Simplified Model of Real Neuron
ub
(Threshold Logic Unit)
et
of 1 to n inputs is written as
w.
n
Output = sgn ( Input i - )
i=1
ww
- Non-linear summation,
- Smooth thresholding,
- Stochastic, and
13
www.csetube.in
www.csetube.in
SC - Neural Network Artificial Neuron Model
2.2 Artificial Neuron - Basic Elements
x1 W1 Activation
Function
x2 W2
i=1
xn Wn
Threshold
Synaptic Weights
Weighting Factors w
in
The values w1 , w2 , . . . wn are weights to determine the strength of
e.
input vector X = [x1 , x2 , . . . , xn]T. Each input is multiplied by the
ub
T
I = X .W = x1 w1 + x2 w2 + . . . . + xnwn = xi wi
i=1
w.
Threshold
ww
14
www.csetube.in
www.csetube.in
SC - Neural Network Artificial Neuron Model
Threshold for a Neuron
The total input for each neuron is the sum of the weighted inputs
to the neuron minus its threshold value. This is then passed through
the sigmoid function. The equation for the transition in a neuron is :
a = 1/(1 + exp(- x)) where
x = ai wi - Q
i
in
Activation Function
e.
An activation function f performs a mathematical operation on the
ub
signal output. The most common activation functions are:
et
15
www.csetube.in
www.csetube.in
SC - Neural Network Artificial Neuron Model
2.2 Activation Functions f - Types
Over the years, researches tried several functions to convert the input into
an outputs. The most commonly used functions are described below.
- I/P Horizontal axis shows sum of inputs .
- O/P Vertical axis shows the value the function produces ie output.
- All functions f are designed to produce values between 0 and 1.
Threshold Function
A threshold (hard-limiter) activation function is either a binary type or
a bipolar type as shown below.
in
I/P
1 if I 0
Y = f (I) = e.0 if I < 0
ub
I/P 1 if I 0
Y = f (I) =
ww
-1 -1 if I < 0
16
www.csetube.in
www.csetube.in
SC - Neural Network Artificial Neuron Model
Piecewise Linear Function
This activation function is also called saturating linear function and can
have either a binary or bipolar range for the saturation limits of the output.
The mathematical model for a symmetric saturation function is described
below.
in
17
e.
ub
et
cs
w.
ww
www.csetube.in
www.csetube.in
SC - Neural Network Artificial Neuron Model
Sigmoidal Function (S-shape function)
The nonlinear curved S-shape function is called the sigmoid function.
This is most common type of activation used to construct the neural
networks. It is mathematically well behaved, differentiable and strictly
increasing function.
in
1 for large +ve values, with
a smooth transition between the two.
e.
is slope parameter also called shape
ub
asymptotic values.
18
www.csetube.in
www.csetube.in
SC - Neural Network Artificial Neuron Model
Example :
The neuron shown consists of four inputs with the weights.
x1=1 +1 Activation
Function
x2=2 I
+1
y
X3=5 -1
Summing
xn=8 Junction
+2 =0
Threshold
Synaptic
Weights
in
+1
e. +1
I = XT . W = 1 2 5 8 = 14
ub
-1
et
+2
cs
= (1 x 1) + (2 x 1) + (5 x -1) + (8 x 2) = 14
y (threshold) = 1;
ww
19
www.csetube.in
www.csetube.in
SC - Neural Network Architecture
3. Neural Network Architectures
- The edges may represent synaptic links labeled by the weights attached.
Example :
e5
V1 V3
V5
e2
e4
in
e5
V2
e3
V4 e.
ub
Vertices V = { v1 , v2 , v3 , v4, v5 }
cs
Edges E = { e1 , e2 , e3 , e4, e5 }
w.
ww
20
www.csetube.in
www.csetube.in
SC - Neural Network Architecture
3.1 Single Layer Feed-forward Network
w11 y1
x1
w21
w12
w22 y2
in
x2
w2m e.
w1m
ub
wn1
wn2
et
xn wnm
ym
cs
Single layer
w.
Neurons
ww
21
www.csetube.in
www.csetube.in
SC - Neural Network Architecture
3.2 Multi Layer Feed-forward Network
Input Output
hidden layer hidden layer
weights vij weights wjk y1
w11
x1 v11
w12
v21 y1 y2
x2 w11
v1m
v2m y3
vn1 w1m
V m
ym
in
Hidden Layer
x e.
neurons yj
yn
Input Layer
ub
neurons xi Output Layer
neurons zk
et
- The input layer neurons are linked to the hidden layer neurons; the
weights on these links are referred to as input-hidden layer weights.
- The hidden layer neurons and the corresponding weights are referred to
the first hidden layers, m2 neurons in the second hidden layers, and n
output neurons in the output layers is written as ( - m1 - m2 n ).
The Fig. above illustrates a multilayer feed-forward network with a
configuration ( - m n).
22
www.csetube.in
www.csetube.in
SC - Neural Network Architecture
3.3 Recurrent Networks
Example :
y1
x1
y1 y2
Feedback
x2 links
Yn
ym
in
X e.
ub
Input Layer Hidden Layer Output Layer
neurons xi neurons yj neurons zk
et
23
www.csetube.in
www.csetube.in
SC - Neural Network Learning methods
4. Learning Methods in Neural Networks
The learning methods in neural networks are classified into three basic types :
- Supervised Learning,
- Gradient descent,
- Competitive and
in
- Stochastic learning.
24
e.
ub
et
cs
w.
ww
www.csetube.in
www.csetube.in
SC - Neural Network Learning methods
Classification of Learning Algorithms
Fig. below indicate the hierarchical representation of the algorithms
mentioned in the previous slide. These algorithms are explained in
subsequent slides.
Neural Network
Learning algorithms
in
Least Mean Back
Square Propagatione.
ub
25
cs
w.
ww
www.csetube.in
www.csetube.in
SC - Neural Network Learning methods
Supervised Learning
- A teacher is present during learning process and presents expected
output.
- Every input pattern is used to train the network.
improved performance.
Unsupervised Learning
- No teacher is present.
in
features in the input patterns.
Reinforced learning
e.
ub
- A teacher is present but does not present the expected or desired output
et
- A reward is given for correct answer computed and a penalty for a wrong
w.
answer.
ww
Note : The Supervised and Unsupervised learning methods are most popular
forms of learning compared to Reinforced learning.
26
www.csetube.in
www.csetube.in
SC - Neural Network Learning methods
Hebbian Learning
Hebb proposed a rule based on correlative weight adjustment.
In this rule, the input-output pattern pairs (Xi , Yi) are associated by
the weight matrix W, known as correlation matrix computed as
n
W= Xi YiT
i=1
T
where Yi is the transpose of the associated output vector Yi
27
in
e.
ub
et
cs
w.
ww
www.csetube.in
www.csetube.in
SC - Neural Network Learning methods
Gradient descent Learning
This is based on the minimization of errors E defined in terms of weights
and the activation function of the network.
- If Wij is the weight update of the link connecting the i th and the j th
Wij = ( E / Wij )
in
Note : The Hoffs Delta rule and Back-propagation learning rule are
the examples of Gradient descent learning.
e.
ub
28
et
cs
w.
ww
www.csetube.in
www.csetube.in
SC - Neural Network Learning methods
Competitive Learning
- In this method, those neurons which respond strongly to the input
Stochastic Learning
- In this method the weights are adjusted in a probabilistic fashion.
in
e.
ub
et
cs
w.
ww
www.csetube.in
www.csetube.in
SC - Neural Network Systems
5. Taxonomy Of Neural Network Systems
AM (Associative Memory)
Boltzmann machines
in
BSB ( Brain-State-in-a-Box)
Cauchy machines
e.
ub
Hopfield Network
et
Neoconition
w.
Perceptron
ww
30
www.csetube.in
www.csetube.in
SC - Neural Network Systems
Classification of Neural Network
A taxonomy of neural network systems based on Architectural types
and the Learning methods is illustrated below.
Learning Methods
in
Table : Classification of Neural Network Systems with respect to
e.
learning methods and Architecture types
ub
31
et
cs
w.
ww
www.csetube.in
www.csetube.in
SC - Neural Network Single Layer learning
6. Single-Layer NN Systems
w11 y1
x1
w21
w12
w22 y2
x2
w2m
w1m
wn1
in
wn2
xn wnm
e. ym
ub
Single layer
Perceptron
et
cs
1 if net j 0 n
y j = f (net j) = where net j = xi wij
ww
i=1
0 if net j < 0
32
www.csetube.in
www.csetube.in
SC - Neural Network Single Layer learning
Learning Algorithm : Training Perceptron
The training of Perceptron is a supervised learning algorithm where
weights are adjusted to minimize error when ever the output does
not match the desired output.
If the output is 1 but should have been 0 then the weights are
decreased on the active input link
K+1 K
i.e. W = W . xi
ij ij
If the output is 0 but should have been 1 then the weights are
in
increased on the active input link
i.e. W
K+1
= W
K
+ . xi
e.
ij ij
ub
Where
et
K+1 K
W is the new adjusted weight, W is the old weight
cs
ij ij
33
www.csetube.in
www.csetube.in
SC - Neural Network Single Layer learning
Perceptron and Linearly Separable Task
S1 S2 S1
S2
in
(a) Linearly separable patterns (b) Not Linearly separable patterns
e.
ub
Note : Perceptron cannot find weights for classification problems that
et
www.csetube.in
www.csetube.in
SC - Neural Network Single Layer learning
XOR Problem :
Exclusive OR operation
X2
Input x1 Input x2 Output
(0, 1) (1, 1)
0 0 0
1 1 0 Even parity
0 1 1
1 0 1 Odd parity (0, 0) X1
(0, 1)
XOR truth table
Fig. Output of XOR in
X1 , x2 plane
- There is no way to draw a single straight line so that the circles are on
one side of the line and the dots on the other side.
- Perceptron is unable to find a line separating even parity input
in
patterns from odd parity input patterns.
e.
35
ub
et
cs
w.
ww
www.csetube.in
www.csetube.in
SC - Neural Network Single Layer learning
Perceptron Learning Algorithm
The algorithm is illustrated step-by-step.
Step 1 :
Iterate through the input patterns Xj of the training set using the
n
weight set; ie compute the weighted sum of inputs net j = xi wi
i=1
for each input pattern j .
Step 4 :
in
1 if net j 0 n
y j = f (net j) =
e. where net j = xi wij
i=1
0 if net j < 0
ub
Step 5 :
If all the input patterns have been classified correctly, then output
w.
Step 6 :
Otherwise, update the weights as given below :
www.csetube.in
www.csetube.in
SC - Neural Network ADALINE
6.2 ADAptive LINear Element (ADALINE)
x1 W1
x2 W2
Output
Neuron
in
xn Wn
e.
Error
ub
et
+
cs
Desired Output
w.
www.csetube.in
www.csetube.in
SC - Neural Network ADALINE
ADALINE Training Mechanism
(Ref. Fig. in the previous slide - Architecture of a simple ADALINE)
in
scalar value.
Usage of ADLINE :
In practice, an ADALINE is used to
- Make binary decisions; the output is sent through a binary threshold.
38
www.csetube.in
www.csetube.in
SC - Neural Network Applications
7. Applications of Neural Network
Neural Network Applications can be grouped in following categories:
Clustering:
A clustering algorithm explores the similarity between patterns and
places similar patterns in a cluster. Best known applications include
data compression and data mining.
Classification/Pattern recognition:
The task of pattern recognition is to assign an input pattern
(like handwritten symbol) to one of many classes. This category
includes algorithmic implementations such as associative memory.
Function approximation :
The tasks of function approximation is to find an estimate of the
unknown function subject to noise. Various engineering and scientific
in
disciplines require function approximation.
e.
Prediction Systems:
ub
System may be dynamic and may produce different results for the
same input data based on system state (time).
ww
39
www.csetube.in