PERCEPTRON

Perceptron: Image input System

Perceptron: Image Sensor

Perceptron: adaptive weight rack

Perceptron: Basic Operation

The single node computes a weighted sum of the input elements and the threshold, and passes the result through a hard limiting nonlinearity such that the output is either +1 or -1.
Original perceptron convergence procedure for adjusting weights was developed by R. Rosenblatt (Principles of neurodynamics, Sparton Books, 1959.

Perceptron: Basic Neuron

Perceptron: Highlights
Applicable to both continuous and binary input Perceptron forms two decision regions separated by a hyperplane Decides whether an input belong to one of two classes (+1 or -1) The equation for the boundary depends on the connection weights and the threshold.

Perceptron: Decision

Perceptron: Linear Separable Function

Perceptron: Image Recognition System

Perceptron:Image Recognition Neuron

Perceptron:Learning Procedure

Perceptron:Learning Procedure
Step 1: Initialize Weights (W1..Wn) and Threshold (W0)
Set all weights and threshold to small bipolar random values ().

Step 2: Present New Input and Desired Output

Present input vector x1, x2, .....xn along with the desired output d(t). Note:** x0 is a fixed bias and always set equal to 1.
** d(t) takes the value of 1 or 0 and 1.

Perceptron:Learning Procedure
Step 3: Calculate Actual Output [y(t)]
n

y(t) = Fh[ wi(t) * xi(t) ]

i=0 where Fh (e) = 1 when e > 0, and = -1 when e <= 0

Step 4: Adapt Weights

wi(t+1) = wi(t) + * [d(t) - y(t)] * xi
where

is the learning rate and usually is a small number ranging from 0 to 1. Note: wi(t+1) = wi(t) if d(t) = y(t)

0<i<n

and

Perceptron:Learning Procedure
Step 5: Repeat step 2 to 4
Repeat until the desired outputs and the actual network outputs are all equal for all the input vectors of the training set.

Perceptron:Example
Train a Perceptron to recognize the following:
X2

X1 -1 -1 +1 +1

X2 -1 +1 -1 +1

Desire O/P +1 +1 +1 -1

x1

Perceptron:Example (Cont.)
+1

X1 X2

W0 W1 W2

OUTPUT

Step 1: set all weights and threshold to small bipolar random values:
Assume: W0 = 0.1, W1 = -0.2, W2=0.15

Perceptron:Example (Cont.)
Excitation (Line Equation):
0.1-0.2X1+0.15X2 = 0
X2

Perceptron:Example (cont.)
Step 2: Present new input and desired output
Lets apply input (-1,-1) and desired output = +1

Step 3: Calculate for actual output, y(t)

Excitation = 0.1 -0.2*(-1) + 0.15*(-1) = 0.15 >0 Actual output y(t) = +1

Step 4: Adapt Weights

Since y(t) = +1 = desired o/p, No weight change!

Step 5: If not all actual outputs = desired outputs,

then Repeat Step 2 to 4 otherwise Stop

Perceptron:Example (cont.)
Repeat : Step 2: Present new input and desired output
Lets apply input (-1,+1) and desired output = +1

Step 3: Calculate for actual output, y(t)

Excitation = 0.1 -0.2*(-1) + 0.15*(+1) = 0.45 >0 Actual output y(t) = +1

Step 4: Adapt Weights

Since y(t) = +1 = desired o/p, No weight change!

Perceptron:Example (cont.)
Repeat : Step 2: Present new input and desired output
Lets apply input (+1,-1) and desired output = +1

Step 3: Calculate for actual output, y(t)

Excitation = 0.1 -0.2*(+1) + 0.15*(-1) = -0.25 < 0 Actual output y(t) = -1

Step 4: Adapt Weights, let = 0.05

W0(new) = 0.1 + 0.05*(1-(-1))*1 = 0.2 W1(new) = -0.2 + 0.05*(1-(-1))*1 = -0.1 W2(new) = 0.15 + 0.05*(1-(-1))*-1 = 0.05

Perceptron:Example (Cont.)
Excitation (Line Equation):
0.2-0.1X1+0.05X2 = 0
X2

Perceptron:Example (cont.)
Repeat : Step 2: Present new input and desired output
Lets apply input (+1,+1) and desired output = -1

Step 3: Calculate for actual output, y(t)

Excitation = 0.2 -0.1*(+1) + 0.05*(+1) = 0.15 > 0 Actual output y(t) = +1

Step 4: Adapt Weights, = 0.05

W0(new) = 0.2 + 0.05*(-1-(+1))*1 = 0.1 W1(new) = -0.1 + 0.05*(-1-(+1))*1 = -0.2 W2(new) = 0.05 + 0.05*(-1-(+1))*1 = -0.05

Perceptron:Example (Cont.)
Excitation (Line Equation):
0.1-0.2X1-0.05X2 = 0
X2

Perceptron:Example (cont.)
Repeat : Until the desired outputs and the actual
network outputs are all equal for all the input vectors of the training set.
X2

x1

Multioutput Perceptron

Multioutput Perceptron

Example training set, with desired responses

Performance: Root Mean Square (RMS) error

Example learning curve, showing the performance of a two layer perceptron on the trainning set. = 0.1 for the training session, with initial weight values were random (0.05 to 0.05)

(d
p j

jp

y jp
o

nn
p

np = # of patterns in the trainning set no = # of units in the output layer

djp = desired output yjp = actual output p = pattern # j = neuron #

Thoughts on Perceptron
Rosenblatt proved that if inputs presented from the two classes are separable (they fell on opposite sides of some hyperplane), then the perceptron convergence procedure converges and positions the hyperplane between those two classes. One problem with the perceptron convergence procedure is that decision boundaries may oscillate continuously when inputs are not separable and distributions overlap.

Thoughts on Perceptron
The value of , learning rate is usually set below 1, and determines the amount of correction make in a single iteration. The overall learning time of the network is affected by . Learning is usually slower for small values and faster of larger values. The training is global, i.e. the network learns over the entire set of data
How the set should be presented to minimize the training time? Apply sequentially? Over and over? or Random?

2- Layer Perceptron
Adjustable Weights

Input Layer

Output Layer

Perceptron:XOR?
Train a Perceptron to recognize the following:
X2

X1 -1 -1 +1 +1

X2 -1 +1 -1 +1

Desire O/P -1 +1 +1 -1

x1 ?

3- Layer Perceptron:Variation A
Non-Adjustable Weights Adjustable Weights

Outputs

Input Layer

Feature Detector Layer**

Output Layer

** have randomized weights that are fixed, not adapted

3- Layer Perceptron: Variation B

Adjustable Weights Non-Adjustable Weights

Outputs

Input Layer

Hidden Layer

Output Layer

Convex Decision Region Produced by 3 layer Perceptron

X
W11 W12 W21

S1 S1

W22

S2

S2

Convex Decision Region Formed by Intersection of Two Convex Regions

Triangle Y Triangle X

Non-Convex Region Y and not X A B C