Neural Networks

a comprehensive foundation

Lecture Yiping Duan
QQ 499151150
Email yipingduan@126.com
 Neural Networks
a comprehensive foundation

(a course for graduates in fully English)

2
 Textbook and references
Textbook:
Simon Haykin, Neural Networks, a comprehensive foundation,
Prentice-Hall, Inc., 1999, Second Edition
References
Robert J. Schalkoff, Pattern Recognition: Statistical, Structural and
Neural Approaches, Clemson Univ.
C.M.Bishop, Neural Networks for Pattern Recognition, Oxford univ.
press (1995)
T. Kohonen, Self Organizing Maps, Springer (2001)
Bart Kosko, Neural Networks and Fussy Systems, A dynamical systems
approach to machine intelligence, Pretice-Hall, 1992
1993
1994

2001
3
2000
 outlines
Chapter I Chapter VII

Introduction Support Vector Machine (SVM)

Chapter II Chapter II
The task Self Organizing Maps
Chapter III Chapter III
Perceptron Restricted Boltzmann Machine
Chapter IV Chapter IV
Multilayer Perceptron (MLP) Some deep models
Chapter V Chapter V
K-means Summary
Chapter VI
Radial-Basis Function (SVM)

4
5
6
7
8
9
 Neural Networks
a comprehensive foundation

Chatper 1 Introduction
 Neural Networks
a comprehensive foundation

Section 1.1 Why NNs?

 outlines
Main difference between brain and computer?

Artificial neural networks

Supervised learning and unsupervised learning

Benefits of neural networks

Applications
 Comparison between brain and
computer
Brain Computer
Biological Integrated circuit
Composed of a huge number Composed of CPU
of neurons Sequential processing
Parallel processing Not fault tolerant
Fault tolerant Fast and precise computation
Fast but not precise Not learnable or trainable
computation Not adaptable to environment
Learnable or trainable
Adaptable to environment
 Introduction
What is neural computing/neural networks?

The brain is a remarkable computer.

It interprets imprecise information from the senses at an incredibly high

speed.

A good example is the processing of visual information: a one-year-old baby

is much better and faster at recognising objects, faces, and other visual
features than even the most advanced AI system running on the fastest super
computer.

Most impressive of all, the brain learns (without any explicit instructions) to
create the internal representations that make these skills possible
 Introduction
(Artificial) Neural networks are computational models which
mimic the brain's learning processes.
They have the essential features of neurons and their interconnections as
found in the brain.
Typically, a computer is programmed to simulate these features.
Other definitions [S. Haykin, Neural Networks a comprehensive
foundation, 2nd edition, 1999, p.2]
A neural network is a massively parallel distributed processor made up of simple
processing units, which has a natural propensity for storing experimental
knowledge and making it available for use. It resemble the brain in two respects:
Knowledge is acquired by the network from its environment through a learning
process
Interneuron connection strengths, known as synaptic weights, are used to store the
acquired knowledge.
 Artificial Neural Networks
Artificial neural network models (or simply neural networks) are typically composed of
interconnected units or artificial neurons. How the neurons are connected depends on some
specific task that the neural network performs.

Two key features of neural networks distinguish them from any other sort of computing
developed to date:

Neural networks are adaptive, or trainable

Neural networks are naturally massively parallel

These features suggest the potential for neural network systems

capable of learning,
autonomously improving their own performance,
adapting automatically to changing environments,
being able to make decisions at high speed
and being fault tolerant.
 Main Types of ANN
Supervised learning
- Feed-forward Layered ANN
- Multi-Layer Perceptron ( with sigmoid hidden neurons)
- Radial Basis Functions (gaussian, wavelets)
- Support vector machine

Unsupervised learning
Self organizing ANN
Kohonen topgraphic maps
SOMself organizing map
Recurrent neural network
Hopfield neural networks
 Supervised Learning
Typically:
Backprop
of errors error = yi - ti

Training set: xi yi
{(xi, ti); -
i=1,2,,N}

desired output
(supervisor)
ti
 Unsupervised Learning
Typically:
minimizing
system
energy energy f ( yi )

Training set: xi yi
{xi;
i}
 Which for What ?
ANN
EC
Artificial neural networks

Learning Optimizing Evolution computation

Capability Capability
Every combi is
possible and used:
Goal is to realize
processing systems
FL Representing
with greater
Fuzzy Logic Capability
intelligence
 ANN a good choice if:

Data-rich / model-deficient problem

Failure of classical mathematical modeling
Nonlinear, multidimension input/output mapping
Failure of classical linear methods (try it first)
Enough time to design the final ANN
Hours to days to get a ANN
 Benefits of neural networks
Nonlinearity VLSI implementability
Nonlinearity is distributed throughout of This is the result of the massively
the network
parallel nature of a neural network
Input-output mapping
Uniformity of analysis and design
Learning from teacher
Learning by its self organizing Neural network is composed of
neurons
Adaptivity
Adaptable to the surrounding It is an universal information
environment processor
Evidential response Neurobiological analogy
Give confidence in the decision making The design of a neural network is
process motivated by analogy with the brain
Contextual information it is helpful to biological analysis
Knowledge is stored in the very structure
and/or as a tool for information
and activation state of a neural network
processing
 Some real ANN usages
Recognition of
hand-written letters
driver license plates
seals
signatures
Error detection
Fabric error detection and
classification
Predicting on-line the quality of
welding spots
Online monitor of the VLSI
manufacturing
Automatic target recognition
Airplane recognition
Warship recognition
Attack effect evaluation
Biometric recognition
Speech/speaker recognition
Iris recognition
Face detection/recognition
Palm print recognition
Visualizing
high-dimensional space visualization
Control system
To learn the controller such that the
performance will become better and
better
Traffic management
Medical image processing
Stock-market prediction
Navigation of a car
etc
 Representing World Events
Handwritten characters
 Representing World Events
Human faces ??
 Representing World Events
1 2 3
A toy example
i 4
6 5

x2(i) 7
x1(i) 9 10
x1 11
12
X(i) =[x1(i), x2(i)] 13
x1(i) 14 8

15 16

x2 Elliptical blobs (objects)

x2(i)
 Representing World Events
x1
X(15) 1 2 3
X(1) X(7) 4
X(16) 6 5
X(3) X(8) 7
X(25) X(12) 9 10
12 11
X(9) X(10)X(13)X(6) 14
13
8
X(4) X(11) 15 16
X(14)
Elliptical blobs (objects)
x2
A toy example
 Neural Networks
a comprehensive foundation

Section 1.2 Structure of a Neuron

 Biological neurons
A typical neuron collects signals from
other neurons through a host of fine
structures called dendrites.

The neuron sends out spikes of

electrical activity through a long, thin
strand known as an axon, which splits
into thousands of branches.

At the end of the branch, a structure

called a synapse converts the activity
from the axon into electrical effects that
inhibit or excite activity in the connected
Axon neurons.

Synapse When a neuron receives excitatory

Dendrites
Schematic drawing of two biological
neurons connected by synapses
input that is sufficiently large compared
with its inhibitory input, it sends a spike
of electrical activity down its axon.
 The brain is composed of approximately
100 billion neurons

Learning occurs by changing

the effectiveness of the synapses
so that the influence of one
neuron on the other changes
 Artificial Neural Networks
Artificial neural network models (or simply neural networks) are
typically composed of interconnected units or artificial neurons. How
the neurons are connected depends on some specific task that the
neural network performs.

Two key features of neural networks distinguish them from any other
sort of computing developed to date:

Neural networks are adaptive, or trainable

Neural networks are naturally massively parallel

These features suggest the potential for neural network systems

capable of learning, autonomously improving their own performance,
adapting automatically to changing environments, being able to make
decisions at high speed and being fault tolerant.
 Models of a neuron
Perceptron
 perceptron
Composition of a perceptron
A set of synapses or connecting links
An adder
An activation function unit
bk
x1
wk 1
x2 wk 2
vk yk
()
wkn
xn
 Activation function of a
perceptron
bk
x1
wk 1
x2 wk 2
vk yk
()
wkn
xn

m m
y k ( wkj x j bk ) ( wkj x j )
j 1 j 0

(v ) {10 v0
v 0
 Activation function of a perceptron

(v)

-0.5 0.5 v

1 vk 0 1 vk 0.5 1
(vk ) (vk )
0 otherwise (vk ) v otherwise
0 v 0.5 1 exp(ax)
k

1 vk 0 (v)

(vk ) 0 vk 0
1 vk 0
v
 bk
x1
wk 1
x2 wk 2
vk yk
()
wkn
xn

bk
x1
wk 1
x2 wk 2 - yk
()
vk
+
wkn
xn

The model of a perceptron: threshold explanation

 Functions of a perceptron
Binary linear classification
Linearly separable problem and nonlinearly
separable problem
Binary classifier with a perceptron
Input space Input space
 A perceptron as a binary linear
classifier / a simple nonlinear mapper
Input space Input space

Binary classifier
 Regression and Classification
Problem statement
Given: A set of examples {xn},
n = 1N, x d and
corresponding targets {tn},
n = 1N
Aim: To learn the functional
relationship of the targets with
the inputs with the aim to make
accurate predictions over unseen
data x
t = f (x, )

Regression: t is real valued

Classification: t is discrete valued
(class labels)
 Applications of NNs: image filtering

Original Noisy
image image

Conventional NN
method method
 Applications of NNs: image enhancement

Original
image
NN
method
Original
image
NN
method
 Q&A
Thank you for your
attention

04/01/17 47