ADIGRAT UNIVERSITY

COLLEGE OF ENGINEERING AND TECHNOLOGY

DEPARTMENT OF COMPUTER SCIENCE

Introduction to Neural Network

(Course Code: CoSc4151)
By Gidena M ( MSc in CS)
Target Department: 4th Year-CS-Students

Chapter-3: Designing and Training an ANN based System

 Designing and Training an ANN based System
Network Design & Training Issues
 Design:
Architecture of network
 Determines number of network weights
 How many layers and nodes per layer?
 Structure of artificial neurons
 Connectivity and choice of activation (transfer) function
Learning rules and algorithms
 Generalized delta rule (steepest descent) and Momentum descent
 Training:
Ensuring optimum training
Learning parameters
 learning rate, momentum and weight-cost
Data preparation
 The quality of results relates directly to quality of the data(70%).
Consolidation, cleaning, selection and preprocessing and Transformation
 BP algorithm accept only continuous numeric values (0 - 1 range). Four basi
data types(nominal,
Neural Network By Gidena ordinal, interval and continous) 12/3/2018
2
 Designing an ANN based System

General Steps
 Get the data set
 Determine the network topology
 Training
 Optimization
 Testing
\

3 Neural Network By Gidena 12/3/2018

 Designing an ANN based System …
 Get the data set
 Example
 Determine the characteristics of the customers of a given business institution
(Bank) in terms of credit risks (low credit risk , or high credit risk)
 The data for this problem is all about the attributes of the customers
Age

Income

Debt

Payment record

4 Neural Network By Gidena 12/3/2018

 Designing an ANN based System …
 Determine the network topology
 Number of nodes in the input layer
 Equal to input parameters (independent variables)

 Number of nodes and layers in the Hidden layer

 There is no a clear cutting rule (Optimization Governs)
o Minimum error rate, and
o Less number of hidden nodes Helpful to simplify the hardware
representation of the network
 Number of nodes in the output layer
 Equal to output parameters (dependent variables)
o e.g. Number of class labels

5 Neural Network By Gidena 12/3/2018

 Designing an ANN based System …
 Training
 Dataset
 Determine the training, validation, and testing dataset

 Activation function
 Back propagation algorithm
 Stopping rule
Error rate

 Number of epoch
 Momentum coefficient

6 Neural Network By Gidena 12/3/2018

 Designing an ANN based System …
 Optimization: to produce an optimal NN topology
 Minimum error rate
 Less number of nodes in the hidden layer

 Testing
 Without input noise
 With input noise
If the error rate is within the acceptable error range the
network is ready for use.

7 Neural Network By Gidena 12/3/2018

 Designing an ANN based System …
 General Rules
 Initial network has a randomly assigned weights

 Learning is done by making small adjustments in the weights to reduce the

difference between the observed and predicted values

 Main difference from the logical algorithms is the need to repeat the update
phase several times in order to achieve convergence

 Updating process is divided into epochs

 Each epoch updates all the weights of the process.

8 Neural Network By Gidena 12/3/2018

 Designing an ANN based System …
 Example 1

 Determine the characteristics of the customers of a given business

institution (Bank) in terms of credit risks (low credit risk , or high
credit risk)
 The data for this problem is all about the attributes of the customers
o Age
o Income
o Debt
o Payment record

9 Neural Network By Gidena 12/3/2018

 Designing an ANN based System …
 Example 1…

10 Neural Network By Gidena 12/3/2018

 NN Application Development
Specifically,
 The development process for an ANN application has eight steps.
Step 1: Data collection
 The data to be used for the training and testing of the network are collected.

Step 2: Training and testing data separation

 Training data must be identified, and
 A plan must be made for testing the performance of the network.
 The available data are divided into training and testing data sets.
 For a moderately sized data set, 80% of the data are randomly selected for training,
10% for testing, and 10% secondary testing.

Step 3: Network architecture

 A network architecture and a learning method are selected.
 Important considerations are the exact number of perceptrons and the number of
layers.

11 Neural Network By Gidena 12/3/2018

 NN Application Development…
Step 4: Parameter tuning and weight initialization
 There are parameters for tuning the network to the desired learning
performance level.
 Part of this step is initialization of the network weights and parameters
 Often, the initial values are important in determining the effectiveness
and length of training.
Step 5:Data transformation
 Transforms the application data into the type and format required by
the ANN.
Step 6: Training
 Training is conducted iteratively by presenting input and desired
output data to the ANN.
 The ANN computes the outputs and adjusts the weights until the
computed outputs are within an acceptable tolerance of the known
outputs
12
for
Neural theByinput
Network Gidena cases. 12/3/2018
 NN Application Development…
Step 7: Testing
 Once the training has been completed, it is necessary to test the network.
 The testing examines the performance of the network using the derived
weights by measuring the ability of the network to classify the testing data
correctly.
 Black-box testing (comparing test results to historical results)
 is the primary approach for verifying that inputs produce the appropriate outputs.

Step 8: Implementation Now a stable set of weights are obtained.

 The network is ready to use as a stand-alone system or as part of another
software system where new input data will be presented to it and its output
will be a recommended decision.

13 Neural Network By Gidena 12/3/2018

  Example Applications

 Pattern Recognition (reading zip codes)

 Signal Filtering (reduction of radio noise)

 Data Segmentation (detection of seismic onsets)

 Data Compression (TV image transmission)

 Database Mining (marketing, finance analysis)

 Adaptive Control (vehicle guidance)

14 Neural Network By Gidena 12/3/2018

Some practical Applications

 Handwriting recognition
 Recognizing spoken words
 Face recognition
 ALVINN
 ….
Face Recognition

 Input: 30 by 32 pictures of people with the following

properties:
 Wearing eyeglasses or not
 Facial expression: happy, sad, angry, neutral
 Direction in which they are looking: left, right, up, straight
ahead
 Output: Determine which category it fits into for one of
these properties (we will talk about direction)
 Face Recognition…

90% accurate learning head pose, and recognizing 1-of-20 faces

17 NN 1 Neural Networks
 Handwritten digit recognition

18 NN 1 Neural Networks
ALVINN

 Autonomous Land Vehicle in a Neural Network

 Robotic car
 Created in 1980s by David Pomerleau
 1995
 Drove 1000 miles in traffic at speed of up to 120 MPH
 30 x 32 image as input, 4 hidden units, and 30 outputs
ALVINN…
 Steer a van down the road
 2-layer feedforward
 using backpropagation for learning

 Raw input is 480 x 512 pixel image

 Color image preprocessed into 960 input units

 4 hidden units

 30 output units, each is a steering direction

 ALVINN…
Drives 70 mph on a public highway

30 outputs
for steering
30x32 weights
4 hidden into one out of
units four hidden
unit
30x32 pixels
as inputs
21
 Neural Network Approaches
ALVINN - Autonomous Land Vehicle In a Neural Network
 NETalk (1987)

 Mapping character strings into phonemes so they can be

pronounced by a computer

 Neural network trained how to pronounce each letter in a word in

a sentence,
 Output was the correct phoneme

 Results
 95% accuracy on the training data
 78% accuracy on the test set
 Neural network for OCR

A
 feedforward network B
C
 trained using Back- D
propagation E

Hidden
 OCR-Optical Character Layer Output
Layer
Recognition
Input
24 Layer
 OCR for 8x10 characters

10 10 10

8 8 8

25
 Engine management

• The behaviour of a car engine is influenced by a large

number of parameters
– temperature at various points
– fuel/air mixture
– lubricant viscosity.
• Major companies have used neural networks to
dynamically tune an engine depending on current settings.

26
 Signature recognition

• Each person's signature is different.

• There are structural similarities which are difficult to
quantify.
• One company has manufactured a machine which
recognizes signatures to within a high level of accuracy.
– Considers speed in addition to gross shape.
– Makes forgery even more difficult.

27
 Stock market prediction

• “Technical trading” refers to trading based solely

on known statistical parameters; e.g. previous
price
• Neural networks have been used to attempt to
predict changes in prices.
• Difficult to assess success since companies using
these techniques are reluctant to disclose
information.

28
 Mortgage assessment

• Assess risk of lending to an individual.

• Difficult to decide on marginal cases.
• Neural networks have been trained to make decisions,
based upon the opinions of expert underwriters.
• Neural network produced a 12% reduction in
delinquencies compared with human experts.

Resource
 Textbook: ”Neural Networks for Pattern Recognition”,
Bishop,
29 C.M., 1996
 Thank You !!!

30 Neural Network By Gidena 12/3/2018