Surface Defect Detection With Neural Net PDF

Univerza v Ljubljani
FAKULTETA ZA Neural Network 2011/12

STROJNIŠTVO
Seminar:
Surface defect detection with Neural

networks
PD-KM Žiga Gosar 17.09.13

Datoteka: seminar_object_detection, Zadnja sprememba: 17.09.13
Seminar: Surface defect detection with Neural networks
CONTENT
LIST OF FIGURES ................................................................................................................................................... 4
Abstract...................................................................................................................................................................... 5
1 INTRODUCTION ............................................................................................................................................ 6
1.1 SURFACE DEFECT DETECTION .................................................................................................................... 7
2 ARTIFICIAL NEURAL NETWORK (ANN) .................................................................................................. 8
2.1 A PERCEPTRON NEURON .............................................................................................................................. 8
2.2 A SINGLE LAYER PERCEPTRON ................................................................................................................... 9
2.3 MULTI-LAYER FEED-FORWARD NEURAL NETWORK ............................................................................ 10
2.4 CREATING IT ................................................................................................................................................. 10
2.4.1 The easy way................................................................................................................................................ 10
2.4.2 Input and outputs explained ......................................................................................................................... 10
2.4.3 Transfer function ......................................................................................................................................... 12
2.4.4 Initalization ................................................................................................................................................. 13
2.4.5 Creating it: the hard way ............................................................................................................................. 13
2.5 TRAINING IT .................................................................................................................................................. 15
2.6 TESTING AND SIMULATION ....................................................................................................................... 16
2.7 COMPUTING THE ERROR ............................................................................................................................ 18
2.8 GENERATING TRAIN SETS .......................................................................................................................... 18
2.9 IM2VEC.M ...................................................................................................................................................... 21
3 FEATURE EXTRACTION ............................................................................................................................ 22
3.1 GRAY-LEVEL FEATURES............................................................................................................................. 22
3.1.1 Contrast and Brightness............................................................................................................................... 23
3.2 HISTOGRAM EQUALIZATION ..................................................................................................................... 23
3.3 ILLUMINATION EFFECT .............................................................................................................................. 24
3.3.1 Background illumination model ................................................................................................................... 24
3.3.2 Adaptive histogram equalization .................................................................................................................. 24
3.3.3 Normalization .............................................................................................................................................. 25
3.4 GABOR FEATURE EXTRACTION ................................................................................................................ 25
3.4.1 2D Gabor wavelet representation................................................................................................................. 25
3.4.2 Why gabor features ...................................................................................................................................... 28
3.4.3 How to perform gabor filtering .................................................................................................................... 28
3.4.4 Fast gabor filtering ...................................................................................................................................... 28
3.4.5 IM2VEC.M .................................................................................................................................................. 29
4 DEFECT DETECTION SYSTEM ................................................................................................................. 31
4.1 PRE-SELECTION ............................................................................................................................................ 31
4.2 SEARCH ALGORITHM .................................................................................................................................. 32
4.3 THE SECOND PASS ....................................................................................................................................... 34
Žiga Gosar Stran 2/54

4.4 POST-PROCESSING ....................................................................................................................................... 34

5 FUTURE WORK............................................................................................................................................ 37
5.1 PRE-SELECTION ............................................................................................................................................ 37
5.2 FAST CONVOLUTIONAL ANN ..................................................................................................................... 37
5.3 FEATURE SELECTION .................................................................................................................................. 37
5.4 CLASSIFICATION .......................................................................................................................................... 37
6 CONCLUSION ............................................................................................................................................... 38
7 REFERENCES ............................................................................................................................................... 39
8 APPENDIX ..................................................................................................................................................... 40
8.1 USER’S GUIDE & PROGRAM’S DESCRIPTION........................................................................................... 40
8.2 MATLAB CODE ............................................................................................................................................. 45
8.2.1 create_gabot.m ............................................................................................................................................ 45
8.2.2 createffnn.m ................................................................................................................................................. 45
8.2.3 drawrec.m .................................................................................................................................................... 46
8.2.4 gabor.m ....................................................................................................................................................... 46
8.2.5 im2vec.m ..................................................................................................................................................... 47
8.2.6 imscan.m...................................................................................................................................................... 48
8.2.7 loadimages.m ............................................................................................................................................... 50
8.2.8 main.m ......................................................................................................................................................... 52
8.2.9 mminmax.m.................................................................................................................................................. 53
8.2.10 trainnet.m................................................................................................................................................ 54
8.2.11 Defect folder ........................................................................................................................................... 54
8.2.12 Non-defect folder .................................................................................................................................... 54

LIST OF FIGURES
Figure 1: A perceptron neuron .........................................................................................................8
Figure 2: A single layer perceptron................................................................................................... 9
Figure 3: A vector-like single layer perceptron ................................................................................. 9
Figure 4: Hard-limit transfer function ............................................................................................. 10
Figure 5: Log-sigmoid transfer function ......................................................................................... 12
Figure 6: Linear transfer function ................................................................................................... 12
Figure 7: Tan-sigmoid transfer function.......................................................................................... 12
Figure 8: Training neural network .................................................................................................. 16
Figure 9: Histogram equalization.................................................................................................... 23
Figure 10: Illumination effect ......................................................................................................... 24
Figure 11: Gabor filters in Time domain ......................................................................................... 26
Figure 12: Example of a facial image response to above Gabor filters ............................................. 26
Figure 13: Gabor wavelets ............................................................................................................. 27
Figure 14: Gabor wavelets form the top view ................................................................................. 27
Figure 15: Sheme of defect detection with neural network in im2vec.m .......................................... 30
Figure 16: Pre-selection of defect-contained window ..................................................................... 32
Figure 17: Scaned image with neural network ................................................................................ 34
Figure 18: Candidate locations for locating defects......................................................................... 35
Figure 19: Treshold image ............................................................................................................. 35
Figure 20: Centroids ...................................................................................................................... 35
Figure 21: Centers ......................................................................................................................... 36
Figure 22: Drawing rectengulars on the centers .............................................................................. 36
Figure 23: Marked defects ............................................................................................................. 36

Abstract
This paper proposes a classification-based object detection method using Gabor filter features.
Considering the desirable characteristics of spatial locality and orientation selectivities of the Gabor
filter, we design filters for extracting object features from the local image. The feature vector based
on Gabor filters is used as the input of the classifier, which is a Feed Forward neural network
(FFNN) on a reduced feature subspace learned by an approach simpler than principal component
analysis (PCA). The effectiveness of the proposed method is demonstrated by the experimental
results on testing a large number of images and the comparison with the state-of-the-art method. The
image will be convolved with Gabor filters by multiplying the image by Gabor filters in frequency
domain. To save time they have been saved in frequency domain before Features is a cell array
contains the result of the convolution of the image with each of the forty Gabor filters. The input
vector of the network will have large values, which means a large amount of computation. So we
reduce the matrix size to one-third of its original size by deleting some rows and columns. Deleting is
not the best way but it save more time compare to other methods like PCA. Object detection and
recognition has many applications in a variety of fields such as quality control, security system,
videoconferencing and identification. Object classification is currently implemented in software. A
hardware implementation will allow us real-time processing but has higher cost and time to-market.
The objective of this work is to implement a classifier based on neural networks (Multi-layer
Perceptron) for object detection. The ANN is used to classify defect and non-defect patterns.
Key Words: Object detection, Gabor wavelet, Gabor Filter, feed forward neural network classifier
(FFNN), Multilayer perceptron, Feature extraction, Artificial Neural Network.

1 INTRODUCTION
Inefficiencies in industrial processes are costly in terms of time, money and consumer satisfaction.
The global economic pressures have gradually led businesses to become more competitive. In order
to sustain or increase current level of performance in the highly competitive global market,
automotive industry should improve quality of the production process. It has been learned that the
price of material is reduced by 45%–65% due to the presence of defects. Early and accurate
detection of defects and classification in injection molded parts is an important aspect of quality
improvement. The accuracy of manual inspection is not good enough due to fatigue and tediousness.
The solution to the problem of manual inspection is the automated, i.e. machine vision based part
inspection system. Automated part inspection systems mainly involve two challenging problems,
namely defect detection and defect classification. Feature selection plays an important role in
developing automated defect classification capability. For an appropriate feature set, the
distinguishing qualities of the features should be high and the number of features should be small.
Moreover, an appropriate set of features takes into account the difficulties that lie in the feature
extraction process [1].
Quality control of production has always been a necessity in automotive industrie. The users of
reflector set ever-increasing requirements on product quality. Many material properties can still only
be measured in laboratory but more and more measurements are now made on-line during the
production. Especially surface defects have to be detected on-line with a surface defect inspection
system because of their random appearance.
The purpose of a surface defect inspection system is to detect and classify surface defects that impair
product quality with regard to the requirements set by the user. The requirements mostly deal with
the suitability of the product to the intended use of it. In the worst case the defects may make the
product functionally deficient or even unusable. Critical defects are also those which cause
production disturbances.
The project of my neural network course is to extract surface defect form images. The aim of the
project is to show my classmates that it is possible to detect defects or objects and make them
familiar with the subject.
Before the neural network lectures I did not know anything about intelligent search methods.
Surface defect detection system is my first serious programming challange that I took to learn
MATLAB. And I learned so much from it. Not only I became a MATLAB progremmer, it is my
entry point to the world of Artificial Neural Networks (ANN) 1.
In this paper I explain the code and the principle to help understand the basics of defect detection (or
generally object detection) using learning algorithms.
In this work, I will work not only on the concept but also I will explain srdefect defect detection
system for MATLAB, line by line. Everything should be in a practical level which can be
implemented and practiced right away.
Defect detection system is not a complete program for oace detection. The best way to learn object
detection is to start with this project. A person can change the parameters, play with them and check
the results. But later he should continue his research on object detection and combine it with
whatever he has learned from his own research.
1
http://en.wikipedia.org/wiki/Artificial_neural_network

1.1 SURFACE DEFECT DETECTION
Flaws are imperfections appearing as local lightness variation with low contrast and blurry contour,
which without fixed area and different shape. The intensity of flaw may be brighter or darker than
background. At present, there are mainly two traits for captured PS Plates images with flaws. One is
the images with repetitive textured backgrounds; the other is pixels with no uniform brightness
within flaw region. In view of these characteristics, flaws are hard to be inspected directly with
traditional edge detection or threshold method [2, 3]. Moreover, flaws can be classified as many as
few hundred types and each type flaw pattern can be inspected with different recipes made by human
eyes and the experiences of inspectors. Therefore, the automatic classification of flaws is also a
crucial task. At present, machine vision works of flaws detection are focused on the image
reconstruction and the background eliminating [4-7]. Principles of these works are removing
influence of background and preserving defects. Then, a threshold is usually used in the
segmentation. The literature [5] used the level set method employed for image segmentation with a
new region-based active contours model. And the detectable rate is about 96%. The literature [7]
proposed a system that can automatically detect four kinds of defect patterns on TFT-LCD.
Defect detection is the first and foremost step in any automated defect recognition system. Its
reliability greatly affects the preformance and usability of the whole system.
Given a single image or a video frame an ideal defect detector should have the ability to locate all the
present defects inside that image, regardless of their position, variations in scale and orientation.
Furthermore, it should be robust against variation in illumination, colour, background…
Several clues max facilitate the detection process. Colour (for defectss in colourful images and
videos) is one that often can be used. Motion (for detecting defectss in videos) is another well-
known clue that can be estimated by subtracting video frames from each other. But the hardest kind
of detection is defect detection in grey-level still images, in which there is no cue of any type like
colour or motion. The processing usually done as follows:
1. an input image is scanned at all possible locations and scales by a sub-window (patch)
2. defect detection classifiers determine the pattern in each sub-window as sither defect or non-
defect. The defect/non-defect classifier has learned hae to classify it according to its one time
training process over a number of defect and non-defect examples using learning models.
So, it becomes simple. First we need to load an image in MATLAB, cut all possible windows or just
crop sub-windows that we think max contain defects. Then we can examine them individually for the
presents of any defect.
There are different methods for that but here we want to focus only on using neural networks and
Gabor feature extraction to classify them.

2 ARTIFICIAL NEURAL NETWORK (ANN)

We can extract defects form an image by using neural networks. I decided to give it a try and see if I
can crack it for my project. The simplest form of the neural network called perceptron.
2.1 A PERCEPTRON NEURON
The perceptron was invented in 1957 by Frank Rosenblatt2 at the Cornell Aeronautical Laboratory.
It can be seen as the simplest kind of feed forward neural network.
Actually it can be seen as a binary classifier which maps its input vector x to a single binary output
value f(x) across the matrix.
1 if w  x  b  0
f ( x)  
0 else
Where w is a vector of weights. All of them are real values. And w.x is the dot product of the input
vector by the weight vector. b is the 'bias', which a constant term that does not depend on any input
value.
Figure 1: A perceptron neuron
Now here is how it works. First we should teach this neuron how to react when it sees and input
pattern. We call this phase, training. When a perceptron is learning something, it means that its
weights (w) and its bias value (b) are changing. After the learning phase, we fix the weights and bias
and the training is over.
When the training phase is over, we can give any input pattern that we want, even if the perceptron
has not seen it before. Regardless of whether it has seen the pattern before or not, it will return a
value. That is amazing and it is exactly what we expect from a neural network. We show it some
patterns and also give it, their correct answers. Then we ask something else and it tries its best to
answer the new unseen pattern correctly. Applications are endless. But the most common
applications is when noise is involved in the process and each captured pattern is different form the
true clean patterns.
To train a perceptron, we have to prepare a training set. The training set is a set of input vectors and
their cesired ouputs (desired targets). They come like pairs. Each input vector has its own target. We
usually show the input vestor with P, and its target with T.
2
http://en.wikipedia.org/wiki/Frank_Rosenblatt

2.2 A SINGLE LAYER PERCEPTRON
Now that we heard about a single perceptron neuron, let us put some of these neurons beside each
other and form a layer. MATLAB is showing the perceptron network consists of a single layer
perceptron neurons like this.
Figure 2: A single layer perceptron
Furthermore, MATLAB tries to compactify it by showing it in a vector-like format.
Figure 3: A vector-like single layer perceptron
In the compactified vector model, R is the number of elements in the input vector. S is the number of
neurons in the layer.
The perceptron network consists of a single layer of S perceptreon neurons connected to R inputs
through a set of weights wi,j. The netwoek indices i and j indicate that wi,j is the potency of the
connection from the j-th input to the i-th neuron. How that we have several neurons instead of one,
we do not have a single weighr vector, now ourweighrs are the elements of matrixand instead of a
single scalar value as the bias, we have a bias vector of size S.
The 'hardlim' function (Figure 4) that we see below a photo is nothing more than out f(x) function
that we descriebed in the previous section. It can be shown like this:

Figure 4: Hard-limit transfer function
2.3 MULTI-LAYER FEED-FORWARD NEURAL NETWORK
It is just several layers of single-layer perceptron neurons bonded to one another. This explains the
term 'multi-layer'. The term 'feed-forward' means on output of any neuron is going to recurrent to
the previous layers of the network.
2.4 CREATING IT
2.4.1 The easy way
Here we do it the easy way. A multi-layer feed-forward neural network can be created with:
net = newff(P,T,[S1 S2 …])
Where P is an R-by-Q matrix of Q input vectors of R elements each and T is an S-by-Q matrix of Q
target vectors of Q elements each.
S1 is the number of neurons in the first layer of the network. In this guide since we want to distinguis
between defects and non-defects we only need one neuron in the last layer which is called usually the
outpur layer.
So, if you want to have 10 neurons in the first layer and 1 output neuron in the output layer, and you
want to create this network, you should write:
net = newff(P,T,[10 1])
2.4.2 Input and outputs explained
The input of a neural network (a multi-layer feed forward), even if it is as simple as a single layer
perceptron is always a vector.
Do not confuse the input of the neural network (small p) with P (capital p). Here is the difference. A
single pattern to be tested or to be fed to the neural network is usually shown by a small 'p' and we
can consider it as a vertical vector (observation) of elements (features). Now, small 't' is called the
target for this pattern. For training the neural network, we must know advace and we need to use it

along with its corresponding pattern for the purpose of training the nework. In this case it is called
the desired output.
In simulation phase, we do not know 't' and we want to ask it from the network in the
simulating/testing phase, in this case, t is called actual output.
It is noteworthy to say that the difference between the actual output of the neural network and the
desired output is called the error, and usually is shown by 'e' and can be computed as a scalar value
for each neuron in the output.
Even if we want to feed a patch of an image, part of an image, some features of any kind, directly to
the neural network we should first convert it to a vertical vector format. We may use the reshape
function in MATLAB.
Now the capital 'P' is a set of patterns which we know their desited target in advance and we want to
train the network with them. P is not a vector. It is a matrix. Each column of the P matrix is
independent from the other columns and is in fact one pattern. The capital 'T' is horizontal vector (in
case we have only one output neuron in the last layer of our neural network) of desired output which
we expect to get from our patterns in P. Each element of the T vector corresponds to the same
column in P matrix.
If we have more than one neuron in the output layer of our neural network our capital 'T' is a matriy
and the number of rows it has is the same as the number of output neurons.
I will explain all this in an example for us to understand them, because it is usually one of the most
problematic areas when people are trying to learn neural networks. I had the same problem, when I
started. MATLAB was keep giving me non-sense to me and I did not know where the problem was,
for the whole week.
Just as a note, MATLAB guides do not defferentiate between small 'p' and capital 'P'. If we are
following the neural network's toolbox guide, just keep that in mind and avoid everything that
involve the word sequence in it. Sequence are for more advanced operations.
Consider that we have 30 images containing defect and 40 images containing non-defect. Each of the
images is of size 27x18. I mean they have a hight of 27 pixels and a width of 18 pixels. So each of
them is actually a matrix and the size of all the matrixes are equal. They cannot be different. We can
prepare our photos in advance.
The first thing we should do is to rechape each of our matrixes to be a vector formate of size 481x1.
Then for the sake of this example consider that our neural network here has only one neuron in its
output layer and the desired output for htat neuron is 0.9 when the input pattern is a defect; and its
desired output is -0.9 when the input pattern is not a defect.
Then we can concate (short for concatenate) all our input vectors together and form a 486x70
matrix as our P matrix. The first 30 columns are the vectors correspond to defects ant the rest belong
to non-defect patterns.
Because we have only one neuron in the output layer of our neural network, T matrix has only one
row and is of size 1x70. The first 30 elements are 0.9 and the 40 next elements are -0.9.

2.4.3 Transfer function
We said that the transfer function inside each perceptron neuron is herdlim. Well, here in multi-layer
feed forward neural networks, we can change the transfer functions and set them for each layer
separately and independent of the other layers.
Here are the most common transfer functions, used in neural networks:
The first one is a logarithm sigmoid:
Figure 5: Log-sigmoid transfer function
The next one is called pure linear:
Figure 6: Linear transfer function
The last one is more popular and is called tangent sigmoid:
Figure 7: Tan-sigmoid transfer function

In order to change the transfer functions, we should announce them when we are creating the
network. Just as an example:
net = newff(P,T,[20 2],{'logsig','tansig'});
In this network we can be sure that no matter what the input, the ourput of the neural network is
bounded between 1 and -1 and can be as close as to them but it can only be 1 and -1 in the infinity or
–infinity.
An importaint note:
Since it can hardly reach to 1 or -1, it is better to chose our desired outputs as 0.9 and -0.9 for defect
and non-defect. Because we will have better and faster convergence in the training phase and we
actually have the chance to see 0 as our training error. But if we choose our desired output for
instance as 1, then it means the input of the transfer function should go as far as infinity to meet our
desired output, which is problematic in some cases and in other cases it may never be feasible.
2.4.4 Initalization
Now that we have created a neural network, we should initalize all its weights and bias values to
random real numbers, because the first moment that qw create a network, every weight is zero. So
no matter what the input, to help the training with the input, to help the training value of the weights
should be uniformly spread. So after creating the neural network, we should always randomize the
weights.
net = init(net);
This line alone, will read the network which is stored in 'net'. Randomize its weights and bias values
for all layers and then save it again in 'net' variable, which is storing the network.
2.4.5 Creating it: the hard way
As we can see in defect detection system for MATLAB, if we open the file 'createffnn.m' we will see
that it is more complicated than what I just explained in that last section, except for the initialization
part which is the last line.
The reason is that by difining the network in a hard way, we can actually control each aspect of the
structure of the network. We can break a link to a specific neuron, define feedbacks and many more
things. It is all explained in the help topics under the name Neural Network Toolbox > Advanced
Topics > Custom Networks.
But here I will just explain them based on the actual lines of the program.
net = network;
This line will define an empty network. It can be any kind of artifficial neural network that MATLAB
supports.
net.numInputs = 1;
net.numLayers = 2;
As the names say, we have one output layer and in the whole network we have two layers. The
second layer is the output layer.

net.biasConnect = [1;1];
This line says that neurons of the first and second layers of the network have adjustable bias inputs
for themselves.
net.inputConnect = [1 ;...
0 ];
This line says that the elements of the input vector are connected to only the first layer and they are
not connected to the output layer.
net.layerConnect = [0 0 ;...
1 0 ];
In this 2x2 matrix, there is only one element which has the value of one. This element means that
there is a connection soming from layer one to layer two. The top-left zero means that there is no
feed-back connection from layer one to itself. The bottom-right zero means, there is no feed-back
connection from the output layer to itself. And finally the top right zero means that no connection is
coming from the output layer to the first layer which is the input layer. Sometimes the outputs can be
feedback to the previous layers (like recurrent neural networks). Note that if any other one of the
zeros had the value of one instead, the network could still work, but it would not be called a feed-
forward network, because it had feedback in it.
net.outputConnect = [0 1];
net.targetConnect = [0 1];
The first line means the output of the simfunction is coming from the output of the second layer. The
second line means, while training the value of the backpropagated value is coming from the output of
the second layer.
netInputs = ones (2160,2);
netInputs (1:2160,1)= -1;
net.inputs{1}.range = netInputs;
Now, here the netInputs is a matrix of size 2160x2. I will explain later why it has 2160, when it
comes to generating the input vectors. But for now just accept this as an explanation that 2160 is the
size of our feature vector which comes from the Gabor Filter Feature Extraction Stage. I will
explain that later.
The first column of the netInputs matrix is -1 and the second column is 1. This means that each
element of our feature vector should be bounded between 1 and -1.
net.layers{1}.size = 100;
The number of neurons in the first layer is 100 and the number of neurons in the output layer is 1. So
for each patch that we are going to feed this network, we get one scalar value, and since the transfer
function of the output neuron is tangent sigmoid, that scalar value is between -1 and 1.
net.layers{1:2}.transferFcn = 'tansig';
net.layers{1:2}.initFcn = 'initnw';
net.initFcn = 'initlay';
The first line indicates that all the neurons in the first and second (1:2) layer of the network have
tangent sigmoid as their transfer functions. The second line means that the initialization function is
Nguyen-Widrow initialization function. The third line, just keep it as it is.
net.performFcn = 'msereg';
net.trainFcn = 'trainscg';
We should pay more attention to these two lines, because they specify the training method and the
performance function that will be used in the training phase. In general there are three performance
functions that can be used; namely msereg which is short for 'Mean squared error with
regularization', mse which is the short for the normal 'Mean squared error'. And msa which is 'Mean
absolute error'. There are aditional sources that we can follow to understand them:

1. MATLAB help centre for the Neural Network Toolbox

2. Neural Networks: A Comprehensive Foundation by Simon Haykin (here3)
2.5 TRAINING IT
The Multy-layer perceptron with the training algorithm of feed propagation is universal mappers,
which can in theory, approximate any continuous decision region arbitrarily well. Yet the
convergence of feed forward algorithms is still an open problem. It is well known that the time cost
of feed forward training often exhibits a remarkable variability. It has been demonstrated that, in
most cases, rapid restart method can prominently suppress the heavy-tailed nature of training
instances and improve efficiency of computation.
Multi-layer perceptron (MLP) with a feed forward learning algorithms was chosen for the proposed
system because of its simplicity and its capability in supervised pattern matching. It has been
successfully applied to many pattern classification problems [8]. Our problem has been considered to
be suitable with the supervised rule since the pairs of input-output are available. For training the
network, we used the classical feed forward algorithm. An example is picked from the training set,
the output is computed.
The training phases are like changing the weights and bias of the neural network and testing it on the
training set that we have. Then it adds small corrections to the initialized weights and again tests it.
To train the neural network we should write:
net = train(net,P,T);
This line simply trains the network, 'net' based on teh P and T inputs and their desired outputs, and
then it puts back the trained network inside 'net'.
But there are some parameters that should be set before we use this line. Most of the parameters
indicate when the training should stop, if the error or the performance function could not converge
to the goal value during the training phase. There is always the option to pause or stop the procedure
manually, but it may happen that we want to automatically train the network and do not push the
stop button.
These parameters are defined in the file 'trainnet.m'. This file is a function that I wrote for the
program.
function NET = trainnet(net,IMGDB)
The first line is the definition of the function and it gets net as the input and prepared P and T data
which are integrated in IMGDB (short for image database).
This line defines the training function. In articles I have read that trainscg which is short for 'scaled
conjugate gradient backpropagation' is the best and fastest training method for our purpose. It is an
iterative training algorithm and uses much less amount of memory compare to other training methods
(e.g. based on levenberg marquardt).
net.trainParam.lr = 0.4;
This is the learning rate and it is the amount of correction to weights from each iteration to the next.
If we choose a grater value for the learning rate, training may converge faster but there is an increase
chance for the training phase to get unstable and never converge to designed output at all.
3
http://search.4shared.com/postDownload/R2rIFx-Z/Neural_Networks_A_comprehensiv.html

net.trainParam.epochs = 400;
The maximum number of training iteration (also called epochs) is 400. After that the training phase
stops.
net.trainParam.show = 10;
This line indicates that after 10 iterations, the result will be updated on the screen for the user.
Experimentally I realised that the larger the number, the faster the training will be. But, we may get
bored if we do not see any activity on the screen for a considerable amount of time.
net.trainParam.goal = 1e-3;
The last parameter defines our goal. It means that the training stops whenever the value of our
choice falls below this value.
Let me add something importaint here as an example. Consider that we have chosen msa as our
performance function. And we have chosen 1e-3 for the goal. It means that after the network has
been trained, if we test each pattern of our training set and compute the difference between the
desired output and the actual output for each pattern and get the absolute value of the results, in a
way that none of the parameters have negative, then the average absolute error becomes less than
0.001. It is not a bad goal when using msereg as our training performance function. The rest of the
function is for extracting the T and P matrix form IMGDB and the actual training command which I
have explained before. Saving the trained network is always a good investment in time.
Figure 8: Training neural network
2.6 TESTING AND SIMULATION
Now that we have trained the neural network, obviosly we need it to work for us. We want to feed it
an input vector and it should be able to classify it based on wath it has learnt. Now there is no talking
about changing the weights or the bias values for each neuron anymore.
The difference between our brain and an artificial neural network is that, we constantly learn from
our environment. We love to study and get feedback from everyone and everything, but in the world
of artifitial neural networks, at least the multi-layer feed-forward backpropagation type that we are
talking about, there is no such thing as an on-going learning.

In most cases when we want to release our application in public or we want to use it in industry. We
usually train it once and it does not matter how much time it takes for it to be trained. Then after
that, we just want to use it. It can happen from time to time that, one wants to update the training set
and perform training again but it seldom happens.
So what we are dealing with here is a well-trained neural network, sitting in MATLAB workspace to
be used.
There are two ways to use it. The first is to give it one pattern and get one scalar value for each
neuron it has in its output layer. The second approach is to give it a bunch of patterns as input vector
in the form of columns of a matrix, just as we have explained in previous section.
Simulating a neural network is easy. We just have to write this command:
a = sim(net,p);
First let us consider that p is actually one pattern (one vertical vector), then if we have only one
neuron in the last layer of our network, 'a' (short for answer) is going to be a scalar value. If we have
more than one neuron in the last layer of our neural network 'a' will be a vector.
Now let us consider that we want to test several input vectors in one pass. All we have to do is to
prepare our data. Of course we can put them in any vairable that we want but here we assume that
we have stored them in matrix P. The command is not really different except that we are doing to get
a matrix instead of a vector. To summarie all that, we compare them in a table.
Table 1: Comparison of dimensions
Typically for distinguishing between defect and non-defect, text and non-text and anything that falls
in this category, we only need one output neuron. This is true as long as we do not want to use
orthogonal cods for the desired output of our network. Just as an example assume that we have one
output neuron and the value of this neuron for defect patterns is 0 and for non-defect pattern is 1 for
logsigmoid neurons (or -1 and 1 for tangent sigmoid). In the case that we want to use orthogonal
codes, we should have two output neurons with tangent sigmoid as their activation functions. For
defect patterns the value of one output neuron is -1 and the second is 1. For non-defect patterns the
values are revered. Here we have an advantage that if the value of two output neurons are both the
same, we can conclude that the network is not sure about the type of the input pattern. Even it can
be added to the training set for clarification.
In defect detection system, I have chosen the high value (near 1) as the representative of defects and
the low value (near -1) as the non-defects. So I have used one output neuron and in each call to the
network, I only ask for simulation of one feature vector; one feature vector at a time. So that means
my answer always will be a scalar value.

2.7 COMPUTING THE ERROR
Consider that we have got the answer of our network and we know the desired output for each
pattern that we have. Now it is time to measure the error of our network. In this way we can actually
say if a training method is working or not.
There are two popular formulas for computing the performance function. The first one is mean
square error (MSE). Assuming that there is only one neuron in our output later, it is computed as
below:
1 Np a
  Oi  Oid 
2
MSE  (1)
Np i 1
where Np is the number of patterns in the train set. Oia is the actual output of the neural network
for the input pattern number i. Oid is the desired output of the network for the pattern number i.
The other popular performance metric is mean absolute error (MAE). Again assuming that there is
only one neuron in our last layer, it is computed as follows:
1 Np a
MAE  
Np i 1
Oi  Oid . (2)
Another performance metric which is not as popular as the last two is maximum absolute error
(MaAE).
MaAE  max Oia  Oid , i  1...Np (3)
Anyway considering that the goal for training a neural network is equal for MaAE and MAE, it is
much harder and even in some cases impossible to reach a goal with MaAE. Thus, if it is not
impossible, then it is very hard. The orher reason why no one is interested in it is because it is not
derivable and most of the optimization techniques cannot find its minimum easily.
Often if one needs to choose MaAE as the performance function of the network, he should go for a
training algorithm based on stochastic search method (like simulated annealing) or population based
training algorithms (like genetic algorithms4 or particle swarm optimization5).
2.8 GENERATING TRAIN SETS
Tje question at one time was, how to create the training set and how it is possible to change the
database to something else like for example faces, birds, sigals, traffic signs…
I created two folders. One is called 'defect' and the other one is 'non-defect'. There are several images
with PNG format in each of them. These images are actually the training sets that I have generated.
As we can see all of the images are 27x18. I searched the internet and I found some of the defect
images and put them in the folder. I added some more defect images to the folder. Finding defects
was not the hard part. Finding non-defect images was usually strange, because a defect is defined,
but a non-defect can be anything. To do this, I started with 5 or 6 random non-defect images.
4
http://en.wikipedia.org/wiki/Genetic_algorithm
5
http://en.wikipedia.org/wiki/Particle_swarm_optimization

First I trained the neural network for the first time and tested it over an image without any defects.
When I say I tested it, it means that I cut every possible 27x18 patches from the image and converted
them into a vector format. Then I gave each vector to the input the trained neural network.
While testing patches, it happed that I got high response for some of them, let's say over 0.9. In
those cases, I got them and put them in the non-defect folder. Then I trained the network for the
second time and did the same procedure again.
One thing is very importaint due. Our training set should always have a balance between the number
of defects and the number of non-defects. Every non-defect image that we add to the training set will
lower the effect of detecting defect images. So it can be considered a challenging task.
When I was doing this boring task of generating the training set, it happened a lot that I added one
non-defect photo to the train data set and the next time I trained the network, it could not detect one
of the previously detected defects.
Now the other way can also happen. We add more defect images to the training set and we find that
it is detecting some defects somewhere it is not supposed to detect anything.
Just to add something importaint. To generate my training data set, I made 10 images out of every
defect image. First of all I generated four other images from each defect image by circular shifting
them one pixel in each direction. The last image was left-to-right flip of the image using fliplr,and I
have done it with any shifted images. Needless to say, we can generate other images. For example
we can rotate some images and add the rotated version too. We have endless possibilities.
Just like this, I made 2 other images of each non-defect image, by flipping them upside down and
flipping them left-to-right. We can do this tricks too if we want to increase the number of samples in
our data sets.
This is a challenging task and it takes a lot of time. Hopefully there are techniques, not only feature
extraction will make the training faster, it also make generating the train sets faster too and the whole
system may become robust against variation of the illumination and contrast.
Consider that we were able to choose our images and we have finalized the train data sets. At this
point on other photo is going to be added or deleted from the training sets and they are ready for
use. When we start the 'defect detection system' program for the first time, we will see some dots. In
that phase the program is generating all the variation of the images that it needs for training and it is
gathering all of them inside a large matrix called IMGDB. This procedure is a one time job only and
the matrix itself will take a considerable size of about 15MB on our hard disc.
The script 'loadimages.m' is responsible for reading the image from the folders and generating the
final input-ready train sets of the network in the workspace under the name of IMGDB. Now it is
time to explain this script here, line by line.
function IMGDB = loadimages
It means that we are dealing with a function without any input arguments.
%~~~~~~~~~~~~~~~
defect_folder = 'defect/';
non_defect_folder = 'non-defect/';
file_ext = '.png';
out_max = 0.9;
out_min = -0.9;
%~~~~~~~~~~~~~~~
Now these lines are pretty clear. They are the folders' names, the extension of the graphic files and
we know all about 0.9 and -0.9.
if exist('imgdb.mat','file')

load imgdb;
else
IMGDB = cell (3,[]);
end
When for the first time the application creates IMGDB, it stores it in a file named 'imgdb.m'. the next
time that we run the application, it will check to see it the file exists or not. If the file exists, it loads
it and if not, it will create an empty IMGDB.
IMGDB is a cell matrix. Cell matrices in MATLAB are different from ordinary matrixes in a sense
that they do not have any computational application. They are used for storage. As we can see, this
cell matrix has three different rows. In each row and column of this cell matrix we can store any type
of data that we want. By data I mean text, another matrix but an ordinary one, a scalar value or that
can be stored in MATLAB.
I am going to store each image in one column of this cell matrix. The first row stores the name of the
image. It may be used for debugging. The second row stores the image in a vector format and the
third row stores the desired output value for that image which can be 0.9 or -0.9.
fprintf ('Loading Defects ');
This is as obvious as it is, and expression to give hope to the user that the application is not dead,
during the loading phase.
folder_content = dir ([defect_folder,'*',file_ext]);
Now, the action begins. Application starts by compiling a list of all the images with the mentioned
extension inside the variable 'folder_content'.
Each image name and its full address are going to be stored in one row of this matrix.
ndefect = size (folder_content,1);
Since we have the list of the defect image files, the size of that vector is equal to the number of
defect images that we have.
for k=1:ndefect
It shows that a for-loop is going to happen and we would like to go through every record that we
have on our file list to read the image fiels one by one.
string = [defect_folder,folder_content(k,1).name];
The name of the file number k is stored in string.
image = imread(string);
The image is read and is stored in a variable called 'image'.
[m n] = size(image);
if (m~=27 || n~=18)
continue;
end
We find the size of the image and if the image is not what we expected, it will pass this image and
will start the loop again with the next image. That is what 'continue' means.
f=0;
for i=1:length(IMGDB)
if strcmp(IMGDB{1,i},string)
f=1;
end
end
if f==1
continue;
end
fprintf ('.');
These lines check tha name of the current image and compate it with the name of all the images that
have been existed on IMGDB till now. If a mach is found, the algorithm will ignore this image and

will go to the next. If not it continues and prints a dot on the screen for the purpose of entertaining
the user.
IM {1} = im2vec (image); % ORIGINAL DEFECT IMAGE
IM {2} = im2vec (fliplr(image)); % MIRROR OF THE DEFECT
IM {3} = im2vec (circshift(image,1));
IM {4} = im2vec (circshift(image,-1));
IM {5} = im2vec (circshift(image,[0 1]));
IM {6} = im2vec (circshift(image,[0 -1]));
IM {7} = im2vec (circshift(fliplr(image),1));
IM {8} = im2vec (circshift(fliplr(image),-1));
IM {9} = im2vec (circshift(fliplr(image),[0 1]));
IM {10} = im2vec (circshift(fliplr(image),[0 -1]));
Now as I told it before, each 10 images is going to be produced from only one. One thing however
remains here which I have to explain with more details in the next section. It is the weird 'im2vec.m'
function. It is not a MATLAB function. Actually it is a function for converting a 27x18 patch (sub-
window image) to a feature vector.
for i=1:10
IMGDB {1,end+1}= string;
IMGDB {2,end} = out_max;
IMGDB (3,end) = {IM{i}};
end
The last lines actually store all this images into IMGDB. The desired output is on the second row
and the third row is the actual feature vectors.
The rest of the file is ecactly the same for reading and appending the non-defect images, except for
the folder's name and the various images that are going to be reproduced from the non-defect image
at hand.
2.9 IM2VEC.M
This function is the most important function in the whole program. It is used more than any other. It
is like an interface between the 27x18 patch and the neural network. Its job is to pass each sub-
window to the network and if there is a feature extraction, it is programmed in this function.
Next chapter is all about it and the methods for feature extraction.

3 FEATURE EXTRACTION
Well, now that we know about how to create a neural network and how to train it, we should focus
on one thing now. This is, how should we prepare our data for the network and what kind of data
should we use?
The 'im2vec.m' function is the answer to all these questions. The input of the function is a 27x18
image patch. The output is a vector which is completely ready to be fed to the neural network. What
happens in between is importaint, because this is the part which is responsible for extracting features
from the image and it has a great influence on the overall preformancce of the software because it
has the highest number of executions.
It is executed in two situations:
- the first is when we want to train the neural network. Before training the network, we
need to generate some data as our training set.
- the second situation is when the training is over and one wants to test the system on
some real images. So consider that we have a large picture of our flaws which I call
'test image'.
Then the simplest approach is to cut every possible 27x18 image patch from it. If we know each sub-
window difined with the pixel in its center, then we have roughly many sub-windows, just a few less
than the number of pixels of the test image. A few less because some of the sub-windows near the
border of the test image are not completely inside and there are not actual data avaliable for them.
3.1 GRAY-LEVEL FEATURES
So the first model that I describe is that the sub-window is reshaped to a vector of size Nx1.
The command is simple:
vector = reshape(subwindow, [486,1]);
It is as simple as that and 486=27x18. To extract all the sub-windows of a test image of size m x n,
we should write:
for y=1:m
for x=1:n
try
subwin=tim(y-13:y+13,x-9:x+8);
catch
continue;
end
.
.
do whatever with subwin
.
end
end
Now there are several things to be described here. First of all subwin is short for sub-window and tim
is short for the test image.
y-13 to y+13 is a range which contains 27 pixels. That is because both of the pixels at the ends are
included, x-9 to x+8 is a range which contains 18 pixels. Catch automatically skips the errors that
occur at the borders if part of the patch is outside of the image borders.

3.1.1 Contrast and Brightness
We can have the same image but contrast means that every pixel of the image is multiplied by a value
(C). And brightness means that every pixel of the image is summed with a value (B). So the equation
is something like this:
new_pixel_value = C  pixel_value + B (4)
and C and B are fixed for all the pixel of the sub-window. As we know the brighter a picture, the
higher is its gray-level values. So changing the gray-level value of the pixels in a linear way means
that we have the same image, same defect, but it is dark or bright and the worst part is that the
neural network is going to see that as two different images, which is not the case. Our brain can
fairly recognize that it is the same image, no matter what the linear equation is.
To solve this problem we need to use histogram equalization.
3.2 HISTOGRAM EQUALIZATION
So in order to neutralize the effect of contrast and brightness, the best tool that can be used in
histogram equalization.
new_subwin = histeq(subwin);
The line above is used to enhance the contrast of images by transforming the values in an intensity
image, so that the histogram of the output image approximately matches a specified histogram.
If we do not specify a histogram, we must specify the number of bins for histogram. It will
automatically compute the histogram using those numbers of bins and will try to equalize the power
of each bins, so that the resultant image has a smooth histogram.
If we do not specify the number of bins, the default value is 64. Here we can compare them:
Figure 9: Histogram equalization
The first two photos are the same. Except that the right image has a 0.8 as its contrast and -40 as its
brightness. Both images are corrected by histogram equalization and as seen in the bottom images,
they are very much the same as each other. There is a difference. The -40 brightness has caused a
problem because some of the values are clipped when they have become negative. We can see the
difference in the output images but for most part, they are the same.

3.3 ILLUMINATION EFFECT
Changing the contrast or brightness actually simulates the change of illumination, but if the light
source changes physically for the subject or for defect, there is a difference here.
In case of changing the light source, the illumination effect in non-linear and is not fixed for the
whole window. This image shows the effect of changing the light source for three subjects.
Figure 10: Illumination effect
There are two ways to correct the problem. The first method is by modeling lighting of the image.
3.3.1 Background illumination model
One of the ways to model background illumination is by averaging the image or window or by
filtering the image using a Gaussian filter.
Then if we subtract the response of the filter from the image, what remains is the object without the
effect of background.
3.3.2 Adaptive histogram equalization
Here is another way to correct this problem which I have used successfully in 'defect detection
system'. It is by using histogram equalization but instead of computing the histogram for the whole
image, we are going to make it adaptive just for a local area of limited pixels.
The command is of course written as:
nww_win = adapthisteq(win,'Numtiles',[8 3]);
And I have chosen the local pixels to be in an area of 8x3. Personally I found it by trial and error.
But we may test other numbers and we may find better results
According to MATLAB, adapthiseq is a contrast-limited adaptive histogram equalization (CLAHE).
We can find it in 'im2vec.m'.

3.3.3 Normalization
We should not forget that each input vector should be normalized to a pre-defined range. The most
popular range is [-1,1] but can be any other range. It is better for it to have symmetry around zero. It
will help for training phase, especially when we are using uniform initialization.
3.4 GABOR FEATURE EXTRACTION
Gabor features are simply the coefficients of the response of Gabor filters. Gabor filters are related to
Gabor wavelets, directly. To understand the concept of Gabor filtering, we must first start with
gabor wavelets. Each gabor wavelet is formed from two components; a complex sinusoidal carrier
placed under a Gaussian envelope.
Thus, apart from the Gaussian envelope in each 2D Gabor wavelet, sinusoidal carrier has a frequency
and orientation of its own and the system is similar to those of human visual system, known as the
visual cortex. Gabor filtering has been found to be appropriate for discrimination and representation
of texture images in particular.
3.4.1 2D Gabor wavelet representation
Since defect recognition is not a difficult task for human beings, selection of biologically motivated
Gabor filters is well suited to this problem. Gabor filters, modeling the responses of simple cells in
the primary visual cortex, are simply plane waves restricted by a Gaussian envelope function [9].
An image can be represented by the Gabor wavelet transform allowing the description of both the
spatial frequency structure and spatial relations. Convolving the image with complex Gabor filters
with 5 spatial frequency (v = 0,…,4) and 8 orientation (μ = 0,…,7) captures the whole frequency
spectrum, both amplitude and phase (Figure 10). In Figure 11, an input face image and the amplitude
of the Gabor filter responses are shown below.
Figure 10 and 11 show one gabor wavelet with a specific frequency and orientation.

Figure 11: Gabor filters in Time domain
Figure 12: Example of a facial image response to above Gabor filters
Figure 11 shows Gabor wavelets family and the result on applying them on a defect image is shown
in the bottom.

Figure 13: Gabor wavelets
The next image is the same Gabor wavelet but it is illustrated from the top just like an image. Note
that the whiter areas have higher wavelets.
Figure 14: Gabor wavelets form the top view
There are many ways to generate Gabor wavelets. In general there are real Gabor wavelets and
compex ones. In my program, I have used the complex one. One way to produce the real wavelet is
described here6 in a very simple form.
I am rewriteing them again in this form:
function kernel = gabor(theta, lambda, gamma, psi)
alpha = 3.5;
sigma = lambda/alpha;
filtersize = floor(lambda/gamma);
[y x] = ngrid(-filtersize:filtersize, -filtersize:filtersize);
ty = x * cos(theta) + y * sin(theta);
tx = -x * sin(theta) + y * cos(theta);
kernel = exp(-0.5 * (gamma^2*ty.^2+tx.^2)/sigma^2).*cos((2*pi/lambda)*tx+psi);
%0<=theta<=pi
%lambda: wavelenth, any number of pixels
%gamma: antistropic ration, 0<gamma<=1
%0<=psi<=2*pi
This function is responsible for computin the latest Gabor wavelet.

As we can see in the Gabor wavelets, the frequency determines how many peaks they have in the
resultant karnel. Frequency is inversely related to Lambda. Thus, by controlling lambda we can
control the frequency. Alpha is a ratio that I have chosen so that regardless to the scale of the kernel,
there is only one peak in the middle and two valleys around.
Other notable things for this function are that any created kernel has odd number of rows and
colums.
6
http://en.wikipedia.org/wiki/Gabor_filter

3.4.2 Why gabor features
When the input data is too large or it is suspected to be redundant (it means the data has not much
useful information) then the input data will be transformed into a reduced representation set, named
as features. The process to obtain this vector of features is feature extraction.
Consider that we have a different defect images from different defects. In defect detection we are
interested in highlighting those parts of the defects that are common for all defects. All the defects
have holes, scraches, small waves, fractures… We need features that can highlight them. Of corse in
defect recognition, then we need features that can successfully distinguish different defects from each
other. Gabor features can do both. In fact, Gabor features have been used for both purposes. What
they can do is that they can remove useless and redundant data and what is left can be used for
defect detection or defect recognition. Now the aim of the later stage, the classifier, which in our
case is a neural network, can be to recognize them or to detect them.
3.4.3 How to perform gabor filtering
Just like any other filtering method, Gabor filtering is done by convolving images with Gabor
kernels. Each window that contains a Gabor wavelet is a Gabor kernel and as described before the
size of the size of the kernel are always odd pixels.
After creating all the required kernels, which are about 40, each kernel should be convolved with the
window. The convolution of f and g is written by f*g. In 2D we can consider that we put both
images on top of each other and we multiply each of their pixels. Then we sum them all into one
scalar value which belongs to the location of the pixel at the centre of the window. After that, one
should move the kernel and compute the results for another location until we have the result for all
the values of the pixels. We can read more about convolution in here7.
But there is a faster way to compute the result of the convolution and I have used it inside the defect
detection system.
3.4.4 Fast gabor filtering
All the convolutions can be done by using Fast Fourier Transform (FFT). First we need to prepare
each kernel in frequency domain by taking the fast Fourier transform from each and every one of the
kernels. It is better to compute them while creating the kernels, so that we can save each kernel in its
frequency form.
After we get a window to process, we should take FFT from that window and for computing the
convolution between each window and each Gabor kernel, the only required thing is to multiply them
pixel by pixel (the dot product) in frequency domain. After the multiplication is done, we should
bring back the result to spatial domain by takeing its IFFT of the inverse fast Fourier transform. The
result is the same as convolving them together.
There are some considerations. In my program each window is 27x18. The size of each kernel is
variable and it depends on the parameters and more specifically the scale of the kernel. To use FFT,
it needs the size of both parts to be a power of two. Now, since our window is of size 27x18, our
only reasonable option is to ectend it to 32x32. It can be done by zero padding and the fft2 function
7
http://en.wikipedia.org/wiki/Convolution

in MATLAB can take care of it. The same should happen for each Gabor kernel and as one can see
in create_gabor.m each of the Gabor kernels has a size of 32x32. The maximum scale of Gabor filter
can not exceed this size because the wavelet should be contained inside a kernel of this size.
The second consideration is that now the result of the multiplication has a size of 32x32. The inverse
fast Fourier transform (IFFT) which is implemented with ifft2 in MATLAB, transforms the data
while taking its inverse to its original size of 27x18.
The third considetation is that sometimes it is needed to shift the result by half the size of the kernel
to produce the correct convolution results. In fact, it is always needed. But here since we want to
pass it to the neural network, the black box will take care of it and it is not necessary which will only
waist time.
3.4.5 IM2VEC.M
Now that we know everything about geature extraction, let us look at im2vec.m which is the most
importaint function in the program. As the name says, it converts an image (or more percisely a
window or patch) to a vector using Gabor feature extraction. The output of the function is a vector
which should be forwarded to the classifier for the task of classification.
function IMVECTOR = im2vec (W27x18)
This line defines im2vec.m as a function. W27x18 is the input argument which is the input window of
size 27x18. IMVECTOR is the output vector of the function. Now here I used W27x18 as the name
of the variable. The number does not have anything to do with the size of the window. It is just a
name for me to know the size.
load gabor;
The next line load the Gabor kernels which have been prepered before and they are stored in matrix
G after they have transformed the frequency domain.
W27x18 = adapthisteq(W27x18,'Numtiles',[8 3]);
Adaptive histogram equilization is used to make the feature vector robust against lighting conditions
and variations in contrast and brightness.
Features135x144 = cell(5,8);
for s = 1:5
for j = 1:8
Features135x144{s,j} =
mminmax(abs(ifft2(G{s,j}.*fft2(double(W27x18),32,32),27,18)));
end
end
The next lines define an empty cell matrix which contains 40 matrixes in 5 rows and 8 columns. Each
matrix is going to hold the response of its corresponding Gabor kernel. Each row presents a scale
and each column represents an orientation. The loops compute the response and everything that I
mentioned about zero padding and size can be seen here. So many things are happening in just one
line. Here because I originally used complex Gabor filters, the results are not real, so abs is used to
get the power of the response which is real and then mmainmax is used which is user-defined
function for normalizing the ourput between -1 and 1.
Features45x48 = cell2mat(Features135x144);
After that, cell2mat is used to convert the cell matrix to a usual matrix by concatenating all of its
separate matrixes.
The new width is going to be 5x27=135 and the new hight is going to be 8x18=144. That is why I
chose the name of the variavle Features135x144 in the first place.

But 135x144 contains 19440 elements in the ouput vector. For a neural network to be trained on a
feature extraction vector like this, we need a high number of inputs neurons which is not reasonable
regarding training and simulation time. That is why we have to cut some of the data and reduce the
dimensionality of the feature vector.
The first thing that came to my mind was to use one of the dimensional reduction algorithms. Some
of them can be found here8. But after testing PCA, I realizde that it takes so much time to complete
it for each possible window and I just decided to pick one out of three consecutive data by simply
deleting (or resizing) the rest. The following lines are going to do that and what is left is a matrix size
45x48.
Features45x48 (3:3:end,:)=[];
Features45x48 (:,3:3:end)=[];
The last line is nothing more than reshaping the remaining data into the form of a vector which can
be done successfully using reshape function in MATLAB.
IMVECTOR = reshape (Features45x48,[2160 1]);
Figure 14 shows the whole diagram of the im2vec.m that I have just explained here.
Figure 15: Sheme of defect detection with neural network in im2vec.m
8
http://en.wikipedia.org/wiki/Dimension_reduction

4 DEFECT DETECTION SYSTEM

Up to now, we know everything we need to detect defects from a sample image using a multi-layer
feed forward neural networks and Gabor wavelets as its feature extraction method.
Of course there is always more. For instance, we may want to learn about feature selection, other
geatures, maybe we want to use it on our webcam and we have to come up with a method to do the
detection in real-time, but there are three more things that I should explain to finish the system.
They are pre-selection and search algorithm (imscan) and post-processing.
When I wrote the code for the first time, I loaded an image and then I checked every possible pixel
of the image to find the defect. It was a huge disappoitment for me, because it was unbelievably
slow. I decided to work on something like a pre-selection step for the program guess some locations
of the image which might contain a defect with a high probability. I call this section the pre-selection
stage.
4.1 PRE-SELECTION
After some testing, I got the idea to cross-corelate a defect-like sub-image of size 27x18 with the
input image. Later I named the defect-like image, a template defect. Actually I ended up using two
template defects, one with a bright background and the second one with a dark (black) background.
Cross-correlation is a measure of similarity that we can read more about here9. It is simply a
convolution but theoretically speaking, we do not rotate the kernel. My only intention is to roughly
guess the location of the defects to avoid inspecting every location.
Here is the first section of the imscan.m script that is responsible for finding the rough location of the
defects.
C1 = mminmax(double(im));
C2 = mminmax(double(imread (TEMPLATE1)));
C1,C2 and C3 are prepared images for the convolution and they are all normalized between values -
1 an 1.
Corr_1 = double(conv2 (C1,C2,'same'));
Cell.state = int8(imregionalmax(Corr_1) | imregionalmax(Corr_2));
Then we compute the convolutions and we use a MATLAB function called imregionalmax function
to find the local maximum peaks inside the both of them. The result of each imregionalmax function
is a binary image with the same size of the original image that most of its pixels are zeros but some
pixels are one, exactly at the location of each regional maxima.
An ideal point for me was an ON pixel around the very light or dark area of each defect. I tested it
and just to be shore that I had not lost any critical points, I added the regional maxima results form
both of the templates.
Cell.state(1:13,:)=-1;
Cell.state(end-13:end,:)=-1;
Cell.state(:,1:9)=-1;
Cell.state(:,end-9:end)=-1;
9
http://en.wikipedia.org/wiki/Cross-correlation

Then I discarded those ON pixels (with value 1) near the borders to avoid the search for those
locations that the 27x18 window cannot fit into and I saved all of the poins inside an image called
Cell.state for further search
Cell.net = ones(m,n)*-1;
Cell.net is another binary image that will hold the results of the neural network for each pixel.
[LUTm LUTn]= find(Cell.state == 1);
LUTm and LUTn are two vectors that hold the location of each ON pixel in Cartesian format. The
find function in MATLAB finds those locations.
imshow(im);
hold on
plot(LUTn,LUTm,'.y');
pause(0.01);
And the last four lines display the image on the screen. First the original image is shown and we hold
the image with 'hold on'.
Then we plot all the points on it using plot function. 'y' will show the ON pixel in yellow.
The pause seems very strange but if I remove that, MATLAB just continues the computation and
does not give time for the display interrupt to come in and thus nothing will be displayed.
Figure 16: Pre-selection of defect-contained window
4.2 SEARCH ALGORITHM
In the pre-selection stage, we recorded the location of all the pixels that should be checked in a
image-like matrix caled cell.state. Actually the word 'state' is coming from the fact that each pixel
contains a 1 or a 0 as its value. The ON pixels which have 1 as their values, are the location of the
center of the 27x18 windows that should be checked.
Now we shoul make sure that when the algorithm finishes, the values of all the pixels are -1. During
this phase the value of some pixels may change. According to defined rules, some pixels may change
their velues form -1 to 1. Other pixels may change their values to -1.

The first thing that we need is an endless loop. The program can not get out of this loop unless it
checks all marked pixels.
There are many ways to get into a loop without an end but one way is to use this loop.
while (1==1)
…
Somewhere inside we will check the stop condition and we will use 'break' to geto out.
…
end
Each time that the algorithm reaches to the end and wants to start over, is called 'iteration'. In each
iteration we check only one picel. Thus in each iteration, the first thing we do is to find only one
pixel with stat 1. This is done with the following command:
[i j] = find(Cell.state==1,1);
This command will find one pixel with value 1. The function 'find' in MATLAB always starts
checking the arrays from top to borrom and from left to right. So, needless to say that when we look
at te processing results on the screen, it appears that the program is processing the image form left to
right. The location of the pixel is going to be stored in i and j veriables.
The next three lines are the stop condition. If (j,i) does not exist, the algorithm assumes that no point
is left for checking and it will get out of the loop.
if isempty(i)
break;
end
Next, the algorithm cuts the patch from the image.
imcut = im(i-13:i+13,j-9:j+8);
Imcut is the result and it has a size of 27x18. Its center is located at the point (j,i). After that to be
sure that we will never check this pixel again, we are going to change its state value to -1, so that in
the next iteration, the algorithm does not pick this one again.
Cell.state(i,j) = -1;
Then, comes the most important line. The window is going to be checked by the neural network. But
before that it should be changed to a vector format using 'im2vec' function that we have discussed
before.
Cell.net(i,j) = sim(net,im2vec(imcut));
Cell.net at pixel (i,j) stores the response of the network. It is fairly true that most of the time of the
program is spent in this line alone. Now, according to the result for this pixel, we should make a
decision about other pixels in the neighbourhood.
THE BLACK ZONE a

If the result of the neural network is less then -0.95 (it is pretty much near -1), it means that
according to our trained network there is no way for defect near this location. In order to save some
time, we make sure that none of the pixels in the neighbourhood of 3 city block are not needed to get
checked. We deliberately delete all the ones arround them and put -1 in cell.state.
THE MAGENTA ZONE a

If the result of the neural network is over -0.95 but is still a negative value less than a pre-specified
treshold (-0.5), it still does not contain any defect but we do not change anything around.

THE BLUE ZONE a

If the returned value is over 0.95, according to our network, that location defiantly contains a defect.
So if the prophecy is true, any window around this location cannot contain any other defect. So we
make sure that in the area equal to 27x18, there is not going to be another pixel with state 1.
THE GREEN ZONE a

The returned value is between 0.5 and 0.95. It probably contains a defect, but it is better to search
areas arround it for better match.
THE RED ZONE a

The returned value is between -0.5 and 0.5. this area probebly does not contain a defect. After that,
we do the same procedure…
4.3 THE SECOND PASS
…for all of the 9 adjacent pixels, but with one difference. If the pixel was already checked, we leave
it alone.
If the pixel was in the blue zone, we mark all the 27x18 pixels around as checked, so that we are
never going to check it again by ahanging their states to -1. If the pixel was in the green zone, we
mark it for the next iteration by changing its state to 1. All of these steps make sure that we check
every single pixel around our original candidate points to the extent that the neural network cannot
find any other defects arround the region.
The figure 16 below shows the result of scanning an image with this algorithm.
Figure 17: Scaned image with neural network
4.4 POST-PROCESSING
From this point forward, we have the neural network responses for most of the candidate locations.
There is only a matter of deciding which locations to choose as defects. In the above example, if we
show the cell.net as an image using the imshow command, we will see something like this:

Figure 18: Candidate locations for locating defects
We first treshold it, with our pre-defined treshols, which was 0.5.
xy_ = Cell.net > Threshold;
The result is this:
Figure 19: Treshold image
Then we dilate a disc structure element to avoid discontinuity. It sometimes happens and forces
duplicate rectangles around the defect area. The command is:
xy_ = imdilate (xy_,strel('disk',2,4));
Figure 20: Centroids
Then we use a connected component analysis to find each region and its centre.
regionprops(LabelMatrix,'centroid')
Regionprops is a MATLAB command that performs connected component analysis and we can ask
for some properties of this components or we can get all of them. Here what I needed was only the
centroid of each component. Thus that is what I asked by putting it as a parameter.
The output is in the form of a 1x2 matrix. It contains the coordinates of the centres. Each connected
component has only one centre. Here are the centres that I hvae shown.

Figure 21: Centers
It is easy to mark the centres on the image. In the program I have used xy_ variableas a temporary
variable whatever I needed it. We just have to write a for…loop to count from one to the number of
components that I have named it nLabeled.
for i = 1:nLabel
xy_(fix(CentroidMatrix(i).Centroid(2)),...
fix(CentroidMatrix(i).Centroid(1))) = 1;
end
The coordinates are in real numbers, so I have used 'fix' to get their integer values for the location of
the pixel. Although that is almost enough for defect recognition or other stages, I wanted to see the
results and be able to show it in my presentation.
Thus, I wrote a function called 'drawrec' to do this for me, it gets the image with marked pixels and
turn it into another image like the one below.
Figure 22: Drawing rectengulars on the centers
At last, I scanned that and changed the colour of every pixel in the original image to green; as we can
see in figure 22.
Figure 23: Marked defects

5 FUTURE WORK
Now we know everything about 'Defect detection system'. I already know that it is not optimized for
a real-word application. It is only educational.
There are lots of new technques and technologies for object recognition. There are endless
possibilities when it comes to combining this program with other feature ectraction, feature selection
ar other classification methods.
We need to search for terms 'defect detection','defect recognition','object detection' and we can find
all about them. The same system can be used to detect defects, signals, similar objects, orc…
Here, I point out some of the possibilities that can be done to improve the detection rate.
5.1 PRE-SELECTION
This can be one of the best places to start with. The pre-selection stage of the algorithm is
responsible for most of the errors that occur during the detection. The problem is that it usually
misses the most important areas. Thus, those areas do not have any chance to get examined
thoroughly with the neural network.
If we want to do a better job, we could try to develop something more accurate or totally skip it and
find other methods with low-computational costs to examine the entire image.
5.2 FAST CONVOLUTIONAL ANN
At this time I was interested to implement neural network with some convolutions just like here10.
Every computation related to the neural networks can be done using convolutions and later by using
Fast Fourier Transform for fast convolutions.
5.3 FEATURE SELECTION
There are lots of methods to examine which features are better for our type of detection. This
program has been done totally by trial and errors
I do not have any doubt that Gabor features can be optimized regarding to their size and orientation
and internal parameters.
We can use feature selection not only to optimize these parameters, but to reduce the feature space.
Do not doubt that we get better detection results and it becomes much faster than this.
For instance if we want to distinguis between handwritten and printed characters, we may need
different features instead of Gabor features. That is also true for sign detection, signature, graffiti
paintings…
5.4 CLASSIFICATION
There are lots of other classification methods that we can work on. SVM, Fisher, other type of
neural networks. Hidden Markov Model (HMM) is another classifier and feature extraction that
promises good results.
10
http://www.springerlink.com/content/a0e28xp4a8axq9n8/

6 CONCLUSION
Defect detection has been an attractive field of research for computer vision scientists. Humans are
able to identify reliably a large number of flaws and neuroscientists are interested in understanding
the perceptual and cognitive mechanisms at the base of the defect detection process. Those
researches illuminate computer vision scientists’ studies. Although designers of defect detection
algorithms and systems are aware of relevant neuropsychological studies, they also should be
prudent in using only those that are applicable or relevant from a practical/implementation point of
view. Since 1888, many algorithms have been proposed as a solution to automatic flaw detection.
Although none of them could reach the human detection performance, currently two biologically
inspired methods, namely eigenfaces and elastic graph matching methods, have reached relatively
high detection rates. Eigenfaces algorithm has some shortcomings due to the use of image pixel gray
values. As a result system becomes sensitive to illumination changes, scaling, etc. and needs a
beforehand pre-processing step [10]. Satisfactory recognition performances could be reached by
successfully aligned defect images. When a new flaws attends to the database system needs to run
from the beginning, unless a universal database exists. Unlike the eigenfaces method, elastic graph
matching method is more robust to illumination changes, since Gabor wavelet transforms of images
is being used, instead of directly using pixel gray values. Although detection performance of elastic
graph matching method is reported higher than the eigenfaces method, due to its computational
complexity and execution time, the elastic graph matching approach is less attractive for commercial
systems [11]. Although using 2-D Gabor wavelet transform seems to be well suited to the problem,
graph matching makes algorithm bulky. Moreover, as the local information is extracted from the
nodes of a predefined graph, some details on a flaw, which are the special characteristics of that flaw
and could be very useful in detection task, might be lost [12,13,14]. In this seminar, a new approach
to flaw detection with Gabor wavelets& feed forward neural network is presented. The method uses
Gabor wavelet transform & feed forward neural network for both finding feature points and
extracting feature vectors. From the experimental results, it is seen that proposed method achieves
better results compared to the graph matching and eigenface methods, which are known to be the
most successive algorithms.
Although the proposed method shows some resemblance to graph matching algorithm, in my
approach, the location of feature points also contains information about the flaw. Feature points are
obtained from the special characteristics of each individual defect automatically, instead of fitting a
graph that is constructed from the general flaw idea. In the proposed algorithm, since the defect
features are compared locally, instead of using a general structure, it allows us to make a decision
from the parts of the flaw. Moreover, having a simple matching procedure and low computational
cost proposed method is faster than elastic graph matching methods. Proposed method is also robust
to illumination changes as a property of Gabor wavelets, which is the main problem with the
eigenface approaches. A new flaw image can also be simply added by attaching new feature vectors
to reference gallery while such an operation might be quite time consuming for systems that need
training. Feature points, found from Gabor responses of the flaw image, can give small deviations
between different conditions (colour, illumination, rotation, etc.), for the same individual flaw.
Therefore, an exact measurement of corresponding distances is not possible unlike the geometrical
feature based methods. Moreover, due to automatic feature detection, features represented by those
points are not explicitly known, whether they belong to a white or a black (dark) area. Giving
information about the match of the overall defection structure, the locations of feature points are
very important. However using such a topology cost amplifies the small deviations of the locations of
feature points that are not a measure of match.

7 REFERENCES
[1] J. Tou, R. Gonzalez, Pattern Recognition Principles. Massachusetts, USA: Addison-Wesley
Publishing Company, 1981.
[2] Qing-xiang Wang, Di Li, et al. Unsupervised defect detection for gold surface defect of flexible
printed board. In Optics and Precision Engineering. Vol.18, No.4,pages 981-987, Apr. 2010.
[3] L. C. Chen and C. C. Kuo, Automatic TFT-LCD mura defect inspection using discrete cosine
transform-based background filtering and just noticeable difference quantification strategies. In
measurement science & technology, vol. 19, pages 1-10, 2008.
[4] Da-qi Zhang, Shi-ru QU , HE Li. Application of automatic image enhancing technique to road
defect detection systems. In Optics and Precision Engineering. Vol.18, No.8, pages 1869-1876,
Aug. 2010.
[5] Xin Bi, Chungang Zhuang, and Han Ding. A New Mura Defect Inspection Way for TFT-LCD
Using Level Set Method. In IEEE Signal Processing Letters, Vol. 16, No. 4, pages 311-
314,Apr. 2009.
[6] Qi Cao,De-jiang Wang, et al. Energy accumulation in infrared point target detection. In Optics
and Precision Engineering. Vol.18, No.3, pages 741-747, Mar. 2010.
[7] L. T. Fang, H. C. Chen, I. C. Yin, S. J. Wang, C. H. Wen, and C. H.Kuo, Automatic mura
detection system for liquid crystal display panels. In Proceedings of Conference SPIE-IS&T
Electronic Imaging, San Jose,California, Jan.15-19, pages 143-152, 2006.
[8] Fan Yang and Michel Paindavoine,”Prefiltering for pattern RecognitionUsing Wavelet Transform
and Neural Networks”, Adavances in imaging and Electron physics, Vol. 127, 2003.
[9] Lyons M. J. , Budynek J. and Akamatsu S., “Automatic Classification of Single Facial Images”,
IEEE Trans. on PAMI, Vol. 21, No.12, December 1999.
[10] Yang F. and Paindavoine M., “Prefiltering for pattern Recognition Using Wavelet Transform
and Neural Networks”, Advances in imaging and Electron physics, Vol. 127, 2003.
[11] Krüger N., Pötzsch M. and Malsburg C.V.D., “Determining of Face Position and Pose with a
learned Representation Based on Labeled Graphs”, Image and Vision Computing,Vol. 15,
pp.665-673,1997.
[12] Gajski D., Dutt v, Wu A., “High-Level Synthesis: Introduction to Chip and System Design”,
Kluwer Academic Publishers, Boston, 1992.
[13] Kanade T., “Picture processing by computer complex and recognition of human faces”,
Technical report, Kyoto University, Dept. of Information Science, 1973.
[14] Turk M. and Pentland A., “Eigenfaces for recognition”, Journal of Cognitive Science, pp.71-
86,1991.

8 APPENDIX
8.1 USER’S GUIDE & PROGRAM’S DESCRIPTION
How to run the program for the first time

1. copy all files and directories to the MATLAB’s work folder WE may also create a folder
there to avoid confliction with other programs
2. find the file named “main.m”
3. double click on the this file or type “main” in the command window for the first time the
program will create three files automatically
gabor.mat : this file contains a cell array matrix called “G”. Forty gabor filters are stored in
“G” in frequency domain each of which has a resolution of 32x32
net.mat : feed forward neural network structure
imgdb.mat : All Images which are going to be used in training
4. a menu will be shown . click on “Train Network” and wait until the program trains our neural
network
5. click on “Test on Photos”. A dialog box will be appeared. Select a .jpg photo. ” im1.jpg” is a
small image which is good for our first visit of the program. The selected photo will be
shown on the screen. We can maximize the window if we want.
6. Wait until the program detects some defects. During this phase we should see some activities
on the selected photo…
Requirments:
1. MATLAB 7.0 or Later
2. Image Processing Toolbox
3. Neural Network Toolbox
PROGRAM’S DEPENDENCY REPORT
gabor.m
This script contains gabor equation and is used to generate one based on some parameters
create_gabor.m
This script uses gabor.m to generate forty 32x32 gabor filters and save them in a cell array matrix
called

“G” and in a file named “ gabor.mat”. This script will be inkoved only once unless we delete
“gabor.mat”.
Slika 1: Gabor Filters in Time Domain
main.m
The main menu and the only file we need to run the program
createffnn.m
This function creates a feed forward neural network with one hundred neurons in the hidden layer
and one neuron in the output layer. The network will be saved in “net.mat” for further use. To learn
more about how to customized neural network see “MATLAB help > Neural Network Toolbox >
Advance Topics”
loadimages.m
This function prepares images for training phase. All data form both “defect” and “non-defect”
folders will be gathered in a large cell array. Each column represents the features of an image which
could be a defect or not
Rows are as follows:

Row 1: File name
Row 2: Desired output of the network corresponded to the feature vector.
Row 3: Prepared vector for the training phase
Also this script saves the database to a file named “imgdb.mat”, so that we do not need to create the
database more than once unless we add or delete some photos to/from “defect” and “non-defect”

folders. Every time we do this, after recreating a database, we should initialize and train the network
again. This script uses “im2vec.m” to extract features from images and vectorize them for the
database.
im2vec.m
This function takes a 27x18 image. It adjusts the histogram of the image for better contrast. Then the
image will be convolved with gabor filters by multiplying the image by gabor filters in frequency
domain. Gabor filters are stored in “gabor.m”. To save time they have been saved in frequency
domain before.
Features135x144 is a cell array contains the result of the colvolution of the image with each of the
forty gabor filters. These matrixes will be concated to form a bif 135x144 matrix of complex
numbers. we only need the magnitude of the result. That is why “abs” is used.
135x144 has 10,400 pixels. It means that the input vector of the network should have 19,400 values
which mean a large amount of computation. So we reduce the matrix size to one-third of its original
size by deleting some rows and columns. Deleting is not the best way but it save more time compare
to other methods like PCA
trainnet.m
This function trains the neural network and returns the trained network

imscan.m
First Section:
Second Section:
In this section the algorithm checks all potential defect-contained windows and the windows around
them using neural network. The result will be the output of the neural network for checked regions.
cell.mat
Third Section:
1. Filtering above pattern for values above threshold (xy_)
2. Dilating pattern with a disk structure (xy_)

3. finding the center of each region
4. Draw a rectangle for each point
5. Final Result

8.2 MATLAB CODE
8.2.1 create_gabot.m
% Date : 12.01.2012
% Author : Žiga Gosar
close all;
clear all;
clc;
% Creating 40 Gabor Filters
G = cell(5,8);
for s = 1:5
for j = 1:8
G{s,j}=zeros(32,32);
end
end
for s = 1:5
for j = 1:8
G{s,9-j} = gabor([32 32],(s-1),j-1,pi,sqrt(2),pi);
end
end
% Showing Gabor Filters
figure;
for s = 1:5
for j = 1:8
subplot(5,8,(s-1)*8+j);
imshow(real(G{s,j}),[]);
end
end
for s = 1:5
for j = 1:8
G{s,j}=fft2(G{s,j});
end
end
save gabor G
8.2.2 createffnn.m
% Date : 12.01.2012
net = network;
% ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
% This function creates a feedforward neural network with 100
% neurons in the input layer and 1 neuron in the output layer.
% to study more about how to create a network like this see help files
% for Neural Network Toolbox - Advance Concepts
%
% The network will be saved to 'net.m' for future use.
net.numInputs = 1;
net.numLayers = 2;
net.biasConnect = [1;1];
net.inputConnect = [1 ;...
0 ];
net.layerConnect = [0 0 ;...
1 0 ];
net.outputConnect = [0 1];
net.targetConnect = [0 1];
netInputs = ones (2160,2);
netInputs (1:2160,1)= -1;
net.inputs{1}.range = netInputs;

net.layers{1:2}.transferFcn = 'tansig';
net.layers{1:2}.initFcn = 'initnw';
net.initFcn = 'initlay';
net.performFcn = 'msereg';
net = init(net);
save net net
8.2.3 drawrec.m
% Date : 12.01.2012
function out = drawrec (in,w)
% ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
% This function will draw a 27x18 rectangle
% Inputs are an image 'in' abd w or(~) is a binary image same size as the
% input image , in the binary image each 1 is the center of a rectangle and
% the function leaves zeros alone.
[m n]=size(in);
[LUTm LUTn]=find(in);
out = zeros (m,n);
for i =1:size(LUTm,1)
try
out (LUTm(i),LUTn(i))=0;
end
try
out (LUTm(i)-14:LUTm(i)+13,LUTn(i)-9)=1;
end
try
out (LUTm(i)-14:LUTm(i)+13,LUTn(i)+8)=1;
end
try
out (LUTm(i)-14,LUTn(i)-9:LUTn(i)+8)=1;
end
try
out (LUTm(i)+13,LUTn(i)-9:LUTn(i)+8)=1;
end
end
8.2.4 gabor.m
% Date : 12.01.2012
function Psi = gabor (w,nu,mu,Kmax,f,sig)
% w : Window [128 128]
% nu : Scale [0 ...4];
% mu : Orientation [0...7]
% kmax = pi/2
% f = sqrt(2)
% sig = 2*pi
m = w(1);
n = w(2);
K = Kmax/f^nu * exp(1i*mu*pi/8);
Kreal = real(K);
Kimag = imag(K);
NK = Kreal^2+Kimag^2;
Psi = zeros(m,n);
for x = 1:m

for y = 1:n
Z = [x-m/2;y-n/2];
Psi(x,y) = (sig^(-2))*exp((-.5)*NK*(Z(1)^2+Z(2)^2)/(sig^2))*...
(exp(1i*[Kreal Kimag]*Z)-exp(-(sig^2)/2));
end
end
8.2.5 im2vec.m
% Date : 12.01.2012
function IMVECTOR = im2vec (W27x18)
%{ The input of the function is a 27x18 window . At first the function adjusts
the histogram of the window . Then to convolve the window with Gabor filters,
the window in frequency domain will multiply by gabor filters Gabor filters are
stored in gabor.mat . to save time they have been saved in frequency domain
before. %}
load gabor; %loading Gabor filters
% Adjust the window histogram , the parameters are set with try and error
W27x18 = adapthisteq(W27x18,'Numtiles',[8 3]);
Features135x144 = cell(5,8);
for s = 1:5
for j = 1:8
Features135x144{s,j} =
mminmax(abs(ifft2(G{s,j}.*fft2(double(W27x18),32,32),27,18)));
end
end
% Features135x144 is a cell array contains the result of the convolution of the
% window with each of the fourty gabor filters. These matrixes will concate to
% form a big 135x144 matrix of complex numbers/
% We only need the magnitude of the result That is why the abs is used
%Features135x144 = abs(cell2mat(Features135x144));
Features45x48 = cell2mat(Features135x144);
%{ 135x144 is very painful to be an input for the network. it has 19,400 pixels.
It means that the input vector of the network should have 19,400 pixels which
means a large amount of computation and waste of time. So we reduce the matrix
size to one-third of it's original size. There are so many ways to reduce the
matrix size like using PCA, using the median of each 3x3 pixels or deleting the
rows and colums deleting is not the best way, but it save more time compare with
others %}
% The numbers in the input vector of the network should be between -1,1
% and the line below will fulfill this concept.
% Features45x48 = premnmx(Features135x144);
% Change the matrix to a vector

IMVECTOR = reshape (Features45x48,[2160 1]);
%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
%{ notes : This function is very critical. consider that we have a big photo say
400x300. Can you say how may 27x18 windows we have ? If we do not preprocess the
photo to predict the location of the defects we have a window for each and every
single pixel of the photo which means 120,000 windows (A little less because of
the borders). And each pixel is the center of a new window. If this function
takes 0.4 sec to be executed, the whole photo will take % about 13 hours only
for the network preprocess. So any unnecessary line in this function can be a
hell for the whole process and we should optimize this function as possible as
we can. %}

8.2.6 imscan.m
% Date : 12.01.2012
%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
{% This function gets the network and an image , select some 27x18 windows of
the image and send them to network. The result of the network determins if the
window is a defect or not it is not as easy as that . follow the comments %}
function im_out = imscan (net,im)
close all
%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
% PARAMETERS
%REPORT_FILE = 'report.txt'; %name of the report file
SCAN_FOLDER = 'imscan/'; %folder for storing the detected defects
UT_FOLDER = 'imscan/under-thresh/';
TEMPLATE1 = 'template1.png';
TEMPLATE2 = 'template2.png';
%if the network result for a 27x18 window gets over thresh , that window is
%detected as a defect.
Threshold = 0.5;
DEBUG = 0;
%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
warning off;
delete ([UT_FOLDER,'*.*']);
delete ([SCAN_FOLDER,'*.*']);
if (DEBUG == 1)
mkdir (UT_FOLDER);
mkdir (SCAN_FOLDER);
end
[m n]=size(im);
%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
%% First Section
%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
{% It is not possible to send all 27x18 windows in a photo to the network (see
notes in 'im2vec.m' so we should chose some windows in a large photo which the
existent probapility of the defect is more in these windows. A window is
determined by it's center which is a pixel on the image. Template is a 27x18
image or window containing a defact without background correlation between the
Template and the image will give us an image which have peaks over the defect
area. imregonalmax is MATLAB function which extract the peaks in the binary mask
each pixel in the binary mask will be used as a center for a window %}
C1 = mminmax(double(im));
figure;imshow (Corr_1,[])
Cell.state = int8(imregionalmax(Corr_1) | imregionalmax(Corr_2));
Cell.state(1:13,:)=-1;
Cell.state(end-13:end,:)=-1;
Cell.state(:,1:9)=-1;
Cell.state(:,end-9:end)=-1;
Cell.net = ones(m,n)*-1;
[LUTm LUTn]= find(Cell.state == 1);
% two lines below show the image on the screen with the preprecessed centers in
% red
imshow(im);
hold on
plot(LUTn,LUTm,'.y');pause(0.01);
%~~~~~~~~~~~~~~~~~~~~~~~ SOS ~~~~~~~~~~~~~~~~~~~~~~~~~~~~

%{ The predication of centers will not always work properly but it's good for
must of the times. If anyone have a suggestion about the above algorithm, I will
be glad to hear%}
%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
%% Second Section
{% Now that we predict some centers for the windows, these loops find the
maximum result of the network around these centers which is the exact location
of the defect. remember each 27x18 window is determined by it's center. type
spiral(18) in the command window and compare the numbers in it with the location
of the centers you see on screen with yellow color.%}
while (1==1)
[i j] = find(Cell.state==1,1);
if isempty(i)
break;
end
imcut = im(i-13:i+13,j-9:j+8);
Cell.state(i,j) = -1;
Cell.net(i,j) = sim(net,im2vec(imcut));
if Cell.net(i,j) < -0.95
for u_=i-3:i+3
for v_=j-3:j+3
try
Cell.state(u_,v_)=-1;
end
end
end
plot(j,i,'.k');pause(0.01);
continue;
elseif Cell.net(i,j) < -1*Threshold
plot(j,i,'.m');pause(0.01);
continue;
elseif Cell.net(i,j) > 0.95
plot(j,i,'.b');pause(0.01);
for u_=i-13:i+13
for v_=j-9:j+9
try
end
end
end
elseif Cell.net(i,j) > Threshold
plot(j,i,'.g');pause(0.01);
elseif Cell.net(i,j) < Threshold
plot(j,i,'.r');pause(0.01);
end
for i_=-1:1
for j_=-1:1
m_=i+i_;
n_=j+j_;
if (Cell.state(m_,n_) == -1 || Cell.net(m_,n_)~=-1)
continue;
end
imcut = im(m_-13:m_+13,n_-9:n_+8);
Cell.net(m_,n_) = sim(net,im2vec(imcut));
if Cell.net(m_,n_) > 0.95
plot(n_,m_,'.b');pause(0.01);
for u_=m_-13:m_+13
for v_=n_-9:n_+9
try
end
end
end
continue;

end
if Cell.net(m_,n_) > Threshold
Cell.state(m_,n_) = 1;
plot(n_,m_,'.g');pause(0.01);
if (DEBUG == 1)
imwrite(imcut,[SCAN_FOLDER,'@',int2str(m_),',',int2str(n_),'
(',int2str(fix(Cell.net(m_,n_)*100)),'%).png']);
end
else
Cell.state(m_,n_) = -1;
plot(n_,m_,'.r');pause(0.01);
if (DEBUG == 1)
imwrite(imcut,[UT_FOLDER,'@',int2str(m_),',',int2str(n_),'
(',int2str(fix(Cell.net(m_,n_)*100)),'%).png']);
end
end
end
end
end
%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
%% Third Section
{% now we have and image called xy_ , to draw the green rectangles we should
find the centers of rectangles which is pixles with nwtrork results over
threshold.%}
hold off
figure;
imshow (Cell.net,[]);
xy_ = Cell.net > Threshold;
xy_ = imregionalmax(xy_);
xy_ = imdilate (xy_,strel('disk',2,4));
[LabelMatrix,nLabel] = bwlabeln(xy_,4);
CentroidMatrix = regionprops(LabelMatrix,'centroid');
xy_ = zeros(m,n);
for i = 1:nLabel
xy_(fix(CentroidMatrix(i).Centroid(2)),...
fix(CentroidMatrix(i).Centroid(1))) = 1;
end
xy_ = drawrec(xy_,[27 18]);
im_out (:,:,1) = im;
im_out (:,:,2) = im;
im_out (:,:,3) = im;
for i = 1:m
for j=1:n
if xy_(i,j)==1
im_out (i,j,1)=0;
im_out (i,j,2)=255;
im_out (i,j,3)=0;
end
end
end
8.2.7 loadimages.m
% Date : 12.01.2012
function IMGDB = loadimages
%{ This function will prepare defects and non-defects for the train set all data
will be gathered in a large cell array . each colume represent a window for the
network , which could be a defects or not. Rows are as follows :
Row 1 ----> File Name
Row 2 ----> Desired output of the net in responding to the vector

Row 3 ----> Prepaired vector for training phase

you can see for yourself this large database out of the program.
type
IMGDB = loadimages
in the command window.
Also the function save the database to a file 'imgdb.mat' to save time
so this function is a one time function except you want to change something
in the 'im2vec.m'. In that case you should build the database again.%}
%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
defect_folder = 'defect/'; %LOCATION OF DEFECT IMAGES
non_defect_folder = 'non-defect/'; %LOCATION OF NON-DEFECT IMAGES
file_ext = '.png';
out_max = 0.9; % DESIRED OUTPUT FOR DETECTING A DEFECT
out_min = -0.9; % DESIRED OUTPUT FOR NOT DETECTING A DEFECT
%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
load imgdb;
else
IMGDB = cell (3,[]);
end
fprintf ('Loading Defects ');
folder_content = dir ([defect_folder,'*',file_ext]);
ndefect = size (folder_content,1);
for k=1:ndefect
string = [defect_folder,folder_content(k,1).name];
if (m~=27 || n~=18)
continue;
end
f=0;
f=1;
end
end
if f==1
continue;
end
fprintf ('.');
% as you see, each defect in the defect folder becomes 10 vectors
IM {1} = im2vec (image); % the original defect
IM {2} = im2vec (fliplr(image));% the mirror of the defect which is a defect
% for better response of the network , 1-pixel shifted defect in any
% direction is used to train network invariant to 1-pixel.
IM {3} = im2vec (circshift(image,1));
IM {4} = im2vec (circshift(image,-1));
IM {5} = im2vec (circshift(image,[0 1]));
IM {6} = im2vec (circshift(image,[0 -1]));
IM {7} = im2vec (circshift(fliplr(image),1));
IM {8} = im2vec (circshift(fliplr(image),-1));
IM {9} = im2vec (circshift(fliplr(image),[0 1]));
IM {10} = im2vec (circshift(fliplr(image),[0 -1]));
for i=1:10
IMGDB {2,end} = out_max;
end
end
fprintf ('\nLoading non-defects ');
folder_content = dir ([non_defect_folder,'*',file_ext]);
nndefect = size (folder_content,1);
for k=1:nndefect
string = [non_defect_folder,folder_content(k,1).name];

if (m~=27 || n~=18)
continue;
end
f=0;
f=1;
end
end
if f==1
continue;
end
fprintf ('.');
% here we need some examples to train the network with non-defect vectors
% so now we do not have a defect here. We can do what ever we want with
% the photo. Strike it with a hammer ,...
% I have chosen only the fliped version of non-defect. Flip left-right
% and up-down
IM {1} = im2vec (image);
IM {2} = im2vec (fliplr(image));
IM {3} = im2vec (flipud(image));
IM {4} = im2vec (flipud(fliplr(image)));
for i=1:4
IMGDB {2,end} = out_min;
end
end
fprintf('\n');
save imgdb IMGDB;
8.2.8 main.m
% Date : 12.01.2012
clear all;
close all;
clc;
% ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
% create_gabor is a s script . it's purpose is to craete a file
% with name 'gabor.mat'.
% This file contains a cell array matrix with name 'G'
% and 'G' has 40 32x32 matrixes coresponding to Gabor filter
% in frequency domain
if ~exist('gabor.mat','file')
fprintf ('Creating Gabor Filters ...');
create_gabor;
end
% ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
{% createffnn create a feedforward neural network with 100 neurons in input
layer and one neuron in output layers the network input is a 27x18 window of the
picture in vector form. It's purpose is to determine if the window is a defect
or not the output of the network is close to 0.9 for a defect and close to -0.9
for a non-defect%}
if exist('net.mat','file')
load net;
else
createffnn
end
% ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

{% loadimages prepare images in defect and non-defect folders for network

traning phase. See the source for more information%}
load imgdb;
else
IMGDB = loadimages;
end
% ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
while (1==1)
choice=menu('Defect Detection',...
'Create Database',...
'Initialize Network',...
'Train Network',...
'Test on Photos',...
'Exit');
if (choice ==1)
IMGDB = loadimages;
end
if (choice == 2)
createffnn
end
if (choice == 3)
% trainnet is a fuction which train the network
net = trainnet(net,IMGDB);
end
if (choice == 4)
pause(0.001);
[file_name file_path] = uigetfile ('*.jpg');
if file_path ~= 0
im = imread ([file_path,file_name]);
% if the input image happenes to be an RGB image
% try-end will convert it to gray image
try
im = rgb2gray(im);
end
% imscan is a function which scan the whole photo for defect
% im_out is the output image of the function
% with green rectangles across the defect
tic
im_out = imscan (net,im);
toc
figure;imshow(im_out,'notruesize');
end
end
if (choice == 5)
clear all;
clc;
close all;
return;
end
end
8.2.9 mminmax.m
% Date : 12.01.2012
function output = mminmax(input)

max_ = max(max(input));
min_ = min(min(input));
output = ((input-min_)/(max_-min_) - 0.5) * 2;
8.2.10 trainnet.m
% Date : 12.01.2012
function NET = trainnet(net,IMGDB)
%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
% define train parameters for the network
net.trainParam.lr = 0.4;
net.trainParam.epochs = 400;
net.trainParam.show = 10;
net.trainParam.goal = 1e-3;
%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
{% IMGDB is a large database of vectors with desired network output
corresponding to them which is loaded to the workspace by 'load imgdb'.
imgdb.mat is created with loadimages the first time of the execution by
loadimages.m %}
T{1,1} = cell2mat(IMGDB(2,:));
P{1,1} = cell2mat(IMGDB(3,:));
net = train(net,P,T);
save net net
NET = net;
8.2.11 Defect folder
8.2.12 Non-defect folder

Surface Defect Detection With Neural Net PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Surface Defect Detection With Neural Net PDF

Uploaded by

Copyright:

Available Formats

Univerza v Ljubljani

FAKULTETA ZA Neural Network 2011/12

Surface defect detection with Neural

PD-KM Žiga Gosar 17.09.13

Žiga Gosar Stran 2/54

4.4 POST-PROCESSING ....................................................................................................................................... 34

Žiga Gosar Stran 3/54

Žiga Gosar Stran 4/54

Žiga Gosar Stran 5/54

Žiga Gosar Stran 6/54

1.1 SURFACE DEFECT DETECTION

Žiga Gosar Stran 7/54

2 ARTIFICIAL NEURAL NETWORK (ANN)

2.1 A PERCEPTRON NEURON

Figure 1: A perceptron neuron

Žiga Gosar Stran 8/54

2.2 A SINGLE LAYER PERCEPTRON

Figure 2: A single layer perceptron

Furthermore, MATLAB tries to compactify it by showing it in a vector-like format.

Figure 3: A vector-like single layer perceptron

Žiga Gosar Stran 9/54

Figure 4: Hard-limit transfer function

2.3 MULTI-LAYER FEED-FORWARD NEURAL NETWORK

2.4.1 The easy way

2.4.2 Input and outputs explained

Žiga Gosar Stran 10/54

Žiga Gosar Stran 11/54

2.4.3 Transfer function

Figure 5: Log-sigmoid transfer function

The next one is called pure linear:

Figure 6: Linear transfer function

The last one is more popular and is called tangent sigmoid:

Figure 7: Tan-sigmoid transfer function

Žiga Gosar Stran 12/54

2.4.5 Creating it: the hard way

Žiga Gosar Stran 13/54

Žiga Gosar Stran 14/54

1. MATLAB help centre for the Neural Network Toolbox

Žiga Gosar Stran 15/54

Figure 8: Training neural network

2.6 TESTING AND SIMULATION

Žiga Gosar Stran 16/54

Table 1: Comparison of dimensions

Žiga Gosar Stran 17/54

2.7 COMPUTING THE ERROR

2.8 GENERATING TRAIN SETS

Žiga Gosar Stran 18/54

Žiga Gosar Stran 19/54

Žiga Gosar Stran 20/54

Žiga Gosar Stran 21/54

3.1 GRAY-LEVEL FEATURES

Žiga Gosar Stran 22/54

3.1.1 Contrast and Brightness

3.2 HISTOGRAM EQUALIZATION

Figure 9: Histogram equalization

Žiga Gosar Stran 23/54

3.3 ILLUMINATION EFFECT

Figure 10: Illumination effect

3.3.1 Background illumination model

3.3.2 Adaptive histogram equalization

Žiga Gosar Stran 24/54

3.4 GABOR FEATURE EXTRACTION

3.4.1 2D Gabor wavelet representation