Advances in DIP

( Fuzzy , Artificial Neural Networks, Expert System and Image Segmentation)

Minakshi Kumar
Photogrammetry and Remote Sensing Division , Indian Institute of Remote Sensing Dehradun

Fuzzy
ORIGIN: Aristotle some 2000 years ago formulated Conventional Logic as the Law of Excluded Middle (ie., Crisp Set) , which has dominated Western Logic ever since.. But the era was over.. Lotfi Zadeh, Professor, University of California at Berkley,had first conceived the idea of Fuzzy Logic during 1960s as a way of processing data by allowing partial set membership rather than crisp set membership.

WHAT IS FUZZY LOGIC (FL).?

FL provides a simple way to arrive at a definite conclusion based upon vague, ambiguous, imprecise, noisy, or missing input information. FL is basically a multi-valued logic that allows intermediate values to be defined between conventional evaluations like yes/no, true/false, black/white etc. Notions like rather warm or pretty cold can be formulated mathematically and processed by computers

Fuzzy Set Basics

Classical (crisp) sets:
Membership in a set is all or nothing Membership function cS: Universe {0, 1} cS(x) = 1 iff x S

Fuzzy sets:

range of real numbers

Membership in a set is a degree membership function cS: Universe [0, 1]

Linguistic Characterizations of Degree of Membership

Consider the set of cold days in Dehradun in December 2005. Was December 31 cold? It might have been called one of:
very cold sort of cold not cold

The answer depends on the observer, time, etc.

Sounds similar to probability, but isnt

Probability deals with uncertainty, likelihood Fuzzy logic deals with ambiguity, vagueness

Fuzzy Membership

Membership Functions
A membership function (MF) is a curve that defines how each point in the input space is mapped to a membership value (or degree of membership) between 0 and 1. The input space is sometimes referred to as the universe of discourse, a fancy name for a simple concept. It associates a weighting with each of the inputs that are processed, define functional overlap between inputs, and ultimately determines an output response. The rules use the input membership values as weighting factors to determine their influence on the fuzzy output sets of the final output conclusion.

Membership function plots

is a pile 1

# grains of sand 0 10 100 1000 10000 100000

Crisp vs. Fuzzy Membership Functions

is a pile 1

fuzzy

crisp grains of sand

0 10 100 1000 10000 100000

Set of Tall people

More Crisp vs. Fuzzy Membership Functions

1 Crisp 0 1 Fuzzy 0 0 0 1 1 grade of membership universe of possibilities

Note: Universe can be continuous or discrete, ordered or unordered.

Alternative Notation

Fuzzy-Set Operations
expressed using membership functions
1
A B

0
Fuzzy OR (union) Fuzzy AND (intersection)

cA U B(x) = max(cA (x), cB(x))

1
AUB

cA B(x) = max(cA (x), cB(x))

1
AB

Fuzzy Complement (not the only possible model)

cA(x) = 1 - cA(x).

Fuzzy Classification
Method In traditional classification, information is represented in a one-pixel -one - class method and its classification result is definite, i.e. a pixel belongs to a class. In fuzzy classification, a pixel is considered having different membership grades obtained from fuzzy classification indicate the area proportion of every cover classes in a mixed pixel. As the traditional method, the fuzzy classification can also be divided into fuzzy supervised and fuzzy unsupervised classification.

Fuzzy Supervised Classification

Sample pixels should be chosen for estimating fuzzy parameters before classification, different to conventional method, chosen sample need not be sufficiently homogeneous. Fuzzy mean can be expressed as

where n is the total number of sample pixels, fc is membership of a pixel to class c, xi is pixel value of sample pixels. Fuzzy covariance matrix can be expressed as

After the fuzzy parameters are determined, every pixel is classified according to its spectral value. To perform a fuzzy classification, the membership function must be defined for each class, In this work, the membership functions are defined based on maximum likelihood classification algorithm with fuzzy and fuzzy covariance matrix replacing the conventional mean and covariance matrix. The following is the definition of membership function for cover class c: N is the pixel
vector dimension, m is number of predefined classes.

Fuzzy Unsupervised Classification

The fuzzy unsupervised classification can be performed in many methods, in this work, kmeans clustering algorithm is adopted and is run using iteration as following: determine classification number k, 1<K< , bij definition distance choose> give a initial membership matrix

Fuzzy Unsupervised Classification Cont

calculate fuzzy clustering where centers of every class calculate new membership matrix U(t+1) using formula if the criterion is satisfied, stop the iteration. Otherwise, repeat process 3 to 5. The criterion of ending iteration can choose

Artificial neural Networks

Introduction of Neural Network: Brain is a highly complex, nonlinear, and parallel computer. It has structural constituents, known as neurons for computations. Definition of Neural Network: A neural network is a massively parallel distributed processor made up of simple processing units, which has a natural propensity for storing experimental knowledge and making it available for use. It resembles the brain in two respects; Knowledge is acquired by the network from its environment through a learning process. Inter-neuron connection strengths, known as synaptic weights, are used to store the acquired knowledge. The produre used to perform the learning process is called a learning algorithm, the function of which is to modify the synaptic weights of the network in an orderly fashion to attain a desired design objective.

Basic components of Neuron:

The Neuron:
The basic neuron consists of; Synapses: are connections between neurons they are not physical connections, but miniscule gaps that allow electric signals to jump across from neuron to neuron. Soma: these electrical signals are then passed across to the soma which performs some operation and sends out its own electrical signal to the axon. Axon: the axon then distributes this signal to dendrites. Dendrites: carry the signals out to the various synapses,and the cycle repeats

Artificial Neuron: Each neuron has a certain number of inputs. Each of which have a weight assigned to them. The weights simply are an indication of how important the coming signal for that input is. The net value of the neuron is then calculated. The net is simply the weighted sum, the sum of all the inputs multiplied by their specific weight. Each neuron has its own unique threshold value, and if the net is greater than the threshold, the neuron fires(or output a 1), otherwise it stays quiet(outputs a 0). The output is then fed into all the neurons it is connected to.

Architecture of Neural Network: The network consists of several layers of neurons An input layer Hidden layer Output layer Hidden layer-because the user cannot see the inputs or outputs for those layers. Ups and downs of neural network: They are excellent as pattern classification/recognizers. Neural Network can handle exceptions and abnormal input data. The power of neural network lies in their ability to process information in a parallel fashion but machines today are serial they only execute on instruction at a time.

General Mechanism of Neural Network:

Schematic representation of an neural network:

An artificial neuron is a model whose components have direct analogs to the components of actual neuron. The input signal are represented by x0, x1, ..xn. Each of these inputs is modified by a weight (synaptic weight) whose function is analogous to that of the synaptic junction in a biological neuron. These weights can be either positive or negative. The processing element consists of two parts The first part simply sums the weighted input resulting in a quantity I. The second part is effectively a non-linear filter, usually called the activation function through which the combined signal flows. Activation function be; It can be threshold function that passes information, when the output of the first part of the artificial neuron exceeds the threshold T.
(I) = 1, I>T 0, I<=T

Perceptron: Perceptron is the most basic term used in ANN. It can be single layered or multi-layered structure. Single perceptron: The structure of a single perceptron is very simple. There are two inputs, a bias and an output In the perceptron, the most common form of learning is by adjusting the weights by the difference between the desired output and the actual output. i = xi = desired output actual output.

Linear associator neural network: The most elementary neural network is a linear associator The fundamental neuron sums the weighted inputs and then subjects this sum to a nonlinear activation function to keep the output with in reasonable range. The architecture of a linear associator is a set of input neurons that are connected to a set of output neurons. (i.e., a two =layer neural network) When the activation functions of the neurons are linear, this network is a linear associator It is not very accurate, if two many items are stored in the associators.

Learning: This talk about weights and thresholds. Which leads to an obvious question, how are all these set? Most popular training methods are; Back-propagation The delta rule Kohonen learning Learning rules can be categorized into two areas: Supervised Un-supervised Supervised learning rules require a teacher. Unsupervised learning do not require teacher because they produce their own output, which is then further evaluated.

Learning and Recall: Learning is the process of adapting the connection weights in an artificial network to produce the desired output vector presented to the input buffer. Recall is the process of accepting an input stimulus and producing an output response in accordance with the network weight structure. Recall is an integral part of the learning process since a desired response to the network must be compared to the actual output to create an error function. In supervised learning, the artificial neural network is trained to give the desired response to a specific input stimulus. In graded learning, the output is graded as good or bad on a numerical scale, and the connection weights are adjusted in accordance with the grade. In unsupervised learning there is no specific response sought, but rather the response is based on the networks ability to organize itself.

Back Propagation:
Back-propagation is a systematic method for training multiple (three or more) layer artificial neural networks. The problem with the perception is that it cannot express non-linear decisions. The perceptron is basically a linear threshold device, which returns a certain value, 1 for example , if the dot product of the input vector and the associated weight vector plus bias surpasses the threshold, and another value, -1 for example, if the threshold is not reached. For example: If the dot product of the input vector and the associated weight plus the bias, is graphed it is obviously linear. More over, all the input vectors that will give a value greater than the threshold are separated into one space, and those that will not, will be separated into another.

Linear plot

Non-linear plot

Multi-layer feed forward network normally consist of three or four layers; with one output and minimum one hidden layer. The raw values of input layer are fed to hidden layer. Once the neurons for the hidden layer are computed, their activations are then fed downstream to the next layer, until all the activations eventually reach the output layer, in which each output layer neuron is associated with a specific classification category. Thus in computing the values of each neuron in the hidden and output layers one must first take the sum of the weighted sums and the bias, and then apply sigmoid function to calculate the neurons activation. Then how does the network learn the problem? The solution is by modifying all the weights.

How to modify the weights; Take partial derivative of the error of the network with respect to each weight, we will learn a little about the direction the error of the network is moving. If the derivative is positive, this tells us that the error is increasing when the weight is increasing, then add a negative value to the weight and vice versa if the derivative is negative. Because the taking of these partial derivatives and then applying them to each of the weights takes place starting from the output layer to hidden layer weights, then the hidden layer to input layer weights. This algorithm has been called the Back Propagation Algorithm.

Basic steps in back-propagation training for a multi-layer neural network: 1. Randomize the weights to small random values (both positive & negative) to ensure that the network is not saturated by large values of weights.(if all weights start at equal values, and desired performance requires unequal weights, the network would not train at all) Select a training pair from the training set. Apply the input vector to network input. Calculate the network output. Calculate the error, the difference between the network output and the desired output. Adjust the weights of the network in a way that minimizes this error. Repeat steps 2 6 for each pair of input output vectors in the training set until the error fro the entire system is acceptably low.

2. 3. 4. 5. 6. 7.

Neural Networks
A NN is a machine learning approach inspired by the way in which the brain performs a particular learning task: Knowledge about the learning task is given in the form of examples.
Inter neuron connection strengths (weights) are used to store the acquired information (the training examples). During the learning process the weights are modified in order to model the particular learning task correctly on the training examples.

Learning
Supervised Learning Recognizing hand-written digits, pattern recognition, regression. Labeled examples (input , desired output) Neural Network models: perceptron, feed-forward, radial basis function, support vector machine. Unsupervised Learning Find similar groups of documents in the web, content addressable memory, clustering. Unlabeled examples (different realizations of the input alone) Neural Network models: self organizing maps, Hopfield networks.

Network architectures
Three different classes of network architectures
single-layer feed-forward multi-layer feed-forward recurrent neurons are organized in acyclic layers

The architecture of a neural network is linked with the learning algorithm used to train

Single Layer Feed-forward

Input layer of source nodes

Output layer of neurons

Multi layer feed-forward

3-4-2 Network

Input layer

Output layer

Hidden Layer

Recurrent network
Recurrent Network with hidden neuron(s): unit delay operator z-1 implies dynamic system

z-1 input hidden output

z-1

z-1

Neural Network Architectures

The Neuron
The neuron is the basic information processing unit of a NN. It consists of: 1 A set of synapses or connecting links, each link characterized by a weight: W1, W2, , Wm 2 An adder function (linear combiner) which computes the weighted sum of the inputs:

u = w jx j
3

Activation function (squashing function) for limiting the amplitude of the output of the neuron.

j =1

y = (u + b)

The Neuron
Bias

x1

b w1
Local Field Activation function

Input signal

x2

w2

()

Output

M
xm

M
wm
Synaptic weights

Summing function

Bias of a Neuron
Bias b has the effect of applying an affine transformation to u

v=u+b
v is the induced field of the neuron

u = w jx j
j=1

Bias as extra input

Bias is an external parameter of the neuron. Can be
modeled by adding an extra input.

x0 = +1 x1
Input signal

w0 w1 w2

v=

wx
j=0

j j

w0 = b

x2

weights

Local Field

Activation function

()

Output

M
xm

wm Synaptic

Summing function

Segmentation
is the process of dividing an image into regions of related content. is a process of partitioning an image into some non-overlapping meaningful homogeneous regions process of subdividing or partitioning of an image into homogeneous (with respect to some criterion of homogeneity) and disjoint (non-overlapping, non uniform) regions of different statistical behaviour, called image segments

Segmentation

Grouping together pixels that have similar properties such as color, texture, motion, etc Each pixels can be treated as a data point in the feature space

Segmentation
These image regions or segments may be later associated with information labels, but the segmentation process simply gives each region a generic label (region 1, region 2, etc.). In the context of earth remote sensing, these labels would generally be a ground cover type or land use category.
The shape of an object can be described in terms of: Its boundary requires image edge detection The region it occupies requires image segmentation in homogeneous regions, Image regions generally have homogeneous characteristics (e.g. intensity, texture)

Segmentation Algorithms
Segmentation algorithms are based on one of two basic properties of intensity values - discontinuity and similarity. First category is to partition an image based on abrupt changes in intensity, such as edges in an image. Second category are based on partitioning an image into regions that are similar according to a predefined criteria. Histogram Thresholding approach falls under this category.

Segmentation Approaches
Thresholding Boundary Detection Region Growing

Thresholding
Suppose that the gray-level histogram corresponds to an image, f(x,y), composed of dark objects in a light background, in such a way that object and background pixels have gray levels grouped into two dominant modes. One obvious way to extract the objects from the background is to select a threshold T that separates these modes. Then any point (x,y) for which f(x,y) > T is called an object point, otherwise, the point is called a background point.

Gray Scale Image - bimodal

Image of rice with black background

Image histogram of rice

Segmented Image

Image after segmentation

Segmentation
Based on Similarity measures
Region Growing Splitting Split and Merge

Region Growing
A simple approach to image segmentation is to start from some pixels (seeds) representing distinct image regions and to grow them, until they cover the entire image For region growing we need a rule describing a growth mechanism and a rule checking the homogeneity of the regions after each growth step

Split / Merge
The opposite approach to region growing is region shrinking ( splitting ). It is a top-down approach and it starts with the assumption that the entire image is homogeneous If this is not true , the image is split into four sub images This splitting procedure is repeated recursively until we split the image into homogeneous regions

Split / Merge
If the original image is square N x N, having dimensions that are powers of 2(N = 2n): All regions produced but the splitting algorithm are squares having dimensions M x M , where M is a power of 2 as well (M=2m,M<= n). Since the procedure is recursive, it produces an image representation that can be described by a tree whose nodes have four sons each Such a tree is called a Quadtree.

Split / Merge

Quadtree

R0

R1 R0 R3 R1

R2

R00

R01

R02

R04

Split / Merge
Splitting techniques disadvantage, they create regions that may be adjacent and homogeneous, but not merged. Split and Merge method It is an iterative algorithm that includes both splitting and merging at each iteration:

Boundary Based Segmentation

Methods based on Edge Detection
Filters
Sobel Roberts Kirsh Laplace

Expert Systems
Knowledge Based Information System (KBIS) Expert System (ES): A KBIS that uses its knowledge about a specific area to act as an expert consultant to the end user

Expert Systems
Expert Systems and DSS:
ES

are built into a DSS to improve the quality of decision making; helps add structure to unstructured problems. Support a specific problem domain. Uses a knowledge database to support decision making.

Characteristics of an Expert System

Can explain their reasoning or suggested decisions. Can display intelligent behavior. Can draw conclusions from complex relationships. Can deal with uncertainty.

Components of Expert Systems

[Figure 11.7]

Advantages of ES
Serve in the absence of a real expert. Provide consistent advice, unaffected by stress and time constraints. Knowledge can be transferred and reproduced. May be less expensive than an actual expert, especially if the knowledge is used over and over.