Pattern Recognition Letters 1 (1983) 435-442 July 1983

North-Holland

An automatic procedure for image segmentation

D.J. B A R T L I F F
GEC Research Laboratories, Hirst Research Laboratories, Wernbley, UK

Received 31 January 1983

Revised 14 April 1983

Abstract: A procedure for image segmentation involving no image-dependent thresholds is described. The method involves not
only detection of edges but also production of closed region boundaries. The method has been developed and tested on head
and shoulder images.

Key words: Image analysis, scene analysis, segmentation, edge detection, thresholding.

1. Introduction were tried and rejected, but finally a scheme was

This paper reports work carried out as part of
current investigations into image-processing al-
gorithms for the purpose of image analysis. Poten-
tial applications of this work include development
o f an image storage or archiving system as well as
for feature extraction in scene analysis and under-
standing.
One approach in initial processing involves seg-
menting an image into regions of approximately
the same texture in order that each region can then
be analysed individually. To this end the boun-
daries of the regions must be located, thus involving
not only the detection of edges but also the edges
must be linked together to form closed contours.
Many techniques for edge detection have been
developed but each has its disadvantages (Pratt,
1978). In many cases, for example, closed region
boundaries are not produced and indeed most
methods are successful only in a narrow field of
application.
This paper covers work which has been carried
out on 'algorithm engineering', i.e. optimisation
o f combinations of several basic techniques to find
a scheme which improves upon individual methods
and has broader applications.
To this end several combinations of algorithms
evolved which combined information from the
Marr-Hildreth and the Sobel edge-detection opera-
tors. Use was also made of two edge-cleaning
algorithms. The edge segments which have been
detected and accepted by the scheme can be finally
described using a 2-bit chain code. This image-
segmentation scheme has been implemented on a
VAX 11/780 computer.
The next section describes briefly the Marr-
Hildreth and Sobel operators and gives reasons for
the combination of the two. Section 3 looks at the
actual implementation of the scheme on the com-
puter, while Section 4 presents two examples.
Finally Section 5 summarises the work which has
been carried out so far and outlines the areas for
further investigation.
2. The edge-detection operators
Segmentation is a necessary first stage in many
applications of image analysis. By way of example,
in order to achieve a very efficient description of
pictures it is necessary to go beyond sophisticated
coding techniques applied to the raw picture data.
The approach which has been adopted here con-
sists of dividing the image into its component parts

0167-8655/83/$3.00 © 1983, Elsevier Science Publishers B.V. (North-Holland) 435

Volume 1, Numbers 5,6 PATTERN RECOGNITION LETTERS
or regions of approximately the same texture. For
each of these regions parameters can then be ex-
tracted which will enable the future reconstruction
of the texture within that region. These parameters,
together with a description of each region's boun-
dary can then form the final description of the
image. It is required, for our purposes, that the
region boundaries are 4-connected, rather than
8-connected or discontinuous (see Figure I and
Duda and Hart (1973), p. 284).
So in this approach the first stage of processing
is to detect the edges between regions. Noise occurs
in various forms in a real digitised image and pro-
duces many spurious edges in processes which use
purely edge-detection operators. Obviously unless
these edges can be eliminated in some way, this will
lead to more storage space than necessary being
taken up by the final description. Noise can also
give discontinuity of edges in some schemes, making
them useless for segmentation purposes.
Several schemes exist which use output from
edge-detection operators to reduce the number of
such spurious edge points (Pratt (1978), Rosenfeld
and Thurston (1972), Heuckel (1973)), but these
have not proved satisfactory. Many of these apply
a threshold to some value calculated at each candi-
date edge point but this is not satisfactory for two
reasons. Firstly, thresholds which reduce the num-
ber of spurious edge points to a reasonable level
must also reject some edge points which are essen-
"k
Id
Fig. 1. The difference between 4-connected and 8-connected
edges. An n-connected edge, for n = 4 or 8, is an edge along
which each pixel has at least two n-neighbours as edge points.
4-neighbours are given in (a); 8-neighbours in (b). An example
of a 4-connected edge is given in (c), and of an 8-connected edge
in (d).
436
July 1983
tial to the correct segmentation of the image,
leaving discontinuous edges. It is much more diffi-
cult to find criteria for the interpolation of such
broken edges than it is to find criteria for the rejec-
tion of edge points.
Secondly thresholds are very often arbitrary in
nature so that, even if a way could be found
around the first problem, the best threshold for
one picture could vary greatly from that for
another. However, provided that the value in ques-
tion is a reasonable measure of the gradient of the
edge at a point, then very low thresholds can be
used to reduce the number of spurious edge points
slightly.
Below are descriptions of the two contrasting
edge-detection operators which are used in this
scheme. The first one was assumed to detect all the
edges necessary for an adequate segmentation of
the image. However it is sensitive to noise and so
produces a large number of spurious edges. The
second operator is much more robust when used
on noisy images but has too many disadvantages to
be used on its own.
2.1. The Marr-Hildreth operator
The basic edge-detection algorithm in this scheme
involves the use of Marr and Hildreth's Edge-
Detection Theory (1980). In a given image, at the
boundary of a region, there is a zero-crossing in
the second directional derivative of the intensity
function of the image, I(x, y). However to detect
relatively large-scale intensity changes, such as are
required for our purposes, some form of smoothing
Fig. 2a. Cross-section of the Gaussian distribution G.
Volume 1, Numbers 5,6 PATTERN RECOGNITION LETTERS
I within 2w of the perimeter of the image cannot be
Fig. 2b. Cross-section of - V 2G.
of the intensity function must first be carried
out. Marr and Hildreth found that the optimal
smoothing filter is a 2-dimensional, rotationally
symmetric Gaussian distribution (see Figure 2a),
and also found there is no need to use directional
derivatives. The use of the only non-directional
second derivative operator, the Laplacian, leads to
a great saving in computational time and so that
operator is used in this implementation. There is
some evidence to suggest that the human eye uses
such filters in its visual processing.
In this implementation candidate edge points are
located by scanning the signal,
S(x, y) = - A 2 G * I(x, y),
pixel by pixel and finding the zero-crossings. Here G
is the 2-dimensional Gaussian distribution para-
metrised by its width, w, (see Figure 2b), and •
denotes a 2-dimensional convolution over x and y.
I
\ Marr and Hildreth's theory in which zero-crosings
2w
JI
Fig. 2c. Storage of the mask. The value of - V 2G, at the centre
of each square above, is stored in the mask.
July 1983
A zero-crossing is defined to be a pixel at which the
signal S has opposite sign to that of the signal at,
at least one of its 8-neighbours (see Figure 1).
- A 2 G can be calculated before the convolution is
performed and so it can be permanently stored in
the form of a rough quadrant of a circle of radius
2w (see Figure 2c). Rotational symmetry is then
used to generate the rest of the mask which is used
in performing the convolution. A mask of this size
is used because - A 2 G is negligible at distances
greater than 2w from the centre. However points
considered since at these points the mask would
overlap the perimeter. So the edge-detection pro-
cess is carried out on a window formed by ignoring
a margin of width 2w around the original image.
Information from points inside this margin is still
used, for instance in the convolution, but no edges
can be detected there.
The edges detected by this algorithm are 4-con-
nected but are two pixels thick because both sides
of a zero-crossing are marked. This differs from
the implementation by Marr and Hildreth of their
own theory. However the Marr-Hildreth edge-
detection algorithm in its original form has been
found to fail where several regions meet in a situa-
tion such as shown in Figure 3. This problem can
be very noticeable when the algorithm is used on
real images. Even when both sides of each zero-
(1)
crossing are marked as in this implementation,
then the Gaussian filter used must have small
width (w = 4), otherwise the problems of curvature
cannot be overcome.
It may be thought that any edges thus detected
will be inaccurate and also that any edges which
are two pixels apart in the image will be merged by
this process. However the edges when subsequently
reduced to only one pixel thick (see Section 3), will
lie less than one pixel away from the zero-crossing.
Compare this with the original implementation of
are asymmetrically marked on the positive side.
It was assumed that, for a small width of the
Gaussian filter ( w = 4 ) , all the region boundaries
required for an adequate segmentation of the
image would be represented by zero-crossing seg-
ments in the output of the algorithm. Unfortunately
with this small width the algorithm is very sensitive
437
Volume 1, Numbers 5,6 PATTERN RECOGNITION LETTERS
to noise and so a large number of spurious zero-
crossings are produced. The smaller the width the
more sensitive the algorithm becomes. Two
methods have been tried to reduce the number o f
such zero-crossings. First the width o f the Gaussian
filter could be increased; even small increases
greatly reduce the amount of noise, however not
only are some features erroneously merged but
also as stated above problems arise with curvature.
Another disadvantage is that the mask size is in-
creased and so not only takes up more storage
space but the process is also made more computa-
tionally complex and a much larger part o f the
image must be ignored in the edge-detection
process. The second method makes use of the fact
that in general noise produces edges of relatively
small gradients. Thus the gradient at each zero-
crossing could be thresholded to eliminate false
meduim
light
/ for i = 1 and 2, then the magnitude of the gradient
verydark / a
Fig. 3. (a) The true regions. (b) The edges detected by the
original Marr-Hildreth algorithm.
438
July 1983
edges. However as stated previously this is a
dangerous operation if it is used as the only process
to reduce the number of spurious edges and it has
been found in this case to produce discontinuous
edges.
In their work Marr and Hildreth suggest that in-
formation from a number of convolutions, using
Gaussian filters of small and large widths, should
be combined. There is some evidence to suggest
that filters of different widths are present in the
human eye, but whilst some work has been done
on this in relation to stereovision it is not at all
clear how the information could be combined for
our purposes.
2.2. The Sobel operator
The Sobel operator (Duda and Hart (1973),
p. 271) produces an approximate gradient at each
pixel, except those on the perimeter o f the image,
by differencing the intensity function in two per-
pendicular directions using the 3 x 3 masks
GI=
I! 0!1 I i il
0
0
, G2= 0
-2 -
Let the inner product of the mask Gi with a 3 x 3
. (2)
window of pixels, centred on a pixel P, be Gi(P)
at P is
A (P ) = ( G 1(p)2 + Gz(p)2)l/2. (3)
When this algorithm is used on its own, the method
of producing edges is to threshold the gradient
magnitudes thus calculated. Again thresholding
does not give good results for reasons given pre-
viously. Other methods of using the information
from the Sobel operator, such as tracing local
maxima of the gradient, have also not produced
satisfactory results. Two reasons being their com-
putational complexity and their use of arbitrary
parameters.
However this operator is much less sensitive to
noise than the Marr-Hildreth operator and again
gives, in general, low values for the gradient at
false edge points. Thus the task of the work
covered here was to take the zero-crossing seg-
ments produced by the modified Marr-Hildreth
Volume 1, Numbers 5,6 PATTERN RECOGNITION LETTERS July 1983

algorithm and by using the gradients calculated in
both algorithms reject whole edge segments thus
leaving 4-connected edges.
3. Implementation of the edge-detection scheme
A flow chart showing the position of each pro-
cess in the scheme is shown as Figure 4.
The output from the modified Marr-Hildreth al-
gorithm is an array of integer gradient values fac-
tored to the range 0-255 for ease of storage and
display. Pixels at which there is no zero-crossing
are given a gradient value of 0. Since the edges are
two pixels thick they are reduced to one pixel by
using a thinning algorithm. Similarly the output

l Digitised Picture Array

1
+

IObtain the gradient values at

Marr-Hildreth zero-crossings
'Obtain the gradients from
I the Sobel operator

ITHIN t h e a r r a y

Calculate the Product of the

two gradient values

l hreshold the product at I I

+
Threshold the Product at 255

ISort the edge array

+
I

IFill-in 8-connected edges 1

f
I
t
Chain code edge segments,
rejecting those without 'probable' points

+
Store codes for accepted segments

Fig. 4. Implementation of the scheme.

439
Volume 1, Numbers 5,6 PATTERN RECOGNITION LETTERS July 1983

from the Sobel algorithm is an array of integer pixels which had previously been junctions may at
gradient values, also factored to the range 0-255. this stage have only one edge segment either en-
These arrays are both windows of the original tering or leaving the junction, where there had
image area as described in the previous section. previously been three or four. The remaining seg-
Some initial experiments, involving calculation ment can then also be rejected as it is not active in
o f the product, at each pixel, of the two gradient the division of the image into regions. Thus each
values, showed that at least one pixel at which the junction is continuously examined until no further
product is 255 or more is included in each edge segments can be removed in this way. Then the re-
essential to the segmentation of the image. More- maining segments have their codes and starting
over many o f the false edges consisted entirely o f points recorded in a data-file for future reforma-
pixels at which the product was" less than 255. Thus tion of the image's segmentation.
the array of product values is thresholded twice. It has been previously stated that thresholding
Once to give an array of 'possible' edge points, was a dangerous operation and yet it is used in this
those with product 1 or greater, and the second scheme on three separate occasions. However two
time to give an array of 'probable' edge points, o f them are very low thresholds which are applied
those with product of 255 or more. The pixels with indirectly when the gradients from the Marr-
product 0 can be safely discarded since these points Hildreth and Sobel operators are factored to the
are either not zerocrossings in the Marr-Hildreth
algorithm or give extremely small gradient values
for either of the Sobel of Marr-Hildreth operators.
The final segmentation required lies somewhere
between the 'possible' and the 'probable' array.
The 'possible' array has too many spurious edges
wilst the 'probable' array does not consist of con-
tinuous edges. However a large number of spurious
edges can be removed from the 'possible' array by
the use of a sorting algorithm, which removes
those edge points not active in the segmentation of
the image.
Advantage is now taken of a fact mentioned
earlier in this section. Namely that many false
edges contain no 'probable' points. For an effi- Fig. 5. The earlier noisy image.
cient description of each region's boundary, edge
segments have to be coded and in this scheme a
2-bit chain code is used (see Duda and Hart
(1973)). An edge segment is defined to be a 4-con-
nected edge linking any two junction points, a
junction to the perimeter of the window of the
image or two pixels on the perimeter. The chain
coding is done at this stage and after each segment
is coded it is recorded whether or not the edge seg-
ment (including the first and last pixels) contained
any 'probable' edge points.
When all the edge segments have been coded,
segments with no 'probable' edge points are re-
jected as a whole, except for segments consisting of
only two points which are not considered long
enough to give trustworthy information. Some Fig. 6. The image with reduced noise.

440
Volume 1, Numbers 5,6 PATTERN RECOGNITION LETTERS July 1983

integer range 0-255. It is thought that there will be

very few pixels at which either gradient is given the
value 0 when in fact both are non-zero. The third
application of a threshold is also permissible since
points which have gradient product less than 255
can still be included in the picture. Moreover the
threshold is not arbitrary in that it is fixed by the
maximum gradients obtained from the Marr-
Hildreth and Sobel algorithms.

4. Examples
The scheme has been developed and tested using
real images. These images are 128x 1 2 8 x 8 bit
monochrome pictures thus having 256 grey levels.
Two examples are presented here, both images
consisting of head and shoulder portraits (see
Figures 5 and 6). The two images were produced at
different times, the earlier one, which is examined
first being very much noisier than the other. Faces
represent tough challenges to edge-detection
schemes because they contain very few sharp boun-
daries between regions. This scheme, however,
works particularly well on the less noisy picture,
and although understandably less well on the
other, still produces an adequate segmentation in
this case.
4.1. Image with a high level o f noise
Obviously with such a large amount o f noise
present in this image, there are going to be many
more unwanted edges in the final segmentation of
this image than the second one. However the
scheme can still discern the important features of
the image in spite of the high level of noise. The
amount of noise in this picture is particularly
reflected in the large number of spurious edges
detected in the background to the face. In the final
segmentation produced by the scheme there are
around 180 regions (see Figure 7).
Fig. 7. The final segmentation of the first image.
w = 3 could be used on this image because as stated
in Section 2, decreasing w has many advantages.
However even on this image with much less noise
too many spurious edges were detected by the
modified Marr-Hildreth algorithm for that mask
size to be used in the scheme, so again w = 4 was
used. There are only about 110 regions in the final
segmentation of this image (see Figure 8). The
original image in this case is certainly no less com-
plex than the previous example and so the differ-
ence in the number of regions found in the images
is explained by the reduction in the level of noise
in the second image. Comparison of Figures 6 and
8 shows that the scheme has produced a reasonably
accurate segmentation with not too many un-
wanted edges.
S. Conclusions
In this paper a scheme has been described for the

4.2. Image with reduced noise

This example is presented in more detail since

the standard of the original image in this case is
suitable for many applications.
It was hoped that a Gaussian filter of width Fig. 8. The final segmentation of the second image.

441
Volume 1, Numbers 5,6 PATTERN RECOGNITION LETTERS
segmentation of images. An important feature of
the scheme is that there are no image-dependent
parameters or thresholds, thus making it suitable
for automatic implementation. The aim of the
method is to provide a complete decomposition of
an image into its constituent regions, and therefore
it involves not only detection of edges but the pro-
duction of closed region boundaries. The segmen-
tation procedure has been developed and tested
using real head and shoulder images, and gives
very good results even when significant amounts of
noise are present.
The basis of the method consists of combining
information from the Marr-Hildreth and the Sobel
edge detection operators in such a way as to utilise
the best qualities of each. On its own the former
has the advantage of tending to produce closed
contours, but was found to be too sensitive to
noise unless wide masks were used resulting in loss
of detail and excessive computational complexity.
The Sobel operator on its own is suitable for hard-
ware implementation but does not give closed con-
tours. The method described here avoids the above
difficulties. It is presently implemented on a VAX
11/780 computer, but real-time hardware imple-
442
July 1983
mentation should be possible in the future using
appropriate VLSI circuits.
Segmentation, as described here, forms an essen-
tial part of the feature extraction process needed
for the efficient description of images, and as the
front end for pattern recognition and scene under-
standing on complex images. As part of the on-
going research texture analysis is also being
studied.
References
Duda, R.O. and P.E. Hart (1973). Pattern Classification and
Scene Analysis. John Wiley, New York.
Freeman, H. (1961). On the encoding of arbitrary geometric
configurations. 1RE Trans. Elect. Comp. 10 (June), 260-268.
Heuckel, M. (1973). A local visual operator which recognises
edges and lines. J. A C M 20(4) 634-647.
Marr, D. and E.C. Hildreth (1980). Theory of edge detection.
Proc. Roy. Soc. B207, 187-217.
Pratt, W.K. (1978). Digital Image Processing. John Wiley, New
York, pp. 478-499.
Rosenfeld, A. and M. Thurston (1972). Edge and curve detec-
tion: further experiments. IEEE Trans. Computers 21(7)
677-715.