A Study on Traffic Sign Recognition in Scene Image Using Genetic Algorithms

and Neural Networks

Yuji Aoyagi and Toshiyuki Asakura
Faculty of Engineering
Fukui University
3-9-1 Bunkyo Fukui, Fukui, 910 Japan
Abstract - This research is concerned with a new technology for
the recognition of traffic sign from a scene image using genetic
algorithms (denoted by GA) and neural networks (denoted by
First, an original video image is obtained as a binary
image by using the smoothing filter and the laplacian filter.
Second, the traffic sign in a scene image is searched using
proposed GA. Third, if there exist two traffic signs in a video
image, the method to detect the second traffic sign is developed
based on the fitness of individuals, Finally, it can be recognized
using NN what kind of traffic sign is taken. Through these
recognition experiments, it can be shown that the new
technology proposed here is sufficiently valid for the
recognition of traffic sign from a scene image.

").

has been proposed [6].However, if the extracted area is not

a circle pattern, the sign might not be completely detected
from the area. Also, though the hue of the traffic sign must
be same, the hue changes hard by the distance, time ,
weather, passing year's deterioration and so on. Then, the
recognition depends largely on the degree of a detected hue.
In this research, the technique based on GA [7] is
proposed to recognize a circle traffic sign by using only
brightness. This method is thought as a searching problem
from the scene image. By treating both the position and the
size of the traffic sign as gene information, the method
detecting the traffic sign is developed. Moreover, by
applying NN [8] to the detected sign, it is recognized what
kinds of traffic sign is found.

I .INTRODUCTION

1 . IMAGE PROCESSING FOR TRAFFIC SIGN

A former image pattern recognition has been chiefly

researched only for an individual object. However, it is an

advanced direction to recognize the object which becomes a
target from a scene image, with the development of the
visual system of the robot [l].But, it is generally difficult
to recognize a specific object in such scene image. Because
there exist many other objects and unexpected targets in an
image, and also, a degree of freedom for the relative
position between a target and a video is large [2], [3].
Especially, in recent years, autonomous vehicles such as
mobile robots is widely researched. For these robots running
on a road, it is necessary to know the information of an
obstacle, a traffic sign and so on. Though there are a lot of
reports in which the object is only the traffic sign, there are
few reports which recognizes the traffic sign from the scene
image ~41,FI.
In order to recognize the object in the image, it is
necessary to separate the object from the background.
However, it is difficult to separate the traffic sign from the
scene image similar to a man's sight. Because the traffic
sign in an image has the extremely many degrees of
freedom with respect to the position, size and background.
As a recognition method, the area candidate which the
traffic sign exists has been specified by using the hue and
then the method to recognize the traffic sign from the area

0-7803-2775-6196 $4.00 0 1996 IEEE

In this research, a scene picture of the traffic sign is

taken with a video camera installed in the car under running
beforehand. The image is memorized in the frame memory
as a digital image. The memorized image is processed on
the computer (CPU: Intel Pentium 90MHz) by the image
processing algorithm proposed here.
Fig. 1 shows the algorithm of the image recognition,
which is composed of two modules. That is, the one is the
traffic sign detection module by GA and the other is it's
recognition module by NN. Fig. 2 shows one example of a
video image which the image processing is performed.
In this research, by using only the brightness, the
existence area of the traffic sign is detected from the image.
In generally, in order to detect the area of the traffic sign,
the hue is used. However, in the case of using the hue,
there are following problems.
(1) The image processing time increases because of an
increase in the amount of the information.
(2) It is difficult to extract the same hue area because of the
traffic sign hue is not always same by the distance from
the camera to a sign, the weather, and so on.
In using only the brightness, there does not arise such
problems as stated above. But, because of the information
which can treat is little, the candidate area of the traffic sign

1838

has increased. In this research, by using only brightness, the

method of detecting the sign is proposed.
In order to detect the traffic sign by the brightness, it is
necessary to separate the object from the background by the
difference of brightness. To extract the outline and the edge
in input image, the laplacian filter is applied, which is one
of differential filters and shown by the following equation.

g(i,j)=f(i,j-1)+f(i-1,j)+f(i+l!j)+f(i,j+l)-af(i,j)
(1)

Here, i and j are the pixel coordinate on the image, f the

brightness before processing and g the one after processing.
In using such differential filters, the noise included in the
image is often emphasized. Then, before applying the
laplacian filter, the smoothing filter is often used for the
noise removal. Among many smoothing filters, the filter
with eight neighborhood weight is used. This filter can be
shown by the next equation.

g(i,j)= { f(i-1,j-1 )+f(i,j-I) +f(i+l,j-I)

f(i-1,j) + W , j ) +f(i+l,j>
f(i--l,j+I )+f(i,j+l) +f(i+l,j+l ) 1/10

C
Fig. 1 Image processing algorithm

(2)

M e r (1) and (2) is applied to Fig. 2, the binary image is

obtained by using next equation.

where T is threshold, which is set as T=3. Then, Fig. 3

shows the binary image of Fig. 2.

II . DETECTION OF TRAFFIC SIGNS BY GA

Fig. 2 An example of image

A. Application of GA
In order to detect the traffic sign from a video image of
Fig. 2, it is necessary to explore a circle pattern which
shows the traffic sign. However, since there exist a lot of
patterns in the background of the traffic sign, it is difficult
to detect the traffic sign in the image. Then, to find the
object which might be a circle among many patterns in the
image, the circular ratio is measured. However, in case that
the- searching pattern separates into some parts, it is
necessary to treat as one pattern by relating the separated
pattern. Therefore, the traffic sign can be detected by
preparing a circle standard pattern, which the pattern
matching is performed on the image. In this research, GA
with the searching ability is applied for this problem and a
circle pattern in the image is matched by giving the gene
information.

1839

Fik. 3 Binary image

By using GA, the circle trafic signs are found from the
image shown in Fig. 3. Though the original image of Fig. 2
is a size of 256 X 256, the searching image is used the one
which reduced to 128 X 128 because of the decrease the
searching points of GA. For the searching problem of the
traffic sign in the image, the k-th genes of individuals are
expressed by following equation [9].

ability in the local and global area has been improved. Each
parameters of GA is as follows.

Gk = ( X k t y k t s k )

The search is desirable to stop when the traffic sign is

found in the image. However, it is difficult to give the
definition of trafic sign for the searched pattern. Then, GA
finished when the alternation generation time becomes 150
times. When the search is finished, the information which
has the maximum fitness on the position and the size of
individual is adopted as the searched result. The fitness of
the k-th individual is shown by the next equation.

(4)

where each parameter, xt , y

following ranges.
Xk
Yk
s k

< [o

c [o
c [o

t
t

1271
1271
151

, and

s k

are defined in the

(5)
(6)
(7)

Total of individuals
Selection rate
Mutation rate
Alternation of Generation time

Hence, the gene is expressed by a 18bit length as follows.

1
fk=

%k

1+O.l

S k

(8)

The gene expressed here decides both the position and the
size of the searched standard pattern, and the circle with the
size of a standard radius ( r=15 ) in the image is expanded
or reduced by using the next equation.

: 32 individuals
: 30%
: 10%
: 150 times

f:(Pi-bi ) +a
i -1

(search jnzttem brightness)

0 (background)
b=
1 (object)
a=10

p=l

where p is a brightness value of the searched pattern, b is a

brightness value of the image, n is a composition point

k%

Individual coding

B. Evolution algorithms of GA
The evolution algorithm in GA is shown in Fig. 4. GA
used here is applying to a multi-points crossover for the
crossover, a two points change for the mutation, and an elite
preservation strategy method for the selection. GA study has
the following characteristics. That is, by doing one point
mutation after calculating the fitness, the fitness is improved
and one point exchange is done. If the fitness is not
improved, the operation is done which returns gene. By
doing this processing, the optimum solution is searched at
high speed.
GA used here is applied to the elite preservation strategy
method. This method has possibility that elite's individual
rapidly extend in population, and there exists a danger that
falls into the local minimum solution. In this research, the
reproduced individuals are compared with the higher
ranking individuals. If the individuals are the same, the
same individual is reproduced by repeating the mutation of
one point change. By adding this process, the searching

1840

Fitness calculation

Selection

Two point crossover

,
I

)+(
Fig. 4 Evolution algorithm in GA

number of the searched pattem and a standard radius is

r=15. Also, a is added when the search pattem is explored
outside the image area.

C. Detection of Another Trafic Signs

It is assumed that traffic signs exist in the image
obtained. But, the number of traffic signs is not necessarily
one in the image. Then, the method is proposed to detect
second sign by the search of all individuals.
The individuals converge to some solutions that show the
traffic sign in an image space. The state of individuals after
the searching is shown in Fig. 5. The solid curve is the
solution and the black points are each individuals. As the
searching is adopted only for an individual with the
maximum fitness, the number of traffic sign obtained is one
for one image. Therefore, in order to detect some traffic
signs, the property of GA is used that individuals converge
on some solutions.
As shown in Fig. 6, among individuals which converge to
each solution, an individual with the higher fitness is
elected, after the searching. An elected individuals as the
second sign is decided.
In the case that there are plural traffic signs in the image,
these signs often line up sideways and vertically in the same
size. Then, the decision of the second solution is given by
the next conditions.

GA could detect two trafic signs, "one sign" the case GA

could detect only one traffic sign, and "don't detect" the
case GA couldn't detect the traffic sign.
In this result, though there are some failure examples, the
reasons are the followings. First, the reason is that GA is
converges easily to the neighborhood of the sign with a
high fitness. That is, even if GA is searching for the sign

Fig. 5 Search resilt in searching space

solution

Fig. 6 Election of individual on each solution

/i-q
second individual

where r 1 is the radius for the individual of first solution,

r z is the radius for the one of second solution, and the 4 is
distance between individuals. In this way, two individuals
are decided as shown in Fig. 7 and then these are adopted
as traffic sign.

Fig. 7 Decision of two individuals

D. Experiment of Trafic Sigm Detection

In order to show the effectiveness of the proposed
evolution algorithm, the application of general GA to the
image of Fig. 3 is discussed. Fig. 8 shows the evolution
process of the optimal individual to the image in Fig. 3.
Here, Type 2 is the proposed GA and Type 1 is GA to
permit for the individual to overlap with Type 2.
As the result, it is shown that the proposed GA is
evolving into a faster and higher fitness than another GA.
For two images, the states of the searching by GA are
shown in Fig. 9. Furthermore, Table 1 shows the searching
results to 30 images in which two signs or one sign exists
in the image. In this table, "two signs" shows the case that

1841

0.7
a,

sg
U

a,

_ _i"... ....*.._
_ . -_..._-._..--..I. _..... _.---.
I
F""'
--.._

0.6

0.5

_.

GA Type2
- . -GATypel
GA

50
100
Alternation of Generation Time

Fig. 8 Evolution process of each GA

150

A. Composition of NN

Table 1 Detection result by GA

(b) the case of one sign
(a) the case of two signs

1-1

two signs
one sign

one sign
no detection

1
4
1

with a low fitness and one individual is searched for the

sign with the maximum fitness, other individuals also follow
to the individual. Second, the maximum fitness is not
always to be a traffic sign. Third, in the case that there are
some searching points which has an fitness near the
maximum fitness, the search to the maximum fitness
becomes insufficient.
However, even if the separation with the background is
not performed sufficiently, or even if a sign pattern is not
obtained as a complete circle pattern, it was shown that the
proposed GA is effective as the detection method.

N . A TRAFFIC SIGN RECOGNlTION MODULE BY

NEURAL NETWORKS

The recognition category is classified into three classes;

the speed sign (speed), other traffic sign (sign), and not a
traffic sign (other). For the size of the detected pattem is
various, the input pattern is made regular for the size of
18 X 18. Fig. 10 shows the construction of NN. Here, the
number of input layer units is 324, hidden layer units 15
and output layer units 3. The learning of NN is performed
by using the back propagation method and finishes when
the square error E of the output satisfies the following
condition.

Here, u f k )shows the output from the j-th output layer

units for the k-th learning pattern and a teacher signal.
The teacher signal is given by "1"for the j-th output layer
units and "0" for other units. Moreover, the number of
learning patterns are 27 patterns for "speed", 26 for "sign1',
28 for "other". Therefore, the sum is k=81 patterns.

Though the traffic sign was detected by the proposed GA ,

it is not obvious whether the detected sign is a speed sign B. Experiment of Trafic Signs Recognition
or other sign. Hence, in this chapter, it is recognized by
The recognition experiment was performed by using NN
using NN whether it is a traffic sign or not.
shown in Fig. 10. However, if the output value of NN is
smaller than 0.50, it is not judged whether it is a traffic
I
1
I
I
sign. Because recognition result with possibility to muse the
misjudgment is that the reliability of the system is
decreased.
Fig. 11 shows one example for the experimental result of
recognition. Moreover, the result of recognition experiments
for all detected image patterns is shown in Table 2. The
number of all detected image patterns is 48, in which
"speed" pattern 24 patterns, "sign" one 24 patterns, and
"other" no patterns.
(a) 50 generation time
In this table, among 24 patterns of the speed sign, 23
@) 150 generation time
pattems succeeded but one pattern was recognized as other

output

speed
sign
Other

(a) 50 generation time

(b) 150 generation time
Fig. 9 The searching by GA

Fig. 10 Neural networks

1842

(2) By evaluating the matching of the searching pattern, it

was verified for a traffic sign to be detected even if
traffic sign has some shapie loss.
(3) As the same individual is not generated in the evolution
process of GA by the repeating of the mutation, both
the global and local searching abilities of GA are
improved.
(4) By evaluating the indiviclual after the searching, it was
clarified that 'the second traffic sign can be easily
detected.
(5) To recognize what the delected traffic sign is, the
method by using NN has been proposed and the
effectiveness was demonstrated through the recognition
experiments of a traffic sign.
From these results, the effectiveness of the recognition
system of the traffic sign proposed here was verified.

(a) Image A

[l]M. Yachida, "Motion vision and environment

@> ImageB
Fig. 11 Experimental result of recognition
Table 2 Experimental result of recognition

sign
other

23
1
0

0
23
1

0
0
0

traffic sign. Moreover, among the 24 patterns other than the

speed sign, it was recognized that only one pattern was not
a sign. From these results, it could be evaluated that the
recognition was excellently performed, although there exist
many other patterns in the background.

v . CONCLUSION
From a scene image which changes by the distance, time,
and the weather, etc, the research to recognize a traffic sign
has been developed. In this research, the method using GA
was proposed which detected the traffic sign only from the
brightness image. Moreover, in order to specify the detected
traffic sign, the speed sign was recognized by using NN. At
a result, the followings were obtained.
(1) The detection method of a traffic sign has been
proposed which searches for a circle pattem by using
GA and judges a sign from the learning of NN.

recognition, "ISCIE Trans., vol. 35, no. 12, pp.

754-761,1991.
[2] M. Asada, "Robot Vision,"ISCIETrans., vol. 34, no. 7,
pp. 386-396,1990.
[3] K. Nagano, K. Miyamoto, M. Tamagawa, Y. Ida, and I.
Fujita, "Application of image processing and pattern
recognition,"Tech. Rep., Mitsubishi Heavy CO., vol. 28,
no. 6, pp. 586-588, 1991
[4] N. Yabuki and N. Miki, "Detection of max speed sign
and discrimination of maK speed in road scene image,
"IPS. Trans., cv, vol. 76, no. 18, pp. 127-134, 1992
T. Hibi, T.Yamashiro, M. Kamata and T. Ohka, "A
recognition of road sign $or automatic vehicle contro1,"in
Proc. SICE. '94, pp. 555-556, 1994
T. Asakura and Y. Aoyagi, "Image recognition of traffic
speed marks using neural networks,"JSME. Trans., vol.
61, no. 589, pp. 206-212,, 1994
D. E. Goldberg, "Genetic algorithms in search,
optimization, and machine learning, "Addison - wesley,
ppl-88, 1989
D. E. Rumelhart, G. E. Hinton, and R. J. Williams,
"Learning representations by back-propagating errors,"
Nature, no. 323, pp. 533-536, 1986
T. Agui and T. Nagao, "CBmputer image processing and
recognition, "Shokoudon Publishing, Co. Ltd., pp.
190-201,1992

1843