Image processing designtask: license plate localization

Tom Jongsma s00662301

March 26, 2008

1 address:
Faldenserwei 28, 8834 XJ Baard - t.m.jongsma@student.utwente.nl - educational
program MSc. EE Embedded systems track
 CONTENTS

1 Introduction 2

2 A first attempt 3

3 A second attempt 4
3.1 Select license plate candidates . . . . . . . . . . . . . . . . . . . . . . . . 4
3.2 Select the true candidate that covers the license plate . . . . . . . . . . . 5
3.3 Crop the image to the area of the license plate . . . . . . . . . . . . . . . 5
3.4 Performance evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

4 Conclusions and recommendations 7

5 Matlab code 8

1
CHAPTER 1
INTRODUCTION

License plate localization is an important part of the subject of license plate recognition,
it limits the search space of the recognition part, so it can be computed faster. License
plate recognition is an important subject in todays life. There are nowadays many motor
vehicles on the road. It useful to identify the motorvehicles for a lot of applications.
For example for automated tolling systems, traffic monitoring, public safety and security,
speed control and road prizing. A license plate localization system consist of the following
subsystems:

• image acquisition

• license plate finder

• char finder

• optical character reading

• syntax check

In this project only the license plate finder stage is implemented in Matlab using
the image processing toolbox. The teacher provided 171 images of the back of different
cars to analyze and find the location of the license plate in those images. There is a lot
information about license plate localization to be found on the internet. A lot of them
make use of a neural network. Some make use of correlation (they correlate the image
with a square wave with the ’frequency’ of the letters on the license plate), Gabor filtering
or Hidden Markov models. In this report the selection of the location of the license plate
is based on edge density.

2
CHAPTER 2
A FIRST ATTEMPT

In the first attempt a mask is made to filter the white lines on the road out. The white
lines are on almost the same position in the images, so that was in easy task to do.
Edge-images of two of the sample images were made and multiplied with each other. So
only the edges that are on the same position remain. Then with dilation operations and
taking the inverse of that, a mask to reject the lines from te images was ready. Then
the white pixels in an edge image was counted by row. The row with the most edges
is selected as the location of the license plate. It had not a good performance. The
main problem in this method is that when the guess of the license plate location was not
correct, there were no other candidate area’s to look at and it stuck, that’s why I started
at the beginning with a new attempt with more candidate area’s. The code of this first
attempt is not nice readable and not part of the final result and therefore not in this
report(only in the .zip file).

3
CHAPTER 3
A SECOND ATTEMPT

3.1 Select license plate candidates

This try is also based on edge detection. A property of the license plate is that it holds a
lot of vertical edges (the transitions between the yellow background and the black letters
are sharp edges). This time a sobel vertical mask is used to select vertical edges in the
image, so the license plate area is selected as a candidate, because that holds a lot of
vertical edges. This method is based on counting of edges in an area. The area to be
evaluated is 10 pixels height and 60 pixels wide. It highlights an area of 20 pixels height
and 60 pixels wide. This choice was made after a lot of trial and error. After that step,
the number of candidate areas is evaluated. When there is more than one candidate area
detected, a next selection criterion can be made. Too small or too large area’s can be
rejected from possible candidates. An example of the output of this stage can be found
in Figure 3.1

Figure 3.1: An example of the output of stage 1

4
 In a lot of images there is more than one candidate license plate area detected. From
the candidates the true license plate has to be selected. The details of that step is made
clear in the next section.

3.2 Select the true candidate that covers the license

plate
There is already made a selection based on edge density and area. This function first
selects on aspect ratio. This cannot be a very discriminating function, because the
previous step sometimes selects almost square area’s for the area that covers the license
plate. But things like large vertical lines can rejected as a candidate by this step. After
that step a step based on edge density is processed.
It selects the candidate area with the most edges per area as the true license plate
candidate. An example of the output of this stage can be found in Figure 3.2

Figure 3.2: An example of the output of stage 2

3.3 Crop the image to the area of the license plate

This function lays a mask with approximately the dimension of a license plate on the
image and selects the area with the most edges as the license plate area and crops the
image to that area. An example of the output of this stage can be found in Figure 3.3

5
 Figure 3.3: An example of the output of stage 3

3.4 Performance evaluation

The performance evaluation can’t be realized by the computer alone, because the next
steps like optical character recognition aren’t realized in this project...then it would have
been easy to do a syntax check and let the computer decide if there was a true license
plate detected. When the syntax is correct the change should be very high a true license
plate was analyzed, but in this case the correct localization of the license plate has to be
evaluated by hand. The following selection criterion is used: when all the characters on
the license plate were readable it is counted as a success, when there is doubt about 1
letter or more it is counted as a fail.
With the functions candsel, truelpselect and lpcrop executed with the standard settings
43 license plates were fully readable. That’s a success rate of 25,1 percent. (the standard
settings are for the candsel function: Threshold = 70, minarea = 1000, maxarea = 10000
and for the truelpselect function: minratio = 4, maxratio = 12)
After searching for possible problems it was clear that the truelpselect function had a
very bad performance.
After that not so good result there was tried to find the license plate with the candsel
function with standard settings (see above for setting details or look in the help file of
the function) and the lpcrop function. That gave a huge performance boost, because 137
license plates were fully readable. That’s a success rate of 80,1 percent.
Because I was curious if the first step was really needed, because the candsel function
and the lpcrop function are very similar, I tried to find the license plate with only the
lpcrop function. It gave nearly the same performance as the candsel combined with the
lpcrop function, but it was a lot faster, because it saved all the computations of the
candsel function.

6
CHAPTER 4
CONCLUSIONS AND RECOMMENDATIONS

The best achieved result in this project is a success rate of 80,1 percent. In a lot of
applications you need a success rate of (almost) 100 percent, because you need the license
plate numbers of every car for most applications. A license plate localization system with
a lot lower performance is not very usable. That doesn’t point out the license plate
localization system made in this project is useless, but when implementing it in a real
application it needs other techniques to achieve a higher success rate, so it has to be
combined with other license plate localization techniques, like the use of a neural network.
In some situations this algorithm make a mistake and detects a lot of edges on the side
window of a car, this can be solved with for example acquiring stereo images instead of a
single image so the algorithm can only look at the back of the car for a license plate and
isn’t influenced by a lot of edges in the side windows. The algorithm used in this project
is not fast, but it can be optimized, for example by first detecting the position of the car
and crop the image to the dimensions of the car so the search space of the algorithm is
much smaller. It can also be computed in parallel by splitting the image in more parts
and process that in parallel and select the candidate with the highest edge density at the
end.

7
CHAPTER 5
MATLAB CODE

8
0001 function [candidates] = candsel(image, threshold, minarea, maxarea)
0002 %CANDSEL Selects candidate license plate areas in an image
0003 % This method selects the candidates based on edge density and candidate
0004 % area.
0005 % CANDIDATES = CANDSEL(image, threshold, minarea, maxarea, customtweaks)
0006 % image = the source image
0007 % threshold = the number of white pixels in an area of 10x60 pixels
0008 % required to become a candidate
0009 % minarea = the minimum area required to become a candidate
0010 % maxarea = the maximum area required to become a candidate
0011 % customtweaks = when this value is set to 1, custom tweaks are executed
0012 % that are only applicable on images of the test set
0013 % Recommended values:
0014 % Threshold = 70
0015 % minarea = 1000
0016 % maxarea = 10000
0017 % customtweaks = 0
0018
0019 filtered1 = edge(image,’sobel’,[],’vertical’);
0020
0021 mask2 = zeros(463,756);
0022 %threshold = 70;
0023 %initialize variables
0024 ouderecord = 0;
0025
0026 %loop that counts the white pixels in a block and when that number of white
0027 %pixels is larger than a threshold it is unmasked
0028 for i=1:1:696
0029 for j=1:1:453
0030 maskedimage = filtered1(j:j+10,i:i+60);
0031 totaalwittepixels = sum(maskedimage(:));
0032 if (totaalwittepixels > threshold)
0033 mask2(j:j+20,i+1:i+60) = 1;
0034 else
0035 end;
0036 end;
0037
0038 end;
0039
0040
0041 %count the number of candidate areas
0042 [L,NUM] = bwlabel(mask2,8);
0043 numvalues = length(unique(L));
0044
0045 %when there is more than one candidate a selection on the area is done
0046 if (numvalues > 1)

9
0047
0048 for i=1:1:NUM
0049 plaatje = zeros(463,756);
0050 for k=1:1:756
0051 for j = 1:1:463
0052 if (L(j,k) == i)
0053 plaatje(j,k) = 1;
0054 end;
0055 end;
0056 end;
0057
0058
0059
0060 new = sum(plaatje(:));
0061
0062 if ((new >= minarea) && (new < maxarea))
0063 ouderecord = ouderecord+plaatje;
0064 ouderecord = ouderecord > 0;
0065 end
0066 end;
0067 end;
0068
0069
0070 ouderecord = uint8(ouderecord);
0071 eindplaatje = ouderecord.*image;
0072
0073 candidates = eindplaatje;
0074

10
0001 function [winningcandidate] = truelpselect(image, minratio, maxratio)
0002 %TRUELPSELECT Selects the true license plate from the candidates
0003 %WINNINGCANDIDATE = TRUELPSELECT(image, minratio, maxratio)
0004 %selects from the candidate license plates the true license plate based on
0005 %aspect ratio and highest edge density
0006 %the minratio defines the minimum aspect ratio
0007 %the maxratio defines the maximum aspect ratio
0008 % recommended values:
0009 % minratio = 4
0010 % maxratio = 12
0011
0012 %initialize variables
0013 rightcandidate = 0;
0014
0015 %get the number of candidate areas
0016 [L,NUM] = bwlabel(image,8);
0017
0018 %evaluate the size of the candidate area
0019 for k=0:1:NUM-1;
0020 candidate = (L > i) & (L < i+2);
0021 miny = 1000; minx = 1000; maxy = 0; maxx = 0;
0022 D = size(image);
0023 for l=1:1:D(1)
0024 for j=1:1:D(2)
0025 pixel = candidate(l,j);
0026 if (pixel == 1)
0027 if (maxy < l)
0028 maxy = l;
0029 elseif (maxx < j)
0030 maxx = j;
0031 elseif (miny > l)
0032 miny = l;
0033 elseif (minx > j)
0034 minx = j;
0035 end;
0036
0037 end;
0038
0039 end;
0040 end;
0041 candidate = uint8(candidate);
0042
0043 %calculate the height and width of the area
0044 height = maxy - miny;
0045 width = maxx - minx;
0046

11
0047 %calculate the aspect ratio
0048 aspectratio = width/height;
0049
0050 %make a selection based on the aspect ratio
0051 if (aspectratio > minratio) && (aspectratio < maxratio)
0052 rightcandidate = rightcandidate + candidate;
0053 end;
0054 end;
0055
0056 %mask the image with the candidates that are remaining after the aspect ratio
0057 %selection
0058 candidatesover = uint8(rightcandidate).*image;
0059
0060 %detect edges in the remaining candidates
0061 edgescandidatesover = edge(candidatesover,’sobel’,’vertical’);
0062
0063 %get the number of remaining candidates
0064 [L,NUM] = bwlabel(rightcandidate,8);
0065 nrofedges = 0;
0066
0067 %check if further processing is needed
0068 if (NUM > 1)
0069 disp(’more than 1 candidate left, further processing needed’);
0070
0071 %calculate the edge density in the remaining candidate areas
0072 for x=0:1:NUM-1
0073 candidate = (L > x) & (L < x+2);
0074 oppervlakte = bwarea(candidate);
0075 edgedensity = candidate.*edgescandidatesover;
0076 nrofedges(x+1) = sum(edgedensity(:))/oppervlakte;
0077 end;
0078
0079 else
0080
0081 end;
0082
0083 %select the remaining candidate with the highest edge density
0084 [Y,I] = max(nrofedges);
0085
0086 winningcandidate = (L > I-1) & (L < I+1);
0087 winningcandidate = uint8(winningcandidate).*image;
0088
0089

12
0001 function [lp] = lpcrop(image)
0002 %LPCROP Crops the license plate from the candidate that covers the true
0003 %license plate
0004 % LP = LPCROP(image)
0005 % where image is the image the license plate has to be extracted from
0006 % This method outputs a smaller image that only holds the license
0007 % plate
0008
0009 %detect the edges in the image
0010 edges = edge(image,’sobel’,’vertical’);
0011
0012 %initialize variables
0013 record = 0;
0014 u = 1;
0015 o = 1;
0016
0017
0018 %look for the area of 20 x 110 pixels with the largest edge density
0019 for i=1:1:646
0020 for j=1:1:443
0021 maskedimage = edges(j:j+20,i:i+110);
0022 totaalwittepixels = sum(maskedimage(:));
0023 if (totaalwittepixels > record)
0024 record = totaalwittepixels;
0025 mask2 = zeros(463,756);
0026 mask2(j:j+20,i+1:i+110) = 1;
0027 u = j;
0028 o = i;
0029 else
0030 end;
0031 end;
0032
0033 end;
0034
0035 %make the output image
0036 lp = image(u:u+20,o:o+110);
0037

13