Professional Documents
Culture Documents
VEHICLE
(FINAL REPORT)
PREPARED AT : INNOVATION CELL IIT BOMBAY
SUBMITTED BY: AMARTANSH DUBEY
MENTOR: EBRAHIM ATTARWALA
ACKNOWLEDGEMENTS
I take this opportunity to express my gratitude to the people who have
been instrumental in the successful completion of this project. We are
thankful to Innovation Cell, IIT Bombay, for providing me with this
opportunity to learn, explore and implement our skills. I thank my
mentor EBRAHIM ATTARWALA and all other mentors for their constant
guidance and support throughout the course of the project.
PROBLEM STATEMENT:
To develop image processing algorithms for autonomous vehicle which
basically involves:
1) Lane Detection.
2) Vehicle Detection and Recognition.
3) Character, Text and Road Sign Detection and Recognition.
4) Controlling motion of vehicle using above three algorithm in Real
time.
INDEX
1.
2.
3.
4.
5.
6.
7.
8.
Part A:
Briefing the concepts of image processing used in the codes for detecting lanes,
removing noise, etc.
SUBHEADINGS:-
HORIZON REMOVAL.
OTHER SMALL CONCEPTS.
Part B:
Algorithms and codes which works well and detected lanes successfully but were
not adaptive in every conditions.
SUBHEADINGS:
ALGORITHM-1 & FLOWCHART AND PROBLEM IN IT
10
Final code
SUBHEADINGS:
Part C:
Final algorithm.
Final flowchart.
RESULT.
11
SUBHEADINGS:
WHY CASCADED CLASSIFIER NEEDED:
SAMPLING:
SETTING REGION OF INTEREST OR CROPPING
Training and Classification:
OUTPUT OF CODE
12
13
IMPORTANT LINKS CONTAINING
CODES, SAMPLE VIDEOS, OUTPUT VIDEOS
- Analyzing and manipulating the image which includes data compression and image
enhancement and spotting patterns that are not to human eyes like satellite
photographs.
- Output is the last stage in which result can be altered image or report that is based on
image analysis.
OPENCV:
OpenCV (Open Source Computer Vision) is a library of programming
functions mainly aimed at real-time computer vision, developed by Intel
Russia research center in Nizhny Novgorod, and now supported by Willow
Garage and Itseez.[1] It is free for use under the open source BSD license.
The library is cross-platform. It focuses mainly on real-time image
processing. If the library finds Intel's Integrated Performance Primitives on
the system, it will use these proprietary optimized routines to accelerate
itself.
OpenCV is written in C++ and its primary interface is in C++, but it still
retains a less comprehensive though extensive older C interface. There are
now full interfaces in Python, Java and MATLAB/OCTAVE (as of version
2.5).
Third is that opencv is dynamic library whereas FPGA are only hardware
with no memory so we cant use OPENCV for image processing, we have to
choose other tools like OPENCL or VIVADO, and these tools are not as
efficient as OPENCV.
PART A:
FOLLOWINGS ARE CONCEPTS AND TOOLS I USED IN LANE
DETECTION:
1)CANNY/SOBEL EGDE DETECTION:
2)HOUGH TRANSFORM.(For shape detection)
3)BIRD EYE VIEW / PERSPECTIVE VISION.
4)GAUSSIAN BLURR.
5)HORIZON REMOVAL.
6)There are many other small concepts which I have dealed in the
explanation of codes.
DETALIED EXPLANATIONS::
1)CANNY/SOBEL EGDE DETECTION:
1. The Canny Edge detector was developed by John F. Canny in 1986. Also known
to many as the optimal detector, Canny algorithm aims to satisfy three main
criteria:
o
Good localization: The distance between edge pixels detected and real
edge pixels have to be minimized.
IMPLEMENTATION:
1) Filter out any noise. The Gaussian filter is used for this purpose. An example of a
Gaussiankernel of size = 5 that might be used is shown below:
2) Find the intensity gradient of the image: A) Apply a pair of convolution masks
(in and directions) and Find the gradient strength and direction with:
The direction is rounded to one of four possible angles (namely 0, 45, 90 or 135)
3) Non-maximum suppression is applied. This removes pixels that are not
considered to be part of an edge. Hence, only thin lines (candidate edges) will
remain.
4) Hysteresis: The final step. Canny does use two thresholds (upper and lower):
If a pixel gradient is higher than the upper threshold, the pixel is accepted as an
edge
If a pixel gradient value is below the lower threshold, then it is rejected.
If the pixel gradient is between the two thresholds, then it will be accepted only if
it is connected to a pixel that is above the upper threshold.
Canny recommended a upper:lower ratio between 2:1 and 3:1.
Example :::
In Hough standard Transform, we will express lines in the Polar system. Hence, a line equation
can be written as:
OR
each pair
represents each line that passes by
In general for each point
we can define the family of lines that goes
through that point as:
.
If for a given
we plot the family of lines that goes through it, we get a
sinusoid. For instance, for x=8, y=6, we get the following plot (in a plane ):
3) We can do the same operation above for all the points in an image. If the curves
of two different points intersect in the plane - , that means that both points
belong to a same line. For instance, following with the example above and
drawing the plot for two more points:
,
and
,
, we get:
(
) or the line in which
,
and
lay. It means that in general, a line
can be detected by finding the number of intersections between curves.The more curves
intersecting means that the line represented by that intersection have more points. In general,
we can define a threshold of the minimum number of intersections needed to detect a line.
4) HORIZON REMOVAL:
it means to remove the horizon (sky and other unwanted area above the road),
this will help in removing unwanted noises and disturbances. It is done by
accessing rows and columns and put pixel =0 to remove unwanted region or it can
be done by using function setROI.
5)GAUSSIAN BLURR:
It is used to remove noise from the input image.
When Gaussian blurr function is applied on matrix of pixel of input image it gives
output image using the kernel:
Function uses above matrix to perform averaging of pixels of the input image.
PART B
Here I have mentioned the algorithms and codes which works well and
detected lanes successfully but were not adaptive in every conditions and
depends on certain factors like type of roads, lanes and surroundings. I
tried 3 such algorithms and finally reached to the final algorithm.
ALGORITHM-1 IT IS
NOT ADAPTIVE FOR
EVERY TYPE OF
LANES
Start
Main function
DISPLAY THE
PROCESSED IMAGE
IMPORTANT NOTE: code, sample videos, output videos are uploaded on google drive:
https://drive.google.com/folderview?id=0BxV8Z1s8nFXWcDRuOXN1WEFRV3c&usp=sharing
THIS WILL BE OUTPUT WHEN THIS ALGORITHM IS EXECUTED:
Advantage of removing horizon can be seen as there are no lines above the line
which differentiate road and sky.
PROBLEM WITH THE ALGORITHM 1:For controlling a self driving car and
maintaining fix distance from both the lanes, I needed the equation of the
detected lines, which is not possible because more than one lines are detecting
one lane, I needed one hough line for each lane so that I can find its equation and
distance of car from the lane.
ALGORITHM 2:
IMPORTANT NOTE:
code, sample videos, output videos are uploaded on google drive:
https://drive.google.com/folderview?id=0BxV8Z1s8nFXWcDRuOXN1WEFRV3c&usp=sharing
this do not allow hough function to print lines which are not in this
theta range, so curves could not be detected. Only straight lanes are
detected successfully.
ALGORITHM 3:
In this code I used distance of hough lines as tool to filter out other hough lines
and obtained one hough line for each lane, this avoids weighted mean and errors
mentioned in algorithm 2.
Here I find equation of the all the hough lines as :
column=(-sin(theta)/cos(theta)*350)+ rho/cos(theta);
and divided the hough lines in two categories, one which are
on left half of the image, and one which are on right
side,then filter out only those lines which are nearest to the center of camera
(on left and right of center respectively), because in between the roads there is
very less probability of having lane like thick straight lines, and small lines which
could be detected as lines are removed by Gaussian blurr or by setting parameters
of hough probabilistic function (increase min gap between points which should be
detected as lines ). Also k-mean clustering can be used to remove other lines .Well
road signs which may cause problem are removed using Haar Classifier.
PROBLEM WITH THE ALGORITHM 3:
This code solves problem of weighted mean but still curved could not be detected
because thresholding which is applied on theta is still there, moreover since lines
are too much slanted and it is not possible to calculate pixel-real distance it is not
possible to localizing car in between lanes.
IMPORTANT NOTE:code, sample videos, output videos are uploaded on google drive:
https://drive.google.com/folderview?id=0BxV8Z1s8nFXWcDRuOXN1WEFRV3c&usp=sharing
PART C
Final algorithm:
To get accurate distance, and relation between real world distance and pixels I
applied BIRD EYE VIEW(PERSPECTIVE VISION), I already mentioned about BIRD
EYE VIEW above. This gives me relation between pixel and real world distance,
and now car can be localized between the lanes.
FINAL FLOWCHART:
IMPORTANT NOTE: code, sample videos, output videos are uploaded on google drive:
https://drive.google.com/folderview?id=0BxV8Z1s8nFXWcDRuOXN1WEFRV3c&usp=sharing
FINAL flowchart
Start
Main function
DISPLAY THE
PROCESSED IMAGE
When code is executed then following results are displayed on four windows:
RESULT
2) SECOND WINDOW: It shows image after applying BIRD EYE VIEW, CANNY
EGDE and GAUSSIAN BLURR.
3)THIRD WINDOW: It shows image on which all the detected hough lines are
shown without filtering.
4)FOURTH WINDOW: It shows final image with only hough lines on lanes.
5) FIFTH WINDOW(shown AT LAST): it is showing the distance of centre of
camera from both of the lanes, and sending data for localizing car to processor
using UART.
IMPORTANT NOTE: code, sample videos, output videos are uploaded on google drive:
https://drive.google.com/folderview?id=0BxV8Z1s8nFXWcDRuOXN1WEFRV3c&usp=sharing
Classifiers can be used for face detection, character and text recognition and many more.
Classifier works on feature extraction . It involves following steps:
1) SAMPLING:
Sampling means to collect sample images of the object which is to be detected. This is very
important step, and for good results sampling should be done accurately. Generally for good
face detection program more than 1000 samples are to be taken. Suppose I want to detect a
traffic sign, for that I have to gather sample images of the sign from all possible angles and
brightness conditions. More are the samples gathered more is the accuracy. In order to train our
own classifier we need samples, which means we need a lot of images that show the object we
want to detect (positive sample) and even more images without the object (negative sample).
POSITIVE IMAGES
It means images of object to be detected, take photos of the object you want to detect, look for
them on the internet, extract them from a video or take some Polaroid pictures generate
positive samples for OpenCV to work with. It's also important that they should differ in lighting
and background.
NEGATIVE IMAGES
Now negative images are needed, the ones that don't show a object to be detected. In the best
case, if one wants to train a highly accurate classifier, he should have a lot of negative images
that look exactly like the positive ones, except that they don't contain the object we want to
recognize. As I want to detect stop signs on walls, the negative images would ideally be a lot of
pictures of walls. Maybe even with other signs. Keep an eye on the ratios of the cropped images,
they shouldn't differ that much. The best results come from positive images that look exactly like
the ones you'd want to detect the object in, except that they are cropped so only the object is
visible.
OUTPUT OF CODE:
Here is OUTPUT OF CODE was run:::
TRAFFIC SIGN, FACE AND VEHICLE RECOGNITION:
IMPORTANT NOTE: code, sample videos, output videos are uploaded on google
drive:https://drive.google.com/folderview?id=0BxV8Z1s8nFXWcDRuOXN1WEFRV
3c&usp=sharing
IMPORTANT NOTE:
code, sample videos, output videos are uploaded on google drive:
https://drive.google.com/folderview?id=0BxV8Z1s8nFXWcDRuOXN1W
EFRV3c&usp=sharing