International Journal of Advance Foundation and Research in Computer (IJAFRC)

Volume 2, Issue 12, December - 2015. ISSN 2348 4853, Impact Factor 1.317

Development Of Control Software For Mobile Robot Using

Image Processing and Artificial Intelligence
Mukul Pathak1,Shwetant Mohapatra2, Kshitij Kamalakar3,Prof. Uma Nagaraj
Dept. of Computer Engineering, MIT Academy Of Engineering, Savitribai Phule Pune University Pune, India
1iammap26@gmail.com, 2shwetant95@gmail.com 3kshitijkamalakar@gmail.com

ABSTRACT
In this project our main aim is to design and develop the control software for the
deployment and alignment of the flippers/wheels mechanism of a stair climbing robot.
The robot platform is a differential drive, with skid steering system. The system is mounted
on a rugged chassis. Vision sensors are mounted on the robot. These are cameras which will
provide motion images of the robots surroundings. The application software will apply
image processing and artificial intelligence techniques to detect stairs at Real-time and align
the flipper/wheel at an appropriate distance from the stair. Use of canny edge detection
method to detect the edges of the stairs, smoothen the image and removing noise from the
image. Neural networking will be used to detect stairs and faults. Machine learning
technology to overcome faults in stairs and act accordingly from the saved experiences. This
will be Linux based application which will have support of OpenCV API.
INDEX TERM :Pattern Recognition, shape detection, automation and Security.Artificial
Intelligence, Image Processing, Neural Network, Opencv API, Mobile Robot.

I. INTRODUCTION
Autonomous ground robots have traditionally been restricted to single floors of a building or outdoor
areas free of abrupt elevation changes such as stairs. Although autonomous traversal of stairways is an
active research area for some humanoid and ground robots, the focus within the vision and sensor
community has been on providing sensor feedback for control of the mechanical aspects of stair
traversal, and on stair detection as a trigger for the initiation of autonomous climbing, rather than
on stair traversability. The restriction to a single floor presents a significant limitation to real-world
applications such as mapping of multi-floor buildings and rescue scenarios. Our work seeks a
solution to this problem and is motivated by the rich potential of an autonomous ground robot
that can climb stairs while exploring a multi-floor building. A comprehensive indoor exploration
system could be capable of autonomously exploring an environment that contains stairways,
locating them and assessing their traversability, and then engaging a platform-specific climbing routine
in order to traverse any climbable stairways to explore other floors. The physical properties of a stairway
may limit the platforms that are capable of climbing it. For example, a robot may not be able to climb
some stairways due to step height, and a ground robot may be restricted to stairways with a low pitch
due to its weight distribution. Our proposed approach is an effort to integrate the existing work in
autonomous stair climbing with autonomous exploration: a system to detect and localize stairways in
the environment during the process of exploration. With a map of the environment and estimated
www.ijfarc.org
72 | 2015, IJAFRC All Rights Reserved

International Journal of Advance Foundation and Research in Computer (IJAFRC)

Volume 2, Issue 12, December - 2015. ISSN 2348 4853, Impact Factor 1.317
locations and parameters of the stairways, the robot could plan a path that traverses the stairs in
order to explore the frontier at other elevations that were previously inaccessible. For example, a
robot could finish mapping the ground floor of a building, return to a stairway that it had previously
discovered, and ascend to the second floor to continue exploring if that stairway is of dimensions
(i.e. step height, width, pitch) that are traversable by that particular platform. Our proposed system
directly addresses the needs of an exploratory platform for solving the problem defined above. We
seek to answer the question Is this traversable? when a stairway is discovered during exploration.
The questions When should I traverse that? and How do I traverse that? are left for the path planner and
stair climbing routines, respectively. The system is composed of a stairway detection module for
extracting stair edge points using single camera. The physical properties can then be used by a path
planner to determine the traversability of stairs in relation to the specific robotic platform.
In this paper, we present a strategy for ascending-stair detection,approach, and traversal for an
autonomous tracked vehicle. Specifically, we focus on the minimum-sensing scenario in which only a
monocular camera are available on the robot. Our algorithm is divided in four phases (P1) Far-approach:
the robot determine possible ascending-stair locations and starts to navigate towards one of them.
(P2) Near-approach: the robot verifies alignment using analysis of the leading stair edge. (P3) Stair
alignment: the robot aligns itself to the ascending stairs. (P4) Stair traversal: the robot steers along a safe
trajectory up ascending staircase
II. LITERATURE SURVEY
The research conducted so far in development of software for mobile robot using image processing
and artificial intelligence are discussed in this section. The set of challenges outlined above span
several domains of research and the majority of relevant work will be reviewed in this section. In this
section, basic concept are discussed for better understanding of the project. The literature survey
regarding which methodology should be applied was taken considering the following existing algorithms
and models.
Mike Fair and David P. Miller[1] presented a stair climbing robot built around the chassis of a
Quest Access wheelchair. They replaced the control electronics and several sensors including a laser
scanner and a set of Sharp range sensors have been added. The original wheelchair hardware was
capable of negotiating a staircase, but it required the human operator to detect the staircase, put
the chair into stair-climbing mode, align the chair with the staircase, and then manually control the
rate of ascent or decent. The sequencing of the chair tilt and easy downs (the hydraulic actuators
which gently allowed the chair to transition from level ground to inclines ) was done automatically.
E.Mihankhah, A.Kalantari [2] presented a Autonomous Stair- case Detection and Stair Climbing for a
Tracked Mobile Robot using Fuzzy Controller. Theoretical analysis and implementation of
autonomous staircase detection and stair climbing algorithms on a novel rescue mobile robot are
presented in this paper. The main goals are to find the staircase during navigation and to implement a
fast, safe and smooth autonomous stair climbing algorithm. Silver is used here as the experimental
platform. This tracked mobile robot is a teleoperative rescue mobile robot with great capabilities
in climbing obstacles in destructed areas. Its performance has been demonstrated in rescue robot
league of international RoboCup competitions. A fuzzy controller is applied to direct the robot
during stair climbing. Controller inputs are generated by processing the range data from two
LASER range finders which scan the environment one horizontally and the other vertically
Daniel M.Helmick Stergios and Michael C Henry [3]designed for Small,tracked mobile robots for
general urban mobility have been developed for the purpose of reconnaissance and/or search and
rescue missions in buildings and cities. Autonomous stair climbing is a significant capability required for
many of these missions. In their paper they presented the design and implementation of a new set of
73 | 2015, IJAFRC All Rights Reserved

www.ijfarc.org

International
national Journal of Advance Foundation and Research in Computer (IJAFRC)
Volume 2, Issue 12,
1 December - 2015. ISSN 2348 4853, Impact Factor 1.317
estimation and control algorithms that increase the speed and effectiveness of stair climbing. They have
developed a Kalman filter that fuses visual/laser data with inertial measurements and provides
attitude estimates of improved accuracy at a high rate,and(ii) a physics based controller that
minimizes the heading error and maximizes the velocity of the vehicle during stair climbing.
Experimental results using a tracked vehicle validate the improved
improved performance of this control and
estimation scheme over previous approaches
Sonda Ammar Bouhamed and Imene Khanfir Kallel, Dorra Sellami Masmoudi [4] proposed a paper on
staircase detection using ultrasonic sensor in (IJACSA) International Journal of Advanced Computer
Science and Applications, Vol. 4, No. 6 on 2013 . A new device is then proposed to enable them
to see the world with their ears. Considering not only system requirements but also technology
cost, they used, for the conception of our tool, ultrasonic sensors and one monocular camera to enable
user being aware of the presence and nature of potential encountered obstacles. In this paper, they
are involved in using only one ultrasonic sensor to detect staircases in electronic cane. The system
was evaluated on a set of ultrasonic signal where stair-cases
stair cases occur with different shapes.
sh
Using a
multiclass SVM approach, recognition rates of 82.4.
III. PROPOSED SYSTEM
PARAMETERS FOR MOBILE ROBOT
1) Raw image captured from the camera.
2) Default Focal length of the camera.
3) Standard value of stair riser.

Figure 1: Architectural Diagram

1.1 Stair Edge Detection:
In this section we shall present the state of the art for edge stair detection algorithms used by the
corresponding robots to climb stairs. In our project we use a robot which has a camera mounted
on it. A single molecular camera
mera is used to capture the surrounding . A video is a sequence of image and
image is a binary representation of visual information. Image can be represented in the form of a matrix ,
where each element represents some value of pixel. Color digital images are made of pixels, and pixels
are made of combinations of primary colors. Color image consists 3-channel
channel of red, blue and green.
Every channel is represented in form of matrix. Element in each channel is used to store the intensity of
red
ed , blue or green .Combination of these intensity provide the colour for every pixel. A gray scale
image has only 1 channel which is used to store the colour intensity between black or white color.
Using image processing we are analyzing
analyzi and manipulating the image to get the desired output. The RGB
74 | 2015, IJAFRC All Rights Reserved

www.ijfarc.org

International Journal of Advance Foundation and Research in Computer (IJAFRC)

Volume 2, Issue 12, December - 2015. ISSN 2348 4853, Impact Factor 1.317
image we get from camera , is to be first converted into gray scale image . The gray scale image is
then filtered so as to remove the noise. A filtering method is used to remove the extra pixels and to fill the
holes in a continuous curve. Then the filtered image is used to detect edges. A edge detection technique
Open CV is used to detect edges. The main purpose of the edge detection is to simplify the image
in order to minimize the data to be processed. There are three methods which can be used to detect
edges. First is Soble second is Laplacian and third is canny . In this project we are using canny edge .
Laplacian is more prone to noise and Sobel edge detection function requires the destination image to
have a larger depth than the source one. So we must make edge frame the 16-bit intermediate in this
case to store the result of the Sobel operation. A 16-bit signed image cannot be shown with the cv
Show Image function properly. So we have to convert it back to 8-bit unsigned format. To save memory
space and code lines, Canny edge detection is used. Using contour detection, the co-ordinates of boundary
are stored in a variable.
Processing on blurred image -If the image is blurred , then it can be rectified using the un-sharp masking
technique .In this technique a slightly blurred image is compared with the original image to detect the
presence of edges and the contrast is then slightly increase along the edges .
1.2 Stair Detection Using Neural Network:
In our method, we are using neural network to make a simple decision to check whether there is stair
or not. Back propagation is an algorithm in neural network that can be used to train a neural
network[5]. Training a neural network is the process of finding a set of weights and bias values so that,
for a given set of inputs, the outputs produced by the neural network are very close to some known
target values. Gradients are values that reflect the difference between a neural networks computed
output values and the desired target values. As it turns out, gradients use the calculus derivative of the
associated activation function. The gradients of the output nodes must be computed before the
gradients of the hidden layer nodes, or in other words, in the opposite direction of the feed forward
mechanism. Supervised learning which that each output unit is told what its desired response to
input signals is used in our system.
1.3 Distance calculation and alignment:
In order to determine the distance we are going to utilize triangle similarity [6]. In this we have stairs
with a known height of one step H. We then place this object some distance D from our camera. We take a
picture of our object using our camera and then measure the apparent height in pixels P. Lets say the
focal length is F. D = (H x F) / P .Using contour detection we can calculate the x co-ordinate of two end
edges of a same horizontal boundary. If the values are different the robot is not aligned otherwise it is
aligned.

IV. PROJECT IDEA

In this project we have made a flow chart which will help us analyze and proceed according to the
processes or steps specified in the flow chart.

75 | 2015, IJAFRC All Rights Reserved

www.ijfarc.org

International
national Journal of Advance Foundation and Research in Computer (IJAFRC)
Volume 2, Issue 12,
1 December - 2015. ISSN 2348 4853, Impact Factor 1.317

Figure 2: Flow Chart

The camera mounted on the ROV will act as the input for the program. When the robot starts moving or
in operational the program runs as it starts capturing stream of images that is the video from the camera
mounted over the robot. The images captured from this molecular camera then applies edge detection
methods to outline the of stairs. Image filtering is applied and stairs are detected using contour detection
methods. Now as proposed in previous section, we will detect an approximate distance
distanc of the stair from
the robot after which robot will be able to climb up the stairs autonomously.
This project will be executed on the ROV DAKSH from the robotics group of the DEFENCE RESEARCH
AND DEVELOPMENT ORGANISATION'S R&DE(E) Pune
V. RELEVANT MATHEMATICS ASSOCIATED WITH THE PROJECT.
System Description:
Input: Image,Actual height of stair , height of stair in pixel , Focal length of camera.
Output: Stair recognition , movement of wheels ,distance between camera and stair.
Identify data structures,
ures, classes, divide and conquer strategies to exploit distributed/parallel/concurrent
processing, constraints.
Related Function:
$f_1$ = database creation for storing the images of types of stairs.
$f_2$ = Camera capture of stairs while robot is mobile
$f_3$ = image processing techniques on the frame captured
$f_4$ = mapping the processed image with the database to detect the type of stair it climbing on
Mathematical formula
Distance = (actual height of one step * Focal length)/pixel height of one step.
Success Conditions: stair detected accurately,
accu
distance between the robot and camera
\item
item Failure Conditions: image inaccuracy , defect in camera.

76 | 2015, IJAFRC All Rights Reserved

www.ijfarc.org

International Journal of Advance Foundation and Research in Computer (IJAFRC)

Volume 2, Issue 12, December - 2015. ISSN 2348 4853, Impact Factor 1.317
VI. CONCLUSION
So far we have explored a big idea to automate a part of the manually controlled ROV DAKSH from the
Defense Research and Development Organizations Research and Development Establishment (E), Pune.
Our work will help the robots to detect and climb stairs on their own. This will help in further more
exploiting the areas of automation and help the country in different needs. This project will help us in
understanding the concepts of Artificial intelligence, Image processing techniques and Neural networks.
VII. FUTURE WORK
Our project opens a lot of doors for further improvising the idea of automation amongst robots. Currently
working on ROV Daksh, we will be working on all other manually controlled robots of the DRDO. This will
help in all other robots of the organization to be somehow automated. Use of Neural network by us can
encourage in making the whole robot automated and hence save lives in the war-zone, terrorist attacks,
diffusing the bombs and other natural or man-made calamities.
VIII. APPLICATIONS AREA FOR PROJECT
Applications of the project are :
[1] Stair climbing autonomously
[2] In war zones
[3] In operations at the urban areas where buildings are high and requires robot to climb stairs
autonomously.
IX. AACKNOWLEDGMENTS
Our sincere thanks to all those who are guiding us in this
project. Our sincere thanks to our guide and mentor :Prof.Uma Nagaraj, HOD, COMP, MIT-AOE, Pune
Our external Guides :
Mr. Alok Mukherjee, Scientist, RDE(E), Pune
Mr. Altaf Mirza Baig, Scientist, RDE(E), Pune
Mr. Debjyoti Das, hi-tech robotic systemz ltd, Pune

X. REFERENCES
[1]

S. Owald, J.-S. Gutmann, A. Hornung, and M. Bennewitz, From 3D point clouds to climbing
stairs: A comparison of plane segmentation approaches for humanoids .In Proc. of the
IEEE-RAS Int. Conf. On Humanoid Robots (Humanoids), 2011.

[2]

Delmerico, J.A., Baran, D., David, P., Ryde, J., Corso, J.J. Ascending stairway modeling from dense
depth imagery for traversability analysis. In: International Conference on Robotics and
Automation (ICRA). pp. 2283 2290 (2013).

A. M. Johnson, M. T. Hale, G. C. Haynes, and D. E. Koditschek, Autonomous legged hill and

stairwell ascent ,In IEEE International Workshop on Safety, Security, Rescue Robotics
(SSRR),pp. 134 142.November 2011
www.ijfarc.org
77 | 2015, IJAFRC All Rights Reserved

[3]

International Journal of Advance Foundation and Research in Computer (IJAFRC)

Volume 2, Issue 12, December - 2015. ISSN 2348 4853, Impact Factor 1.317
[4]

J. Delmerico, J. Corso, D. Baran, P. David, and J. Ryde, Toward autonomous multi-floor

exploration: Ascending stairway localization and modeling , U.S. Army Research Laboratory,
Tech. Rep. ARL-TR, 2012

[5]

R. B. Rusu and S. Cousins, 3D is here: Point Cloud Library (PCL), In IEEE International
Conference on Robotics and Automation (ICRA), Shanghai, China, May 9-13 2011

[6]

Tung-Sing Leung , Medioni G. , Real-time staircase detection from a wearable stereo system
.In: Pattern Recognition (ICPR), 2012 21st International Conference on Nov. 2012,pp 3770 3773[7]

[7]

Hernandez, D.C., Jo, K.H. Stairway tracking based on automatic target selection
directional filters . In: Frontiers of Computer Vision (FCV). pp. 16 (2011)

[8]

Cloix, S., Weiss, V., Bologna, G., Pun, T., Hasler, D. Obstacle and planar object detection
using sparse 3D information for a smart walker. In: 9th International Conference on Computer
Vision Theory and Applications. vol. 2, pp. 305 312. Lisbon, Portugal (Jan 2014)

[9]

Tseng, C.K., Li, I., Chien, Y.H., Chen, M.C., Wang,W.Y., Autonomous stair detection and climbing
systems for a tracked robot . In: System Science and Engineering (ICSSE), 2013 International
Conference on. pp. 201 204. IEEE (2013).

78 | 2015, IJAFRC All Rights Reserved

using

www.ijfarc.org