2010 Second International Conference on Machine Learning and Computing

Real-Time Robotic Hand Control using Hand Gestures

Jagdish Lal Raheja, Radhey Shyam, Umesh Kumar, P Bhanu Prasad
Digital Systems Group
Central Electronics Engineering Research Institute (CEERI)/Council of Scientific & Industrial Research (CSIR)
Pilani 333031, Rajasthan
India
jagdish@ceeri.ernet.in, er.radheyshyam@gmail.com, umesh.kr1983@gmail.com, bhanu@ceeri.ernet.in
Glove based technique is well-known means of recognizing
hand gestures. It utilizes sensor-detached mechanical glove
devices that directly measure hand and/or arm joint angles
and spatial position. Although glove-based gestural
interfaces give more precision, it limits freedom as it
requires users to wear cumbersome patch of devices.
Jae-Ho Shin [5] used entropy analysis to extract
hand region in complex background for hand gesture
recognition system. Robot controlling is done by Fusion of
Hand Positioning and Arm Gestures [6] using data glove.
Although it gives more precision, it limits freedom due to
necessity of wearing gloves. For capturing hand gestures
correctly, proper light and camera angle are required. A
technique to manage light source and view angle in efficient
way for capturing hand gesture is given in [7]. A review has
been written on the subsets of hand posture and gesture
recognition in [8], [9]. The problem of visual hand
recognition and tracking is quite challenging. Many early
approaches used position markers or colored bands to make
the problem of hand recognition easier, but due to their
inconvenience, they cannot be considered as a natural
interface for the robot control [10].
We have proposed a fast as well as automatic hand
gesture detection and recognition system. 3D Matlab
Kinematic model of a PUMA [11] Robot, located in the
robotics lab of Walla, Walla University, is used for
executing action by hand gesture. This approach of gesture
identification
On the basis of recognized hand gesture, some
specific command for moving different arms in specified
direction has been designed. But this system does not stop
here. It can be extended to any robotic system with a number
of specific commands suitable to that system.

Abstract this paper presents a new approach for controlling

robotic hand or an individual robot by merely showing hand
gestures in front of a camera. With the help of this technique
one can pose a hand gesture in the vision range of a robot and
corresponding to this notation, desired action is performed by
the robotic system. Simple video camera is used for computer
vision, which helps in monitoring gesture presentation. This
approach consists of four modules: (a) A real time hand
gesture formation monitor and gesture capture, (b) feature
extraction, (c) Pattern matching for gesture recognition, (d)
Command determination corresponding to shown gesture and
performing action by robotic system. Real-time hand tracking
technique is used for object detection in the range of vision. If a
hand gesture is shown for one second, the camera captures the
gesture. Object of interest is extracted from the background
and the portion of hand, representing the gesture, is cropped
out using the statistical property of hand. Extracted hand
gesture is matched with the stored database of hand gestures
using pattern matching. Corresponding to the matched
gesture, action is performed by the robot.
Keywords- Computer Vision, Hand gesture Recognition,
Pattern Recognition, Robot arm control.

I.

INTRODUCTION

Interpretation of human gestures by a computer is used for

human-machine interaction in the area of computer vision
[1]. The main purpose of gesture recognition research is to
identify a particular human gesture and convey information
to the user pertaining to individual gesture. From the corpus
of gestures, specific gesture of interest can be identified, and
on the basis of that, specific command for execution of
action can be given to robotic system. Overall aim is to
make the computer to understand human body language [2],
thereby bridging the gap between machine and human.
Hand gesture recognition can be used to enhance human
computer interaction without depending on traditional input
devices such as keyboard and mouse.
Hand gestures are extensively used for telerobotic
control and applications [3]. Robotic systems can be
controlled naturally and intuitively with such telerobotic
communication. [4] A prominent benefit of such a system is
that it presents a natural way to send geometrical
information to the robot such as: left, right, etc. Robotic
hand can be controlled remotely by hand gestures. Research
is being carried out in this area for a long time. Several
approaches have been developed for sensing hand
movements and corresponding by controlling robotic hand.
978-0-7695-3977-5/10 $26.00 2010 IEEE
DOI 10.1109/ICMLC.2010.12

12

x
Take one frame as input from video camera

Determine control instruction, corresponding to

matched gesture, and give that instruction to
specified robotic system.

Figure 1 shows the flow diagram of whole system, i.e.

performing hand gesture identification and robot control. On
the basis of gesture, identified by pattern matching, control
instruction is determined from the stored instructions set.
The selected instruction set, corresponding to recognized
hand gesture is given to robot for carrying out the control
action.

Does frame
contain object of
interest (i.e.
contains some
)?

2.1 Gesture finding from online Camera: From video

stream one frame is captured in each 1/3 second. Target is to
identify the frame that contains hand gesture shown by
human. For this we are searching the frame in which there is
no movement. Required frame is identified by comparing
three continuous captured frames. Motion parameter is
determined for each frame by counting total pixels of
mismatch. If motion parameter is less than the specified
threshold value, it is considered as a frame having less
movement of object i.e. the frame contains some hand
gesture, user wants to show.
Analysis of frames to find the frame of interest is done by
converting captured frame into a binary frame. Differences
between newly captured frame and two previously captured
frames are determined and they are added together to find
motion parameter. Differences between values of
corresponding pixels are counted with both frames and
added to find motion parameter. Since binary image has
values of one or zero, XOR function can give locations
where mismatches occur.
If frame1, frame2, and frame3 are three matrixes containing
three frames captured in three continuous time slots
respectively then:

Crop out object of interest (i.e. extract

hand part showing gesture from
captured frame

Resize extracted hand gesture to fixed

dimension and match it with database of
sample gestures

Find Command corresponding to

matched gesture

fr1= frame1 XOR frame3

Execute instruction shown by hand gesture

fr2 = frame2 XOR frame3

mismatch_matrix = fr1 OR fr2
Figure 1: Flow Diagram of Robot Control using
Hand Gesture
II.

sum =
motion parameter =

METHODOLOGY

Proposed methodology is able to use live video camera for

gesture identification. It sniffs frames of live video stream in
some time interval. In our case frame capture rate for
gesture search is 3 frames per second. Proposed technique to
control robotic system using hand gesture display is divided
into four subparts:
x Capture frame containing some gesture
presentation.
x Extract hand gesture area from captured frame.
x Determine gesture by pattern matching using PCA
algorithm.

Here r and c are the number of rows and columns in image

frames. Threshold value is set as 0.01. i.e. if total pixels of
mismatch is less than 1% of total pixels in a frame, then it is
considered as frame of interest. Required frame is forwarded
for further processing and this module again start searching
frame of interest.
2.2 Extraction of Hand Gesture Area from Frame of
Interest: The frame with a gesture contains extra part along
with required part of hand i.e. background objects, blank
spaces etc. For better result in pattern matching, unused part
must be removed. Therefore hand gesture is cropped out

13

from obtained frame. Cropping of hand gesture from the

obtained frame contains three steps:

Figure 2: Boundary Selection of Hand Cropping

First step is to convert selected frame into black-and-white

image using global thresholding.
Second step is to extract object of interest from the frame. In
our case, object of interest is the part of human hand
showing gesture. For this, extra part other than the hand is
cropped out so that pattern matching can give more accurate
results. For cropping extra parts row and column number is
determined, from where object of interest appears. This is
done by searching from each side of binary image and
moving forward until white pixels encountered are more
than the offset value. Experimental results shows that offset
value set to 1% of total width gives better result for noise
compensation. If size of selected image is mXn then:
Hor_offset= m/100
Ver_offset=n/100

Figure 3: Histogram Showing White pixels in each column, with cropping

point for hand gesture.

Statistical analysis of hand shape shows that either we pose

palm or fist, width is lowest at wrist and highest at the
middle of palm. Therefore extra hand part can be cropped
out from wrist by determining location where minimum
width of vertical histogram is found. Figure 3.c and 3.d
show global maxima and cropping points for hand gestures
in figure 3.a and 3.b respectively.
Cropping point is calculated as:

Min_col= minimum value of column number where total

number of white pixels are more than Hor_offset.

Global Maxima = column number where height of

histogram is highest (i.e. column number for global maxima
as shown in figure 3).

Max_col= maximum value of column number where total

number of white pixels are more than Hor_offset.

Cropping point = column number where height of

histogram is lowest in such a way that cropping point is in
between first column and column of Global Maxima

Min_row= minimum value of row number where total

number of white pixels are more than Ver_offset.
Max_row= maximum value of row number where total
number of white pixels are more than Ver_offset.

If gesture is shown from opposite side (i.e. from other hand)

then cropping point is searched between column of Global
Maxima and last column. Direction of the hand is detected
using continuity analysis of object during hand gesture area
determination. Continuity analysis shows that whether the
object continues from column of Global maxima to first
column or to last column. i.e. whether extra hand is left side
of palm or right side of palm.

Third step is to remove parts of hand not used in gesture

presentation i.e. removal of wrist, arm etc. Because theses
extra parts are of variable length in image frame, pattern
matching with gesture database gives unwanted results, due
to limitations of gesture database. Therefore, parts of hand
before the wrist need to be cropped out.

2.3 Resizing and Pattern Matching: After cropping,

cropped gesture is scaled to fit in frame of 60X80 pixels so
that it can be matched with databases without distorting
original shape of hand gesture. Figure 4.b shows cropped
hand gesture part from figure 4.a. And figure 4.c shows
hand gesture part after scaling and fitting with 60X80 pixels

14

were used to control robotic hand. Figure 6 shows all known

gestures for which proposed system works. It gives 90%
accuracy in proper light arrangement and its robustness
decreases with poor light arrangement.

without changing the aspect ratio of cropped input gesture

of figure 4.a Figure 4.d represent final cropped and resized
gesture of desired dimension.
Principal Component Analysis (PCA) method is used for
pattern matching. There are two reasons behind using PCA:
(i) PCA method is suitable for pattern matching as human
hand is used for gesture expression and components present
in hand (e.g. fingers, palm, fist, etc.) are large enough, when
compared to noise. (ii) PCA method is very fast compared
to neural network method which requires training database
and more time along with high computation power.

Figure 4: Scaling cropped hand gesture to fit into desired dimension

Figure 5: Movement of PUMA 762 corresponding to some specific gesture.

2.4 Control Instruction Generation for Robot: Different

functions corresponding to each meaningful hand gesture
are written and stored in database for controlling robotic
hand. For implementation of robotic hand control, PUMA
robotic model has been used. 10 different hand gestures are
used for designing 10 different moves of robotic hand.
Movement command is designed and written as function in
robot specific language corresponding to each meaningful
hand gesture. For controlling PUMA model, a set of
commands is written for movement of each arm in precise
angle and direction. Whenever a gesture is matched with
meaningful gesture from database, instruction set
corresponding to that gesture is identified and passed to
robot for execution of those commands. In this way robotic
system can be controlled by hand gesture using live camera
as eyes. Figure 5 shows movements of robotic hand
corresponding to specific hand gestures.
III.

IV.

CONCLUSION AND FUTURE WORK

In todays digitized world, processing speeds have increased

dramatically, with computers being advanced to the levels
where they can assist humans in complex tasks. Yet, input
technologies seem to cause a major bottleneck in
performing some of the tasks, under-utilizing the available
resources and restricting the expressiveness of application
use. Hand Gesture recognition comes to rescue here.
Robotic control is dependent on accurate hand gesture
detection and hand gesture detection directly depends on
lighting quality. Beside 10 different hand gestures, one
blank gesture (no object of interest) is added to give no
operation command. When system matches with any blank
gesture, then it performs no action. Besides robustness of
the system, proposed method for controlling robot using
hand gesture is very fast. This methodology can be extended
for more complex robots in the fields of computer vision
and robotics. Future work of will be focused on more
complicated gesture and precise movement of robot.

RESULTS

Proposed technique to control robot is tested in lab

environment using PUMA 762 robotic model. Hand gesture
is shown in front of web cam which is fixed at a place.
Virtual robot is used for controlling. 11 different gestures

15

Region Extraction and Gesture Recognition using entropy analysis,

IJCSNS International Journal of Computer Science and Network
Security, VOL.6 No.2A, 2006 pp. 216-222.
[3] Moezzi and R. Jain, "ROBOGEST: Telepresence Using Hand
Gestures". Technical report VCL-94-104, University of California,
San Diego, 1994.
[4] D. Kortenkamp, E. Huber, and R. P. Bonasso, "Recognizing and
Interpreting Gestures on a Mobile Robot", AAAI96, 1996.
[5] Didier Coquin, Eric Benoit, Hideyuki Sawada, and Bogdan Ionescu,
Gestures Recognition Based on the Fusion of Hand Positioning and
Arm Gestures, Journal of Robotics and Mechatronics Vol.18 No.6,
2006. pp. 751-759.
[6] L. Brthes, P. Menezes, F. Lerasle, and J. Hayet. Face tracking and
hand gesture recognition for human robot interaction. In International
Conference on Robotics and Automation, pages 19011906, 2004.
New Orleans, Louisiana.
[7] Jagdish Raheja, Radhey Shyam and Umesh Kumar, Hand Gesture
Capture and Recognition Technique for Real-time Video Stream, In
The 13th IASTED International Conference on Artificial Intelligence
and Soft Computing (ASC 2009), September 7 - 9, 2009 Palma de
Mallorca, Spain.
[8] Ying Wu, Thomas S Huang, Vision based Gesture. Recognition: A
Review, Lecture Notes In Computer Science; Vol. 1739,
Proceedings of the International Gesture Workshop on Gesture-Based
Communication in Human-Computer Interaction, 1999.
[9] K G Derpains, A review of Vision-based Hand Gestures, Internal
Report, Department of Computer Science. York University, February
2004.
[10] Seyed Eghbal Ghobadi, Omar Edmond Loepprich, Farid Ahmadov
Jens Bernshausen, Real Time Hand Based Robot Control Using
Multimodal Images, IAENG international journal of computer
science IJCS_35_4_08.
[11] Walla,
Walla
University,
Robotic
Lab
PUMA762,
http://www.mathworks.com/ matlabcentral/fileexchange/14932.

Figure 6: Set of specific gesture used for Robot Control

ACKNOWLEDGMENT
This research is being carried out under a CSIR project
activity Image processing based Smart Camera system for
Human Gesture Identification under Supra Institutional
project Technology Development for Smart Systems at
Central Electronics Engineering Research Institute
(CEERI), Pilani, India. Authors would like to thank
Director, CEERI for his active encouragement and support.
REFERENCES
[1]

[2]

Reza Hassanpour, Stephan Wong, Asadollah Shahbahrami,

VisionBased Hand Gesture Recognition for Human Computer
Interaction: A Review, IADIS International Conference on Interfaces
and Human computer Interaction. 25 - 27 July 2008 Amsterdam,
Netherlands .
Jae-Ho Shin, Jong-Shill Lee, Se-Kee Kil, Dong-Fan Shen, Je-Goon
Ryu, Eung-Hyuk Lee, Hong-Ki Min, Seung-Hong Hong, Hand

16