ICVES 2009

A Smart Mobility Solution for Physically Challenged

Rajat Garg, N.Shriram, Vikrant Gupta, and Vineet Agrawal
Department of Electronics and Communication Engineering
VIT- University, Vellore- 600014, India
{rajatgarg2006, nshriram2006, vikrantgupta2006, vineetagrawal2006} @ vit.ac.in

Abstract— This paper presents a hand gesture and a vision
driven wheelchair system for physically handicapped people. The
system is contrived to have a utilitarian design that caters to
most of the disabilities rather than just one. In order to minimize
system-human interaction, we have endeavored to present two
distinct but closely related concepts viz. the hand gesture and
vision based control system in conjunction for the wheelchair
control. The wheelchair system is designed such that it provides
for a natural and an intuitive user interface which can
comprehend and react to in compliance with the users’
instinctive volition which eases the mobility of the physically
challenged people. All image processing was performed using NI
Vision Assistant. NI LabVIEW was used to train and implement
Neural Networks for Hand Gesture Classification and also to
enable motion control. A prototype was developed on which all
our experiments were successfully carried out. The system has
been tested for users with varied hand shapes and proved to be
extremely reliable.
Keywords— Thresholding, Morphological operations, Image
Moments, Artificial Neural Network (ANN).
• An eclectic gesture set is used to offer higher accuracy
with minimized confusion. By contrast, conventional
wheelchairs comprise of buttons, joysticks, paddles and
levers to perform various control maneuvers. The
subject ought to know the whereabouts of the controls
and must be sufficiently agile to reach and operate
them, which is difficult in the case of physically
challenged and infirm people.
• Very robust to lighting variations.
• Uses Image Moments instead of template matching for
classification which has a significant impact on the total
processing time, due to simplified algorithm. The
acquired images are invariant to scaling, rotation and
translation.
• Image classification was done using Artificial Neural
Networks which made our system more intuitive and
adaptive in nature. It was also significantly quicker and
more efficient than Template-Matching and nearest
neighbor classifiers.

I. INTRODUCTION
Old age results in infirmity which eventuates in bodily
enervations. This state in old people often leads to adverse
scenarios where locomotion may pose a major challenge. This
situation is more tragic in the case of the paralyzed and the
quadriplegic [1]. In perspective of these issues, a locomotive
contrivance would serve a monumental purpose, especially
one that comprehends and complies with the users’ will. The
challenge would thus be to devise and construct a locomotive
contraption viz. a wheelchair which accords with the subject’s
instinctive volition and reciprocates in a manner that will
assist navigation. Fig.1 The conceived architecture of the proposed electronic wheelchair
Various wheelchair systems are found aplenty. Voice
operated devices are by far the most popular. However, voice In perspective of the affording a more natural user
operated wheelchairs certainly inhibit the user’s natural ability communication and in order to minimize human-system
to communicate. Moreover, it doesn’t conform to the user’s interaction, we have endeavored to present two distinct but
instinctive ability to enable locomotion and may prove to be closely related concepts viz. the hand gesture[2,3] and vision
very difficult to control and maneuver. Numerous attempts based control system in conjunction [4] which are used as
have been made to map eye movements onto a display and to modes of communication to the electronic wheelchair
perform certain tasks by drifting the cursor across various contingent on the users’ will, subsequently facilitating
options available across the terminal. Though successful, this locomotion. The idea here is to improve the independence of
mode of operations exerts a huge amount of stress on the user physically challenged people over complicated systems with
and may make the system impracticable. Certain endeavors complex functionalities thus offering a locomotive device that
have also been made to develop thought-controlled comprehends the users’ instinctive thought process enabling
wheelchairs. simplified motion.
Our application distinguishes from the previous attempts This paper is organized into three main sections. Firstly, a
because of few pronounced differences viz. detailed elucidation of the principles involved in hand gesture
control is presented. This section also illustrates the various

978-1-4244-5441-9/09/$26.00 ©2009 IEEE 168

Authorized licensed use limited to: Thiagarajar College of Engineering. Downloaded on February 17,2010 at 07:31:48 EST from IEEE Xplore. Restrictions apply.
image processing methods adopted followed by a discussion
on Image Moments and Artificial Neural Networks, which
facilitates convenient classifications This is followed by a
description of the methodology for vision based control. The
final section presents the test results of our experimentations
which uses the aforementioned concepts in concurrency.
II. SYSTEM CONFIGURATION
The conceived architecture of the wheelchair is shown in
fig. 1. The illustration shows the presence of two IEEE-1394,
Monochrome, CMOS cameras for image acquisition. For
capturing hand gestures, a monochrome camera is situated
facing downward as shown. For enabling vision control, eye
movements are captured. This is done by the second
lightweight camera affixed to a band around the forehead,
focused on the eye as outlined. The cameras can be
repositioned based on users’ convenience. The wheelchair
consists of a black colored hand rest. To avoid system
complexity the user is restrained to issue the hand gestures
while placing the hand on the hand rest. Also the hand rest is
black colored to make the background uniform and to
alleviate succeeding complications in image processing.
Besides the cameras, the system consists of an image
processing and a decision-making unit. The image processing
was performed exclusively using LabVIEW RT with Vision
Development Module and decision-making was established
using NI LabVIEW RT with the help of features extracted
from the image-processing unit. The result of the decision-
making unit would be signaled as commands to the motors
through the digital I/O module 9472.
Firstly, video acquisition of the eye movement and hand
motion is either done in synchronism or separately based on
the nature of control to be executed. Appropriate information
in the gestures is extracted and issued as commands to
maneuver the wheelchair. Fig. 2 demonstrates the general
system setup.
Fig. 2 System Configuration
III. SYSTEM IMPLEMENTATION
Compact Vision System [NI CVS 1454[5]] and Compact
RIO [NI cRIO 9074[6]] were used as image processing and
decision making unit respectively .The cameras are interfaced
with CVS-1454.
NI cRIO Digital I/O Module- 9472 was used to signal the
decision made by the NI cRIO-9074 to the respective motors.
An NI cRIO analog input module 9201 was used to take
analog input from sensors and microphone and send the
169
Authorized licensed use limited to: Thiagarajar College of Engineering. Downloaded on February 17,2010 at 07:31:48 EST from IEEE Xplore. Restrictions apply.
information to the NI cRIO for immediate processing. Four
proximity sensors were used to check whether the desired
direction is obstacle- free or not.
The Firewire, CMOS cameras are made to monitor the
hand or eye movements constantly. The wheelchair is initially
in stand-by mode. A switch is pressed to initialize the system.
Following this activation, the camera performs image
acquisition for processing. Depending on the results of this
image processing, commands are signaled to the motors.
However, prior to image processing we perform motion
detection on this sequence of acquired frames to discard
intermediate and inappropriate frames between any two
consecutive legitimate gestures. Thus only those frames in
which the hand or eyeball is static for a considerable amount
of time is taken for image processing and all others with
motion or blurring are discarded. This time interval is entirely
dependent on the convenience of the user. Considering only
relevant frames is of primary importance as the system
requires only legitimate gestures to issue commands and every
other gesture is unneeded. Thus it is wise to discard all
unnecessary frames in concern of not overloading the
processor.
The following sections will elaborate the image processing
techniques that were adopted for A) Hand Gesture Control B)
Vision Based Control respectively.
A. Hand Gesture Control
Hand Gesture control plays a primal role is our application.
In this method of control, the user gesticulates to the system a
specific, pre-defined hand gesture. The wheelchair in turn
reacts to the specific gesture issued. The user has at his
disposal the control of the wheelchair and can maneuver it the
way he wants.
(a) (b)
Fig.3 (a) Flowchart describing Hand Gesture control and (b) Flowchart
describing Motion Detection procedure
As mentioned earlier, in view of reducing imaging
processing complexity, the user is constrained to give the
gesture while placing the hand on the hand rest. This however
will not be of inconvenience as placing the hand on the hand
rest is an ergonomically viable posture. Also in perspective of
reducing complexity of non-uniform background, the hand
rest is made black in color. The basic steps followed in hand to reproduce an exact hand gesture that is already stored in the
gesture control are shown in the flowchart in fig.3 (a). database. It is also prone to errors when the image is scaled
which is likely when the camera is repositioned. The gesture
1) Image Acquisition: Our application focuses on the hand
variations because of rotation, scaling, and translation can be
segment enclosed between the wrist and the fingertips, and
circumvented using a set of features that are invariant to these
any image processing in done only on this portion of the hand.
operations.
This consideration is of chief importance in assisting the
system in much easier interpretation of the gestures. The
images are continuously acquired by the camera that is fitted
over the hand rest and are pre-processed to check whether the
gesture given is static or not.
2) Removal of Erratic Gestures (Motion Detection): In
order to eliminate the necessity to process every frame we Fig.5 Image Moments of normal, scaled, translated, and rotated images
resort to performing motion detection on the sequence of
Table I— The 7 Image Moments of the gestures in fig.5
images prior to any image processing. The user is stipulated to
issue a legitimate gesture for a specific interval of time such
that there is insignificant hand motion in that interval. Thus
with motion detection all the intermediate frames between any
two legitimate gestures which are either blurred or
unrecognized can be discarded and only the valid gestures are
considered for future processing. The concomitant flowchart
shown in fig. 3(b) elaborates the process of motion detection.
3) Thresholding: The relevant image is now thresholded
using Adaptive Thresholding which will automatically set the
threshold value to figure out brighter objects. Since we are
using a black background, our hand will behave as a bright
object and will be perfectly separated from the background
after Thresholding. To indentify the region of interest, which
in this case is the region enclosed between the wrist and the The fig.5 shows an example of a threshold hand gesture,
fingertips, we resort to usage of bands worn around the wrist. which is scaled, translated, and rotated respectively. The table
This proved to be of immense assistance in delineating the 1 shows their respective image moments. It is clear from the
ROI from the extraneous segment of the hand. table that the image moments are significantly close to each
other. This example substantiates the usability of image
4) Morphological Operations: Despite Thresholding, in moments in object recognition applications as they are
majority of the cases, the resultant image fails to contain only considerably invariant to scaling, rotation and translation. We
the concerned hand figure. The image may consist of border have taken 3 image moments Φ1, Φ2, and Φ3 for our
objects or stray pixels which may debilitate precise feature experimentations. The usage of Image Moments largely
extraction. In order to remove these we resort to certain simplifies our algorithm and decreases the processing time as
morphological operations which essentially includes removal compared with template-matching.
of border objects, small objects, extraneous or noisy pixels Having achieved the above stages, we have successfully
resulting in a clean binary image with the requisite segment of extracted the image moments from an image of a user’s hand
the hand. Fig. 4(a)-(e) depict the aforementioned step, in that gesture. However, these moments would remain meaningless
order. Fig. 4(e) is the final thresholded image which is unless we are able to classify the newly issued gesture into
qualified for the subsequent gesture classification process. one of the pre-defined gestures which would correspond to a
particular command to control the wheelchair. We have
utilized the concept of Artificial Neural Network [ANN] for
Hand Gesture Classification.

(a) (b) (c) (d)
Fig. 4 (a) Original image; (b) Thresholded image; (c) Removing border; (d)
Removing small objects ; (e) Filling Holes.
5) Gesture Classification:
Calculation of Moment Invariants [7]: For feature extraction
we use image moments. Classification using Template
Matching and Nearest Neighbor classifiers were tried which
proved to be highly susceptible to errors when a user is unable
(e) Estimating the Best Set of Gestures: Faulty gesture
classification due to overlapping moment invariants can be
obviated by selecting the most appropriate gestures for our
application. Over 5000 sample images with corresponding
moment invariants, representing 12 gestures, were chosen
insuring that these were distinctly identifiable and
ergonomically feasible to issue. Accuracy scores for these
gestures as a result of succeeding experimentation is exhibited
in fig.6

170
Authorized licensed use limited to: Thiagarajar College of Engineering. Downloaded on February 17,2010 at 07:31:48 EST from IEEE Xplore. Restrictions apply.
 were used as the activation functions in layer 1 and layer 2
respectively. We used NI LabVIEW to train the system using
Back Propagation Algorithm and Delta Rule.
The index of the neuron which fires in the last stage
represents the type of gesture recognized. We obtained an
output vector of size 8x1 with one element ‘1’ and remaining
‘0’s. Whichever command corresponds to the output of index
1 is executed. For example, if the output is [0 1 0 0 0 0 0 0 ]’,
gesture 2 is recognized and the command corresponding to it
is carried out. If more than one neuron fires at the same time
or if none fires then the gesture is considered unrecognized [9].
B. Vision Based Control:
Fig.6 The 12 distinct gestures with their respective classification accuracy
scores Vision based control has been included in our application in
concern of enhancing the scope of the functionalities offered
Furthermore, a Φ1-Φ2-Φ3 3D plot was used to rule out the
by the electronic wheelchair. It isn’t hard to notice that eye
gestures with overlapping spatial distributions. Fig.7 shows
movements are far more comfortable than moving any other
the resultant 8 gestures that were culled out by discarding
part of the body. This simple notion plays a momentous role
gestures with overlapping image moments and low accuracy
in establishing motion control for maneuvering the wheelchair.
score. Notice that the moment invariant clusters are distinctly
Fundamentally, the notion is to afford a system that directs the
distributed in space and non-overlapping.
subject to the place desired merely by eye movements i.e. the
direction of motion is determined by where the subject is
looking. This rationale can be corroborated by everyday
experience. Whenever a person desires to move to a place in
close proximity, his instinctive reaction would be to look
toward the place before making a move. It is possible to track
the movement of the iris, which is an obvious indicator of the
direction of travel.

Fig.7 A Φ1-Φ2-Φ3- 3D plot illustrating the spatial distinction in the clusters

of image moments for the hand gesture chosen.

Training of ANN using optimal gesture set: Neural networks

[8]
have been applied to perform complex functions in
numerous applications, including pattern recognition,
classification, identification etc. In comparison with other
methods, a neural network classifier is more effective and less
time-consuming. The property of ANN to learn and predict
over time makes it a powerful classification tool ideal for
complex computation. ANN has been used for hand gesture
classification thus providing for a more intuitive system.
Following the elimination process, the neural network was
trained using these optimal set of gestures. The system was Fig. 8 Image Processing using NI Vision on a sample image to determine the
trained with approximately 1000 gestures corresponding to the center coordinates of the iris.
8 gestures selected earlier, each with around 120 sample
Following the acquisition of applicable images using
images. The number of iterations was taken as 1000 with
motion detection as indicated in section II, we perform the
mean square error to be 0.01 and the ANN was trained twice
following sequence of processes to procure the centre
to develop an accurate transfer function between the input and
coordinates of the iris. The following image calibration tools
the output. We have developed a two layer Perceptron model
are used to remove nonlinear and perspective errors caused by
for our applications in order to recognize the correct gestures
lens distortion and camera placement.
issued to the system.
Firstly, we begin by smoothing the acquired image using a
The first layer consists of ten neurons and the second layer
smoothing filter. This is done in order to remove noises from
contains eight neurons. Sigmoid and Pure Linear functions
the acquired image. This process is followed by Thresholding

171
Authorized licensed use limited to: Thiagarajar College of Engineering. Downloaded on February 17,2010 at 07:31:48 EST from IEEE Xplore. Restrictions apply.
of the image so as to separate the pupil from the rest of the
image. This step is accompanied by morphological operations
such as removal of border pixels and small objects and
particle filtering to improve the quality of the processed
images and to ensure its independence over noise and
distortions. We then eventually find the centre coordinates of
the iris with the help of its curvature using the ‘circle detector’
function. All concomitant snapshots shown in fig.8 would
attest to the aforementioned steps.
Fig. 9 Depiction of the Vision Controlled algorithm
The Algorithm: In order to determine where the subject is
looking, the following methodology is adopted.
• CR: X-Coordinate of the Reference Point
• C: X-Coordinate of the Centre of the Eye
• b: Parameter that determines the direction of
motion— this can be calibrated based on users’
convenience and surrounding conditions
• If CR>C and |CR –C|>b then, turn RIGHT.
• If CR<C and |CR -C|>b then, turn LEFT.
• If |CR –C| <b then, go STRAIGHT.
We have taken the reference point as the centre of the iris
in the case when the subject is looking straight. With respect
to this reference we can decide where the subject is
looking, for e.g. if he is looking toward left, his intention is
to turn left. The extent to which the negotiation of turn is to be
executed can be calibrated based on the surroundings and
users’ comfort level. The accompanying fig.9 shows the
reference point. When the subject moves his eye beyond the
parameter ‘b’ as shown, there’s an indication to the system to
execute a turn in that direction to the extent defined by the
user.
C. Motion Control:
In order to establish full fledged motion control, we have
incorporated both vision based and hand gesture based control
in conjunction. Fig.10 shows a flow chart that illustrates the
general conception of motion control implemented using a
combination of both these methods. The issued hand gesture is
identified to be one of the qualified 8 gestures. If this issued
gesture is gesture 1, then the responsibility of direction control
is transferred to the eye motion. The wheelchair then traverses
172
Authorized licensed use limited to: Thiagarajar College of Engineering. Downloaded on February 17,2010 at 07:31:48 EST from IEEE Xplore. Restrictions apply.
in the direction of the line of sight (LoS). If the issued gesture
isn’t gesture 1, then whichever command corresponds to the
identified gesture is executed. Five of the 8 gestures are
shown in fig. 11 with the corresponding commands performed.
Fig.10 Flowchart illustrating the motion control implementation
IV. LIMITATION, ISSUES AND SOLUTIONS
The following were the issues that we confronted during
the course of our experiments and correspondingly the
proposed solutions.
1) Lighting Conditions: Poor illumination of captured
images will have dramatic declination on the effectiveness of
the system. In order to combat this issue, the light source is
placed in such a way that the maximum light that is reflected
back is from the hand. The hand is illuminated using a 60W
light source. To prevent the light source from perturbing the
user, an 850nm filter is positioned over the source. The
illumination cannot be detected by the human eye, as 850nm
falls in the infrared region, and hence does not agitate the user.
Also a diffused, non-obtrusive light source is used to
illuminate to subject’s eye in event of poor illumination.
2) Defining the Region of Interest (ROI): In view of
identifying the region of interest, which in this case is the
region enclosed between the wrist and the fingertips, we resort
to usage of a band worn around the wrist. This proved to be of
immense assistance in delineating the ROI from the
extraneous segment of the hand. The color of the band was
chosen black to match the background, which facilitated the
image processing.
V. EXPERIMENTAL RESULTS:
Fig. 11 Hand gestures numbered 1-5 corresponding to the command they
execute
 Fig. 12 (a) The Front Panel of our application; (b) LabVIEW VI — The entire Block Diagram for Hand Gesture and Vision Controlled Wheelchair
Fig.11 above shows five of the eight eclectic gestures we
had obtained earlier, essential for our application. Gesture 1
transfers the responsibility of direction control to eye motion.
The wheelchair travels in direction of line of sight. Gesture 2
halts the wheelchair. Gesture 3, 4, 5 make the wheelchair
traverse straight, rotate left and right respectively. Fig.12 (a)
and 12(b) show the front panel and the LabVIEW VI block
diagram of our application.
All image-processing experimentations were performed
using NI Vision Assistant and NI LabVIEW was used for both
training and implementing the Neural Network and also to
effect motion detection and control of wheelchair. We used a
computer running on Windows XP with a 2GHz processor
and 512MB for all our initial experimentations.
Fig. 13 NI Vision script —Image processing for Hand Gesture Control
Fig. 14 NI Vision script —Image processing for Vision Based Control
The NI Vision scripts which were prepared for Hand
Gesture and Vision based control are shown in fig. 13 and
fig.14 respectively. Our image processing algorithm to
process an image on an average required only about 9ms and
556ms for Hand Gesture and Vision based Control
respectively. We computed a total of less than 650ms for the
entire operation on our computer, right from image acquisition
to an action being executed.
Our hardware experimentation includes the usage of CVS-
1454 and cRIO- 9074 with cRIO digital I/O module 9472. The
Compact Vision System [CVS-1454] was interfaced with two
Firewire [IEEE-1394], CMOS cameras.
The CVS-1454 was configured such that it resides on the
same subnet as the development computer. The VI was
deployed on CVS-1454 enabling it to run the application
without further connection to the development computer. The
CMOS cameras are made perform image acquisition, and the
image processing was performed in Real-Time by CVS-1454.
The Image Moments, which forms the output of the CVS is
dispatched to cRIO-9074. An Ethernet cable is used to link
173
Authorized licensed use limited to: Thiagarajar College of Engineering. Downloaded on February 17,2010 at 07:31:48 EST from IEEE Xplore. Restrictions apply.
cRIO and CVS. The requisite information, once passed onto
cRIO, was subject to further processing. The VI prepared on
the development computer was also deployed onto the cRIO.
The cRIO processes the Image Moments and with the help of
Neural Network classifies the issued gesture and dispatches
the command to the motors via the cRIO digital I/O module
9472.
VI. SUMMARY AND FUTURE WORK
The paper describes a novel vision and hand gesture based
locomotive system ideal for physically challenged individuals.
The electronic wheelchair also focuses on complying with
user’s instinctive volition enabling much easier mobility. An
eclectic set of 8 hand gestures were selected by statistical
analysis and were used in our experimentations. Our system
excels other systems in that it uses only a minimum number of
gestures and facilitates easy implementations with accurate
results. The system proved to be highly responsive to the user
inputs and behaved precisely in the manner it was
programmed to. We plan to incorporate dynamic, three
dimensional gestures[10] in the future for increased
functionalities .We also plan to incorporate GPS based failure
notification system, which can also be used to monitor the
users’ whereabouts.
REFERENCES
[1] Mark Weiser, “The Computer for the Twenty-First Century,” Scientific
American, pp. 94–104, September 1991.
[2] Davis, J., and Shah, M.: ‘Recognizing hand gestures’. Proc. European
Conf. Computer Vision, Stockholm, 1994, pp. 331–340
[3] C. Metzger, M. Anderson, and T. Starner. Freedigiter: A contact-free
device for gesture control. In Eighth IEEE International Symposium on
Wearable Computers (ISWC’04), pages 18–21, 2004.
[4] Marius et al.: ‘Face detection using color thresholding, and Template
matching’, Stanford University, CA,May
2003.http://www.stanford.edu/class/ee368/Project_03/Project/reports/e
e368group15.pdf
[5] http://www.ni.com/pdf/products/us/2005_5732_221_101_lo.pdf
[6] http://www.ni.com/pdf/manuals/374639c.pdf
[7] M. Hu, ‘‘Visual pattern recognition by moment invariants,” IRE
Trnrisnctioiis oii biforriiatioii Tlieory, vol. IT& pp. 179-187, 1962.
[8] Haykin, S.: ‘Neural networks’ (Prentice-Hall, 1999, 2ndedn.)
[9] Gonzalez, R.C., Woods, R.E., and Eddins, S.L.: ‗Digital image
processing using MATLAB‘ (Pearson Prentice-Hall, 2004)
[10] Y. Xiaoming and X. Ming, "Estimation of the fundamental matrix from
uncalibrated stereo hand images for 3D hand gesture recognition",
Pattern Recognition, vol. 36, pp. 567-584, 2003.