VIRTUAL MOUSE USING COLOUR
SEGMENTATION
Samyak Jain 17BCE0139
VIT UNIVERSITY
Vellore samyak.jain2017@vitstudent.ac.in
Abstract— Since the computer technology continue to
grow up, the importance of human computer interaction is
enormously increasing. Nowadays most of the mobile devices
are running using a touch screen technology. However, this
technology is still not cheap enough to be used in desktop
systems. Creating virtual human computer interaction device
such as mouse or keyboard using a webcam and computer
vision techniques can be an alternative for the touch screen. In
this study, finger tracking based on a
virtual mouse application has been designed and implemented
using a regular webcam.
The motivations is to create an object tracking application to
interact with computer, and develop a virtual human computer
interaction device.
Keywords— HUMAN COMPUTER
INTERACTION, VIRTUAL MOUSE,FEATURE
EXTRACTION, VISUALIZATION, MATLAB
Introduction
In this project, the mouse cursor movement and click events are
controlled using a camera based on colour detection and
segmentation technique. Here real time video has been
captured using a Web- Camera. The user wears coloured tapes
to provide information to the system. Individual frames of the
video are separately processed. The processing techniques
involve an image subtraction algorithm to detect colours. Once
the colours are detected, the system performs various
operations to track the cursor and performs control actions. No
additional hardware is required by the system other than the
standard webcam which is provided in every laptop computer.
In one proposed system we can control the mouse cursor
movement and click events of the mouse using hand gestures.
Proposed system was worked on eight steps; Capturing the real
time video (webcam captures the real time video at a fixed
frame rate), Flipping of Images (camera captures an image
which is in the form of inverted, means that if we move the
colour pointer towards the left, the image of the pointer moves
towards the right and vice-versa), Conversion of Flipped Image
into Gray scale Image (computational complexity is reduced in
a grayscale image as compared to a coloured image), Colour
Detection (This process creates an image which contains the
detected object as a patch of grey surrounded by black space),
Conversion of gray scale Image into Binary scale Image (grey
region of the image obtained after subtraction needs to be
converted to a binary image for XXX-X-XXXX-XXXX-
X/XX/$XX.00 ©20XX IEEE
finding the region of the detected object), Finding Centroid of
an object and plotting (MATLAB is used to find the centroid
of the detected region) , Tracking the Mouse pointer(Once the
coordinates has been determined, the mouse driver is accessed and the
coordinates are sent to the cursor), and finally Performing Clicking
Actions (Clicking action is based two colours 1. If Red along with
Green colour is detected, Left clicking action is performed and 2. If
Red along with Blue colour is detected, Right clicking action is
performed.
Another one proposed was the Virtual Mouse using Hand
Gesture and Color Detection technology, in which three
technologies are mainly used: object detection, image
processing and color recognition. The proposed system
consists of four parts: Hand Gloves (gloves will have
different colors painted on different fingers, which will
make the system faster and easier for the algorithm),
Webcam (necessary component for detecting the image),
image processing and mouse movement.
In this proposed system, image processing is the main part.
Image processing was done in 5 steps; Color Detection
(MATLAB built in function “imsubstract” can be used),
Filtering the Images (Median filtering gives optimum
results), Conversion of Images (after filtering was done
over, For conversion of image one may also use in built
function
“im2bw”), Removing small areas (here MATLAB's
“bwareaopen” function was used) and Find Centre (Here
“bwlabel” Matlab function can be used for cropping the
genuine area). Next is Move the cursor and Mouse click
event, after Find Centre step centroid from final image,
movement has to take place and finalize (X,Y) coordinates
where clicking action is taken.
A third example is the virtual mouse implementation using
color pointer detection. In the first step Webcam is used to
capture and recognize an object in view and tracks the user's
hand gestures using computer vision techniques. In the
second step Video is divided into Image frames based on the
FPS (Frames per second) of the camera. In the third step red,
green and blue color object is extracted or detected by
subtracting the flipped color suppressed channel from the
flipped Gray Scale Image. In the fourth step MATLAB is
used to find the centroid of the detected region i.e. Matrix
consisting of the X (horizontal) and Y (vertical) coordinates
of the centroid. Finally Perform Clicking Actions, where
JAVA is used to access these flags / centroids.
SURVEY
Virtual mouse implemented with hand gesture tracking based on
image is one of studies in human-computer interaction. In this
study that human-computer interaction is implemented with virtual
mouse is purposed. This study consists of three main steps that are
hand gesture tracking, features of hand region extraction and
classification of these features. In this study hand gesture tracking
is generated with Camshift (Continuously Adaptive Mean Shift)
algorithm, features of hand gestures are extracted with bag of
visual words and these features are classified with support vector
machines. Detailed tests are performed to compare success of
tracking, features extraction and classisfication methods and that
this system works successfully is shown..
W.-W.M. Tsang and Kong-Pang Pun (2005), presented a virtual
mouse that is based on tracking the movement of a finger is
presented. The idea is for users to control their computer or TV
system simply by moving their fingers, without any contact on
realistic objects. A fast and robust method that can trace the
position of finger tip is proposed. The method consists of
four steps. First, the skin-color pixels of figures are detected
by color segmentation using the chrominance component of
the input from a CMOS image sensor. Second, density
regularization processes are carried out to reinforce the
regions of skin-colour pixels. Third, an effective window
search technique is applied for minimizing computational
cost. Fourth, a finger-tip tracking algorithm is applied to
find out the finger-tip position. Clicking action is also
implemented by a specific movement. The virtual mouse
has been designed in an embedded Linux system. The
device works successfully, the response is quick and
accurate positioning is obtained.
Javeria Farooq and Muhaddisa Barat Ali (2014), presented that
Hand gesture recognition is a natural and intuitive way to
interact with the computer, since interactions with the computer
can be increased through multidimensional use of hand gestures
as compare to other input methods. The purpose of this paper is
to explore three different techniques for HGR (hand gesture
recognition) using finger tips detection. A new approach called
“Curvature of Perimeter” is presented with its application as a
virtual mouse. The system presented, uses only a webcam and
algorithms which are developed using computer vision, image
and the video processing toolboxes of Matlab.
Li Wensheng, Deng Chunjian and Lv Yi (2010) presented A
method to implement virtual mouse through machine vision are
proposed. Firstly we present an efficient algorithm based on
color to track the movement of fingertips in real time, then we
present a set of messages (including elementary fingertip
message and simulated mouse message) that are generated
according to the result of fingertip tracking, These messages
can be used to control the window based applications. We also
presents a framework to help programmers realize human-
computer interaction based on virtual mouse. The performance
of virtual mouse shows a great promise that we will be able to
use this methodology to control computers and achieve a more
natural human-computer interaction.
Alexandre Henzen and Percy Nohama (2016), present a new
solution that allows people with special needs to use all the
features of the keyboard and mouse via sensors connected to a
user's body part in which it maintains motor control. The
application developed constitutes a keyboard and mouse
emulator, configurable and adaptable, which was named ETM
(Emulator Keyboard and Mouse). The software captures
trigger's signals connected to the USB (Universal Serial Bus),
through the adaptation of the contacts of the buttons of a
joystick. By means of a command processor and a key layout
tailored to user needs, it sends keyboard and mouse commands
for any Windows application. Tests conducted with cerebral
palsy in the project participants schools and volunteers with
amyotrophic lateral sclerosis, have shown satisfactory results.
With the adaptation of sensors and programs developed, it
optimized communication and user learning, minimized motor
limitations due to neurological diseases, and it has expanded his
accessibility and digital and social inclusion.
Ashish Mhetar (2014) proposed a product which aims to be a
virtual marker and also has additional advantage that it has
features of a Mouse incorporated into it. The existing Virtual
Marker is modified to function more than a marker and act as a
mouse pointer also giving it the advantage of all the
functionality of a mouse. In this paper, we have proposed a
hardware implementation of a Virtual Mouse which has an
improvement in performance of the existing "Virtual Marker" by
making it highly responsive in real time.
Ramazan Özgür Doğan and Hülya Doğan (2015) proposed a study
study that human-computer interaction is implemented with virtual
mouse is purposed. This study consists of three main steps that are
hand gesture tracking, features of hand region extraction and
classification of these features. In this study hand gesture tracking is
generated with Camshift (Continuously Adaptive Mean Shift)
algorithm, features of hand gestures are extracted with bag of visual
words and these features are classified with support vector machines.
Detailed tests are performed to compare success of tracking, features
extraction and classisfication methods and that this system works
successfully is shown.
Tomas Palleja (2008) introduced a work conducted to create a
relative virtual mouse based on the interpretation of the head
movements through a low cost camera. This virtual device is
designed as a non-contact alternative for people with mobility
impairments. First validation results show performance close to a
digital joystick.
Farzana Jabeen and Linmi Tao (2016) described that Virtual reality
is an emerging paradigm that is trying in context to enhance their
independence and quality of life. But it is not so easy to interact
with the VR applications. When you wear the virtual glasses you
won't be able to see the physical world in real. The common user
problem is to use the input devices (keyboard, mouse, remote
control) to change the menu and access the different features of
VR. But in case of people with special needs they will not be able
to handle interaction with such devices. To make maximum use of
virtual reality technique effective, we are proposing one bit virtual
mouse. By using one key, virtual mouse provides the control access
to VR application with ease as well as training environment for the
users. The primary evaluation of the system is done with the
experiment. The post experiment results and disable people
feedback show performance of the system is satisfactory.
Changhyun Jeon and Dongil Shin (2017) presented a virtual
monitor that allows a virtual space to be controlled by the hand
mouse. It is possible to map the coordinates on the virtual monitor
to the coordinates on the real monitor accurately. A hand-mouse
interface based on the virtual monitor concept maintains the
outstanding intuitiveness that is the strength of the previous study
and enhances the accuracy of mouse functions. In order to evaluate
the intuitiveness and accuracy of the interface, we conducted an
experiment with 50 volunteers ranging from teenagers to those in
their 50s. The results of this intuitiveness experiment showed that
84% of the subjects learned how to use the mouse within 1 min. In
addition, the accuracy experiment showed the high accuracy level
of the mouse functions [drag (80.9%), click (80%), double-click
(76.7%)]. This is a good example of an interface for controlling a
system by hand in the future.
Byungsung Lee and Junchul Chun (2010) presented a vision-based
framework to manipulate the augmented reality (AR) objects
robustly in a marker-less AR system. It is known that one of the
promising ways to develop a marker-less AR system is using
human's body such as hand or face for replacing fiducial markers.
This paper presents a real-time method to manipulate the AR
objects dynamically in a developed 3D hand coordinate system. To
build the marker-less system, we utilize YCbCr skin-color model
for hand shape detection and curvature-based fingertip detection
from an input video images. Using the detected fingertips the
camera pose are estimated to register virtual objects on the hand
coordinate system. In order to manipulate the movements of the
virtual objects rendered on the marker-less AR system
dynamically, a vision-based hand control interface, which utilizes
pattern recognition for the hand command initiation, is
developed. From the experiments we can prove that the
proposed hand mouse can manipulate the objects in a
feasible fashion.
Several authors have suggested various techniques for
preprocessing and enhancement in the last two decades. The
task of virtual image enhancement is to sharpen the edges
to increase the contrast between suspicious regions and the
background. Image enhancement includes intensity and
contrast manipulation, noise reduction, background
removal, edges sharpening, filtering, etc.
We have studied different techniques for segmentation.. Some
studies focused on trained data while other targeted untrained
data; some studies used atlas, and some did not. A critical
review of the studied literature is summarized in the compare
and contrast table (Table I).
PROBLEM STATEMENT
Virtual Mouse using Color
Segmentation
ARCHITURES
IMPLEMENTATION
Camera Settings
The runtime operations are managed by the webcam of the
connected laptop or desktop. To capture a video, we need to create
a Video Capture object. Its argument can be either the device
index or the name of a video file. Device index is just the number
to specify which camera.Since we only use a single camera we
pass it as ‘0’. We can add additional camera to the system and
pass it as 1,2 and so on. After that, you can capture frame-
byframe. But at the end, don’t forget to release the capture. We
could also apply color detection techniques to any image by doing
simple modifications in the code.
Capturing frames
The infinite loop is used so that the web camera captures the frames
in every instance and is open during the entire course of the
program. We capture the live feed stream, frame by frame. Then
we process each captured frame which is in RGB(default) color
space to HSV color space. There are more than 150 colorspace
conversion methods available in OpenCV. But we will look into
only two which are most widely used ones, BGR to Gray and BGR
to HSV
Masking technique
The mask is basically creating some specific region of the image
following certain rules. Here we are creating a mask that comprises
of an object in red color. After that we perform a bitwise AND
operation on the Input image and theThreshold image, which result
in only the red colored objects are highlighted. This result of the
AND operation is stored in res. We then display the frame, res and
mask on 3 separate windows using imshow() function.
Display the frame
The imShow() is a function of HighGui and it is required to call the
waitKey regulerly. The processing of the event loop of the
imshow() function is done by calling waitKey. The function
waitKey() waits for key event for a “delay” (here, 5 milliseconds).
Windows events like redraw, resizing, input event etc. are
processed by HighGui. So we call the waitKey function, even with
a 1ms delay.
Mouse Movement
We have to first calculate the center of both detected red object
which we can easily do by taking the average of the bounding boxes
maximum and minimum points. now we got 2 co-ordinate from the
center of the 2 objects we will find the average of that and we will
get the red point shown in the image. We are converting the
detected coordinate from camera resolution to the actual screen
resolution. After that we set the location as the mouse_position. but
to move the mouse pointer it will take time. So we have to wait till
the mouse pointer reaches that point. So we started a loop and we
are not doing anything there we are just waiting will the current
mouse location is same as assigned mouse location. That is for the
open gesture.
Clicking
The next step is to implement the close gesture. The operation is
performed by clicking the object and dragging it. It is similar to the
open gesture, but the difference is we only have one object here so
we only need to calculate the center of it. And that will be placed
on the location where we will position our mouse pointer. Instead
of mouse release operation we will be performing a mouse press
operation.
Drag
In order to implement the dragging we indroduce a variable
‘pinchflag’. It will be set to 1 if it was clicked earlier. So
after clicking whenever we find the open gesture we check
if the pinchflag is set to 1. If it is set to one then Drag
operation is performed otherwise the mouse move operation
is performed. 5.8 DnD Frame First we create the
MyFileDropTarget class. Inside that we have one
overridden method, OnDropFiles. This method accepts the
x/y position of the mouse along with the file paths that are
dropped.
File Transfer
Using the file path obtained from MyFileDropTarget class
we send the file to the other system using the python socket
module. And there it recieves and creates the file.
ALGORITHM
The project works on the concept of colour segmentation.
The colour detected are mapped on the live cam feed and
are processed to find the appropriate gesture. The methods
and concept used are: For Image Processing:
● Thresholding
● Segmentation
● Contouring
● Morphological Transformations (Dilation/Erosion)
For Gesture Recognition:
● Calculated Centroid Distance
● Detected the no of contours
● Calculated maximum area of the contours detected
MAIN MODULE:
The main part of the project is within main module. The main
module is divided into 3 parts
· Input
· Processing
· Output
Input:
This part contains all the passing of the live cam feed. All the
input comes from reading the video frames. Each frame is
captured and send for processing.
Processing:
This is the main part of the project. This module covers all the
preprocessing done on the live cam frames. This processing is
done on the individual frames. It include all basic image
processing techniques like segmentation, thresholding,
morphological transformation and contouring. These
techniques will refine the image and removes noise and detect
colors and segment it according to colors. After detection of
colors we find contours on respective colors. There can be
maximum of 3 contours and minimum of 1 for the cursor to
perform any operation.
Output:
The contours are represented as circles and is displayed in an
interface window.
SCREENSHOTS

For Mouse actions:

● Maintained the aspect ratio of screen ● Creating a stack
of action for smooth execution of action ● Mapped the
mouse pointer to the movement of contours

STEPS FOR IMPLEMENTED ALGORITHM

1. Start Acquisition
2. Input: Image using RGB
3. Convert the RGB image into color space. (Eg. YCbCr,
LAB, HSV)
4. Extract the hand from an image using the color
Models by applying thresholding.
5. Color Based Segmentation using ROI properties.
6. Real time color based Tracking.

MODULES

CALIBRATION MODULE:
This module include the capturing of the pixel value of the
colored objects used to operate the software. For the first
time user it is mandatory for the user to run the calibrate
module so that the software can calibrate the colors and will
able to detect. Since in our project we our using red green
and yellow colors. We can also go with RGB colors but it
will be good if we go by the standard color of the software.
Bright colors are preferable and not metallic colors. The
module function works accordingly: · Just left click the
color you want to capture. · Press the respective color key
from the keyboard Egs : R for red , G for green etc · The
values will be written in the pickle file for future use
 Mouse Movement using a Real Time Camera”, 2008,
Brown University,Providence ,RI ,USA, Department of
computer science
4- Kamran Niyazi, Vikram Kumar, Swapnil Mahe,
Swapnil Vyawahare, “Mouse Simulation Using Two
Coloured Tapes”, Department of Computer Science,
University of Pune, India, International Journal of
Information Sciences and Techniques (IJIST) Vol.2, No.2,
March 2012
5- K N. Shah, K R. Rathod and S. J. Agravat, “A survey on
Human Computer Interaction Mechanism Using Finger
Tracking” International Journal of Computer Trends and
Technology, 7(3), 2014, 174-177
6- Rafael C. Gonzalez and Richard E. Woods, Digital
Image Processing, 2nd edition, Prentice Hall, Upper
Saddle River, New Jersey, 07458 7 - Shahzad Malik,
“Real-time Hand Tracking and Finger Tracking for
Interaction”, CSC2503F Project Report, December
18, 2003

REFRENCES
1- A. Erdem, E. Yardimci, Y. Atalay, V. Cetin, A. E.
“Computer vision based mouse”, Acoustics,
Speech, and Signal Processing, 2002.
Proceedings. (ICASS). IEEE International
Conference
2- Chu-Feng Lien, “Portable Vision-Based HCI – A
Real-time Hand Mouse System on Handheld
Devices”, National Taiwan University, Computer
Science and Information Engineering Department
3- Hojoon Park, “A Method For Controlling The