SIXTH SENSE DEVICE

NAME OF THE PARTICIPANT 1: K.TEJASWINI

BRANCH: ECE
NAME OF THE INSTITUTION: G.N.I.T.S
CONTACT NUMBER: 8885149199
EMAIL ID: kttejaswini@gmail.com

NAME OF THE PARTICIPANT 2: M.SIVANI
BRANCH: ECE
NAME OF THE INSTITUTION: G.N.I.T.S
CONTACT NUMBER: 9550724974
EMAIL ID: sivani.mallajosyula@gmail.com


SIXTH SENSE DEVICE
ABSTRACT
Our basic five senses are sight, hearing,
touch, taste and smell. These five senses
have been helping us over millions of
years to interact with the objects in our
everyday life. But actually the most useful
information that can help us make the right
decision is not naturally apparent with our
five senses. This Sixth sense device is an
attempt to interlink the digital world with
the physical world with the information
via natural hand gestures which is rather
enclosed traditionally on paper or digitally
on a screen keeping us continually
connected to the digital world. Sixth sense
technology was developed by Pranav
Mistry, a PhD student in the Fluid
Interfaces Group at the MIT Media Lab.
The hardware components, a pocket
projector, a mirror and a camera are
brought together in a pendant like mobile
wearable device.
INTRODUCTION
'Sixth Sense' is a wearable gestural
interface that augments the physical world
around us with digital information and lets
us use natural hand gestures to interact
with that information. Sixth sense
technology is the newest jargon that has
proclaimed its presence in the technical
arena. This technology has emerged,
which has its relation to the power of these
six senses. Our ordinary computers will
soon be able to sense the different feelings
accumulated in the surroundings and it is
all a gift of the sixth sense technology.
Sixth sense device is a wearable gesture
based device that augments the physical
world with digital information and lets
people use natural hand gestures to interact
with that information.
Sixth sense in scientific terms is
defined as extra sensory perception or in
short ESP. It involves the reception of
information not gained through any of the
five senses nor is it taken from any
experiences from the past or known. Sixth
sense aims to more seamlessly integrate
online information and technology into
everyday life. By making available
information needed for decision making
beyond what we have access to with our
five senses, it effectively gives users a
sixth sense.
The Sixth Sense prototype is comprised of
a pocket projector, a mirror and a camera.
The hardware components are coupled in a
pendant like mobile wearable device.




HARDWARE COMPONENTS
Camera
The camera is the key input device of the
Sixth Sense system. The camera acts as a
digital eye of the system. It basically
captures the scene the user is looking at.
The video stream captured by the camera
is passed to mobile computing device
which does the appropriate computer
vision computation. The major functions
of the camera can be listed as:
Captures users hand movements
and gestures (used in
reorganization of user gestures)
Captures the scene in front and
objects the user is interacting with
(used in object reorganization and
tracking)
Takes a photo of the scene in front
when the user performs a framing
gesture
Captures the scene of projected
interface (used to correct the
alignment, placement and look and
feel of the projected interface
components)
Projector
The projector is the key output device of
the Sixth Sense system. The projector
visually augments surfaces, walls and
physical objects the user is interacting with
by projecting digital information and
graphical user interfaces. The Projector
projects visual information enabling
surfaces and physical objects to be used as
interfaces. The projector itself contains a
battery inside, with 3 hours of battery life.
A tiny LED projector displays data sent
from the smart phone on any surface in
view object, wall, or person. Pocket
projector Pk101 from Optoma can be used.
The downward facing projector projects
the output image onto the mirror. Mirror
reflects the image on to a desired surface
thus the digital image is freed from its
confines and placed in the real world. The
mobile computing device provides the
projector with the content to be projected.
The major functions of the projector can
be listed as:
Projects graphical user interface of
the selected application onto
surfaces or walls in front.
Augments the physical objects the
user is interacting with by
projecting just-in-time and related
information from the Internet.
Mirror
The mirror reflects the projection coming
out from the projector and thus helps in
projecting onto the desired locations on
walls or surfaces. The user manually can
change the tilt of the mirror to change the
location of the projection. For example in
application where the user wants the
projection to go on the ground instead of
the surface in front, he can change the tilt
of the mirror to change the projection.
Thus, the mirror in the Sixth Sense helps
in overcoming the limitation of the limited
projection space of the projector.
Microphone
The microphone is an optional component
of the Sixth Sense device. It is required
when using a paper as a computing
interface. When the user wants to use a
sheet of paper as an interactive surface, he
or she clips the microphone to the paper.
The microphone attached this way
captures the sound signals of users
touching the paper. This data is passed to
computing device for processing. Later,
combined with the tracking information
about users finger, the system is able to
identify precise touch events on the paper.
Here, the sound signal captured by the
microphone provides time information
whereas the camera performs tracking.
Colored Markers
Marking the tip of users fingers with red,
yellow, green, and blue tape helps the
webcam recognize gestures. The
movements and arrangements of these
makers are interpreted into gestures that
act as interaction instructions for the
projected application interfaces.


SOFTWARE COMPONENT OF
SIXTH SENSE
The prototype system runs on windows
platform and majority of the code is
written in C++ and C#. Applications are
implemented using JAVA 2 MICRO
edition, a Java platform designed for
embedded systems where target devices
range from industrial controls to mobile
phones. Computer vision library is written
in symbian C++ (used in Gesture
tracking). The software for the sixth sense
prototype is developed on a Microsoft
Windows platform using C#, WPF and
open CV. The software works on the basis
of computer vision. A small camera acting
as an eye, connecting us to the world of
digital information. Processing is
happening in the mobile phone, and
basically works on computer vision
algorithms. Approx 50,000 lines of code
are used..
Tokens: This version of code uses colored
fingertips as markers. One needs 4
different colors in order to run most of the
Apps. We recommend using Red, Yellow,
Blue and Green (or colors that are different
enough). We have also noticed that the
matt colored tape/objects works better as
they have less reflective changes. When
you click on Add Markers, the screen
starts blinking and after 4-5 seconds it
stops with a still picture so that you can
add those markers on your hand as new
markers. You can simply drag-select the
colored marker on the screen by mouse
cursor. Once you have all four markers (O,
P, M, and N) added the software will start
tracking the four markers.
Apps: There are three ways to launch any
application. Apps tab can let you choose
the apps you want to load and run. One can
also load apps from Menu. Menu is
triggered using Namaste gesture
(according to your customisation). Once
on Menu screen you can choose the App
by hand-click (Hand Click gesture works
by moving the O marker and hiding P
marker.


RELATED TECHNOLOGY
USED IN SIXTH SENSE
DEVICE

KINDS OF GESTURES
RECOGNIZED
MULTI-TOUCH GESTURES are like the
ones we see in the iphone where we
touch the screen and make the map move
by pinching and dragging. FREEHAND
GESTURES are like when you take a
picture or a namaste gesture to start the
projection on the wall. ICONIC
GESTURES drawing an icon in the air.
Like, whenever we draw a star, show us
the weather details. When we draw a
magnifying glass, it shows us the map.
This system is very customizable. We can
make our own gesture which our sixth
sense device can understand. We can
change the Sixth Sense to our need.

Gesture recognition
Gesture recognition is a topic in computer
science and language technology with the
goal of interpreting human gestures via
mathematical algorithms. Gestures can
originate from any bodily motion or state
but commonly originate from the face or
hand. Current focuses in the field include
emotion recognition from the face and
hand gesture recognition. Many
approaches have been made using cameras
and computer vision algorithms to
interpret sign language. However, the
identification and recognition of posture
and human behaviours is also the subject
of gesture recognition techniques. Gesture
recognition can be seen as a way for
computers to begin to understand human
body language, thus building a richer
bridge between machines and humans than
primitive text user interfaces or even GUIs
(graphical user interfaces), which still limit
the majority of input to keyboard and
mouse. Gesture recognition enables
humans to interface with the machine
(HMI) and interact naturally without any
mechanical devices. Using the concept of
gesture recognition, it is possible to point a
finger at the computer screen so that the
cursor will move accordingly. This could
potentially make conventional input
devices such as mouse, keyboards and
even touch screens redundant. Gesture
recognition can be conducted with
techniques from computer vision and
image processing. The literature includes
ongoing work in the computer vision field
on capturing gestures or more general
human pose and movements by cameras
connected to a computer. This is computer
interaction through the drawing of symbols
with a pointing device cursor.



Augmented reality

Augmented reality (AR) is a term for a live
direct or indirect view of a physical real-
world environment whose elements are
augmented by virtual computer-generated
sensory input such as sound or graphics. It
is related to a more general concept called
mediated reality in which a view of reality
is modified (possibly even diminished
rather than augmented) by a computer. As
a result, the technology functions by
enhancing ones current perception of
reality. In the case of Augmented Reality,
the augmentation is conventionally in real-
time and in semantic context with
environmental elements, such as sports
scores on TV during a match. With the
help of advanced AR technology (e.g.
adding computer vision and object
recognition) the information about the
surrounding real world of the user
becomes interactive and digitally usable.
Artificial information about the
environment and the objects in it can be
stored and retrieved as an information
layer on top of the real world view. The
term augmented reality is believed to have
been coined in 1990 by Thomas Caudell,
an employee of Boeing at the time.
Augmented reality research explores the
application of computer-generated imagery
in live-video streams as a way to expand
the real-world. Advanced research
includes use of head-mounted displays and
virtual retinal displays for visualization
purposes, and construction of controlled
environments containing any number of
sensors and actuators.

Computer vision
Computer vision is the science and
technology of machines that see, where see
in this case means that the machine is able
to extract information from an image that
is necessary to solve some task. As a
scientific discipline, computer vision is
concerned with the theory behind artificial
systems that extract information from
images. The image data can take many
forms, such as video sequences, views
from multiple cameras, or
multidimensional data from a medical
scanner. As a technological discipline,
computer vision seeks to apply its theories
and models to the construction of
computer vision systems. Examples of
applications of computer vision include
systems for: Controlling processes (e.g., an
industrial robot or an autonomous
vehicle),Detecting events (e.g., for visual
surveillance or people counting)
Organizing information e.g., for indexing
databases of images and image sequences,
Modelling objects or environments e.g.,
industrial inspection, medical image
analysis and Interaction (e.g., as the input
to a device for computer-human
interaction). Computer vision is closely
related to the study of biological vision.
The field of biological vision studies and
models the physiological processes behind
visual perception in humans and other
animals. Computer vision, on the other
hand, studies and describes the processes
implemented in software and hardware
behind artificial vision systems. Computer
vision is, in some ways, the inverse of
computer graphics. While computer
graphics produces image data from 3D
models, computer vision often produces
3D models from image data. There is also
a trend towards a combination of the two
disciplines, e.g., as explored in augmented
reality. Sub-domains of computer vision
include scene reconstruction, event
detection, video tracking, object
recognition, learning, indexing, motion
estimation, and image restoration.

Radio Frequency Identification
Radio frequency identification (RFID) is a
generic term that is used to describe a
system that transmits the identity (in the
form of a unique serial number) of an
object or person wirelessly, using radio
waves. It is basically an electronic tagging
technology that allows the detection,
tracking of tags and consequently the
objects that they are affixed to. It's
grouped under the broad category of
automatic identification technologies.
RFID is in use all around us. If you have
ever chipped your pet with an ID tag, used
EZPass through a toll booth, or paid for
gas using SpeedPass, you've used RFID. In
addition, RFID is increasingly used with
biometric technologies for security. Unlike
ubiquitous UPC bar-code technology,
RFID technology does not require contact
or line of sight for communication. RFID
data can be read through the human body,
clothing and non-metallic materials.

Applications
1. Make a call
If one wants to make a call, one
can extend their hand in front of the
projector.

2. Call up a map
With the map application we can call up
the map of our choice by drawing a
magnifying glass and then use thumbs and
index fingers to navigate the map.
3. Time details
The user can draw a circle on wrist to get a
virtual watch that gives the correct time
4. Multimedia reading experiences
Sixth Sense can enrich a users multimedia
experiences. It can be programmed to
project related videos onto newspaper
articles being read.


5. Drawing applications
The drawing application lets the user draw
on any surface by tracking the fingertip
movements of the users index finger.
6. Zooming features
The user can zoom in or zoom out using
intuitive hand movements.
7. Access book information
The system can project Amazon ratings on
that book, as well as reviews and other
relevant information.
8. Access product information
Sixth Sense uses image recognition or
marker technology to recognize products
we pick up, and then feeds us information
on those products.
9. Flight updates
The system will recognize your boarding
pass and let you know whether your flight
is on time and if the gate has changed
10. Take pictures
If you fashion your index fingers and
thumbs into a square ("framing" gesture),
the system will snap a photo. After taking
the desired number of photos, we can
project them onto a surface, and use
gestures to sort through the photos, and
organize and resize them.





ADVANTAGES
Sixth sense is a user friendly interface
which integrates digital information into
physical world and its objects, making the
world your computer.
1. It uses hand gestures to interact
with digital information.
2. It supports multi-touch and multi-
user interactions.
3. It is cost effective.
4. It is portable and is easy to carry as
we can wear it around on our neck.

DISADVANTAGES
1. When a person is taking a picture
using the Sixth Sense, he just snaps
with his fingers which have little
marker caps on them. People dont
want to be captured on a camera of
a random person. One can never
tell when Sixth Sense is taking a
picture, because it is not a big
object and nothing is being help in
your hand in order to take a picture
like a camera or a phone. This is
invading others privacy which can
be a huge problem.
2. Projection is better in the night
time and dark areas rather than
mornings and bright areas.
3. When the device is projecting on a
hard surface, it is not private
enough for just the user. People
around him can see the projection
since it is very detailed.
FUTURE ENHANCEMENTS
1. To get rid of colour markers.
2. To incorporate camera and
projector inside mobile computing
device.
3. To apply this technology in various
interests like gaming, educational
systems etc.
4. To have 3D gesture tracking.
5. To make sixth sense work as fifth
sense for disabled person.

CONCLUSION
Information is often confined to paper or
computer screens. Sixth Sense frees data
from these confines and seamlessly
integrates information and reality. With the
miniaturization of computing devices, we
are always connected to the digital world,
but there is no link between our
interactions with these digital devices and
our interactions with the physical world.
Sixth Sense bridges this gap by
augmenting the physical world with digital
information, bringing intangible
information into the tangible world. The
idea is that the sixth sense tries to
determine not only what someone is
interacting with, but also how he/she is
interacting with it. This software
recognizes the objects around us, searches
the internet for information that is
potentially relevant to that situation ,and
then the projector takes over. The Sixth
Sense prototype implements several
applications that demonstrate the
usefulness, viability and flexibility of the
system, allowing us to interact with this
information via natural hand gestures. The
potential of becoming the ultimate
"transparent" user interface for accessing
information about everything around us.
The Sixth Sense prototype implements
several applications that demonstrate the
usefulness, viability and flexibility of the
system.