SIXTH SENSE TECHNOLOGY

information. It is based on the

ABSTRACT

concepts of augmented reality and

Sixth Sense is a wearable gestural
interface

that

augments

the

physical world around us with

digital information and lets us use
natural hand gestures to interact
with that information.

PhD

student

well

implemented

the

perceptions of it. Sixth sense

technology has integrated the real
world objects with digital world.
The fabulous 6th sense technology
is a blend of many exquisite

It was developed by PranavMistry,

a

has

in

the

Fluid

technologies. The
makes

it

thing

magnificent

which
is

the

Interfaces Group at the MIT Media

marvelous integration of all those

Lab.

technologies and presents it into a

single portable and economical

SixthSense bridges the gap by

bringing

intangible,

digital

product. It associates technologies

like

hand

gesture

recognition,

information out into the tangible

image capturing, processing, and

world, and allowing us to interact

manipulation, etc. It superimposes

with this information via natural

the digital world on the real world.

Sixth sense technology is a

hand
SixthSense comprises a pocket
projector, a mirror and a camera.
The

hardware components

are

coupled in a pendant like mobile

wearable device

perception of augmented reality

concept. Like senses enable us to
perceive information about the
environment in different ways it
also

aims

at

perceiving

information. Sixth sense is in fact,

about comprehensing information

INTRODUCTION

more than our available senses.

WHAT IS SIXTH SENSE?

And today there is not just this

physical world from where we get

Sixth Sense is a wearable gestural

information but also the digital

interface that enhances the physical

world which has become a part of

world

digital

our life. This digital world is now

information and lets us use natural

as important to us as this physical

hand gestures to interact with that

world. And with the internet the

around

us

with

digital world can be expanded

many times the physical world.
God hasnt given us sense to
interact with the digital world so
we have created them like smart
phones, tablets, computers, laptops,
net books, PDAs, music players,
and others gadgets. These gadgets
enable us to communicate with the

Rather than waiting for that time

digital world around us.

to come, I want people to make

their own system. Why not?,

EVALUATION OF SIXTH SENSE

Mistry says in an article on Rediff

TECHNOLOGY

Business. People will be able to

make their own hardware. I will

Steve Mann is considered as the

give them instructions how to make

father of Sixth Sense technology

it. And also provide them key

who made a wearable computer in

softwaregive them basic key

1990. He implemented the Sixth

software layersthey will be able

Sense technology as the neck worn

to build their own applications.

projector with a camera system. He

They will be able to modify base

was a media lab student at that

time. Then his work was carried

level and do anything.

So it can be expected that the

forward by PranavMistry, an Indian

software will be open source and

research assistant in MIT Media

there will be a wide market of apps

Lab. He came up with exciting new

too.

applications from this technology.

Sixth

sense

technology

was

developed at media labs in MIT

and coined as Wear Ur World
(WUW). The inventors have filed
patent under the name Wear Ur
World (WUW) in February 2010.

COMPONENTS

CAMERA

Captures an object in view and

tracks the users hand gestures

The hardware components are coupled in a

pendant like mobile wearable device.
camera
Projector

It sends the data to smart phone

It acts as a digital eye, connecting you to

the world of digital information

PROJECTOR

Mirror
Smart Phone
Colored Markers

The

projector

projects

visual

information enabling surfaces and

physical objects to be used as

interfaces
The projector itself contains a
battery inside, with 3 hours of
battery life.
A tiny LED projector displays data
sent from the smart phone on any
surface in viewobject, wall, or

A Web-enabled smart phone in the

users pocket processes the video

person.

data
Other software searches the Web
and interprets the hand gestures

COLOR MARKERS
MIRROR

It is at the tip of the users fingers .

Marking the users fingers with

red, yellow, green, and blue tape

The

usage

of

the

mirror

is

significant as the projector hang

loosely pointing downwards from
the neck.

SMARTPHONE

helps

the

webcam

recognize

gestures
The movements and arrangements
of these markers are interpreted
into gestures that act as interaction
instructions

for

the

application interfaces.

projected

recognition, image processing, etc.

HOW IT WORKS?

At present the commercial product

Send

for

processing

capture

isnt launched butImages

the prototype is
prepared. The sixthPictures
sense prototype
is made using very
common and
gestures
easily available equipments like
pocket projector, a mirror, mobile
components, color markers and a
camera.

Send info

The

projector

projects

visual

images on a surface. This surface

Information
can be wall, table, book
or even

your hand. Thus, the entire

world is
On surface
available on your screen now.
project

image

When user moves their hands to

form different movements with
colored markers on the finger tips,
the

camera

captures

these

movements. Both the projector and

the camera are connected to the
mobile computing device in the
users pocket. Recognition is made
reflection on

The sixth sense technology uses

different technologies like gesture

using computer vision technique.

These

markers

act

as

visual

tracking fiducials. The software

program

processes

this

video

stream data and interprets the

movements

into

gestures.

The

gestures are different from one

another and are assigned some
commands. These gestures can act
as input to application which is
projected by the projector. Since,
the projector is aligned downwards
for compactness; therefore images
would be formed at the users feet
if mirror wasnt used. The mirror
reflects the image formed by the
projector to front. The entire
hardware is fabricated in the form
of a pendent. The entire product
cost around $ 350 and that also
because of projector. It works very
similar like a touch screen phone
with entire world as the screen.

Information

is

sent

to

the

Smartphone for processing

The

downward-facing

projector

projects the output image on to the

mirror
Mirror reflects image on to the
desired surface
Thus, digital information is freed
from its confines and placed in the
physical world
The

SixthSense

prototype

comprises a pocket projector, a

mirror and a camera contained in a
pendant-like,

wearable

device.

Both the projector and the camera

are

connected

to

mobile

computing device in the users

pocket.

The hardware that makes Sixth

visual

The

projector

information

projects
enabling

Sense work is a pendant like

surfaces, walls and physical objects

mobile wearable interface

around us to be used as interfaces;

It has a camera, a mirror and a

projector

and

is

connected

wirelessly to a bluetooth smart

phone that can slip comfortably
into ones pocket

pictures,

tracks users' hand gestures and

physical objects using computervision based techniques.[1] The
software program processes the
video stream data captured by the

The camera recognizes individuals,

images,

while the camera recognizes and

gestures

makes with their hands

one

camera and tracks the locations of

the

colored

markers

(visual

tracking fiducials) at the tips of the

users fingers. The movements and

arrangements of these fiducials are

interpreted into gestures that act as
interaction
projected

instructions
application

for

the

interfaces.

SixthSense supports multi-touch

and multi-user interaction.

APPLICATIONS
Fingers as brush:
The user can draw anything on
paint with the help of his fingers.
This drawing can be 3D also.
Hence, no need to use mouse.
using the fingers the user can
capture photos hence, no need to
carry an additional gizmo. The box
created by the fingers act as frame

Capture photos with fingers:

for capturing photo.

Feed information on people:

news headline and then projects the

relevant video.

Check your Flight Status:

The system will project relevant

information about a person such as
what they do, where they work, and
so on.

Palm is the new dialer:

Just place the ticket in front of the

projector and it checks its status
from the internet.
Clock:

This technology enables the user to

call without using the dialer. The
dialer will be projected on palm
and the user can dial the number
using other hand.

Read Books easily:

Check out the ratings of the Book

the user just needs to make gesture

you are going to buy, it checks the

of clock and the watch will be

ratings from the internet. And

projected on the users hand.

another amazing thing is that it

reads the book for you.

Video Newspapers:
like the video newspapers of Harry
Potter this technology identifies the

Access

anywhere

internet:

Sixth Sense can be programmed to

project

related

videos

onto

newspaper articles you are reading

Get product information:

the users can browse internet on
any surface even on their palm.

Create multimedia reading

experiences:

Sixth

Sense

uses

image

recognition or marker technology

to recognize products we pick up,
then feeds us information on those
product

Get book information:

 Cost effective
Data access directly from machine
in real time
Mind map the idea anywhere
It is an open source

LIMITATIONS
Hardware

limitations

of

the

devices, that we currently carry

The system can project Amazon
ratings on that book, as well as
reviews

and

other

relevant

information

around with us.

Many phones will not allow the
external

camera

feed

to

be

manipulated in real time.

Post processing can occur

CONCLUSION

COST AND LICENSE1

SixthSense prototypes cost approximately

This

$350

the

applications. This can be used as a

computer).the main cost being the micro-

replacement of the 5th senses for

projector. Mistry had announced in Nov

handicapped peoples. This can

2009 that the source code will be released

provide

under Open Source.The open source code

machineries in industry. This will

for the project can be found at SixthSense

have

Google Code and SixthSenseGithub Repo.

different developers just depending

to

build

(not

including

technology

has

easy

different

seamless

control
application

over
for

upon how he imagines and what he

wants.

So,

widespread

ADVANTAGES

considering
applications

its
the

inventor PranavMistry has decided

to make its software open source.

Portable
Supports multi-touch and multi-

This will enable individuals to

user interaction
Connectedness between world and

depending

information

make

their

own
upon

application
needs

and

imagination. As this technology

will emerge may be new devices

and hence forth new markets will
evolve. Some existing devices and
technologies will be discontinued
but one thing is guaranteed it will
write a new chapter in history of
science and technology.