ABOUT

'SixthSense' is a wearable gestural interface that augments the physical world around
us with digital information and lets us use natural hand gestures to interact with that
information.

We've evolved over millions of years to sense the world around us. When we encounter something,
someone or some place, we use our five natural senses to perceive information about it; that
information helps us make decisions and chose the right actions to take. But arguably the most
useful information that can help us make the right decision is not naturally perceivable with our five
senses, namely the data, information and knowledge that mankind has accumulated about
everything and which is increasingly all available online. Although the miniaturization of computing
devices allows us to carry computers in our pockets, keeping us continually connected to the digital
world, there is no link between our digital devices and our interactions with the physical world.
Information is confined traditionally on paper or digitally on a screen. SixthSense bridges this gap,
bringing intangible, digital information out into the tangible world, and allowing us to interact with this
information via natural hand gestures. ‘SixthSense’ frees information from its confines by seamlessly
integrating it with reality, and thus making the entire world your computer.

The SixthSense prototype is comprised of a pocket projector, a mirror and a camera. The hardware
components are coupled in a pendant like mobile wearable device. Both the projector and the
camera are connected to the mobile computing device in the user’s pocket. The projector projects
visual information enabling surfaces, walls and physical objects around us to be used as interfaces;
while the camera recognizes and tracks user's hand gestures and physical objects using computer-
vision based techniques. The software program processes the video stream data captured by the
camera and tracks the locations of the colored markers (visual tracking fiducials) at the tip of the
user’s fingers using simple computer-vision techniques. The movements and arrangements of these
fiducials are interpreted into gestures that act as interaction instructions for the projected application
interfaces. The maximum number of tracked fingers is only constrained by the number of unique
fiducials, thus SixthSense also supports multi-touch and multi-user interaction.

The SixthSense prototype implements several applications that demonstrate the usefulness, viability
and flexibility of the system. The map application lets the user navigate a map displayed on a nearby
surface using hand gestures, similar to gestures supported by Multi-Touch based systems, letting
the user zoom in, zoom out or pan using intuitive hand movements. The drawing application lets the
user draw on any surface by tracking the fingertip movements of the user’s index finger. SixthSense
also recognizes user’s freehand gestures (postures). For example, the SixthSense system
implements a gestural camera that takes photos of the scene the user is looking at by detecting the
‘framing’ gesture. The user can stop by any surface or wall and flick through the photos he/she has
taken. SixthSense also lets the user draw icons or symbols in the air using the movement of the
index finger and recognizes those symbols as interaction instructions. For example, drawing a
magnifying glass symbol takes the user to the map application or drawing an ‘@’ symbol lets the
user check his mail. The SixthSense system also augments physical objects the user is interacting
with by projecting more information about these objects projected on them. For example, a
newspaper can show live video news or dynamic information can be provided on a regular piece of
paper. The gesture of drawing a circle on the user’s wrist projects an analog watch.
The current prototype system costs approximate $350 to build. Instructions on how to make your
own prototype device can be found here (coming soon)

Despite the advances and advantages of computer-aided design tools, the traditional pencil and
paper continue to exist as the most important tools in the early stages of design. The goal of the
project ‘inktuitive’ is to combine the intuitive process of creation that is inherent in paper and
pencil with the power of computing that the digital design tools provide. Inktuitive also extends the
natural work-practice of using physical paper by giving the pen the ability to control the design in
physical three-dimensional space, freeing it from its tie to the paper. The intuition of pen and paper
are still present, but lines are be captured and translated into shapes in the digital world. The
physical paper is augmented with overlaid digital strokes. Furthermore, the platform provides a novel
interaction mechanism for drawing and designing using above the surface pen movements.
‘inktuitive’ is an intuitive physical design workspace that aims to bridge the gap and bring
together the conventional design tools such as paper and pencil with the power and convenience of
the digital tools for design. .

Abstract
Despite the advances and advantages of computer-aided design (CAD) tools such as Autodesk
Revit, ArchiCad and SketchUp, the traditional pencil and paper continue to exist as some of the most
important tools in the process of design. We present ‘Inktuitive’, an intuitive physical design
workspace that aims to bridge the gap and bring together the conventional design tools such as
paper and pencil with the power and convenience of the digital tools for design.
Keywords: Design tools, paper and pencil, 3D drawing, workspace, connecting physical and digital
world.
1 Introduction
The development of computer-aided design (CAD) tools has leveraged the power of the digital world
in the process of design by allowing designers to express their creations in new ways. As opposed to
the conventional method of drawing primitives on multiple 2-dimensional (2D) representations
(views), it is increasingly becoming popular to directly model 3-dimensional (3D) representations of
the actual component objects being used and to specify the relationships between them
parametrically to design an architectural product. Digital tools such as Autodesk Revit, ArchiCad and
SketchUp are examples of tools that support this new paradigm also known as Building Information
Modelling (BIM) [BIM]. However, many architects and designers still prefer to use physical tools such
as paper and pen to articulate their ideas especially at early stages of the design, where it is critical
that your hand motion have direct and immediate effect on the object being designed. As the new
design paradigm proliferates, challenges in adopting such affordances of physical design tools into
the digital realm will become of increased importance. In addition, as designers start thinking in
terms of 3D objects rather than in 2D views of objects, a new set of interactions that augments the
conventional use of pen on paper needs to be sought. Instead of replacing the traditional tools,
paper and pencil, it will be more fruitful to merge the advantages and functionalities of digital design
tools with traditional work practice of using paper and pen, hence connecting the physical and digital
experiences.
2 Inktuitive - A physical design workspace
Inktuitive is an intuitive physical design workspace where designers can create and manipulate
digital models and representations of ideas in a more direct and natural way. The goal of Inktuitive is
to combine the intuitive process of creation that is inherent in paper and pencil with the power of
computing that the modern digital design tools provide. Inktuitive also extends the natural work-
practice of using physical paper by giving the pen the ability to control the design in physical 3D
space, freeing it from its tie to the paper. The following use scenario outlines major features of the
‘Inktuitive’ platform.
Figure 1 presents the working prototype of ‘Inktuitive’ system [Mistry and Sekiya. 2008]. The
‘Inktuitive’ system provides a physical workspace environment where digital content is projected
from below onto the paper placed over the frosted-glass surface of the desk. Pen movement on the
surface of the paper is tracked by ultra-sonic digital-pen hardware. Two stationary sensors receive
ultra-sonic waves that are emitted by the
Figure 1. ‘Inktuitive’ prototype.
transmitter placed at the tip of the pen. The device measures the location of the pen tip on the paper
using the calculation of receiving time of the waves received by the two stationary receivers. Further,
the pen tip is also augmented with infra-red (IR) LED array for tracking the 3D coordinates {X,Y,Z} of
the pen above the paper surface using stereo vision mechanism. Two USB cameras with IR-pass
filters mounted above the workspace detect the IR light emitted by the IR-LEDs which are used to
calculate the {X,Y,Z} coordinates of the pen tip via triangulation mechanism. The projector, ultra-
sonic pen hardware, cameras and the navigational knob that lets user navigate through 3D space
are all connected to a computer. A software program processes the data collected by the digital-pen
hardware and two cameras. Image processing of the two camera inputs provides the system with
absolute location of the pen tip (IR LEDs) in 3D space above the paper surface while the ultra-sonic
components help capturing precise user strokes on the paper surface. The software program uses
these two streams of user hand motion in augmenting the paper on the desk with projected digital
information. A vertical display screen provides the user with 3D view of the created objects.
3 Conclusion
In this paper, we present ‘Inktuitive’, an intuitive physical design workspace which brings the power
of computing and digital design tools to the traditional work practice for designing using paper and
pencil. Furthermore, the platform provides a novel interaction mechanism for drawing and designing
using above-the-surface pen movements in 3D.
References
BIM - BUILDING INFORMATION MODELING.
http://en.wikipedia.org/wiki/Building_Information_Modeling
As accessed on 15th Dec, 2008.
MISTRY, P. AND SEKIYA, K. 2008. Inktuitive: An Intuitive Physical Design Workspace. In
Proceedings of 4th International Conference on Intelligent Environments (IE08). Seattle, USA

An Invisible Computer Mouse – Yet Another Invention by Pranav Mistry

Pranav Mistry, who earlier had made headlines for his invention Sixth Sense and even received
Popular Science 2009 Invention Award for it, has now invented yet another similar device and this
time its invisible – A mouse and amusingly it costs just 20$ to build its prototype.

The perpetual changes in computer technology & web has seen many evolutions, right from large
room size CPUs to miroprogrammed slim netbooks, heavy bulky monitors to thin LCDs, few MBs
capacity hard disks to trillion capacity HDs but in all these what remained nearly unchanged and un-
evolved is mouse – moving it around to help us interact computer.

Mouseless is an invisible computer mouse project done in in MIT Fluid Interfaces Group headed by
Pranav Mistry, this invisible mouse provides the familiarity of interaction of a physical mouse without
actually needing a real hardware mouse, hence removes the requirement of having a physical
mouse altogether but still provides the intuitive interaction of a physical mouse that everyone is
familiar with.
Mouseless consists of – (1) an Infrared (IR) laser beam, (2) an Infrared camera and both embedded
in the computer itself. The laser beam module is modified with a line cap and placed such that it
creates a plane of IR laser just above the surface the computer sits on. The user cups their hand, as
if a physical mouse was present underneath, and the laser beam lights up the hand which is in
contact with the surface. The IR camera detects those bright IR blobs using computer vision. The
change in the position and arrangements of these blobs are interpreted as mouse cursor movement
and mouse clicks. As the user moves their hand the cursor on screen moves accordingly. When the
user taps their index finger, the size of the blob changes and the camera recognizes the intended
mouse click.

INTRODUCTION
³Sixth Sense is a wearable gestural interface device that

augments the physical world with digital information and lets people use natural

hand gestures to interact with that information. ´

It was developed by PRANAV MISTRY,

below and start sharing.

Link account

Readcast this Document

Readcast Complete!
Click 'send' to Readcast!
edit preferences

Set your preferences for next time...Choose 'auto' to readcast without being
prompted.
Top of Form

Sowjanya Sri...

Sowjanya Srinivasan

Link account

Advanced Cancel

Bottom of Form
Top of Form
 e8a0bc300d6b5e

Add a Comment

Submit

View comments

1 document_comme 4gen

Bottom of Form

Share & Embed

Add to Collections

Download this Document for Free

Auto-hide: off
1
Outline
Introduction

 C o m p o n e n t s

 T e c h n o l o g y B e h i n d S i x t h S e n s e .

 H o w I t W o r k s ?
 A p p l i c a t i o n s

 Cost

 Advantages & Disadvantages

 C o n c l u s i o n A n d F u t u r e
2

³Sixth Sense is a wearable gestural interface device that

augments the physical world with digital information and lets people use natural

hand gestures to interact with that information. ´

It was developed by PRANAV MISTRY,
3

4

First Page

Previous Page

Next Page

/ 32
Zoom Out

Zoom In

Fullscreen

Exit Fullscreen

Select View Mode

View Mode
BookSlideshowScroll

Search com

Readcast

Add a Comment

Embed & Share

Reading should be social! Post a message on your social networks to let others
know what you're reading. Select the sites below and start sharing.

Link account
Readcast this Document

Readcast Complete!
Click 'send' to Readcast!
edit preferences

Set your preferences for next time...Choose 'auto' to readcast without being
prompted.
Top of Form

Sowjanya Sri...

Sowjanya Srinivasan

Link account

Advanced Cancel

Bottom of Form
 Top of Form
e8a0bc300d6b5e

Add a Comment

Submit

View comments

1 document_comme 4gen

Bottom of Form

Share & Embed

Add to Collections

Download this Document for Free

Auto-hide: off
1
Outline
Introduction

 C o m p o n e n t s

 T e c h n o l o g y B e h i n d S i x t h S e n s e .

 H o w I t W o r k s ?
 A p p l i c a t i o n s

 Cost

 Advantages & Disadvantages

 C o n c l u s i o n A n d F u t u r e
2

³Sixth Sense is a wearable gestural interface device that

augments the physical world with digital information and lets people use natural

hand gestures to interact with that information. ´

It was developed by PRANAV MISTRY,
3

4
 A pocket projector,

 A mirror,

 A camera.
 Mobile component

 Colored markers
5


The Projector:
projects visual information, enabling surfaces, walls and
physical objects around the wearer to be used as interfaces;
 The Camera And Hands: recognizes and tracks the user's
hand gestures and physical objects using
computer-vision based techniques.
 The Software Program: processes the video stream data
took by the camera and tracks the locations of the colored markers (visual
tracking of colored fingers) at the tip of the user¶s fingers using simple computer-
vision techniques.
 The maximum number of tracked fingers is only
constrained by the number of unique fiducials, thus Sixth
Sense also supports multi-touch and multi-user interaction

Technology behind sixth

sense
 Hardware Setup.

 Software Setup.

 Events & Gestures.

 System Architecture

Hardware setup
 MOBILE
 Nokia n95 smart phone is used (running symbianO.S s60
edition).
 It has multitasking capability.
 Built-in camera provides execution of both:
 Gesture tracking engine.
 Gesture enabled application

PROJECTOR
 Pocket projector Pk101 from Optoma is used.

 It augments near by surfaces.

 It is a LED based projector.

 Suitable for mobile usa

Software setup
 Applications are implemented using
JAVA 2MICRO
edition.
 Computer vision library is written in symbian c++(used in
Gesture tracking).
 The software for the sixth sense prototype is developed on a
Microsoft Windows platform using C#, WPF and open CV

HOW S/W WORKS

The software works on the basis of
computer vision.
A small camera acting as an eye, connecting
us to the world of digital information.
Processing is happening in the mobile phone,
and basically works on computer vision
algorithms

THE SOFTWARE RECOGNIZES THREE

KINDS OF GESTURES:

MULTITOUCH:like the ones we see in the iphone ± where we
touch
the screen and make the map move by pinching and
dragging.
FREE HAND: like when you take a picture or a namaste gesture to
start the projection on the wall.

ICONIC: drawing an icon in the air. Like, Whenever we draw
a star, show us the weather details. When we draw a
magnifying glass, show us the map
Events & Gestures
 Events & Gestures are detected using colored markers
attached to fingers.
The user can zoom in or out by moving his hands/fingers
farther or nearer to each other, respectively
The user can draw on any surfaces using the movement of
the index finger used as a pen.
System also supports freehand gestures (postures). one
example is to touch both the index fingers with the opposing
thumbs, forming a rectangle or framing gesture
This gesture activates the photo taking application.
Another example of such gestures is the µNamaste¶ posture
that lets the user navigate to the home screen.
 System lets the user draw icons or symbols in the air using
the movement of the index finger
16
TECHNIQUE BEHIND
The hardware that makes Sixth Sense work is a pendant like
mobile wearable interface
 It has a camera, a mirror and a projector and is connected
wirelesslyto aBluetooth smart phone that can slip comfortably
into one¶s pocket
 The camerarecognizesindividuals, images, pictures, gestures
one makes with their hands
 Information is sent to the Smartphone forprocessing
 The downward-facing projectorprojects the output image on to
the mirror
 Mirrorreflectsimage on to the desired surface
 Thus, digital information isfreed from its confinesand placed
in the physical world

COST
T
he current prototype system costs
approximately$350 to build, mainly due to the micro-
projector. The software may be available for free on the
model of open and editable freeware

ADVANTAGES

Portable

 Inexpensive

 Multi-sensory

 Connectedness between the world and

information
 It is an open source
 Data access directly from machine in real time

LIMITATIONS

Software does support the ability to use real time

video streams in order to produce augmented reality.
 Hardware limitations of the devices, that we
currently carry around with us.
For example many phones will not allow the
external camera feed to be manipulated in real time.
 Post
processing
can
occur
however
FUTURE OF 6TH SENSE
 Interactive Advertisements.
 3d visualizations.
 Solar batteries via small solar panel.
 Camera can act as a third eye for the blind
person
³According to researchers, after 10years we
will be here with the ultimate sixth-sense
brain implant.´
CONCLUSION
Sixth Sense recognizes the objects around us,
displaying information automatically and letting
us to access it in any way we need.
 The Sixth Sense prototype implements several
applications that demonstrate the usefulness,
viability and flexibility of the system.
 Allowing us to interact with this information via
natural hand gestures.
 The potential of becoming the ultimate
"transparent" user interface for accessing
information about everything around