CHI 2009 ~ Techniques for Mobile Interaction
Graspables: Grasp-Recognition as a User Interface
Brandon Taylor
MIT Media Lab
20 Ames St. E15-346
Cambridge, MA 02139 USA
bttaylor@media.mit.edu
ABSTRACT
The Graspables project is an exploration of how measuring
the way people hold and manipulate objects can be used as
a user interface. As computational ability continues to be
implemented in more and more objects and devices, new
interaction methods need to be developed. The Graspables
System is embodied by a physical set of sensors combined
with pattern recognition software that can determine how
users hold a device. The Graspables System has been
implemented in two prototypes, the Bar of Soap and the
Ball of Soap. Applications developed for these prototypes
demonstrate the effectiveness of grasp-recognition as an
interface in multiple scenarios.
Author Keywords
Grasp, User Interface.
ACM Classification Keywords
H.1.2: Models and Principles: User/Machine Systems.
H.5.2: Information Interfaces and Presentation: User
Interfaces. K.8.0: Personal Computing: General. J.7:
Computer Applications: Computers in Other Systems.
INTRODUCTION
The Graspables are devices developed to explore how
measuring the way people grasp objects can enhance user
interfaces. This paper will first attempt to explain the
rationale and inspiration behind using grasp-recognition as
an interface. It will provide a detailed description of how
the sensors and software were implemented in the two
Graspables prototypes. Next, follows a discussion of the
applications that have been developed to explore and
demonstrate the capabilities of grasp-recognition. Lastly,
there will be a discussion of our experiences with grasp-
recognition, including methods for evaluation and
improvement.
The origin of the Graspables can be traced back to a high
level discussion of ways to improve multi-function
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise,
or republish, to post on servers or to redistribute to lists, requires prior
specific permission and/or a fee.
CHI 2009, April 4–9, 2009, Boston, MA, USA.
Copyright 2009 ACM 978-1-60558-246-7/09/04…$5.00.
917
April 7th, 2009 ~ Boston, MA, USA
V. Michael Bove, Jr.
MIT Media Lab
20 Ames St. E15-368B
Cambridge, MA 02139 USA
vmb@media.mit.edu
handheld devices. It was suggested that an ideal multi-
function device would need to be capable of two things: it
would need to automatically infer what users want to do
with it and it would need to be able to alter its affordances
accordingly. When it wasn’t being used, the device would
simply appear to be an undifferentiated block, like a bar of
soap.
While the Graspables may not completely fulfill this vision,
the idea of creating devices that implicitly understand users’
intentions, without the need for menus and direct
commands, was the launching point for the project. As the
project evolved, emphasis shifted away from multi-function
handhelds to exploring how basic manipulations of objects
can contain useful information. In other words, what can
you learn from the way people hold an object? Can you
distinguish whether a user wants to make a phone call or
just look up a contact by the way they hold their phone?
Can a golf club predict a slice if it is gripped improperly?
In pursuing these questions, the Graspables were
constrained by the desire to have a system that could be
realistically implemented in existing objects and devices.
This led us to shy away from approaches that require
elaborate sensing environments or expensive input devices.
The hope was that the right combinations of sensors and
software could give objects an enhanced understanding of
their users’ actions without limiting portability or
affordability.
Another key aspect of the research was the focus placed on
objects themselves. Instead of focusing on just creating a
new interface method or a specific type of controller, we
were very interested in understanding and exploring how
people interact with a variety of different objects. Our view
was that understanding how people grasp and interact with
a coffee cup is potentially just as valuable as how they
interact with electronics. Thus, we wanted a system that
could be implemented into arbitrary geometries.
Background
Nearly twenty years ago, Mark Weiser coined the term
Ubiquitous Computing to describe the idea of a vast
network of computing devices interacting unobtrusively to
enhance productivity. While the proliferation and
dispersion of computational power has certainly occurred, it
has not yet “vanish[ed] into the background” [17].
CHI 2009 ~ Techniques for Mobile Interaction
Projects like the Graspables fit into the realm of Ubiquitous
Computing by trying to expand the ways in which
computers are controlled. By developing grasp-recognition
as a user interface, it is hoped that users can be presented
with a more natural method of interacting with devices.
Instead of seeing a device and trying to imagine how its
menu system and buttons are mapped, grasp-recognition
can leverage users’ intuitions about how devices should be
used for certain functions.
Studies have been performed demonstrating how certain
computerized tasks can be more easily accomplished when
properly modeled by physical devices [5]. Early work on
the concept of Graspable User Interfaces suggested that by
“facilitating 2-handed interactions, spatial caching, and
parallel position and orientation control” physical
computing could provide a richer interface than virtual,
graphics-based interfaces [4].
A grasp-recognition based interface would, by virtue of its
nature, capitalize on these advantages. Rather than creating
controls through arbitrary key-mappings, the physical
nature of the Graspables provides suggestive physical
affordances and passive haptic feedback by approximating
and representing real-world objects.
Motivation
Portable device interfaces provide a distinct challenge for
designers. For more complex portable systems, there is a
natural desire to mimic the operational semantics of the
computer as much as possible. People are accustomed to
the window metaphor of most desktop computer GUIs, so it
makes sense to leverage this knowledge to some extent.
A common approach in portable devices is to imitate the
clicking and dragging functions of a mouse with a
touchscreen. Full keyboards are often implemented either
in the form of physical buttons or virtual ones. Both
approaches have drawbacks. Physical buttons are
necessarily small and always present, even when an
application only needs a subset of the keys. Virtual buttons
on the other hand, provide no tactile feedback, which can
render them unusable to certain groups of users.
These issues have led researchers to explore other
interaction methods that may end up being more
appropriate for handheld devices. A common example is
the use of accelerometers in many cameras and phones for
switching between portrait and landscape views. Another
approach is to capitalize on the inherent mobility of
handheld devices by exploring gestures as an interaction
method. Studies have explored using gestures for things
such as the detection of common usage modes [9] to the
mapping of functions to relative body positions [1]. While it
is hard to predict what new interfaces will catch on,
successes like the iPhone’s Multi-Touch display provide
encouragement for continuing research.
In implementing the Graspables system into the Bar of
Soap and the Ball of Soap, we were interested in how the
918
April 7th, 2009 ~ Boston, MA, USA
devices’ geometries impact what objects they can easily
represent.
The most common input devices, such as mice, keyboards
and even video game controllers, generally sacrifice
representation in favor of more robust, general controls.
Over time, these systems develop semantics of their own
(think how similar most video game controllers are or how
people expect to be able to click icons in graphical
interfaces) and people hardly even think about the control
metaphors. However, there are exceptions.
Tablet and stylus systems exist to better bridge the gap
between writing or drawing and computers. Video games
can use specialized peripherals such as steering wheels or
guns. These examples highlight the importance of objects
for some tasks. While there is likely no way to completely
avoid the tradeoff between robust and representative
controls, it is certainly worth exploring how new interfaces
can create more literal interactions and of what value these
may be.
RELATED WORK
The Huggable is a robotic Teddy Bear being designed by
the Personal Robots group at the MIT Media Lab to provide
therapeutic interactions similar to those of companion
animals. The Huggable is being developed with the goal of
“properly detecting the affective content of touch” [15].
Towards this end, the Huggable is thus equipped with an
array of sensors that detect the proximity of a human hand,
measure the force of contact and track changes in
temperature. The data from these sensors is then processed
to distinguish interactions such as tickling, poking or
slapping [14].
From a technical perspective, the goals of the Huggable are
very similar to those of the Graspables System. Both seek
to identify and understand the ways users manipulate an
object. In many ways, the Huggable could be viewed as a
sophisticated example of a grasp-recognition system. That
said, there are obvious differences between the Graspables
System described in this paper and the hardware/software
system of the Huggable. The sensing hardware of the
Huggable, for example, relies on dense arrays of Quantum
Tunneling Composite (QTC) force sensors and broad
electric field sensors for touch sensing, whereas the
Graspables are implemented with a dense set of capacitive
sensors. Additionally, whereas the Huggable is intimately
connected to the Teddy Bear form factor, our work
demonstrates a system that can readily be adapted into
various geometries for different uses.
The Tango is a whole-hand interface designed by the
Multisensory Computation Laboratory at Rutgers for the
manipulation of virtual 3D objects [11]. The device is a
hand-sized spherical object with a 3-axis accelerometer and
an 8x32 capacitive sensing grid housed in a compressible
dielectric material. The Tango is calibrated to detect
CHI 2009 ~ Techniques for Mobile Interaction April 7th, 2009 ~ Boston, MA, USA

variations in pressure from which a simplified hand model devices. It also served as a test bed for the Graspables
can be estimated. System’s hardware.
The Tango uses spherical harmonics to create a rotationally The Bar of Soap prototype, shown in Figure 1, is a
invariant map of pressures [8]. These pressure maps can 11.5x7.6x3.3 cm rectangular box containing a 3-axis
then be reduced using principal component analysis and accelerometer and 72 capacitive sensors. The capacitive
classified using K-nearest neighbors. A 3D virtual sensors are controlled by three Qprox QT60248 chips,
environment in which the Tango was used to manipulate which treat each one as a discrete, binary sensor. An Atmel
virtual objects was also developed. Atmega644 microcontroller in the device samples these
sensors and can communicate results to a PC via Bluetooth.
While the Tango clearly shares certain objectives with the
Low-power cholesteric LCD screens on the two largest
Graspables, there are significant differences in their
faces can provide user feedback. Transparent capacitive
respective implementations. First, the grid structure of the
sensors were developed and placed over both screens. This
capacitive sensors and the classification software of the
allowed the display surfaces to also function as sensing
Tango would not directly translate to other device
surfaces, which in turn allowed the Bar of Soap to better
geometries, severely limiting the number of objects it could
emulate functional devices with interactive touchscreens.
represent. Also, since the Tango is actually attempting to
Having screens on both sides preserved the symmetry of the
infer general hand poses it requires additional constraints,
device and allowed the Bar of Soap to function as a generic
such as single hand use. In the end, while the sensing
rectangular sensing device with two customizable faces.
techniques and software analysis provide interesting
references, the goals of the Tango require a significantly The transparent sensors were created by placing thin film
different approach than those of the Graspables. coated with Indium Tin Oxide (ITO) on opposite sides of a
clear piece of acrylic. ITO is a transparent conductive
When development began on the first version of the Bar of
material that fills the role of the interdigitated copper traces
Soap, a similar study was being conducted by the Samsung
of the other capacitive sensors. We tested ITO sensors in a
Advanced Institute of Technology (SAIT) [2,7]. After
variety of patterns and sizes before choosing strips of
receiving encouraging results from an initial study in which
approximately 5mm width. This design has the advantage
painted gloves where used to create image maps of grasp
of being relatively simple to construct while providing
patterns, a prototype device was built for real-time grasp
sensitivity to finger touches comparable to the copper traces
detection. The SAIT device contained a 3-axis
used elsewhere. Settings in the QT60248 chip were able to
accelerometer and 64 capacitive sensors. A user study was
amplify the responses of sensors located further along the
performed to try and classify 8 different use modes with the
ITO to compensate for its resistance. Response was further
device.
improved by lining the edges of the device with a grounded
The results from the SAIT study match up well with those copper strip.
of the initial Bar of Soap study [16], correctly classifying
75% to 90% of grasps across multiple users. The SAIT
device uses non-binary capacitive sensors, a different
sensor layout on a device of different physical dimensions,
a unique set of use modes and different classification
techniques from the Bar of Soap. In addition to these
differences, our research goes beyond static grip
recognition to explore how changing grasps and gestures
can enhance interactions. We also look beyond common
handheld electronics to see how grasp recognition can
impact interactions with physical objects and virtual
environments.

DESIGN
The Graspables System is a hardware and software platform
capable of detecting how a user is manipulating a device.
The system needs to be flexible enough to accommodate
distinct sensor layouts for objects with different physical Figure 1. The Bar of Soap
geometries. It is also important that the system be able to
process and transmit data in real-time.

The Bar of Soap The Ball of Soap

The Bar of Soap was designed to explore how grasp- As the Bar of Soap evolved from an exploration of ways to
recognition could be of use in a variety of modern handheld improve multi-function handhelds into a more general
platform to explore grasp-recognition, we began to consider

919
CHI 2009 ~ Techniques for Mobile Interaction April 7th, 2009 ~ Boston, MA, USA

the limitations of its physical form. While a small section will discuss the applications that have been
rectangular box provides an adequate representation of developed for the Graspables implementations.
many handheld electronics, it has inherent limitations.
Multi-Function Handheld
In order to explore interactions with different physical As handheld electronics have become more powerful, it has
forms, the Ball of Soap was developed. Since a truly become possible to add more functionality to individual
spherical object would create difficulties in laying out the devices. Whereas phones used to be just phones, most
capacitive sensors used in the Bar of Soap, we built the Ball modern cell phones now take pictures, play music, browse
of Soap as a small rhombicosidodecahedron. This 62-sided the internet and more. One difficulty for designers has been
Archimedean solid provides flat surfaces near the quarter how to layout the appropriate affordances for these various
square inch size and surface density of the Bar of Soap’s functions given the small size of the devices. We noticed
sensors when the overall diameter approaches three inches. that many of the functions now provided by multi-function
handhelds have been adopted from devices that people are
accustomed to holding and operating in distinct ways. The
Bar of Soap as a multi-function handheld application is
designed to capitalize on people’s previous experiences by
inferring how they want to use the device based on how
they are holding it.
This application provides a very self-contained
demonstration of the potentials of grasp-recognition. The
device passively senses its orientation and the position of a
user’s hands, and then displays an interface corresponding
to the most likely functionality mode. For demonstration
purposes the sampling and classification routine is
performed every three seconds, but it could easily be
triggered by some sort of gesture. Figure 3 shows the Bar
of Soap’s multi-function handheld mode switching
application in use.
Figure 2. The unassembled Ball of Soap
The surface structure of the Ball of Soap prevented the
simple printing of interdigitated copper traces used as
sensors on the Bar of Soap. Instead, adhesive copper pads
were cut and attached to the faces with wires running to
circuit boards inside the ball. We explored using a variety
of trace shapes for each of the different face geometries, but
found that only using the smallest, triangle arrangement
provided a more consistent response across the capacitive
sensors.
As can be seen in Figure 2, the small
rhombicosidodecahedron shape also allowed the Ball of
Soap to be separated into three sections for easier assembly.
The two end pieces are identical and each contains a Qprox
chip that controls the 23 capacitive sensors on its surface.
The center piece has 16 faces, 15 of which have capacitive Figure 3. The Bar of Soap being held in a ‘phone’ grasp
sensors and one that houses the power button and
Currently, the application switches between five
programming interface. Inside the Ball, attached to the
functionality modes: camera, gamepad, phone, personal
center piece is the main circuit board with the
data assistant (PDA), and remote control. These modes
microcontroller, accelerometer, battery and Bluetooth chip.
were chosen to represent common, pre-existing handheld
A grounding bracelet can be attached to improve sensor
devices that were assumed to have relatively distinct
response.
interaction methods. The data used to train the Bar of Soap
to distinguish the different modes was gathered from users
APPLICATIONS
who were given a functionality mode (camera, gamepad,
In the process of developing the Graspables System,
phone, PDA or remote) and asked to hold the device
applications were always a consideration. While it is hoped
however they felt was appropriate given the mode.
that the grasp-recognition technique is general enough to be
applied to many other scenarios, specific objectives
strongly influenced the design of the prototypes. This

920
CHI 2009 ~ Techniques for Mobile Interaction
Rubik’s Cube
Increasingly, gesture recognition using accelerometers and
other sensors is being incorporated into handheld device
interfaces [13,6]. To demonstrate that the Graspables are
not just limited to distinguishing a small number of static
grasps we designed a virtual Rubik’s cube application. This
application makes use of gesture recognition to control a
virtual object using a tangible real-world interface.
Figure 4. The Bar of Soap controlling a virtual Rubik’s cube
This application, shown in Figure 4, exists as a Matlab
script that streams raw sensor data from the Bar of Soap via
Bluetooth. A graphical version of a Rubik’s cube is
displayed on screen and is mapped to the Bar of Soap’s
orientation as determined by the accelerometer data.
Rotating ends of the virtual Rubik’s cube is accomplished
by sliding a finger across the sensors that are most closely
mapped to that end on the Bar of Soap.
The Rubik’s cube application demonstrates how grasp-
recognition can be used to provide a more tangible interface
to virtual environments. By mapping the orientation of the
virtual object to the real world device, changing viewpoints
is incredibly intuitive. Selecting different virtual objects
can be as simple as picking up a different grasp-recognition
equipped device. Lastly, this application shows how the
sensors used for distinguishing static grasps can also be
used to recognize dynamic manipulations and gestures.
Pitch Selection
In baseball, subtle differences in the way the pitcher grips
the ball have a profound effect on the outcome of the pitch.
Thrown correctly, a slider can become nearly unhittable as
it darts away from the batter at the last second. However,
the slightest error in its delivery can see the pitch landing
hundreds of feet in the wrong direction.
Given the importance of fine finger manipulations on
pitching, this seems an ideal scenario for the Graspables
System. A baseball that can detect how it’s being held
could be extremely useful in training players to throw
921
April 7th, 2009 ~ Boston, MA, USA
certain pitches or diagnose potential delivery issues. On the
other hand, baseball video games could use such a device to
provide a method of pitch selection that is more realistic
and engaging than pushing a button on a controller.
While individual grasps may vary slightly from pitcher to
pitcher, in general the outcome (pitch type) is mapped to a
certain grip relative to the baseball’s seams. Given these
previously defined grasps, training data was acquired by
having a single user appropriately hold the Ball of Soap for
a set of pitch types. For this application, the skin of a
baseball was wrapped around the Ball of Soap. Due to the
four-way symmetry of a baseball, each individual pitch type
can be thrown with the ball in four unique absolute
orientations. Since the sensors on the Ball of Soap are
aligned to the absolute orientation of the Ball rather than
relative to the baseball’s seams, training data was collected
for each pitch being held in the four different absolute
orientations.
Figure 5. The Ball of Soap with a baseball cover as a pitch
selector
The pitch selection application, shown in Figure 5, operates
as a Matlab script. Upon activation, Matlab opens a serial
port for communication with the Ball of Soap and a screen
presents the user with a pitcher ready to throw. The user
then grips the Ball appropriately to throw a fastball, a
curveball, or a splitter and makes a throwing gesture. The
acceleration of the Ball triggers the classification routine,
which in turn triggers an animation taken from Nintendo’s
Mario Super Sluggers videogame to display the selected
pitch.
DATA ANALYSIS
In addition to building the Graspables hardware, we also
had to develop methods for making sense of the data that it
produced. A significant amount of work was put into
creating appropriate data features and classification
methods for each of the applications.
CHI 2009 ~ Techniques for Mobile Interaction
Multi-Function Handheld Data
Each time the Bar of Soap is sampled, data are gathered
from 75 independent sensors. In order to create a more
manageable feature space, we needed a way to process
these data before applying classification techniques.
While we were testing the first version of the Bar of Soap
(which had no capacitive sensors on one of the largest
faces) we developed a Matlab script to visualize the data.
This visualization tool, shown in Figure 6, had checkboxes
laid out to represent the capacitive buttons, a 3D plot of the
accelerometer readings and icon display that could either
present a sample’s class label or the result of a classifier.
Figure 6. A visualization of the data of a ‘phone’ grasp. This
visualization layout was designed for the Bar of Soap V1,
which did not have sensors on the front face.
Using this visualization tool, we noticed that the capacitive
sensors had a strong tendency to be activated in regional
groups. Additionally it became clear that treating the
sensors as having an absolute location reduced the accuracy
of classifiers. To account for these facts, we began
reducing the dataset by counting the number of activated
capacitive sensors along each face (splitting the largest two
faces into two halves) and orienting these groups according
to the accelerometer readings. This method outperforms
datasets reduced using Principal Component Analysis and
Fischer Linear Discriminants and automatically adjusts the
device when it is rotated or flipped.
Rubik’s Cube Data
In order to control the virtual Rubik’s cube, data is rapidly
sampled via Bluetooth. The accelerometer data is
smoothed and mapped to the orientation of the virtual cube.
Additionally, sliding a finger across different faces of the
Bar of Soap triggers a rotation of the corresponding part of
the Rubik’s cube. While it might be possible to detect
sliding gestures in a simpler manner, it was desired that the
method be generalizable to more complex gestures. Thus,
we chose to implement hidden Markov models [12] to
detect the sliding gestures.
In order to train the Hidden Markov Models, data was
collected and labeled as a single user slid his finger over the
922
April 7th, 2009 ~ Boston, MA, USA
face of the Bar of Soap. The sliding gesture was recorded
in both directions along each edge of the Bar of Soap and
along the outermost rows and columns of sensors on the
largest two faces. While the sliding gesture was being
recorded on a specific side, no particular attention was
placed on how the user was holding the Bar of Soap. This
insured that data about manipulations that were not sliding
gestures was also recorded.
The sliding gestures were modeled using a left-right Hidden
Markov Model. The states represented the position of the
finger activating either a single capacitive sensor or two as
it slides between them. The number of states in the model
depended upon the number of capacitive sensors on the side
that was being modeled. In addition to the left-right HMM’s
modeling the sliding gestures, ergodic models exist to
model general, non-sliding interactions.
These models are trained using the raw sensor data as
observation sequences. The sliding models are trained
using the corresponding sliding gestures. The general
ergodic model is trained using the data from sliding
gestures that do not correspond to the modeled area. For
example, the data set that represents a sliding gesture along
the short edge of the Bar of Soap is used to train the ergodic
model of the long side.
The time sequences of activated capacitive sensors are then
broken up into observation sequences corresponding to the
different gesture models. The trained models are used to
calculate the probability of observing such a sequence. If a
sequence has a higher probability of being observed given
one of the sliding gesture models, a sliding event is
triggered.
If the sliding gesture is performed on either of the largest
faces, the rotation will occur in the direction of the sliding
gesture and on the corresponding end of the virtual cube.
To rotate an end along the axis that is mapped
perpendicular to the front and back face, the user simply
places their hand over one of the large faces and slides their
finger along one of the edge faces in the direction of desired
rotation. The virtual cube will interpret the covered side as
the stationary side of the cube and rotate the opposite side.
Pitch Selection Data
Like with the Bar of Soap as a multi-function handheld, the
capacitive sensors are grouped in order to reduce the size of
the feature space. Instead of grouping by sides, as in the
Bar of Soap, capacitive buttons are grouped around the 12
pentagonal faces on the small rhombicosidodecahedron.
Each pentagonal face is surrounded by five square faces,
shared with a single other pentagonal face, and five triangle
faces, shared with two other pentagonal faces. These faces
are weighted inversely to the number of groups they
inhabit: an active pentagonal face receiving a weight of 6, a
square 3, and a triangle 2. Thus each of the twelve groups
creates a feature with a value between 0 and 31 depending
on the number of activated faces.
CHI 2009 ~ Techniques for Mobile Interaction
The pitch recognition application for the Ball of Soap
operates very similarly to the multi-function mode
switching application for the Bar of Soap. The capacitive
sensors are grouped and processed as discussed above, then
Bayesian discriminants are calculated. Each of the four
pitch orientations is treated as a separate class. Thus, for N
pitch types, 4xN determinants are calculated.
The classification routine is triggered by a throwing
gesture. Thus, though the sensors are continually sampled,
the discriminant functions are not calculated until the
accelerometer values surpass a threshold. When the
threshold is crossed, the discriminants are calculated and
the most likely pitch is selected.
EVALUATION
One of the greatest difficulties our research faced was
finding an effective method for evaluating the Graspables.
On a general level, this is a difficulty facing any new
method of interface. If it really is novel, then task
completion comparisons with existing interfaces overlook
new possibilities offered by the interface. On a more
specific level, grasp-recognition is difficult to evaluate
because of the lack of any ground truth. Even in situations
where a grasp is universally recognized, such as the
relations of fingers to baseball seams when throwing a four-
seam fastball, the exact grip will vary from person to person
based on hand size. The problem becomes even more
challenging when dealing with less defined grasps. After
all, who’s to decide what the proper way to grasp a phone
is?
This isn’t to say that no evaluation can be done. Obviously,
for grasp-recognition to be of any value, grasps must at
least have enough meaning to be remembered and repeated
by individuals. We conducted a two-part user study to
examine first, how reliably our system could recognize
grasps, and second, how grasps associated with various
devices vary across a population.
This study was conducted using the first version of the Bar
of Soap’s Multi-Function Handheld application. The
procedure was the same for both parts of the study, the only
difference being that the first part was conducted using a
single user, whereas the second part was conducted with a
total of thirteen individuals.
For this study, users were seated with the Bar of Soap in
front of them on a table. They were told that they would be
given a specific functionality mode (camera, gamepad,
PDA, phone or remote control) and that they should then
pick up the device and interact with it however they saw fit
until instructed to set it back down.
After giving these instructions, we would begin recording
data from the Bar of Soap and then verbally indicate what
functionality mode the device should be treated as. Once
the user had established a relatively stable pose with the
device, we would label and save the data sample and have
the user place the device back on the table. We then
923
April 7th, 2009 ~ Boston, MA, USA
repeated this process with each user until we had data
samples from each of the five tested functionality modes.
For the single user part we collected 39 sample grasps in
each functionality mode for a total of 395 grasps. For the
multiple users part, each of the 13 users provided three
sample grasps per mode to create a matching data set.
Single User Test Multi-User Test
Templates 82.2% 75.4%
Nnets 92.5% 79%
KNN 95% 75.8%
Parzen 95.4% 72.3%
Bayes 95% 79%
GLD 87.5% 70.3%
Table 1. Recognition rates for different classification
techniques for both single and multiple user datasets.
For this study we tested a wide range of classifiers
including Templates, Neural Networks, Bayesian
Classification, k-Nearest Neighbors, Parzen Windows and
General Linear Discriminants [3]. To test the reliability of
the system, we used the single user dataset. Each classifier
was trained using a randomly selected set of 29 sample
grasps from each mode, then tested on the remaining 10
samples. This process was repeated 10 times using
different training sets. To see how consistently grasps are
associated with devices across multiple users, we trained
the classifiers using the entire single user dataset and tested
them with the multiple user dataset. The recognition rates
for both studies are shown in Table 1.
We were pleased to find that multiple classification
techniques were able to correctly identify a single user’s
grasps with over 90% accuracy. This led us to conclude
that our sensor design was adequate for grasp-recognition.
The relatively high rates of recognition across multiple
users further encouraged us that even without prompting or
demonstration, people do have similar models of how to
grasp things. In the end, we chose to implement the
Bayesian classifier due to its high performance and minimal
computational complexity.
DISCUSSION
While we were generally pleased with the results of our
user study, there are a few caveats that must be kept in
mind. Our study group varied in hand size and handedness,
however, all the participants were fairly homogenous in age
(18 to 36) and familiar with electronic devices.
Additionally, the tested functionality modes were selected
based on the assumption that they would have distinct
grasps associated with them. A large source of error in the
multi-user study was when this assumption failed. For
example, when users held the phone as though they were
CHI 2009 ~ Techniques for Mobile Interaction
looking up a number rather than up to their ear as though
speaking on it, the grasp was often classified as a PDA.
Another problem the Graspables encountered is that the
system tends to show a significant bias regarding hand size.
While this can be overcome by explicitly training the
classifiers for different users or by implementing a learning
algorithm, it would be desirable to have a more universal
response.
There is also the common question of how many different
grasps the system is capable of recognizing. Unfortunately,
we do not have a simple answer. From our experience the
limiting factor is not the sensing system as much as the
application. Theoretically, the binary capacitive sensors on
the Bar of Soap can distinguish 722 different grasps in a
number of different orientations. While the anatomy of the
hand would make that infeasible, the fact is there are not
many applications where shifting a finger a fraction of an
inch creates a meaningful difference in the user’s mind.
The Ball of Soap’s pitch selection is an example where
minor changes in grip could have a meaningful impact.
However, even in that case it would be difficult to draw
distinctions between minor grip changes without
extensively measuring and studying live pitchers and ball
trajectories.
While the limited number of meaningful ways to hold a
rectangular box may make the Graspables System seem like
overkill, we feel that our sensor density has other benefits.
As the Rubik’s cube application demonstrates, a grasp-
recognition system can be used for more than just
recognizing distinct grasps. Virtual environments could
make use of more precise mappings of hand position.
Another possibility is a twister-like finger dexterity game
could make use of far more grasp combinations that would
naturally be used.
Aside from the study used to test the accuracy of the grasp-
recognition, many users have informally interacted with the
Graspables. It is interesting how quickly users respond to
the idea of grasp-recognition. For the Bar of Soap as a
mode switching handheld in particular, users are quick to
adjust how they are holding the device and seem to enjoy
trying to figure out how the grasps have been trained in the
device. Similarly, many users of the Ball of Soap as a pitch
selector begin experimenting with various grasps even
before the trained pitch grips are explained. Whether this
enthusiasm would transfer to real implementations of grasp-
recognition is questionable, but it does seem to indicate that
making better use of people’s senses of touch and
proprioception could provide better interfaces.
FUTURE WORK
The prototypes discussed in this paper only represent a
small fraction of the potential implementations of the
Graspables System. Another implementation that was
discussed and would be worth developing is a stylus
prototype. In the graphics art world alone, pencils,
924
April 7th, 2009 ~ Boston, MA, USA
paintbrushes, erasers and wands could all be represented by
different ways of grasping a stylus.
For more of departure from the work in this paper,
implementing Graspables into existing devices would be
interesting. Using the handheld device mode switching that
was demonstrated by the Bar of Soap in a fully functional
handheld would be worth studying. Questions about when
to trigger the classification algorithms, what error rates
would be acceptable to users, and the general effectiveness
of the natural mode switching would be better explored by
longer studies with fully functional devices.
Applying what has been learned from the existing
applications to other scenarios also has potential. Can the
Graspables System be used as a safety check to ensure that
power tools are being operated properly? What could be
gained by expanding the scale of the system from handheld
objects to whole body-sized arrays?
There is also room to perform further tests to improve the
reliability and robustness of the system. Optimizing sensor
densities could be valuable. Exploring how environmental
factors such as humidity impact the capacitive sensors
could improve system reliability. Exploring additional
inputs such as pressure sensors could be beneficial. Lastly,
the software and classifiers could always benefit from more
training data.
CONCLUSIONS
The Graspables demonstrate how grasp-recognition can
provide a unique and intuitive user interface. We presented
a design rationale, our system design, a variety of
application scenarios, and a discussion of our experiences
with the system. The system we developed can
implemented in multiple geometries to provide a better
representation of different objects in virtual environments.
It can also provide devices with additional awareness of
users’ intentions via their manipulations. As mobile
devices continue to grow in power and popularity, new
interaction methods will need to be developed
accommodate them. We feel that grasp-recognition has the
potential to provide significant enhancement to current
interfaces.
ACKNOWLEDGEMENTS
We thank Jeevan Kalanithi, Daniel Smalley and Matt
Adcock for their work on the classification techniques.
Thanks also to Quinn Smitwick, James Barabas and Ana
Luisa Santos for their assistance. This work was supported
by the CELab, Digital Life, and Things That Think
consortia at the MIT Media Lab.
REFERENCES
1. Ängeslevä, J., Oakley, I. and Hughes, S. Body
Mnemonics: Portable Device Interaction Design
Concept. In Proc. of Info. Vis. (2003).
2. Chang, W., Kim, K.E., Lee, H., Cho, J.K., Soh, B.S.,
Shim, J.H., Yang, G., Cho, S. and Park, J. Recognition
CHI 2009 ~ Techniques for Mobile Interaction
of Grip-Patterns by Using Capacitive Touch Sensors.
IEEE ISIE 2006, 4 (2006), 2936-2941.
3. Duda, R., Hart, P., Stork, D. Pattern Classification,
John Wiley & Sons, Inc., 2001.
4. Fitzmaurice, G.W., Ishii, H. and Buxton, W. Bricks:
Laying the Foundations for Graspable User Interfaces.
CHI 1995: Proc. of the SIGCHI Conf. on Human
Factors in Computing Systems. ACM Press (1995),
442-449.
5. Fitzmaurice, G.W. and Buxton, W. An Empirical
Evaluation of Graspable User Interfaces: Towards
Specialized, Space-Multiplexed Input. CHI 1997: Proc.
of the SIGCHI Conf. on Human Factors in Computing
Systems. ACM Press (1997), 43-50.
6. Harrison, B.L., Fishkin, K.P., Gujar, A., Mochon, C.,
and Want, R. Squeeze Me, Hold Me, Tilt Me! In Proc.
of the SIGCHI Conf. on Human Factors in Computing
Systems. ACM Press (1998), 17-24.
7. Kim, K.E., Chang, W., Cho, S., Shim, J., Lee, H., Park,
J., Lee, Y., Kim, S. Hand Grip Pattern Recognition for
Mobile User Interfaces. In Proc. of AAAI 2006. (2006),
1789-1794.
8. Kry, P.G. and Pai, D.K. Grasp Recognition and
Manipulation with the Tango. Int. Symposium on
Experimental Robotics. (2006), 551-559.
9. Mäntylä, V.M., Mantyjarvi, J., Seppanen, T. and
Tuulari, E. Hand Gesture Recognition of a Mobile
Device User. IEEE Int. Conf. on Multimedia and Expo.
1 (2000), 281-284.
925
April 7th, 2009 ~ Boston, MA, USA
10. MacKenzie, C.L., Iberall, T. The Grasping Hand.
Elsevier, 1994.
11. Pai, D.K., VanDerLoo, E.W., Sadhukhan, S. and Kry,
P.G. The Tango: A Tangible Tangoreceptive Whole-
Hand Human Interface. In Proc. of World Haptics.
(2005), 141-147.
12. Rabiner, L.R. A Tutorial on Hidden Markov Models
and Selected Applications in Speech Recognition. In
Proc. of the IEEE. Vol. 77, no.2, (1989), 257-286.
13. Rekimoto, J. Tilting Operations for Small Screen
Interfaces. In Proc. of 9th ACM Symposium on User
Interface Software and Technology. (1996), 167-168.
14. Stiehl, W.D. and Breazeal, C. Affective Touch for
Robotic Companions. At 1st Int. Conf. on Affective
Computing and Intelligent Interaction. (2005).
15. Stiehl, W.D., Leiberman, J., Breazeal, C., Basel, L.,
Lalla, L., and Wolf, M. Design of Therapeutic Robotic
Companion for Relational, Affective Touch. 2005 IEEE
Int. Workshop of Robots and Human Interactive
Communication. (2005), 408-415.
16. Taylor, B.T. and Bove, V.M. The Bar of Soap: A Grasp
Recognition System Implemented in a Multi-Functional
Handheld Device. CHI 2008: Extended Abstracts on
Human Factors in Computing Systems. (2008), 3459-
3464.
17. Weiser, M. The Computer for the 21st Century.
Scientific American. Vol. 265, no. 3, Sept. 1991, 94-104.