Literature Review

Designing a gesture based touchscreen interface: A

Literature Review
Kent Hawkings
Department of Computer Science
University of Cape Town
kenthawkings@gmail.com

Abstract different types of gestures one can perform using dif-

ferent input modalities offered by most of todays
This paper reviews the various approaches suggested handheld devices. In particular, the user-centered
towards interface design. In particular, gesture based design approach is considered as well as the psycho-
interaction with touchscreen devices is considered. logical and sociological impacts that gestures may
As the most important aspect of an interface is ar- carry. This lack of implementation of extensive ges-
guably its ease of use a focus on Human-Computer In- tural interaction implies that further research in the
teraction (HCI) and Interaction Design (ID) method- field is needed.
ologies is presented. In addition, current approaches
to gesture based interaction are analysed so as to es- Contemporary research into interface design has
tablish the potential offered. To this end it is sug- largely been based on studying the way in which
gested that gesture based interaction methods poten- humans interact with computer. In particular there
tially present a novel and effective way of increasing has been something of a paradigm shift towards the
usability of the system in terms of ease of use and psychological and sociological aspects of interface de-
simplicity. sign (Yusof, et al., 2004). Most notably the fields of
Human-Computer Interaction (HCI) and Interaction
Design (ID) have become highly prominent in this
Introduction field of research.

Touchscreen devices are becoming ever more ubiqui-
tous as time passes. This is no more evident than in
the mobile technology industry. The majority of mo-
bile devices use touchscreen technology as their main
source of input. Although touchscreen devices have
been in common use for several years the vast major-
ity of current devices fail to fully utilise the potential
offered by touchscreen technology. Gesture based
interaction has not been successful in widespread use
(Rico, Crossnan and Brewster, 2011). Many touch-
screen devices still abide by the paradigm of using
an intermediate device (i.e. on-screen/soft keyboard)
for data input. While gesture based input is some-
what prevalent in most devices (i.e. pinch gesture
for zooming) users still largely rely on poking ges-
tures when interacting with touchscreen devices and
are forced to navigate through large menu interfaces
to find applications. The term gesture carries no
formal definition largely because it is such a broad
concept. Saffer (2010) defines a gesture as any phys-
ical movement that a digital system can sense and
respond to without the aid of a traditional pointing
device such as a mouse or stylus. A wave, a head
nod, a touch, a toe tap, and even a raised eyebrow
can be a gesture. As such this paper will consider
This paper will analyse literature regarding these
these potential methodologies and how they can be
applied to implementing an effective interface. In
addition, literature concerning gesture based interac-
tion with regard to user centered design is analysed.
Given that simplicity and ease of use in an inter-
face are arguably the most important aspects of a
system this paper will focus on user focused design
approaches.
Interaction Design
The early days of computer technology application
development saw a very engineer-centric focus. This
focus on technology with little regard to user interac-
tion resulted in the inability of products to meet the
cognitive and psychological tolerance of users (Yang
and Chen, 2009). From this stemmed the field of
Interaction Design. The Interaction Design process
presents a stark contrast to the conventional design
paradigm in that it focuses almost entirely on the
end user of the product and gives little regard to
the actual technology used (Yang and Chen, 2009).
Essentially it focuses on how things should be based

University of Cape Town 1

 Literature Review

on how they currently are. considered as basic guidelines for all GUI design. In
particular it was noted that the ability to retain infor-
Interaction design is largely driven by investigating mation varied greatly among users (ling, et al., 2012).
the psychological aspects behind user interaction.
This can be seen in the paper GUI Design Based In contrast to the psychological aspects of interaction
on Cognitive Psychology: Theoretical, Empirical and design presented by Ling, at al. (2012), Hartman,
Practical Approaches by Ling, et al. (2012) which Klemmer and Takayama (2006) emphasise the im-
presented a study on the effectiveness of applying portance of the physical body in our experience,
cognitive psychology to graphical user interface de- interaction and understanding of our world. They
sign. In addition to the four main principles of GUI attempt to synthesize five themes drawn from the
design (focus on users rather than the technologies theories of embodiment in order to inspire an origi-
used, consider function first and presentation later, nal interaction design approach focusing on human
ensure GUI simplicity and the promote learning and embodied engagement:
delivering information) four cognitive psychology the-
ories were used as guidelines for GUI design: Thinking by doing
It is asserted that physically interacting with objects
Schema Theory asserts that knowledge is or- accelerates learning as it involves both physical and
ganised into basic building blocks simply known mental activity as posited by prominent psychologist
as units. Jean Piaget. Apparently studies have shown that
Cognitive Load Theory is an instruc- gesturing lightens cognitive load for both adults and
tional design theory defining information pro- infants alike. Furthermore, constrained gestural abil-
cess involving long-term processing and short- ities actually hinders thinking and communication.
term/working memory.
Performance
Retention theory refers to the amount of in- One can take better advantage of kinesthetic memory
formation that can be stored and processed by by assigning dedicated actions to different functions
a user in a certain period of time. of a user interface. Furthermore one can afford kines-
thetic learning and memorisation over prolonged pro-
Gestalt law is a foundation of instructional longed use by consistently dedicating physical move-
screen design that is explained by 11 additional ments to interface functions.
laws (which we will not outline).

The methodology by the authors applied was a two Visibility

pronged approach. Firstly case studies on the appli- Studies have shown that the production and manip-
cation of cognitive psychology and their evaluation ulation of visible artifacts in the workplace facilitate
were carried out. From this a GUI was created. Cer- coordination.
tain aspects of the GUI using the outlined cognitive
psychology theories whereas other aspects were cre- Risk Making the consequences of decisions more
ated in such a way that they negated them. The directly visible to people alters the outcome of the
GUI was then analysed under three aspect of human decision process.
cognition: human memory limitation, attention and
perception based on user tests. Secondly a survey Thickness of practice
was conducted in order to collect qualitative data Simply put this implies that if technology is supposed
outlining how users choose software (in particular, to be advantageous to the user it must provide some
how users choose their favourite media player, so- sort of previously unavailable functionality. More
cial networking site and search engine). The data specifically, the correspondence between the technol-
collected from the survey was then analysed based ogy and the real world must at some point deteriorate
on its compatibility with the cognitive psychological in order for the technology to be successful.
theories.
Clearly the physiological aspects of interaction de-
Through the use of the experiment and the sur- sign are highly pertinent to the concept of gesture
vey it was concluded that the proposed cognitive based interaction. If nothing else they actually en-
theories were proven effective when applied to GUI courage the use of gesture based interaction as it can
design. However it was also pointed out that not all lighten cognitive load and afford kinesthetic learning
the proposed cognitive psychology theories could be and memorisation. Furthermore, the themes of risk

University of Cape Town 2

 Literature Review

and performance could both provide compelling ar- published by different authors. As a result the struc-
guments for the use of a direct manipulation interface ture vary as a result of the authors presence. In turn
(which will be discussed later). Despite the intuitive this results in lack of consensus as to how patterns
appeal of these concepts there remains a need for should be formed and categorised to provide the in-
both qualitative and quantitative analysis. formation required for good design (Krushcitz and
Hitz, 2009). As stated by Seffah (2010): A univer-
On the other hand the empirical evidence provided sally accepted taxonomy for [design] patterns is still
for cognitive psychological is compelling with regard missing in HCI. Among the most popular patterns
to its application in interface design. However the forms are the following (Kruschitz and Hitz; 2009):
large variation in user retention capabilities poses an
Alexandrian Form: The initial concept of de-
obstacle to designers. No less so than with gestural
sign patterns was developed by by Christopher
interaction were users may require to memorise mul-
Alexander. He posed design patterns as a prob-
tiple gestures. This suggests that the total number
lem/solution pair and presented them in a com-
of gestures used in a system require a large amount
mon format.
of consideration so as to lessen the learning curve of
the user. UI Pattern Form: In INTERACT patterns
workshop in 1999 the UI pattern form was de-
veloped consisting of seven elements;
Human-Computer Interaction Tidwell: The Tidwell forms is a highly mini-
malist design pattern consisting of several fac-
An important aspect of designing any interface is the
tors.
way in which humans interact with it. A large field
of study regarding this is HCI which, as defined by These are but a fraction of the numerous design
Mathew, Haff and Abri (2011) is the design and im- patterns that have been suggested over the years.
plementation of interactive computing systems that Despite best efforts to apply some form of standard-
users can interact with. isation there has been little to no success (Kruschitz
and Hitz; 2009).
Human computer interaction is a highly diverse field
of study in the field of computer science. This could Contemporary approaches to HCI are that of adap-
arguably be attributed to the fact that it was initially tive and intelligent HCI as outlined by Abri, Hajj
developed with multiple disciplines including (among and Mathew (2011). The intelligent HCI module
others) computer science, cognitive psychology, social aims at learning the users reasoning, plans and goals
and organization psychology, design, sociology, and resulting in addressing their needs in a tailored way
anthropology. Summarily it can be described as not in order to ease interface interaction. An example of
just the study of humans nor the study of technology intelligent HCI is speech recognition. Adaptive HCI
but rather a bridging between these two disciplines remains one step ahead of intelligent HCI. As the
(Yusof, et al.; 2004). name suggests, it alludes to the concept of adapting
the user interface to satisfy the needs of the current
At its core, HCI is built around design patterns; user. This can be done in several ways; ranging from
important tools for the re-use and propagation of simplifying the interface to rearranging task details
knowledge within the HCI domain (Krushcitz and to predict the users next command. An example
Hitz, 2009; Seffah, 2010). Design patterns can be of adaptive HCI can be seen in web browsers that
thought of forms or templates that can be used to automatically personalises the interface based on the
solve a commonly recurring problem (Seffah, 2010). most frequently visited websites.
The power of HCI design patterns lie the fact that
they essentially both concrete and abstract as a An argument against conventional HCI is posed by
method of solving a problem. They are abstract Thimbleby and Thimbleby (2007). Technology as a
enough to draw on the common elements that hold discipline supports two alternate points of view; in-
between all instances of the resulting solution (Sef-ternalist and externalist. The internalist view focuses
fah, 2010; Kruschitz and Hitz; 2009). on the world internal to the user interface; computer
interaction, logic, computation, etc. The externalist
Design patterns have been proven to be successful view on the other hand focuses on the world external
tools, however an issue that exists with them is the to the user interface; human-interaction, observa-
fact that there are such a vast amount of patterns tion, cognition, etc. Conventional HCI is externalist

University of Cape Town 3

 Literature Review

being highly concerned with human-centered issues. Physical actions or presses of buttons rather
Thimbleby and Thimbleby (2007) argue that this than complex syntactical commands.
can result in HCI ignoring important internalist is-
Rapid, incremental and irreversible operations.
sues. As an example the authors draw on the work
of Clayton Lewis, who posed the concepts of inner These principles were largely guided and inspired by
and outer HCI. These concepts of similar to the dis- Schneidermans studies of what an enthusiastic and
tinction between internalist and externalist except happy user is. According to Schneiderman:
that the distinction between them is psychological
...enthusiastic users of the system report
rather than technological. As such, Lewiss distinc-
the following positive feelings
tion completely excludes the interaction system in
its distinction. Thimbleby and Thimbleby (2007) Mastery of the interface
assert that its as if the interactive system is a given Competence in performing tasks;
that is taken completely for granted.
Ease in learning the system origi-
nally and in assimilating advanced
The validity of this argument was further motivated
features;
by describing a selection of distinct products of in-
ternalist HCI. None were developed through nor was Confidence in the capacity to retain
supported by research that would have met conven- mastery over time;
tional externalist HCI criteria. Despite the fact that Enjoyment in using the system;
none of the products followed any recommended ex- Eagerness to show the system off to
ternalist HCI development cycles all were successful. novices;
Its important to note that the paper did not outright Desire to explore more powerful as-
dismiss externalist HCI they merely proposed that pects of the system.
an internalist approach to HCI can be effective.
These feelings convey an image of a truly
In general, it seems that HCI has become something pleased user (Schneiderman; 1983).
of a tried and tested methodology that proves to be Schneiderman essentially recognised the intuitive ap-
effective. Criticisms aimed at the heavily psycholog- peal of maintaining direct control over appropriate
ical/externalist approach seem somewhat unjustified graphical representations of objects. Furthermore
due to the lack of empirical evidence brought for- he realised reasoned that suitable representations of
ward. Furthermore, the concepts of adaptive and problems are crucial to finding solutions as well as
intelligent HCI provide an intuitively appealing way the learning process. An example of this, as posed by
of approaching system design. Schneiderman (1983), can be seen with maths prob-
lems. Drawing a picture to represent the problem is
Although the use of design patterns provide a pow- often highly beneficial in understanding the problem
erful way to solve problems that exist in HCI the and finding a solution.
sheer number of patterns available is somewhat over-
whelming. Moreover, it is arguable impossible to In order to aid his own understanding of direct ma-
objectively choose a superior design pattern due to nipulation Schneiderman (1983) considered the syn-
the highly subjective nature of problems that one tactic/semantic cognitive model. Essentially this
faces in the field of HCI. model asserts that there are two kinds of knowledge
in long-term memory; syntactic and semantic. Syn-
Direct Manipulation tactic knowledge is essentially the arbitrary knowl-
edge about a system. It varies between systems and
Direct manipulation is a style of Human-Computer
is thus system dependent. Semantic knowledge refers
Interaction created by Ben Schneiderman through
to the concepts or functionality of the system. It is
studies involving user interaction with video games
structured hierarchically from low-level fuctions to
(Schneiderman, 1983). Simply put, the concept of
higher level concepts As such it is highly volatile and
direct manipulation entails the following three princi-
forgotten unless frequently evoked. In contrast to
ples that Scheiderman defined as the integrated por-
syntactic memory, semantic knowledge is stable in
trait of direct manipulation (Schneiderman; 1983,
that in can easily be anchored to familiar concepts in
1997):
memory via explanation, analogy or example; thus
Objects and actions of interest should be con- it is system independent. Schneiderman (1983) then
tinually represented. stressed the implications of this cognitive model in

University of Cape Town 4

 Literature Review

order to highlight the benefits of direct manipula- absolutely no empirical evidence is given to support
tion. New users to the system begin with a close its effectiveness. This clearly represents a large gap
link between their semantic and syntactic knowledge. in the overall impact of the paper. Despite this lack
Users begin by memorising commands within the of empirical evidence however ADM is an intuitively
interfaces to perform certain tasks (thus using syn- appealing concept. Despite the aforementioned lack
tactic knowledge) and as users review/recall these of empirical the paper does provide a compelling ar-
commands this acts as a stimulus for recalling se- gument through inference. It stands to reason that
mantic knowledge. Thus as users gain experience human-human interaction should serve as a model
they start exploring higher level concepts. With di- for human-computer interaction. This serves as an
rect manipulation objects are shown directly to the interesting field of investigation.
user; thus all user actions exist within the high level
problem domain. There is little or no need to break Despite the huge success of direct manipulation it
these tasks into multiple commands with complex has still managed to garner criticism. Kwon, et
syntactic form (Schneiderman, 1983). al. (2011) highlighted three main issues with con-
ventional direct manipulation. Firstly manipulating
Despite the proposed superiority in direct manip- objects directly is not effective or even possible under
ulation interfaces as well its intuitive appeal, the the following conditions:
psychological theory behind direct manipulation has
not been extensively developed. Furthermore, from Items that are small or distant
a psychological point of view, direct manipulation
has actually been noted to confound several factors. Items that have a large amount of attributes
Among others, these factors include: recognition
rather than recall, visual search instead of verbal Items that are densely packed or in a limited
memory and concrete rather than abstract metaphors space
(Kieras, Meyer, Ballas; 2001).
Secondly manipulating multiple objects as a group
can pose a large challenge to users. Finally direct
The concept of direct manipulation has evolved over
manipulation provides no effective way to manipu-
time and alternate approaches have been suggested.
late intangible object properties (for example, the
One such suggestion is that of Adaptive Direct Ma-
amount of space between multiple objects is difficult
nipulation (ADM) as posed by Riecken (1991). ADM
to present as a domain object thus direct manipu-
is a graphical interface that dynamically adapts
lation is not feasible). In order to address these is-
presentation and management of interface objects
sues Elmqvist, et al. (2011) posed the notion of us-
based on the currents users actions/behaviour as
ing surrogate objects. Simply put, users interact with
well as the anatomical characteristics of their hand.
these surrogate objects instead of the domain objects.
Riecken (1991) motivated this by essentially extend-
Changes made to the surrogate objects then propa-
ing human-human interaction to human-computer in-
gate to the domain object. Although the presence
teraction. He noted that gesticulation is a large part
of surrogate objects certainly decreased the direct
of communication and realised this could be applied
nature of the manipulation it is proposed that this
to interface interaction. Furthermore it is noted that
approach provides better support for attribute-rich
it provides feedback to the user that the system is
objects, seamless and visible manipulation of mul-
attending to a dialogue with the user by directly re-
tiple objects simultaneously, and access to abstract
sponding to the users behaviour/mannerisms. This
and domain objects. In addition to an increase in
is analogous to handing someone an object. When
the indirect nature of the system it was noted that
you are handing an object to someone they must
trade-offs were required in order to accommodate for
respond by position their hand correctly based on
surrogate objects:
where your hand is. Finally the paper provides
an overview of the design and implementation of
Divided attention: with the addition of sur-
a touchscreen interface designed to support a set of
rogate objects the users attention becomes di-
user requested tasks. The system was provided with
vided between it and the domain object.
a system to define probable hand and finger trajec-
tories based on user interaction with the system. Ex Situ: Because surrogates are removed from
their context they cannot convey positions, thus
Although the design and function of the ADM sys- absolute operations such as resizing and moving
tem are quite extensively described in the paper are best left to direct manipulation.

University of Cape Town 5

 Literature Review

Visual Representation: Choosing good repre- Surface Gestures

sentations can be challenging, particularly for
abstract entities. The first (and arguably; foremost) input modality
offered by touchscreen devices is the touchscreen dis-
play itself. Using this one can create gestures in two
The use of surrogate objects provides an interesting dimensions by way of using the device touchscreen
and compelling alternative to direct manipulation. as a mobile surface computer. Ruiz, Li and Lank
However it remains little more than an alternative as (2011) refer to these two dimensional gestures as sur-
certain scenarios certainly lend themselves to direct face gestures.
manipulation as opposed to indirect manipulation.
The use of either paradigm largely depends on the In his book Designing Gestural Interfaces, Saffer
given problem domain. As such neither ca be consid- (2008) states:
ered as either superior or inferior to the other.
Weve entered the era of interactive ges-
Schneiderman (1983) himself noted problems with tures. The next several years will be sem-
direct manipulation. Among others, Schneiderman inal years for interaction designers and
outlined two important issues with direct manipula- engineers who will create the next gen-
tion. Firstly, with graphic representations user are eration of interaction design inputs, pos-
forced to learn the meanings of them. While the sibly defining them for decades to come ...
designed of a system might find a graphical object We have an opportunity that comes along
meaningful it may have little/none to users. Sec- only once in a generation, and we should
ondly, these graphical representations can also be seize it
highly misleading to users. While users may quickly
Saffer (2008) motivates this with the fact that touch-
grasp the analogical representation of a graphical
screen interfaces are becoming ever more prevalent in
object it is possible that they may make incorrect
handheld devices. Furthermore, he states that touch-
assumptions about what operations are permissible.
screen technology has evolved to such a point that it
However these issues can be largely fixed by ample
is no longer a limiting factor. These two points when
user testing.
taken together imply the opportunity to define ges-
tural interaction based on what is most natural and
Schneiderman formulated his arguments on the pit-
efficient for the user rather than what the technology
falls of direct manipulation during his conception of
can do.
it. They extended directly from the empirical re-
search used to formulate the actual concept of direct
Despite this potential however there exists a distinct
manipulation. With this in mind it seems foolish
lack in the understanding of the surface gestures de-
to disregard them. Furthermore these pitfalls can
sign space (Wobbrock, Morris and Wilson, 2009).
largely be avoided simply through user testing.
This is further exacerbated by the fact that most
surface gestures have simply defined by the designers
of the system in question (Wobbrock, Morris and
Wilson, 2009). This would suggest that most of the
gestures chosen would be done so out of the concern
Gesture Based Interaction for reliable recognition rather than what gestures
would be chosen by users of the system. Taken in
Touchscreen devices (smartphones in particular) are conjunction with the fact that user-centered design
currently the fastest growing and predominant con- is the cornerstone of HCI the development of a user
sumer computing devices (Li, 2012). Touchscreen defined gesture set seems only natural.
devices give users unique capabilities and form fac-
tors, allowing for a massive impact in the field HCI A study done by Mauney, et al. (2010) analysed the
through their ability to recognise gestures. similarities and differences in user defined gestures
across different cultures. This was done through an
Before continuing it is pertinent to accurately de- experiment whereby 340 across nine different coun-
fine the concept of a gesture. Ruiz, Li and Lank, tries were asked to define 28 common actions (i.e.
(2011) categorized gesture types based on the two zoom and copy) using gesture input. All ac-
primary input modalities provided by the majority tions were described pictorially to the user as the
of handheld touchscreen devices. describing the action verbally or through text could

University of Cape Town 6

 Literature Review

skew the data due to semantic and syntactical differ- exist a high level of agreement as to what gestures
ences between languages. The gestures were recorded are to be used for certain commands. Although users
by simply having users dip their fingers in charcoal with limited experience exhibit less agreement this
which would then leave an impression on the screen problem should fade with time as touchscreen de-
showing the gesture. vices become increasingly commonplace. Needless
to say this is highly encouraging for the design of
The initial analysis of study suggested that there was gestural interfaces as it suggests a limited learning
a large degree of agreement about gestures across the curve. However one must keep in mind that although
cultures. Using the chi-squared test for each gesture one could design a system based entirely on gestural
across the nine countries it was concluded that no ges- interaction it would pose a a steep learning curve to
ture was favoured by any country (p > 0.1). However the user (Wobbrock, Morris and Wilson, 2009). At
it was found Chinese participants used significantly the very least at least one taxonomy of surface ges-
more symbolic gestures (p < 0.01). Analysis of the tures now exists as a result of Wobbrocks, Morris and
results also suggested that the use of direct manip- Wilsons (2009) study. Furthermore the preference
ulation resulted in a higher agreement among the among users of only using one hand indicates that
gestures chosen. gestural interfaces are suitable for handheld devices.
Intuitively, direct manipulation seems like the best
Although the moderators in the study requested that paradigm to around which to built gestural inter-
users created gestures that did not require a menu action. This concept that was discussed by Rubine
the majority of participants requested a menu in the (1992) in his paper in which a score editor and draw-
event that they had difficulty thinking of a gesture. ing program were created with gestural interfaces
A significant portion of users requested a menu that using the direct manipulation paradigm. Unfortu-
could be accessed by simply tapping the screen. nately no user testing was done of the system, thus
the potential offered is only up to speculation. This
The study done by Mauney, et al. (2010) suggested is further encouraged by the study done by Mauney,
that users who had past experience with gesture et al. (2010). Moreover, the lack of agreement seen
based devices had a higher agreement rate. On the between users with a non-technical background could
other hand the amount of agreement among non- suggest the need for widgets within the system that
technical users is somewhat lower as suggested in guide the users interaction.
the study done by Wobbrock, Morris and Wilson
(2009). The study involved 20 users with no techni-
cal background to perform 27 basic commands using Motion Gestures
gestures on a 24 x 18 surface. All users were asked The second input modality supported by many (if
to perform the task using both one handed and two not the majority) of handheld touchscreen devices is
handed gestures. Throughout the course of the ex- the set of motion sensors they carry (i.e. accelerom-
periment the think-aloud method was used to gauge eter, gyroscope, orientation sensor, etc).
users thought processes. Despite the somewhat dis-
appointing lack of agreement there did exist a high As with surface gestures there exists lack of un-
degree of consistency and symmetry with each users derstanding in the design space for motion gestures
gesture set (i.e. similar operations usually elicited
(Ruis, Li and Lank, 2011). This could largely be
similar gestures from the user). From the results a attributed to the fact that in the past the design
taxonomy was created by manually classifying each of gesture based interfaces has been predominantly
gesture along form dimensions: form, flow, nature focus on issues of gesture recognition with very little
and binding. Qualitative analysis of the indicated consideration given to the social, practical and psy-
that users preferred using one hand as a opposed to chological factors affecting usability (Rico, Crossnan
two. However it was also shown that there was a and Brewster, 2011). Unlike surface gestures which
higher degree of agreement among the two handed are often rather small and subtle movements, mo-
gestures employed by the users. Interestingly most tion gestures usually require expressive gesticulation
users didnt care about the number of fingers used to
to execute. This brings up the issue of social ac-
elicit the gestures. ceptability as motion gestures require users to adopt
new and potentially idiosyncratic behaviour in public
Overall current literature seems to encourage em- spaces (Rico, Crossnan and Brewster, 2011). This is
ploying of user defined gestures in a system. For of particular importance to failure scenarios which
users with experience using gestures there seems to alone could define a motion gesture as social accept-

University of Cape Town 7

 Literature Review

able or not. Rico, Crossnan and Brewster, (2011) can build. A possible avenue of future research into
present the following example: gesture interaction could possibly look at creating a
system using a combination of both surface gestures
...our previous work shows that users were
and motion gestures.
concerned that an interface would be un-
able to successfully recognize inputs, re-
sults in repeated and increasingly erratic
motions to successfully complete a ges- Conclusion
ture. While the error could happen to any
gesture, errors executing a foot tap versus Interface design is a multifaceted issue which appears
a shrug would lead to very different be- to depend on several determinants including psycho-
haviors. Users were also concerned about logical, social factors. No more so than with gestural
the possibility of false positive recogni- interfaces which introduce physiological facts to the
tions by a system, and those gestures that design process. In addition, all of these factors are
users felt they were more likely to perform highly dependent on each other. As such, in order
by accident were less acceptable. to create an effective interface one must take all of
these factors into account.
One could use this as the basis for the argument that
a formal model of motion gestures is required. Some-
Human-Computer Interaction and Interaction de-
thing that is currently non-existent. This lack of
sign principles have proved to be a tried and tested
formality with regard to gesture design is addressed
approach to interface design. Intelligent and adaptive
by Ruis, Li and Lank (2011) in their paper User-
HCI take the concept further providing users with a
Defined Motion Gestures for Mobile Interaction. In
system that dynamically responds to their inactions
it, a study was performed whereby 20 users were
in order to help emulate human-human interaction.
asked to design and perform a total of 19 commands
The use of design patterns can help designers find a
using motion gestures. As in the study done by Wob-
starting point to solving design problems and prevent
brock, Morris and Wilson (2009) no regard was given
them from having to re-invent the wheel. However
to the systems ability to recognise gestures, instead
the sheer number of design patters coupled with their
users were told to assume that the device could recog-
subjectivity to the situation may pose something of a
nise any gesture they performed. The actions were
challenge to the designer. Despite criticisms towards
recorded by both measuring the sensors as well as
the overtly externalist approach presented by HCI
making video records of the users interaction. From
it remains a powerful technique for improving the
the results the authors suggested that it yielded two
user experience. In particular, direct manipulation
specific research questions:
has become arguably the most successful of all HCI
As seems to be the case with surface gestures, styles, largely due to its intuitive appeal.
users showed a broad and agreement on the ges-
tures used. Gestural interfaces provide a relatively new and
potentially effective method for user interaction.
Using both the recorded measurements from
Current literature provides encouraging support for
the sensors and reviewing the video records of
user-centered approaches to gestural interface de-
the user interaction the authors attempted to
sign. Studies suggest that there is a large agreement
specifYy the taxonomy of the parameters that
among user defined gestures for both motion and sur-
can be manipulated to differentiate between dif-
face gestures (e; Ruiz, Li and Lank, 2011). Thanks
ferent gestures. It is suggested that this taxon-
to research by Wobbrock, Morris and Wilson (2009)
omy could be used as a design space for motion
there now exists a taxonomy for surface gestures.
gestures.
Similarly the research done by Ruis, Li and Lank
It was interesting to note that, as with surface ges- (2010) provides designers with a taxonomy for mo-
tures, there is generally a high level of agreement tion gestures. If nothing else these are promising
between gesture defined by the users. The social starting points for gestural interface design. However
acceptability of motion gestures presents a new di- gestural interfaces that use motion gestures intro-
mension to its implementation and remain a valid duce a sociological factors that need to be addressed
concern for designers. by the designer. In particular, the social appropri-
ateness of certain motion gestures in public places
The work done by Ruiz, Li and Lank (2009) pro- requires close attention by designers.
vides a taxonomy of motion gestures upon which one

University of Cape Town 8

REFERENCES
Interestingly, analysis of results seems to suggest
that direct manipulation is a particularly prudent
HCI style to apply to gestural interfaces. Firstly,
agreement in gestures between users is higher in sys-
tems that use direct manipulation as opposed to
those that dont (Mauney, et al., 2010). Further-
more, it has been suggested that on-screen widgets
can be used to provide hints to users as to appropri-
ate gestures to use (Wobbrock, Morris and Wilson
2009). This would also lessen the cognitive strain on
users to memorise copious amounts of gesture which
is particularly important as memory retention varies
greatly among users (Ling, et al., 2012).
Overall the topic of interface design is complex and
multi-faceted with many authors proposing different
approaches. However, a fundamental component of
the majority of interfaces is to provide efficiency and
ease of use. As such this remain an important con-
cept in interface design. Touchscreen devices have
the potential to provide this ease of use. Thus by
considering psychological, sociological and physio-
logical factors of gestural input in conjunction with
user-centered design perspective one can gain some
insight into what may be done to improve users
interaction experience.
References
[1] Adikari, S., McDonald, C. 2006. User and Us-
ability Modeling for HCI/HMI: A Research De-
sign. International Conference on Information
and Automation. Colombo, Sri Lanka. 15 - 17
December.
[2] Amin, R., Baker, O. F., Yusof, R. J. R., Zain-
udin, R. 2004. Human Computer Interaction
(HCI) - Aspect in Developing Information Ac-
cess System. TENCON 2004. Chiang Mai, Thai-
land. 21-24 November.
[3] Hartman, B., Klemmer, S. R., Takayama, L. [12] Rico, J., Crossnan, A., Brewster, S. 2011.
2006. How Bodies Matter: Five Themes for In-
teraction Design. Proceedings of the 6th confer- tions of Gestures in Real World Mobile Settings.
ence on Designing Interactive systems. ACM,
New York, USA. pp. 140-149.
[13] Rieken, R. D. 1991. Adaptive Direct Manipu-
[4] Kieras, D., Meyer, D., Ballas, J. 2001. Towards
Demystification of Direct Manipulation: Cog-
nitive Modeling Charts the Gulf of Execution.
Proceedings of the SIGCHI Conference on Hu-
man Factors in Computing Systems. ACM, New [14] Rubine, D. 1992. Combining Gestures and Di-
York, USA. pp. 128-135.
University of Cape Town
Literature Review
[5] Kruschitz, C., Hitz, M. 2009. The Anatomy of
HCI Design Patterns. Computation World: Fu-
ture Computing, Service Computation, Adap-
tive, Content, Cognitive, Patterns. Athens,
Greece. 15-20 November.
[6] Kruschitz, C., Hitz, M. 2010. Analyzing the HCI
design pattern variety. Proceedings of the 1st
Asian Conference on Pattern Languages of Pro-
grams. Tokyo, Japan. March 16-17.
[7] Kwon, B., Javed, W., Elmqvist, N., Soo Yi,
J. 2011. Direct Manipulation through Surro-
gate Objects. Proceedings of the SIGCHI Con-
ference on Human Factors in Computing Sys-
tems. ACM, New York, USA. pp. 627-636.
[8] Li, Y. 2012. Gesture-Based Interaction: A New
Dimension for Mobile User Interfaces. Proceed-
ings of the International Working Conference on
Advanced Visual Interfaces. ACM, New York,
USA. pp. 6-6.
[9] Ling, C. S., Zainon, W. M. N. W., Yee, C. K.,
Yee, W. S. 2012. GUI Design Based on Cognitive
Psychology: Theoretical, Empirical and Prac-
tical Approaches. International Conference on
Computing Technology and Information Man-
agement (ICCM). IEEE, Seoul, Korea. pp. 836-
841.
[10] Mathew, A. R., Al Hajj, A., Al Abri, A.,
2011. Human-Computer Interaction (HCI): An
Overview. International Conference on Com-
puter Science and Automation Engineering.
Shanghai, China. 10-12 June.
[11] Mauney, D, Howarth, J., Wirtanen, A., Capra,
M. 2010. Cultural Similarities and Differences
in User-Defined Gestures for Touchscreen User
Interfaces. CHI 10 Extended Abstracts on Hu-
man Factors in Computing Systems. ACM, New
York, USA. pp. 4015-4020.
Gesture-Based Interfaces: Practical Applica-
Whole Body Interaction. London: Springer. pp.
173-186.
lation. Decision Aiding for Complex Systems,
Conference Proceedings, IEEE, Charlottesville,
USA. pp. 1115-1120.
rect Manipulation. Proceedings of the SIGCHI
9
REFERENCES
Conference on Human Factors in Computing
Systems. ACM, New York, USA. pp. 659-660.
[15] Ruiz, J., Li, Y., Lank, E. User-defined motion [20] Thimbleby H., Thimbleby, W. 2007. Internal-
gestures for mobile interaction.Proceedings of the
SIGCHI Conference on Human Factors in Com-
puting Systems. ACM, New York, USA. pp 197-
206.
[16] Saffer, D. 2008. Introducing Interactive Ges-
tures. Designing Gestural Interfaces. California:
OReilly Media, Inc. pp. 1 - 30.
[17] Schneiderman, B. 1983. Direct Manipulation:
A Step Beyond Programming Languages. Com-
puter. 16(8). pp. 57-69.
[18] Schneiderman, B. 1997. Direct Manipulation for
Comprehensible, Controllable Predictable and
User Interfaces, Proceedings of the 2nd interna-
tional conference on Intelligent user interfaces,
ACM, New York, pp. 33-39.
[19] Seffah, A. 2010. The evolution of design pat-
terns in HCI: from pattern languages to pattern-
oriented design. PEICS 10 Proceedings of the
1st International Workshop on Pattern-Driven
University of Cape Town
Literature Review
Engineering of Interactive Computing Systems.
ACM, New York. pp. 4-9.
ist and Externalist HCI. Proceedings of the 21st
BCS HCI Group Conference. British Computer
Society, Lancaster, UK. pp. 111-114.
[21] Wobbrock, J. O., Morris, M. R., Wilson, A.
D. 2009. User-Defined Gestures for Surface
Computing. Proceedings of the SIGCHI Confer-
ence on Human Factors in Computing Systems.
ACM, New York, USA. pp. 1083-1092.
[22] Yang, X., Chen G. 2009. Human-Computer In-
teraction Design in Product Design .Education
Technology and Computer Science, 2009. ETCS
09. First International Workshop on. IEEE,
Hubei, China. pp. 437-439.
[23] Yusof, R. J. R., Amin, R., Zainudin, R., Baker,
O. F. 2004. Human Computer Interaction (HCI)
Aspect in Developing Information Access Sys-
tem. TENCON 2004. 2004 IEEE Region 10
Conference. IEEE, Chiang Mai, Thailand. pp.
387-390.
10