Professional Documents
Culture Documents
erezposner@gmail.com
Nick Starzicki
Department of Electrical Engineering,
Afeka, Tel Aviv College of Engineering
218 Bney Efraim Rd., Tel Aviv 69107, Israel
nicks@mail.afeka.ac.il
Eyal Katz
Department of Electrical Engineering,
Afeka, Tel Aviv College of Engineering
218 Bney Efraim Rd., Tel Aviv 69107, Israel
eyalk@afeka.ac.il
In this work we have implemented a single standard camerabased virtual keyboard, by improving shadow-based touch
detection. The proposed solution is applicable to any surface.
The system has been implemented on an Android phone,
operates in real time, and gives excellent results.
I. INTRODUCTION
This paper describes an improved method implementing a
virtual keyboard using the single integrated standard two
dimensional (2D) camera of a smart phone.
Virtual keyboards had been proposed based on different methods
[1]-[13]. Camera based virtual keyboards can be implemented
using a single or multiple cameras. One of the major challenges
is how to determine if the finger touches the surface or not.
Touch detection based on real three-dimensional (3D) model
built from stereoscopic camera based systems is more accurate
than single camera based solutions. However, since stereoscopic
cameras are not common in mobile phones, this method is less
applicable to mobile solutions. The challenge of accurate touch
detection is even greater when using a single camera and almost
any surface. Floating virtual keyboards, which are portable and
enable directing the camera to any surface, are even more
challenging to implement.
The proposed solution is based on single standard mobile phone
camera; it implements a floating keyboard; and presents an
improved touch detection method, based on shadow analysis.
Thus enables working on any surface, as long as both the finger
and its shadow are visible to the camera.
, =
= 1
255 ,
0 ,
( , )
33255 (
33<255 (
(1)
(2)
= ( ())
(3)
3. Tip Detection
In this stage the tip of the object is found using the approach
presented in [8]. The finger's discrete outline is converted into a
list of consecutive coordinates representing the contour of the
finger. Each three j consecutive pixel coordinates [C(j k),
C(j) , C(j + k)] represent three vertexes of a triangle, and the
head angle is calculated under the assumption that the middle
coordinate is the head vertex. The angle is calculated using the
law
of
cosines.
= (
+ 2
) 2
(7)
2. Edge Detection
The edge detection phase is essential in order to find the
hand's contour and then the finger tip. Hence, to extract the
contour we use canny filter. Additional modifications are needed
due to the fact that the contour acquired is not continuous and to
make it so we use 8-point connected component analysis
neighborhood system that produces a set of counter-clockwise
perimeter coordinates which trace the outline of the hand
I(j) = { (xj, yj) }
(6)
this enables a complete traversal of the hand's edge used in 3.
and small bumps that may have remained.
In order to remove the remaining bumps we threshold the
image under the assumption that there should be only one big
object in the frame- the hand- . The area of each remained object
is calculated and the largest one is determined to be the hand. To
reduce the complexity, the continuous contour is transformed into
a discrete one.
(a)
(b)
4. Shadow Extraction
In most Smart Phones, in particular Samsung Galaxy s9000I
there are many features. Among them is an ISO Camera. ISO
determines how sensitive the image sensor is to light ISO 100
was found affective to revoke dynamic operations that could
affect the shadow isolation.
The image obtained from 1. and the captured image are
subtracted from the initial reference image leaving us with only
the shadow within the new obtained image (after transforming
the result image to a binary image). Then, the image containing
the shadow is processed as if a hand is being detected through
(a)
(b)
(c)
(d)
(a)
(b)
(c)
(d)
Figure 5: Shadow Processing. (a) shows the shadow extraction from the
captured image, (b) shows the shadow's contour consisting of the main
shadow and an unnecessary small shadow, (c) shows the shadow's
continuous curve, (d) The shadow's tip detected is shown as a green dot
on the shadow's contour
5. Touch Detection
The fingertip detected in 3. and its following shadow's tip
found in 4. is an estimate of the finger and shadow locations in
the image. The fingertip is marked on its hand's curve ( xSF , ySF )
shown in RED in Figure 6.a, Figure 6.b. The corresponding point
on the shadow's curve ( xS , yS ) is marked in GREEN in Figure
6.a, (theorem: the corresponding point on the shadow curve will
always be visible). Then the distance between ( xSF , ySF ) and
( xS , yS ) is calculated as:
d = ( xSF xS , ySF yS )
(8)
(9)
As seen from Figure 6.a where the finger is distant from the
surface, the distance d is large and there is no touch, and when
they are close, shown in Figure 6.b the distance is small and a
touch has occurred. It must be noted that when the distance d is
small enough to represent a touch, both the fingertip and shadow
tip must be in the same region representing a certain letter in
order for the touch to be valid. Special cases are treated or
flagged by the system.
(e)
(f)
6. Mapping
The bottom half of the phones screen is divided into 30
buttons. The button coordinates are known, making the mapping
quite simple: once a touch is detected, its coordinate is known,
and is compared to the button ranges to obtain the requested key
and add user feedback. The keyboard's language was
implemented in both English and Hebrew.
IV. RESULTS AND COMPARISONS
From examining the ratio, r, of Eq. (9), [5] it is apparent that
the shadow area has little or no change between "Almost-touch"
and actual touch, but is not reduced to an area of zero. Unlike
the above measure the proposed measure calculated distances
between the fingertip and the shadow-tip. The proposed measure
of Eq. (8) reduces to zero upon actual touch.
Furthermore, looking at the possibility for a false touch
detection, since the existing method of [5] needs a threshold
which depends on lightning conditions and the virtual keyboard
surface texture, this threshold acts on variable and noisy input.
The proposed solution is always searching for a fingertip and
shadow-tip distance that ideally reduces to one pixel, depending
on the light source position. Therefore, it appears that this
No touch
Almost Touch
Touch
1.2
0.1
0.07
20.5
7.3
0.9
VI. REFERENCES
H. Du, T. Oggier, F. Lustenburger and E. Charbon, A virtual
keyboard based on true-3D optical ranging, Proc. British Machine
Vision Conference (BMVC), Oxford, pp. 220-229, Sept. 2005.
[2] Katz I, Gabayan K, Aghajan H, "A Multi-Touch Surface Using
Multiple Cameras". Dept. of Electrical Engineering, Stanford
University, Stanford, CA 94305, 2007.
[3] F. Echtler, M. Huber, G. Klinker, Shadow Tracking on MultiTouch Tables", Technische Universitt Mnchen - Institut fr
InformatikBoltzmannstr. 3, D-85747 Garching, Germany., AVI
08, 28-30 May, 2008, Napoli, Italy.
[4] Andrew D. Wilson, PlayAnywhere, A compact interactive
tabletop projection-vision system, Proceedings of the 18th annual
ACM symposium on User interface software and technology,
October 2326, 2005, Seattle, WA, USA
[5] Y. Adajania, J. Gosalia, A. Kanade, H. Mehta, N. Shekokar.
Virtual Keyboard Using Shadow Analysis, IEEE Conference on
Emerging Trends in Engineering and Technology (ICETET),
2010.
[6] Kenkichi Yamamoto, Satoshi Ikeda, Tokuo Tsuji, and Idaku Ishii,
A Real-time Finger-tapping Interface Using High-speed Vision
System, 2006 IEEE International Conference on Systems, Man,
and Cybernetics, October 8-11, 2006, Taipei, Taiwan
[7] Y. Hirobe, T. Niikura, Y. Watanabe, T. Komuro, M. Ishikawa,
Vision-based Input Interface for Mobile Devices with High-speed
Fingertip Tracking, UIST09, October 47, 2009, Victoria, BC,
Canada.
[8] Malik S, "Real-time Hand Tracking and Finger Tracking for
Interaction", CSC2503F Project Report, December 18, 2003.
[9] T. .Niikura, Y. Hirobe, A. Cassinelli, Y. Watanabe, T. Komuro, M.
Ishikawa, In-air Typing Interface for Mobile Devices with
Vibration Feedback, SIGGRAPH 2010.
[10] Jani Mantyjarvi', Jussi Koivumaki2,
Petri Vuori3 ,
KEYSTROKE RECOGNITION FOR VIRTUAL KEYBOARD,
2002
[11] H. A. Habib and M. Mufti: Real Time Mono Vision Gesture Based
Virtual Keyboard System, IEEE Transactions on Consumer
Electronics, Vol. 52, No. 4, NOVEMBER 2006
[12] Shumin Zhai, Michael Hunter, Barton A Smith, The Metropolis
Keyboard An Exploration of Quantitative Techniques for
Virtual Keyboard Design, Proceedings of ACM Symposium on
User Interface Software and Technology (UIST 2000), November
5-8, 2000, San Diego, California. pp 119-128.
[13] Mathias Klsch, Matthew Turk, Keyboards without Keyboards:
A Survey of Virtual Keyboards, UCSB Technical Report 200221, July 12, 2002
[1]