Using Leap Motion sensor and game environment for carpal

tunnel syndrome treatment

Diana Tsilidi1
1

Informatics Engineering Department, T.E.I of Crete, Heraklion Crete, Greece.

Abstract
There has been extended research regarding the negative effect of using computers
and playing video games, but in recent years researchers suggest that video games,
that motivate players to move, especially the games that require the use of sensors and
generally motion sensing input devices, can be useful tools for encouraging healthy
physical activity. Another important impact of the use of sensors in Human Computer
Interaction (HCI), is that the technology is not limited to the healthy users anymore,
these devices can be used for the development of different applications that can be
easily used by patients with motor disabilities.
Keywords: Leap Motion, Carpal tunnel syndrome, Physical therapy, Sensors

I.

Introduction

A physical injury or a stroke can lead a patient to a long and difficult recovery, while
physical therapy is a slow process that often takes months before positive effects can
be felt. During the therapy, patients have to perform boring, time consuming and
repetitive exercises that are not very pleasant. Also, while the presence of the physical
therapist is not necessary for the performance of the exercises, there is not enough
motivation for the patient to do them alone at home. [1] The idea of converting the
boring physical exercises into a very interesting and fun game is very promising, since
it can provide a motivation for the patients to continue therapy at home, even when
they are not under therapist supervision. Games often use repetitive actions to get
players into a rhythm. Especially when the game includes elements like achievements,
high scores, rewards and positive feedback, it can become incredibly fun and
addictive and is going to keep patients want to play more and help to speed up their
recovery. [2] It creates an environment where the repetition becomes an advantage,
and something to be enjoyed rather than dreaded.
Rehabilitation and treatment can thus be more fun, less time consuming, and, most
important, measurable. The game can be designed to collect data on the progress of
the patient, by keeping the game scores and comparing the results over certain time
period, and inform the patient and the therapist about the results of the treatment. The
therapist can analyze the data to be able to determine how to proceed with the
treatment or whether the therapy has positive effects on the patient. Further, it adds
the feeling of accomplishment for the patients, which can be very beneficial for the

treatment. [3] Of course, the system has to be individually calibrated to the abilities of
each patient, so that no patient will become frustrated with a lack of progress or
progress too quickly.
II. Carpal tunnel syndrome
Carpal tunnel syndrome (CTS) is a medical condition in which the median nerve is
compressed as it travels through the wrist at the carpal tunnel and causes pain,
numbness and tingling, in the part of the hand that receives sensation from the median
nerve. Although painful sensations may indicate other conditions, carpal tunnel
syndrome is the most common and widely known of the entrapment neuropathies in
which the bodys peripheral nerves are compressed or traumatized. The mechanism is
not completely understood but there are a variety of contributing factors. Some of the
individual predisposing factors include: diabetes, obesity, pregnancy, hypothyroidism,
and a narrow-diameter carpal tunnel. CTS may also result from an injury that causes
internal scarring or mis-aligned wrist bones. Occupational causes involve use of the
hand and arm, such as heavy manual work, work with vibrating tools, and highly
repetitive tasks even if they involve low force motions. Symptoms usually start
gradually, with frequent burning, tingling, or itching numbness in the palm of the
hand and the fingers, especially the thumb and the index and middle fingers. Some
carpal tunnel sufferers say their fingers feel useless and swollen, even though little or
no swelling is apparent. The symptoms often first appear in one or both hands during
the night, since many people sleep with flexed wrists. A person with carpal tunnel
syndrome may wake up feeling the need to shake out the hand or wrist. As
symptoms worsen, people might feel tingling during the day. Decreased grip strength
may make it difficult to form a fist, grasp small objects, or perform other manual
tasks. In chronic and/or untreated cases, the muscles at the base of the thumb may
waste away. Some people are unable to tell between hot and cold by touch. [4]

Image 1 Carpal tunnel syndrome.

III. Physical therapy treatment

The microcirculation of the nerve is being affected, because of the entrapment
neuropathies, also venous congestion occurs, and axoplasmic transport is reduced.
The pressure within the carpal tunnel is minimized, when the position of the wrist is
neutral (0) , but it is increased during flexion and extension of the wrist. Also, the
external pressure of the palm, the incursion of the lumbricals into the carpal tunnel,
forearm position of full supination, the sustained grip and isolated finger flexion
against resistance tend to increase the pressure within the carpal tunnel. Lower
pressures have been observed with the forearm in 45 of pronation and the
metacarpophalangeal (MCP) joints flexed to 45, as compared to forearm supination
and 0 or 90 of MCP joint flexion. Thus, any changes in the wrist and forearm
position, activity levels, repetitive gripping, and pressure on the volar aspect of the
wrist could possibly lead to altered nerve compression and reduce the symptoms
related to CTS. The main goal of the intervention techniques is to reduce the pressure
within the carpal tunnel and improve neural circulation and nutrition.
In case of Carpal Tunnel Syndrome patients with painful symptoms and diminished
hand functionality the main action of a physical therapist is to reduce the pain and to
give the patient exercises for mobility and strength. Tendon gliding of the finger
flexor tendons and nerve gliding of the median nerve exercises are appropriate for
conservative management of symptoms related to CTS. [5] The therapy, always
depending on the particular case of the patient condition, may also include exercises
to increase the strength of the muscles in hand, fingers, and forearm (and in some
cases, the trunk and postural back muscles) and stretching exercises to improve the
flexibility of the wrist, hand, and fingers.

Image 2 Exercises for Carpal Tunnel Syndrome treatment

The goal of physical therapy is to reduce the symptoms and eliminate the need for
surgery and to permit the patient to be active and functional in everyday life. Physical
therapy treatment, like strength and flexibility improvement exercises for the wrist
and fingers muscles, is also important after surgery, so that the patient is able to
restore strength to the wrist and to learn to modify bad habits that may have led to
symptoms in the first place. [4]
IV. Game based therapy versus traditional therapy

A different and much more interesting way to perform the prescribed home exercises
for the physical therapy is by using interactive technologies such as video games with
motion-based input devices. The use of motion-based games provides the patient with
the necessary motivation needed to complete therapeutic exercises at home by
presenting a rich graphical and multimodal game context. Also, it is important for the
game environment and subject to be customizable, so that it can be appropriate and
attractive for every individual patient. The quality of home game-based therapy is
expected to be improved compared to the traditional therapy at home, because the
game environment will encourage and motivate the patient by providing interesting
content and performance feedback to the patient, instead of repetitive and monotonous
procedure like traditional therapy usually is.
V. Leap motion sensor
The Leap Motion controller is a small USB peripheral device which is designed to be
placed on a physical desktop, facing upward. Using two monochromatic IR cameras
and three infrared LEDs, the device observes a roughly hemispherical area, to a
distance of about 1 meter (3.28084 feet). The LEDs generate pattern-less IR light and
the cameras generate almost 300 frames per second of reflected data, which is then
sent through a USB cable to the host computer, where it is analyzed by the Leap
Motion controller software using mathematical calculations in a way that has not been
disclosed by the company, in some way synthesizing 3D position data by comparing
the 2D frames generated by the two cameras.
The Leap Motion API presents motion tracking data to the application as a series of
snapshots called frames. Each frame of tracking data contains the measured positions
and other information about each entity detected in that snapshot. Each Frame object
contains an instantaneous snapshot of the scene recorded by the Leap Motion
controller. Hands, fingers, and tools are the basic physical entities tracked by the Leap
Motion system.
A Frame object containing tracking data from a connected Controller object can be
obtained by using the frame() method of the Controller class:

The frame() function takes a history parameter that indicates how many frames back
to retrieve. The last 60 frames are maintained in the history buffer. The Frame class
defines several functions that provide access to the data in the frame.
A hand is described by its position, orientation, posture, and motion:
isRight, isLeft Whether the hand is a left or a right hand.
Palm Position The center of the palm measured in millimeters from the
Leap Motion origin.
Palm Velocity The speed and movement direction of the palm in
millimeters per second.
Palm Normal A vector perpendicular to the plane formed by the palm of
the hand. The vector points downward out of the palm.
Direction A vector pointing from the center of the palm toward the fingers.
grabStrength, pinchStrength Describe the posture of the hand.
Motion factors Provide relative scale, rotation, and translation factors for
movement between two frames.
The Vector class defines functions for getting the pitch (angle around the x-axis), yaw
(angle around the y-axis), and roll (angle around the z-axis) [6].
VI. Other sensors used for therapy and rehabilitation
Kinect is a motion sensing input device that was at first built for Xbox 360 video
game console, but later was released also for Windows PC. Kinect enables the users
to interact and control the game environment with a Natural User Interface (NUI), by
using either gesture or voice commands. The system is composed of an RGB camera,
a depth sensor, which includes an infrared projector and a monochrome CMOS
(complimentary metal-oxide semiconductor), and a multi-array microphone. The
depth sensor offers more robust 3D data capturing, regardless of the lighting
conditions. Kinects microphone includes an array of four microphones that can
isolate the voices of the players from the noise in the room and can process 4 channels
of 16-bit audio at a sampling rate of 16 kHz. Thus, the user can be at a distance from
the microphone and still use voice controls. The sensor can recognize 6 individuals
but only track 2 of them, providing their skeletal information, up to 20 joints for each
active individual. The SDK for Windows can be programmed in C++, C#, and Visual
Basic.Net. [7]
Since the use of Kinect has become very popular in physical therapy and
rehabilitation, it is important to understand the possibilities and the limitations of the
sensor. The research regarding the evaluation of reliability and robustness of Kinect
sensor in medical applications comes to the conclusion that it is an appropriate choise

for motion capture device and precise joint tracking and it is considered accurate
enough for use in most of the physical therapy and rehabilitation treatments. Some of
the skeleton tracking issues can be noise and false recognition of some objects of the
scene as users parts, for example a chairs leg can be detected as the individuals leg
in the seated mode. This can create a serious problem in the development of an
application for individuals using wheelchairs or walkers.
The results of clinical experiments with volunteers, asked to perform abduction and
flexion motions, showed that Kinect tracks trunks movement accurately but it may
underestimate arm movements and overestimate leg movements by up to 30%. [8]
There are also different techniques, such as computational algorithms, that reduce
tracking errors and improve Kinects motion tracking accuracy, making it acceptable
therapeutic tool for monitoring and treatment. Also, Kinect is not the most appropriate
therapeutic tool for wrist/hand joints tracking. In this case, Leap motion sensor can be
much more accurate and thorough.
NintendoWii was released in 2006 as a home video game console and was the first
gaming device to use motion-sensing input technology. [9] The primary Wii remote
controller detects movement in three dimensions. In addition to the usual buttons it
has a built in accelerometer combined with infrared camera which allows it to sense
its position in 3D space and detect acceleration along three axes . The controller
connects to the Wii using Bluetooth with an approximate range 9 meters. Also, there
are some additional controllers that Wii can support ,such as the Nunchuck, which
features an accelerometer and the traditional analog stick with trigger buttons and a
Wii Balance Board to detect the weight and weight distribution of a person standing
on it.
The Wii has gained the support of physical therapists regarding its use for
rehabilitation, because it is usable, it has a wide variety of games available and the
environment that it creates motivates the patients to continue the therapy. There have
been studies that showed a significant improvement in post-stroke patient condition
after the use of Wii Fit Balance Board for rehabilitation. However, the games that it
provides and generally the whole system are not designed specifically for
rehabilitation, leading to issues to be arise regarding the use of Wii in physical
therapy treatment. The main problem is that it cannot accurately monitor and track
patients progress and most of the times the games are too challenging for users with
physical disabilities. Although, there have been attempts to combine the benefits of
the Nintendo Wii with other projects to produce a new integrated rehabilitation
system. [10]
VII. Experimental results and assessment
The assessment of a therapeutic game or system is usually based on the observation of
the patient during the process of rehabilitation or therapy and estimation of the results.

Sensors and computing technology can contribute significantly to a precise clinical

assessment and efficient therapeutic decision making.
A noteworthy study that has been conducted [11] and should be mentioned used Leap
Motion sensor and an open source JavaScript version of a game called Fruit Ninja for
stroke patients rehabilitation. The original game was modified for the needs of the
study and mouse events were replaced by hand movement tracking. The pilot study
involved 14 patients with stroke, who were asked to play the game for one minute,
while they were able to choose the appropriate difficulty level for their condition. The
results showed a general satisfaction of the users, who considered this way of
rehabilitation engaging, useful and would use it at home. Although some of the
participants didnt find the game responsive enough regarding the tracking of their
hand, generally the clinical results were promising and highly correlated with the
game score, which makes it a useful assessment tool.
VIII. Future work and challenges
It is very important for the further development and progress of the therapeutic and
rehabilitation systems that use Leap Motion sensor to conduct experimental studies, as
the one mentioned above, so there are real clinical results, based on which such
system can be adapted and improved. Different studies that concern a specific
physical and mobility impairments would significantly contribute to the advancement
of these treatment applications.
Another important step would be to create open source libraries of different sets of
gestures that would be useful for the development of the therapeutic applications,
since that at the present time the existing gestures sets are very limited and intended
for simple application and game controlling.
Of course, the use of Leap Motion sensor in physical therapy and rehabilitation is
limited, because it is only appropriate for the arm, hand and fingers tracking, so it can
be used for specific therapeutic applications. However, it can be combined with other
existing platforms to be able to calculate all the metrics and detect and track a rich set
of body gestures. [1] For example, Leap Motion sensor can compensate for the lack of
precision of Microsoft Kinect in detecting hand gestures like pronation/supination and
flexion/extension of metacarpophalangeal and the proximal interphalangeal joints of
the hand, while Kinect is more appropriate for other body movements tracking. This
combination of sensors would provide the therapist with sufficient data for the
comprehensive assessment of the patients condition and therapy progress.

References
[1] Abdur M. Rahman, "Multisensor Serious Game-Based Therapy," International Journal
of Distributed Sensor Networks, 2014.
[2] Making physical therapy fun with Ten Ton Raygun. [Online].
http://blog.leapmotion.com/axlr8r-spotlight-making-physical-therapy-fun-with-ten-tonraygun/
[3] www.notimpossiblenow.com. [Online].
http://www.notimpossiblenow.com/lives/what-if-physical-rehabilitation-were-as-easyas-playing-a-video-game.
[4] American Physical Therapy Association, "Physical Therapist's Guide to Carpal Tunnel
Syndrome," 2013.
[5] Susan L. Michlovitz, "Conservative Interventions for Carpal Tunnel Syndrome," Journal
of orthopaedic & sports physical therapy, October 2004.
[6] LeapMotion API. [Online].
https://developer.leapmotion.com/documentation/csharp/api/Leap_Classes.html?prog
lang=csharp.
[7] Hossein Mousavi Hondori and Maryam Khademi, "A Review on Technical and Clinical
Impact of Microsoft Kinect on Physical Therapy and Rehabilitation," Journal of Medical
Engineering, no. 2014, p. 16, December 2014.
[8] S. Obdrzalek, G. Kurillo, and F. Ofli, "Accuracy and robustness of Kinect pose estimation
in the context of coaching of elderly population," in Proceedings of the Annual
International Conference of the IEEE Engineering in Medicine and Biology Society, San
Diego, USA, 2012, pp. 1188-1193.
[9] J.C. Lee, "Hacking the nintendo wii remote," Pervasive Computing, IEEE, vol. 7, no. 3, pp.
39-45, July 2008.
[10] Fraser Anderson, Michelle Annett, and Walter Bischof, Lean on Wii: Physical
Rehabilitation With Virtual Reality and Wii Peripherals, 2010.
[11] Maryam Khademi et al., "Free-Hand Interaction with Leap Motion Controller for Stroke
Rehabilitation," , Toronto, Canada, 2014.