Omek Beckon GAT - User Manual

Omek Gesture Authoring Tool
For Beckon 3.0

User Manual
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Legal Notice
Copyright 2012 by Omek Interactive. All rights reserved. All information included in this document, such as text, graphics, photos, logos and images, is
the exclusive property of Omek Interactive. No part of this publication may be reproduced, transmitted, transcribed, stored in a retrieval system, or
translated into any language or computer language, in any form or by any means, electronic, mechanical or otherwise without prior written permission
of Omek Interactive.
This document is provided strictly on an as is basis without warranty of any kind, either express or implied, including, but not limited to, the implied
warranties of merchantability or fitness for a particular purpose Omek Interactive assumes no responsibility for any errors that may have been included in
this document, and reserves the right to make changes to the document without notice. Omek Interactive disclaims any responsibility for incidental or
consequential damages in connection with the furnishing, performance, or use of this document.
2012 Omek Interactive, Ltd. All rights reserved. | www.omekinteractive.com
Table of Contents
Preface ................................................................................................................................................................................ 5
Gesture Authoring Tool (GAT) Overview ............................................................................................................................ 5
What is a Gesture? .......................................................................................................................................................... 5
Training GAT to Recognize Gestures ............................................................................................................................... 5
Machine Learning ............................................................................................................................................................ 6
Basic Entities and Terms .................................................................................................................................................. 7
Frame .......................................................................................................................................................................... 7
Gesture........................................................................................................................................................................ 7
Gesture Pack ............................................................................................................................................................... 7
Sequence ..................................................................................................................................................................... 8
Clip .............................................................................................................................................................................. 8
Building a Pack................................................................................................................................................................. 8
Designing a Gesture Pack ................................................................................................................................................ 9
Guidelines for Designing a Gesture ............................................................................................................................. 9
Gesture Training Overview ............................................................................................................................................10
Stage One: Creating a Pack and Adding Gestures ............................................................................................................12
Creating a Gesture Pack ................................................................................................................................................12
Pack Dashboard......................................................................................................................................................... 14
Adding and Modifying Gestures ....................................................................................................................................15
Adding a Basic Gesture ............................................................................................................................................. 15
Adding a Preview Image to a Gesture ....................................................................................................................... 17
Creating Composite Gestures ................................................................................................................................... 18
Importing Gestures ................................................................................................................................................... 20
Adding Rules ..................................................................................................................................................................21
Stage Two: Preparing Sequences......................................................................................................................................23
Adding Sequences .........................................................................................................................................................23
Recording Sequences Guidelines and Tips ............................................................................................................. 23
Importing Sequences ................................................................................................................................................ 25
Marking Sequences .......................................................................................................................................................27
Marking Sequences Guidelines and Tips ................................................................................................................ 28
Stage Three: Building the Pack .........................................................................................................................................32
Choosing a Build Mode..................................................................................................................................................32
Training Gestures Separately ........................................................................................................................................33
Initiating the Build .........................................................................................................................................................34
Understanding and Improving Results ..........................................................................................................................35
Build History .............................................................................................................................................................. 37
Tips for Dealing with Misses ..................................................................................................................................... 38
Tips for Dealing with False Hits ................................................................................................................................. 38

Stage Four: Testing the Pack.............................................................................................................................................39
Stage Five: Publishing the Pack.........................................................................................................................................44
Appendix A: Glossary ........................................................................................................................................................45
Appendix B: GAT Configuration ........................................................................................................................................47
Appendix C: Gesture Dashboard.......................................................................................................................................48
Setting Up the Viewing Pane .........................................................................................................................................50
Appendix D: Marking Sequences ......................................................................................................................................52
Playback Toolbar ...........................................................................................................................................................53
Appendix E: Statistics ........................................................................................................................................................54
Appendix F: FAQs ..............................................................................................................................................................57
Preface
This document describes the process of training Gestures using the Omek Gesture Authoring Toolkit (GAT). It
provides an overview of the basic training process and workflow, explains the different stages, and introduces
the interactive application for training basic Gestures.
Gesture Authoring Tool (GAT) Overview

What is a Gesture?
A gesture is a human body motion or pose. A motion involves the movement of some or all of the body, while a
pose is a single, static body position. Specific gestures are often defined and used in natural user interface (NUI)
applications to trigger some response in the application. To enable this, the application must be able to
recognize these gestures from among all the possible motions that a person can make while interacting with the
application. When we use the capitalized term Gesture in this document, we are referring to one of the
gestures that an application has defined as triggering an application reaction.
Training GAT to Recognize Gestures

The Gesture Authoring Tool (GAT) allows you to define new Gestures and train a software module to recognize
them. It provides a simple, graphical user interface, designed for non-technical users. You can rely on GAT to
walk you through the Gesture creation process without writing a single line of code.
In GAT, Gesture recognition is trained by analyzing recorded examples of the Gestures. This means that rather
than trying to define how parts of the body move or pose in a Gesture, all you have to do is record examples of
the Gesture. GAT then analyzes these examples using an approach called machine learning, described in the
next section.
While GAT training process greatly simplifies the work of defining and implementing new Gestures, there still is
a short learning curve involved in understanding how best to use the tool and streamline the process. We
recommend first training simple Gestures, and then going on to increasingly more complicated Gestures.
A set of Gestures trained together using GAT is called a Gesture Pack. You will need to use the Beckon SDK to
access your Gesture Pack, together with the rest of the motion tracking functionality that Beckon provides. Once
a Gesture Pack is generated, you can integrate it into your application. See the Beckon SDK Developer Guide for
detailed instructions.
Machine Learning
Machine Learning is a type of computer algorithm, which learns to identify certain patterns by analyzing
positive and negative examples of the pattern. This method has many applications. For example, using machine
learning, you can train software to identify subject categories of a given text, musical genres of recorded music
clips, or a specific persons face from among photographs of different people. In our case, we train software to
recognize certain Gestures from among many possible gestures.
In order to learn by example, a machine learning algorithm must receive as input both instances of the pattern it
is trying to identify, and instances that do not conform to the pattern. These are often referred to as positive
and negative instances. In the case of gesture recognition, a positive instance is a video clip that contains the
gesture, while a negative instance is a video clip that does not contain the gesture.
When you are working with GAT, and you mark Gestures within their sequences, what you are doing is defining
positive instances for the machine learning algorithm to work with. Negative instances are taken from randomly
selected locations throughout the unmarked portions of the sequences.
The machine learning algorithm then analyzes all of the examples according to features that a programmer
defines. In the case of Gestures, the features might include the relative joint positions of the tracked person, the
speed of their motions, and so on. By comparing these features between positive and negative instances, the
algorithm learns to identify positive instances of the defined Gestures, not only those it encounters in the
training set, but also new instances that it encounters at runtime.
Since everything the algorithm learns is derived from the instances it receives during training, its important to
represent the real world as far as possible in the set of training instances. This means that you should record
the same gesture performed by several people, from various positions in relation to the sensor. You should
record gestures performed by men, women and children, depending on who will use the application.
When the algorithm is trained well, it can achieve highly accurate gesture identification. This may take several
iterations of training, involving adding more instances, marking instances more accurately, and preventing GAT
from trying to learn from noisy data or ambiguous instances.
Basic Entities and Terms

This section describes some of the central concepts used in this guide.
Note:
See Appendix A: Glossary for a complete list of terms.
Frame
A video recording is not a continuous stream, but is in fact composed of several discrete images or frames. The
number of frames recorded per second depends on the specific sensor used. A typical frame rate for the types of
sensors used for motion tracking is 30 frames per second. The longer a video sequence is, the more frames it
contains.
Gesture
A Gesture is a unique, distinctive pose or motion of one or more body joints. This may be a static (single-frame)
pose, such as the right hand stretched forward, or a dynamic (multi-frame) motion, such as the right hand
drawing a circle.
GAT supports the following types of gestures:
Basic Gestures, which are the fundamental building blocks of GAT. These Gestures are created using GAT
training process. GAT distinguishes between three types of Basic Gestures:
Single Motion Gesture: a gesture that is completed in a short period of time, such as a salute.
Pose: a single, static body pose, such as standing with arms stretched upwards.
Continuous Gesture: an ongoing motion, such as running or spinning.
When creating a Basic Gesture, you can create its Mirror Gesture with no additional effort. A Mirror
Gesture is the mirror image of a Basic Gesture. For example you can define a jumpRight Gesture and at
the same time define a jumpLeft Gesture as its mirror image. You dont have to train a Mirror Gesture
separately GAT trains it automatically.
Composite Gestures, which are a combination of two Basic Gestures performed one after the other, in a
certain order and within a specified time range.
Gesture Pack
A Gesture Pack is a collection of Gestures that are trained and packaged together. Typically you create a single
Gesture Pack that contains all the custom gestures that your application uses. When you add a Basic Gesture to a
Gesture Pack, you must retrain all of the gestures in the pack. This is because GATs machine learning algorithm must
take into account all of the types of gestures it can encounter. A positive instance of a new Basic Gesture also serves
as a negative instance of other Basic Gestures, thereby affecting their training process.
Sequence
In order for GAT to be able to recognize a Basic Gesture, you need to teach it what the Gesture looks like. You do
this by recording short video segments of people performing the Gesture. A video segment, together with its tracking
data produced by the Beckon SDK, is called a Sequence. A sequence is a recording of a person performing the
Gesture, with a length of up to 60 seconds.
Once you have recorded sequences, you need to indicate which frame or frames contain the required Gesture. This is
called Marking.
Clip
You can mark three types of segments or Clips:
Gestures - Each performance of a given Gesture in a sequence is called an Instance. Typically, there
are several instances of a Gesture in a sequence. You must mark all of the Gesture instances in the
sequence to ensure correct learning.
Bad Data - You can assist the training process by marking faulty tracking. This refers to places in the
sequence where you see that identified joint locations are incorrect in relation to the true joint
positions. When you mark the faulty frame or frames as Bad Data, GAT will ignore these portions
when learning Gestures.
XNegative - Sometimes you may encounter segments which are confusing to the learning process, as
they contain a movement or pose that resembles one of the defined Gestures but should not be
identified as a Gesture. In this case, you can mark the segment as XNegative, which ensures that this
specific segment will be taken as a negative instance for all Gestures.
The marked segment of a sequence which represents a specific event (Gesture, Bad Data or XNegative) is called
a Clip.
Building a Pack
Once all Gestures are created in the Pack, and sequences are marked for each Gesture, you can build the Pack. During
the build process, GAT learns from the examples you have provided and creates Classifiers for the Gestures. A
Classifier is a software component that identifies a given segment as the target Gesture, or rejects it.
The build process not only creates Classifiers, but also tests their quality. To do this, GAT takes a random subset
of the marked sequences, and uses them to train the Classifiers. Then it tests them on the rest of the marked
sequences. This means that GAT runs the Classifiers on the marked sequences that were not used for training,
and produces a Build Report, which shows what Gestures were identified in this stage.
The test results include:
The number of Hits (true positives in machine learning terms), which are Gestures that were detected
correctly.
The number of False Hits (false positives in machine learning terms), which are cases where a Gesture
was identified although it was not performed.
The number of Misses (false negatives in machine learning terms), which are cases where a Gesture was
performed but was not identified.
Designing a Gesture Pack

A Gesture Pack groups together a set of Gestures, which will be built and published together. You can create one
or more Gesture Packs for your application. The main principle to follow, when deciding which Gestures to
group into a Pack, is that Gestures that tend to appear at the same stage of the application should be in the
same Pack. This is because its important to train GAT to differentiate among these gestures by comparing their
attributes at the Pack build stage, so that it identifies them accurately at runtime.
Before defining a Gesture Pack in GAT, you should first conceptualize each Gesture. This means that you should
define the motions that should be detected and the conditions in which the motions are performed. There are
several attributes and conditions of the Gestures that should be considered and documented when defining
them. These are detailed in the next section.
Guidelines for Designing a Gesture
Name: The Gesture name should be unique and comprehensive, for example, rightFrontPunch".
Note:
A Gesture name must be unique within a single Pack. The same Gesture name may appear in
more than one Pack. However, if several Packs are used in the same application, there may not
be duplicate Gesture names among these Packs.
Granularity: Each Basic Gesture should be short and simple. Avoid defining combinations of more than one
basic movement as a Single Motion Gesture. Optionally, two simple Gestures can be combined into a
Composite Gesture. For example, instead of training a leftRightFrontPunch Gesture (which includes a left
front punch and then a right front punch), you can train a rightFrontPunch and a leftFrontPunch, and
then you can easily define any combination of these punches.
Participants: Determine whether the target participants include men, women and children. Make sure that
the video clips show several different people from among the target groups, performing the relevant
Gestures. The Gestures should be performed several times, with various people in various locations.
Description: When defining a Gesture, make sure its description is clear and specific. For example: The
right hand starts from the right side of the body, at the height of the head or the torso, and moves fast and
straight towards an opponent, who stands in front of the player. A clear, detailed description will help
when recording video segments.
Conditions: Detail the conditions and situations under which the Gesture will be performed, regarding start
and end positions, speed of motion, and any other factors of the gesture. For example: A punch can be
executed from a neutral pose (while the right hand is at the side of the body, in front of the torso, or close
to the right shoulder), or from a defense pose (where the hands defend the face). A punch can be executed
from a standing position, while dodging an opponents hit, or while stepping forward. Also, if there are
conditions in which the Gesture will not be performed, describe these as well.
Avoid confusion: If there are similar motions or poses that should not be detected as the Gesture, describe
them as well. For example: A slow movement of the hand should not be detected as a front-punch. A
hook-punch should not be detected. If there such confusing motions or poses, make sure to record
them as part of the input to the Build process. Mark them as XNegative clips, to ensure that GAT does not
learn them as Gesture instances.
Gesture Training Overview

The overall workflow for training Gestures is as follows:
Create New Pack
Add Sequences (Record or

Import)
Create All Gestures
Mark Sequences
Pack Rules and Settings
Build
Refine
Analyze/Live Test
Publish
Figure 1: Gesture Pack Training Process Diagram
Note:
Before building the Pack with all Gestures enabled, you can also choose to build one Gesture at
a time. See Training Gestures Separately for more details.
10
These are the stages of creating and building a Gesture Pack:

Stage One: Creating a Pack and Adding Gestures: Create a new Gesture Pack and add Gestures.
Stage Two: Preparing Sequences: Add video sequences for each of the Gestures in the Gesture Pack.
Record sequences of the Gesture.
Add the sequences to the Gesture in the Pack.
In each sequence, mark Gesture instances, Bad Data and XNegative instances.
Stage Three: Building the Pack: Build the Pack. GAT creates a Classifier for each gesture.
Stage Four: Testing the Pack: Test and improve the Pack:
Evaluate by observing Gesture detection in real-time.
Check the performance statistics in the Build Report.
If the Pack performs well, you are done!
If you want to improve performance, return to stage 2, add sequences and improve marking, then build and
test.
Stage Five: Publishing the Pack: Create the software module that you will integrate into your application.
11
Stage One: Creating a Pack and Adding Gestures

First, conceptualize and define the Gestures you want to include in your Gesture Pack, using the guidelines
described in Designing a Gesture Pack.
Then launch GAT tool and start creating your Gesture Pack.
Creating a Gesture Pack

You can create a new Pack or add new Gestures to an existing Pack. When you create a new Gesture, its related
files are located in the Pack folder by default. You can choose to change the default location of the Gesture files
(see Appendix B: GAT Configuration).
To create new Pack:
1.
Open GAT tool and click New Pack.
Figure 2: Welcome to GAT
The New Pack dialog appears.
Figure 3: New Pack Dialog
12
1.
In the Name text box, type a name for the new Pack.
Note:
Gesture Pack names can only contain lower-case letters, upper-case letters, and underscores.
2.
Click Create.
The Pack Dashboard window appears. For a new Pack with no Gestures defined, the Pack Dashboard looks
like this:
Figure 4: Pack Dashboard for an Empty Pack
Note:
To rename a Pack after it is created, on the File menu, click Rename Pack. Do not rename Pack
files manually as this could cause the Pack to malfunction.
13
Pack Dashboard
The Pack Dashboard displays all of the Gestures defined in the Pack. From this window you can add and modify
Gestures in the Pack.
Figure 5: Pack Dashboard
To display the Pack Dashboard:

3.
Open GAT to display the Welcome screen.
Figure 6: Welcome Screen
4.
You can do one of the following:
Create a new Pack (see Creating a Gesture Pack).
14
Click Open Pack to display the Open window. Select the .gpp file you want to open.
Click one of the Pack names in the Recent Packs list.
The Pack Dashboard appears with all the Gestures defined in the pack.
Adding and Modifying Gestures

Adding a Basic Gesture
A Basic Gesture is a Single Motion, Pose or Continuous Gesture (see Gesture). The first Gestures you define in
your Pack are Basic Gestures. Once they are defined, you can create Composite Gestures, composed of two
Basic Gestures.
To add a Basic Gesture to a Pack:
1.
From the Gesture menu, choose Add Gesture.

The New Gesture dialog appears.
Figure 7: New Gesture Dialog
2.
In the Name text box, type a name for the new Gesture.
Note:
Gesture names can only contain lower-case letters, upper-case letters, and underscores.
3.
From the Type drop-down list, select the type of Basic Gesture you want to define: Single Motion, Pose or
Continuous.
4.
Optionally, you can choose to create a Mirror Gesture for this Basic Gesture. (See Gesture for an
explanation of Mirror Gestures). When you check the Create Mirror Gesture box, the Mirror Name area is
displayed. Type a name for the Mirror Gesture.
Figure 8: New Gesture Dialog with Mirror Name

15
5.
Click OK.
The Pack Dashboard displays an icon for the new Gesture.
Figure 9: Gesture Icon with Mirror gesture
Note:
If you added a Mirror Gesture, the Mirror Gesture icon
Mirror Gesture.
appears next to the name of the
Note:
Defining a new Mirror Gesture requires a rebuild of the Gesture Pack, but deleting an existing
Mirror Gesture does not.
16
Adding a Preview Image to a Gesture

When you create a new Gesture, its icon is displayed in the Pack Dashboard with the default Preview Image:
Figure 10: Default Preview Image
You can change this image to a different image that illustrates the actual Gesture you are defining. The following
image files types are supported: JPEG, TIFF, PNG, GIFF.
To add a preview image:
1. Right-click the icon of the Gesture to which you want to add the image, and choose Add Preview Image.
The Open window appears.
2.
Select the image file type from the list, and browse to the location of the file you want to use as a preview
image.
3.
Click Open.
The Gesture icon now displays the image you chose.
Figure 11: Preview Image
17
Creating Composite Gestures

You can combine two Gestures into one Composite Gesture. When creating the Gesture, you define the order of
the two Gestures. You can also configure the minimum and maximum time gaps (in numbers of frames) that can
occur between the Gestures. The maximum time gap determines whether the sequence of Gestures will be
identified as the Composite Gesture or as two separate Gestures. The minimum time gap is useful for preventing
a single gesture from being identified as a Composite Gesture.
To add a Composite Gesture:
1.
From the Gesture menu, choose Add Gesture.

The New Gesture dialog appears.
Figure 12: New Gesture Dialog

2.
In the Name textbox, enter a name for the new Gesture.
3.
From the Type drop-down list, select Composite Gesture.

The Composite Gesture dialog appears.
Figure 13: Composite Gesture Dialog
4.
From the first Gesture dropdown list, select the first Gesture of the Composite Gesture.
5.
In the Minimum Gap textbox, type the minimum gap between the two Gestures in frames.
6.
In the Maximum Gap textbox, type the maximum gap between the two Gestures in frames.
7.
From the second Gesture dropdown list, select the second Gesture of the Composite Gesture.
18
8.
Click OK.
The Pack Dashboard displays the new Composite Gesture icon.
Figure 14: Composite Gesture Icon
Note:
When you create a Composite Gesture from Gestures that are already trained, you do not need to
rebuild the Gesture Pack. Deleting a Composite Gesture also does not require a rebuild.
19
Importing Gestures
You can import a Gesture from one Pack to another. When you import the Gesture, you import all its definitions as
well as sequences associated with the Gesture and their markings.
To import a Gesture:
1.
Right-click anywhere in the Pack Dashboard and choose Import Gesture.

The Import Gesture dialog appears.
Figure 15: Import Gesture Dialog
2.
Click the Browse button

to browse to the location of the Pack project .gpp file from which you want to
import the Gesture, select the .gpp file and click Open.
A new area appears below the textbox, with a list of available Gestures in the Pack.
Figure 16: Import Gesture Dialog with Gesture List
3.
Check the Gesture or Gestures you want to add and click Add.
The new Gestures are added to the Pack Dashboard.

Note:
Composite Gestures cannot be imported from another Pack. Instead, import the relevant Basic
Gestures and create a Composite Gesture in the new Pack.
Note:
If the Create a copy of sequences when I add them to the pack option is checked (see Appendix B:
GAT Configuration), the imported sequence files will be copied to your Pack folder.
20
Adding Rules
A Rule is a condition or constraint that must be met in order for a Gesture to be recognized, in addition to its
motion properties. The following Rule is supported:
Refractory Period Rule - This Rule dictates that a Gesture will only be recognized once within a specific number
of frames. A new instance of the Gesture is detected only if the specified number of frames has passed since the
previous occurrence of the Gesture.
To define a Rule:
1.
In the Pack Dashboard, from the Pack menu, choose Pack Rules or click the Rules button
the Gesture Pack Rules window.
, to display
Figure 17: Gesture Pack Rules Window
Note:
If you define a rule for a Gesture that has a Mirror Gesture, the rule will automatically be applied to
the Mirror Gesture as well. You can override this behavior by explicitly adding a different rule for the
Mirror Gesture.
Note:
Creating a new rule or updating an existing rule does not require rebuilding the Gesture Pack.
Deleting a rule does require a rebuild.
21
2.
Click the
button to add a Rule.
A line appears on the display, allowing you to define the Rule.
Figure 18: Define Rule
3.
From the drop-down list, select the Gesture for which you want to define the Rule.
4.
Enter the number of frames for the delay period.
Figure 19: New Rule with Data
5.
Click OK.
Once the Pack is built, this Rule will affect the way the Gesture is recognized.
22
Stage Two: Preparing Sequences

The input to GAT consists of video sequences of people performing the target Gesture as well as other actions.
You will need to prepare sequences for each of the Basic Gestures in the Pack. This includes two steps:
1.
Adding sequences by recording new sequences or importing existing sequences
2.
Marking the sequences
Adding Sequences
You can record new video sequences or import pre-recorded sequences. A GAT sequence is composed of two
files: a file that contains the feed from the sensor, and an additional file that contains the Beckon motion
tracking information (the .seq file). Both files should be located in the same folder and have the same name with
different suffixes.
By default, the sequences recorded by GAT are saved in the Pack Folder. You can change the target folder for the
recorded sequences (see Gesture Pack Locations).
Recording Sequences Guidelines and Tips

When recording sequences, its important to cover a broad range of movements and people, which accurately
represent the input your application will receive at runtime. Follow these guidelines to produce a correct,
representative set of sequences.
Short sequences: record short sequences (typically, sequences should be no longer than 30 seconds, with
no more than 4 Gesture instances).
Single person: Only one person should be in the sensors field of view when recording sequences.
Person variation: record several different people (one at a time) of different sizes.
Location variation: record people in different locations and angles, relative to the sensor.
Clarity: Try to record typical, unambiguous instances of the Gesture. In particular, it should be obvious
when the Gesture begins, and when it ends.
Demonstrate XNegatives: In addition, if there are movements that look similar to one of the Packs
Gestures, but should not be identified as a Gesture, record these too and mark them as XNegative.
23
Note:
In order to record sequences, you need to connect a sensor to your computer and make sure it is
operating properly.
To record a sequence for a Gesture:
1.
Double-click the Gesture to which you want to add the sequence. The Gesture Dashboard is displayed.
2.
Click Add Sequence, choose Record New and wait for the sensor to initialize.
The Viewing Pane displays the sensor input.
Figure 20: Add a New Recording
3.
In the Recording dialog, you can click the Options arrow to display the recording options.
Figure 21: Recording Options
In the Options area, you can:

Check the Countdown box to add a 5 second delay before recording begins.
In the Time Limit textbox change the length of the recording in seconds (between 3 and 60 seconds).
Click the Browse button to select the target directory, where the sequence files are saved.
24
4.
When you are ready, click the Record button
The Record button changes to the Stop button

which you can click to stop the recording. The
recording automatically stops when the selected time limit is reached. Each sequence is saved in a different
sub-directory inside the target directory, with an automatic name (seq1, seq2, etc.).
The new sequence appears in the left pane of the Gesture Dashboard. It is automatically associated with
the Gesture for which it was recorded.
Figure 22: New Sequence
Note:
When you record a new sequence, GAT will name it automatically as seq followed by a number.
Sequences are numbered in order of creation, starting from 0.
After a sequence is created, you can right-click on it and choose Rename to rename it.
Sequence names can only contain lower-case letters, upper-case letters, underscores and digits.
Importing Sequences
You can also import pre-recorded sequences for training Gestures. You can import a single file or a folder with
several files.
By default, importing sequences does not copy the sequence files into the Pack folder. If you want to copy the
files, activate this option in the Options window (see Appendix B: GAT Configuration). A sequence file that
already exists in the Pack will not be recopied when importing sequences.
If the tracking file (.seq) file is missing (for example if the sequence was not recorded using GAT, or if the .seq file
was created using an older version of the Beckon SDK) you have the option of retracking the sequence. This
means that GAT processes the sensor sequence and creates tracking information in a new .seq file.
25
IMPORTANT!
Importing a folder containing a large amount of sequences can take a long time, especially if the
sequences need to be retracked.
If a sequence needs retracking after it has been added, GAT will indicate this on the specific sequence in the
Sequences pane on the left of the Gesture Dashboard.
To retrack a sequence, right-click on the sequence and click Retrack.
Figure 23: Retracking a Sequence

To import a sequence:
1.
Open the Pack Dashboard.
2.
Double-click on the Gestures icon to open its Gesture Dashboard.
3.
Click Add Sequence, and choose From File.

The Open dialog appears.
4.
Browse to the sequence file you want to import and click Open.
The Sequence is added to the Gesture Dashboard.
You can also import several sequences at once, if they reside in the same folder.
To import a folder:
1.
Open the Pack Dashboard.
2.
Double-click on the Gestures icon to open its Gesture Dashboard.
3.
Click Add Sequence, choose From Folder.

The Browse for Folder dialog appears.
4.
Browse to the folder you want to import and click Open.

All the sequences in that folder appear in the sequence pane of the Gesture Dashboard.
26
Marking Sequences
Once sufficient sequences have been added, you must mark the Gesture instances in the sequences. While you
play back the sequence in GAT, you can mark a clip the frames in which the Gesture appears.
The number of clips or instances required to train and test a Gesture can vary, depending on the complexity and
the uniqueness of the Gesture. Single Motion gestures require more instances than Pose or Continuous
Gestures.
Note:
As a rule of thumb, before building a Pack, make sure that you have:
1. 30 marked clips for each Single Motion Gesture.
2. 10 marked clips for each Pose and Continuous Gesture.
You may have to add more clips after the first build, to improve results.
Here are some numbers taken from the real training scenario of a RightClick gesture:
15 sequences
40 marked Gesture clips
15 XNegative clips
In addition to marking Gesture clips, you can also mark segments as Bad Data and XNegative.
Bad Data refers to places in the sequence where you see that identified joint locations are incorrect in relation
to the true joint positions. When you mark the faulty frames as Bad Data, GAT will ignore these portions when
learning Gestures.
XNegative refers to motions that are similar to one of the Gestures in your Pack, and could be confused with it,
but should not be detected as a Gesture. For instance, if you see that hopping on one foot is sometimes
confused with jumping on both feet, mark the hopping-on-one-foot sequences as XNegative clips (assuming
hopping-on-one-foot is not a Gesture in the Pack).
IMPORTANT!
XNegative clips are learned as negative instances of all the Gestures in a Pack (they do not relate to
one specific Gesture). Therefore, take care not to mark an XNegative clip on a motion that should be
recognized as one of the Gestures in the Pack, because then GAT will try to learn it as both a positive
and a negative instance.
Note:
If a single sequence contains several different types of Gestures, add it to each Gesture in turn. Then
mark the clips of each Gesture within the Gesture Dashboard.
Note:
If a sequence is not manually marked with any type of clip (Gesture, XNegative or Bad Data) it will
not be used when the Pack is built. A sequence which has automatically-marked Bad Data clips will
still be excluded unless it has user-defined clips.
27
Marking Sequences Guidelines and Tips
Completeness: Each and every instance of the Gesture must be marked. If you do not mark a correct
instance of a Gesture, its frames could potentially be used as a negative instance, causing GAT to learn from
incorrect input and harming its accuracy.
Consistency: When marking instances of a specific Gesture, be consistent about marking the beginnings and
ends of the Gesture. Each instance should contain the same motion (and not only part of the motion).
Partial Gestures: All unmarked frames can potentially be used as negative instances. If only part of a
Gesture appears at the beginning or end of a sequence, it should not be marked as a positive instance, since
only part of the Gesture is available. On the other hand, it should not be considered a negative instance
either. In this case, mark the partial Gesture instance as Bad Data, so that it will be ignored by the learning
process.
Accurate Marking: make sure to mark the start and end of a Gesture accurately, and not to include frames
in the clip that dont belong to the Gesture. When in doubt, its better to make a clip shorter rather than
too long.
No Overlap: make sure that there is no overlap between clips of any type (Gesture, XNegative or Bad Data).
If the same sequence was added to more than one Gesture, take special care not to mark Gestures that
overlap with other Gestures clips in the sequence.
3D View: We recommend working in 3D View when you are marking clips. In this view, you can adjust the
angle in which you view the tracked skeleton, so that you can optimally detect the Gestures start and end.
For instance, if a person stretches their arm forward and back, its easier to see the boundaries of this
motion from a side view rather than a frontal view.
XNegatives: You should use XNegative clips to mark specific motions that could be confused with one of the
Gestures you are training. Usually you will detect these segments when they are identified as false hits
during the build. As with marking Gesture clips, its important to mark XNegative clip ranges accurately, and
its better to make a clip shorter rather than too long.
28
Bad Data: Gestures should only be marked on areas in the sequence where the tracking is accurate.
Inaccurate joint data reduces the quality of the Pack. Therefore, you should pay attention to the Low
Confidence indicator, which appears as a red underline in the sequence playback and automatically marks
low-quality segments as Bad Data, as in the example below:
Figure 24 : Sequence with Auto-Marked Bad Data
Do not mark Gestures on segments which GAT has automatically marked as Bad Data. Additionally, if there
are segments where you can see that the joint tracking is inaccurate, even if GAT has not marked them as
such, you can explicitly mark them as Bad Data.
29
To mark a Gestures sequences:

1.
In the Pack Dashboard, double-click the Gesture icon.

The Gesture Dashboard appears, with the Gestures sequences in the left pane.
2.
Double-click the sequence you want to mark.

The sequence starts playing in the Video panel (See Appendix C: Gesture Dashboard for a detailed
description of the parts of this window).
Figure 25: Sequence Playback
Tip:
You can decrease the playback speed by clicking the Speed arrow and selecting or . It is easier to
mark clips in slow motion.
3.
You can use the Playback toolbar icons

playback of the sequence (see Playback Toolbar for a detailed description)
4.
Use the Mark In and Mark Out buttons to mark the beginning and end frames of a clip.
to control the
30
Figure 26: Mark In/Mark Out Options
Click the Mark arrow to display the clip options and click the type you want to mark. Choose:
5.
Gesture: to mark a clip as a positive Gesture instance.
Bad Data: to mark a clip as data to be ignored.
XNegative: to mark a clip as a segment that should not to be identified as a Gesture (although it is
similar enough to one of the Gestures to be confusing).
Make sure to mark all the occurrences of the Gesture. You should also take care to mark as XNegative all events
that might be mistakenly detected as a Gesture.
Figure 27: Marked Sequence
31
Stage Three: Building the Pack

Choosing a Build Mode
Once there are enough marked clips for each of the Basic Gestures, you can build the Pack. At this stage, GAT
applies its machine learning algorithm (see Machine Learning) to create Classifiers for each Gesture.
When you build a Pack you can choose the build mode. This may be one of the following values:
Fast
Normal
Slow
Although the names of the build modes appear to indicate their speed, they actually differ in the number of
learning iterations that each one performs. The Fast mode performs one learning iteration, the Normal mode
performs three iterations, and the Slow mode can perform a varying number of iterations, up to half the
number of sequences in the pack. The modes also differ in the number of sequences that they use to train the
Classifiers. Therefore, the quality of results that each mode produces may differ. There is no absolute rule as to
which mode is always fastest in practice, or which mode produces the best results.
Note:
As a rule of thumb, select the Fast build mode to start with. Use this mode to examine the initial
results and improve your sequences and their marking. Then you can try the Normal and Slow
modes, to see which mode produces the best results.
32
Training Gestures Separately

In our experience, it is often effective to train Gestures separately before training them all together. You can
achieve this by excluding all Gestures but one from the Build.
The reason this method is useful is that most of the work, which follows the analysis of the initial test results,
involves improving the training sequences for the Gesture. This means adding and marking more sequences,
improving marking of existing Gesture clips, marking more Bad Data and XNegative clips, and so on. Its faster to
build a Pack containing a single gesture, and easier to focus on the results of a single Gesture Classifier. Once
youre satisfied with the quality of the sequences, clip marking, and results of each stand-alone Gesture
Classifier, you can move on to the next stage of building the Pack with all Gestures included.
Note:
Building Gesture Classifiers separately is optional. If this method is used, it is only a preliminary stage
before building the entire Pack. Ultimately, GAT must train on all Gestures together, so that it can
learn to differentiate among them.
To exclude a Gesture from the Build:

Select the Gesture in the Pack Dashboard, and from the Gesture menu choose Exclude Gesture from Build.
To re-include a Gesture in the Build:
Select the Gesture in the Pack Dashboard, and from the Gesture menu choose Include Gesture in Build.
33
Initiating the Build

To build a Pack:
1.
On the menu bar, click Pack.
2.
In the Pack menu, click Build Settings, then click the build mode you want.
Figure 28: Select Build Mode
3.
In the Pack menu, click Build.

The Preparing Build message appears.
4. The build process can take from a few minutes up to hours depending on the number of sequences in the
Pack. Building the Pack creates a build sub-directory in the Pack directory.
When the build is finished, GAT displays the test scores for the Basic Gestures.
Note:
By default, GAT will try to build several classifiers in parallel, to shorten build time. If your Gestures
contain large numbers of sequences, this mode may fail due to lack of memory. If this happens, you
will see this message:
In this case, click Build Serial to build in serial mode. This mode takes longer but requires less
memory.
34
Understanding and Improving Results

GAT produces a single, overall build score, computed by testing the Packs performance on all the test
sequences. This will give you a high-level indication of the Packs quality.
In order to get a more detailed breakdown of the Packs performance, from the Pack menu choose Show Build
Report. The Build Reports window appears.
Figure 29: Build Report Window
35
The following table summarizes the Build Reports information. The numbers in the first column refer to the areas
marked in Figure 29: Build Report above.
Table 1: Build Report Items
Name
Description
Score
The total score of the Gesture, on a scale of 0-100. (See note below
regarding the calculation of the score value).
Hits
The number of correctly detected Gesture instances.
False Hits
The number of events detected as Gestures, which were not marked as

Gestures in the sequences.
Misses
The number of events that were marked as Gestures but were not detected.
Sequence
The sequence files used in the build.
False Hits/
Misses
For each of the sequences, the number of False Hits and Misses and specific
frames where they were detected.
Note:
The score value is calculated according to this formula:
Score = (# Hits) / (# Hits + # Misses + # False Hits)
Note:
To analyze an error in detection, double-click on the error frame number in the False Hits or Misses
column. The Gesture Dashboard opens, showing the sequence and frame of the error.
Figure 30: Double-Click on Error Frame in Build Report
36
Build History
In addition to the results of the current build, GAT saves the build results of the previous 10 versions.
To see the results of previous builds:
1. Click the Show Build Reports button
The Build Reports window appears.
in the bottom-right corner of the Pack Dashboard.
Figure 31: Build Reports History Window
2. Click the Build Reports History tab. A list of the last 10 build date-times is displayed.
3. Click on the arrow to the right of each builds date-time to see its results.
Note:
If you want to retrieve the Gesture Packs classifiers from older builds, you can find them in subfolders under the Gesture Packs Build folder. The sub-folders are named according to the date and
time of the build, as they are displayed in the Build Reports History window.
37
Note:
The Build stage uses some of the marked clips for training the Classifiers, and the rest for testing
them. The Build Report refers only to the test set results. The division between train and test sets is
constant, so that while youre improving results, you can examine the changes on the same set.
If there are Gestures in the Pack that were not included in the build, they are also listed.
Examine each of the falsely identified events, review their sequences, and see whether their clip marking should
be adjusted. Below are some tips on how to do this.
When you are satisfied with the build results, you can go on to test the Pack on live input from a sensor or on a
recorded video sequence.
Tips for Dealing with Misses

Misses are events that were not identified as Gestures even though they were marked.
Here are some error scenarios and possible solutions.
Add and mark more sequences with Gesture instances, so that GAT has more examples to learn from.
If you notice a tracking problem, you can use the Bad Data option to exclude the specific frame or
frames from the build.
If the beginning or the end of a marked event is not accurately marked, you can drag either of the
borders of the clip to fine-tune the marking.
Tips for Dealing with False Hits

False Hits are events that were identified as Gestures even though they were not marked.
Here are some error scenarios and possible solutions.
If a particular movement is identified incorrectly as a Gesture, you can use the XNegative option to mark
the specific segments as explicit negative instances. (Remember: XNegative clips will be used as negative
instances of all Gestures, so make sure you dont mark a valid Gesture as an XNegative clip). You can
also record new instances of this confusing movement, and mark them as XNegative clips as well.
If you see a False Hit within the range of a marked Gesture instance, this is due to the fact that GAT
refines the marked Gesture ranges during the build. In this case, we recommend correcting the start and
end positions of the clip manually, so that only the Gesture is included.
Note:
Bad Data clips are ignored in the Classifier training stage, but False Hits and Misses may occur in
faulty data segments during a Live Test. This represents a real-life scenario and can occur at
application runtime as well.
38
Stage Four: Testing the Pack

In the Live Test window, you can observe how the Pack detects Gestures at runtime.
You can test the Pack on live input from a sensor or on a recorded video sequence. While observing the Packs
behavior with the new input, write down the problems in as much detail as you can, noting the reasons for the
failures. Try to understand which movements cause the Pack to produce either False Hits or Misses. Perhaps the
player is standing in a particular area or angle with respect to the sensor, which is not well represented in the
training sequences.
Note:
When testing a Pack on a pre-recorded sequence, avoid frequent presses on the Play/Pause button,
and do not move the playback thumb
. These actions disrupt the timing of the sequences
input and can cause incorrect Gesture recognition.
You can now refine the Pack by adding marked training sequences according to the problems you detected and
rebuilding the Pack.
Figure 32 describes the fine-tuning feedback cycle.
Figure 32: Testing Feedback Cycle
39
Figure 33: Test Window
The following table summarizes the Test window parts and controls:
Table 2: Test Window Items
Option
Use to
Gestures and Alerts
Display detected Gestures and Alerts, including name and frame number
Filters pane
Select the Gestures and Alerts to display
Video panel
Display the live / play-back video
Playback Timeline
Displays the playback progress with any marked clips, and the Confidence
indicator.
Input source
Allows you to choose the live sensor or a sequence file as input.
Play
Start sequence playback.
40
To activate live testing:

4.
In the Pack Dashboard click Live Test.

The Live Test window appears.
5.
In the Filters pane, click Gestures to display the Gestures tab and select the gestures you want to detect.
You can click the arrow next to the color and select a different color for the display of the detected Gesture.
The Gestures you selected will appear during the test on the timeline and in the Gestures and Alerts pane.
Figure 34: Gesture Filters
Note:
Gestures whose classifiers have not been built cannot be viewed in Live Test.
6.
In the Filters pane, click Alerts to display the Alerts tab, and select the Alerts you want to display.
The Alerts you selected are displayed in the Gestures and Alerts pane.
Figure 35: Alerts Filter
7.
You can now do one of the following:
Test live sensor input. Make sure you have a sensor connected, then click Live Camera. After the
sensor initializes, the video panel displays the input from the sensor. Have a person in view of the
sensor perform the Gesture, and observe the detection.
41
Test from a video sequence: Click Sequence, the click the Browse
location of the sequence. Click
button to browse to the
to start playback and observe the Gesture detection.
42
Each time a Gesture is detected, the event is marked on the timeline in the color selected for the specific
Gesture.
Figure 36: Detected Gestures
The Gestures and Alerts pane displays occurrences of the detected Gestures and Alerts, according to the
filters selected in the Filters tabs.
Figure 37: Test Window Gestures and Alerts Pane
You now need to analyze the results and if necessary, add or modify training sequences according to the
problems you detected.
43
Stage Five: Publishing the Pack

Once you have improved the build results to your satisfaction, you can publish your Gesture Pack. In the Pack
Dashboard, click Publish to create the files and folders you need to use in your application.
See the Beckon SDK Developers Guide for instructions on how to implement Omek Gesture Recognition in your
application.
Note:
If you want to use a Gesture Pack in the Beckon Viewer, please retrack all of its sequences in the
Beckon Viewer. This is so that the new Gesture information will be added to these sequences.
44
Appendix A: Glossary
Bad Data
A marked segment of a sequence that will not be used when building the Pack. This option is used to mark faulty
tracking data or partial Gestures.
Classifier
A software component that identifies a given motion as the target Gesture, or rejects it.
Clip
A marked segment of a sequence which represents a specific event (Gesture, Bad Data, XNegative).
False Hit
A segment that was identified as a Gesture but not marked as such.

Gesture
A unique, distinctive motion or pose of one or more body joints. This may be a static pose, such as the right
hand stretched forward, or a motion, such as the right hand drawing a circle.
Gesture, Basic
One of the following types of Gestures:
Single Motion Gesture
Pose Gesture
Continuous Gesture
Gesture, Composite
A Gesture that is defined as a combination of two Basic Gestures, performed in a specific order within a
specified time range.
Gesture, Mirror
A Gesture that is a mirror image of a Basic Gesture. For example, the Mirror Gesture of a jumpRight Gesture
would be a jumpLeft Gesture.
Hit
A Gesture that was both marked and detected correctly.

Instance
One occurrence of a given Gesture in a sequence. There can be several instances of a Gesture in a single
sequence.
45
Miss
A segment that was marked as a Gesture but not identified as such.

Pack
A collection of Gestures that are trained and packaged together.

XNegative
A pose or motion that is similar to one of the defined Gestures, but should not be identified as a Gesture.
XNegative examples are used to train the Classifier to reject such instances.
Sequence
A short video clip, which includes the tracking data as computed by the Beckon SDK.
46
Appendix B: GAT Configuration

GAT supports various configuration options. To set these options, in the Pack menu, click Options.
Figure 38: GAT Setup Options
These are the options you can configure:

Gesture Pack Locations
Sequences Folder: The default location for sequences recorded by GAT.
Build Folder: The default location for the classifier files that GAT produces. Older build files are also stored
here, under sub-folders that are named according to the date and time of the build.
Sequence Handling
Create a copy of sequences when I add them to the pack: When checked, sequences imported into the
Pack are copied into the Pack folder. This option is turned off by default.
Clear all previous marking when I create a copy of the sequences: When checked, clip markings on the
imported sequences are removed from the copied versions. This option is turned off by default.
Startup
Show Step-by-Step Guide: When checked, the Step-by-Step Guide is displayed by default.
Open Last Pack: When checked, when GAT is opened, it automatically opens the last Pack that was edited.
This option is turned off by default.
47
Appendix C: Gesture Dashboard

The Gesture Dashboard allows you to manage and mark the Gestures sequences.
Figure 39: Gesture Dashboard (with Step-by-Step Guide hidden)
48
The following table summarizes the parts and controls of the Gesture Dashboard.
Table 3: Gesture Dashboard Items
Description
Use to
Pack > Gesture
Select a Gesture to work on.
Add Sequence
Record or import sequences.
Sequences Pane
Display all sequences added to the Gesture. The right-click menu enables
you to rename a sequence, delete a specific sequence or all sequences,
retrack a specific sequence or all sequences, clear all markings or display
statistics.
Viewing Pane Setup
Set up the viewing panes 2D, 3D and overlays views (see Setting Up the
Viewing Pane).
Video Pane
Display live sensor input or play back sequences.
Zoom In / Zoom Out
Zoom in or out on the video display.
Display / Hide Arrow
Display/hide the Step-by-Step guide.
Playback Timeline
View playback progress. Click any location to go forward/backward in the

sequence playback.
Marker
Mark the current location in the sequence
Navigate Playback
Timeline
Zoom in/zoom out and scroll.
Playback Toolbar
Control playback.
Mark In / Mark Out
Mark Gestures, Bad Data and XNegative clips on the sequence.
Speed
Set the playback speed to one of these values: , ,1, 2, 4.
Go
Go to a specific location in the sequence.
49
Setting Up the Viewing Pane

The Viewing pane can be set to display one of the following views:
2D View: The 2D view displays information in a 2-dimensional space. This includes:
The sensor feed: RGB or depth images
Tracking info (bone and joint overlay)
3DView: The 3D view displays information in a 3- dimensional space. This includes:
Tracking info (bone and joint overlay)
The Viewing pane can display additional optional information. The following table lists the overlays available in
the 2D and 3D Views.
Table 4: 2D View/3D View Overlays
Overlay
Description
Available in
RGB
Displays the RGB color data from the sensor sequence.

If the sensor does not support RGB, the infra-red (grayscale) image is displayed.
2D
Depth
Displays the depth image from the sensor or sequence.
2D
Show Joints
Displays joints as colored dots. You can turn joint display on and off, and also
display only a subset of joints that you select in the Select Joints window.
2D/3D
Show Bones
Displays bones as colored lines. You can turn bone display on and off.
2D/3D
Joint Info
Displays the rotation and translation of the joints
2D/3D
Select Joints
Opens a window that allows you to select the joints you want to display in the
overlay.
2D/3D
50
To select joints in 2D view/3D view:

1. From the Overlays dropdown list, choose Select Joints.
The Select Joints image appears, with the currently selected joints displayed in orange.
Figure 40: Joints Image
Figure 41: Joints List
2.
Click each joint to select/de-select it. You can also click Select All to select all joints or Clear to de-select all
joints.
3.
You can also click Joints List to display a list of the joints. Click a joint name in the list to select/de-select the
joint.
Note:
Selecting joints in the 2D/3D view only affects their display. It does not affect the set of tracked
joints.
51
Appendix D: Marking Sequences
Figure 42: Sequence Playback

Table 5: Sequence Playback Options
Options
Use to
Sequence
Select a sequence to mark.
Video Panel
Display the video of the sequence.
Playback Timeline
Displays the playback progress with any marked clips, and the Confidence
indicator.
Navigate Playback
Timeline
Zoom in/out and scroll left/right on the timeline.
Playback Toolbar
Control the playback of a sequence.
Mark In / Mark Out
Mark clips as Gesture, Bad Data, or XNegative.
Speed
Use to set the playback speed to one of these values: , ,1, 2, 4.
Frame
Enter a frame number and click Go, to skip to that frame in the playback.
52
Playback Toolbar
You can use the Playback toolbar icons
playback of a sequence.
to control the
Table 6: Playback Options
Button
Click to
Stop playback and go to the beginning of the sequence.
In a marked sequence, go to the previous clip.
Go back one frame.
Play / Pause the playback.
Go forward one frame.
In a marked sequence, go to the next clip.
Repeat playback in a loop.
53
Appendix E: Statistics
The Statistics feature provides information about the Pack, the Gestures and the sequences in the Pack.
To display Pack statistics, do one of the following:
From the Pack menu choose Pack Statistics.

Right-click any area of the Pack Dashboard and choose Show Pack Statistics.
Click the Statistics button
at the bottom right of the Pack Dashboard.
The Statistics window appears.
Figure 43: Pack Statistics Window
The Pack Statistics include the following information:
Number of Gestures: the number of Gestures in the Pack.
Total Number of Sequences: the number of sequences in all the Gestures in the Pack.
Number of Marked Sequences: the number of marked sequences in the Pack.
Number of Unmarked Sequences: the number of unmarked sequences in the Pack.
Average Score: the average build score for the Gestures in the Pack.
Last Build Date: the date of the latest build.
Last Build Total Runtime: the time that the latest build took to complete.
Last Build Type: The build type (Normal, Fast, or Slow) chosen for the last build.
54
To view a Gestures Statistics, on the left pane, click the Gesture you want to view.
The statistics for the Gesture you selected appear in the right pane.
Figure 44: Gesture Statistics Window
The Gesture Statistics include the following information for each of the Gestures in the Pack:
Score: the latest build score for the Gesture.
Gesture Type: one of the values Single Motion, Pose or Continuous.
Mirror Gesture: the name of the corresponding mirror gesture, if there is one.
Excluded from Build: whether this Gesture will be ignored during the next build.
Total Number of Sequences: the number of sequences for the Gesture.
Number of Marked Sequences: the number of marked sequences for the Gesture.
Number of Unmarked Sequences: the number of unmarked sequences for the Gesture.
55
To view sequence statistics:

1.
In the left pane, click the Gesture arrow to view its sequences.
2.
Click the sequence you want to view.

The right pane displays the sequence statistics.
Figure 45: Sequence Statistics Window
The sequence statistics include the following information for each of the sequences in the Pack:
Duration: the length of the sequence in seconds.
Number of Frames: the number of frames in the sequence.
Location: the location of the video file of the sequence.
Need Retrack: whether retracking is required for this sequence.
Clips Marked as Gesture: the number of clips in the sequence that are marked as Gesture instances.
Clips marked as Bad Data: the number of clips in the sequence that were marked as Bad Data.
Clips Marked as XNegative: the number of clips in the sequence that were marked as XNegative.
56
Appendix F: FAQs
Question:
How many sequences should I record to train a Classifier?
Answer:
The number of required sequences varies for different Gestures, as some are easy to detect and some are more
difficult. In general, use the following approach. Begin by recording at least 10 training sequences, of which the
majority will show the target Gesture. A few of the sequences should exhibit other motions that are similar to
the Gesture, but should not be detected as the Gesture.
Since the examples that the Build learns from are the marked clips of the gesture (and not the entire sequence),
you need to make sure that the set of sequences includes:
At least 30 marked clips for a Single Motion Gesture.
At least 10 marked clips for a Pose or Continuous Gesture.
Question:
I trained a new Gesture Classifier, but it gets very few hits. What should I do to improve it?
Answer:
Add more Gesture instances to the training sequences.
Question:
I need to train the same Gesture for the left and right hands. Do I have to record sequences for each side?
Answer:
You do not need to mark and record sequences for each side separately. Instead, you can use the Mirror Gesture
feature. While defining a Gesture, check the Create Mirror Gesture option. Name the original and mirror
Gestures informatively. For instance, if the original Gesture is jumpRight, name the mirror Gesture jumpLeft.
You only need to add sequences to the original Gesture. GAT will automatically train the mirror Gesture on a
flipped version of the original sequences.
Question:
I trained a new Gesture and it works well, with no misses at all. But I am getting quite a few false hits, what
should I do?
Answer:
Record sequences of a person performing the motion that causes the false hits, and add a few such sequences
to the training set.
57
Question:
The trained Gesture is not reliable, although I have many sequences.
Answer:
Make sure the input data is accurate. Pay particular attention to the Gestures clips and the tracking in the
training sequences. Make sure all the Gestures are marked. If the tracking is not accurate for a particular
sequence, mark the problematic frames as Bad Data.
Question:
What are the typical numbers of sequences and clips in a real-life Gesture training scenario?
Answer:
Here are some numbers taken from the real training scenario of a RightClick gesture:
15 sequences
40 marked Gesture clips
15 XNegative clips
This Gesture took 4 cycles of building and refinement, which were completed in a few hours.
58

Omek Beckon GAT - User Manual

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Omek Beckon GAT - User Manual

Uploaded by

Copyright:

Available Formats

Omek Gesture Authoring Tool

For Beckon 3.0

2012 Omek Interactive, Ltd. All rights reserved. | www.omekinteractive.com

Tips for Dealing with False Hits ................................................................................................................................. 38

2012 Omek Interactive, Ltd. All rights reserved. | www.omekinteractive.com

Gesture Authoring Tool (GAT) Overview

Training GAT to Recognize Gestures

2012 Omek Interactive, Ltd. All rights reserved. | www.omekinteractive.com

2012 Omek Interactive, Ltd. All rights reserved. | www.omekinteractive.com

Basic Entities and Terms

2012 Omek Interactive, Ltd. All rights reserved. | www.omekinteractive.com

2012 Omek Interactive, Ltd. All rights reserved. | www.omekinteractive.com

Designing a Gesture Pack

Guidelines for Designing a Gesture

2012 Omek Interactive, Ltd. All rights reserved. | www.omekinteractive.com

Gesture Training Overview

Create New Pack

Add Sequences (Record or

Pack Rules and Settings

Figure 1: Gesture Pack Training Process Diagram

2012 Omek Interactive, Ltd. All rights reserved. | www.omekinteractive.com

These are the stages of creating and building a Gesture Pack:

Record sequences of the Gesture.

Add the sequences to the Gesture in the Pack.

Evaluate by observing Gesture detection in real-time.

Check the performance statistics in the Build Report.

If the Pack performs well, you are done!

2012 Omek Interactive, Ltd. All rights reserved. | www.omekinteractive.com

Stage One: Creating a Pack and Adding Gestures

Creating a Gesture Pack

Open GAT tool and click New Pack.

Figure 2: Welcome to GAT

The New Pack dialog appears.

Figure 3: New Pack Dialog

2012 Omek Interactive, Ltd. All rights reserved. | www.omekinteractive.com

Figure 4: Pack Dashboard for an Empty Pack

2012 Omek Interactive, Ltd. All rights reserved. | www.omekinteractive.com

Figure 5: Pack Dashboard

To display the Pack Dashboard:

Open GAT to display the Welcome screen.

Figure 6: Welcome Screen

You can do one of the following:

Create a new Pack (see Creating a Gesture Pack).

2012 Omek Interactive, Ltd. All rights reserved. | www.omekinteractive.com

Click one of the Pack names in the Recent Packs list.

Adding and Modifying Gestures

From the Gesture menu, choose Add Gesture.

Figure 7: New Gesture Dialog

Figure 8: New Gesture Dialog with Mirror Name

Figure 9: Gesture Icon with Mirror gesture

appears next to the name of the

2012 Omek Interactive, Ltd. All rights reserved. | www.omekinteractive.com

Adding a Preview Image to a Gesture

Figure 10: Default Preview Image

Figure 11: Preview Image

2012 Omek Interactive, Ltd. All rights reserved. | www.omekinteractive.com

Creating Composite Gestures

From the Gesture menu, choose Add Gesture.

Figure 12: New Gesture Dialog

In the Name textbox, enter a name for the new Gesture.

From the Type drop-down list, select Composite Gesture.

Figure 13: Composite Gesture Dialog

2012 Omek Interactive, Ltd. All rights reserved. | www.omekinteractive.com

Figure 14: Composite Gesture Icon