Introduction to Kinect SDK Beta

CS4201 Interactive System Project Lab 1 Instructor: Dr. Shengdong Zhao NUS-HCI Lab Slides prepared by Lan Ziquan

Agenda
Introduction Preparation Explanation for sample code Practice together Show your understanding

From http://www.xbox.com/en-US/Kinect/Home

INTRODUCTION
4

All about Kinect

First announced on 01/06/09 at E3 2009 Release Date
US: 4/11/10 SG: 18/11/10

Application development
Microsoft released a non-commercial Kinect for Windows SDK on 16/06/11 You are about to learn and use it

Programming environment
power supply

Windows 7 machine with software installed

Kinect sensor

Kinect Sensor
3D DEPTH SENSORS RGB CAMERA

MULTI-ARRAY MIC

MOTORIZED TILT
7

Programming environment
power supply

Windows 7 machine with software installed

Kinect sensor

System Requirements (1/2)

Hardware
Windows 7 (x86 is recommended) Dual-core, 2.66-Hz (recommended) Graphics card that supports DirectX 9.0c 2GB of RAM (recommended)

Software
Microsoft Visual Studio 2010 (Prof/Express/) Microsoft .NET Framework 4.0 Audio control (x86 only) is not covered here
9

System Requirement (2/2)

Software for audio control (not covered in this lecture)
Microsoft Speech Platform Server Runtime, version 10.2 (x86) Microsoft Speech Platform Software Development Kit, version 10.2 (x86) Kinect for Windows Runtime Language Pack, version 0.9

10

PREPARATION

11

Download and Install Kinect SDK Beta

Download Kinect SDK Beta
X86: http://download.microsoft.com/download/3/C/B/3CB546 D6-C04E-4097-9393-C81C51F7FDA2/KinectSDK32.msi X64: http://download.microsoft.com/download/3/C/B/3CB546 D6-C04E-4097-9393-C81C51F7FDA2/KinectSDK64.msi Caution:
Uninstall ALL other Kinect drivers before installation Only install on NATIVE Windows 7 instead of Virtual Machine DISCONNECT Kinect device while installing

12

Skills Requirements
IDE Integrated Development Environment
Visual Studio 2010
http://msdn.microsoft.com/enus/vstudio/ff431702 (official)

Programming Language
Program logic
C# (or C++) http://www.homeandlearn.co.uk/csharp/ csharp.html

UI Design (with C# only)

WPF Windows Presentation Foundation http://www.codeproject.com/KB/WPF/Be ginWPF1.aspx

13

DEMO TIME
14

Kinect Basic Functions

Color stream Depth stream Skeletal tracking Audio processing (not covered here)
15

Color Stream

16

Depth stream

17

Skeletal tracking

18

Piece 1 Data field

Piece 2 Three event handlers

Piece 3 Helper methods

EXPLANATION FOR SAMPLE CODE

19

MainWindow
skeleton

depth

frameRate video
20

Piece 1 Using Microsoft.Research.Kinect.Nui

MainWindow class starts Data field initialization Piece 2 Three event handlers

Piece 3 Helper methods

21

nui represents the Kinect sensor instance Initialize variables to calculate the # of fps

Define constants for indexing array

Define an array that is large enough to hold an entire frame of depth data at 32bits per pixel

22

Define a dictionary to store colors for each joint

23

Piece 1

Window_Loaded Piece 2 Three event method handlers

Piece 3 Helper methods

24

Before calling any other methods on the Runtime object, i.e. nui, an application must call Runtime.Initialize to initialize the Kinect sensor The Initialize method throws InvalidOperationException if it fails to find a Kinect sensor

Parameter is a bitwise OR combination of the following values from RuntimeOpitions enumeration: - UseDepthAndPlayerIndex [recommended] - UseColor [recommended] - UseSkeletalTracking [recommended] - UseDepth

25

The value of VideoStream and DepthStream are both ImageStream objects The ImageStream.Open has 4 parameters - ImageSteamType is either Video or Depth - Number of lookahead buffers. By using 2 buffers, the application can draw with one buffer and the capture driver can fill the other - ImageResolution is a value of the enumeration - ImageType is also a value of the enumeration

VideoStream.Open Required RuntimeOptions Valid Image Resolution Valid Image Type UseColor Resolution1280x1024 Resolution640x480 Color,ColorYUV,ColorYUVRaw

DepthStream.Open UseDepthAndPlayerIndex Resolution320x240 Resolution80x60 DepthAndPlayerIndex

26

Application can use either a polling model or an event-driven model to retrieve frames Here we introduce the event-driven model in this sample

Event handler methods to be explained later

To implement the event-driven model, we creates DepthFrameReady, SkeletonFrameReady and VideoFrameReady events for nui, then associates them with the event handlers respectively When a frame from the Kinect sensor is ready for the application, the runtime signals the relevant event and the associated event handler method runs
27

Piece 2

nui_ColorFrameReady method

Piece 1 Data field Piece 3 Helper methods

28

Get the image to be displayed from parameter e

Use the WPF BitmapSource.Create method to create a bitmap for the display Check up the library to know the parameters here NOTE: Image.Bits is an array of bits that form the image The next method is about to construct that array for the display
29

Piece 2

nui_DepthFrameReady method Piece 1 Data field Piece 3 Helper methods

30

Convert the image array to the desired display format Then display the convertedDepthFrame Depth data (Image.Bits) is an array of 16-bit values

This means a time span of 1 second

Update variables to calculate the number of frames per second

31

Piece 1 Data field

Piece 3

Piece 2 Three event handlers

convertDepthFrame method
32

This method is to convert 16-bit gray-scale data into 32-bit RGB data in order to be well displayed depthFrame16 is an array of 16-bit values, as follows: - The low-order 3 bits contain a skeleton ID - The remaining bits contain the depth value in millimeters Get the player skeleton ID

Zero-initialize the RGB data in 32-bit frame

- Retrieve the 11 bits of actual depth data - Invert the result to generate an 8-bit intensity value that is appropriate for display
33

Color different players with different RGB values

Player 0 indicates that NO player is present

34

Piece 2

nui_SkeletonFrameReady method

Piece 1 Data field Piece 3 Helper methods

35

Retrieve SkeletonFrame object from parameter e

Initialize player skeleton index

Create an array of SolidColorBrush objects for drawing different player skeletons (according to iSkeleton)
36

Clear any current displayed skeleton

The TrackingState field of the SkeletonData structure indicates that this skeleton is being tracked

The two pieces of code is explained next

37

getBodySegment method has the following 3 parameters - data.Joints is a collection of joints for the skeleton - Brush is to draw the (poly)line - a variable number of JointID values identify a sequence of joints on the skeleton The code is explained later

38

jointPos

39

Piece 1 Data field

Piece 3

Piece 2 Three event handlers

getDisplayPosition method

40

Convert skeleton coordinates in the [-1.0, 1.0] range to depth coordinates, which returns x and y coordinates as floating point number in the range from 0.0 to 1.0 Convert the floating point coordinates to values in the 320x240 depth coordinate space, which will be used next

Convert depth coordinates to color image coordinates, which return color image coordinates as values in the 640x480 color image space
41

Piece 1 Data field

Piece 3

Piece 2 Three event handlers

getBodySegment method

42

Construct a point collection that contains all joints

Construct a polyline using data required

43

PRACTICE TOGETHER

44

SHOW YOUR UNDERSTANDING

45

Quick Assignment
Write a small program that can recognize the following gestures
Left/Right Arm +
UP/DOWN/FORWARD/LEFT/RIGHT

E.g. Left arm FORWARD & Right arm DOWN

Image from www.123fr.com

46