Nyart Processing

nyARToolKit+Processing: A short tutorial
Dr. J.B. Hayet

EN MATEMATICAS
CENTRO DE INVESTIGACION
August 2011
,
J.B. Hayet
ERA2011, August 2011
1 / 57
Outline
Overview
Dissection of an example
Handling 3D models
Conclusions
,
J.B. Hayet
2 / 57
Overview
Outline
Overview
Handling 3D models
Conclusions
,
J.B. Hayet
3 / 57
Overview
ARToolKit
C and C++ library to
easily develop Augmented
Reality applications.
Uses computer vision to
detect the markers;
track the markers;
estimate the camera
position/orientation relative
to planar markers.
Re-projects virtual objects

(e.g. VRML data) in the
markers coordinate system.
[From ARToolKit documentation]
,
J.B. Hayet
4 / 57
Overview
ARToolKit
Features:
Multiplatform library (Windows, Linux, Mac OS X, SGI).

Overlays 3D virtual objects on real markers.
Supports:
multiple input sources (USB, Firewire, capture card),
multiple format (RGB/YUV420P, YUV),
multiple camera tracking schemes,
Fast, cheap 6D marker tracking (real-time planar pattern

detection).
Configurable markers patterns, support for multi-markers
objects.
,
J.B. Hayet
5 / 57
Overview
NyARToolKit
NyARToolKit is a port of ARToolkit in Java (named

after Nyatla, the main contributor).
Same functionalities for visual interpretation and projection of
3D models.
Slower in execution than the original, but
architecture-independent.
http://nyatla.jp/nyartoolkit/
,
J.B. Hayet
6 / 57
Overview
NyARToolKit
NyARToolkit has been ported to numerous flavors:

Java,
Android,
C#,
SilverLight, . . .
,
J.B. Hayet
7 / 57
Overview
NyARToolKit: Licensing
GPL
Free to use for non-commercial applications under the GPL license.
Commercial Licence
License managed by ARToolWorks.
,
J.B. Hayet
8 / 57
Overview
Processing
A very simplified Java dialect.

Minimalistic IDE.
A priori no difficulty to C, C++, Java programmers.
Focuses on a non-expert audience, in search for visualization,
fast prototyping, interactive applications.
Built-in functions for interaction, sound, video, 3D. . .
Produces local applications or applets.
,
J.B. Hayet
9 / 57
Overview
Processing
Applications particularly well adapted:

learning programming: bases of programming concepts,
learning CS concepts: animated visualization of data,
physical effects,
fast prototyping of applications, sketches,
statistical data visualization.
,
J.B. Hayet
10 / 57
Overview
Processing
,
J.B. Hayet
11 / 57
Overview
Processing
A typical Processing application manages one configurable window

and its corresponding mouse, keyboard event and mixes:
definitions of classes,
definitions of pre-defined functions : setup, draw.
It can be used to emphasize either programming paradigm (OOP,
functional. . . ).
,
J.B. Hayet
12 / 57
Overview
Processing
setup: serves as an initialization function, it is called once at the

moment the play button is pressed. Allows to set up the window size,
drawing properties, frame rate. . .
draw: is called as a callback function at a framerate specified by the
user. Put there everything related to your RA application: the display
of the current camera image, the VR stuff. . .
,
J.B. Hayet
13 / 57
Overview
Processing
Video library:
display QuickTime video files,
grab video data from a camera (USB Cameras, IEEE 1394
(Firewire) Cameras, or Video Cards),
make QuickTime videos directly from a running program.
Class Capture for RA applications.
http://processing.org/reference/libraries/video/index.html
,
J.B. Hayet
14 / 57
Overview
Processing
Data used in the program: the directory data inside your

sketchbook in which you will store all data files necessary for your
application.
In the AR application, you will need:
camera parameters files (ARToolkit),
pattern description files (ARToolkit),
3D model files.
,
J.B. Hayet
15 / 57
Overview
NyARToolKit with Processing
Installation:
Download the NyARToolkit library for Processing at
http://nyatla.jp/nyartoolkit/wiki/index.php?NyAR4psg.en
Unzip the downloaded file in the directory

PROCESSING DIR/libraries
Same process for all external libraries in Processing.
,
J.B. Hayet
16 / 57
Overview

Installation:
,
J.B. Hayet
17 / 57
Overview

Installation:
,
J.B. Hayet
18 / 57
Overview
NyARToolKit: Coordinate systems

Zc
Xc
xc
Yc
yc
Zm
Xm
Ym
,
J.B. Hayet
19 / 57
Overview

The quantities estimated by the tracking algorithm are the 6
parameters of Tcm :

Rcm tcm
Tcm =
,
0
1
where
Rcm is a 3 3 rotation matrix that is specified by 3 free
parameters (e.g., angles),
tcm is a 3 1 translation vector,
Tcm is the resulting 4 4 transformation matrix.
,
J.B. Hayet
20 / 57
Overview

This mathematical object transforms points in the Rm frame
(marker) into the Rc (camera) frame:
Xc
Xm
Yc
= Tcm Ym ,
Zc
Zm
1
1
they are the extrinsic parameters of the camera, that vary while
the camera moves.
,
J.B. Hayet
21 / 57
Overview

Note that if we take Xm = 0, Ym = 0, Zm = 0 (i.e. the marker
center),
Xc

Yc
= tcm ,
Zc
1
1
i.e tcm are the coordinates in the camera frame of the center
of the marker.
,
J.B. Hayet
22 / 57
Overview

Once calculated in the camera frame, the 3D points are projected
onto the image frame through the perspective projection,
Xc
sx f 0 x0 0
xc
yc = 0 sy f y0 0 Yc ,
Zc
0
0 1 0
1
1
they are the intrinsic parameters of the camera, that are supposed to
stay stationary while the camera moves. To determine them, the
camera has to be calibrated, ARToolKit has tools to determine them.
,
J.B. Hayet
23 / 57
Overview
Intrinsic parameters:
(x0 , y0 ): coordinates of the projection onto the image of the
projection center;
f : focal distance (generally in mm);
(sx , sy ): scale parameters to give the focal distance in pixels;
: scale factor; not a parameter (can be expressed through the
last line).
,
J.B. Hayet
24 / 57
Overview
Hence, when
the intrinsic parameters are known,
the extrinsic parameters are known,
I can project on the image any virtual shape that I have defined
relatively to the marker. If it is done dynamically and fast enough, I
will have the impression that the virtual shape is sticked to the
marker. That is the principle of augmented reality.
,
J.B. Hayet
25 / 57
Overview
Intrinsic parameters need to be calibrated.

The calibration is stored in a binary file (format specific to
ARToolkit and variants).
You need to use the original calibration program, or use default
.dat files.
http://www.hitl.washington.edu/artoolkit/download/
,
J.B. Hayet
26 / 57
Overview
The main problem is the computation of the extrinsic parameters, i.e.

a camera localization problem: thats why AR is mainly done by
computer vision people.
This localization can be done with markers (i.e. deduce from the
distorted shape in the image of a well-known planar object), like in
ARToolKit or without markers (most recent works in the computer
vision community).
,
J.B. Hayet
27 / 57
Overview
The steps:
detect markers (as black squares on a white background),
recognize the markers (i.e. see whats inside and if it looks like
one of the known markers),
compute the extrinsic parameters by putting in correspondence
points from the detected markers with points fron the known
marker and resolving the corresponding optimization problem.
,
J.B. Hayet
28 / 57
Overview
NyARToolKit: System flow
[From ARToolKit documentation]
,
J.B. Hayet
29 / 57
Overview
1.
2.
3.
4.
5.
6.
Initialize the application;

Grab a video input frame;
Detect and recognize the markers;
Calculate camera transformation;
Draw the virtual objects;
Close the video capture.
The program does cycles from 2 to 5.
,
J.B. Hayet
30 / 57
Overview
NyARToolKit: Implemented techniques

Image processing:
Binarization (threshold on grey level).

Extraction of connected components.
,
J.B. Hayet
31 / 57
Overview

Image processing:
Contour extraction.
Straight line fitting.
Sub-pixel line intersections (xi , yi ).
,
J.B. Hayet
32 / 57
Overview
Pattern recognition:
Image pattern normalization.
Template matching.
Deduce the nature and orientation of the true pattern,
and its corner coordinates (Xi , Yi ).
,
J.B. Hayet
33 / 57
Outline
Overview
Handling 3D models
Conclusions
,
J.B. Hayet
34 / 57
1.
2.
3.
4.
5.
6.
Initialize the application;

Grab a video input frame;
Detect and recognize the markers;
Calculate camera transformation;
Draw the virtual objects;
Close the video capture.
The program does cycles from 2 to 5.
,
J.B. Hayet
35 / 57
NyARToolKit: a simple example
,
J.B. Hayet
36 / 57
Needed libraries
,
J.B. Hayet
36 / 57
Object to capture images from video.
,
J.B. Hayet
36 / 57
Object to manage markers stuff.
,
J.B. Hayet
36 / 57
Step 1 of the cycle.
,
J.B. Hayet
36 / 57
Constructor.
,
J.B. Hayet
36 / 57
Adds a given marker (0) and sets its dimension.
,
J.B. Hayet
36 / 57
Steps 2 to 5.
,
J.B. Hayet
36 / 57
Check camera.
,
J.B. Hayet
36 / 57
Step 2.
,
J.B. Hayet
36 / 57
Steps 3, 4.
,
J.B. Hayet
36 / 57
Checks if marker 0 has been detected.
,
J.B. Hayet
36 / 57
Step 5.
,
J.B. Hayet
36 / 57
NyARToolKit: MultiMarker
The main class for managing multiple markers with the Processing
version is MultiMarker
It inherits from NyARPsgBaseClass.
Serves as a container for several markers.
Manages traditional ARToolkit markers OR id-Markers.
,
J.B. Hayet
37 / 57
Traditional ARToolkit markers: image patterns.
Need training.
Template matching not super-fast.
A lot of possible appearances.
Id-markers: binary pattern on a regular grid
Dont need training.
Fast recognition.
Can cover up to 512 different appearances.
,
J.B. Hayet
38 / 57
[From http://flash.tarotaro.org]
,
J.B. Hayet
39 / 57
void setup ( ) {
s i z e ( 6 4 0 , 4 8 0 , P3D ) ;
c o l o r M o d e (RGB, 1 0 0 ) ;
p r i n t l n ( M u l t i M a r k e r . VERSION ) ;
cam=new C a p t u r e ( t h i s , 6 4 0 , 4 8 0 ) ;
nya=new M u l t i M a r k e r ( t h i s , width , h e i g h t , c a m e r a p a r a . d
nya . addARMarker ( p a t t . h i r o , 8 0 ) ; // i d =0
nya . addARMarker ( p a t t . k a n j i , 8 0 ) ; // i d =1
nya . addNyIdMarker ( 0 , 8 0 ) ; // i d =2
nya . addNyIdMarker ( 1 , 8 0 ) ; // i d =3
}
,
J.B. Hayet
40 / 57
Methods
addARMarker(String fileName, float width): Add a marker.
detect(PImage image): Detect markers in the image.
isExistMarker(int id): Check if a given marker has been detected.
beginTransform(int id): 3D objects will be specified relatively to
the marker.
endTransform().
,
J.B. Hayet
41 / 57
Get methods
getConfidence(int id): Get confidence on marker recognition.
getCurrentThreshold(): Get binarization threshold.
getLife(int id): Get number of consecutive marker detection.
getLostCount(int id): Consecutive lost detections before fail.
getMarkerVertex2D(int id): Get 2D vertices of the markers.
,
J.B. Hayet
42 / 57
Set methods
setConfidenceThreshold(double val): Set threshold on
confidence.
setThreshold(int ith): Set binarization threshold.
setLostDelay(int val): Delay before considering fail.
,
J.B. Hayet
43 / 57
getMarkerMatrix(int id): gives the aforementioned rigid
transform between the marker frame and the camera frame

Rcm tcm
Tcm =
,
0
1
getProjectionMatrix(): Get the intrinsics parameter matrix (with
clipping).
You get the full projection matrix by multiplying them
PMatrix3D t r a n s f = nya . g e t M a r k e r M a t r i x ( 0 ) ;
PMatrix3D p r o j
= nya . g e t P r o j e c t i o n M a t r i x ( ) ;
proj . apply ( t r a n s f ) ;
,
J.B. Hayet
44 / 57
Important when you want to do multi-marker applications based on

marker interactions:
1
Tm0 m = Tcm
0 Tcm
Example: triggers events when the distance between the two markers
frame centers is inferior to some quantity.
,
J.B. Hayet
45 / 57
To perform the projection of a point by hand:
float
i n [ ] = new f l o a t [ 4 ] ;
in [0] = x ;
in [1] = y ;
in [2] = z ;
in [3] = 1.0;
f l o a t o u t [ ] = new f l o a t [ 4 ] ;
p r o j . mult ( i n , o u t ) ;
f l o a t x = w i d t h /2 + w i d t h ( o u t [ 0 ] / o u t [ 3 ] ) / 2 ;
f l o a t y = h e i g h t /2+ h e i g h t ( o u t [ 1 ] / o u t [ 3 ] ) / 2 ;
pp [ k ] = new PV e c t o r ( x , y ) ;
e l l i p s e (x , y ,10 ,10);
,
J.B. Hayet
46 / 57
pickupMarkerImage
pickupRectMarkerImage
define a quadrilateral in the marker frame and map it in given size
image.
,
J.B. Hayet
47 / 57
,
J.B. Hayet
48 / 57
Handling 3D models
Outline
Overview
Handling 3D models
Conclusions
,
J.B. Hayet
49 / 57
Handling 3D models
Processing: rendering modes
P3D (Processing 3D) - Fast 3D renderer for the web.

Sacrifices rendering quality for quick 3D drawing.
OPENGL - High speed 3D graphics renderer that makes
use of OpenGL-compatible graphics hardware is
available.
,
J.B. Hayet
50 / 57
Handling 3D models
Processing: Handling 3D shapes
Built-in functions:
box()
sphere()
vertex()
Sufficient for many cool applications.
,
J.B. Hayet
51 / 57
Handling 3D models
Processing: Handling 3D shapes
External libraries with more 3D primitives. As an example, shapes3d

provides classes for more 3d shapes (tubes, cones, ellipsoids. . . ), 3d
terrain models. . .
http://processing.org/reference/libraries/
http://www.lagers.org.uk/s3d4p/distribution/web/index.html
[Install in the same way as nyArToolkit]
,
J.B. Hayet
52 / 57
Handling 3D models
NyARToolKit: handling 3D models

shapes3D:
,
J.B. Hayet
53 / 57
Handling 3D models
NyARToolKit: handling 3D models
objimp: library to import .obj files, and render them (in OpenGL
mode)
http://www.pixelnerve.com/processing/libraries/objimport/
,
J.B. Hayet
54 / 57
Conclusions
Outline
Overview
Handling 3D models
Conclusions
,
J.B. Hayet
55 / 57
Conclusions
NyARToolKit + Processing
Very fast prototyping of AR applications.

Ideal for teaching purposes.
Built-in functionalities (audio, video, . . . ) sparkling creativity.
,
J.B. Hayet
56 / 57
Conclusions
Contact
Jean-Bernard Hayet (jbhayet arrobase cimat.mx)
Slides + material at
http://www.cimat.mx/jbhayet/era2011.zip
,
J.B. Hayet
57 / 57

Nyart Processing

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Nyart Processing

Uploaded by

Copyright:

Available Formats

nyARToolKit+Processing: A short tutorial

Dr. J.B. Hayet

ERA2011, August 2011

ERA2011, August 2011

ERA2011, August 2011

Re-projects virtual objects

ERA2011, August 2011

Multiplatform library (Windows, Linux, Mac OS X, SGI).

Fast, cheap 6D marker tracking (real-time planar pattern

ERA2011, August 2011

NyARToolKit is a port of ARToolkit in Java (named

ERA2011, August 2011

NyARToolkit has been ported to numerous flavors:

ERA2011, August 2011

ERA2011, August 2011

A very simplified Java dialect.

ERA2011, August 2011

Applications particularly well adapted:

ERA2011, August 2011

ERA2011, August 2011

A typical Processing application manages one configurable window

ERA2011, August 2011

setup: serves as an initialization function, it is called once at the

ERA2011, August 2011

ERA2011, August 2011

Data used in the program: the directory data inside your

ERA2011, August 2011

NyARToolKit with Processing

Unzip the downloaded file in the directory

ERA2011, August 2011

NyARToolKit with Processing

ERA2011, August 2011

NyARToolKit with Processing

ERA2011, August 2011

NyARToolKit: Coordinate systems

ERA2011, August 2011

NyARToolKit: Coordinate systems

ERA2011, August 2011

NyARToolKit: Coordinate systems

ERA2011, August 2011

NyARToolKit: Coordinate systems

ERA2011, August 2011

NyARToolKit: Coordinate systems

ERA2011, August 2011

NyARToolKit: Coordinate systems

ERA2011, August 2011

NyARToolKit: Coordinate systems

ERA2011, August 2011

NyARToolKit: Coordinate systems

Intrinsic parameters need to be calibrated.

ERA2011, August 2011

NyARToolKit: Coordinate systems

The main problem is the computation of the extrinsic parameters, i.e.

ERA2011, August 2011

NyARToolKit: Coordinate systems

ERA2011, August 2011

NyARToolKit: System flow

[From ARToolKit documentation]

ERA2011, August 2011

NyARToolKit: System flow

Initialize the application;

The program does cycles from 2 to 5.

ERA2011, August 2011

NyARToolKit: Implemented techniques