STEREOPSIS 1

Camera geometry and 3D-reconstruction

Norbert Krger and Dirk Kraft based on slides by Nicolas Pugeault

Outline

A. One camera

Lm

M1

perspective considerations projective geometry camera model

C

M'' M'
m

B. Two cameras

extrinsic parameters

M m1 m2

calibration epipolar geometry reconstruction

C1

C2

A. One camera

Lm

M1

M'' M'
m

Perspective

visual sensors form images by concentrating light on a sensitive surface using lenses Images are a projective mapping of the 3D space understanding this mapping is essential for inverting it

pinhole camera model projective geometry calibration

Perspective (cont'd)

Intuitively, we have a relation between apparent size and depth of the object

far objects appear small nearby objects appear big

Thales theorem yields:

f x= X d

The Projective plane

Point is designated by the three coordinates:

vectors are points at infinity

u m@ v A w
w=0

point at infinity

m1 m2
l1
line at infinity

l2

parallel lines meet at points at infinity

The Projective plane (cont'd)

There is a point-line duality Lines are represented by:

ax + by + cw = l m = 0
x y a +b +c =0 w w

Such that the following relation holds 8m 2 l

a l @ b A c

m
l3

l1 l2

m1 m2 m3

The Projective space

3D points are represented 0 1 by

Vectors are points at infinity They lie on the plane at infinity

plane at infinity

x B y C C M B @ z A w

M1

M L

The Projective space: 3D-planes

Planes in projective space are dual to points They are described by

h O

They satisfy the relation

nx B ny C C B @ nz A h

n is the normal vector h the distance to the origin O

M = 0; 8M 2

The Projective space: Pluecker lines

Pluecker coordinates represent 3D lines using

(normalized such that kk = 1) line moment , defined for all point m 2 L

line orientation vector

= m

L O m

L (; )

Conversion to and from homogeneous coordinates

we can easily convert from and to Homogeneous and Euclidian coordinates:

x x B B y C C homogeneous B @ y A B @ @ z A z 1 m m M
to

Euclidian

0 1 x x C 1 @ y C y A z A w z w M
from

Euclidian

Projective geometry

points at infinity (equivalent to Euclidian vectors): e.g. parallel lines meet at infinity allows for a linear representation of translations:

t = (tx ; ty ; tz )>
0 1 0 e.g., rotation 0 1 0 0 0 0 1 0
translation

x + tx 1 B y + ty C B 0 B C=B @ z + tz A @ 0 1 0

tx x ty C B y C CB C tz A @ z A 1 1

1 0

Projection matrix

Using homogeneous coordinates, the pinhole model can then be formalized as: where

m= AM

f A=@ 0 0

0 f 0

0 0 1

0 0 A 0

Projection matrix (cont'd)

We want to include the imaging parameters:

s is the skew (generally null) pixel arrangement u and v are the pixel resolution in u and v c = (u0 ; v0 )> is the projection of the optical center.

m= K AM 0 1 u s u 0 K = @ 0 v v0 A 0 0 1

Calibration

The result is the following matrix called:

The parameters (f; u ; v ; s; u0 ; v0 ) are called Intrinsic parameters

f u 0 KA=@ 0

f s f v 0

u0 v0 1

0 0 A 0

Calibration, is the process to estimate the following parameters:

s f u ; f v ; ; u0 ; v0 u

Distortion

Real lenses are only an approximation of a pinhole model => distortion e.g. radial distortion with wide-angle (small focal length) lenses. Lead to non-linear projection model

=> image undistortion

Calibration (2)

Tools exist for performing calibration

openCV
http://www.intel.com/technology/computing/opencv/

matlab calibration toolbox

http://www.vision.caltech.edu/bouguetj/calib_doc/

Based on regular grid patterns of known sizes, with manual or automatic corner extraction provides an estimation of all parameters + distortion provides tool for image rectification

Calibration(3)

B. Two cameras

M m1 m2

C1

C2

3D Pose

We need to model the relative position of the two cameras Combination of

translation between the two optical centers rotation between optical axes

m1
C1

m2
C2

This is a rigid transformation in 3D space (=> 6 DoF)

Extrinsic parameters

General projection matrix formulation:

m = (K A H) M

Where the Extrinsic parameters are

H=

R 013

t 1

R is the rotation that brings the optical axis to the z -axis t is the translation vector that brings C onto the origin

O = (0; 0; 0; 1)>
t = R C

Stereo Calibration

http://www.vision.caltech.edu/bouguetj/calib_doc/htmls/example5.html

Stereo calibration (cont'd)

Practical

Objectives:

Calibrate a stereo pair of cameras

compute the projection matrices P1 and P2 compute the fundamental matrix F12 display both left and right images when a point is clicked in the left image, displays the corresponding epipolar line in the right image reconstruct the corresponding 3D point when a stereo pair of points is selected.

Make a program that:

Outline

A. One camera

Lm

M1

perspective considerations projective geometry camera model

C m

M'' M'

B. Two cameras

extrinsic parameters

calibration epipolar geometry reconstruction

C1

M m1 m2

C2

Plucker line - prereq.

Classical line representation Point on line closest to origin:

L O
~ c ~ a

.
~ b

Every point on line can be decomposed: Cross product properties:

To Plucker lines and back (1)

Plucker line: To Plucker line:

To Plucker lines and back (2)

Back:

Epipolar geometry

Plausible 3D origins for an image point m form a halfline in space. This line passes through the optical center of the camera.

Lm
M'' M'

M1

Epipolar Geometry (cont'd)

If we consider the projection matrix

Here italic denotes to projective quantities, e.g., P and bold denotes Euclidean quantities, e.g., P P is a 3x3 matrix. p is a dimension 3 column vector.

Epipolar Geometry (cont'd)

Note that p contains the projective coordinates of the projection on the image plane of the world coordinate system's origin.

If there is not translation then we also have and

Epipolar Geometry (cont'd)

Then p is the projection of the optical center on the image plane

C = R 1 t = P 1 p

and P 1 gives the back-projection vector of a given image point

M1 =

P 1 m 0

Note that m denotes the projective coordinates of a point in the image.

Back-Projected ray

From this we can compute the coordinates of a line in space where lie all plausible 3D origins of the image point

Lm =

1 kM
1k

(M

1; C

1)

Effectively we can even say that the plausible origins lie on the half line (or ray) starting at the camera's optical center C This limits the possible locations of the correspondence in the second image to a line, a half line, or even a segment depending on the set-up.

Epipolar line
Xl X l

Lxl Il

Ir xl

xr

x r

Cl

er

Cr

Epipolar line (cont'd)

The plausible correspondences on the right image form a 2Dline that is the re-projection of the optical ray on the image It can be a line, a half-line or even a segment, depending if the endpoints lie at infinity or not. One endpoint is fixed and is e = P C 2 2 1 called the Epipole Note that if the optical center lie behind the camera, the epipole is not a boundary The second is the reprojection at infinity of the point considered:

e2
C2 e1

m2 m1

m2;1 = P2 M1;1

C1

Epipolar line (cont'd)

Then the line between those two points, where lie all plausible correspondences, is called the Epipolar line

(a)

(b)

l = e2 m2;1

(c)

(d)

e2

e2

e2

e2

C1 C1 C2 C1 C2 C2

C2 C1

Example (left image)

Example (right image)

Example (stereo)

C1
LEFT STEREO

C2
RIGHT

Fundamental matrix (1/2)

The fundamental matrix is defined as:

> 2

F12 m

=0

for m 1 and m 2 two corresponding points, respectively in the left and right image.

It can be computed from the two projection matrixes:

+ F12 = [e2 ] P2 P1

0 [v] = @ v3 v2

v3 0 v1

v2 v1 A 0

Fundamental matrix (1/2)

Where P + is the pseudo-inverse of P such that:

In this case P + can be decomposed into:

P P+ P = P

P + = H 1 A+ K 1

The fundamental matrix can be used to compute the epipolar line:

l = F12 m

Computation of the Fundamental matrix

1) find pairs of corresponding points in the two images (m1 ; m2 ) 2) Effectively 7 DOF, so at least 7 points are required.

3) Select a first set and generate an hypothesis

the distance from the correspondence to the epipolar line gives one constraint m> F12 m1 = 0 2 Solve a system of linear equations (RANSAC will be discussed in the Motion Estimation lecture)

4) outliers => use RANSAC

5) refine using non-linear optimization

Hartley and Zisserman, ch.11.6

Feature correspondences

left

right

correspondences

Computed Epipolar lines

left

right

Stereopsis: epipolar constraint

All potential correspondences of a primitive lie on its epipolar line:

left camera

right camera

Point reconstruction

Two corresponding stereo points back-project two optical rays in space. Theoretically, M lie at the intersection between the two lines. Practically, the two lines are unlikely to intersect. Often the closest point to both lines is chosen.

a)
m1

m2

C1

C2

b)

L2 M1

L1 M2

1 (2 2 ) (1 2 ) 1 (2 1 ) = 1 + 1 1 2 jj1 2 jj

Point reconstruction (cont'd)

Computing the point on L1 that is closest to L2

a)
m1

m2

= arg min d(x; L2 )

x2L1
C1

L1 = (1 ; 1 ) L2 = (2 ; 2 ) k1 k = k2 k = 1

C2

b)

L2 M1

L1 M2

1 (2 2 ) (1 2 ) 1 (2 1 ) = 1 + 1 1 2 jj1 2 jj

Practical

Objectives:

Calibrate a stereo pair of cameras

compute the projection matrices P1 and P2 compute the fundamental matrix F12 display both left and right images when a point is clicked in the left image, displays the corresponding epipolar line in the right image reconstruct the corresponding 3D point when a stereo pair of points is selected.

Make a program that:

Linear alternative

We can alternatively formulate the reconstruction as a linear optimization problem: Al bl M = Ar br Namely if we rewrite the projection matrices as: P = @
0 Q> 1 Q> 2 Q> 3

Then we can write the projection as the system of linear equations: 0 1 0 1 > Q1 q14 u M @ Q> q24 A = @ v A 2 1 > 1 Q3 q34

q14 q24 A q34

Linear formulation

Which translates into the system of equations 8 > < Q1 M + q14 = u Q> M + q24 = v : 2 Q> M + q34 = 3 By replacing we have:

Q> M uQ> M + q14 uq34 1 3 Q> M vQ> M + q24 vq34 2 3 > > Q1 uQ3 And when reordering: M = > > Q2 vQ3

= =

0 0

Ai

uq34 q14 vq34 q24

bi

Solving the system

We concatenate the equations provided by the corresponding points in both images into one system:

A M = b with

8 > > A < > > b :

= =

Al Ar bl br

The solution to this system is given by:

M = (A> A)1 A> b

(Linear least square)

Line reconstruction

The plausible origins of 2D lines form a plane in space The intersection of the two planes generated from the line in both images give a reconstructed 3D-line This process is ill-defined if the lines are parallel to the epipolar line -> then the two lines back-project the same plane in 3D !

L = (1 2 ; h2 1 h1 2 )

Image rectification

Make epipolar lines horizontal Note: there is still an horizontal shift of the images !
Points infinitely far do not have zero disparity

e2
rectification

e2

C1

C2

C1

C2

Image rectification example

before

after

left

right