Professional Documents
Culture Documents
Peter Corke
CyberPhysical Systems Lab.
a university for the real world CRICOS No. 00213J
What is a robot?
a university for the real world CRICOS No. 00213J
1956
1977
Robot: the word
Karel apek
1921
a university for the real world CRICOS No. 00213J
2004
Robot: a definition
where am I?
where are you?
A goal oriented machine that can
sense, plan and act.
a university for the real world CRICOS No. 00213J
What about GPS?
GPS is not perfect
Satellites are obscured in urban areas
Multi-pathing in industrial sites
Not available in many important work
domains such as
underwater
underground
deep forest
Only tells where I am
a university for the real world CRICOS No. 00213J
a university for the real world CRICOS No. 00213J
What about vision?
a university for the real world CRICOS No. 00213J
Evolution of the eye
a university for the real world CRICOS No. 00213J
Consider the bee
brain
1g
106 neurons
a university for the real world CRICOS No. 00213J
Anterior Median and Anterior Lateral Eyes of an Adult Female
Phidippus putnami Jumping Spider
a university for the real world CRICOS No. 00213J
Compound Eyes of a Holocephala fusca Robber Fly
a university for the real world CRICOS No. 00213J
Face of a Southern Yellowjacket Queen- Vespula squamosa
a university for the real world CRICOS No. 00213J
a university for the real world CRICOS No. 00213J
a university for the real world CRICOS No. 00213J
a university for the real world CRICOS No. 00213J
a university for the real world CRICOS No. 00213J
Pupil
Sclera
Iris
Pupil
Iris Sclera
a university for the real world CRICOS No. 00213J
a university for the real world CRICOS No. 00213J
brain
1.5 kg
1011 neurons
bee brain
1g
106 neurons
Seeing is an active process
a university for the real world CRICOS No. 00213J
Robots and vision
a university for the real world CRICOS No. 00213J
a university for the real world CRICOS No. 00213J
A great sensor for robots
a university for the real world CRICOS No. 00213J
Vision is the process of discovering from images
what is present in the world and where it is.
David Marr
a university for the real world CRICOS No. 00213J
Robots that read
a university for the real world CRICOS No. 00213J
a university for the real world CRICOS No. 00213J
a university for the real world CRICOS No. 00213J
Dual-camera image-based visual servo
a university for the real world CRICOS No. 00213J
Watching whales with UAVs
a university for the real world CRICOS No. 00213J
a university for the real world CRICOS No. 00213J
Robots underwater
a university for the real world CRICOS No. 00213J
a university for the real world CRICOS No. 00213J
Image
geometry
a university for the real world CRICOS No. 00213J
Reflection of light
i r
Specular reflection
- angle of incidence equals
angle of reflection
a university for the real world CRICOS No. 00213J
Reflection of light
I cos r
Lambertian reflection
- diffuse/matte surface
- brightness invariant to
observers angle of view
a university for the real world CRICOS No. 00213J
Extramission theory
a university for the real world CRICOS No. 00213J
Image formation
a university for the real world CRICOS No. 00213J
Pin hole images
a university for the real world CRICOS No. 00213J
The worlds largest pin hole camera
http://www.legacyphotoproject.com
a university for the real world CRICOS No. 00213J
Image formation
a university for the real world CRICOS No. 00213J
Use a lens to gather more light
a university for the real world CRICOS No. 00213J
Image formation
f
bigger
area
F = f /
image plane
a university for the real world CRICOS No. 00213J
Pin-hole image geometry
Y Y y
y =
Z z Z z
a university for the real world CRICOS No. 00213J
No unique inverse
a university for the real world CRICOS No. 00213J
Thin lens model
The pin-hole camera p
luminance in units of
brighter images is to c
pin A convex lenses can f
hol equivalent
object er
ay lens
pin holeallows more light
The
zi elementary aspec
z zo
inverted
f11.1.
f The
image
positive z-a
focal points and its image are relat
image plane
ideal
thin
lens
1 1 1
+ =
zo zi f
where zo is the distan
a university for the real world
focal length of the le
CRICOS No. 00213J
Thin lens model
pin
hol equivalent
object er pin hole
ay
zi
z zo
f inverted
f
image
focal points
image plane
ideal
thin
Focussing on distant objects lens
zo
zi f
a university for the real world CRICOS No. 00213J
Thin lens model
pin
hol equivalent
object er pin hole
ay
zi
z zo
f inverted
f
image
focal points
image plane
ideal
thin
lens
a university for the real world CRICOS No. 00213J
Pin-hole image geometry (3D)
Y y
=
Y x Z f
y
Z f X x
X =
Z f
fX fY
x= ,y= (X,Y, Z) (x, y)
Z Z
3 2
R R
a university for the real world CRICOS No. 00213J
Perspective transform
a university for the real world CRICOS No. 00213J
Perspective projection
Lines lines
parallel lines not necessarily parallel
Conics conics
a university for the real world CRICOS No. 00213J
a university for the real world CRICOS No. 00213J
Ideal city (1470)
a university for the real world CRICOS No. 00213J
a university for the real world CRICOS No. 00213J
Homogeneous coordinates
Fig. 2.1.
a " ! ! '
" $ " % " ! " "
! # " " b
Cartesian homogeneous
"! ! % " ! "
" " " *! " ( )
% "# ! ! '
P = (x, y, 1)
" $ ! & !
P = (x, y) " ' " ! % "
% $ " ! ' " ! % "
P R 2 !% "
"
%
P P % " !
2 ! '
homogeneous Cartesian
P = (x,
y,
z) P = (x, y)
x y
x= , y=
z z
Fig. 2.2.
" ! '
" $ " ! " $ "
a university for the" real ( ) ( ) !
world CRICOS No. 00213J
( ) " $ " ( ) !
Pin-hole model in homogeneous form
X
x f 0 0 0
Y
y = 0 f 0 0
Z
z 0 0 1 0
1
x y
x = f X, y = fY, z = Z x= , y=
z z
fX fY
x= ,y=
Z Z
Perspective transformation, with the pesky divide by Z, is linear in
homogeneous coordinate form.
1 0 0 0 f 0 0 0
0 1 0 0 0 f 0 0
0 0 1 0 0 0 1 0
0 0 0 1
a university for the real world CRICOS No. 00213J
Central projection model
P = (X, Y, Z) X
x f 0 0 0
y = 0 Y
f 0 0
Z
z 0 0 1 0
1
z
opt
ica
l ax
is
p x
z=f
l
c
prin t
i pa xC
zC
poin camera
{C} origin
e p l an e yC
g
ima
y
a university for the real world CRICOS No. 00213J
Change of coordinates
P = (X, Y, Z)
1
u 0 u0
W
1 u 1
0 v v0
z 0 0 1
opt
ica 0
l ax
is 0
p
z=f
c ipal xC
H n
pri t zC
poin ) camera
( u 0, v0 {C} origin
e p lan
e yC
g
i ma
1
H
W
v
a university for the real world CRICOS No. 00213J
Complete camera model C
P = (X, Y, Z)
z
0
X {C}
x P
x f 0 0 0
Y
y = 0 f 0 0
y C
Z
z 0 0 1 0 {0}
1
extrinsic parameters
1 X
u u 0 u0 f 0 0 0 1
1 R t Y
v = 0 v0 0 f 0 0
v 013 1 Z
w 0 0 1 0 0 1 0
1
K
C
intrinsic
parameters
camera matrix
a university for the real world CRICOS No. 00213J
MATLAB example
>> cam = CentralCamera('focal', 0.015, 'pixel', 10e-6, ...
'resolution', [1280 1024], 'centre', [640 512], ...
'name', 'mycamera')
cam =
name: mycamera [central-perspective]
focal length: 0.015
pixel size: (1e-05, 1e-05)
principal pt: (640, 512)
number pixels: 1280 x 1024
T:
1 0 0 0
0 1 0 0
0 0 1 0
0 0 0 1
>> cam.C
ans =
1.0e+03 *
1.5000 0 0.6400 0
0 1.5000 0.5120 0
0 0 0.0010 0
a university for the real world CRICOS No. 00213J
MATLAB example
>> T = transl(-1,0,0.5)*troty(0.8);
>> cam.mesh(X, Y, Z, Tcam, T)
a university for the real world CRICOS No. 00213J
Fish-eye lens
a university for the real world CRICOS No. 00213J
Fish-eye imaging model
a university for the real world CRICOS No. 00213J
Imaging by reflection
a university for the real world CRICOS No. 00213J
a university for the real world CRICOS No. 00213J
a university for the real world CRICOS No. 00213J
Catadioptric imaging model
a university for the real world CRICOS No. 00213J
Multi-view
correspondence
The problem of finding the
same point in different views of
the same scene
a university for the real world CRICOS No. 00213J
The correspondence problem
a university for the real world CRICOS No. 00213J
The correspondence problem
a university for the real world CRICOS No. 00213J
The correspondence problem
a university for the real world CRICOS No. 00213J
Corner detector
a university for the real world CRICOS No. 00213J
Corner detector
a university for the real world CRICOS No. 00213J
Corner
! " %
l% %"
13.3.1
'" ' ! detector
' Corner Detectors
Classical ' ' # " ! ' " !
" * ) % % ! ' ' ! & " * *
&'%" ! ! % ! ' !&
! & ' ' # +
% '
' % ! * % " " " ) %' , ( ' ' ' & ! # " ! ' " " (! % & " ! ! ' & ! & '! % " !" ! ' % % & ' ! ! ' !
! " % & ' "# " ' ! ' ' ! ' "& * ) % % ! '% ! ' ' ! " % ' ! " &" ! " * * % ' " ! & ' ! & ' ' '
&' ! ! # + * ' " ' " & ) % & , ! ( ! ' , ' & % ! ! ' ! ' " ! (& % &' , " ' ! " ' " ! ' & ! ! " ! " ' (% % & ' " %
' & ' % # ", ! ' ' # + ' " & ! ' " % ! % " ! %" ! ' ' ! ! " % ' " % " ! % " ! # " ! ' & % % ' $ " (! ' &
'& ! ! ' ' # , + ) ' ' ( & & %! ! "' " , " % ! ! ' % ! ' ! ,& ' , ' ' " ' !" '& %! ! ' )
" '
' & ' % & ,! # , + % " ' ! ' % " % " % ! , % ' " " (! ' " ) * ' ' ! ! $ ( & " %& (! % # & " & '! ' % & " % !
" ' " ! &' ' " ! * * * & (& & ! ' ! + ' # ' %
' ! ' ' , ) ( % " " " ! % , ' ' !
% &' " % ! % # " ! ' ' '" % * & " % ) .& & "
" ( '& ' & ! &' ! # " ! ' & !, ' % & ' ! % ' " % ' * % , ' " ( ' ) % * " ' ' % ! $ (! & # & (% & # & ' )
' * " ' & " ! & & ' " ! ' ' " ! ! '* ( ' " ! '* ' * ! & ( & &% ! " ' ! ! & + ' '" (# ! ' %
( , )
! ! " ' u% % v& ' ' " % ! ( & % ' # " & ! ( ' ' ! ' ,' " % * % & ! ' ' " " % ) " ).& % # # !
(" (& , " '
! ' % & "" !
" (% & ) ' ! ! ' ' # & " ! ' & % ! ' ,' '& * ! ! ' % ' * " ! % ! ' % ' %" ! '! % ! ! '# %
'"*! & & # & " ! ,' ! ' ( ' & " ! ' ' ! % " ! & '" (! (" (
! ! " ' % ' (& ' &( ! ' , % ! ' '" " ) % # # ! !
" % ) ! ' & % ', '* ! % " ! ! '% ' ! !
" ! &# , &
* & & "% " % " ! ! ' ,# , ! &$ ( % * ! " * & & '
& % ', & (% % " ' ' * & (& & # % ) " (& , % ', & )
' " % &# ! '& ! ' % ! !
% ' " ! & ! ' ! (
) ( & ' ! ' % &' & (%
* %
& & " " % " ! ! ' ,# , ! &$ ( % * ! " *
& % ', & (% % " ' ' * & (& & # % ) " (& , %
' " % &# ! '& ! ' % ! !
% ' " ! & ! '
) ( & ( ' ! ' " ! ! ' % & / ' & ) & (( % ' " % ) % , # + ! ' ! ! ' % &' # " ! '&
' " & * % & ! ' ' " ! " ' " % ) ' '" % & ' ' ' & ! "
& " '%" # & ! ' + ! & ! && ! ' , % ! ' ! ' ! ( %
% ' " ! & " ! & $ ( ! ' , ' ' '" % ! ) & ' % " ! " (' # (' " % # " ! ' " ! !
a university for the real* world & ! (" ! ' ' " & ! % / & ) ( ' " % ) % , # +
CRICOS No. 00213J
! ' ! ! ' % &'
! ! % - ' # # %" , ! ! ' & % ', & ' * ' &(
) ( & ' ! ' % &' & (%
Corner detector
(! ' " ! / & ) ( ' " % ) %, # + ! ' ! ! ' % &' # " ! '& %
' " & * % & ! ' ' " ! " ' " % ) ' ' " % & ' ' ' & ! 13.3 " ! Point Features
More general approach
&" '%" # & !
& ' +
% ' " ! & " ! & $ ( ! ' ,
' '
!
'" %
!
!
&& ! '
)
, % ! ' ! '
& ' % " ! " (' # (' " % # " ! ' " !
! ( % "
!
Gaussian weighting matrix
* & ! " ' & %
! ! % - ' # # %" , ! ! ' & % ', & ' * ' &( "
&$ ( % % ! & '* ! ' % " ! ! ' &# % " ! &
, '
13.3 Point Features
* % & * ' ! '% + ' ' # & & & # " ! '& " & % '" ' ! '% " '
* ! " * ! ' ' % ! # # %" + ' , ' % (! ' , " % & % &
) , ) # " )' - ( * (( # ' # ! # , ' % ! ) (* " "
,) $ # ' . $ # + $ !* )$ # ) #
* % ! % ' " % -" ! ' ! ) % ' % ! '& % &# ' ) , !
! " * * % '
structure tensor
) , ) # " )' - ( * (( # ' # ! # , ' % ! ) (* " "
), $ # . ( $ # (.
+ $ " ! * )" $ # )') # " )' - ' '' )$ + ' $ * (! . ( ) ()' * )* ' ) # ($ ' * )$
$ ' ' ! ) $ # " )' - $ ' ( $ # " $ " # ) " )' - ) % )* ' ( ) # ) # ( ). ()' * )* ' $ )
!$ ! # $ * ' $ $ # )( # + !* (% ' $ + ' $ ) )$ # !!. # + ' # ) ( ' % )$ #
$ ) # $ * ' $ $ ! " # )( $ ) " )' - ' $ " % * ) '$ " ) "
'
a university # for)(the(& real
* ' world $ ' " * ! ) % ! # ) # (" $ $ ) * (# , CRICOS) No.
# 00213J
" )' - (
,* ( ! (. " * " % ' ' )' ! " # " ' ,)' & - ' '' )$ + ' $ * (! . ( ) ()' * )* ' ) # ($ ' * )$
' * (# $ ( # " % ' $ + ( ) () ! ). # ' ! ! ). $ ) ) )$ ' ' # )
Convolution
12.4.1 lConvolution
kernel
& " ) " $ $ " # $ " $ " # & % $
Defocus involves a kernel which is a & '" & " " ' ! (% % ! " % !
K
2-dimensional Airy pattern or sinc func- % ( , * * & & " ! ' " % -" ! '
tion. The Gaussian function is similar in
% ! ! " % &( ' % ! " % & " " '
shape, but is always positive whereas
the Airy pattern has low amplitude
negative going rings.
a university for the real world * & &, '% " (' ' " %
CRICOS No. 00213J ! !
0.02
0.015
0.01
0.005
0.005
0.01
0.015
0.02
15
10
1
5 10
0 5
5 0
5
10 10
15 15
a university for the real world CRICOS No. 00213J
a university for the real world CRICOS No. 00213J
' # )( (& * ' $ ' " * !)% ! # ) # (" $ $ ) * (# , ) # " )' -
)$ ' ( ' !& ' ) $ #! ' " % & ' )' - & $ ( % ' ( $ # " $ " # ) " )' - ) % )* ' ( ) # ) # ( ). ()' * )
' * (# $ ( # " % ' $ + ( ) () ! ). # ' ! ! ). $ ) ) )$ ' '
! $" ! # ( # $ * Corner detector
' $
' ). % $ # )(
!!. ! * ! ) * (# # + ! * ( % ' $ + ' + )+ $ ' $ ) ) $ # ! !
* (( # . # + ' '# ! "# )
$ )) # $ ,* ' ) $ $ (" $ $ ) # ! " % ' # )( " $ ) ' ) " )' - ' $ " % * ) '$ "
' # # ( #! )()' " (&' ! () * %' / $ & # $ ))' (" ' * ! )" % % ( ! )$ # % , # + #$ ' ! ,)' # (" 1$ ! $ )( ! ' % & * ' (# " $ # ! ' ' & 13.3 Point Features
% , ' )) # $ # (" $
' + ' " & *)$ * ' (% # $ ( & # ) (" % # ' $ ,! + ()' )+ ' " ' ! ()" ' ' ! ) ).$ " # % ),# " ' ' $ ! + ' " % )& ! ' )., ' $ ' # & )$! " , ! ) ' ) % )$ ! ' .
More" & " general '%" # & ! ' + approach ! & ! && ! ' , % ! ' ! ' ! (
% (( ' " " (! ! & ' " )$! # & )$ ( ! $ ' ' ' , ' ).# % ! ' ' ' ! " ! % . $ # ! ! * ) ,! )& ' % $ " # ! * ((" # ( ' '# ()' " % $ ' ' # + " )! # ' + " ! ! $ " ! * ((( # (* ' ' #
% "
ner , ( (. " " )' " )' - ' ' ' * (! . ( ) ()' * )* ' ) # ($ ' * )$
hatdetector.
have not been # $ (' ' ) ! )), $ * # )" ) # )' # " - $ ()'' -($ ' (# $ # ' ()'
* " (($ " # # )# )' # " (! )' - ) % )*# ' , ( )' % ! # ) )# ( ).(* " ()'" * )* ' $ )
compressed. )$ # . $ # + $ !* )$ # ) # 1 small constant edge
!$ ! # $ * ' $ $ # )( # + !* (% ' $ + ' $ ) )$ # !!. # + ' # ) ( ' % )$ #
$ ) # $ * ' $ $ ! " # )( $ ) " )' - ' $ " % * ) '$ " ) "
, ' ' # )( (&' * )' $ ' " # * + ! ) % ! * ! ( $ # ) $ # # (")( $ # $ )) 1 large *"(#
edge
,$ ' , ) # " ) )'( -"
peak (*(
to in the literature '' # * ' ( ' # # ' $ ( ! )$ ' # ( + "/ % ! * ' $ + ' (% )' ( ()# )( ! ).()' $ # # 0' ! () !# ).) ' $ ' $ )(' # ' ) ) )$# )$ ' ' ! () (' $ # # )
er detector.
a university for the real
#,)'* " ( ()! (.* "# )* " % ' ' )' ! )# ' world
(# % " ' ' ' # ,! )'(! .' & - ()'
" ). $ ' )! * ' # )$!' )' ))$' * (%+ (' $ # $ # * (!(. , (' )+ # )()'+ $ * )$ )* ' *)CRICOS
# ($+ ' # ! No.
(( * * 00213J
)$(' # ' ! " ! ' ) $ #
Harris corner value
a university for the real world CRICOS No. 00213J
Harris corner features
>> C1(1:4)
ans =
(600,662), strength=0.0054555, descrip= ..
(24,277), strength=0.0039721, descrip= ..
(54,407), strength=0.00393328, descrip= ..
(116,515), strength=0.00382975, descrip= ..
>> C1.plot()
a university for the real world CRICOS No. 00213J
Harris corner features
a university for the real world CRICOS No. 00213J
Image motion sequence
a university for the real world CRICOS No. 00213J
Comparing features
1 1 1
The pixel values I [ u, v] themselves are not
sufficiently unique
a university for the real world CRICOS No. 00213J
Feature matching
Chapter 12 Image Processing
a university for the real world CRICOS No. 00213J
MATLAB example
>> m = C1.match(C2)
m =
39 corresponding points (listing suppressed)
>> whos m
Name Size Bytes Class
Attributes
m 1x39 416 FeatureMatch
a university for the real world
>> idisp({b1, b2});
>> m.plot()
CRICOS No. 00213J
For tracking
>> t = Tracker(im, c)
200 continuing tracks, 41 new tracks, 0 retired
241 continuing tracks, 46 new tracks, 0 retired
287 continuing tracks, 34 new tracks, 0 retired
.
.
a university for the real world CRICOS No. 00213J
Tracking results
a university for the real world CRICOS No. 00213J
Problems with feature matching
Non-unique matches
many to one
visual similarity
left right
a university for the real world CRICOS No. 00213J
More problems with feature matching!
Problems:
finds only small scale features
simple neighbourhood window matching is problematic
with changes in scale and rotation
with missing parts
leading to erroneous matches
a university for the real world CRICOS No. 00213J
Epipolar
geometry
The geometry underlying
different views of the same
scene
a university for the real world CRICOS No. 00213J
Epipolar geometry
P!
P
epipolar plane
I1 ima
ge p
lane I2
ne
la
p !
2
1
p p
ge p 2
epipolar ima
1 line 2
z1 ! !
z2 x2
{1}
1
e 2
e
x1 {2}
y1
y2
1
2
a university for the real world CRICOS No. 00213J
Homogeneous coordinates
Fig. 2.1.
a " ! ! '
" $ " % " ! " "
! # " " b
Cartesian homogeneous
"! ! % " ! "
" " " *! " ( )
% "# ! ! '
P = (x, y, 1)
" $ ! & !
P = (x, y) " ' " ! % "
% $ " ! ' " ! % "
P R 2 !% "
"
%
P P % " !
2 ! '
homogeneous Cartesian
P = (x,
y,
z) P = (x, y)
x y
x= , y=
z z
Fig. 2.2. lines and points are duals
" ! '
" $ " ! " $ "
" ( ) ( ) !
a university for the real( )
world " $ " ( ) ! CRICOS No. 00213J
A line in homogeneous form
such that
T p = 0
p = (x,
y,
z)
= (l1 , l2 , l3 ) Point equation of a line
y = mx + c
a university for the real world CRICOS No. 00213J
Line joining two points
p 2 = (d, e, f )
p 1 = (a, b, c)
= p 1 p 2
a university for the real world CRICOS No. 00213J
Point joining two lines
2 = (d, e, f )
p
handles the case of
non-intersecting lines
p = 1 2 automatically
a university for the real world CRICOS No. 00213J
Fundamental matrix
2 2
! ! p =0
Fundamental
matrix
2 T 1
p F p=0
a university for the real world CRICOS No. 00213J
Testing match validity
2 T 1
p F p=0
Now we just have to figure out F...
a university for the real world CRICOS No. 00213J
Computing F
a university for the real world CRICOS No. 00213J
The RANSAC algorithm
a university for the real world CRICOS No. 00213J
>> F = m.ransac(@fmatrix, 1e-4, 'verbose')
62 trials
6 outliers
5.94368e-05 final residual
F =
-0.0000 -0.0000 0.0052
0.0000 -0.0000 0.0010
-0.0053 -0.0023 1.0682
>> idisp({b1,b2})
>> m.outlier.plot(r);
a university for the real world CRICOS No. 00213J
How did the camera move?
essential
matrix
2 T 1
x E x = 0
must be F
p Kx
2 T T 1 1
p K2 EK1 p =0
T
E= K2 FK1 E S(t)R
a university for the real world CRICOS No. 00213J
Dealing with
scale
a university for the real world CRICOS No. 00213J
More problems with feature matching!
a university for the real world CRICOS No. 00213J
+ )1" / + 1& 3" * " + 0 1, # & + ! 1% " - , & + 1, # * 5& * 2* $ / ! & " + 1& 0 1, , * - 21" 1% " 0" , + !
! " / & 3 1& 3" + ! ! " 1" / * & + " 4 % " / " 1% & 0 & 0 7 " / , % " - ) & + , - " / 1, /
Laplacian of Gaussian sequence
Fig. 12.18. , * - / & 0, + , # 14 , + ! / " 12/ + 0 + & * $ " 4 % " / " 1% " " ! $ " 0
" ! $ " , - " / 1, / 0 a + + 6 , - " / 1, / 0- , + ! & + $ 1, $ / ! & " + 1* $ + & 12! " 11%
4 & 1% ! " # 2)1 - / * " 1" / 0 b $
1% " " ! $ " 0 / " * 2 % 1% & + + " / 1% + 1% ,
+ & 12! " , # ! " / & 3 1& 3" , # 200& +
(" / + " ) % " , , -" / , - " / 1, / 4 % & % & 0 0% , 4 + & + & $
1, / / " . 2& / " 0)" 00 , * - 21 1& , + 1% + 200& + , - " / 1& , + % " % 601" / " 0& 0 1% /
+ + 6 21$ " + " / 1" 01% & ( " / " ! $ " 0 $ 2* " + 10
& 0 1% " 02* , # 1% " 0" , + ! 0- 1& )! " / & 3 1& 3" & + 1% " % , / & 7 , + 1 ) + ! 3" / 1& )! & / " 1& , + 0 , /
, / , 1% 0" 0 / " 02)10 / " 0% , 4 + , # / 4 " % 3" , + 0& ! " / " ! + " ! $ "
& + 3" / 1" ! 4 % & 1" & 0 7 " / ,
02- - / " 00& , + % 0 " " + 20" ! 1, #,
! & 0 / " 1" & * $ " 1% & 0 + " , * - 21" ! 6 , + 3, )21& , + 4 & 1% 1% " - ) & + (" / + " ) + )1" / + 1& 3" * " + 0 1, # & + ! 1% " - , & + 1
! " / & 3 1& 3" + ! ! " 1" / * & + " 4 % " / " 1% & 0 &
>> L = klaplace()
L =
0 1 0
& 0 1% " 02* , # 1% " 0" , + ! 0- 1& )! " / & 3 1
1 -4 1 ! & 0 / " 1" & * $ " 1% & 0 + " , * - 21" !
0 1 0 >> L = klaplace()
L =
0 1 0
4 % & % & 0 & 0, 1/ , - & 8 & 1/ " 0- , + ! 0 " . 2 ))6 1, " ! $ " 0 & + + 6 ! & / " 1& , + % " 0" , + ! ! " / & 3 1 -4 1
0 1 0
1& 3" & 0 " 3" + * , / " 0" + 0& 1& 3" 1, + , & 0" 1% + 1% " # & / 01 ! " / & 3 1& 3" + ! & 0 $ & + , 4* % & *% & 0,& 0,+ 1/ )6
, - & 8 & 1/ " 0- , + ! 0 " . 2 )
20" ! & + , + ' 2+ 1& , + 4 & 1% 200& + 0* , , 1% " ! & * $ " 1& 3" & 0 " 3" + * , / " 0" + 0& 1& 3" 1, + , & 0" 1%
20" ! & + , + ' 2+ 1& , + 4 & 1% 200& + 0
4 % & % 4 " , * & + " & + 1, 1% " -) & + , # 200& + ( " / + " ) , + ! & 0 1% " -) & +
( " / + " Mexican
) $ & 3" + , 3" hat
% & 0 function
+ " 4 / & 11" + + )61& ))6 0
a university for the real world CRICOS No. 00213J
Simple scale-space example
a university for the real world CRICOS No. 00213J
Laplacian of Gaussian sequence
a university for the real world CRICOS No. 00213J
Magnitude of LoG function
this peak is the
characteristic scale of the
feature
a university for the real world CRICOS No. 00213J
Characteristic scale
a university for the real world CRICOS No. 00213J
Scale Invariant Feature Transform
100 100
200 200
300 300
v (pixels)
v (pixels)
400 400
500 500
600 600
700 700
800 800
200 400 600 800 1000 1200 200 400 600 800 1000 1200
u (pixels) u (pixels)
a university for the real world CRICOS No. 00213J
Epipolar magic example
image 1 image 2
a university for the real world CRICOS No. 00213J
484 Using multiple images
200
v (pixels)
400
600
800
Figure 14.3:
image 1 image 2
Feature matching. Subset (100 out of 1664) of matches based on SURF de-
scriptor similarity. We note some (low in the image) that some are clearly incorrect.
which results in
a university the realof SurfPointFeature objects. ManyCRICOS
two forvectors thousands
No. 00213J of
world
200
v (pixels)
400
600
800
200
v (pixels)
400
600
800
500 1000
1500 2000 2500
a university for the real world
u (pixels)
CRICOS No. 00213J
Geometry of multiple views 497
Epipolar magic
noname
100
200
300
v (pixels)
400
500 image 2
600
P
700
epipolar plane
I 1 image plan I2
e
800 lan
e
ge p
1
p epipolar ima
p
2
1 line
200 400 600 800 1000 1200 z1 ! 2
! z2 x2
{1} e
1 2
e
u (pixels) x1 {2}
image 1
re 14.10: Image from Figure 14.1(a) showing epipolar lines converging on the projection
y 1 epipolar
point
1
y2
2
e second cameras centre. In this case the second camera is clearly visible in the bottom
of the image.
a university for the real world CRICOS No. 00213J
3 dimensional
vision
a university for the real world CRICOS No. 00213J
How big is it?
a university for the real world CRICOS No. 00213J
a university for the real world CRICOS No. 00213J
The Ames room
a university for the real world CRICOS No. 00213J
The Ames room
a university for the real world CRICOS No. 00213J
How do we estimate distance?
1.Occlusion
PERCEPTION, LAYOUT, AND VIRTUAL REALITY 29
9.Motion perspective server, from 0.5 to 10,000 m, for nine sources of information about layout. I assume that more potent sources of
information are associated with smaller depth-discrimination thresholds; and that these thresholds reflect
suprathreshold utility. This array of functions is idealized for the assumptions given in Table 1. From Perceiving
Layout and Knowing Distances: The Integration, Relative Potency, and Contextual Use of Different Information
About Depth, by J. E. Cutting and P. M. Vishton, 1995, in W. Epstein and S. Rogers (Eds.), Perception of Space
and Motion (p. 80), San Diego: Academic Press, Copyright 1995 by Academic Press. Reprinted with permission.
How the eye measures reality and virtual reality.
JAMES E. CUTTING, Cornell University, Ithaca, New York.
Chauvet et al., 1995; Hobbs, 1991) and Egyptian art (see image) to the top, and assuming the presence of a ground
Behavior
Hagen, Research
1986; Hobbs, 1991),Methods, Instruments,
where it is often used alone, & Computers
plane, 1997,
of gravity, and 29 (1),
the absence of a27-36
ceiling (see Dunn,
with no other information to convey depth. Thus, one Gray, & Thompson, 1965). Across the scope of many
can make a reasonable claim that occlusion was the first different traditions in art, a pattern is clear: If one source
source of information discovered and used to depict spa- of information about layout is present in a picture be-
tial relations
in depth. yond occlusion, that source is almost always height in the
a university for the real worldBecause occlusion can never be more than ordinal in-
formationone can only know that one object is in front
visual field. The conjunction
CRICOS No. of occlusion
00213J and height,
with no other sources, can be seen in the paintings at Chau-
of another, but not by how muchit may not seem im- vet; in classical Greek art and in Roman wall paintings;
How do we estimate distance?
1.Occlusion
2.Height in visual field
3.Relative size
4.Texture density
5.Aerial perspective
6.Binocular disparity
7.Accomodation
8.Convergence
9.Motion perspective
a university for the real world CRICOS No. 00213J
How do we estimate distance?
1.Occlusion
2.Height in visual field
3.Relative size
4.Texture
Texture density
5.Aerial perspective
6.Binocular disparity
7.Accomodation
8.Convergence
9.Motion perspective
a university for the real world CRICOS No. 00213J
How do we estimate distance?
1.Occlusion
PERCEPTION, LAYOUT, AND VIRTUAL REALITY 29
9.Motion perspective server, from 0.5 to 10,000 m, for nine sources of information about layout. I assume that more potent sources of
information are associated with smaller depth-discrimination thresholds; and that these thresholds reflect
suprathreshold utility. This array of functions is idealized for the assumptions given in Table 1. From Perceiving
Layout and Knowing Distances: The Integration, Relative Potency, and Contextual Use of Different Information
About Depth, by J. E. Cutting and P. M. Vishton, 1995, in W. Epstein and S. Rogers (Eds.), Perception of Space
and Motion (p. 80), San Diego: Academic Press, Copyright 1995 by Academic Press. Reprinted with permission.
How the eye measures reality and virtual reality.
JAMES E. CUTTING, Cornell University, Ithaca, New York.
Chauvet et al., 1995; Hobbs, 1991) and Egyptian art (see image) to the top, and assuming the presence of a ground
Behavior
Hagen, Research
1986; Hobbs, 1991),Methods, Instruments,
where it is often used alone, & Computers
plane, 1997,
of gravity, and 29 (1),
the absence of a27-36
ceiling (see Dunn,
with no other information to convey depth. Thus, one Gray, & Thompson, 1965). Across the scope of many
can make a reasonable claim that occlusion was the first different traditions in art, a pattern is clear: If one source
source of information discovered and used to depict spa- of information about layout is present in a picture be-
tial relations
in depth. yond occlusion, that source is almost always height in the
a university for the real worldBecause occlusion can never be more than ordinal in-
formationone can only know that one object is in front
visual field. The conjunction
CRICOS No. of occlusion
00213J and height,
with no other sources, can be seen in the paintings at Chau-
of another, but not by how muchit may not seem im- vet; in classical Greek art and in Roman wall paintings;
How do we estimate distance?
1.Occlusion
PERCEPTION, LAYOUT, AND VIRTUAL REALITY 29
9.Motion perspective server, from 0.5 to 10,000 m, for nine sources of information about layout. I assume that more potent sources of
information are associated with smaller depth-discrimination thresholds; and that these thresholds reflect
suprathreshold utility. This array of functions is idealized for the assumptions given in Table 1. From Perceiving
Layout and Knowing Distances: The Integration, Relative Potency, and Contextual Use of Different Information
About Depth, by J. E. Cutting and P. M. Vishton, 1995, in W. Epstein and S. Rogers (Eds.), Perception of Space
and Motion (p. 80), San Diego: Academic Press, Copyright 1995 by Academic Press. Reprinted with permission.
How the eye measures reality and virtual reality.
JAMES E. CUTTING, Cornell University, Ithaca, New York.
Chauvet et al., 1995; Hobbs, 1991) and Egyptian art (see image) to the top, and assuming the presence of a ground
Behavior
Hagen, Research
1986; Hobbs, 1991),Methods, Instruments,
where it is often used alone, & Computers
plane, 1997,
of gravity, and 29 (1),
the absence of a27-36
ceiling (see Dunn,
with no other information to convey depth. Thus, one Gray, & Thompson, 1965). Across the scope of many
can make a reasonable claim that occlusion was the first different traditions in art, a pattern is clear: If one source
source of information discovered and used to depict spa- of information about layout is present in a picture be-
tial relations
in depth. yond occlusion, that source is almost always height in the
a university for the real worldBecause occlusion can never be more than ordinal in-
formationone can only know that one object is in front
visual field. The conjunction
CRICOS No. of occlusion
00213J and height,
with no other sources, can be seen in the paintings at Chau-
of another, but not by how muchit may not seem im- vet; in classical Greek art and in Roman wall paintings;
No unique inverse
a university for the real world CRICOS No. 00213J
Binocular disparity
a university for the real world CRICOS No. 00213J
a university for the real world CRICOS No. 00213J
Stereo disparity
50
100
150
200
v (pixels)
250
300
350
400
450
500
550
200 400 600 800 1000 1200
u (pixels)
a university for the real world CRICOS No. 00213J
1954-59
a university for the real world CRICOS No. 00213J
Disparity
The horizontal displacement in an image point due
to horizontal translation of the camera
fb
d
Z
b
f
f
a university for the real world CRICOS No. 00213J
2010
a university for the real world CRICOS No. 00213J
a university for the real world CRICOS No. 00213J
Prague
a university for the real world CRICOS No. 00213J
a university for the real world CRICOS No. 00213J
Anaglyph images
a university for the real world CRICOS No. 00213J
a university for the real world CRICOS No. 00213J
Anaglyph image
a university for the real world CRICOS No. 00213J
a university for the real world CRICOS No. 00213J
a university for the real world CRICOS No. 00213J
left
a university for the real world CRICOS No. 00213J
right
a university for the real world CRICOS No. 00213J
Shutter glasses
a university for the real world CRICOS No. 00213J
Computational
stereo
Depth from 2 images
a university for the real world CRICOS No. 00213J
Disparity
Fig. 14.21.
( & # ! ( " ' &
" (& ( " ( & (
fb
! ' ' + $ ( # & .# " ( -
)" ( (! ( ' (
d
+ " # + &# ! ( ( !
Z
a university for the real world CRICOS No. 00213J
T =
vL
d
dmax
left right
uL uL
Computational stereo
50
100
150
50 85
200
100
v (pixels)
250
300
80
350
400
450
150 75
500
550
200 400 600 800 1000 1200 200
u (pixels)
70
v (pixels)
250
65
300
60
350
55
400
450 50
500 45
550 40
100 200 300 400 500 600
u (pixels)
a university for the real world CRICOS No. 00213J
50 85
100 80
150 75
200
70
v (pixels)
250
65
300
60
350
55
400
450 50
500 45
550 40
100 200 300 400 500 600
u (pixels)
Planar
homgraphy
The problem of finding the
same point in different views of
the same scene
a university for the real world CRICOS No. 00213J
Homography
n object plane
P
epipolar plane
I 1 image plane I2
H ne
e pla
1
p imag
p
2
z1 z2 x2
{1}
x1 {2}
y1
1 y2
2
2 1
p=H p
a university for the real world CRICOS No. 00213J
Homography
2 1
p=H p
a university for the real world CRICOS No. 00213J
Corresponding points
p1 q1
p2
q2
q3
p3 q4
p4
pi = (ui , vi ) qi = (ui , vi )
u1 u2 u3 u4 u1 u2 u3 u4
v1 v2 v3 v4 v1 v2 v3 v4
>> H = homography(P, Q)
>> Q = homtrans(H, P)
a university for the real world CRICOS No. 00213J
Perspective rectification
100
150
200
250
v (pixels)
300
350
400
450
500
550
100 200 300 400 500 600 700 800 900 1000
u (pixels)
2 1
p=H p >> homwarp(H, im, 'full')
a university for the real world CRICOS No. 00213J
Virtual camera
# & % &
& & % &
0 /
(" (
% &
"
>> tr2rpy(sol(2).T, 'deg')
0
ans = & &
-76.6202 9.4210 -13.8081
/ - # % "
a university for the real world CRICOS No. 00213J
Optical flow
a university for the real world CRICOS No. 00213J
Optical flow patterns
tx tz
100
100
200
200
300
300
400
400
v (pixels)
v (pixels)
500
500
600
600
700
700
800
800
900
900
1000
1000
100 200 300 400 500 600 700 800 900 1000
100 200 300 400 500 600 700 800 900 1000
u (pixels)
u (pixels)
z
100
200
300
400
>> cam.flowfield([1 0 0 0 0 0])
v (pixels)
500
600
700
800
900
1000
100 200 300 400 500 600 700 800 900 1000
u (pixels)
a university for the real world CRICOS No. 00213J
+ +
' "
$ $
Optical
"
flow
" & $ & #
" equation
$ ( " $ # " $ & $ $ "
#% #$ $% $ ! ! $ ! # $
a university for the real world CRICOS No. 00213J
Ambiguity between translation and rotation
tx 100
y 100
200 200
300 300
400 400
v (pixels)
v (pixels)
500 500
600 600
700 700
800 800
900 900
1000 1000
100 200 300 400 500 600 700 800 900 1000 100 200 300 400 500 600 700 800 900 1000
u (pixels) u (pixels)
a university for the real world CRICOS No. 00213J
tx #% #$ $% $ ! ! $ ! # $
100
200
300
400
v (pixels)
500
600
700
800
900
1000
100 200 300 400 500 600 700 800 900 1000
u (pixels)
y small f y large f
100 100
200 200
300 300
400 400
v (pixels)
v (pixels)
500 500
600 600
700 700
800 800
900 900
1000 1000
100 200 300 400 500 600 700 800 900 1000 100 200 300 400 500 600 700 800 900 1000
u (pixels) u (pixels)
a university for the real world CRICOS No. 00213J
* / * # !& % ,! + ! & - & ,! ' & $ ' * % +
+ & Motion
- of $ multiple
+)/ . points
. ## - % ,## * ' .
$ ) $ & + & % * & )) * ' & % * + & )& + + & % )& , % # %
Consider the& case
) + ) of' & three
% + * points, in matrix form
!. / / * !,
$
a university for the real
07/ % 0world $
%/ $ .07/3% 0% (( $ .% / % 2 !+ *0.
+ *0. + ((! 0$ !+ ((! .)3 !% ((
. /. % + 2 !0
CRICOS No. 00213J
Desired view
a university for the real world CRICOS No. 00213J
Current view
a university for the real world CRICOS No. 00213J
Image plane motion
a university for the real world CRICOS No. 00213J
Image plane motion
(u,
v)
(u, v)
a university for the real world CRICOS No. 00213J
IBVS simulation
a university for the real world CRICOS No. 00213J
New concepts
Image convolution
Gaussian, Laplacian operators
Homogeneous coordinates
Feature detectors
Scale-space
Geometry of image formation
Fundamental and essential matrix
Homography
Image correspondence
RANSAC
a university for the real world CRICOS No. 00213J
Further reading
and
rithms and examples in a narrative that covers robotics and com-
Control
author over many years as a practitioner of both robotics and
computer vision. It is written in a light but informative style, it is
easy to read and absorb, and includes over 1000 matlab and
Simulink examples and figures. The book is a real walk through
the fundamentals of mobile robots, navigation, localization, arm-
robot kinematics, dynamics and joint level control, then camera
models, image processing, feature extraction and multi-view
geometry, and finally bringing it all together with an extensive
FUNDAMENTAL
discussion of visual servo systems. ALGORITHMS
IN MATLAB
isbn 978-3-642-20143-1
9 783642 201431
springer.com
a university for the real world CRICOS No. 00213J
Homework
a university for the real world CRICOS No. 00213J