You are on page 1of 40

Shape-from-X

Class 12

Some slides from Shree Nayar. others


Shape-from-X

X =Shading
X =Multiple Light Sources
(photometric stereo)
X =Texture
X =Focus/Defocus
X =Specularities
X =Shadows
X =
Shape from shading

Shading as a cue for shape reconstruction


What is the relation between intensity and
shape?
Reflectance Map
Reflectance Map
Relates image irradiance I(x,y) to surface
orientation (p,q) for given source direction and
surface reflectance
Lambertian case: I (x, y )
k : source brightness

: surface albedo (reflectance)


c : constant (optical system)
s v
i n
Image irradiance:

I = kc cos i = kcn s


Let kc = 1 then I = cosi = n s

Reflectance Map
Lambertian case

I = cos i = n s =
( pps + qq s + 1) = R ( p, q )
p 2 + q 2 + 1 pS2 + qS2 + 1

Reflectance Map
Iso-brightness contour (Lambertian)

cone of constant i
Reflectance Map
Lambertian case
iso-brightness
contour
q
0.8
0.9
1.0 R( p, q) = 0.7
( pS , qS )

p
i = 90
( ppS + qqS + 1 = 0)
0.3

0.0
Note: R( p, q ) is maximum when ( p, q ) = ( pS , qS )
Shape from a Single Image?
Given a single image of an object with known
surface reflectance taken under a known light
source, can we recover the shape of the object?
Given R(p,q) ( (pS,qS) and surface reflectance)
can we determine (p,q) uniquely for each image
point?
q

NO
Solution

Add more constraints


Shape-from-shading

Take more images


Photometric stereo
Stereographic Projection

(p,q)-space (gradient space) (f,g)-space z


z

S 1 z =1 s 1 z =1
N q n g

s
n s n
p f
y
y
x
x
Problem
1
(p,q) can be infinite when = 90
2p 2q
f = g=
1+ 1+ p2 + q2 1+ 1+ p2 + q2

Redefine reflectance map as R( f , g )


Occluding Boundaries

e
n v

e
n
n e, n v n = e v e and v are known

The n values on the occluding boundary can be used as


the boundary condition for shape-from-shading
Image Irradiance Constraint

Image irradiance should match the


reflectance map

Minimize
2
ei = (I (x, y ) R( f , g )) dxdy
image

(minimize errors in image irradiance in the image)


Smoothness Constraint

Used to constrain shape-from-shading


Relates orientations (f,g) of neighboring
surface points
Minimize

es = ( x y ) ( x y )dxdy
f 2
+ f 2
+ g 2
+ g 2

image

( f , g ) : surface orientation under stereographic projection


f f g g
fx = , f y = , gx = , g y =
x y x y

(penalize rapid changes in surface orientation f and g over the image)


Shape-from-Shading

Find surface orientations (f,g) at all image


points that minimize
weight

e = es + ei
smoothness
constraint

image irradiance
Minimize error
2
( f )+ (g + g ) + (I (x, y ) R( f , g )) dxdy
2 2 2 2
e= x +f y x y
image
Results
Results
Solution

Add more constraints


Shape-from-shading

Take more images


Photometric stereo
Photometric Stereo

(p , q )
1
S
1
S

p
(p , q )
2
S
2
S

(p , q )
3
S
3
S
Photometric Stereo k : source brightness
: surface albedo (reflectance)
Lambertian case: c : constant (optical system)

kc
I= kc cos i = n s = 1

n Image irradiance:
s2 s3 I1 = n s1
s1 v I 2 = n s2
I 3 = n s 3

We can write this in matrix form:


T
I
1 s 1

I = sT n
2 2
I 2 sT3

Solving the Equations
T
I
1 s 1

I = sT n
2 2
T
I
2 s
3
I S ~
n
31 3 3 31

~ 1
n=S I inverse
~
=n
~ n
n ~
n= ~ =
n
More than Three Light Sources
Get better results by using more lights
T
I1 s 1
= n

I N sTN

Least squares solution:


I = Sn~ N 1 = (N 3)(31)
T T ~
S I = S Sn
~ ( )
T
n= S S S I
1 T

Solve for , n as before


Moore-Penrose pseudo inverse

Color Images
The case of RGB images
get three sets of equations, one per color channel:

I R = RSn
I G = G Sn
I B = BSn
Simple solution: first solve for n using one channel
Then substitute known n into above equations to get

( R , G , B )
Or combine three channels and solve for n
I = I 2R + I 2G + I 2B = Sn
Computing light source directions

Trick: place a chrome sphere in the scene

the location of the highlight tells you the source


direction
Specular Reflection - Recap
For a perfect mirror, light is reflected about N

n Ri if v = r
s i i r Re =
v 0 otherwise

We see a highlight when v=r


Then s is given as follows:

s = 2(n r )n r
Computing the Light Source Direction
Chrome sphere that has a highlight at position h in the image
N

H h

rN

C c

sphere in 3D

image plane

Can compute N by studying this figure


Hints:
use this equation:
can measure c, h, and r in the image
Depth from Normals

V2
V1

Get a similar equation for V2


Each normal gives us two linear constraints on z
compute z values by solving a matrix equation
Limitations

Big problems
Doesnt work for shiny things, semi-
translucent things
Shadows, inter-reflections

Smaller problems
Camera and lights have to be distant
Calibration requirements
measure light source directions, intensities
camera response function
Trick for Handling Shadows
Weight each equation by the pixel brightness:

I i (I i ) = I i (n s i )
Gives weighted least-squares matrix equation:

I12 I1sT1

= n
I N2 I N sTN

Solve for ,n as before
Original Images
Results - Shape

Shallow reconstruction Accurate reconstruction


(effect of interreflections) (after removing interreflections)
Results - Albedo

No Shading Information
Original Images
Results - Shape
Results - Albedo
Results

1. Estimate light source directions


2. Compute surface normals
3. Compute albedo values
4. Estimate depth from surface normals
5. Relight the object (with original texture and uniform albedo)
Shape from texture

Obtain normals from texture element (or


statistics) deformations,

Examples from Angie Loh


Shape from texture

Obtain normals from texture element (or


statistics) deformations,

Examples from Angie Loh


Depth from focus

Sweep through focus settings


most sharp pixels correspond to depth
(most high frequencies)
Depth from Defocus

More complicates, but needs less images


Compare relative sharpness between
images
Shape from shadows

S. Savarese, M. Andreetto, H. Rusmeier, F.


Bernardini, P. Perona, 3D Reconstruction by
Shadow Carving: Theory and Practical Evaluation,
International Journal of Computer Vision (IJCV) ,
vol 71, no. 3, pp. 305-336, March 2007.
Shape from specularities

Toward a Theory of Shape from Specular Flow


Y. Adato, Y. Vasilyev, O. Ben-Shahar, T. Zickler
Proc. ICCV07

You might also like