Professional Documents
Culture Documents
be Minimized Using
Graph Cuts?
Shai Bagon
Advanced Topics in Computer Vision
June 2010
E
a number
-20
237
Foreground Selection
Let
yi color of ith pixel
xi {0,1} BG/FG labels (variables)
F energy:
E(x)=iEi(xi)+ijEij(xi,xj)
2
xi
xm
xn
Submodular
Known concept from set-functions:
f x f y f x y f x y x, y S
f 1,0 f 0,1 f 1,1 f 0,0
S 2
xi
Eij(xi,xj):
What does it
mean?
B+C-A-D 0
F2 submodular
How to
Minimize?
E(x) = i Ei(xi) + ij Eij (xi, xj), xi {0,1}
Local beliefs:
Prior knowledge:
Data term
Smoothness term
Graph Partitioning
V EE ww )
A weighted graph G=( V
Special Nodes: s t
s-t cut: S V, T V , s S, t T
S T , S T V
Cost of a cut: Cut ) S , T ( wij
iS , jT
wij
D-C
s
Ej(1)
B+C-A-D
j
i
Eij(xi,xj)
xj
xi
Ei(0)
=A+
C-A C-A
0 D-C
0 D-C
0 B+C-A-D
0
C-A
D-C
s
Ej(1)
B+C-A-Dassignment
st cut binary
i
0 vS
cut
xv cost energy of assignment
Ei(0)
1 v T
t
min cut Energy min.
cut S , T A wij
iS , jT
A E j 1 Ei 0 D C B C A D
E x
B=Eij(0,1)
Recap
F2 submodular:
E(x) = i Ei(xi) + ij Eij (xi, xj)
Eij(1,0)+Eij(0,1)Eij(0,0)+Eij(1,1)
Next
Multi-label F2
E(x)=i Ei(xi) + ij Eij(xi,xj) s.t. xi {1,,L}
Fusion moves: solving binary sub-problems
Applications to stereo, stitching, segmentation
Solve Binary problem:
xi=0
xi=1
Current
labeling
=
suggested
labeling
Fusion
Alpha
expansion
Stereo matching
see http://vision.middlebury.edu/stereo/
Input:
Pairwise MRF
[Boykov et al. 01]
Ground
truth
slide by Carsten Rother, ICCV09
Panoramic stitching
Panoramic stitching
AutoCollage
Next
Multi-label F2
E(x)=i Ei(xi) + ij Eij(xi,xj) s.t. xi {1,,L}
Fusion moves: solving binary sub-problems
Applications to stereo, stitching, segmentation
Non-submodular
Beyond pair-wise interactions: F3
Merging Regions
regions (Ncuts)
input image
edge prob.
j
i
pi
weak edge
i: x i 0
i: xi 0
Taking -log
Merging Regions
x
i: xi 1
log pi
i
i: xi 0
i: xi 0
log1 p log p
i
i: xi 0
i:xi 0
pi Adding and
1 xi C wi xi
C log
1 pisubtracting
the i
i
same number
pi 1
likely to be an edge
2
1 1 pi likey to be merged
pilog
2
i: x 0
wi 0
pi
wi log
x1pi : wlog
p
i 0 i
def
i: xi 1
Merging Regions
Solving for edges:
arg min x C i wi xi
Consistency constraints:
No dangling edge
J
x1
x2
x3
EJ
xi
wi
1 longer
E J 1 x1No
x2 x3
x2 x3 1 x1 x3
1 x1 pair-wise:
F3
Minimization trick
1 x1 x2 x3 min z 1 x1 x2 x3 2
z 0 ,1
xi min z i 1 xi K 1
i 1
z 0 ,1
Freedman D., Turek MW, Graph cuts with many pixel interactions:
theory and applications to shape modeling. Image Vision Computing 2010
Merging Regions
The resulting energy:
E x
min z n wn xn
l ,m ,nJ zlmn xl xm xn 1 1 xl xn
+ Pair-wise
- Non submodular!
s
j
i
i
Kolmogorov V., Carsten R., Minimizing non-submodular functions with graph cuts a review.
PAMI07
yi 1 if i T , i S
otherwise
partial labeling
s
i
j
t
s
i
j
t
QBPO - Probing
QPBO:
Probe Node p:
0
QBPO - Probing
Probe nodes in an order until energy
unchanged
Simplified energy preserves global
optimality and (sometimes) gives the global
minimum
Merging Regions
Result using QPBO-P:
input image
regions (Ncuts)
Result
Recap
F3 and more
Minimization trick
Non submodular
QPBO approx. partial labeling
Beyond F3
[Kohli et. al. CVPR 07, 08, PAMI 08, IJCV 09]
Image Segmentation
E(X) =
ci xi + dij |xi-xj|
i
Image
n = number of
pixels
E: {0,1}n R
0 fg, 1bg
i,j
Unary Cost
Segmentation
Pn Potts Potentials
Patch
Dictionary
(Tree)
h(Xp) =
0
if xi = 0, i p
Cmax otherwise
Cmax 0
Pn Potts Potentials
n = number of
pixels
E: {0,1}n R
0 fg, 1bg
E(X) =
h(Xp) =
ci xi + dij |xi-xj|
i
i,j
0
if xi = 0, i p
Cmax otherwise
hp (Xp)
Image Segmentation
n = number of
pixels
E: {0,1}n R
0 fg, 1bg
E(X) =
ci xi + dij |xi-xj|
i
Image
i,j
Pairwise
Segmentation
hp (Xp)
Final Segmentation
Application:
Recognition and Segmentation
Image
One
superpixelizatio
n
another
superpixelization
Pn Potts
0 if xi = 0, i p
f(xp) otherwise
p
Pn Potts
p
Robust Pn
Potts
Application:
Recognition and Segmentation
Image
another
One
supersuperpixelizatio pixelization
n
Pn Potts
One input
image
Ground
truth depth
Stereo with
hardsegmentation
Stereo with
robust Pn Potts
How is it done
Most general binary
function:
H (X) = F ( xi ) concave
H (X)
xi
min f(x)
x
Higher Order
Function
xi = 0
xi > 0
{
=
0 if all xi = 0
C1 otherwise
x {0,1}n
x,a {0,1}
Quadratic Submodular
Function
f(x) = 0
a=0
f(x) = C1
a=1
min f(x)
x
x,a {0,1}
xi
Quadratic Submodular
Function
C1xi
C1
xi
min f(x)
x
x,a {0,1}
Higher Order
Submodular Function
xi
Quadratic Submodular
Function
C1xi
a=1
a=0
Lower envelope
of concave
functions is
concave
C1
xi
Summary
Submodular F2
F3 and beyond: minimization trick
Non submodular
QPBO(P)
a=1
a=0
f1(x)
xi
s
i
j
t