Graph Cuts based

segmentation
methods:

Daniel Heilper,
CS Department, Haifa University
Topics:

Segmentation as Graph cuts problems.
MIN-CUT and Normalized CUT approaches
Examples of Cost/Energy functions
Performance issues feasibility and
approximations
Relations to Clustering and other methods
Motivation:

Human perception takes into account
several perceptual aspects and blends
them together:
Proximity:

Human perception
Proximity:

Closure:

Human perception
Similarity


Continuation

Early segmentation methods
Boundary based methods
- based on local information and cannot
guarantee continuous closed edge contours
(Derivatives kernels, Harris, Canny)
Region based methods
+ statistics inside the region.
- often generates irregular boundaries and small
Holes.

Boundary based methods
Original text Canny:
Boundary based methods
Derivatives:

Region based methods

Global optimization approaches based
on energy functions
Construct a heuristic energy function which
take into account both the local information
and the global information.
Graph Representation:
The set of points in an arbitrary feature
space are represented as a weighted
undirected
graph G = (V,E,w) where the nodes of the
graph are the points in the feature space,
and an edge is formed between every pair
of nodes. The weight on each edge, Wij is
a function of the similarity between nodes i
and j.
Graph Representation:
Energy cost functions:
Energy Function is heuristic for
quantization of a combination of Data
Features on an N-D Image.
Simple Examples: Distance metric Image
is sent as a binary image, graph is
represented as points in the plane
Energy functions for Graph Cuts
In some cases, energy minimization is
NP-Hard, or has high polynomial order
which make the method impractical
Choosing a good energy function is not
Pure science
Energy functions for Graph Cuts
The most Natural Energy form is:



V - Boundary Penalty caused by choosing
labels for 2 pixels
D Regional Penalty for assigning a pixel
to some label.
First approach: Max-Flow Min-Cut
Refresher: Flow Networks:

Network flow graphs
Given A flow Networks where:
is a directed graph
is a source (not necessary belongs to G)
is a sink (not necessary belongs to G)
a non-negative capacity function on the
edges (if we can assume )
) , ( E V G =
( ) c t s G N , , , =
V s e
V t e
+
R E c :
( , ) 0 c u v = ( , ) u v E e
Network flow graphs
Function is a valid flow function if holds:
Capacity constraints: ,
counter symmetry: for each ,


flow conservation: , where
, are the edge set that leave V.
Similarly are the sedge set that
enters V
( ) ( )
, , f u v c u v s
( ) E v , u e
, v u V e
( ) ( ) u , v f v , u f =

+
e e
=
) ( ) (
) ( ) (
v N e v N e
e f e f
) (v N
+
) (v N

{ } t s V v , \ e
Network flow graphs
The value of the flow function is


( ) =

eV v
v , s f

+
e e

) ( ) (
) ( ) (
s N e s N e
e f e f
Network flow graphs
Network flow graphs
Network flow graphs
Max-Flow Min-Cut Theorem:
The maximum amount of flow is equal to
the capacity of a minimum cut
Interactive Segmentation application
using Graph Cuts:
Motivation: for some applications: Fully
automatic segmentation does not give
sufficient results.
for example, in medical diagnostics,
Doctors handlings a medical scan images
where soft tissues, blood vessels, and
bones are blended together , are very
difficult to segment.
Above: CAT scan of Pancreas.
Interactive Segmentation application
using Graph Cuts
In a nutshell:
User marks some pixels as seeds of Background and
Object, and runs the application several times until a
satisfactory result achieved.
Interactive inputs:
seeds pixels
the ratio between regional importance VS boundary
importance
Energy function (Intensities, distances, textures, etc..)
Seeds in the flow graph
Interactive Segmentation application
using Graph Cuts
First run, application fully solves max-flow task
Each time the application outputs an insufficient
result, the user adds more seeds, (changes the
ratio ) and rerun the application.
The segmentation is improved until satisfactory by
user.
Only the first run is computationally heavy, while
the consecutive runs are fast, and can be seen as
an addition to capacities, and hence finding
additional augmenting paths in the flow function
Past Interactive tools for segmentation:
intelligent scissors
live wire
shortest path between certain pixels using
some energy function
Difficulties:
the user inputs have to be very accurately
positioned at the desired boundary
can not be easily generalized to 3D data.
Energy function:
General form of the energy function:


Where



Energy function:

R(A) stands for the sum of the regional
penalties of a given labeling
Similarly B(A) is the sum of the boundry
penalties of the labeling
3D - Data
Video sequence is treated as a 3D volume
image:

Second Method: Normalized Cut:
The normalized cut criterion measures both
the total dissimilarity between the different
groups as well as the total similarity within
the groups.
The key point is that image partitioning is to
be done from the big picture downward,
rather like a painter first marking out the
major areas and then filling in the details.

Normalized Cut
Unfortunately, minimizing normalized cut
exactly is NP-complete, even for the
special case of graphs on
grids(Papadimitriou). However, an
approximate discrete solution can be found
efficiently
MINCut problem
Why minimum cut isnt always favorable?
the minimum cut criteria favors cutting
small sets of isolated nodes in the graph.
(for example when cost function is inverse
to the distance between nodes)
A case where minimum cut gives a
bad partition





.
Definition: Normalized cut



where assoc(A,V) is the total connection
from nodes in A to all nodes in the graph
Normalized cut
It is easy to see that single nodded cuts
produce MINCut value of at least 1, so
minimum normal cut is not likely to be
single nodded.
Definition: Normalized Association:
In the same spirit, we can define a measure
for total normalized association within
groups for a given partition:

NAssoc vs NCut
A direct conclusion:
NCut(A,B) = 2 Nassoc(A,B)
Ncut is normalized measure of
disassociation
minimizing Ncut is equivalent to maximizing
association
Computing the Optimal Partition
An approximate discrete solution can be
found efficiently by converting the problem
into a generalized eigenvector problem
NCut as Eigenvector system
Matrix representation
W is the cost Matrix
D is the sum of the costs from node i:



The Partition is a binary matrix size |P|XK where
|P| is the number of pixels and K is the number of
labels

NCut as Eigenvector system

The Segmentation matrix y:

Each column is a label vector

NCut can be written as:

Recursive two-way Ncut grouping
algorithm
1. Given an image or image sequence, set
up a weighted graph G=(V, E) and set
the weight on the edge connection two
nodes to be a measure of the similarity
between the two nodes.
2. Solve for eigenvectors
with the smallest eigenvalues

Recursive two-way Ncut grouping
algorithm
3. Use the eigenvector with the second
smallest eigenvalue to bipartition the
graph
4. Decide if the current partition should be
subdivided and recursively repartition the
segmented parts if necessary.
Experiments on energy functions:






where X(i) is the spatial location of node i, and F(i)
is a feature vector based on intensity, color, or
texture information
If can separate the regional weight from the
boundary weight using log operation





(a) A synthetic image showing a noisy step image. Intensity
varies from 0 to 1, and Gaussian noise with sigma = 0.2 is
added.
(b) eigenvector with the second smallest eigenvalue
(c) resulting partition.


Performance and optimality
Solving an eigenvector system takes at
least (|P|^3) time
Using iterative power method we can find
the second lowest eigenvector in a
O(|P|*|n|) where |P| is #pixels, and |n| is
#iterative steps
The NCut at the end of the algorithm has
proven constant proximity to the MIN NCut
Finding the Minimum Normalized Cut
VS interactive MINCut finding:

Min NCut Pros:
Avoids segmenting isolated small sections (VS
blind calculation of MINCut)
This NP-Hard can be well approximated
Min NCut cons:
Difficult to control and analyse
Finding the Minimum Normalized Cut
VS interactive MINCut finding
interactive MINCut Pros:
Allows a fast correction of the result in an
interactively manner, and converges
Easy to control
Energy Minimization Techniques for
Graph Cuts
Goal: giving a faster algorithm to the MIN-
Cut problem or a general K-Labeling
partition of an Image
Some energy functions makes the MinCut
problem to be NP-Hard
Other functions reach optimal solution in
impractical polynomial time
Known algorithms for energy
minimization:
Simmulated Anealing converges well but for
small temperature deltas is exponential in time,
and for large deltas it converges to local
minimum which may be far from the minimum
Combinatorial Max flow Min cut algorithms.
Combinatorial algorithms solves the problem in
at least O(|P|^3) time (Tarjan & Goldberg).
dynamic programming restricted to 1-D energy
function

linear time algorithm for finding the
Max-Flow-Min-Cut/
Updating the labeling function:
two types of large moves: -expansion and
--swap

-Exapnsion

Take a subset of pixels which are not
labeled to have that label
Works for a metric energy function:


- swap
Changing a label of some pixels to label

Works for semimetric Energy functions:

-Exapnsion
-Exapnsion and {, }Swap
-Exapnsion and {, }Swap
these moves allow large number of pixels
to change their labels simultaneously
Note that a move which assigns a given
label to a single pixel is both an - swap
and an -expansion.
(- swap can be seen as a private case of
-Exapnsion)
{ } Expansion
1. Start with an arbitrary labeling f
2. Set success := 0
3. For each label belongs L
3.1. Find f = argminE(f0) among f0 within one
-expansion of f
3.2. If E(f*) < E(f), set f := f* and success := 1
4. If success = 1 goto 2
5. Return f

{, } Swap algorithm
1. Start with an arbitrary labeling f
2. Set success := 0
3. For each pair of labels {, } belongs to L
3.1. Find f* = argminE(f) among f within one - swap
of f
3.2. If E(f*) < E(f), set f := f* and success := 1
4. If success = 1 goto 2
5. Return f
e
e
Algorithms summary
A cycle is successful if a strictly better
labeling is found at any iteration.
The algorithms stop after the rst
unsuccessful cycle since no further
improvement is possible.
-Exapnsion much faster than - swap :
|L| VS o(|L|^2) iterations (L is the labeling)
Benchmark results






(Results of running the algorithms on the previous
image)



References:

[1] J. Shi and J.Malik,
Normalized Cuts and Image Segmentation, Proc. CVPR 1997.
[2] Boykov, Y., Jolly, M., " Interactive graph cuts for optimal
boundary and region
segmentation of objects in N-D images." In: International
Conference on Computer Vision, Vancouver , BC. (2001) 105112
[3] Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy
minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell.
23 (2001) 12221239

[4] Goro Obinata, Ashish Dutta: Vision Systems: Segmentation and
Pattern Recognition, I-Tech Education and Publishing, Vienna,
Austria