Professional Documents
Culture Documents
Abstract—Efficient segmentation of globally optimal surfaces representing object boundaries in volumetric data sets is important and
challenging in many medical image analysis applications. We have developed an optimal surface detection method capable of
simultaneously detecting multiple interacting surfaces, in which the optimality is controlled by the cost functions designed for individual
surfaces and by several geometric constraints defining the surface smoothness and interrelations. The method solves the surface
segmentation problem by transforming it into computing a minimum s-t cut in a derived arc-weighted directed graph. The proposed
algorithm has a low-order polynomial time complexity and is computationally efficient. It has been extensively validated on more than
300 computer-synthetic volumetric images, 72 CT-scanned data sets of different-sized plexiglas tubes, and tens of medical images
spanning various imaging modalities. In all cases, the approach yielded highly accurate results. Our approach can be readily extended
to higher-dimensional image segmentation.
Index Terms—Optimal surface, medical image segmentation, graph algorithms, graph cut, minimum s-t cut, geometric constraint.
1 INTRODUCTION
This extension remains fully compatible with graph search- of all these approaches has been revealed and generalized by
ing, i.e., in 2D, it produces identical result when the same the Image Foresting Transform (IFT) proposed by Falcão et al.
objective function and hard constraints are employed. [24]. Yet another class of powerful approaches to multi-
Consequently, many existing problems that were tackled dimensional image segmentation is based on level sets
using graph-searching in a slice-by-slice manner can be [39], [40].
migrated to our new framework with little or no change to
the underlying objective function formulation. The pro- 2.2 Energy Minimization Using Graph-Cuts
posed method is limited to handling terrain-like (height- The work most relevant to the method presented here is the
field) and cylindrical surfaces. This limitation seems severe energy minimization framework using minimum s-t cuts
at first sight. However, as we will demonstrate, the established by Boykov et al. [36], [41] and Kolmogorov et al.
guarantee of global optimality and the freedom to design [42]. The cost function follows the “Gibbs” model [43]:
a problem-specific objective function allow the method to
EðfÞ ¼ E data ðfÞ þ E smooth ðfÞ;
be applied to a variety of medical image segmentation
problems. In the context, the method is referred to as a where f is a labeling of the image pixels. To minimize EðfÞ,
3D approach regardless of the dimension of the graph being a special class of arc-weighted directed graphs Gst ¼
used, as opposed to the slice-by-slice (2D) approaches. Note ðV [ fs; tg; EÞ was employed. In addition to the nodes
that the preliminary results related to this research have corresponding to image pixels (voxels), the node set of Gst
been presented in a conference paper [15]. contains two special terminal nodes, namely, the source s and
the sink t. In image segmentation, the terminals typically
correspond to the labels (e.g., object, background) that can
2 BACKGROUND AND RELATED WORK
be assigned to pixels. The arcs in Gst can be classified into
2.1 Graph-Based Image Segmentation two categories: n-links and t-links. n-links connect pairs of
Graph-based approaches have been playing an important neighboring pixels whose costs are derived from the
role in image segmentation over the past several years. The smoothness term E smooth ðfÞ. t-links connect pixels with
general theme of these approaches is the formation of a terminals, whose costs are derived from the data term
weighted graph G ¼ ðV ; EÞ with node set V and arc set E. E data ðfÞ. An s-t cut (briefly, a cut) in Gst is a set of arcs
The nodes
v 2 V correspond to image pixels (or voxels) and whose removal partitions the nodes into two disjoint
arcs vi ; vj 2 E connect the nodes vi ; vj according to some subsets S and T , such that s 2 S and t 2 T and no dipath
neighborhood system. Every arc vi ; vj 2 E has a cost (or can be established from s to t. The cost of a cut is the total
weight) representing some measure of preference that the cost of arcs in the cut. A minimum s-t cut is a cut whose cost
corresponding pixels belong to the object of interest. is the minimum. The minimum s-t cut problem and its dual,
Depending on the specific application and the graph the maximum flow problem, are classic combinatorial
algorithm being used, the constructed graph can be directed problems that can be solved by various polynomial-time
undirected.
or In a directed graph (or digraph), the arcs algorithms [44], [45], [46].
vi ; vj and vj ; vi (i 6¼ j) are considered distinct and they It was proven that, if the arcs of Gst are properly
may have different costs. If a directed arc vi ; vj exists, the constructed and their costs properly assigned, a minimum
node vj is called a successor of vi . A sequence of consecutive s-t cut in Gst can be used to exactly or approximately
directed arcs hv0 ; v1 i; hv1 ; v2 i; . . . ; hvk1 ; vk i form a directed minimize EðfÞ of certain forms in an efficient way. In turn, if
path (or dipath) from v0 to vk . an energy function is appropriately designed, a minimum
Typical graph algorithms that were exploited for image s-t cut can segment an image into objects and background
segmentation include minimum spanning trees [16], [17], as desired. Several medical image segmentation techniques
[18], [19], [20], shortest paths [7], [12], [21], [22], [23], [24], based on this framework were developed by Boykov and
[25], [26], [27], and graph-cuts [28], [29], [30], [31], [32], [33], Jolly [33], [34] and Kim and Zabih [38]. Kim and Zabih’s
[34], [35], [36], [37]. Graph-cuts are relatively new and method was designed specifically for contrast-enhanced
arguably the most powerful among all graph-based MR images. Boykov and Kolmogorov’s algorithm [47] is
mechanisms for image segmentation. They provide a clear flexible and shares some elegance with the level set
and flexible global optimization tool with considerably methods. However, it needs the selection of object and
good computational efficiency. background seed points, which is difficult to achieve for
The introduction of graph-cuts into medical image many applications. Besides, without taking advantage of
analysis happened only recently [14], [33], [38]. Classic the prior shape knowledge of the objects to be segmented,
optimal boundary-based techniques (e.g., dynamic program- the results are topology-unconstrained and may be sensitive
ming, A graph search, etc.) were used on 2D problems [27]. to initial seed point selections.
However, their 3D generalization, though highly desirable,
has been unsuccessful for over a decade [8], [9], [10]. As a 2.3 Segmentation of Coupled Surfaces
partial solution, region-based techniques such as region Due to the imperfections of medical imaging techniques,
growing or watershed transforms were used. However, they insufficient image-derived information may be available
suffer from an inherent problem of “leaking.” Advanced for defining an object boundary or surface. This insuffi-
region growing approaches incorporated various knowl- ciency can be remedied by using clues from the other
edge-based or heuristic improvements (e.g., fuzzy connect- mutually related boundaries or surfaces. Cooptimization of
edness) [21], [22]. The underlying shortest-path formulation multiple coupled surfaces thus frequently yields superior
LI ET AL.: OPTIMAL SURFACE SEGMENTATION IN VOLUMETRIC IMAGES—A GRAPH-THEORETIC APPROACH 121
where wðV Þ is fixed and is the cost sum of all nodes with a priori knowledge about expected border position can be
negative costs in G. Since S fsg is a closed set in G [57], modeled:
[58], the cost of a cut ðS; T Þ in Gst and the cost of the
corresponding closed set in G differ by a constant. Hence, eðx; y; zÞ ¼ ð1 j!jÞ ðI ? Mfirst derivative Þðx; y; zÞ
ð6Þ
the source set S fsg of a minimum cut in Gst corresponds þ_ ! ðI ? Msecond derivative Þðx; y; zÞ:
to a minimum closed set C in G.
The þ_ operator stands for a pixel-wise summation, and ? is
Because the graph Gst has OðknÞ nodes and OðknÞ arcs,
a convolution operator. The weighting coefficient 1 !
the minimum closed set C in G can be computed in
1 controls the relative strength of the first and second
T ðkn; knÞ time, herein, T ðkn; knÞ is the time for finding a
derivatives, allowing accurate edge positioning. The values
minimum s-t cut in an arc-weighted directed graph with
of !, p, and q may be determined from a desired boundary
OðknÞ nodes and OðknÞ arcs.
surface positioning information in a training set of images
The optimal k surfaces correspond to the upper envelope
(Section 6.2).
of the minimum closed set C . They can be recovered in the
following way: For each i (i ¼ 1; . . . ; k), recall that the 5.2 Nonedge-Based Cost Functions
subgraph Gi is used to search for the target surface N i . For The object boundaries do not have to be defined by
every x 2 x and y 2 y, let ViB ðx; yÞ be the subset of nodes in gradients. For example, a piecewise constant minimal
both C and the ðx; yÞ-column Coli ðx; yÞ of Gi , i.e., variance criterion based on the Mumford-Shah functional
ViB ðx; yÞ ¼ C \ Coli ðx; yÞ. Denote by Vi ðx; y; z Þ the node [60] was proposed by Chan and Vese [61] to deal with such
in ViB ðx; yÞ with the largest z-coordinate. Then, voxel situations:
I ðx; y; z Þ is on the ith optimal surface N i . In this way, Z
the minimum closed set C of G uniquely defines the EðS; a1 ; a2 Þ ¼ ðI ðx; y; zÞ a1 Þ2 dxdydz
optimal k surfaces fN 1 ; . . . ; N k g in I . insideðSÞ
Z ð7Þ
To sum up, we have the following theorem:
þ ðI ðx; y; zÞ a2 Þ2 dxdydz:
Theorem 1. The optimal k surfaces in a 3D image Iðx; y; zÞ with outsideðSÞ
n voxels can be computed in T ðkn; knÞ time. The two constants a1 and a2 are the mean intensities in the
interior and exterior of the surface S, respectively. The
Finally, the outline of the algorithm is: energy EðS; a1 ; a2 Þ is minimized when S coincides with the
. Input: k, x , y , l , u , and the cost function(s). object boundary and best separates the object and back-
. Construct the graph Gst (¼ ðV [ fs; tg; E [ Est Þ). ground with respect to their mean intensities.
. Compute the minimum s-t cut ðS ; T Þ in Gst . The variance functional can be approximated using our
. Recover the k optimal surfaces from S fsg. per-voxel cost model and, in turn, can be minimized using
our graph-based algorithm. Since the application of the
5 COST FUNCTIONS Chan-Vese cost functional may not be immediately obvious,
let us consider a single-surface segmentation example.
Designing appropriate cost functions is of paramount
Any feasible surface uniquely bipartitions the graph into
importance for any graph-based segmentation method. In
two disjoint subgraphs. One subgraph consists of all nodes
real-world problems, the cost function usually reflects
that are on or below the surface, and the other subgraph
either a region-based or edge-based property of the surface
consists of all nodes that are above the surface. Without loss
to be identified.
of generality, let a node on or below a feasible surface be
5.1 Edge-Based Cost Functions considered as being inside the surface; otherwise, let it be
A typical edge-based cost function aims to accurately outside the surface. Then, if a node V ðx0 ; y0 ; z0 Þ is on a
position the boundary surface in the volumetric image. feasible surface N , then the nodes V ðx0 ; y0 ; zÞ in Colðx0 ; y0 Þ
Such a cost function may, e.g., utilize a combination of the with z z0 are all inside N , while the nodes V ðx0 ; y0 ; zÞ with
first and second derivatives of the image intensity function z > z0 are all outside N . Hence, the voxel cost cðx0 ; y0 ; z0 Þ is
[59], and may consider the preferred direction of the assigned as the sum of the inside and outside variances
identified surface. computed in the column Colðx0 ; y0 Þ as follows:
Let the analyzed volumetric image be Iðx; y; zÞ. Then, X
the cost cðx; y; zÞ assigned to the image voxel I ðx; y; zÞ can cðx0 ; y0 ; z0 Þ ¼ ðI ðx0 ; y0 ; zÞ a1 Þ2
zz0
be constructed as: X ð8Þ
þ ðI ðx0 ; y0 ; zÞ a2 Þ2 :
cðx; y; zÞ ¼ eðx; y; zÞ pððx; y; zÞÞ þ qðx; y; zÞ; ð5Þ z>z0
where eðx; y; zÞ is a raw edge response derived from the first Then, the total cost of N will be equal to EðN ; a1 ; a2 Þ
and second derivatives of the image and ðx; y; zÞ denotes (discretized on the grid ðx; y; zÞ). However, the constants a1
the edge orientation at location ðx; y; zÞ that is reflected in and a2 are not easily obtained since the surface is not well-
the cost function via an orientation penalty pððx; y; zÞÞ. 0 < defined before the global optimization is performed.
p < 1 when ðx; y; zÞ falls outside of a specific range around Therefore, the knowledge of which part of the graph is
the preferred edge orientation; otherwise, p ¼ 1. A position inside and outside is unavailable. Fortunately, our graph
penalty term qðx; y; zÞ > 0 may be incorporated so that construction guarantees that, if V ðx0 ; y0 ; z0 Þ is on N , then the
LI ET AL.: OPTIMAL SURFACE SEGMENTATION IN VOLUMETRIC IMAGES—A GRAPH-THEORETIC APPROACH 125
Fig. 5. Effect of intrasurface smoothness constraints. (a) Cross-section of the original image. (b) The first cost image. (c) The second cost image.
(d) Results obtained with smoothness parameters x ¼ y ¼ 1. (e) Results obtained with smoothness parameters x ¼ y ¼ 5.
separate training subset of expert-defined independent Maximum and mean surface positioning errors (signed
standard. The training image data were not used for and unsigned) were computed for each point on a regular
performance evaluation. In phantoms, the cost functions grid of the independent standard surface as the shortest
were optimized to maximize the agreement between the distances between the independent standard and the
ground truth and the computer-defined segmentation computer-identified surface. These errors were reported as
results. Specifically for the pulmonary CT data, a physical mean
standard deviation in absolute measurements and
phantom containing six plexiglas tubes with sizes ranging as percentages of the diameter. The diameter errors were
from 1:98 to 19:25 mm was imaged by multidetector CT and reported as absolute and diameter-percentage errors, both
analyzed using our surface segmentation method. The signed and unsigned.
corresponding outer wall diameters ranged from 4:45 to The surface-to-surface wall thickness was defined as the
25:50 mm. The CT scans were taken at four distinct angles local distance between the outer and inner wall surfaces
of 0 , 5 , 30 , and 90 , rotated in the coronal plane to and was measured at 15 intervals along angular projec-
represent oblique airway positioning with respect to the CT tions from the tubular structure centerline in individual
imaging planes. cross-sections. Linear regression analysis was used to
compare the per-slice mean airway wall thicknesses
6.3 Execution Time
computed from the observer-defined and computer-seg-
The execution times were recorded and compared in
mented airway wall borders. The regression equations were
242 computer phantoms of varying sizes to gain a basic
compared to the line of identity using t-statistics for the
understanding of the speed/size relationship. All experi-
slope and intercept.
ments were conducted on a standard 1.67 GHz workstation
On tubular structures for which segmentation results
with 3.5 GB of memory. The execution time for each test
obtained by a 2D dynamic programming approach were
case was measured three times and the results were
available [62], these results were compared against those
averaged. The execution time included the graph initializa-
obtained using the reported 3D surface segmentation. The
tion time and the actual computation time. Two standard
two approaches shared the same pre/postprocessing steps
implementations of the proposed algorithm were tested,
and utilized training-optimal cost functions. Linear regres-
which used two different minimum s-t cut/maximum flow
sion analysis of the minor and major inner diameters
algorithms: the “Boykov-Kolmogorov” (BK) algorithm [36]
measured from the two segmentation results was per-
and the highest-level “push-relabel” (PR) algorithm with
gap relabeling and global relabeling heuristics [66]. For a formed. The measurements were compared using paired
graph with n nodes and m arcs, the theoretical worst-case t-text for equivalence. In cases for which an expert-defined
time-complexities for these algorithms are Oðn2 mcÞ and independent standard was available, the unsigned border
pffiffiffiffiffi (surface) positioning errors were compared using a paired
Oðn2 mÞ, respectively, where c is the cost of the minimum
cut [36]. To ensure a fair comparison, the two implementa- t-text. In all cases, p ¼ 0:05 was considered statistically
tions used near identical “forward-star” graph representa- significant.
tions. For single-surface detection, the BK algorithm was In all reported cases, the segmentation was performed
implemented using a memory-efficient “implicit-arc” re- fully automatically with no human interaction.
presentation [14] and was compared to the standard
schemes. 7 RESULTS
6.4 Segmentation Accuracy Assessment Indices 7.1 Computer Phantoms
Surface detection performance was assessed using surface The first group of 3D phantoms contained three separate
positioning errors in all cases for which independent surfaces embedded in the image. The goal was to identify
standard was available. On tubular structures, the perfor- two of the three surfaces based on some supplemental
mance was also assessed using major and minor diameter surface properties (smoothness in this case). The lower
errors. The errors of the measured diameters were two surfaces were smoother in comparison with the
determined whenever possible. In multiple-surface detec- topmost surface. The lowest surface exhibited a slightly
tion tasks, surface-to-surface (wall) thickness errors were darker brightness and, thus, was fixed by setting the cost-
calculated. The independent standards were defined manu- function to be attracted by low magnitude brightness
ally by expert observers. (Fig. 5a). The cost function for the second surface yielded
LI ET AL.: OPTIMAL SURFACE SEGMENTATION IN VOLUMETRIC IMAGES—A GRAPH-THEORETIC APPROACH 127
Fig. 6. Effect of surface separation constraints. (a) Cross-section of the original image. (b) The first cost image. (c) The second/third cost image.
l u l u l
(d) Results obtained with separation parameters 12 ¼ 5, 12 ¼ 15 and 13 ¼ 16, 13 ¼ 25. (d) Results obtained with separation parameters 12 ¼ 5,
u l u
12 ¼ 15 and 13 ¼ 26, 13 ¼ 35.
Fig. 7. Single-surface versus coupled-surfaces. (a) Cross-section of the original image. (b) Single surface detection using our method with standard
edge-based cost function. (c) MetaMorphs method segmenting the inner border in 2D. (d) MetaMorphs with the texture term turned off. (e) Single
surface detection using our algorithm and a cost function with a shape term. (f) Double-surface segmentation obtained with our approach.
Fig. 8. Segmentation using the minimum-variance cost function. (a), (b), and (c) Original images. (d), (e), and (f) The segmentation results.
an identical magnitude for the middle and topmost surface characteristics (texture) and edge properties (shape) of the
positions (Fig. 5c). Consequently, the resulting surface image into a single model. For this particular example, due
detection could be fully controlled by the smoothness to the smoothly changing intensity and some exterior
constraints. Figs. 5d and 5e show controlled detection of the intensities being similar to interior, the Mumford-Shah
lower surface interacting with either the middle or the style texture term did not have a positive influence on the
upper surface. In the first case, the smoothness parameters final segmentation. The shape term, being derived using the
x and y were both set to 1. In the second case, the Canny edge detector followed by an unsigned distance
smoothness parameters were set so that the resulting transform, played a leading role in producing the successful
surface was the topmost surface. boundary (Fig. 7d). Our approach using a cost function
Fig. 6 shows the result of a triple-surface detection formulated in the same way as Huang’s shape term
experiment. The surfaces in the data set are 10 voxels apart. produced an equally good result as the MetaMorphs model
The algorithm was set to always identify the lowest surface (Fig. 7e). Fig. 7f demonstrates our method’s ability to
and select two out of the three surfaces above it, which was segment both borders of the sample image. In comparison,
fully controlled by the separation constraints. the current MetaMorphs implementation was unable to
A volumetric image shown in Fig. 7a consisted of segment the outer contour.
three identical slices stacked together to form a 3D volume. Fig. 8 presents segmentation examples obtained by our
As shown, the gradual change of intensity causes the algorithm using the minimum-variance cost function (Sec-
gradient strengths to locally vanish. Consequently, border tion 5.2). The objects and background were differentiated by
detection using an edge-based cost function fails locally their different textures. In Figs. 8a and 8d, the cost function
(Fig. 7b). By using an appropriate cost function, the inner was computed directly on the image itself. For Figs. 8b and
elliptical boundary could be successfully detected using the 8e and Figs. 8c and 8f, the curvature and edge orientation in
2D MetaMorphs method developed by Huang et al. when each slice were used instead of the original image data [61].
attempting to segment the inner contour in one image slice The two boundaries in Figs. 8c and 8f were segmented
[67] (Fig. 7c). Huang et al.’s method combines the regional simultaneously.
128 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 28, NO. 1, JANUARY 2006
TABLE 1
7.4 Airway Wall Segmentation
Average Execution Times While inner wall surfaces are well visible in CT images,
outer airway wall surfaces are very difficult to segment due
to their blurred and discontinuous appearance. Adjacent
blood vessels further increase the difficulty of this task. The
currently used 2D dynamic programming method works
reasonably well for the inner wall segmentation but is
unsuitable for the segmentation of the outer airway wall. By
optimizing the inner and outer wall surfaces and consider-
ing the geometric constraints, our new optimal coupled-
7.2 Execution Times surfaces segmentation approach produces good segmenta-
The average execution times of our simultaneous k-surface tion results for both airway wall surfaces in a robust
(k ¼ 2; 3) detection algorithm are shown in Table 1 for the manner (Figs. 11 and 12).
implementation using the Boykov-Kolmogorov maximum Compared to the manual tracings in 39 randomly
flow algorithm on a “forward-star” represented graph. selected slices, the automated 3D approach yielded signed
Comparisons of different implementations of the pro- border positioning errors of 0:01
0:15 mm and 0:01
posed algorithm for the single, double, and triple surfaces 0:17 mm for the inner and outer wall surfaces, respectively.
detection cases are shown in Fig. 9. The Boykov-Kolmol- The corresponding unsigned errors were 0:10
0:11 mm
gorov (BK) implementation typically performed better and 0:12
0:12 mm, respectively. The maximum unsigned
when the cost function is smooth and k (the number of border positioning errors for the inner and outer wall
surfaces to be segmented) is small, as were the cases in surfaces were 0:37
0:18 mm and 0:41
0:20 mm, respec-
Figs. 9a and 9b. However, cost functions that exhibit many tively. Linear regression analysis revealed close correlation
local minima and a larger k tend to push the algorithms to between the observer-defined and computer-detected air-
the worst case performance bound. When this happens, the way wall thicknesses (r2 ¼ 0:978) in the 39 slices. The
push-relabel (PR) implementation tends to be more effi- regression equation closely approximates the line of
cient. By exploiting the regularity in the graph structure, the identity with neither the slope nor the intercept being
“implicit-arc” (IA) graph representation was shown to significantly different from one and zero (p > 0:3).
improve both the speed and memory efficiency (50 percent The 3D method-generated inner airway wall surfaces
less compared to “forward star”) of the algorithm. exhibited higher overall accuracy in comparison with the
2D slice-by-slice method. The major and minor diameters
7.3 Accuracy Assessment in CT-Imaged Physical yielded by the 2D and 3D approaches in all 317 airway
Phantoms segments correlated closely (r2major ¼ 0:994, r2minor ¼ 0:998).
Signed percent errors of the computer segmented and However, only the mean minor diameters were statistically
measured diameters are presented in Fig. 10. When com- equivalent between the two methods (p ¼ NS). The major
pared with segmentation errors achieved by the traditional diameters obtained from the 2D approach were signifi-
2D slice-by-slice approach, the new 3D coupled-surfaces cantly larger than those from the 3D approach (p ¼ 0:003).
method was statistically significantly more accurate Segmenting and measuring the inner and outer wall
(p 0:001), although the differences were small: The signed surfaces in one complete airway tree (about 30 airway
errors of the measured diameters of the 2D and 3D approaches segments) took approximately 6 minutes excluding the time
were 0:02
0:11 mm and 0:01
0:10 mm, respectively. used for presegmentation and skeletonization. About
The corresponding unsigned errors were 0:09
0:07 mm and 50 percent of the running time was spent on the measure-
0:08
0:07 mm, respectively. ment stage.
Fig. 9. Execution time comparisons. (a) Single-surface detection case. (b) Execution times in smooth and noisy cost functions for triple-surface
detection (3S = smooth, 3N = noisy).
LI ET AL.: OPTIMAL SURFACE SEGMENTATION IN VOLUMETRIC IMAGES—A GRAPH-THEORETIC APPROACH 129
Fig. 10. Signed percent errors of inner and outer-diameter measurements in the CT-imaged phantom. Mean errors
standard deviations shown as
a function of phantom tube diameter.
Fig. 11. Comparison of observer-defined and computer-segmented inner and outer airway wall borders. (a) Expert-traced borders. (b) Three-
dimensional surface obtained using our double-surface segmentation method.
Fig. 12. Segmentation of inner airway wall surface in five consecutive slices of an airway segment, shown with 3D surface rendering of the obtained
surface. (a) Slice-by-slice dynamic programming segmentation—notice the “leaking” in one of the slices using the 2D method. (b) Optimal coupled-
surface segmentation approach using the 3D method.
130 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 28, NO. 1, JANUARY 2006
Fig. 13. Three-dimensional segmentation of the diaphragm surface from volumetric in vivo CT image. (a) Coronal and sagittal views of the original
image with the manually traced independent standard segmentation. (b) Three-dimensional surface segmentation approach.
Fig. 14. Segmentation of MR arterial walls and plaque. (a) Two original MR slices. (b) Manually identified lumen, IEL, and EEL borders.
(c) Segmentation obtained using slice-by-slice dynamic programming. On average, 2.4 interaction points per image slice were needed.
(d) Four borders identified by the reported fully automated optimal four-surface segmentation method.
Fig. 15. IVUS image segmentation result. (a) Polar and longitude cross-sections of the original image. (b) Segmentation result by optimal coupled-
surface segmentation. (c) Surface reconstruction.
into the surface detection process is considered. Second, the manipulating the cost values of the graph nodes corre-
steps necessary allowing interactive guidance of the search sponding to the landmark voxels, i.e., by assigning them an
process in difficult images are described. Finally, the extremely low cost such that the feasible surface including
proposed method is compared to the existing segmentation these voxels will have the minimum cost globally. How-
methods. ever, a more reliable and efficient way to incorporate
landmarks would be to change the graph construction itself.
8.1 Variable Geometric Constraints It is clear that the ðx; yÞ-column with a landmark node will
In the previous sections, homogeneous smoothness and contain only a single node corresponding to the landmark
separation constraints whose values remain constant along voxel. Due to the introduction of landmarks, some nodes in
each axial direction were considered. In fact, the proposed the original graph that do not satisfy the geometric
algorithm allows incorporation of variable geometric con- relationship with the landmarks can be automatically
straints. The constraints can be specified adaptively based pruned, further reducing the graph complexity. The
on the local image context. For example, the surface placement of landmarks is especially useful in human-
smoothness and surface-to-surface interrelations may vary computer interactive segmentation.
at different locations. Variable geometric constraints can be 8.3 Relations to the Open-Pit Mining Problem
incorporated into the algorithm by rearrangements of the The single surface segmentation algorithm is closely related
graph edges. As a result, the graph arcs may no longer be to the open-pit mining problem [69], [70], [71], which seeks
parallel as the case in the constant-constraint case. to excavate the earth surface to extract ore (e.g., gold)
Nevertheless, there are practical obstacles that need to be contained in the earth. Each block of earth is associated with
overcome for this approach to become useful. For the a net profit, which is the value of the ore it contains minus
variable-constraint setup, a key problem remains challen- the excavation cost of the block. A key objective in open-pit
ging: How to automatically adjust the constraints as mining is to excavate earth blocks until a pit surface is
needed. We expect that employing on-the-fly machine reached such that the total net profit is maximized. Due to
learning techniques will help solving this issue. the use of the large mining machines, the pit surface is
expected to be “smooth.”
8.2 Surface Guidance and Interactive Segmentation
In practice, prior knowledge about the shape and/or 8.4 Comparison to Other Segmentation Methods
position of the desired optimal surfaces may be available. In this paper, performance of our new method was directly
Such knowledge can be incorporated into the algorithm by compared with that of the dynamic programming in several
placing “landmark” points in the image, which are the medical image segmentation tasks. This is because of their
voxels that the detected surfaces must pass through. There common graph-based formulation and similar domain of
are essentially two ways to achieve this. One is by application. In recent years, many novel and powerful
132 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 28, NO. 1, JANUARY 2006
[18] Y. Xu and E.C. Uberbacher, “2-D Image Segmentation Using [41] Y. Boykov, O. Veksler, and R. Zabih, “Fast Approximate Energy
Minimum Spanning Trees,” Image and Vision Computing, vol. 15, Minimization via Graph Cuts,” IEEE Trans. Pattern Analysis and
no. 1, pp. 47-57, 1997. Machine Intelligence, vol. 23, no. 11, pp. 1222-1239, Nov. 2001.
[19] A. Jagannathan and E. Miller, “A Graph-Theoretic Approach to [42] V. Kolmogorov and R. Zabih, “What Energy Functions Can Be
Multiscale Texture Segmentation,” Proc. Int’l Conf. Image Proces- Minimized via Graph Cuts?” IEEE Trans. Pattern Analysis and
sing, vol. 2, pp. 777-780, Sept. 2002. Machine Intelligence, vol. 26, no. 2, pp. 147-159, Feb. 2004.
[20] P.F. Felzenszwalb and D.P. Huttenlocher, “Efficient Graph-Based [43] S. Geman and D. Geman, “Stochastic Relaxation, Gibbs Distribu-
Image Segmentation,” Int’l J. Computer Vision, vol. 59, pp. 167-181, tions, and the Bayesian Restoration of Images,” IEEE Trans. Pattern
Sept. 2004. Analysis and Machine Intelligence, vol. 6, pp. 721-741, Nov. 1984.
[21] J.K. Udupa and S. Samarasekera, “Fuzzy Connectedness and [44] L.R.J. Ford and D.R. Fulkerson, “Maximal Flow through a
Object Definition: Theory, Algorithms, and Applications in Image Network,” Canadian J. Math., vol. 8, pp. 399-404, 1956.
Segmentation,” Graphics Models and Image Processing, vol. 58, [45] A.V. Goldberg and R.E. Tarjan, “A New Approach to the
pp. 246-261, May 1996. Maximum-Flow Problem,” J. ACM, vol. 35, pp. 921-940, 1988.
[22] P.K. Saha and J.K. Udupa, “Relative Fuzzy Connectedness among [46] A.V. Goldberg and S. Rao, “Beyond the Flow Decomposition
Multiple Objects: Theory, Algorithms, and Applications in Image Barrier,” J. ACM, vol. 45, pp. 783-797, 1998.
Segmentation,” Computer Vision and Image Understanding, vol. 82, [47] Y. Boykov and V. Kolmogorov, “Computing Geodesics and
pp. 42-56, 2000. Minimal Surfaces Via Graph Cuts,” Proc. Int’l Conf. Computer
[23] L.G. Nyul, A.X. Falcão, and J.K. Udupa, “Fuzzy-Connected Vision (ICCV), pp. 26-33, Oct. 2003.
3D Image Segmentation at Interactive Speeds,” Graphical Models, [48] X. Zeng, L. Staib, R. Schultz, and J. Duncan, “Segmentation and
vol. 64, pp. 259-281, Sept. 2002. Measurement of the Cortex from 3-D MR Images Using Coupled-
[24] A.X. Falcão, J. Stolfi, and R. de Alencar Lotufo, “The Image Surfaces Propagation,” IEEE Trans. Medical Imaging, vol. 18,
Foresting Transform: Theory, Algorithms, and Applications,” pp. 927-937, Oct. 1999.
IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 26, [49] D. MacDonald, N. Kabani, D. Avis, and A. Evans, “Automated 3-D
no. 1, pp. 19-29, Jan. 2004. Extraction of Inner and Outer Surfaces of Cerebral Cortex from
[25] J.H.C. Reiber, P.M.J. van der Zwet, C.D. von Land, G. Loois, I. MRI,” NeuroImage, vol. 12, no. 3, pp. 340-356, Sept. 2000.
Zorn, M. van den Brand, and J.J. Gerbrands, “On-Line Quantifica- [50] R. Goldenberg, R. Kimmel, E. Rivlin, and M. Rudzsky, “Cortex
tion of Coronary Angiograms with the DCI System,” Medicamundi, Segmentation: A Fast Variational Geometric Approach,” IEEE
vol. 34, pp. 89-98, 1989. Trans. Medical Imaging, vol. 21, pp. 1544-1551, Dec. 2002.
[26] M. Sonka, M.D. Winniford, and S.M. Collins, “Robust Simulta- [51] L. Spreeuwers and M. Breeuwer, “Detection of Left Ventricular
neous Detection of Coronary Borders in Complex Images,” IEEE Epi- and Endocardial Borders Using Coupled Active Contours,”
Trans. Medical Imaging, vol. 14, pp. 151-161, Mar. 1995. Computer Assisted Radiology and Surgery, pp. 1147-1152, June 2003.
[27] M. Sonka, V. Hlavac, and R. Boyle, Image Processing, Analysis, and [52] R. Goldenberg, R. Kimmel, E. Rivlin, and M. Rudzsky, “Fast
Machine Vision, second ed. Brooks/Cole Publishing Company, Geodesic Active Contours,” IEEE Trans. Image Processing, vol. 10,
1999. pp. 1467-1475, Oct. 2001.
[28] Z. Wu and R. Leahy, “An Optimal Graph Theoretic Approach to [53] X. Han, C. Xu, and J.L. Prince, “A Topology Preserving Level Set
Data Clustering: Theory and Its Application to Image Segmenta- Method for Geometric Deformable Models,” IEEE Trans. Pattern
tion,” IEEE Trans. Pattern Analysis Machine Intelligence, vol. 15, Analysis and Machine Intelligence, vol. 25, no. 6, pp. 755-768, June
no. 11, pp. 1101-1113, Nov. 1993. 2003.
[29] I. Cox, S. Rao, and Y. Zhong, “Ratio Regions: A Technique for [54] T.F. Cootes, A. Hill, C.J. Taylor, and J. Haslam, “The Use of Active
Image Segmentation,” Proc. Int’l Conf. Pattern Recognition, vol. 2, Shape Models for Locating Structures in Medical Images,” Proc.
pp. 557-564, Aug. 1996 13th Int’l Conf. Information Processing in Medical Imaging (IPMI),
[30] H. Ishikawa and D. Geiger, “Segmentation by Grouping Junc- pp. 33-47, 1993.
tions,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, [55] T. Cootes, G. Edwards, and C. Taylor, “Active Appearance
pp. 125-131, June 1998. Models,” IEEE Trans. Pattern Analysis and Machine Intelligence,
[31] I. Jermyn and H. Ishikawa, “Globally Optimal Regions and vol. 23, no. 6, pp. 681-685, June 2001.
Boundaries as Minimum Ratio Cycles,” IEEE Trans. Pattern [56] S. Mitchell, B. Lelieveldt, R. van der Geest, H. Bosch, J. Reiber, and
Analysis and Machine Intelligence, vol. 23, no. 10, pp. 1075-1088, M. Sonka, “Multistage Hybrid Active Appearance Model Match-
Oct. 2001. ing: Segmentation of Left and Right Ventricles in Cardiac MR
[32] J. Shi and J. Malik, “Normalized Cuts and Image Segmentation,” Images,” IEEE Trans. Medical Imaging, vol. 20, pp. 415-423, May
IEEE Trans. Pattern Analysis Machine Intelligence, vol. 22, no. 8, 2001.
pp. 888-905, Aug. 2000. [57] D. Hochbaum, “A New-Old Algorithm for Minimum-Cut and
[33] Y. Boykov and M.-P. Jolly, “Interactive Organ Segmentation Using Maximum-Flow in Closure Graphs,” Networks, vol. 37, pp. 171-
Graph Cuts,” Proc. Medical Image Computing and Computer-Assisted 193, 2001.
Intervention (MICCAI), pp. 276-286, 2000. [58] J. Picard, “Maximal Closure of a Graph and Applications to
[34] Y.Y. Boykov and M.-P. Jolly, “Interactive Graph Cuts for Optimal Combinatorial Problems,” Management Science, vol. 22, pp. 1268-
Boundary and Region Segmentation of Objects in N-D Images,” 1272, 1976.
Proc. Int’l Conf. Computer Vision (ICCV), vol. 1, pp. 105-112, July [59] M. Sonka, G.K. Reddy, M.D. Winniford, and S.M. Collins,
2001. “Adaptive Approach to Accurate Analysis of Small-Diameter
[35] S. Wang and J. Siskind, “Image Segmentation with Ratio Cut,” Vessels in Cineangiograms,” IEEE Trans. Medical Imaging, vol. 16,
IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, no. 6, pp. 87-95, Feb. 1997.
pp. 675-690, June 2003. [60] D. Mumford and J. Shah, “Optimal Approximation by Piecewise
[36] Y. Boykov and V. Kolmogorov, “An Experimental Comparison of Smooth Functions and Associated Variational Problems,” Comm.
Min-Cut/Max-Flow Algorithms for Energy Minimization in Pure and Applied Math., vol. 42, pp. 577-685, 1989.
Vision,” IEEE Trans. Pattern Analysis and Machine Intelligence, [61] T.F. Chan and L.A. Vese, “Active Contour without Edges,” IEEE
vol. 26, no. 9, pp. 1124-1137, Sept. 2004. Trans. Image Processing, vol. 10, no. 2, pp. 266-277, Feb. 2001.
[37] Y. Li, J. Sun, C.-K. Tang, and H.-Y. Shum, “Lazy Snapping,” ACM [62] J. Tschirren, “Segmentation, Anatomical Labeling, Branch-Point
Trans. Graphics (TOG), vol. 23, pp. 303-308, Aug. 2004. Matching, and Quantitative Analysis of Human Airway in
[38] J. Kim and R. Zabih, “A Segmentation Algorithm for Contrast- Volumetric CT Images,” PhD dissertation, Univ. of Iowa, Aug.
Enhanced Images,” Proc. IEEE Int’l Conf. Computer Vision, pp. 502- 2003.
509, Oct. 2003. [63] K. Palágyi, E. Sorantin, E. Balogh, A. Kuba, C. Halmai, B.
[39] J. Sethian, Level Set Methods and Fast Marching Methods Evolving Erdohelyi, and K. Hausegger, “A Sequential 3D Thinning
Interfaces in Computational Geometry, Fluid Mechanics, Computer Algorithm and Its Medical Applications,” Proc. 17th Int’l Conf.
Vision, and Materials Science, second ed. Cambridge, U.K.: Cam- Information Processing in Medical Imaging (IPMI), pp. 409-415, 2001.
bridge Univ. Press, 1999. [64] M. Unser, A. Aldroudi, and M. Eden, “B-Spline Signal Processing:
[40] Geometric Level Set Methods in Imaging, Vision and Graphics, S. Osher Part I—Theory,” IEEE Trans. Signal Processing, vol. 41, pp. 821-833,
and N. Paragios, eds. Springer, 2003. Feb. 1993.
134 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 28, NO. 1, JANUARY 2006
[65] M. Unser, A. Aldroubi, and M. Eden, “B-Spline Signal Processing: Xiaodong Wu received the BS and MS degrees
Part II—Efficient Design and Applications,” IEEE Trans. Signal both in computer science from Peking University,
Processing, vol. 41, no. 2, pp. 834-848, Feb. 1993. China, in 1992 and 1995, respectively, and the
[66] B. Cherkassky and A. Goldberg, “On Implementing Push-Relabel PhD degree in computer science and engineer-
Method for the Maximum Flow Problem,” Algorithmica, vol. 19, ing from the University of Notre Dame in August
pp. 390-410, 1997. 2002. He is currently an assistant professor in
[67] X. Huang, D. Metaxas, and T. Chen, “Metamorphs: Deformable the Departments of Electrical and Computer
Shape and Texture Models,” Proc. IEEE CS Conf. Computer Vision Engineering and Radiation Oncology at the
and Pattern Recognition (CVPR), vol. 1, pp. 496-503, June 2004. University of Iowa. From 2002 to 2004, he was
[68] F. Yang, G. Holzapfel, C. Schulze-Bauer, R. Stollberger, D. an assistant professor of computer science at
Thedens, L. Bolinger, A. Stolpen, and M. Sonka, “Segmentation the University of Texas—Pan American. His research interests are
of Wall and Plaque in in Vitro Vascular MR Images,” The Int’l J. primarily in biomedical computing and computer algorithms with a
Cardiac Imaging, vol. 19, pp. 419-428, Oct. 2003. particular emphasis on the development and implementation of efficient
[69] I.G.H. Lerchs, “Optimum Design of Open-Pit Mines,” Trans. CIM, algorithms for solving challenging problems arising in computer-assisted
vol. 58, pp. 17-24, 1965. medical diagnosis and treatment, biomedical image analysis, and
[70] B. Denby and D. Schofield, “Genetic Algorithms: A New bioinformatics. He is a senior member of the IEEE.
Approach to Pit Optimization,” Proc. 24th Int’l APCOM Symp.,
pp. 126-133, 1993. Danny Z. Chen received the BS degree in
[71] D. Hochbaum and A. Chen, “Improved Planning for the computer science and mathematics from the
Open—Pit Mining Problem,” Operations Research, vol. 48, pp. 894- University of San Francisco, San Francisco,
914, 2000. California, in 1985 and the MS and PhD degrees
[72] X. Zhang, C. McKay, and M. Sonka, “Tissue Characterization in in computer science from Purdue University,
Intravascular Ultrasound Images,” IEEE Trans. Medical Imaging, West Lafayette, Indiana, in 1988 and 1992,
vol. 17, pp. 889-899, Dec. 1998. respectively. He is a professor in the Department
[73] A.Ü. Coşkun, Y. Yeghiazarians, S. Kinlay, M.E. Clark, O.J. of Computer Science and Engineering at the
Ilegbusi, A. Wahle, M. Sonka, J.J. Popma, R.E. Kuntz, C.L. University of Notre Dame. In 1996, he received
Feldman, and P.H. Stone, “Reproducibility of Coronary Lumen, the Faculty Early Career Development (CA-
Plaque, and Vessel Wall Reconstruction and of Endothelial Shear REER) Award from the US National Science Foundation. His research
Stress Measurements In-Vivo in Humans,” Catheterization and interests include parallel computing, computational geometry, algo-
Cardiovascular Interventions, vol. 60, no. 1, pp. 63-78, Sept. 2003. rithms, and data structures. He is a senior member of the IEEE.
[74] S. Mitchell, J. Bosch, B. Lelieveldt, R. van der Geest, J. Reiber, and
M. Sonka, “3-D Active Appearance Models: Segmentation of Milan Sonka received the PhD degree in 1983
Cardiac MR and Ultrasound Images,” IEEE Trans. Medical from the Czech Technical University in Prague.
Imaging, vol. 21, pp. 1167-1178, Sept. 2002. He is a professor of electrical and computer
[75] R. Beichel, G. Gotschuli, E. Sorantin, F.W. Leberl, and M. Sonka, engineering at the University of Iowa and a
“Diaphragm Dome Surface Segmentation in CT Data Sets: A fellow of the IEEE. His research interests include
3D Active Appearance Model Approach,” Proc. SPIE Int’l Symp. medical imaging and knowledge-based image
Medical Imaging: Image Processing, vol. 4,684, pp. 475-484, May analysis. A major focus of his research in the last
2002. several years has been on the development of
clinically applicable automated techniques for
Kang Li received the BS degree from Nanjing cardiovascular analysis, pulmonary CT image
University, China, in 2002 and the MS degree analysis, cell tracking and cellular shape analysis, and augmented
from the University of Iowa in 2003, both in reality image-based surgical planning. He is the first author of the book
electrical engineering. He is currently working Image Processing, Analysis and Machine Vision published in 1993 by
toward the PhD degree in the Department of Chapman and Hall in London, second edition 1998 by PWS, Pacific
Electrical and Computer Engineering at the Grove, California. He has coauthored or coedited 10 other books
Carnegie Mellon University. His research inter- including the Handbook of Medical Imaging, Volume II—Medical Image
ests are in biomedical image processing, com- Processing and Analysis published in 2000. He has authored seven book
puter vision, graph algorithms, and machine chapters, more than 60 journal papers, 160 conference papers, and
learning. He is a student member of the IEEE 60 abstracts. He is associate editor of the IEEE Transactions on Medical
and the IEEE Computer Society. Imaging and member of the editorial board of the International Journal of
Cardiovascular Imaging.