2574 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 22, NO.

7, JULY 2013
A Generalized Random Walk With Restart and
Its Application in Depth Up-Sampling and
Interactive Segmentation
Bumsub Ham, Student Member, IEEE, Dongbo Min, Member, IEEE,
and Kwanghoon Sohn, Senior Member, IEEE
AbstractIn this paper, the origin of random walk with
restart (RWR) and its generalization are described. It is well
known that the random walk (RW) and the anisotropic diffusion
models share the same energy functional, i.e., the former provides
a steady-state solution and the latter gives a ow solution.
In contrast, the theoretical background of the RWR scheme is
different from that of the diffusion-reaction equation, although
the restarting term of the RWR plays a role similar to the
reaction term of the diffusion-reaction equation. The behaviors
of the two approaches with respect to outliers reveal that they
possess different attributes in terms of data propagation. This
observation leads to the derivation of a new energy functional,
where both volumetric heat capacity and thermal conductivity
are considered together, and provides a common framework that
unies both the RW and the RWR approaches, in addition to
other regularization methods. The proposed framework allows
the RWR to be generalized (GRWR) in semilocal and nonlocal
forms. The experimental results demonstrate the superiority of
GRWR over existing regularization approaches in terms of depth
map up-sampling and interactive image segmentation.
Index TermsAnisotropic diffusion, depth up-sampling,
diffusion-reaction equation, interactive segmentation, random
walk with restart (RWR), thermal diffusivity.
I. INTRODUCTION
M
ANY researchers have attempted to answer the
following questions: How can a computer extract
useful information from digital photographs or videos, as a
human being does?", or What is the optimal way of com-
pleting this process?" Some physiological observations have
been translated into mathematical models and subsequently
implemented with an approximation for simplicity. This allows
Manuscript received May 18, 2012; revised November 3, 2012; accepted
February 25, 2013. Date of publication March 20, 2013; date of current
version May 13, 2013. This work was supported by the MKE (The Ministry
of Knowledge Economy), Korea, under the Information Technology Research
Center support program supervised by the National IT Industry Promotion
Agency (NIPA) (NIPA-2012-H0301-12-1008). The work of D. Min was
supported by the research grant for the Human Sixth Sense Programme at
the Advanced Digital Sciences Center from Singapores Agency for Science,
Technology, and Research. The associate editor coordinating the review of
this manuscript and approving it for publication was Prof. Xilin Chen.
B. Ham and K. Sohn are with the School of Electrical and Elec-
tronic Engineering, Yonsei University, Seoul 120-749, South Korea (e-mail:
mimo@yonsei.ac.kr; khsohn@yonsei.ac.kr).
D. Min is with the Advanced Digital Sciences Center, 138632, Singapore
(e-mail: dongbo@adsc.com.sg).
Color versions of one or more of the gures in this paper are available
online at http://ieeexplore.ieee.org.
Digital Object Identier 10.1109/TIP.2013.2253479
machinery to (semi-) automatically handle complicated prob-
lems such as object recognition, surveillance, object tracking,
and 3D reconstruction. One of the most important technolo-
gies employed to address the aforementioned questions is to
regularize an image while preserving universal features. Image
regularization can be classied into two categories, local and
nonlocal approaches, according to how the neighborhood used
in the regularization process is dened.
A number of approaches to using local regularization have
been proposed, including the anisotropic diffusion [1], [2],
the total variation [3], [4], the Mumford-Shah regularization
[5], [6], the bilateral lter [7], [8], the random walk (RW)
[9], [10], and the random walk with restart (RWR) [11] (see
also [12]). Perona and Malik proposed an anisotropic diffusion
model, the thermal diffusivity of which changes from a con-
stant to a space-variant function called the edge-stopping"
function [1]. You et al. addressed the anisotropic diffusion
in an optimization problem and proposed its energy func-
tional [13]. The work was extended within a robust statistics
framework, resulting in the robust anisotropic diffusion [14].
The bilateral lter, rst proposed by Tomasi and Manduchi,
is a nonlinear lter that combines tonal and spatial kernels. It
regularizes homogeneous regions while preserving important
features [7]. However, the bilateral lter is an intuitive method
with no theoretical links to other existing methods [15], [16].
Elad proposed an energy functional corresponding to the
bilateral lter and showed how the bilateral lter can be
improved and expanded in order to handle more compli-
cated reconstruction problems [16]. The RW approach is a
classical method that estimates the probability of a random
walker on a graph [9]. Theoretically, the method shares the
same energy functional with the anisotropic diffusion [10].
Specically, the RW model provides a steady-state solution,
while the anisotropic diffusion gives a ow solution. Shen
et al. [17] generalized the RW model by introducing an
augmented node similar to graph cuts [18], which can be
thought of as imposing a prior knowledge on the RW [19].
Recently, the RWR method has become increasingly popular,
as its restarting term gives meaningful information in the
steady-state, allowing the global relation to be considered
at all scales (or at all times). Thus, the RWR approach
is more suitable to some applications, such as interactive
segmentation [20], cost aggregation [21], and information
retrieval [22].
1057-7149/$31.00 2013 IEEE
HAM et al.: GENERALIZED RWR 2575
Nonlocal regularization has been intensively studied such
that (semi-) local methods have been extended to the
corresponding nonlocal form, thus allowing texture informa-
tion to be successfully leveraged without altering a signal to
be preserved [23]. Gilboa and Osher proposed the nonlocal
diffusion [24], a nonlocal counterpart of the anisotropic dif-
fusion, to better capture texture information through nonlocal
processing. The symmetric energy ow preserves the overall
energy, as in the anisotropic diffusion scheme, and prevents
singular regions from being blurred [24], [25]. Buades et al.
proposed the nonlocal mean lter [26] as a nonlocal extension
of the bilateral lter by utilizing a patch-wise afnity function.
Protter et al. presented an energy functional of the nonlocal
mean lter and further generalized it within a weighted least
square framework [27]. Similarly, Pizarro et al. generalized
the nonlocal mean lter by utilizing a patch similarity for
both the delity and smoothness terms [28]. Recently, the total
variation [3], [4] and the MumfordShah regularization [5], [6]
were also extended to the corresponding nonlocal formulation
for better handling of ne structures and textures [29], [30].
To the best of our knowledge, there have been no studies
on the origin of the RWR model. In general, the RWR can be
described as an ad-hoc method of the RW approach in that a
restarting term is simply added to obtain a non-trivial steady-
state solution. To explore the origin of the RWR, we rst
investigate the relationship between the RW and the diffusion
models, as the two approaches have been known to share the
same energy functional [10]. Namely, the RW model provides
a steady-state solution, while the diffusion approach yields a
ow solution. On the other hand, the behavior of the RWR is
different from that of the diffusion-reaction equation, although
the restarting term of the RWR constrains the steady-state
solution to an initial condition in the same manner as the
reaction term in the diffusion-reaction equation.
In this paper, we show that the RWR and the diffusion-
reaction equation have different theoretical backgrounds. The
behavior of the two models with respect to outliers reveals
that each approach possesses different attributes in terms of
data propagation in the presence of the outliers. Specically,
it is shown that the RWR is more robust against outliers than
the diffusion-reaction equation. Based on this observation, we
propose a new energy functional, where both volumetric heat
capacity and thermal conductivity are considered together, and
provide a common framework that unies both the RW and
the RWR approaches, as well as other regularization methods.
The proposed energy functional allows us to generalize the
RWR in semi-local and nonlocal frameworks. To verify its
performance, the generalized RWR (GRWR) scheme is applied
to depth map up-sampling and interactive image segmentation.
The experimental results show that: 1) the GRWR approach is
more robust to outliers; 2) the GRWR can aggregate texture
information better; and 3) a global relation can be captured
by the GRWR model with no stopping criterion, which is not
feasible in classical diffusion.
The remainder of this paper is organized as follows. The
anisotropic diffusion and the RW model are briey reviewed
in Section II. A common energy functional that unies the RW
and the RWR schemes and generalizes them in semi-local and
nonlocal frameworks is described in Section III. An extensive
analysis of experimental results is presented in Section IV.
Finally, conclusions and suggestions for future work are given
in Section V.
II. DIFFUSION AND RANDOM WALK
Let u(x) : R
+
be a function with a continuous image
domain where R
2
is an open and bounded space with
x and y being 2D vectors, which represent spatial
coordinates.
A. Anisotropic Diffusion
The heat equation, also known as the diffusion equation, is a
fundamental partial differential equation that models the distri-
bution of heat or temperature over a given domain with respect
to time. Perona and Malik proposed the anisotropic diffusion
model, and applied this physics model to image processing,
particularly for an edge preserving regularization [1]. With an
initial condition f (x), the anisotropic diffusion is dened as
follows [24]:

t
u(x) =
_

(u(y) u(x)) w(x, y)dy (1)

u
t =0
(x) = f (x) (2)
where
t
denotes a partial derivative with respect to time t. The
afnity function w(x, y) is positive w(x, y) > 0 and
symmetric w(x, y) = w(y, x), playing a role as a discontinuity
marker that stops diffusion across different features.
w(x, y) =
_
exp
_
| f (x) f (y)|
2
/h
2
_
, y
L
(x)
0, y /
L
(x)
(3)
with
L
(x) = {y \x : |y x| 1}. The range bandwidth
is represented as h. Note that w(x, y) corresponds to the
thermal diffusivity in physics [24].
B. Random Walk
The RW is a classical method in the eld of random
processes and is closely related to circuit theory [9]. It formu-
lates the trajectory of a random walker that takes successive
random steps. The RW model is usually dened on a discrete
graph, but without a loss of generality, its continuous formal-
ization can be represented, for a starting position f (x), as
u
t +1
(x) =
_

u
t
(y)w(x, y)dy
_

w(x, y)dy
(4)
u
0
(x) = f (x) (5)
where u
t
(x) represents the trajectory or the position of a
random walker at time t . Similar to the anisotropic diffusion
model in (1), any monotonically decreasing function can be
used as the afnity function w(x, y).
2576 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 22, NO. 7, JULY 2013
C. Relationship Between Diffusion and Random Walk
Theoretically, the RW and the anisotropic diffusion schemes
share the same origin and thus, they have the same energy
functional [10]. Let us consider the following energy func-
tional:
E(u) =
1
4
_

(u(x) u(y))
2
w(x, y)dydx (6)
where
w(x, y) =
_
exp
_
| f (x) f (y)|
2
/h
2
_
, y
L
(x)
0, y /
L
(x)
(7)
with
L
(x) = {y \x : |y x| 1}.
Since this energy functional is linear and strictly convex,
a global minimum is guaranteed. This minimum can be
calculated via the steepest descent method or the GaussJacobi
iteration as follows:
1) Flow Solution Via Steepest Descent Method: Since the
derivative of the energy functional is
E

(u) =
_

(u(x) u(y)) w(x, y)dy (8)

the ow solution with an initial condition f (x) is obtained as

t
u(x) = E

(u) =
_

(u(y) u(x)) w(x, y)dy (9)

which is identical to the anisotropic diffusion model in (1).
2) Steady-State Solution Via GaussJacobi Iteration: When
a solution reaches the steady-state, an energy transition with
respect to time approaches 0, i.e.,
t
u(x) = 0. Thus, the
steady-state solution is given by
0 =
_

(u(x) u(y)) w(x, y)dy. (10)

This can be solved by GaussJacobi iteration as follows:
u
t +1
(x) =
_

u
t
(y)w(x, y)dy
_

w(x, y)dy
(11)
which is equivalent to the RW model in (4).
Accordingly, it can be seen that the RW and the anisotropic
diffusion models seek the same global minimum on a given
energy functional: the former provides the steady-state solu-
tion, while the latter gives the ow solution.
III. GENERALIZED RANDOM WALK WITH RESTART
In this section, we rst observe the relationship between the
RWR and the diffusion-reaction equation, and then describe a
unied energy functional for the RW and the RWR models.
The RWR approach is ultimately generalized in both semi-
local and nonlocal forms.
A. Problem Statement
The steady-state solutions of the anisotropic diffusion and
the RW models give no meaningful information, i.e., a constant
image. To avoid this problem, the diffusion-reaction equation
constrains the steady-state solution to an initial condition [31]
as follows:

t
u(x) =
_

(u(y) u(x)) w(x, y)dy +( f (x)u(x)) (12)

where (> 0) represents the regularization parameter that con-
trols the leverage between a delity term and a regularization
term. Similarly, the RWR model is dened as
u
t +1
(x) = (1 c)
_

u
t
(y)w(x, y)dy
_

w(x, y)dy
+cf (x) (13)
where the restarting probability means that a random walker
goes back to the starting position f (x) with the probability c.
As described in Section II-C, the anisotropic diffusion and
the RW model are based on the same energy functional.
The following questions should then be considered: What
is the energy functional of the RWR model?" and Is the
energy functional of the RWR model equivalent to that of
the diffusion-reaction equation?" Although the RWR approach
has been successfully applied to many applications (e.g., seg-
mentation [20], image matting [32], information retrieval [22],
annotation renement [33], graph matching [34]), the origin of
the RWR has not yet been extensively investigated. Knowledge
of the origin of the RWR model will allow us to better
understand its behavior from the perspective of energy ow
and further generalize its energy functional.
Proposition 1: The energy functional of the diffusion-
reaction equation is different from that of the RWR model.
Proof: Let us consider the following energy functional that
is similar to (6), with the exception of an additional delity
term:
E
DR
(u) =
1
4
_

(u(x) u(y))
2
w(x, y)dydx
+

2
_

(u(x) f (x))
2
dx. (14)
The solution can be computed in two ways:
1) Flow Solution Via Steepest Descent Method:

t
u(x) =
_

(u(y) u(x)) w(x, y)dy+( f (x)u(x)). (15)

2) Steady-State Solution Via GaussJacobi Iteration:
u
t +1
(x) =
_

u
t
(y)w(x, y)dy +f (x)
_

w(x, y)dy +
. (16)
The ow solution is the same as that for the diffusion-
reaction equation in (12). However, the steady-state solution
does not correspond to that of the RWR model in (13), leading
to the conclusion that the RWR and the diffusion-reaction
equation have a different energy functional.
Proposition 1 also means that the energy functional in (14)
unies the anisotropic diffusion ( = 0) and the diffusion-
reaction equation ( = 0). It, however, does not unify both
the RW and the RWR models although it becomes the energy
functional of the RW when = 0. Note that the nonlocal
extension of (16) is similar to the NDS model proposed
in [28].
B. Derivation of Random Walk With Restart
1) Behavioral Analysis Against Outliers: Before deriving
the energy functional of the RWR model, we rst compare
the behaviors of the diffusion-reaction equation and the RWR.
Shown in Fig. 1(a) are input images that, from left to right,
HAM et al.: GENERALIZED RWR 2577
(a)
(b)
(c)
Fig. 1. Different behaviors of the diffusion-reaction equation and the RWR
model. (a) Initial image (from left to right) that is noise-free, corrupted by
the Gaussian noise with a standard deviation of 20, and corrupted by the
impulsive noise with a density of 0.05. (b) Results obtained with the diffusion-
reaction equation. (c) Results obtained with the RWR model. The number of
iterations is set to 50 in the diffusion-reaction equation and 20 in the RWR
model, respectively, for making the ltering results with similar extent of
blurring. The regularization parameter in (12) and the restarting probability
c in (13) are set to 1/10 and 1/11, respectively. Both methods show similar
ltering behaviors for both the original image and the Gaussian noise image.
However, we found that the RWR model is more robust against impulsive
noise than the diffusion-reaction equation. (a) Initial image. (b) Diffusion-
reaction equation. (c) RWR.
are noise-free, corrupted by the Gaussian noise with a standard
deviation of 20, and corrupted by the impulsive noise with
a density of 0.05, respectively. These images are ltered by
the diffusion-reaction equation and the RWR model. Both
methods show similar ltering behaviors for the noise-free
and the Gaussian noisy images. However, against impulsive
noise, the RWR method shows more robust ltering results
than the diffusion-reaction equation, i.e., the most impulsive
noises, which are rarely handled by the diffusion-reaction
equation, are effectively eliminated by the RWR within a
small number of iterations. The anisotropic diffusion and
the RW models also showed similar results and thus, the
ndings are not shown here. The results imply that there is
an energy functional that unies the RW and RWR models.
This interesting observation gives us new insights into ltering
algorithm design. Consequently, we must investigate why the
RW and the RWR methods are robust to outliers. The answer
is closely related to the thermal diffusivity in physics, which
will be described in the next section.
2) Thermal Diffusivity in the Diffusion and the RWR
Models: First, let us explain the physical meaning
of the thermal diffusivity. Thermal diffusivity
in the diffusion is dened as T = k
_
[35],
where k and represent the thermal conductivity
and the volumetric heat capacity, respectively. Materials
with a low (high) thermal diffusivity slowly (rapidly) adapt
their temperature to the surrounding environment. This implies
that the afnity function w(x, y) and thermal diffusivity T
play a similar role, e.g., a low afnity w(x, y) corresponds
to a low thermal diffusivity, thus preventing diffusion and
vice versa.
Next, let us describe the volumetric heat capacity in the
diffusion and the RW (or the RWR) models. Note that the
volumetric heat capacity is closely related to the diffusion
velocity and the purity of the material. For instance, pure mate-
rials have a higher volumetric heat capacity (a lower diffusion
velocity) than mixtures. In other words, materials with a high
volumetric heat capacity slowly adapt their temperature to the
surrounding environment and vice versa.
However, thermal diffusivity, as dened in the classical
diffusion, does not include the volumetric heat capacity,
i.e., rapidity is not taken into account. Specically, the dif-
fusion as described by (12) rarely occurs when a center node
has a different distribution from its neighborhood, making the
diffusion process sensitive to impulsive outliers, as shown in
Fig. 1(b). Classical anisotropic diffusion approaches model the
thermal diffusivity with the thermal conductivity only, i.e., the
volumetric heat capacity is set to 1 as follows:
T
D
= k w(x, y). (17)
In contrast, the thermal diffusivity of the RW and the RWR
models includes the thermal conductivity w(x, y) as well as
the volumetric heat capacity
_

w(x, y)dy as follows:

T
R
=
k

=
w(x, y)
_

w(x, y)dy
. (18)
The denominator
_

w(x, y)dy in (18) indicates the purity

of the image (the volumetric heat capacity); it becomes small
when a signal within a neighborhood belongs to a mixture
(the outlier) and vice versa. Therefore, the diffusion velocity
increases when outliers exist, making the RW and the RWR
methods more robust to outliers, as shown in Fig. 1(c).
To summarize, the ltering properties of the two approaches
completely differ according to the denition of the volumetric
heat capacity.
3) Energy Functional Unifying the RW and RWR Models:
We have observed that RW-based approaches are more robust
to impulsive outliers when compared to conventional diffusion
methods due to the volumetric heat capacity. Based on this
observation, a new energy functional unifying the RW and the
RWR models is proposed as follows:
E
RWR
(u) =
1
4
_

(u(x) u(y))
2
w(x, y)dydx
+

2
_

(u(x) f (x))
2
dx (19)
where
w(x, y) =
w(x, y)
_

w(x, y)dy
. (20)
Note that the volumetric heat capacity
_

w(x, y)dy
is directly incorporated into the energy functional. From
2578 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 22, NO. 7, JULY 2013
TABLE I
COMPARISON OF THE SOLUTIONS OF THE ENERGY FUNCTIONAL E
DR
(u)
IN (14) AND THE ENERGY FUNCTIONAL E
RWR
(u) IN (19)
= 0 Flow Solution Steady-State Solution
E
DR
(u) Anisotropic diffusion [1] Random walk (RW) [9]
E
RWR
(u) Robust scale-space
lter [36]
Random walk (RW) [9]
= 0 Flow Solution Steady-State Solution
E
DR
(u) Diffusion-reaction
equation [31]
Local version of
NDS [24]
E
RWR
(u) Robust diffusion-reaction
equation
Random walk with restart
(RWR) [33]
a probabilistic point of view, the thermal diffusivity
w(x, y)
__

w(x, y)dy corresponds to the probability that a

random walker at x transits to y in a single step. Two solutions
can also be obtained as follows.
a) Flow solution via steepest descent method:

t
u(x) =
_

(u(y) u(x)) w(x, y)dy

_

w(x, y)dy
+( f (x) u(x)).
(21)
b) Steady-state solution via gaussjacobi iteration:
u
t +1
(x) =
1
1 +
_

u
t
(y)w(x, y)dy
_

w(x, y)dy
+

1 +
f (x). (22)
The steady-state solution is exactly the same as the RWR
model in (13) when c is substituted for
_
(1 +) in (22).
In contrast to the energy functional of (14), the steady-state
solution becomes the RWR model when = 0; otherwise,
the solution is the RW model. Interestingly, when = 0,
the ow solution corresponds to the robust scale-space lter
recently proposed in [36], which is more robust to outliers than
the classical anisotropic diffusion scheme in (1). Therefore,
(21) can be thought of as the diffusion-reaction counterpart of
the robust scale-space lter, i.e., the robust diffusion-reaction
equation. The energy functionals E
DR
(u) and E
RWR
(u) are
compared in Table I. Note that when = 0, the steady-state
solutions of E
DR
(u) and E
RWR
(u) become equivalent to those
of the RW model.
C. Generalized RWR (GRWR)
The unied energy functional in (19) allows us to generalize
the RWR model in nonlocal forms. Let us consider the
following energy functional:
E
GRWR
(u) =
1
4
_

(u(x) u(y))
2
w
S
(x, y)dydx
+

2
_

(u(x) f (y))
2
w
D
(x, y)dydx (23)
where
w
i
(x, y) =
w
i
(x, y)
_

w
i
(x, y)dy
, i = S, D. (24)
The afnity functions w
S
(x, y) or w
D
(x, y) can be repre-
sented by the following local, semi-local, and nonlocal afnity
functions, respectively.
a) Local afnity function:
w
L
(x, y) = w(x, y)
=
_
exp
_
| f (x) f (y)|
2
/h
2
_
, y
L
(x)
0, y /
L
(x)
(25)
with
L
(x) = {y \x : |y x| 1}.
b) Semi-local afnity function:
w
SL
(x, y)
=
_
_
_
exp
_
| f (x) f (y)|
2
/h
2
|x y|
2
/2s
2
_
, y
SL
(x)
0, y /
SL
(x)
(26)
with
SL
(x) = {y \x : |y x| r}. The window radius
and spatial bandwidth are denoted as r and s, respectively.
c) Nonlocal afnity function:
w
NL
(x, y)
=
_
exp
_
f
B
(x) f
B
(y)
2
/h
2
_
, y
NL
(x)
0, y /
NL
(x)
(27)
where
NL
(x) = {y \x} and f
B
(x) denotes a vector
consisting of a patch centered at pixel x.
1) Flow Solution Via Steepest Descent Method: The ow
solution of (23) can be obtained as:

t
u(x) =
_

(u(y) u(x)) w
S
(x, y)dy
_

w
S
(x, y)dy
+
_

( f (y) u(x)) w
D
(x, y)dy
_

w
D
(x, y)dy
(28)
which can be seen as a generalized robust diffusion-reaction
equation.
2) Steady-State Solution Via GaussJacobi Iteration: The
steady-state solution of (23) can be derived as
0 =
_

(u(x) u(y)) w
S
(x, y)dy
+
_

(u(x) f (y)) w
D
(x, y)dy. (29)
The nal solution is given by
u
t +1
(x) =
1
1 +
_
_

u
t
(y)w
S
(x, y)dy
_

w
S
(x, y)dy
_
+

1 +
_
_

f (y)w
D
(x, y)dy
_

w
D
(x, y)dy
_
. (30)
The RWR becomes a special case of (30) when w
S
(x, y)
and w
D
(x, y) are set to the local afnity function in (25) and
a constant, respectively. The RW model can be derived by
setting w
S
(x, y) to the local afnity with being 0.
3) Relationship Between the GRWR and Other Regulariza-
tion Methods: The solution of the energy functional in (23)
is related to other regularization methods according to the
solution type and/or the afnity function used, as summarized
in Table II.
HAM et al.: GENERALIZED RWR 2579
TABLE II
RELATIONSHIP BETWEEN THE PROPOSED ENERGY FUNCTIONAL IN (23) AND OTHER REGULARIZATION METHODS
Method Solution Type w
D
(x, y)
()
(x) of w
D
(x, y) w
S
(x, y)
()
(x) of w
S
(x, y)
AD [1] Flow = 0 - - w
S
(x, y) {y \x : |y x| 1}
RSS [36] Flow = 0 - - w
S
(x, y) {y \x : |y x| 1}
NLD [24] Flow = 0 Constant {y : y = x} w
S
(x, y) {y \x}
RW [9] Steady-state = 0 - - w
S
(x, y) or w
S
(x, y) {y \x : |y x| 1}
BL [7] Steady-state
(noniterative)
= 0 - - w
S
(x, y) or w
S
(x, y) {y : |y x| r}
NLM [26] Steady-state
(noniterative)
= 0 - - w
S
(x, y) or w
S
(x, y) {y }
RWR [20] Steady-state = 0 Constant {y : y = x} w
S
(x, y) {y \x : |y x| 1}
GRDR Flow = 0 w
D
(x, y) {y : |y x| r}
or {y \x}
w
S
(x, y) {y : |y x| r} or
{y \x}
GRWR Steady-state = 0 w
D
(x, y) {y : |y x| r}
or {y \x}
w
S
(x, y) {y : |y x| r} or
{y \x}
AD: anisotropic diffusion [1], RSS: robust scale-space lter [36], NLD: nonlocal diffusion [24], RW: random walk [9], BL: bilateral lter [7],
NLM: nonlocal mean lter [26], RWR: random walk with restart [20], GRDR: generalized robust diffusion-reaction equation,
GRWR: generalized random walk with restart.
First, the regularization methods are classied according
to the solution types: the ow solution and the steady-state
solution. The ow solution is more exible than the steady-
state solution, since the diffusion time, i.e., the number of
iterations, can be adjusted, resulting in varying solution with
respect to time. In contrast, the steady-state solution is unique
(piecewise smooth) for the given energy functional, since there
is no energy transition in the steady-state. It means their
usage depends on the applications. For example, the nonlocal
diffusion [24] is more suitable in denoising images than the
GRWR, although the number of iterations should be specied
to yield visually improved results. In the image segmentation,
the GRWR achieve better results than the nonlocal diffusion,
since the GRWR gives the piecewise constant solution in the
steady-state, which is preferred in the image segmentation.
Second, the regularization parameter heavily inuences
the smoothness of the solution. Especially, it determines
whether the solution is meaningful in the steady-state. When
= 0, the solution diffuses only, and thus gives a trivial solu-
tion in the steady-state. In contrast, the steady-state solution
becomes meaningful when = 0. As this parameter is set to
smaller, the solution becomes smoother, and vice versa. It also
has a similar role in the ow solution.
Finally, the afnity function also determines the type of
the regularization. Generally, the nonlocal afnity function is
based on a patch, which is closely related to a self-example
concept. The nonlocal afnity function is capable of discrimi-
nating texture information from noisy images, in contrast to the
local and semi-local afnity functions. Therefore, the nonlocal
mean lter [26] has been widely used in the image denoising
despite its huge computational overhead.
D. Implementation
Let u
k
: R
+
be a function on a discrete image domain,
where N
2
is an open and bounded space with k and
l being 2D vectors, representing spatial coordinates. The
u
n
k
term denotes the intensity of the pixel k at time n. The
discrete afnity function w[k, l] between two nodes
k = (k
1
, k
2
) and l = (l
1
, l
2
) is dened within an interest
domain N
()
(x), which is a discrete counterpart of
()
(x).
The thermal diffusivity function in (24) is discretized as
w
i
[k, l] =
w
i
[k, l]

l
w
i
[k, l]
, i = S, D. (31)
We can set w
i
[k, l] to the following functions.
a) Local afnity function:
w
L
[k, l] =
_
exp
_
| f
k
f
l
|
2
/h
2
_
, l N
L
0, l / N
L
(32)
with N
L
= {l \k : |l k| 1}.
b) Semi-local afnity function:
w
SL
[k, l]
=
_
exp
_
| f
k
f
l
|
2
/h
2
|k l|
2
/2s
2
_
, l N
SL
0, l / N
SL
(33)
with N
SL
= {l \k : |l k| r}.
c) Nonlocal afnity function:
w
NL
[k, l] =
_
exp
_

_
_
f
B,k
f
B,l
_
_
2
/h
2
_
, l N
NL
0, l / N
NL
(34)
where N
NL
= {l \k} and f
B,k
denotes the patch around
pixel k. Theoretically, this function measures a patch-wise
similarity with an entire image except a reference pixel k, but
in practice, for computational efciency, the interest domain
N
NL
is usually constrained using a set of neighboring pixels
within a certain spatial distance, i.e., N
NL
= {l \k :
|l k| r
N
}. From here on, we use the constrained interest
domain when dening the nonlocal afnity function w
NL
.
2580 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 22, NO. 7, JULY 2013
1) Flow Solution Via Steepest Descent Method: By approx-
imating the partial derivative with respect to time via the
forward Euler equation with an evolution step size n, we
can discretize (28) as follows:
u
n+1
k
= u
n
k
+n
_
l
w
S
[k, l]u
n
l

l
w
S
[k, l]
u
n
k
_
+n
_
l
w
D
[k, l] f
l

l
w
D
[k, l]
u
n
k
_
. (35)
While (35) has a similar form to the nonlocal diffusion
equation [24], it can have a larger evolution step size than
that of nonlocal diffusion. The evolution step size in the
nonlocal diffusion model should decrease according to the
neighborhood size so as to ensure stability (see Appendix I).
2) Steady-State Solution Via GaussJacobi Iteration: The
steady-state solution of (30) can be approximated as:
u
n+1
k
=
1
1 +

l
w
S
[k, l]u
n
l

l
w
S
[k, l]
+

1 +

l
w
D
[k, l] f
l

l
w
D
[k, l]
.
(36)
Note that the solution remains unchanged when it reaches
the steady-state, i.e., u
n+1
k
= u
n
k
.
The steady-state solution can also be written in matrix
form [37]. We will re-formulate (36) using combinatorial nota-
tion. Let f and u
S
denote an M1 column vector representing
the initial condition and the steady-state solution, respectively.
Also, let W
S
= [w
S
[k, l]]
MM
and W
D
= [w
D
[k, l]]
MM
denote the smoothness and the data afnity matrix with a
size M M. The corresponding degree matrix is represented
by D
S
= diag(D
1
S
, . . . , D
M
S
) and D
D
= diag(D
1
D
, . . . , D
M
D
),
where D
k
S
=

l
w
S
[k, l] and D
k
D
=

l
w
D
[k, l].
When the solution reaches the steady-state, (36) can be re-
written in combinatorial form as
u
S
=
1
1 +
P
S
u
S
+

1 +
P
D
f (37)
where P
S
= D
1
S
W
S
(P
D
= D
1
D
W
D
) represents a transition
matrix whose elements p
S
k,l
( p
D
k,l
) can be interpreted as the
total probability with which a random walker reaches u
k
( f
k
)
from u
l
( f
l
) after a single iteration [38]. In contrast to the
RWR model in (13), where a random walker moves back to
an initial position with a xed probability c =
_
(1 +), a
random walker in the proposed method goes back to an initial
position with a probability cP
D
, which takes the local structure
P
D
into account. Thus, the GRWR scheme can capture the
ne structure and texture information better. The steady-state
solution can be expressed as:
u
S
=
_

1 +
__
I
1
1 +
P
S
_
1
P
D
f
=
_

1 +
_

n=0
_
1
1 +
_
n
P
n
S
P
D
f (38)
where I denotes the identity matrix of size M.
E. Relationship Between the Flow and Steady-State Solutions
We have shown that E
DR
(u) in (14) unies the anisotropic
diffusion and the diffusion-reaction equation, and E
RWR
(u)
in (19) and E
GRWR
(u) in (23) unify the RW and the RWR
models. In this section, we explore the relationship between
their ow and steady-state solutions.
Proposition 2: The steady-state solutions of E
DR
(u),
E
RWR
(u), and E
GRWR
(u) are equivalent to the corresponding
ow solutions with a maximum evolution step size.
Proof: Let us approximate
t
u(x) as
u
n+1
k
u
n
k
n(k)
(39)
with an evolution step size n(k). Then, (15), (21), and (28)
can be represented, respectively, as
u
n+1
k
=
_
1 n(k)

l
w[k, l] n(k)
_
u
n
k
+n(k)

l
w[k, l]u
n
l
+n(k) f
k
(40)
u
n+1
k
= (1 n(k) n(k)) u
n
k
+n(k)

l
w[k, l]u
n
l

l
w[k, l]
+n(k) f
k
(41)
and
u
n+1
k
= (1 n(k) n(k)) u
n
k
+n(k)

l
w
S
[k, l]u
n
l

l
w
S
[k, l]
+n(k)

l
w
D
[k, l] f
l

l
w
D
[k, l]
. (42)
Note that (40), (41), and (42) represent the ow solutions
for E
DR
(u), E
RWR
(u), and E
GRWR
(u), respectively. Therefore,
the corresponding stability conditions for the ow solutions are
computed as
0 n(k)
1

l
w[k, l] +
(43)
0 n(k)
1
1 +
(44)
and
0 n(k)
1
1 +
. (45)
When the evolution step size is set to its maximum
value, (40)(42) can be written as
u
n+1
k
=

l
w[k, l]u
n
l
+ f
k

l
w[k, l] +
(46)
u
n+1
k
=
1
1 +

l
w[k, l]u
n
l

l
w[k, l]
+

1 +
f
k
(47)
and
u
n+1
k
=
1
1 +

l
w
S
[k, l]u
n
l

l
w
S
[k, l]
+

1 +

l
w
D
[k, l] f
l

l
w
D
[k, l]
(48)
which are identical to the steady state solutions of E
DR
(u),
E
RWR
(u), and E
GRWR
(u) in (16), (22), and (36).
Remark 1: When = 0 in (40), the RW model can be
referred to as anisotropic diffusion with an adaptive evolution
step size n(k) = 1
_
l
w[k, l], which plays the same role
as the volumetric heat capacity. Thus, the RW approach is
HAM et al.: GENERALIZED RWR 2581
0 500 1000 1500 2000 2500 3000 3500
0.88
0.9
0.92
0.94
0.96
0.98
1
iteration
n
o
r
m
a
l
i
z
e
d

e
n
e
r
g
y
RWR
GRWR
Fig. 2. Normalized energy of the RWR and the GRWR methods, according
to the number of iterations. A test image of Fig. 1(a) (the rst column)
is regularized by the two methods, and then a normalized energy of both
methods is experimentally measured using (19) and (23), respectively. The
range parameter and the restarting probability are set to 10 and 0.01,
respectively, for both methods. In the GRWR method, the nonlocal afnity
function is used in which the patch radius r
P
and neighborhood radius r
N
are set to 2 and 5, respectively. Although both methods guarantee nontrivial
steady-state solutions, the GRWR method achieves a lower normalized energy
in the steady-state as well as a faster convergence rate.
more robust to outliers than the anisotropic diffusion scheme,
although both are derived from the same energy functional.
IV. EXPERIMENTAL RESULTS
In this section, the GRWR method is applied to depth
image up-sampling and interactive image segmentation. The
GRWR could be an alternative to other regularization methods
since a global relation can be effectively captured due to the
steady-state property of the GRWR. Furthermore, the GRWR
approach can effectively handle the complicated texture in
highly cluttered regions.
A. Implementation
The RWR and the GRWR methods can be implemented via
two ways [37]. One is to yield the steady-state solution through
the power iteration, i.e., by repeatedly applying GaussJacobi
iteration as in (36). The other is to pre-compute and store
the pseudo inversion of the matrix I (1 c)P
S
as in (38).
In general, the computational load of calculating the pseudo
inversion depends heavily on the sparseness of the matrix.
As opposed to the local afnity function, the semi-local
and nonlocal afnity functions utilize many neighbors inside
N
SL
and N
NL
, resulting in the semi-dense transition matrix.
Thus, the GRWR method using these afnity functions was
implemented via the power iteration, which does not require
huge pre-computation and memory usage. In this case, it is
important to pre-set the number of iterations for computational
efciency. Namely, the steady-state solution can be efciently
obtained by seeking the minimum number of GaussJacobi
iterations in order for the solution to converge.
To nd the minimum iteration number, a test image of
Fig. 1(a) (the rst column) was regularized by the RWR and
the GRWR method, and then each normalized energy was
experimentally measured using (19) and (23), respectively,
according to the number of iteration, as shown in Fig. 2.
Note that the normalized energy of the RWR method, which
uses the local afnity function, was also measured using the
power iteration to compare the convergence rate, even though
its transition matrix I (1 c)P
S
is sparse. The range para-
meter and the restarting probability were set to 10 and 0.01,
respectively, for both methods. In the GRWR method, the
nonlocal afnity function in (34) was used in which the patch
radius r
P
and neighborhood radius r
N
were set to 2 and 5,
respectively. It is shown that both methods guarantee non-
trivial steady-state solutions, but the GRWR method achieves
a faster convergence rate. Following this observation, we xed
the iteration number to 300 for the GRWR and 3000 for the
RWR in all experiments. Although it might vary according
to a signal type, resolution and/or application, we found
through various experiments that excellent results could be
obtained.
B. Depth Map Up-sampling
1) Background: In the eld of 3D computer vision, it
is important to nd visual correspondence between images.
While many stereo algorithms have been extensively studied,
most methods are still far from being implemented in practical
use due to their heavy computational complexity and unstable
accuracy. Alternatively, active depth sensors can be used to
obtain depth information, but their quality is not satisfactory,
e.g., depth maps captured by Mesa Imaging SR4000 have
low-resolution and are noisy [39]. Many studies have been
performed so as to overcome this limitation [40][43]. The
GRWR approach can be an appropriate solution to depth
map up-sampling due to its steady-state property. Note that
the steady-state solution cannot be obtained by conventional
methods, which are based on the bilateral lter [41], [42] or
mode lter [43].
2) Experimental Environments: The semi-local afnity
function (w
S
and w
D
) in (33) was utilized to up-sample the
depth map. The nonlocal afnity is not appropriate for use in
depth up-sampling. Since it is based on the patch similarity,
neighboring pixels with high afnity values can be found at
different depth layers, causing serious depth fatting problems
on depth discontinuities.
We compared the performance of the GRWR model with
that of the 2D joint bilateral up-sampling (2D JBU) [41],
the 3D joint bilateral up-sampling (3D JBU) [42], and the
weighted mode lter (WMF) [43]. All parameters were xed
during the experiments: In Section IV-B-3), the spatial and the
range bandwidths were set to 3.0 and 5.0, respectively, for all
of the algorithms, while the histogram bandwidth of the WMF
was set to 21. The neighborhood radius r was set equal to the
spatial bandwidth. The restarting probability c =
_
(1 +) in
the GRWR model was set to 0.01. For a proper comparison,
no pre-processing or post-processing (e.g., the multiscale color
measure (MCM) used in the WMF) was employed [43].
In Section IV-B.4), all parameters were set to the same as
that of section IV-B-3) except that the spatial bandwidth and
the neighborhood radius r were set to 7.0.
3) Performance Evaluation With Noisy Depth Maps: The
reference images (from top to bottom) Teddy and Cones are
2582 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 22, NO. 7, JULY 2013
(a) (b) (c) (d) (e) (f)
Fig. 3. Depth up-sampling results for Middlebury test bed images. (a) Reference images (from top to bottom) Teddy and Cones. (b) Initial low-resolution
depth maps corrupted by the Gaussian noise with a standard deviation of 30. The down-sampling ratio is eight in each dimension. (c) 2-D JBU [41].
(d) 3D JBU [42]. (e) WMF [43]. (f) GRWR results. In the GRWR, the semilocal afnity function in (33) is utilized.
(a) (b) (c) (d) (e)
Fig. 4. Results of the synthesized virtual views. The virtual views are synthesized using the reference images (from top to bottom) Teddy and
Cones, and the corresponding up-sampled depth map as in Fig. 3. (a) Ground truth depth map. (b) 2D JBU [41]. (c) 3D JBU [42]. (d) WMF [43].
(e) GRWR. The outliers in the up-sampled depth degenerate the quality of the synthesized view. The depth map corrupted by the outlier leads to geometric
distortion in the synthesized view.
shown in Fig. 3(a). Initial low-resolution depth maps obtained
by down-sampling with a factor of 8 in each dimension were
corrupted by the Gaussian noise with a standard deviation
of 10, 20, and 30. Fig. 3 shows the up-sampling results when
the standard deviation is 30. As evident in Fig. 3(c), the
2D JBU method suppresses the noise to some extent, but also
smooths important features such as depth boundaries due to
its inherent averaging property [43]. Note that the up-sampled
depth maps with noises have inaccurate depth information,
leading to geometric distortion. In Fig. 3(d), the 3D JBU
scheme shows similar results to those obtained by the 2D JBU,
since it also leverages the summation (averaging) property
of the bilateral lter when aggregating the cost. The results
obtained by the WMF are shown in Fig. 3(e). The WMF
regularizes the depth map by utilizing the mode, i.e., it nds
a global maximum in a localized histogram for each pixel.
Thus, the WMF is more robust to noises than the 2D JBU
and the 3D JBU methods and it preserves object boundaries
well. However, some artifacts are still observed. It should be
noted that the results of the WMF may be different from
those of the original paper [43], since the MCM proposed
in [43] is not used for fair performance comparison. Depth
maps up-sampled by the GRWR method are shown in Fig. 3(f).
In contrast to the aforementioned ltering-based approaches,
the GRWR approach yields a steady-state solution that is non-
trivial due to the restarting term. The noise was successfully
suppressed while universal features were effectively preserved.
Note that the robustness against noises is consistent with the
characteristics of the robust scale-space lter [36], which is a
specic case of the ow solution of the GRWR model. In [36],
it was shown that the robust scale-space lter is robust against
various outliers such as the Gaussian noise, the impulsive
noise, and a combination of the two.
For a quantitative evaluation, the percent of bad matching
pixels at all regions was measured with ground truth depth
maps [44], as shown in Table III. It shows that the 2D JBU
slightly outperformed the 3D JBU in the noisy environment.
The performance of both methods drastically varied according
to the standard deviation of the noise. In contrast, the WMF
and the GRWR give consistent results even though the initial
depth map is degraded by severe noise. We also found that
the performance of the GRWR approach is superior to those
of the other methods.
To visualize the inuence of the degraded depth maps,
virtual views were synthesized using the reference images
and the corresponding up-sampled depth maps as in Fig. 3.
As shown in Fig. 4, the degraded depth maps signicantly
inuence the quality of the synthesized view. In addition, there
HAM et al.: GENERALIZED RWR 2583
(a) (b) (c) (d) (e) (f)
Fig. 5. Up-sampling results for depth maps captured by the active range sensor (the rst and second rows) ToF sensor (SR4000) [39] and (the third row)
structured light sensor (Kinect) [47]. In the ToF (structured light) sensor setup, the sizes of the input depth and reference images are 176 144 (80 60)
and 1024 768 (640 480), respectively. The depth maps are normalized between 0 and 255. (a) Reference images. (b) Initial low-resolution depth maps.
(c) 2D JBU [41]. (d) 3D JBU [42]. (e) WMF [43]. (f) GRWR results. In the GRWR, the semilocal afnity function in (33) is utilized.
TABLE III
OBJECTIVE COMPARISON (THE PERCENT OF BAD MATCHING PIXELS) OF
THE UP-SAMPLED DEPTH MAPS
Tsukuba = 10 = 20 = 30
2D JBU [41] 16.30 28.50 42.30
3D JBU [42] 17.20 29.40 42.50
WMF [43] 20.50 18.80 21.30
GRWR 20.90 21.60 27.20
Venus = 10 = 20 = 30
2D JBU [41] 17.60 48.00 61.10
3D JBU [42] 18.60 48.80 63.80
WMF [43] 5.55 12.10 26.60
GRWR 3.34 9.99 19.30
Teddy = 10 = 20 = 30
2D JBU [41] 51.30 73.00 80.60
3D JBU [42] 52.40 73.90 81.20
WMF [43] 34.10 44.70 64.20
GRWR 26.40 41.70 57.30
Cones = 10 = 20 = 30
2D JBU [41] 53.90 73.80 81.00
3D JBU [42] 55.10 74.20 81.30
WMF [43] 43.60 53.00 64.50
GRWR 38.10 49.50 62.60
are many holes in the synthesized view, since the distorted
depth values warp several pixels into the wrong location of
the virtual view.
4) Performance Evaluation With Depth Maps from Active
Range Sensor: We also up-sampled depth maps obtained from
the active range sensors (the Time-of-Flight (ToF) sensor and
the structured light sensor) in Fig. 5. For the ToF sensor
conguration, the SR4000 depth sensor [39] and the Point
Grey Flea camera [45] were used to capture single depth map
and its corresponding color image. For a spatial alignment,
the calibration parameters estimated for two cameras were
utilized to warp the depth data into the corresponding spatial
coordinate of the color camera. The sizes of the input depth
and color images are 176 144 and 1024768, respectively.
The depth maps were normalized between 0 and 255. For
simulating the up-sampling with the structured light sensor,
we used the the data set provided by [46] which offers the
depth map and corresponding color image captured by the
Microsoft Kinect depth sensor [47]. The sizes of the input
depth and color image are 8060 and 640480, respectively.
An original depth map (640480) from the Kinect was down-
sampled with a factor of 8 in order to verify the upsampling
performance. Note that the depth value describes a distance
from the sensor, different from the disparity value in Fig. 3.
The 2D JBU and the 3D JBU enhance the depth boundaries
to some extent compared to the initial depth map, but the
quality on depth boundaries is still dizzy. The WMF gives
better results than those of the 2D JBU and the 3D JBU.
It was already shown in [43] that by applying the MCM to
the input sparse depth maps, the WMF approach can achieve
very accurate results on the depth boundaries, but in our
experiments, the bilinearly interpolated depth maps, which
may be even more seriously noisy across the depth boundaries
than the input original sparse depth maps, were used as initial
inputs for fair comparison with other methods. Thus, Fig. 5(e)
shows a bit worse results around the depth boundaries than
those of the original WMF paper [43]. In Fig. 5(f), we found
that the GRWR shows the best performance among all the
methods, even though the MCM is not used like the case of the
WMF in Fig. 5(e), since it always gives a piecewise constant
solution in the steady-state.
C. Interactive Image Segmentation
1) Background: Interactive image segmentation has become
increasingly important for digital image editing since it yields
satisfactory results that are unattainable by state-of-the-art
2584 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 22, NO. 7, JULY 2013
(a)
0.49 0.57 0.47 0.54
(b)
0.75 0.72 0.61 0.60
(c)
0.77 0.78 0.69 0.71
(d)
0.69 0.81 0.67 0.70
(e)
0.83 0.85 0.90 0.81
(f)
Fig. 6. Segmentation results on texture images. (a) Original images
(the rst and third columns) are corrupted by both the Gaussian noise with a
standard deviation of 20 and the impulsive noise with a density of 0.05 (the
second and fourth columns). Green and blue strokes: the foreground and the
background seeds, respectively. (b) RW [10], (c) ARW [48], (d) RWR [20],
(e) nonlocal diffusion [24], and (f) GRWR results. In the GRWR, the nonlocal
afnity function in (34) is used. The normalized overlap scores are given under
the results. As expected, the GRWR approach is more robust to impulsive
outliers than the RW, the RWR, and the nonlocal diffusion (the second and
fourth columns) schemes. The GRWR method also segments the texture in
the highly cluttered regions very well, in the absence of noise (the rst and
third columns). (a) Input image. (b) RW [10]. (c) ARW [48]. (d) RWR [20].
(e) Nonlocal diffusion [24]. (f) GRWR.
automatic methods. One of the most difcult tasks in segmen-
tation is to separate the texture in highly cluttered regions.
There have been many attempts to address the above
problem [10], [19], [20], [24], [32]. The graph cuts-based
approach mostly produces allowable results in a general image.
However, the method suffers from the small cut problem due
to the inherent nature of the max-ow/min cut algorithm [18].
While the RW and the RWR schemes (which are based on a
probabilistic framework) do not cause the small cut problem,
they still do not work in highly textured images [10], [20],
[48]. This texture problem could be handled to some extent
by leveraging prior models based on a statistical distribution
of the foreground and the background [19], but it signicantly
increases the computational complexity. Furthermore, the per-
formance of the approach depends heavily on the statistical
model used. The nonlocal diffusion method recently proposed
by Gilboa and Osher can segment cluttered regions using the
nonlocal neighborhood [24], but, unlike the RWR, it does
not consider the global relation and thus, it requires that
the number of iterations be set manually for an accurate
segmentation.
The GRWR method with the nonlocal afnity function
in (34) can be an alternative solution to handle the texture
problem in image segmentation since the nonlocal afnity
function inherently considers both texture and structure infor-
mation, similar to the nonlocal diffusion model. Furthermore,
the global relation can be easily captured in the steady-state
solution.
2) Experimental Environments: The performance of the
RW [10], the anisotropic RW (ARW) [48], the RWR [20],
the nonlocal diffusion [24], and the GRWR approaches was
compared. All parameters were xed during the experiments:
the range parameter was set to 10 for all of the algorithms,
while the restarting probability of the RWR and the GRWR
models was set to 0.01. In the nonlocal diffusion approach,
the patch radius r
P
and neighborhood radius r
N
were set
to 2 and 5, respectively. Note that the performance of the
nonlocal diffusion model depends heavily on the number of
iterations, and its steady-state solution gives no meaningful
information. Thus, the number of iterations was carefully set
through intensive experiments, meaning that it should be set
differently for each image. In the GRWR, the sizes of the patch
and neighborhood in the nonlocal afnity function in (34) were
set to be equal to those of the nonlocal diffusion model.
3) Performance Evaluation With Synthetic Images: The
segmentation results obtained on the texture images are shown
in Fig. 6. The RWR [20], the ARW [48], and the nonlocal
diffusion models [24] discriminate the texture better than the
RW approach. Note that, although the RWR and the nonlocal
diffusion models show similar behavior, the number of iter-
ations in nonlocal diffusion should be carefully set through
exhaustive experiments. The ARW and the RWR models show
similar behavior, while the GRWR approach shows the best
performance in that it discriminates the texture along sharp
boundaries very well. In contrast to the nonlocal diffusion,
the GRWR model considers the global relations at all scales
due to the steady-state property and thus, the stopping criterion
is not required. Note that all RW-based approaches, including
the RW, the ARW, the RWR, and the GRWR methods, were
not affected by impulsive outliers due to the volumetric heat
capacity used in the afnity function. The nonlocal diffusion
scheme is also robust to outliers since the nonlocal operator
can, to some extent, discriminate outliers from the true signal
by using a patch similarity [24]. However, the robustness of
the nonlocal diffusion approach is still worse than that of the
GRWR scheme. For a quantitative comparison, the similarity
between the segmentation results and the ground truth was
HAM et al.: GENERALIZED RWR 2585
0.81 0.83 0.84 0.86
0.78 0.77 0.81 0.82
0.69 0.72 0.76 0.78
Fig. 7. Segmentation results on natural images. From left to right: initial input images, ARW [48], RWR [20], nonlocal diffusion [24], and GRWR results.
Green and blue strokes: the foreground and the background seeds, respectively. The nonlocal afnity function in (34) is used in the GRWR. The nonlocal
diffusion, the ARW, and the RWR approaches exhibit similar performance. The GRWR method has the combinatorial property of nonlocal diffusion and the
RWR model: texture information is successfully extracted and a nontrivial steady-state solution is guaranteed. The normalized overlap scores are given below
each result.
measured by a normalized overlap score O [49]:
O =
|A B|
|A B|
(49)
where A and B are the sets of pixels assigned as the fore-
ground from the segmentation results and from the ground
truth, respectively. A higher score means better segmentation
performance. As expected, the GRWR model is found to yield
the best segmentation results.
4) Performance Evaluation With Natural Images: We con-
ducted additional experiments with natural images [50] which
are highly textured and have a similar color distribution
between the foreground and the background. The results
obtained with the GRWR, the ARW [48], the RWR [20],
and the nonlocal diffusion approaches [24] were compared
in Fig. 7. The graph cuts and the RW schemes generally
exhibit worse performance than the RWR and thus, the results
obtained with these methods are not shown here (see the
results in [20]). Although the nonlocal diffusion and the RWR
approaches exhibit similar performance, the latter guarantees a
non-trivial steady-state solution that cannot be achieved by the
former. The GRWR scheme possesses the combinatorial prop-
erty of the two methods: the structure and texture information
are successfully extracted (see the tigers head and the birds
breast) and a non-trivial steady-state solution can be achieved
within a few iterations.
The normalized overlap score was also measured with the
ground truth data [50] in Fig. 7. The GRWR approach yielded
slightly higher scores, even though its segmentation results are
subjectively better to those of other methods. Note that this
metric does not fully reect the human visual system; it cannot
capture the coherence or the connectivity of the boundary
0.89 0.89 0.89 0.88
Fig. 8. Limitation of the objective evaluation using normalized overlap score:
segmentation results on natural images for (from left to right) the ARW [48],
the RWR [20], the nonlocal diffusion [24], and the GRWR are shown. The
normalized overlap score given below each result cannot capture the coherence
or the connectivity of the boundary completely as it simply counts the number
of segmented pixels on the entire image.
completely since it simply counts the number of segmented
pixels on the entire image. As a result, the connectivity
or the coherence along the segment boundaries is ignored.
Furthermore, when a segmented region is larger than a ground
truth, the score decreases regardless of the segmented results
(see the upper part of the stone image of Fig. 8) along the
boundaries. Consequently, the GRWR approach has a slightly
lower score than the RWR in the stone image of Fig. 8 in spite
of its better results along object boundaries.
To further verify the effectiveness of the proposed method,
we measured the percentage of mislabeled pixels (PMP)
with the ground truth data [50]. Let us denote M
A
=
_
m
1
A
, . . . , m
M
A
_
T
and M
B
=
_
m
1
B
, . . . , m
M
B
_
T
as the sets of
pixels (mask) from the segmentation results and from the
ground truth, respectively, the components of which is 1 or 0
2586 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 22, NO. 7, JULY 2013
0 200 400 600 800 1000 1200
0.88
0.9
0.92
0.94
0.96
0.98
1
iteration
n
o
r
m
a
l
i
z
e
d

e
n
e
r
g
y
(1, 2)
(2, 2)
(2, 3)
Fig. 9. Normalized energy of the GRWR method obtained using the patch
( f
B
) and neighborhood (N
NL
) with varying sizes, according to the number
of iterations. The normalized energy is measured in a manner similar to
Fig. 2. (A, B) means the patch radius r
P
and the neighborhood radius r
N
,
respectively. For instance, (1, 3) means that the patch and neighbors consists
of 3 3 and 7 7 windows, respectively. It shows that as the patch and/or
the neighborhood size increases, the convergence rate becomes faster.
TABLE IV
AVERAGE PERCENT OF MISLABELED PIXELS (PMP) IN FIG. 7 WITH
THE GROUND TRUTH DATA [50]
Method Error
Rate(%)
RW [10] 5.95
ARW [48] 4.10
RWR [20] 3.46
Nonlocal diffusion [24] 3.40
GRWR 2.47
if they belong to the foreground or background. Then, the PMP
is dened as follows:
PMP =

M
i=1
m
i
A
m
i
B
M
100 (50)
where represents the XOR operator. As shown in Table IV,
the GRWR approach has a lower error rate than the other
methods on an average.
5) Convergence: The nonlocal afnity function measures
the patch ( f
B
) similarity using many neighboring pixels inside
N
NL
so as to capture the textural information well. To explore
the relationship between the patch ( f
B
) and neighborhood
(N
NL
) size and convergence rate, we measured the normal-
ized energy of the GRWR in a manner similar to Fig. 2 by
varying the two parameters ( f
B
and N
NL
) as shown in Fig. 9.
(A, B) means the patch radius r
P
and neighborhood radius r
N
,
respectively. It shows that as the patch and/or the neighborhood
size increases, the convergence rate becomes faster.
Next, we will investigate how the convergence rate inu-
ences the accuracy of the GRWR method. Fig. 10 shows
the segmentation results obtained using the patch ( f
B
) and
neighborhood (N
NL
) with varying sizes: the patch radius r
P
was set to (from top to bottom) 0, 1, and 2, respectively. The
neighborhood radius r
N
was set to (from left to right) 3, 5,
and 7, respectively. The normalized overlap scores are given
under the results. We can obtain the following observations.
First, enlarging the patch and the neighborhood size is slightly
(0, 3), 0.91 (0, 5), 0.95 (0, 7), 0.97
(1, 3), 0.92 (1, 5), 0.97 (1, 7), 0.98
(2, 3), 0.78 (2, 5), 0.90 (2, 7), 0.97
Fig. 10. Segmentation results on texture image obtained using the patch
( f
B
) and the neighborhood (N
NL
) with varying sizes. (A, B) means the
patch radius r
P
and the neighborhood radius r
N
, respectively. (0, a) can be
thought of the semilocal function in that a pixel-wise similarity measure is
used. The normalized overlap scores are given below each result. It shows
that enlarging the patch ( f
B
) and the neighborhood (N
NL
) size accelerates
the convergence rate, but it does not always guarantee a better performance.
helpful to achieve better segmentation results. Note that it
also enables the fast convergence. Second, in order to prevent
an isolated segmentation, it is essential to use the patch-wise
similarity. In highly cluttered region, the local and/or the semi-
local afnity function cannot discriminate the structure infor-
mation since it only consider the point-wise similarity. Finally,
enlarging the patch and the neighborhood size accelerates the
convergence rate, but it does not always guarantee a better
performance. Therefore, it is important to nd the proper patch
( f
B
) and the neighborhood (N
NL
) size for different inputs.
V. CONCLUSION
A. Summary
In this paper, the origin of the RWR model was investigated.
We showed that the RWR approach has a different theoretical
foundation from the diffusion-reaction equation and has better
ltering behaviors with respect to impulsive outliers. This
allowed us to propose a new energy functional unifying the
RW and the RWR schemes and further generalize the RWR
within semi-local and nonlocal forms. The GRWR approach
with the nonlocal afnity function can aggregate texture infor-
mation better than the RWR method, while maintaining the
steady-state property of the RWR, i.e., the global relation
can be captured. To verify the performance of the GRWR
approach, it was applied to depth image up-sampling and inter-
active image segmentation. The experimental results showed
the superiority of the GRWR over existing regularization
methods. However, the parameters used in the experiment
were not fully optimized such as the range parameter, the
patch and the neighborhood size, the restarting probability,
and the number of iterations. Thus, it is expected that the
HAM et al.: GENERALIZED RWR 2587
performance of the GRWR method could be further improved
with optimized parameters.
B. Discussion
The GRWR approach can be combined with more powerful
techniques. First, it can be applied to soft image segmentation
in a similar manner to that described in [32], which results in
the ability to composite or edit an image seamlessly. Second,
the GRWR method can be accelerated in two ways. It can be
thought of as the RWR approach on which semi-local or non-
local operators are imposed, so that the fast algorithms used in
the RWR [37], [51] can be applied to reduce the complexity.
In addition, the GRWR scheme has a similar form to the
bilateral lter [7] or the nonlocal mean lter [24], which allows
it to be accelerated via signal processing techniques [52].
The proposed energy functional can give new insights into
related elds, such as the generalized PageRank [22]. The
PageRank has been widely used as a ranking criterion to
retrieve web pages in a search engine. Theoretically, the RWR
model is a special case of the PageRank. This means that
the PageRank considers the local topology or structure that
is directly linked to only a current node. Thus, the accuracy
and/or the convergence rate of retrieval algorithms can be
improved by considering a nonlocal hyperlinking topology in
the generalized PageRank. Next, the RWR and the PageRank
can be translated to even more powerful regularization or
retrieval methods by modifying their energy functional. In a
manner similar to the robust anisotropic diffusion [14], the
robust RWR scheme based on l
p
norm [49] can be derived
using a robust energy functional, e.g., the total variation-RWR
model.
APPENDIX
VI. CONVERGENCE COMPARISON BETWEEN (28) AND
NONLOCAL DIFFUSION [24]
In the ow solution, the evolution step size is crucial and
closely related to the stability and convergence. We compare
the evolution step size of the proposed method with that of
nonlocal diffusion [24]. The energy functional of nonlocal
diffusion is as follows:
E(u) =
1
4
_

(u(x) u(y))
2
w
S
(x, y)dydx
+

2
_

(u(x) f (x))
2
dx. (51)
Similar to (35), the corresponding discretized ow solution
can be found as
u
n+1
k
= (1 n

l
w
S
[k, l] n)u
n
k
+n

l
w
S
[k, l]u
n
l
+n f
k
. (52)
Since the weight of the center node u
n
k
should be between
0 and 1 so as to ensure numerical stability, the stability
condition is derived as follows:
0 n 1
_
(|N| +) (53)
where |N| denotes the number of neighbors in the window.
In contrast, the stability condition of the GRWR model is
given by
0 n 1
_
(1 +). (54)
We can see that the maximum evolution step size is xed
to 1
_
(1 +) regardless of the number of neighbors, while
that of nonlocal diffusion decreases according to the number
of neighbors. In conclusion, the proposed method can have
a larger evolution step size than nonlocal diffusion, thus
guaranteeing faster convergence without a loss of stability.
ACKNOWLEDGMENT
The rst and the second authors contributed equally to
this work. The authors would like to thank the advanced
digital science center (ADSC) for providing data sets of the
ToF sensor, and the anonymous reviewers for their valuable
comments and suggestions.
REFERENCES
[1] P. Perona and J. Malik, Scale-space and edge detection using
anisotropic diffusion, IEEE Trans. Pattern Anal. Mach. Intell., vol. 12,
no. 7, pp. 629639, Jul. 1990.
[2] D. Tschumperle and R. Ceriche, Vector-valued image regularization
with PDEs: A common framework for different applications, IEEE
Trans. Pattern Anal. Mach. Intell., vol. 27, no. 4, pp. 506517,
Apr. 2005.
[3] T. F. Chan, S. Osher, and J. Shen, The digital TV lter and nonlinear
denoising, IEEE Trans. Image Process., vol. 10, no. 2, pp. 231241,
Feb. 2001.
[4] A. Chambolle, An algorithm for total variation minimization and appli-
cations, J. Mach. Imag. Vis., vol. 20, nos. 12, pp. 8997, Jan. 2004.
[5] D. Mumford and J. Shah, Optimal approximations by piecewise smooth
functions and associated variational probelems, Commun. Pure Appl.
Math., vol. 42, no. 5, pp. 577685, Jul. 1989.
[6] T. Pock, D. Cremers, H. Cischof, and A. Chambolle, An algorithm for
minimizing the Mumford-Shah functional, in Proc. IEEE 12th Conf.
Comput. Vis., Oct. 2009, pp. 11331140.
[7] C. Tomasi and R. Manduchi, Bilateral ltering for gray and color
images, in Proc. 6th Int. Conf. Comput. Vis., Jan. 1998, pp. 839846.
[8] S. Paris, P. Kornprobst, J. Tumblin, and F. Durand, A gentle introduction
to bilateral ltering and its applications, in Proc. ACM SIGGRAPH
Classes, 2008, pp. 150.
[9] P. G. Doyle and J. L. Snell, Random Walks and Electric Networks.
Washington, DC, USA: Mathematical Association of America, 1984.
[10] L. Grady, Random walks for image segmentation, IEEE Trans. Pattern
Anal. Mach. Intell., vol. 28, no. 11, pp. 17681783, Sep. 2006.
[11] J. Y. Pan, H. J. Yanh, C. Faloutsos, and P. Duygulu, Automatic
multimedia cross-modal correlation discovery, in Proc. 10th ACM
SIGKDD Int. Conf. Knowl. Discovery Data Mining, Aug. 2004,
pp. 653658.
[12] G. Steidl, J. Weickert, T. Brox, P. Mrazek, and M. Welk, On the
equivalence of soft Wavelet Shrinkage, total variation diffusion, total
variation regularization, and SIDEs, SIAM J. Numer. Anal., vol. 42,
no. 2, pp. 686713, Sep. 2005.
[13] Y. L. You, W. Xu, and M. Kaveh, Behavior analysis of anisotropic
diffusion in image processing, IEEE Trans. Image Process., vol. 5,
no. 11, pp. 15391553, Nov. 1996.
[14] M. J. Black, G. Sapiro, D. H. Marimont, and D. Heeger, Robust
anisotropic diffusion, IEEE Trans. Image Process., vol. 7, no. 3,
pp. 421432, Mar. 1998.
[15] D. Barash, A fundamental relationship between bilateral ltering,
adaptive smoothing, and the nonlinear diffusion equation, IEEE Trans.
Pattern Anal. Mach. Intell., vol. 24, no. 6, pp. 844847, Jun. 2002.
[16] M. Elad, On the origin of the bilateral lter and ways to improve it,
IEEE Trans. Image Process., vol. 11, no. 10, pp. 11411151, Oct. 2002.
[17] R. Shen, I. Cheng, J. Shi, and A. Basu, Generalized random walks for
fusion of multi-exposure images, IEEE Trans. Image Process., vol. 20,
no. 12, pp. 36343646, Apr. 2011.
2588 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 22, NO. 7, JULY 2013
[18] Y. Boykov and G. Funka-Lea, Graph cuts and efcient N-D image
segmentation, Int. J. Comput. Vis., vol. 70, no. 2, pp. 109131,
Nov. 2006.
[19] L. Grady, Multilabel random walker image segmentation using prior
models, in Proc. IEEE Conf. Comput. Vis. Pattern Recogn., Jun. 2005,
pp. 763770.
[20] T. Kim, K. Lee, and S. Lee, Generative image segmentation using
random walks with restart, in Proc. Eur. Conf. Comput. Vis., 2008,
pp. 264275.
[21] C. Oh, B. Ham, and K. Sohn, Probabilistic correspondence matching
using random walk with restart, in Proc. Brit. Mach. Vis. Conf., 2012,
pp. 3757.
[22] T. H. Haveliwala, Topic-sensitive pageRank: A context-sensitive rank-
ing algorithm for web search, IEEE Trans. Knowl. Data Eng., vol. 15,
no. 4, pp. 784796, Sep. 2003.
[23] A. Elmoataz, O. Lezoray, and S. Bougleux, Nonlocal discrete regu-
larization on weighted graphs: A framework for image and manifold
processing, IEEE Trans. Image Process., vol. 17, no. 7, pp. 10471060,
Jul. 2008.
[24] G. Gilboa and S. Osher, Nonlocal linear image regularization and
segmentation, Multiscale Model. Simul., vol. 6, no. 2, pp. 595630,
Jul. 2007.
[25] M. Subry, S. Paris, S. W. Hasinoff, J. Kautz, and F. Durand,
Fast and robust pyramid-based image processing, in Proc.
MIT-CSAIL-TR-2001049, Nov. 2011, pp. 13.
[26] A. Buades, B. Coll, and J.-M. Morel, Nonlocal image and movie
denoising, Int. J. Comput. Vis., vol. 76, no. 2, pp. 123139, Feb. 2008.
[27] M. Protter, M. Elad, H. Takeda, and P. Milanfar, Generalizing the
nonlocal-means to super-resolution reconstruction, IEEE Trans. Image
Process., vol. 18, no. 1, pp. 3651, Jan. 2009.
[28] L. Pizarro, P. Mrazek, S. Didas, S. Grewenig, and J. Weickert, Gener-
alised nonlocal image smoothing, Int. J. Comput. Vis., vol. 90, no. 1,
pp. 6287, Apr. 2010.
[29] M. Werlberger, T. Pock, and H. Bischof, Motion estimation with non-
local total variation regularization, in Proc. IEEE Conf. Comput. Vis.
Pattern Recognit., Feb. 2010, pp. 24642471.
[30] M. Jung, X. Bresson, T. F. Chan, and L. A. Vese, Nonlocal Mumford-
Shah regularizers for color image restoration, IEEE Trans. Image
Process., vol. 20, no. 6, pp. 15831598, Jun. 2011.
[31] G. Plonka and J. Ma, Nonlinear regularized reaction-diffusion lters
for denoising of images with textures, IEEE Trans. Image Process.,
vol. 17, no. 8, pp. 12831294, Aug. 2008.
[32] L. Grady, T. Schiwietz, S. Aharon, and R. Westermann, Random walks
for interactive alpha-matting, in Proc. Int. Conf. Visualizat. Imag. Image
Process., 2005, pp. 423429.
[33] C. Wang, F. Jing, L. Zhang, and H.-J. Zhang, Image annotation
renement using random walk with restarts, in Proc. 14th Annu. ACM
Int. Conf. Multimedia, 2006, pp. 647650.
[34] J. Lee, M. Cho, and K. Lee, Hyper-graph matching via reweighted
random walks, in Proc. IEEE Conf. Comput. Vis. Pattern Recognit.,
Jun. 2011, pp. 16331640.
[35] E. Kreyszig, Advanced Engineering Mathematics, 9th ed. New York,
NY, USA: Wiley, 2006, pp. 552560.
[36] B. Ham, D. Min, and K. Sohn, Robust scale-space lter using second-
order partial differential equations, IEEE Trans. Image Process., vol. 21,
no. 9, pp. 39373951, Sep. 2012.
[37] H. Tong, C. Faloutsos, and J.-Y. Pan, Fast random walk with restart and
its application, in Proc. Int. Conf. Data Mining, 2006, pp. 613622.
[38] A. Singer, Y. Shkolnisky, and B. Nadler, Diffusion interpretation of
nonlocal neighborhood lters for signal denoising, SIAM J. Imag. Sci.,
vol. 2, no. 1, pp. 118139, Jan. 2009.
[39] [Online]. Available: http://www.mesa-imaging.ch
[40] B. Huhle, T. Schairer, P. Jenke, and W. Staber, Fusion of range and
color images for denoising and resolution enhancement with a non-local
lter, Comput. Vis. Image Understand., vol. 114, no. 12, pp. 11361345,
Dec. 2010.
[41] J. Kopf, M. F. Cohen, D. Lischinski, and M. Uyttendaele, Joint bilateral
upsampling, ACM Trans. Graph., vol. 26, no. 3, pp. 96100, Aug. 2007.
[42] Q. Yang, R. Yang, J. Davis, and D. Nister, Spatial-depth super res-
olution for range images, in Proc. IEEE Conf. Comput. Vis. Pattern
Recogn., Jun. 2007, pp. 18.
[43] D. Min, J. Lu, and M. N. Do, Depth video enhancement based on
weighted mode ltering, IEEE Trans. Image Process., vol. 21, no. 3,
pp. 11761190, Mar. 2012.
[44] D. Scharstein and R. Szeliski. (2002, Apr.Jun.). A Taxonomy and
Evaluation of Dense Two-Frame Stereo Correspondence Algorithms
[Online]. Available: http://vision.middlebury.edu/stereo/
[45] [Online]. Available: http://www.ptgrey.com
[46] [Online]. Available: http://kinectdata.com
[47] [Online]. Available: http://kinectforwindows.org
[48] J. Zhang, J. Zheng, and J. Cai, A Diffusion Approach to Seeded Image
Segmentation, in Proc. IEEE Conf. Comput. Vis. Pattern Recogn., Jan.
2010, pp. 21252132.
[49] A. K. Sinop and L. Grady, A seeded image segmentation framework
unifying graph cuts and random walker which yields a new algorithm,
in Proc. Int. Conf. Comput. Vis., Feb. 2007, pp. 18.
[50] D. Martin, C. Fowlkes, D. Tal, and J. Malik, A database of human
segmented natural images and its application to evaluating segmentation
algorithms and measuring ecological statistics, in Proc. Int. Conf.
Comput. Vis., Jul. 2001, pp. 416423.
[51] X. An and F. Pellacini, AppProp: All-pairs appearance-space edit
propagation, ACM Trans. Graph., vol. 27, no. 3, pp. 401409, 2008.
[52] S. Paris and F. Durand, A fast approximation of the bilateral lter
using a signal processing approach, Int. J. Comput. Vis., vol. 81, no. 1,
pp. 2452, Jan. 2009.
Bumsub Ham (S09) received the B.S. degree in
electrical and electronic engineering from Yonsei
University, Seoul, Korea, in 2008, where he is cur-
rently pursuing the joint M.S. and Ph.D. degrees in
electrical and electronic engineering.
His current research interests include variational
methods and geometric partial differential equations,
both in theory and applications in computer vision
and image processing, particularly regularization,
stereo vision, super-resolution, and HDR imaging.
Mr. Ham was a recipient of the Honor Prize in
17th Samsung Human-Tech Prize in 2011 and the Grand Prize in Qualcomm
Innovation Fellowship in 2012.
Dongbo Min (M09) received the B.S., M.S., and
Ph.D. degrees in electrical and electronic engineer-
ing from Yonsei University, Seoul, Korea, in 2003,
2005, and 2009, respectively.
He was with the Mitsubishi Electric Research Lab-
oratories, Cambridge, MA, USA, as a Post-Doctoral
Researcher from June 2009 to June 2010. He is
currently with the Advanced Digital Sciences Center,
which was jointly founded by the University of
Illinois at Urbana-Champaign, Urbana, IL, USA, and
the Agency for Science, Technology, and Research,
a Singapore Government Agency. His current research interests include 3-D
computer vision, video processing, 3D modeling, and hybrid sensor systems.
Kwanghoon Sohn (M92SM12) received the B.E.
degree in electronic engineering from Yonsei Univer-
sity, Seoul, Korea, in 1983, the M.S.E.E. degree in
electrical engineering from the University of Min-
nesota, Minneapolis, MN, USA, in 1985, and the
Ph.D. degree in electrical and computer engineering
from North Carolina State University, Raleigh, NC,
USA, in 1992.
He was a Senior Member of the Research Staff
with the Satellite Communication Division, Elec-
tronics and Telecommunications Research Institute,
Daejeon, Korea, from 1992 to 1993, and as a Post-Doctoral Fellow with
the MRI Center, Medical School, Georgetown University, in 1994. He was a
Visiting Professor with Nanyang Technological University, Singapore, from
2002 to 2003. He is currently a Professor with the School of Electrical
and Electronic Engineering, Yonsei University. His current research interests
include 3-D image processing, computer vision, and image communication.
Dr. Sohn is a member of SPIE.