596
1, FEBRUARY 2016
Multipath TCP: Analysis, Design, and
Implementation
Qiuyu Peng, Anwar Walid, Fellow, IEEE, Jaehyun Hwang, Member, IEEE, and Steven H. Low, Fellow, IEEE
AbstractMultipath TCP (MP-TCP) has the potential to greatly
improve application performance by using multiple paths trans-
parently. We propose a uid model for a large class of MP-TCP al-
gorithms and identify design criteria that guarantee the existence,
uniqueness, and stability of system equilibrium. We clarify how al-
gorithm parameters impact TCP-friendliness, responsiveness, and
window oscillation and demonstrate an inevitable tradeoff among
these properties. We discuss the implications of these properties
on the behavior of existing algorithms and motivate our algorithm
Balia (balanced linked adaptation), which generalizes existing al-
gorithms and strikes a good balance among TCP-friendliness, re-
sponsiveness, and window oscillation. We have implemented Balia
in the Linux kernel. We use our prototype to compare the new al-
gorithm to existing MP-TCP algorithms.
Index TermsComputer networks, convergence, nonlinear
dynamical systems, TCPIP.
I. INTRODUCTION
T RADITIONAL TCP uses a single path through the net-
work even though multiple paths are usually available in
today's communication infrastructure; e.g., most smartphones
are enabled with both cellular and WiFi access, and servers in
data centers are connected to multiple routers. Multipath TCP
(MP-TCP) has the potential to greatly improve application
performance by using multiple paths transparently. It is being
standardized by the MP-TCP Working Group of the Internet
Engineering Task Force (IETF) [2]. In this paper, we present
a uid model of MP-TCP and study how protocol parameters
affect structural properties such as the existence, unique-
ness, and stability of equilibrium, the tradeoffs among TCP
friendliness, responsiveness, and window oscillation. These
properties motivate a new algorithm that generalizes existing
MP-TCP algorithms.
Manuscript received August 13, 2013; revised May 16, 2014 and
September 27, 2014; accepted November 20, 2014; approved by IEEE/ACM
TRANSACTIONS ON NETWORKING Editor Y. Liu. Date of publication December
31, 2014; date of current version February 12, 2016. This work was supported
by the ARO MURI under Grant W911NF-08-1-0233, the NSF NetSE under
Grant CNS 0911041, Bell Labs, Alcatel-Lucent, and the Seoul R&BD Program
funded by the Seoul Metropolitan Government under Grant WR080951. A
preliminary version of this work has appeared in the Proceedings of the ACM
SIGMETRICS/International Conference on Measurement and Modeling of
Computer Systems, Pittsburgh, PA, USA, June 1721, 2013.
Q. Peng and S. H. Low are with California Institute of Technology, Pasadena,
CA 91125 USA (e-mail: qpeng@caltech.edu; slow@caltech.edu).
A. Walid is with Bell Laboratories, Murray Hill, NJ 07974 USA (e-mail:
anwar@research.bell-labs.com).
J. Hwang is with Bell Labs, Alcatel-Lucent, Seoul 121-904, Korea (e-mail:
jh.hwang@alcatel-lucent.com).
Color versions of one or more of the gures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identier 10.1109/TNET.2014.2379698
1063-6692 2014 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 24, NO.
Various congestion control algorithms have been proposed
as an extension of TCP NewReno for MP-TCP. A straightfor-
ward extension is to run TCP NewReno on each subpath, e.g.,
[3] and [4]. This algorithm, however, can be highly unfriendly
when it shares a path with a single-path TCP user. This mo-
tivates the Coupled algorithm, which is fair because it has the
same underlying utility function as TCP NewReno, e.g., [5] and
[6]. It is found in [7]however that the Coupled algorithm re-
sponds slowly in a dynamic network environment. A different
algorithm is proposed in [7] (which we refer to as the Max al-
gorithm), which is more responsive than the Coupled algorithm
and still reasonably friendly to single-path TCP users. Recently,
opportunistic linked increase algorithm (OLIA) is proposed as
a variant of Coupled algorithm that is as friendly as the Coupled
algorithm but more responsive [8]. See [9] for more references
to early work on multipath congestion control.
Our goal is to develop structural understanding of MP-TCP
algorithms so that we can systematically trade off different prop-
erties such as TCP friendliness, responsiveness, and window
oscillation that can be detrimental to applications that require
a steady throughput. For single-path TCP, one can associate a
strictly concave utility function with each source so that the
congestion control algorithm implicitly solves a network utility
maximization problem [9][11]. The convexity of this under-
lying utility maximization guarantees the existence, uniqueness,
and stability of most single-path TCP algorithms. For many
MP-TCP proposals considered by the IETF, it will be shown that
the utility maximization interpretation fails to hold in general,
necessitating the need for a different approach to understanding
the equilibrium properties of these algorithms. Moreover, the re-
lations among different performance metrics, such as fairness,
responsiveness, and window oscillation, need to be claried.
The main contributions of this paper are threefold. First, we
present a uid model that covers a broad class of MP-TCP algo-
rithms and identify the exact property that allows an algorithm
to have an underlying utility function. This implies that some
MP-TCP algorithms, e.g., the Max algorithm [7], has no asso-
ciated utility function. We prove conditions on protocol param-
eters that guarantee the existence and uniqueness of the equi-
librium and its asymptotical stability. Indeed, algorithms that
fail to satisfy these conditions, e.g., the Coupled algorithm, can
be unstable and can have multiple equilibria as shown in [7].
Second, we clarify how protocol parameters impact TCP friend-
liness, responsiveness, and window oscillation and demonstrate
the inevitable tradeoff among these properties. Finally, based
on our understanding of the design space, we propose the Balia
(Balanced linked adaptation) MP-TCP algorithm that gener-
alizes existing algorithms and strikes a good balance among
these properties. This algorithm has been implemented in the
PENG et al.: MULTIPATH TCP: ANALYSIS, DESIGN, AND IMPLEMENTATION 597

Linux kernel, and we evaluate its performance using our Linux . Let represent the sending rate on route
prototype. . Each link maintains a congestion price at time . Let
We now summarize our proposed Balia MP-TCP algorithm. be the aggregate price on route . In
Each source has a set of routes . Each route maintains a this paper, represents the packet-loss probability at link ,
congestion window and measures its round-trip time . The and represents the approximate packet-loss probability on
window adaptation is as follows. route .
For each ACK on route We associate three state variables
for each route . Let
(1) .
Then, represents the corresponding
For each packet loss on route state variables for each source . For each link , let
be its aggregate trafc rate.
(2) Congestion control is a distributed algorithm that adapts
and in a closed loop. Motivated by the AIMD algorithm of
where and .
TCP NewReno, we model MP-TCP by
The rest of the paper is structured as follows. In Section II, we
develop a uid model for MP-TCP and use it to model existing (3)
algorithms. In Section III, we prove several structural proper-
ties, focusing on design criteria that determine the existence, (4)
uniqueness, and stability of system equilibrium, TCP-friend-
where for and for . We
liness, responsiveness, window oscillation, and an inevitable
omit the time in the expression for simplicity. Equation (3)
tradeoff among these properties. In Section IV, we discuss the
models how sending rates are adapted in the congestion avoid-
implications of these properties on existing algorithms. This
ance phase of TCP at each end system, and (4) models how
motivates our new MP-TCP algorithm, and we explain our de-
the congestion price is (often implicitly) updated at each link.
sign rationale. In Section V, we compare the performance of the
The MP-TCP algorithm installed at source is specied by
proposed algorithm with existing algorithms using Linux imple-
, where and
mentations of these algorithms. We conclude in Section VI.
. Here, is a vector of positive gains
II. MULTIPATH TCP MODEL that determines the dynamic property of the algorithm.
determines the equilibrium properties of the algorithm. The link
In this section, we rst propose a uid model of MP-TCP algorithm is specied by , where is a positive gain that
and then use it to model MP-TCP algorithms in the lit- determines the dynamic property. This is a simplied model for
erature. Unless otherwise specied, a boldface letter the RED algorithm that assumes the loss probability is propor-
denotes a vector with components . We use tional to the backlog and is used in, e.g., [10] and [11].
to denote the -di-
mensional vector without and to denote B. Existing MP-TCP Algorithms
the -norm of . Given two vectors , means
We rst show how to relate the uid model (3) to the window-
for all components . A capital letter denotes a matrix
based MP-TCP algorithms proposed in the literature. On each
or a set, depending on the context. A symmetric matrix is
route , the source increases its window at the return of each
said to be positive (negative) semidefinite if
ACK. Let this increment be denoted by , where is
for any , and positive (negative) definite if
the vector of window sizes on different routes of source . The
for any . For any matrix , dene
source decreases the window on route when it sees a packet
to be its symmetric part. Given two arbitrary matrices and
loss on route . Let this decrement be denoted by . Then,
(not necessarily symmetric), means is
most loss-based MP-TCP algorithms take the form of the fol-
positive semidenite. For a vector is a diagonal
lowing pseudocodes.
matrix with entries given by .
For each ACK on route .
A. Fluid Model For each loss on route .
We now model the above pseudocodes by the uid model
Consider a network that consists of a set (3). Let be the net change to window on route in each
of links with nite capacities . The network is shared by a set round-trip time. Then, is roughly
of sources. Available to source is a
xed collection of routes (or paths) . A route consists of a set
of links . We abuse notation and use both to denote a source
and the set of routes available to it, depending on the context.
Likewise, is used both to denote a route and the set of links since the loss probability is small. On the other hand
in the route. Let be the collection of all
routes. Let be the routing matrix: if
link is in route (denoted by ), and 0 otherwise.
For each route denotes its round-trip time (RTT). Hence
For simplicity, we assume are constants. Each source main-
tains a congestion window at time for every route
598 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 24, NO. 1, FEBRUARY 2016

From (3) we have where we have ignored taking the minimum with the term
since the performance is mainly captured by .
(5) Recently, OLIA MP-TCP algorithm [8] is shown to achieve
good performance in many scenarios. OLIA uses complicated
We now apply this to the algorithms in the literature. We rst feedback congestion control signals and cannot be modeled by
summarize these algorithms in the form of a pseudocode and (3)(4). We do, however, include OLIA in our Linux-based per-
then use (5) to derive parameters and of the uid formance evaluation in Section V.
model (3).
Single-Path TCP (TCP-NewReno): Single-path TCP is a spe- III. STRUCTURAL PROPERTIES
cial case of MP-TCP algorithm with . Hence, is Throughout this paper, we assume, for all
a scalar, and we identify each source with its route . and only if for some .A
TCP-NewReno adjusts the window as follows. point is called an equilibrium of (3)(4) if it satises, for
For each ACK on route . all and
For each loss on route .
From (5), this can be modeled by the uid model (3) with

or equivalently
We now summarize some existing MP-TCP algorithms, all
of which degenerate to TCP NewReno if there is only one route and if (6)
per source. and if (7)
EWTCP [3]: EWTCP algorithm applies TCP-NewReno-like
algorithm on each route independently of other routes. It adjusts We make two remarks. First, an equilibrium does not
the window on multiple routes as follows. depend on , but only on . The design however
For each ACK on route . affects dynamic properties such as stability and responsiveness
For each loss on route . as we show below. Second, since and
From (5), this can be modeled by the uid model (3) with only if for some by assumption, any nite
equilibrium must have for all . In the following,
we always restrict ourselves to nite equilibria.
In this section, we denote an MP-TCP algorithm by
where is a constant. . We characterize MP-TCP designs
Coupled MPTCP [5], [6]: The Coupled MPTCP algorithm that guarantee the existence, uniqueness, and stability of
adjusts the window on multiple routes in a coordinated fashion system equilibrium. We identify design criteria that determine
as follows. TCP-friendliness, responsiveness, and window oscillation and
For each ACK on route . prove an inevitable tradeoff among these properties. We discuss
in Section IV the implications of these structural properties on
For each loss on route . existing algorithms. All proofs are relegated to the appendices.
From (5), this can be modeled by the uid model (3) with
A. Summary
We rst present some properties of an MP-TCP algorithm
that we have identied. We then interpret them and sum-
Semicoupled MPTCP [7]: The Semicoupled MPTCP algo- marize their implications.
rithm adjusts the window on multiple routes as follows. C0: For each and each , the Jacobians of
For each ACK on route . are continuous and symmetric, i.e.,
For each loss on route .
From (5), this can be modeled by the uid model (3) with

C1: For each , there exists a nonnegative solution

to (6) for any nite such that
Max MPTCP [7]: The Max MPTCP algorithm adjusts the for all . Moreover
window on multiple routes as follows.
For each ACK on route
.
where is the aggregate trafc at
For each loss on route .
link from source .
From (5), this can be modeled by the uid model (3) with
C2: For each and each is contin-
uously differentiable; moreover, the symmetric part
of the Jacobian is negative denite.
PENG et al.: MULTIPATH TCP: ANALYSIS, DESIGN, AND IMPLEMENTATION 599

TABLE I and ) and interpret (3)(4) as a distributed algorithm

MP-TCP ALGORITHMS to maximize aggregate users' utility, e.g., [9][12]. Indeed, for
SP-TCP, an is an equilibrium if and only if is optimal
for

s.t. (8)

and is optimal for the associated dual problem. Here,

means the aggregate trafc at each link does not exceed its
capacity . In fact, this holds for a much wider class of SP-TCP
C3: For each if and only if . algorithms than those specied by (3)(4) [12]. Furthermore, all
The routing matrix has full row rank. the main TCP algorithms proposed in the literature have strictly
C4: For each where concave utility functions, implying a unique stable equilibrium.
. The case of MP-TCP is much more delicate: Whether an un-
C5: For each and each . derlying utility function exists depends on the design choice of
These design criteria are intuitive and usually (but not al- , and not all MP-TCP algorithms have one. Consider the mul-
ways) satised; see Table I. tipath equivalent of (8)
Condition C0 guarantees the existence of utility functions
that an equilibrium of a multipath TCP/AQM s.t. (9)
(3)(4) implicitly maximizes (Theorem 3.1). It is always sat-
ised when there is only a single path ( for all ), but not where is the rate vector of ow and
when . is a concave function.
Conditions C1C3 guarantee the existence, uniqueness, and Theorem 3.1 (Utility Maximization): There exists a twice
global asymptotic stability of the equilibrium (Theorems continuously differentiable and concave such that an
3.2 and 3.3). C1 says that the aggregate trafc rate through a equilibrium of (3)(4) solves (9) and its dual problem
link from source decreases when the congestion price on if and only if condition C0 holds.
that link increases, and it decreases to 0 as increases without Condition C0 is satised trivially by SP-TCP when .
bounds. C2 implies that at steady state, if are perturbed For MP-TCP , the models derived in Section II-B show
by , respectively, then . In the case of that only EWTCP and Coupled algorithms satisfy C0 and have
single-path TCP ( for all ), C2 is equivalent to the cur- underlying utility functions. It therefore follows from the theory
vature of the utility function being negative, i.e., for SP-TCP that EWTCP has a unique stable equilibrium while
is strictly concave. C3 means that the rate on route is zero if Coupled algorithm may have multiple equilibria since its cor-
and only if it sees innite price on that route. responding utility function is not strictly concave. The other
Condition C4 is natural and satised by all the algorithms MP-TCP algorithms all have asymmetric Jacobian and do
considered in this paper. It allows us to formally compare not satisfy C0.
MP-TCP algorithms in terms of their TCP-friendliness (see
formal denition below): Under C1C4, an MP-TCP algorithm C. Existence, Uniqueness, and Stability of Equilibrium
is more friendly if is smaller (Theorem 3.4).
The existence of in C4 is ensured by C2. To interpret C4, Even though a multipath TCP algorithm may not have
note that Lemma B.2 in Appendix B implies that a utility maximization interpretation, a unique equilibrium ex-
at equilibrium. The implicit function theorem then implies ists if conditions C1C3 are satised.
at equilibrium for all . Hence, C4 Theorem 3.2 (Existence and Uniqueness):
says that the aggregate throughput at equilibrium over all 1) Suppose C1 holds. Then, (3)(4) has at least one
routes of an MP-TCP ow is a nonincreasing function of equilibrium.
the price . 2) Suppose C2 and C3 hold. Then, (3)(4) has at most one
Condition C5 is also satised by all the algorithms considered equilibrium
in this paper. It means that the sending rate on a route grows Thus (3)(4) has a unique equilibrium under C1C3.
unbounded when the congestion price is zero. Under C1C3, Conditions C1C3 not only guarantee the existence and
an MP-TCP algorithm is more responsive (see formal uniqueness of the equilibrium, they also ensure that the equi-
denition below) if the Jacobian of is more negative librium is globally asymptotically stable, when the gain
denite (Theorem 3.5). C5 then implies an inevitable tradeoff: is only a function of itself, i.e., for all
An MP-TCP algorithm that is more responsive is necessarily . This is satised by all the existing algorithms presented
less TCP-friendly (Theorem 3.6). in Section II-B.
We now elaborate on each of these properties. Theorem 3.3 (Stability): Suppose C1C3 hold and
for all . Then, the unique equilibrium
is globally asymptotically stable. In particular, starting from
B. Utility Maximization
any initial point and , the trajectory
For single-path TCP (SP-TCP), one can associate a utility generated by the MP-TCP algorithm (3)(4) con-
function with each ow ( is a scalar verges to the equilibrium as .
600 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 24, NO. 1, FEBRUARY 2016

Here, is the Jacobian of (3)(4) at the equilibrium

where , and
is evaluated at .
The stability and responsiveness of the linearized system (10)
Fig. 1. Test network for the denition of TCP friendliness. The link in the (how fast does the system converges to the equilibrium locally)
middle is the only bottleneck link with capacity . is determined by the real parts of the eigenvalues of . Speci-
cally the linearized system is stable if the real parts of all eigen-
values of are negative; moreover, the more negative the real
Our proposed algorithm does not satisfy parts are, the faster the linearized system converges to the equi-
even though it seems to be stable in our experiments. This con- librium. We now show that the linearized system (10) is stable
dition is only sufcient and needed in our Lyapunov stability (i.e., converges exponentially fast to locally) and characterize
proof; see Appendix C. When depends on , one can its responsiveness in terms of the design choices .
replace in the denition of the Lyapunov function in Let .
(21) with evaluated at the equilibrium, and the same ar- Theorem 3.5 (Responsiveness): Suppose C1C3 hold. Then,
gument there proves that is (locally) asymptotically we have the following.
stable. Also see Theorem 3.5 for an alternative proof of local 1) The linearized system (10) is stable, i.e., for
stability. any eigenvalue of . Moreover, where

D. TCP Friendliness
Informally, an MP-TCP ow is said to be TCP friendly
if it does not dominate the available bandwidth when it shares where and are evaluated at the equilibrium point
the same network with a SP-TCP ow [2]. To dene this pre- .
cisely, we use the test network shared by a SP-TCP ow and an 2) For two MP-TCP algorithms and
MP-TCP ow under test as shown in Fig. 1. provided
All paths traverse a single bottleneck link with capacity ,
with all other links with capacities strictly higher than . The and for all
links have xed but possibly different delays. To compare the
friendliness of two MP-TCP algorithms and Theorem 3.5 motivates the following denition of respon-
, suppose that when shares the test network siveness. Given two MP-TCP and , we say that is more
with an SP-TCP, it achieves a throughput of in equilib- responsive than if . Theorem 3.5 (2) implies
rium aggregated over the available paths (the SP-TCP there- that an MP-TCP algorithm with a larger or more nega-
fore attains a throughput of ). Suppose achieves tive denite is more responsive, in the sense that the
a throughput of in equilibrium when it shares the test net- real parts of the eigenvalues of the Jacobian have a smaller,
work with the same SP-TCP. Then, we say that is friendlier more negative upper bound.
(or more TCP-friendly) than if , i.e., if re- Then, the next result suggests an inevitable tradeoff between
ceives no more bandwidth than does when they separately responsiveness and friendliness.
share the test network in Fig. 1 with the same SP-TCP ow. Theorem 3.6 (Tradeoff): Consider two MP-TCP algorithms
From the theory for single-path TCP ( for all ), and with the same gain . Suppose both satisfy
it is known that a design is more TCP-friendly if it has a smaller C1C3 and C5. Then, for all
marginal utility . The same intuition holds for
MP-TCP algorithms even though the utility functions may not
exist for MP-TCP algorithm.
Theorem 3.4 (Friendliness): Consider two MP-TCP algo-
In light of Theorems 3.4 and 3.5, Theorem 3.6 says that a more
rithms and . Suppose both satisfy
responsive MP-TCP design is inevitably less friendly if they
C1C4. Then, is friendlier than if for
have the same .
all .
The theorem is easier to understand in the case of SP-TCP,
i.e., when for all and . Then, it
E. Responsiveness Around Equilibrium implies that a more concave utility function has a larger
Suppose conditions C1C3 hold and there is a unique equi- marginal utility, and hence is less friendly.
librium . Assume all links in are active with
; otherwise remove from all links with prices . F. Window Oscillation
Let . The behavior of (3)(4) around the equi- Window oscillations are inherent in loss-based additive
librium is dened by the linearized system increase multiplicative decrease (AIMD) TCP algorithms. We
close this section by discussing informally why a larger design
(10) generally creates more severe window oscillations.
PENG et al.: MULTIPATH TCP: ANALYSIS, DESIGN, AND IMPLEMENTATION 601

This implies a tradeoff between responsiveness (which is The discussion motivates a new design that generalizes the ex-
enhanced by a large and oscillation (which is reduced isting MP-TCP algorithm.
with a small .
The effect of on window uctuations can be under- A. Implications on Existing Algorithms
stood by studying how it affects the decrease per packet Recall Table I that summarizes the conditions satised by
loss in the following packet-level model. the various algorithms. Only EWTCP and Coupled algorithms
For each ACK on route . satisfy C0. Their equilibrium properties can be studied in the
For each loss on route . standard utility maximization model as done for single-path
Let be an indicator variable of whether a packet TCP. Semicoupled and Max algorithms do not satisfy C0, and
loss is observed on route at an arbitrary time in steady state. therefore analysis through utility maximization is not appli-
Then cable. However, Theorem 4.1 implies that both Semicoupled
and Max algorithms satisfy C1C3 provided they enable no
more than eight routes. Theorems 3.2 and 3.3 then imply
that they have a unique and globally stable equilibrium. It is
also easy to show that EWTCP satises C1C3. The Coupled
represents the expected relative reduction in aggregate algorithm does not satisfy C2 and is found to have multiple
throughput , given that there is at least one equilibria in [5].
packet loss on some route . It is a measure of throughput Next, we discuss friendliness of existing MP-TCP algo-
uctuation for each packet loss that an application experiences. rithms. It can be shown that the corresponding to these
For TCP-NewReno (for which and is a scalar), the algorithms satisfy
window size is halved on each packet loss, ,
and hence .
To understand for MP-TCP algorithms, we need the
following result. for all if all routes have the same round-trip
Lemma 3.1: Let with elements. time. Since all of them satisfy C4, Theorem 3.4 implies that their
Each element is an independent binary random variable with friendliness will be in the same order, i.e., their throughputs in
. Dene the test network of Fig. 1 are ordered as follows:1
. Then

This is conrmed by the Linux-based experiment.

Third, we will discuss responsiveness of existing MP-TCP
algorithms. These algorithms have the same gain function
Suppose each route has a xed loss probability . Then, and
within each RTT, Lemma 3.1 implies

Substituting and from (5),

we get, ignoring the high-order terms Theorem 3.5 then implies that their responsiveness should be in
the same order, as conrmed by our experiments in Section V.
Finally, we discuss window oscillation of existing MP-TCP
(11) algorithms using as the metric. As mentioned in
Section III-F, for TCP NewReno, a bench-
to the rst order. Note that does not affect the equilib- mark single-path TCP algorithm. According to (11), if
rium rates . Hence, with the assumption that are constants, , we have, to the rst order
is determined by the functions in steady state.
Specically, an MP-TCP algorithm with a larger
tends to have a larger and hence more severe window
oscillations. Theorem 3.5, however, suggests that a larger All existing MP-TCP algorithms have the same
also leads to better responsiveness, suggesting an , with strict inequality if and
inevitable tradeoff between responsiveness and window for at least two . Thus, enabling MP-TCP always
oscillation. tends to reduce window oscillation for existing algorithms
compared to TCP NewReno. Moreover, the window oscil-
IV. IMPLICATIONS AND A NEW ALGORITHM lation is always reduced compared to TCP NewReno when
.
In this section, we discuss the implications of these struc-
tural properties on the behavior of existing MP-TCP algorithms. 1When , the MP-TCP source can obtain even smaller throughput than
They are further illustrated in experiment results in Section V. the competing single-path TCP source.
602 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 24, NO. 1, FEBRUARY 2016

TABLE II
HOW DESIGN CHOICES AFFECT MP-TCP PERFORMANCE

B. Generalized Algorithm
Consider the class of algorithms parametrized by as
follows: Fig. 2. Network for our Linux-based experiments on TCP friendliness and re-
sponsiveness, with MP-TCP ows and single-path TCP ows sharing
two links of capacity and propagation delay (single trip) . MP-TCP
(12) ows maintain two routes with rate . Single-path TCP ows maintain one
route with rate .
This class includes the Max , Coupled
, and Semicoupled
algorithms as special cases when all RTTs on different paths of by subpaths with large throughput. The parameter in the
the same source are the same, i.e., . generalized algorithm (12) scales in the right way: A
The next result characterizes a subclass that has a unique and path that has a large has and hence a
locally stable equilibrium point. similar degree of window oscillation as existing algorithms,
Theorem 4.1: Fix any and . For any , while a path with a small has larger than that under
the in (12) satises the following: a design with zero and therefore is more responsive.
1) C1 if ; Our experiments show that Max algorithm
2) C2 and C3 if and are the same for overtakes too much of the competing single-path
all (assuming has full row rank). TCP ows. Hence, we can only use a smaller since is
The requirement that is not restrictive since in practice already innite in order to improve friendliness. To compensate
a device may typically enable no more than three paths. The the responsiveness performance, we will use a larger , which
requirement that are the same for all is used in proving will sacrice window oscillation performance. The Balia
the negative deniteness of the (symmetric part of the) Jacobian MP-TCP algorithm given at the end of Section I corresponds
of . Since a negative denite matrix remains negative to the choice . Instead of allowing the
denite after small enough perturbations of its entries, Theorem window size to drop to 1 for a packet loss, we add a cap for the
4.1 holds if the RTTs of the subpaths do not differ much. This decrement of window size, which improves the performance
(sufcient) condition seems reasonable as two paths between according to our experiments. Note that there is no best
the same sourcedestination pair often have similar RTTs if both parameter settings since there are tradeoffs among all the
are wireline paths. Note that our experiments in Section V show performance metrics and we choose
that the algorithm also converges even if the RTTs on different based on our experiments in Section V, which show that this
paths differ dramatically, e.g., the RTT of WiFi is usually much parameter setting strikes a good balance among responsiveness,
smaller than that of 3G. friendliness, and window oscillation.
For the class of algorithms specied by (12), Theorem 4.1
motivates a design space dened by , V. EXPERIMENT
where and control the tradeoff between friendliness and re- In this section, we summarize our experimental results that
sponsiveness and controls the tradeoff between responsive- illustrate the above analysis. In addition to the MP-TCP algo-
ness and window oscillation. In Table II, we summarize how rithms illustrated in Section II-B, we also include the recently
the parameters affect the performance. developed OLIA MP-TCP algorithm [8]. We evaluate the
We now describe our design philosophy. As discussed above MP-TCP algorithms using a reference Linux implementation
the design of MP-TCP algorithms involves inevitable trade- of MP-TCP, Multipath TCP v0.88 [13]. Since it currently in-
offs among responsiveness, friendliness, and the severity of cludes only Max and OLIA algorithms, we implement EWTCP,
window oscillation. Specically, a design is more responsive Semicoupled, Coupled, and the proposed Balia algorithm in the
if it has a higher gain or a more negative denite Jacobian reference implementation. For the Coupled and our algorithms,
(Theorem 3.5). However, a larger usually the minimum ssthresh is set to 1 instead of 2 when more than
creates a bigger window oscillation; a more negative denite one path is available.
implies a larger , usually hurting friendliness The network topology is shown in Fig. 2. In the testbed, all
(Theorems 3.6 and 3.4). This is summarized in Table II. Since nodes are Linux machines with a quad-core Intel i5 3.33 GHz
enabling multiple paths already reduces window oscillation processor, 4 GB RAM, and multiple 1 Gbps Ethernet inter-
compared to single-path TCP (Section IV-A), MP-TCP can faces, running Ubuntu 13.10 (Linux kernel 3.11.8). The net-
afford to use a relatively large gain for responsiveness. This work parameters such as , and are controlled by
does not compromise too much on window oscillation, but Dummynet [14].
allows us to use a less negative denite Jacobian Our experiments are divided into three parts. First, we com-
with a smaller to maintain sufcient TCP friendliness. pare TCP friendliness of Balia algorithm and prior algorithms.
Moreover, responsiveness is mainly affected by subpaths with The result conrms that the Couple algorithm is the friendliest,
small throughput, while window oscillation is mainly affected and Balia algorithm is close to the Coupled algorithm and
PENG et al.: MULTIPATH TCP: ANALYSIS, DESIGN, AND IMPLEMENTATION
TABLE III
TCP FRIENDLINESS (SAME RTTS): AVERAGE THROUGHPUT (Mb/s) AND
95% CONFIDENCE INTERVAL OF MP-TCP AND SINGLE-PATH TCP USERS
( ms, Mb/s, AND ) Mb/s,
friendlier than the other algorithms. Second, we compare the
responsiveness of each algorithm in a dynamic environment
where ows come and go. The result shows that the Coupled
and OLIA algorithms are unresponsive (illustrating the tradeoff
between responsiveness and friendliness). EWTCP is the most
responsive; Balia is similar in responsiveness but friendlier
to single-path TCP ows. Finally, we show that all MP-TCP
algorithms have smaller average window oscillations than
single-path TCP.
These experiments conrm our analytical results and suggest
our design choice strikes a good balance among friendliness,
responsiveness, and window oscillation.
A. TCP Friendliness
We study TCP friendliness of each algorithm, rst with paths
of similar RTTs and then with paths of different RTTs, which
emulates the wireless scenario. We assume all the ows are
long-lived and focus on the steady-state throughput.
In the rst set of experiments, we let ms,
Mb/s, and . We repeat the experi-
ments 20 times. The average aggregate throughput of MP-TCP
and single-path TCP users and the 95% margin of error for con-
fidence interval (CI) are shown in Table III. The Coupled algo-
rithm is the friendliest, and Balia algorithm is closer to Coupled
algorithm than the others.
In the second set of experiments, we assume highly heteroge-
neous RTTs by emulating the scenario of a mobile device with
both 3G and WiFi access. WiFi access usually has higher ca-
pacity and lower delay compared to 3G. Specicially, we set
ms, Mb/s for the rst link to emulate WiFi
access and ms, Mb/s for the second link to
emulate 3G access. When there exists single-path TCP ows,
i.e., , the behaviors of all the algorithms are similar to
the equal-RTT case in the rst set of simulation. The Coupled
algorithm is the friendliest, and Balia algorithm is closer than
other algorithms. However, when there is no single-path TCP
ow, i.e., and , the performance of OLIA is
not stable to effectively take all the available capacity, while the
other algorithms do not have such a problem. We repeat the ex-
periments 20 times, and we nd sometimes OLIA does not use
the 3G access link. The average throughput of MP-TCP user
and the 95% margin of error for condence interval is shown in
Table IV.
B. Responsiveness
We use the network in Fig. 2 with Mb/s,
ms, and . To demonstrate
603
TABLE IV
BASIC BEHAVIOR (WIFI/3G): THROUGHPUT (Mb/s) OF AN MP-TCP USER AND
95% CONFIDENCE INTERVAL ( ms, ms, Mb/s,
)
TABLE V
RESPONSIVENESS: CONVERGENCE TIME (S) OF MP-TCP AND
TOTAL THROUGHPUT (Mb/s) OF ALL SINGLE-PATH TCP USERS
( ms, Mb/s, AND )
the dynamic performance of each algorithm, we assume the
MP-TCP ow is long-lived while the single-path TCP ows
start at 40 s and end at 80 s. We record the aggregate throughput
of the single-path TCP ows from 40 to 80 s, which measures
the friendliness of MP-TCP. We also measure the time for the
congestion window on the second path to recover2 of MP-TCP
users. It measures the responsiveness of MP-TCP. These mea-
surements are shown in Table V, and the congestion window
and throughput trajectories of all algorithms are shown in Fig. 3.
To clearly show the responsiveness performance, we record the
longest convergence time found in our experiment in Table V,
and the corresponding trajectories are shown in Fig. 3.
EWTCP is the most responsive among all the algorithms.
Ours is as responsive as the Max algorithm, yet signicantly
friendlier than EWTCP. Both Coupled and OLIA algorithms
take an excessively long time to recover. For Coupled algo-
rithm, the excessively slow recovery of the congestion window
on the second path (see Fig. 3) is due to the design that in-
creases the window roughly by on each ACK
assuming the RTTs are similar. After the single-path TCP ow
has left, is small while is large, so that
is very small. It therefore takes a long time for to increase
to its steady-state value. In general, under the Coupled algo-
rithm, a route with a large throughput can greatly suppress the
throughput on another route even though the other route is un-
derutilized. The reason of the poor responsiveness performance
of OLIA can be explained using a similar argument as Cou-
pled algorithm since they have the same increment/decrement
for each ACK/loss in this scenario.
C. Window Oscillation
We use a single-link network model to compare window os-
cillation under MP-TCP and single-path TCP. First, an MP-TCP
ow initiates two subpaths through that link, and we measure
the window size of each subpath and their aggregate window
size. Then, a TCP-Reno ow traverses the same link and we
measure its window size. The results are shown in Fig. 4 for our
algorithm in comparison to single-path TCP (other MP-TCP al-
gorithms have a similar behavior). They conrm that enabling
2Dened as the rst time the congestion window on the second path reaches
the average congestion window (e.g., 60) after the single-path users have left.
604 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 24, NO. 1, FEBRUARY 2016

Fig. 4. Window oscillation: The red trajectories represent throughput uctu-

ations experienced by the application in the case of MP-TCP and the case of
single-path TCP.

VI. CONCLUSION
We have presented a model for MP-TCP and identied
designs that guarantee the existence, uniqueness and stability
of the network equilibrium. We have characterized the de-
sign space and studied the tradeoff among TCP friendliness,
responsiveness, and window oscillation. We have proposed
Balia MP-TCP algorithm that generalizes existing algorithms
and strikes a good balance among these properties. We have
implemented Balia in the Linux kernel and used it to evaluate
the performance of our algorithm.

APPENDIX A
PROOF OF THEOREM 3.1 (UTILITY MAXIMIZATION)
The Lagrangian of (9) is

where are the dual variables and .

Then, the dual problem is

where . The KKT condition

implies that, at optimality, we have

and
(13)
and (14)

Compared to (6)(7), we conclude that if an MP-TCP algorithm

dened by (3)(4) has an underlying utility function , then we
must have

Fig. 3. Responsiveness performance: congestion window trajectory of (15)

MP-TCP for each path (left column). SP-TCP starts at time 40 s and ends at
80 s. The throughput of SP-TCP and total throughput of MP-TCP are shown in Given , (15) has continuously differentiable solutions
the right column. Parameters: ms, Mb/s, and
. if and only if the Jacobian of is symmetric, i.e.,
if and only if
multiple paths reduces the average window oscillation com-
pared to only using a single path.
PENG et al.: MULTIPATH TCP: ANALYSIS, DESIGN, AND IMPLEMENTATION 605

APPENDIX B
PROOF OF THEOREM 3.2 (EXISTENCE AND UNIQUENESS)

A. Proof of Part 1
For any link , let Lemma B.2: Suppose C3 holds. Then, at equilibrium
for all .
Proof: Suppose . Then, by C3,
whose component composes of all the elements in except .
and hence there is a link with . However, then for
For , let
all paths and hence by C3. This implies
, and hence by (7), contradicting .
Recall the vector notations that
and . According to C1, we have the following and
two facts, which will be used in the proof. . To prove uniqueness of the equilibrium, suppose for
is a nondecreasing function of on since the sake of contradiction that there are two distinct equilibrium
is a nonincreasing function of . points and . By Lemma B.2, we have and
since . . Hence, (6) implies and
Next, we will show that is a quasi-concave function . By Lemma B.1 and Assumption C2, we then have
of . In other words, for any xed , the set
is a convex set. If , then

which means is a nonincreasing function of , Hence

hence is a quasi-concave function of and
(18)
(16) Equilibrium condition (7) implies
On the other hand, if , then there exists a and (19)
such that since is continuous
and (20)
and . Note that is a
nondecreasing function of , then is nondecreasing Substituting (19) into (18) yields
for and nonincreasing for . Hence,
is also a quasi-concave function of in this case
and
However, (20) implies that the left-hand side of the last in-
(17) equality is nonnegative (since ), a contradiction.
By Nash theorem, if is a quasi-concave function of Hence, the equilibrium is unique.
for all and is in a bounded set, then there exists a
such that APPENDIX C
PROOF OF THEOREM 3.3 (STABILITY)
We will construct a Lyapunov function and use LaSalle's in-
According to (16) and (17), for any , either or variance principle [15] to prove global asymptotic stability of
but not both holds at any time. Therefore, satis- the unique equilibrium point . Dene
es (7). Since , there exists an to (6). Hence, there . Consider the candidate Lyapunov function
exists at least one solution that satises (6) and (7).
(21)
B. Proof of Part 2
Lemma B.1: Assume a function is continu-
ously differentiable and is negative denite for all . By denition, for all and
Then, for any if . Furthermore, is radially
unbounded, i.e., as . Finally

Proof: Fix any . Dene

. Since is continuous, there exists a
such that the eigenvalues of over the compact If , then we have (since )
set . Then
606 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 24, NO. 1, FEBRUARY 2016

The rst inequality holds since where the inverse exists by Condition C2. C2 also guaran-
if and if . The tees the inverse of , denoted by ; C4 ensures
last equality holds since by Lemma B.2 and (6). . Let
Hence
and

Then
where the last inequality holds since
by Lemma B.1 and Assumption C2. Similarly

Thus
where the last inequality holds since by the equi-
librium condition (7). Hence
(22)

Therefore, if ,then By matrix inverse formula

and if , then . This means and

is indeed a Lyapunov function.
Consider the set
for all Substitute it into (22), we have
of trajectories on which . If the only trajectory in is the
trivial trajectory , then LaSalle's invariance
principle implies that is globally asymptotically stable.
We now show that this is indeed the case.
As shown above, implies , i.e., any trajectory
in must have for all . This
means , and hence, for all since
by Lemma B.2. That is, for all
and hence since has full row rank by C3.
Therefore, is indeed the only trajectory in . where the inequality follows because is negative denite,
This completes the proof of Theorem 3.3. and . Thus, we have
for , i.e., the aggregate throughput of the
APPENDIX D MP-TCP over its available paths is increasing in . This means
PROOF OF THEOREM 3.4 (FRIENDLINESS) (corresponding to ) will attain a higher throughput
than (corresponding to ) when separately sharing the
Let the MP-TCP source be dened by
test network in Fig. 1 with the same SP-TCP.

Algorithm and corresponds to and , re-

spectively. Let and be the throughput and RTT of the APPENDIX E
TCP NewReno source in Fig. 1. The equilibrium is dened by PROOF OF THEOREM 3.5 (RESPONSIVENESS)
, where , and is given by
A. Proof of Part 1
Fix any eigenvalue of . Let be the
corresponding eigenvector with . Then, we have
where the rst equation follows from

Hence
and is the congestion price at the bottleneck link. Applying
the implicit function theorem, we get

Premultiplying on both sides, we have

PENG et al.: MULTIPATH TCP: ANALYSIS, DESIGN, AND IMPLEMENTATION 607

The denominator is real and positive, and When , it is easy to show that satises C1 and
in the numerator is imaginary. Hence is negative semidenite under the conditions of the theorem.
We hence prove the theorem for .

A. Proof of Part 1
Fix any and . Fix any nite such that
for all . Fix any . We now show that there exists
an that satises (6), in particular , in two
where the last inequality holds because the numerator is nega- steps.
tive by Condition C2 and the denominator is positive. Since this First, there exists an that satises if and only
holds for all eigenvalues of , the linearized system (10) is if
stable. Moreover, as desired.

B. Proof of Part 2
(27)
Consider two MP-TCP algorithms and such
that which is equivalent to

and for all (28)

For any (nonzero) , we have Since this holds for all , we have

(23)

(24)
(29)
Hence, .
Clearly, as . Let
APPENDIX F (30)
PROOF OF THEOREM 3.6 (TRADEOFF)
Fix an . Let and Then, since for all by assumption. Moreover,
. Suppose for the sake for all and hence
of contradiction that but
does not hold, i.e., there exists a nite
and a such that

(25) where is a minimizing in (30). Since is continuous,

Since by assumption, a trivial modication there exists an with . Moreover, such a
of Lemma B.1 shows that, for all is unique since is strictly decreasing.
, i.e., Finally, consider the set of with . All such
satisfy (28) with
(26)
(31)
Choose an as follows: For all , choose and
then use condition C5 to choose an large enough so that However, , implying
and . With this , (26) becomes

In summary, given any nite such that for all , a

solution to (28) is uniquely given by
where the last inequality follows from (25). This is a contradic-
tion, and hence . (32)

where
APPENDIX G
PROOF OF THEOREM 4.1
We will show the results hold for any . Since
, the results also hold for . and is the unique value at which .
608 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 24, NO. 1, FEBRUARY 2016

We now prove the other conditions in C1 Proof: It is obvious that . To show

, we have

According to (29), we can show that is a decreasing function

of and is an increasing function of for . Thus,
is a decreasing function of and is an increasing of
if because . For each , let
, then by denition and (32), we have

Since is a decreasing function of , it is also a de-

creasing function of if . Recall that is also a de-
creasing function of is thus a decreasing function of
, in other words, . Lemma G.2: Let that satises and
On the other hand, as for all paths . Then, for any nonzero , we have
traversing . Then, by (27) for , which shows
.

B. Proof of Part 2
To prove satises C2 and C3 for , we will show provided .
that the Jacobian is negative denite if Proof: Given any , let . It
and are the same for . Other properties then sufces to show that, for every for
of C2 and C3 are easy to prove and we omit the proof. Fix an . Given any , consider
and let , the common round-trip time for all .
Let and (35)

Its Lagrangian is

Then, the Jacobian of at is

where is the Lagrange multiplier. Setting for all

and substituting it into , we obtain the
and it is negative denite for if is positive
unique minimizer given by and .
denite. We now show that this is indeed the case when ,
Then
i.e., for any

(33)

By Lemma G.1, and . Then, (33) fol- Hence, when if . When is

lows from Lemma G.2 provided . Hence, the Jacobian is nonzero but , then from (35).
negative denite.3 The proof of Theorem 4.1 is complete after
Lemmas G.1 and G.2 are proved.
APPENDIX H
To show that it satises C3, it follows directly from (27) that
PROOF OF LEMMA 3.1
if , then . It is also clear from (27) that the
converse holds. This proves C3. By the denition of , we have
Lemma G.1: Fix any integer . Given any ,
dene a vector in as follows:

Then, and .
3If , the Jacobian degenerates to

(34)
where the last equality follows from the independence of
which is merely negative semidenite. and
PENG et al.: MULTIPATH TCP: ANALYSIS, DESIGN, AND IMPLEMENTATION
. Anwar Walid (S89M90SM02F09) received
Thus
REFERENCES
[1] Q. Peng, A. Walid, and S. H. Low, Multipath tcp algorithms: Theory
and design, in Proc. ACM SIGMETRICS/Int. Conf. Meas. Model.
Comput. Syst., 2013, pp. 305316.
[2] A. Ford, C. Raiciu, M. Handley, and O. Bonaventure, TCP extensions
for multipath operation with multiple addresses, IETF MPTCP pro-
posal, 2009.
[3] M. Honda, Y. Nishida, L. Eggert, P. Sarolahti, and H. Tokuda, Mul-
tipath congestion control for shared bottleneck, in Proc. PFLDNeT
Workshop, 2009.
[4] J. R. Iyengar, P. D. Amer, and R. Stewart, Concurrent multipath
transfer using SCTP multihoming over independent end-to-end paths,
IEEE/ACM Trans. Netw., vol. 14, no. 5, pp. 951964, Oct. 2006.
[5] F. Kelly and T. Voice, Stability of end-to-end algorithms for joint
routing and rate control, Comput. Commun. Rev., vol. 35, no. 2, pp.
512, 2005.
[6] H. Han, S. Shakkottai, C. Hollot, R. Srikant, and D. Towsley, Overlay
tcp for multi-path routing and congestion control, in Proc. IMA Work-
shop Meas. Model. Internet, 2004.
[7] D. Wischik, C. Raiciu, A. Greenhalgh, and M. Handley, Design, im-
plementation and evaluation of congestion control for multipath tcp,
in Proc. 8th USENIX Conf. Netw. Syst. Design Implementation, 2011,
pp. 88.
[8] R. Khalili, N. Gast, M. Popovic, and J.-Y. Le Boudec, MPTCP
is not Pareto-optimal: Performance issues and a possible solution,
IEEE/ACM Trans. Netw., vol. 21, no. 5, pp. 16511665, Oct. 2013.
[9] S. Shakkottai and R. Srikant, Network optimization and control,
Found. Trends Netw., vol. 2, no. 3, pp. 271379, 2007.
[10] F. P. Kelly, A. K. Maulloo, and D. K. Tan, Rate control for commu-
nication networks: Shadow prices, proportional fairness and stability,
J. Oper. Res. Soc., vol. 49, no. 3, pp. 237252, 1998.
[11] S. H. Low and D. E. Lapsley, Optimization ow control. I: Basic al-
gorithm and convergence, IEEE/ACM Trans. Netw., vol. 7, no. 6, pp.
861874, Dec. 1999.
[12] S. H. Low, A duality model of TCP and queue management algo-
rithms, IEEE/ACM Trans. Netw., vol. 11, no. 4, pp. 525536, Aug.
2003.
[13] MultiPath TCP Linux implementation, [Online]. Available: http://
multipath-tcp.org
[14] M. Carbone and L. Rizzo, Dummynet revisited, Comput. Commun.
Rev., vol. 40, no. 2, pp. 1220, 2010.
[15] H. K. Khalil, Nonlinear Systems, 2nd ed. Upper Saddle River, NJ,
USA: Prentice-Hall, 1996.
Qiuyu Peng received the B.S. degree in electrical
engineering from Shanghai Jiaotong University,
Shanghai, China, in 2011, and is currently pursuing
the Ph.D. degree in electrical engineering at the
California Institute of Technology, Pasadena, CA,
USA.
His research interests are in the distributed opti-
mization and control for power system and commu-
nication networks.
609
the Ph.D. degree in electrical engineering from Co-
lumbia University, New York, NY, USA, in 1992.
He is a Distinguished Member of Technical Staff
with the Mathematics of Network Systems Research,
Bell Labs, Murray Hill, NJ, USA. He holds 10 patents
on computer and communication networks and sys-
tems. He has contributed to the Internet Engineering
Task Force (IETF) with RFCs.
Dr. Walid is an elected member of the Tau Beta
Pi National Engineering Honor Society and the
IFIP Working Group 7.3. He served as an Associate Editor of the IEEE/ACM
TRANSACTIONS ON NETWORKING. He is an Associate Editor of the IEEE/ACM
TRANSACTIONS ON CLOUD COMPUTING and IEEE Network. He was Co-Chair
of the Technical Program Committee of IEEE INFOCOM 2012. He received
the Best Paper Award from ACM SIGMETRICS, IFIP Performance.
Jaehyun Hwang (M'10) received the B.S. degree
from The Catholic University of Korea, Seoul,
Korea, in 2003, and the M.S. and Ph.D. degrees from
Korea University, Seoul, Korea, in 2005 and 2010,
respectively, all in computer science.
Since 2010, he has been with Bell Labs, Al-
catel-Lucent, Seoul, Korea, as a Member of
Technical Staff. His current research interests include
data center protocols, software-dened networking,
multipath TCP, and HTTP adaptive streaming.
Steven H. Low (S88M92SM99F'08) received
the B.S. degree from Cornell University, Ithaca, NY,
USA, in 1987, and the Ph.D. degree from the Univer-
sity of California, Berkeley, CA, USA, in 1992, both
in electrical engineering.
He is a Professor with the Department of
Computing and Mathematical Sciences and the
Department of Electrical Engineering, California
Institute of Technology (Caltech), Pasadena, CA,
USA. Before that, he was with AT&T Bell Labora-
tories, Murray Hill, NJ, USA, and the University of
Melbourne, Melbourne, Australia.
Prof. Low is on the Technical Advisory Board of Southern California
Edison and was a member of the Networking and Information Technology
Technical Advisory Group for the US President's Council of Advisors on
Science and Technology (PCAST) in 2006. He is a Senior Editor of the
IEEE TRANSACTIONS ON CONTROL OF NETWORK SYSTEMS and the IEEE
TRANSACTIONS ON NETWORK SCIENCE & ENGINEERING, is on the editorial
boards of Foundations and Trends in Networking, Foundations and Trends in
Electric Energy Systems, as well as the Journal on Sustainable Energy, Grids
and Networks. He was a co-recipient of IEEE Best Paper awards, the R&D 100
Award, and an Okawa Foundation Research Grant.