Linear Quadratic Control For Sampled-Data Systems With Stochastic Delays

CONFIDENTIAL. Limited circulation. For review only.
Linear Quadratic Control for Sampled-data Systems

with Stochastic Delays
Masashi Wakaiki1 , Masaki Ogura2 , and Joao P. Hespanha3
Abstract We study optimal control for sampled-data systems with stochastic delays. Assuming that the delays can
be modeled by a Markov chain and can be measured by
controllers, we design a control law that minimizes an infinitehorizon continuous-time quadratic cost function. The resulting
optimal control law can be efficiently computed offline by the
iteration of a certain Riccati difference equation. We also obtain
sufficient conditions in terms of linear matrix inequalities for
stochastic stabilizability and detectability, which are used for
the optimal controller design.
I. I NTRODUCTION
Time-varying delays often appear in networked communication and real-time implementation of control systems, as
surveyed in [1][3]. Since the variation of such delays can
be large, it may not be suitable to use optimal controllers
that are constructed for the case of constant delays, e.g., in
[4, Chapter 6]. One solution to compensate for time-varying
delays is to measure delays by time-stamping messages and
exploit these measurements in the control algorithm. An
example of this scenario is inter-area power systems [5]. Our
goal is to design optimal control laws that employ real-time
information about time-varying delays.
Delay-dependent controllers have been widely used for
systems with time-varying delays, and the stabilization by
such controllers has been studied in [5][13] and many
references therein. The design of linear quadratic (LQ)
controllers has also been developed for scenarios where delay
measurements are available [14][17] and where they are not
[18][22].
Although delay-dependent stabilizing controllers can be
obtained offline, the LQ controllers in the earlier studies [14][16], [18][22] require some online computation.
Furthermore, those LQ controllers are based on simplistic
assumptions that the delays can take only finitely many
values or follow identical and independent and identicallydistributed distribution, but networked-induced delays generally take a continuum of values in a given interval and are
often modeled by a Markov chain (see, e.g., [23][25] for
This material is based upon work supported by the National Science
Foundation under Grant No. CNS-1329650. M. Wakaiki acknowledges
The Telecommunications Advancement Foundation for their support of this
work.
1 Masashi Wakaiki is with the Department of Electrical and Electronic
Engineering, Chiba University, 1-33 Yayoi-cho, Inage-ku, Chiba 263-8522,
Japan (email: wakaiki@chiba-u.jp).
2 Masaki Ogura is with the Department of Electrical and Systems Engineering at the University of Pennsylvania, Philadelphia, PA 19104, USA
(email: ogura@seas.upenn.edu).
3 Jo
ao P. Hespanha is with the Center for Control, Dynamical-systems and
Computation (CCDC), University of California, Santa Barbara, CA 93106,
USA (email: hespanha@ece.ucsb.edu).
the modeling of network-induced delays by a Markov chain).

A notable exception can be found in [17], where LQ controllers can be computed offline and can address continuousvalued delays modeled by a Markov chain. However, in
the controller design of [17], we need to solve a nonlinear
vector integral equation called the Riccati integral equation.
Moreover, the earlier studies for the sampled-data case [14]
[22] have considered discrete-time LQ criteria that take into
account the state and the input only at sampling instants.
In this paper, we study LQ control for sampled-data linear systems and present a computationally efficient method
for finding an optimal control law. Since we consider
continuous-time LQ criteria, the derived control law is optimal including intersample behaviors. To obtain the optimal
controller, we reduce our LQ problem to an LQ problem
for discrete-time Markov jump systems whose jumps are
modeled by a Markov chain taking values in a general Borel
space, as done in [17]. We can therefore obtain the optimal
control law offline and can deal with Markovian delays that
take arbitrary values in a given interval.
Additionally, we show that the LQ controller can be
efficiently computed by the iteration of a Riccati difference
equation, based on the results of [26]. Our results require the
assumption that certain pairs of matrices are stochastically
stabilizable and detectable. To facilitate testing this assumption, we provide novel sufficient conditions for stochastic
stabilizability and detectability in terms of linear matrix
inequalities (LMIs). These conditions are inspired by the
gridding methods for establishing the stability of networked
control systems with aperiodic sampling and time-varying
delays in [7], [27][29].
This paper is organized as follows. In Section II, we
introduce the closed-loop system and basic assumptions,
and then formulate our LQ problem. In Section III, we
reduce our LQ problem to an LQ problem for discrete-time
Markov jump systems, which allows us to apply the general
results in [26] to the reduced LQ problem in Section IV.
Section V is devoted to the derivation of sufficient conditions
for stochastic stabilizability and detectability. We illustrate
the proposed method with a numerical simulation in Section
VI and give concluding remarks in Section VII.
Notation: Let Z+ denote the set of nonnegative integers.
For a complex matrix M , let us denote its complex conjugate
by M . We denote by B(Cn , Cm ) the normed linear space
of all n m complex matrices. For a Borel set M, we
the space of matrix-valued functions P () :
denote by Hn,m
1
M B(Cn , Cm ) that are measurable and integrable in M,
and similarly, by Hn,m
sup the space of matrix-valued functions
Preprint submitted to 2017 American Control Conference.

Received September 12, 2016.

P () : M B(Cn , Cm ) that are measurable and essentially
bounded in M. For simplicity, we will write Hn1 := Hn,n
1 ,
,
Hnsup := Hn,n
sup
n
Hn+
sup := {P Hsup : P (t) 0, -almost everywhere in M},
and
e n,m := {A Hn,m : A(t) is real for all t M}.
H
sup
sup
For a bounded linear operator T on a Banach space, Let
r (T ) denote the spectral radius of T .
II. P ROBLEM S TATEMENT
A. Plant
Consider the following linear continuous-time plant:
c : x(t)
= Ac x(t) + Bc u(t),
x(0) = x0
(1)
where x(t) Rn and u(t) Rm are the state and the input
of the plant. This plant is connected to a controller through a
time-driven sampler with period h > 0 and an event-driven
zero-order hold.
The state x is measured at each sampling time t = kh
(k Z+ ). The controller receives the sampled state x(kh)
with a sensor-to-controller delay k and generates the control
value uk at time t = kh + k . We assume that the lengths
of the delays are known to the controller at the time the
sample arrives. One way to measure the delays is to mark
every output of the sampler with a time-stamp. The zeroorder hold transforms the discrete-time control input uk to
the continuous-time control input
(
u1 0 t < 0
u(t) =
uk
kh + k t < (k + 1)h + k+1 , k Z+ .
(2)
where u1 is a known deterministic initial condition for
the zero-order hold. In (2), we have ignored controller-toactuator delays. However, if the controller-to-actuator delays
ca are constant (see, e.g., [1, Section 2.3.2] for this situation)
and if we consider the total delays k + ca instead of the
sensor-to-controller delays k , then we can use the proposed
method in the presence of the controller-to-actuator delays.
B. Problem Formulation
Throughout this paper, we fix the probability space
(, F, P ) and assume that the delays {k : k Z+ } is
modeled by a continuous-state Markov chain.
Assumption 2.1 (Delays modeled by Markov chains):
The delay sequence {k : k Z+ } is modeled by a timehomogeneous Markov chain taking values in an interval
M := [min , max ] [0, h) and having transition probability
kernel G(|) with a density g(|) with respect to a Borel
measure on M, so that for every kR Z+ and every Borel
set B of M, G(k+1 B|k = ) = B g(t| )(dt).
See Remark 3.5 below for the discussion of the case where
delays can be larger than the sampling period.
We assume that the pair of the initial state and delay
(x0 , 0 ) has a distribution
b on Rn M. Define
bM by
n
bM (B) :=
b(R , B) for all Borel sets B of M. We place
the following mild assumption about the initial distribution
b:
Assumption 2.2 (Initial distribution): The initial distribution
b of (x0 , 0 ) satisfies A1) E(kx0 k2 ) < and A2)
bM is absolutely continuous with respect to .

The absolute continuity assumption guarantees the existence
of the Radon-Nikodym derivative of
bM .
Let {Fk : k Z+ } denote a filtration, where Fk
represents the -field generated by {x(0), 0 , . . . , x(kh), k }.
Set Uc as the class of control inputs u = {uk : k Z+ } such
that uk is Fk measurable and the system (1) and (2) is mean
square stable in the sense that E(kx(t)k2 ) 0 as t
and E(kuk k2 ) 0 as k for every initial distribution
b satisfying Assumption 2.2. For all u Uc , we consider

the infinite-horizon continuous-time quadratic cost function
Jc defined by
Z

Jc (b
, u) := E
x(t) Qc x(t) + u(t) Rc u(t)dt , (3)
0
where Qc 0 and Rc > 0 are weighting matrices. The

, u).
optimal cost is defined as inf uUc Jc (b
In this paper, we study the following LQ problem:
Problem 2.3: Consider the system (1) and (2), and let
Assumptions 2.1 and 2.2 hold. Find an optimal control law
, u) for
uopt Uc achieving Jc (b
, uopt ) = inf uUc Jc (b
every distribution
b on (x0 , 0 ).
III. R EDUCTION TO D ISCRETE - TIME LQ P ROBLEM
In this section, we transform Problem 2.3 to the LQ
problem of discrete-time Markov jump linear systems.
Defining

x(kh)
,
(4)
k :=
uk1
this stochastic process can be viewed as the state of the
following stochastic discrete-time linear system
d : k+1 = A(k )k + B(k )uk ,
(5)
where

( )
(6)
B( ) :=
I
Z h
Z h
eAc s Bc ds.
eAc s Bc ds, ( ) :=
Ad := eAc h , Bd :=

Ad
A( ) :=
0

Bd ( )
,
0
This discretized system is widely used for the analysis of

time-delay systems, e.g., in [7], [14][17], [28], [29].
We denote by Ud the discrete-time counter-part Uc , defined
as the class of control inputs {uk : k Z+ } such that
uk is Fk measurable and E(kk k2 ) 0 as k for
every initial distribution
b satisfying Assumption 2.2. The
following result estabilishes that Ud = Uc :
Lemma 3.1: Consider the plant c and its discretization
d . We have that
lim E(kk k2 ) = 0
lim E(kx(t)k2 ) = 0 and

lim E(kuk k2 ) = 0.

Proof: The statement that limt E(kx(t)k2 ) = 0 and
limk E(kuk k2 ) = 0 imply limk E(kk k2 ) = 0,
follows directly from the definition of k . To prove the
converse statement, we note that for the system dynamics
(1) and (2), there exist constants M1 , M2 , M3 > 0 such that
for all [0, h),
kx(kh + )k M1 kx(kh)k + M2 kuk1 k + M3 kuk k.
By Youngs inequality, this leads to
kx(kh + )k2 N1 kx(kh)k2 + N2 kuk1 k2 + N3 kuk k2
for appropriate constants N1 , N2 , N3 > 0. It then follows
that if E(kk k2 ) 0 as k , then E(kx(t)k2 ) 0 as
t . which completes the proof.
The cost Jc in (3) can be expressed
as the following
P
(discrete-time) summation Jc = k=0 Jk , where
Z (k+1)h
x(t) Qc x(t) + u(t) Rc u(t)dt.
Jk :=
kh
The next lemma shows that each Jk is a quadratic form on

the state and input of the discrete-time system d in (4).
Lemma 3.2: For every k 0, we have

Q(k ) W (k ) k
,
(7)
Jk = k
?
R(k ) uk
uk
where the matrices Q, W , R are defined by

Q11 ( ) Q12 ( )
Q( ) :=
?
Q22 ( )

Z h
() Qc (, )
d
W ( ) :=
() Qc (, ) (, ) Qc (, )
Z h
(, ) Qc (, )d
R( ) := (h )Rc +
Z h
() Qc ()d
Q11 ( ) :=
0
Z h
Z h
() Qc (, )d
() Qc ()d
Q12 ( ) :=
0
Z h
() Qc ()d
Q22 ( ) := Rc +
0
Z h
(, ) Qc (, )d
+
Z h
(() Qc (, ) + (, ) Qc ()) d
and the functions , , are defined by

Z
eAc s Bc ds
() := eAc , () :=
0
Z
eAc s Bc ds.
(, ) :=
0
and we have
x(kh + ) Qc x(kh + ) + u(kh + ) Rc u(kh + )
= xk () Qc ()xk + 2xk () Qc ()uk1
+ uk1 () Qc ()uk1 + uk1 Rc uk1 .
On the other hand, if k < h, then
x(kh + ) = ()xk + (() (, ))uk1 + (, )uk .
Hence
x(kh + ) Qc x(kh + ) + u(kh + ) Rc u(kh + )
= xk () Qc ()xk + 2xk () Qc (()
(k , ))uk1 + 2xk () Qc (k , )uk
+ uk1 (() (k , )) Qc (() (k , ))uk1
+ 2uk1 (() (k , )) Qc (k , )uk
+ uk (k , ) Qc (k , )uk + uk Rc uk .
Substituting these equations into
Z kh+k
Jk =
(x(t) Qc x(t) + u(t) Rc u(t)) dt
kh
Z (k+1)h
(x(t) Qc x(t) + u(t) Rc u(t)) dt,
+
kh+k
we obtain the desired expression (7).

As in the linear time-invariant case [30, Section 3.4], we
can remove the cross term in (7) by transforming the input
uk into u
k as
u
k = uk + R(k )1 W (k ) k .
Under this transformation, we can obtain the next lemma.
Lemma 3.3: Assume that the weighting matrix in
Lemma 3.2 satisfies

Q( ) W ( )
0, M.
(8)
R( )
Define
) := A( ) B( )R( )1 W ( ) ,
A(
(9)
and let C( ) and D( ) be obtained from the following

Cholesky decomposition:
C( ) C( ) = Q( ) W ( )R( )1 W ( )
D( ) D( ) = R( ).
(10)
Consider the following discrete-time Markov jump system

(11) and LQ cost (12):
k )k + B(k )
k+1 = A(
uk
(11)
and
Jd (b
, u
) :=

E kC(k )k k2 + kD(k )
uk k2 .
(12)
k=0
Proof: If 0 < k , then

x(kh + ) = ()xk + ()uk1 ,
If u
opt Ud is the optimal control law achieving
, u) for the discrete-time Markov
Jd (b
, uopt ) = inf uUd Jd (b
jump system (11) and the discrete-time LQ cost (12), then


the desired optimal input uopt Uc for the continuous-time
system (1) and the continuous-time LQ cost (3) is given by
1
uopt
=u
opt
W (k ) k
k
k R(k )
(13)
for every k Z+ .
Remark 3.4: If (8), we have strict positive definiteness

Q( ) W ( )
> 0, M = [min , max ]
(14)
R( )
instead of the semidefiniteness, then
We recall a relationship among stochastic stability, the

spectral radius r (L ), and a Lyapunov inequality condition.
Theorem 4.2 ([33]): The following assertions are equivalent:
1) System (15) is stochastically stable.
2) The spectral radius r (LA ) < 1, where LA is as in
(16) above.
3) There exists S Hn+
sup and > 0 such that -almost
everywhere on M
Q( ) W ( )R( )1 W ( ) > 0
S( ) TA (S)( ) I,
from the Schur complement formula. Hence C( ) and D( )

derived from the Cholesky decompositions (10) are continuous with respect to . Moreover, C( ) and D( ) are unique
in the following sense: For all [min , max ], there exist
unique upper triangular matrices C( ) and D( ) with strictly
positive diagonal entries such that (10) holds; see, e.g., [31]
for Cholesky decompositions.
Remark 3.5: When delays are larger than the sampling
period h, we can discretize the system (1) and (2) as in [32].
Under the assumption that certain variables calculated from
the delays can be modeled by a Markov chain, we can extend
the proposed approach to the case with large delays.
Remark 3.6: Although the authors in [14], [17] also
designed LQ optimal controllers by transforming timedelays systems to discrete-time Markov jump systems, only
discrete-time LQ criteria were considered.
IV. LQ CONTROL FOR D ISCRETE - TIME M ARKOV J UMP
S YSTEMS
In this section, we recall the results of [26] for the LQ
problem of discrete-time Markov Jump systems.
First we define stochastic stability for discrete-time
Markov jump linear systems. Consider the following autonomous system
k+1 = A(k )k ,
(15)
e n and the sequence {k : k Z+ } is a timewhere A H

sup
homogeneous Markov chain in a Borel space M. Throughout
this section, we assume that the initial distribution
b of
(0 , 0 ) satisfies the following conditions as in Assumption
2.2: A1) E(k0 k2 ) < and A2)
bM (B) =
b(Rn , B) is
absolutely continuous with respect to .
Definition 4.1 (Stochastic stability, [33]): The
autonomous Markov jump P
linear system (15) is said
to be stochastically stable if k=0 E(kk k2 ) < for any

initial distribution
b satisfying A1) and A2).
Let g(|) be the density function with respect to a Borel
measure on M for the transition of the Markov chain
e n . For all M and all
{k : k Z+ }. Fix H
sup
n
n
V H1 , S Hsup , define the operators L and T by
Z
g( |t)(t)V (t)(t) (dt)
(16)
L (V )( ) :=
M

Z
T (S)( ) := ( )
g(t| )S(t)(dt) ( ). (17)
M
(18)
where TA is as in (17) above.

We next provide the definition of stochastic stabilizability
and stochastic detectability.
Definition 4.3 (Stochastical stabilizability, [26]):
e n and B H
e n,m . We say that (A, B) is
Let A H
sup
sup
e m,n such
stochastically stabilizable if there exists K H
sup
that r (LA+BK ) < 1, where LA+BK is as in (16) above.
In this case, F is said to stochastically stabilize (A, B).
Definition 4.4 (Stochastic detectability, [26]): Let A
e n and C H
e r,n . We say that (C, A) is stochastically
H
sup
sup
e n,r such that r (LA+LC ) <
detectable if there exists L H
sup
1, where LA+LC is as in (16) above.
Define the opeators E, V, K, R as follows: For all M
and all Z Hn+
sup ,
Z
E(Z)( ) :=
Z(t)g(t| )(dt)
M
V(Z)( ) := D( ) D( ) + B( ) E(Z)( )B( )

K(Z)( ) := V(Z)( )1 B( ) E(Z)( )A( )
R(Z)( ) := C( ) C( ) + A( ) E(Z)( )A( )
A( ) E(Z)( )B( )V(Z)( )1 B( ) E(Z)( )A( ).
Using the above operators, we can obtain the optimal control
law for discrete-time Markov jump linear systems from the
iteration of a Riccati difference equation.
Theorem 4.5 ([26]): Consider the Markov jump system
B) is stochastically
(11) with LQ cost Jd in (12). If (A,
stabilizable and (C, A) is stochastically detectable, then

e n+ such that S = R(S)
there exists a unique function S H
sup
B). The
and K := K(S) stochastically stabilizes (A,
opt
opt
optimal control input u
k Ud is given by u
k := K(k )k
and achieves
Jd (b
, uopt ) = inf Jd (b
, u) = E(x0 S(0 )x0 )
uUd
for every initial distribution

b satisfying A1) and A2).
e n+ is computed by the following way: For
Moreover, S H
sup
e n+ and the sequence Y = R(Y ), Y = ,
any H
sup
k
k+1
we have that Y0 ( ) S( ) as -almost everywhere
in M.
If the weighting matrix of Jd is positive definite, that is,
if (14) holds, then we do not have to compute the Cholesky
decompositions in (10) for the design of the optimal controller. This follows from the definition of the operators V,
R and Proposition 4.6 below.


Proposition 4.6: Define A and C as in (9) and (10),
respectively. The pair (A, B) is stochastically stabilizable
B) is stochastically stabilizable. Moreif and only if (A,
over, under the positive definiteness of the weighting matrix
is stochastically detectable if and only if
in (14), (C, A)
(C C, A) is stochastically detectable.
e m,n stochastically
KH
Proof: By the definition of A,
sup
e m,n
stabilizes (A, B) if and only if K R1 W H
sup
B).
stochastically stabilizes (A,
As mentioned in Remark 3.4, C is invertible if (14)
is stochastically detectable with
holds. Hence if (C, A)
n,r
e
is also stochastically
observer gain L Hsup , then (C C, A)
1
detectable with observer gain L(C ) , and vice versa.
V. S UFFICIENT CONDITIONS FOR S TOCHASTIC
S TABILIZABILITY AND D ETECTABILITY
From the results of Sections 3 and 4, we can obtain
the optimal controller under the assumption that (A, B) in
in Lemmas 3.2
(6) is stochastically stabilizable and (C, A)
and 3.3 is stochastically detectable. The major difficulty
in this controller design is to check the stabilizability and
detectability. In this section, we provide sufficient conditions
for stochastic stabilizability and detectability in terms of
LMIs. To this end, we use the following technical result:
Lemma 5.1 ([34]): For all square matrix U and all positive definite matrix S,
U S 1 U U + U S.
For simplicity, we assume that {k : k Z+ } is a timehomogeneous Markov chain taking values in the interval
M = [min , max ]. We can easily extend the results here to
more general Borel spaces M such as bounded sets in R` .
Let g(|) be the density function with respect to a Borel
measure on M for the transition of the Markov chain
{k : k Z+ }.
A. Stochastic Stabilizability
We first study the stochastic stabilizability of the pair A
e n and B H
e n,m . In our LQ problem, we need to check
H
sup
sup
the stochastic stabilizability of the pair (A, B) defined by
(6).
e m,n
Consider the piecewise-constant feedback gain K H
sup
defined by
K( ) := Ki R
m,n
[si , si+1 ),
(19)
Also for each i = 1, . . . , N , define

sm
i :=
min =: s1 < s2 < < sN +1 := max .
(20)
We provide a sufficient condition for the feedback gain

K in (19) to stochastically stabilize (A, B), based on the
gridding approach developed, e.g., in [7], [27][29].
Theorem 5.2: Define
Z
g(t| )dt 0, [min , max ].
i ( ) :=
(21)
and
p
p
m
m
1 (sm
1 (sm
i )A(si )
i )B(si )
..
..
FB,i :=
.
.
.
p
p
m
m
m
m
N (si )A(si )
N (si )B(si )

FA,i
Take a scalar i > 0 such that for all [si , si+1 ),

p

p

1 ( )A( )
1 ( )B( )

..
..

FA,i FB,i i .
.
.

p
p

N ( )A( )
N ( )B( )
(22)
If there exist positive definite matrices Ri > 0, (not neces i , and scalars i > 0 such
sarily symmetric) matrices Ui , K
that the following LMIs are feasible for all i = 1 . . . , N :

F i U K
Ui +Ui Ri 0 Ui FA,i
+K
i
i
i B,i
?
i I
i I
0
> 0,
?
?
R
0
?
?
?
i I
(23)
where R := diag(R1 , . . . , RN ), the pair (A, B) is stochasti i U 1
cally stabilizable and the controller (19) with Ki := K
i
stochastically stabilizes (A, B).
Proof: We employ a piecewise-constant matrix function
S for the Lyapunov inequality (18). Define
S( ) := Si ,
[si , si+1 )
(24)
e n+ , and we have
with Si > 0. By construction, S H
sup
Z
g(t| )S(t)dt =
M
N
X
i ( )Si .
(25)
i=1
R
Since M g(t| )dt = 1 for all M, it follows that {i }N
i=1
satisfies
N
X
i ( ) = 1
i=1
for all M.
Note that
p
N
X
` ( )S` =
p
`=1
where
si + si+1
2
1 ( )I
S1
..
N ( )I
0
..
1 ( )I
..
.
.
p
SN
N ( )I
Using the Schur complement formula, we see from Theorem

4.2 and (25) that K stochastically stabilizes (A, B) if

p
1 ( )I
..
S( ) I (A( ) + B( )K( ))
S 0
.
( )I
N
S
(26)
[si ,si+1 )

Ui + Ui
?
?
?
0
i I
?
?
?
?
?
i FC,i
Ui FA,i + L
i I
Si
?
?
?
?

i
A,i Ui L
0
0
i I
?
?
?
0
0
0
0
i I
?
?
F,i S
0
0
0
,i S
S
?
i I
0
0
> 0,
0
0
i I
where S := diag(S1 , . . . , SN ).
($)
for some > 0, where S := diag(S1 , . . . , SN ). We obtain
p
1 ( )I
..
(A( ) + B( )K( ))
p .
N ( )I
#
"p
p

1 ( )A( )
N ( )A( )
p
p
.
= I K( )
1 ( )B( )
N ( )B( )
that i Hi in i has the product of the variables i and

Note
. However, from the similarity transformation
Ui K
i
with diag(I, 1/i I, I, 1/i I), we see that i is similar to
Ui + Ui Ri
0
Gi Hi
?
i I i I
0
?
?
R
0
?
?
? i I
for all
[si , s
i+1 ), there exists
Moreover, by (22),
A B RnN,n+m with A B < 1 such that
"p
#
p

1 ( )A( )
N ( )A( )
FA,i
A
p
p
+
.
=
i
FB,i
B
1 ( )B( )
N ( )B( )
in which i := 1/i and the variables appear in a linear

form. Thus if the LMI (23) is feasible, then the controller
i U 1 stochastically stabilizes (A, B).
(19) with Ki := K
i
The controller obtained in Theorem 5.2 is assumed to
know to which interval [si , si+1 ) the delay k belongs for
each k Z+ . For the case when no information about delays
can be obtained, we present the design of delay-independent
stabilizing controllers.
Corollary 5.3: Under the same hypothesis in Theorem 5.2, if there exist positive definite matrices Ri > 0, (not
and scalars i > 0
necessarily symmetric) matrices U, K,
such that the following LMIs are feasible for all i = 1 . . . , N :

F
K
U +U Ri 0 U FA,i
+K
B,i i U
?
i I
i I
0
> 0,
?
?
R
0
?
?
?
i I
Hence the matrix inequality (26) is feasible for all M

if

FA,i
A
I
K
S
+
S
i
i
i
FB,i
B
>0
?
S

for every i = 1 . . . , N
and for every A B RnN,n+m
satisfying A B < 1.
Let Ui be a nonsingular matrix. Defining Ri := Si1 and
Ki := Ki Ui , we have from Lemma 5.1 that

FA,i
Ui
0 Si I Ki
+ i A S
B,i
B
0 S 1
S

Ui
0
Ui + Ui Ri Gi + Hi
0 S 1
R
where R = diag(R1 , . . . , RN ) and the matrices , Gi , and
Hi are defined by

Ui
U
:= A B , Gi := FA,i FB,i , Hi := i i .
Ki
Ki
Since kk < 1, it follows that for every i > 0, we have
i Hi (I )Hi 0. Moreover,

Ui + Ui Ri Gi + Hi
H (I )Hi 0
i i
?
R
?
0
= Vi i Vi ,
where
I
Hi
Vi :=
0
Hi
0
Ui +Ui Ri
0
?
, i :=
I
?
0
?
0
i I
?
?
Gi
I
R
?
i Hi
0
.
0
i I
where R := diag(R1 , . . . , RN ), then the delay-independent

1 stochastically stabilizes (A, B).
controller K := KU
Remark 5.4: For the stabilization of discrete-time systems with continuous-valued Markovian jumping parameters,
we here approximate the function S of the Lyapunov inequality (18) by a piecewise constant function; see (24). Another
approach is to approximate the system
{(A( ), B( )) : [min , max ]}
by
{(A(si ), B(si )) : min = s1 < < sN +1 = max }
and then consider the stabilization of discrete-time systems
that have discrete-valued Markovian jumping parameters but
time-varying uncertainty. Although Markov jump systems
with static uncertainty have extensively been studied, e.g.,
in [35], [36], relatively little work has been done on Markov
jump systems with time-varying uncertainty. Therefore we
do not proceed along this line.


B. Stochastic Detectability
0.12
0.12
0.1
0.08
(t)
Next we study the stochastic detectability of the pair A

e n and C H
e r,n . In our LQ problem, we need to check
H
sup
sup
or (Q W R1 W , A)
the stochastic detectability of (C, A)

in Lemmas 3.2 and 3.3.
e n,r as the piecewiseDefine the observer gain L H
sup
constant function:
0.06
0.06
0.04
L( ) := Li Rn,r
[si , si+1 ),
(27)
0.02
where the interval [si , si+1 ) is defined as in (20) for each

i = 1, . . . , N . Note that the positions of the variables K, L
are different between A + BK (stabilizability) and A + LC
(detectability). Moreover, unlike the case of countable state
Markov chains (see, e.g., [37]), the duality of stochastic stabilizability and stochastic detectability is not proved yet for the
case of continous-state Markov chains. Hence we cannot use
Theorem 5.2 directly, but the gridding method also provides
a sufficient condition for stochastic detectability in terms of
LMIs. We omit the proof due to space constraints.
Theorem 5.5: For each i = 1, . . . , N , define i , sm
i as in
Theorem 5.2 and

p

p
A(sm
FA,i
i ) , F
:=
1 (sm
N (sm
)I
)I .
,i :=
m
i
i
C(si )
FC,i
Take scalars A,i , ,i > 0 such that for all [si , si+1 ),

A( )
FA,i

A,i
C( )
FC,i
(28)

p
p

1 ( )I
N ( )I F,i ,i .
If there exists positive definite matrices Si > 0, (not nec i , and scalars i , i > 0
essarily symmetric) matrices Ui , L
such that the LMIs in ($) are feasible for all i = 1, . . . , N ,
then (C, A) is stochastically detectable, and the desired
i.
observer gain L is given by (27) with Li := Ui1 L
VI. N UMERICAL E XAMPLE
Consider the unstable batch reactor studied in [38], [39],
where the system matrices Ac and Bc are given by
1.38
0.2077 6.715 5.676
0.5814
4.29
0
0.675
Ac :=
1.067
4.273
6.654 5.893
0.048
4.273
1.343 2.104
0
0
5.679
Bc :=
1.136 3.146 .
1.136
0
This model is widely used as a benchmark example, and for
reasons of commercial security, the data were transformed
by a change of basis and of time scale [38], [39]. We
take the sampling period h = 0.3 and the delay interval
[min , max ] = [0, 0.4h].
The delay sequence {k } is modeled by a Markov chain
with transition probability kernel G satisfying

Fs (s2 ) Fs (s1 )
G(k+1 [s1 , s2 ]k = s) =
Fs (max ) Fs (min )
10
10
15
20
20
25
30
30
Fig. 1: Sample path of {k : 0 k 30} with 0 =

0.5max = 0.2h.
for every closed interval [s1 , s2 ] [min , max ], where Fs (x)

is the probability distribution function of the standard normal
distribution with mean s and standard deviation 1/150.
Fig. 1 illustrates a sample path of the delays {k } with
0 = 0.5max = 0.2h, where (t) is defined by
(t) := k ,
t [kh + k , (k + 1)h + k+1 ).
The weighting matrices Qc , Rc for the state and the input

in (3) are the identity matrices with compatible dimensions.
From Theorem 5.2, we see that (A, B) in (6) is stochastically
stabilizable. Also, from Theorem 5.5, (Q W R1 W , A)

in Lemmas 3.2 and 3.3 is stochastically detectable. Hence it
follows from Theorem 4.5 that we can derive the optimal
controller uopt from the iteration of a Riccati difference
equation.
Time responses are computed for the initial state x(0) =
[1 1 2 3]> and the initial input u1 = [0 0]> .
Fig. 2 depicts 10 sample paths of kx(t)k2 with initial delay
uniformly distributed in the interval [min , max ] = [0, 0.4h].
We observe that if the initial delay 0 is small, the time
response is similar to the response with no delays by the
conventional discrete-time LQ regulator with same weighting
matrices. Since u1 = 0, the system is not controlled
until the controller receives the initial state x(0) at time
t = 0 , which affects the time response as the initial delay
0 is larger. However, after receiving the sampled state, the
optimal controller achieves almost the same decrease rate of
kx(t)k2 .
VII. C ONCLUDING R EMARKS
We presented an efficient method to obtain optimal controllers for sampled-data systems whose delays can be
measured and be modeled by a continuous-state Markov
chain. The proposed controller minimizes an infinite-horizon
continuous-time quadratic cost function and can be efficiently computed offline by the iteration of a Riccati difference equation. To exploit the results on LQ control for
Markov jump systems, we also derived sufficient conditions
for stochastic stabilizability and detectability in terms of

kx(t)k2

35
35
30
30
25
25
20
20
15
15
10
10
Larger initial delay
0.5
1.5
2
2
2.5
Fig. 2: 10 sample paths of kx(t)k2 : The blue lines are the

time responses with initial state x0 = [1 1 2 3]> and initial delay uniformly distributed in the interval [min , max ] =
[0, 0.4h]. The dotted red line is the time response with no
delays, for which we used the (conventional) discrete-time
LQ regulator computed with same weighting matrices.
LMIs. Future work will focus on addressing more general systems by incorporating controller-to-actuator delays,
packet losses, and output feedback.
R EFERENCES
[1] T. C. Yang, Networked control system: a brief survey, IEE Proc.
Control Theory Appl., vol. 153, pp. 403411, 2006.
[2] J. P. Hespanha, P. Naghshtabrizi, and Y. Xu, A survey of recent results
in networked control systems, Proc. IEEE, vol. 95, pp. 138162,
2007.
[3] L. Zhang, H. Gao, and O. Kaynak, Network-induced constraints
in networked control systemsA survey, IEEE Trans. Ind. Inform.,
vol. 9, pp. 406416, 2013.
[4] R. F. Curtain and H. J. Zwart, An Introduction to Infinite-Dimensional
Linear Systems Theory. New York: Springer, 1995.
[5] F. R. P. Safaei, S. G. Ghiocel, J. P. Hespanha, and J. H. Chow, Stability of an adaptive switched controller for power system oscillation
damping using remote synchrophasor signals, in Proc. 53rd IEEE
CDC, 2014.
[6] S. Hirche, C. Chen, and M. Buss, Performance oriented control over
networks: Switching controllers and switched time delay, Asian J.
Control, vol. 10, pp. 2433, 2008.
[7] L. Hetel, J. Daafouz, J.-P. Richard, and M. Jungers, Delay-dependent
sampled-data control based on delay estimates, Systems Control Lett.,
vol. 60, pp. 146150, 2011.
[8] A. Kruszewski, W.-J. Jiang, E. Fridman, and A. Richard, J.P. Toguyeni, A switched system approach to exponential stabilization
through communication network, IEEE Trans. Control Systems Tech.,
vol. 20, pp. 887900, 2012.
[9] L. Zhang, Y. Shi, T. Chen, and B. Huang, A new method for
stabilization of networked control systems with random delays, IEEE
Trans. Automat. Control, vol. 50, pp. 11771181, 2005.
[10] D. Huang and S.-K. Nguang, State feedback control of uncertain
networked control systems with random time delays, IEEE Trans.
Automat. Control, vol. 53, pp. 829834, 2008.
[11] Y. Shi and B. Yu, Output feedback stabilization of networked control
systems with random delays modeled by Markov chains, IEEE Trans.
[12] Y. Sun and S. Qin, Stability analysis and controller design for
networked control systems with random time delay, Int. J. Systems
Science, vol. 42, pp. 359367, 2011.
[13] B. Demirel, C. Briat, and K. Johansson, Deterministic and stochastic approaches to supervisory control design for networked systems
with time-varying communication delays, Nonlinear Anal.: Hybrid
Systems, vol. 10, pp. 94110, 2013.
[14] J. Nilsson and B. Bernhardsson, LQG control over a Markov communication network, in Proc. 36th IEEE CDC, 1997.
[15] J. Nilsson, B. Bernhardsson, and B. Wittenmark, Stochastic analysis
and control of real-time systems with random time delays, Automatica, vol. 34, pp. 5764, 1998.
[16] H. Shousong and Z. Qixin, Stochastic optimal control and analysis of
stability of networked control systems with long delay, Automatica,
vol. 39, pp. 18771884, 2003.
[17] I. Kordonis and G. P. Papavassilopoulos, On sability and LQ control
of MJLS with a Markov chain with general state space, IEEE Trans.
[18] H. Xu, S. Jagannathanm, and F. L. Lewis, Stochastic optimal control
of unknown linear networked control system in the presence of random
delays and packet losses, Automatica, vol. 48, pp. 10171030, 2012.
[19] K. Kobayashi and K. Hiraishi, Modeling and design of networked
control systems using a stochastic switching systems approach, IEEJ
Trans. Electr. Electron. Eng., vol. 9, pp. 5661, 2014.
[20] S. Elvira-Ceja, E. N. Sanchez, and S. Jagannathan, Stochastic inverse
optimal control of unknown linear networked control system in the
presence of random delays and packet losses, in Proc. ACC15, 2015.
[21] Z. Wang, X. Wang, and L. Liu, Stochastic optimal linear control of
wireless networked control systems with delays and packet losses,
IET Control Theory Appl., vol. 10, pp. 742751, 2016.
[22] I. V. Kolmanovsky and T. L. Maizenberg, Optimal control of
continuous-time linear systems with a time-varying, random delay,
Systems Control Lett., 2001.
[23] C. Hermenier, R. Kissling and A. Donner, A delay model for satellite
constellation networks with inter-satellite links, in Proc. IWSSC09,
2009.
[24] A. Ghanaim, G. Borges, and G. Frey, Estimating delays in networked
control systems using colored Petri nets and Markov chain models,
in Proc. 14th IEEE ETFA, 2009.
[25] A. Ghanaim and G. Frey, Markov modeling of delays in networked
automation and control systems using colored Petri net models simulation, in Proc. 18th IFAC WC, 2011.
[26] O. L. V. Costa and D. Z. Figueiredo, LQ control of discrete-time
jump systems with Markov chain in a General Borel space, IEEE
Trans. Automat. Control, vol. 60, pp. 25302535, 2015.
[27] H. Fujioka, A discrete-time approach to stability analysis of systems
with aperiodic sample-and-hold devices, IEEE Trans. Automat. Control, vol. 54, pp. 24402445, 2009.
[28] M. C. F. Donkers, W. P. M. H. Heemels, N. van de Wouw, and L. Hetel,
Stability analysis of networked control systems using a switched
linear systems approach, IEEE Trans. Automat. Control, vol. 56, pp.
21012115, 2011.
[29] S. J. L. M. van Loon, M. C. F. Donkers, N. van de Wouw, and W. P.
M. H. Heemels, Stability analysis of networked and quantized linear
control systems, Nonlinear Anal.: Hybrid Systems, vol. 10, pp. 111
125, 2013.
[30] B. D. O. Anderson and J. Moore, Optimal Control-Linear Quadratic
Methods. Englewood Cliffs: Prentice Hall, 1990.
[31] M. Schatzman, Numerical Analysis: A mathmatical introduction. Oxford : Oxford Univ. Press, 2002.
[32] M. B. G. Cloosterman, N. van de Wouw, W. P. M. H. Heemels, and
H. Nijmeijer, Controller synthesis for networked control systems,
IEEE Trans. Automat. Control, vol. 54, pp. 15751580, 2009.
[33] O. L. V. Costa and D. Z. Figueiredo, Stochastic stability of jump
discrete-time linear systems with Markov chain in a general Borel
space, IEEE Trans. Automat. Control, vol. 59, pp. 223227, 2014.
[34] J. C. Geromel, R. H. Korogui, and J. Bernussou, H2 and H robust
output feedback control for continuous time polytopic systems, IET
Control Theory Appl., vol. 1, pp. 15411549, 2007.
[35] J. Xiong, J. Lam, H. Gao, and D. W. C. Ho, On robust stabilization
of Markovian jump systems with uncertain switching probabilities,
Automatica, vol. 41, pp. 897903, 2005.
[36] C. E. de Souza, Robust stability and stabilization of uncertain
discrete-time Markovian jump linear systems, IEEE Trans. Automat.
Control, vol. 51, pp. 836841, 2006.
[37] O. L. V. Costa, M. D. Fragoso, and R. P. Marques, Discrete-Time
Markovian Jump Linear Systems. London: Springer, 2005.
[38] N. Munro, Design of controllers for open-loop unstable multivariable
system using inverse Nyquiest array, Proc. IEE, vol. 119, pp. 1377
1382, 1972.
[39] H. H. Rosenbrock, Computer-Aided Control System Design. New
York: Academic Press, 1974.


Linear Quadratic Control For Sampled-Data Systems With Stochastic Delays

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Linear Quadratic Control For Sampled-Data Systems With Stochastic Delays

Uploaded by

Copyright:

Available Formats

CONFIDENTIAL. Limited circulation. For review only.