Lectures On Stochastic Control and Nonlinear Filtering: M. H. A. Davis

Lectures on
Stochastic Control and Nonlinear Filtering
By
M. H. A. Davis
Tata Institute of Fundamental Research

Bombay
1984
Lectures on
Stochastic Control and Nonlinear Filtering
By
M. H. A. Davis
Lectures delivered at the

Indian Institute of Science, Bangalore
under the
T.I.F.R.I.I.Sc. Programme in Applications of
Mathematics
Notes by
K. M. Ramachandran
Published for the

Tata Institute of Fundamental Research
Springer-Verlag
Berlin Heidelberg New York Tokyo
1984
Author
M. H. A. Davis
Department of Electrical Engineering
Imperial College of Science and Technology
London SW 7
United Kingdom
c
Tata Institute of Fundamental Research, 1984
ISBN 3-540-13343-7 Springer-Verlag, Berlin. Heidelberg.

New York. Tokyo
ISBN 0-387-13343-7 Springer-Verlag, New York. Heidelberg.
Berlin. Tokyo
No part of this book may be reproduced in any

form by print, microfilm or any other means with-
out written permission from the Tata Institute of
Fundamental Research, Colaba, Bombay 400 005
Printed by M. N. Joshi at The Book Centre Limited,

Sion East, Bombay 400 022 and published by H. Goetze,
Springer-Verlag, Heidelberg, West Germany
Printed in India
Preface
These notes comprise the contents of lectures I gave at the T.I.F.R. Cen-
tre in Bangalore in April/May 1983. There are actually two separate
series of lectures, on controlled stochastic jump processes and nonlin-
ear filtering respectively, and the corresponding two parts of these notes
are almost disjoint. They are united however, by the common philoso-
phy (if that is not too grand a work for it) of treating Markov processes
by methods of stochastic calculus, and I hope the reader will, at least,
be convinced of the usefulness of this and of the extended generator
concept in doing calculations with Markov precesses.
The first part is aimed at developing optimal control theory for a
class of Markov processes called piecewise-deterministic (PD)proce-
sses. These were only isolated rather recently but seen general enough
to include as special cases practically all the non-diffusion continuous
time processes of applied probability. Optimal control for PD processes
occupies a curious position just half way between deterministic and Sto-
chastic optimal control theory in such a way that no standard theory
from either side is adequate to deal with it. The only applicable theory
that exists at all is very recent work of D. Vermes based on the gener-
alized dynamic programming ideas of R.B. Vinter and R.M. Lewis, and
this is what I have attempted to describe here. Undoubtedly, further de-
velopment of control theory for PD processes will be a fruitful field of
enquiry.
Part II concentrates on the pathwise theory of filtering for diffu-
sion processes and on more sophisticated extensions of it due primarily
to H. Kunita. The intriguing point here is to see how stochastic partial
v
vi
differential equations can be dealt with by stochastic flow theory through

what amounts to a doubly stochastic version of the FeynmanKac for-
mula. Using this, Kunita has given an elegant argument to show the
existence of smooth conditional densities under Hormander-type con-
ditions. This is included. Ultimately, it rests on results obtained by
Bismut and others using Malliavin calculus, since one needs a version
of the Hormander theorem which is valid for continuous (rather than
C ) t-dependence of the coefficients. It was unfortunately impossible
to go into such questions in the time available.
I would like to thank Professor K.G. Ramanathan for his kind invita-
tion to visit Bangalore and K.M. Ramachandran for his heroic efforts at
keeping up-to-date notes on a rapidly accumulating number of lectures,
and for preparing the final version of the present text. I would also like
to thank the students and staff of the T.I.F.R. Centre and of the I.I.Sc.
Guest House for their friendly hospitality which made my visit such a
pleasant one.
Contents
I Stochastic Jump Processes and Applications 1
1 Stochastic Jump Processes 3

0 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
1 Martingale Theory for Jump Processes . . . . . . . . . . 4
2 Some Discontinuous Markov Processes . . . . . . . . . 23
2 Optimal Control of pd Processes 45
II Filtering Theory 63
0 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 65
1 Linear and Nonlinear Filtering Equations.... . . . . . . . 65
2 Pathwise Solutions of Differential Equations . . . . . . . 81
3 Pathwise Solution of the Filter Equation . . . . . . . . . 86
Bibliography 101
vii
Part I
Stochastic Jump Processes

and Applications
1
Chapter 1
Stochastic Jump Processes
0 Introduction
Stochastic jump processes are processes with piecewise constant paths. 1

The Poisson process, the processes arising in Inventory problems (stocks
of items in a store with random ordering and replacement) and queuing
systems (arrivals at a queue with each customer having random demand
for service) are examples of stochastic jump processes. Our aim here
is to develop a theory suitable for studying optimal control of such pro-
cesses.
In Section 1, martingale theory and stochastic calculus for jump pro-
cesses are developed. Gnedenko-Kovalenko [16] introduced piecewise-
linear process. As an example of such a process, consider virtual waiting
time process (VWT ) for queueing systems, where VWT (t) is the time
customer arriving at time t would have to wait for service, see Fig. (0.1).
Later Davis [7] and Vermes [25] introduced the concept of piece-
wise deterministic processes which follow smooth curves (not necessar-
ily straight lines) between jumps. In Section 2, we will study some ap-
plications to piecewise-deterministic processes. The idea there is to de-
rive Markov properties, Dynkins formula, infinitesimal generators etc.,
using the calculus developed in Section 1.
3
4 1. Stochastic Jump Processes
Service
Demand
Figure 0.1: Arrival time of customers

2
1 Martingale Theory for Jump Processes

Let (X, S ) be a Borel space.
Definition 1. A jump process is defined by sequences T 1 , T 2 , T 3 , . . .,
Z1 , Z2 , Z3 , . . . of random variables, T i R+ and T i+1 > T i a.s. and
Zi (X, S ). Set
T = lim T k .
k
Let z0 , z be fixed elements of X. Define the path (xt )t0 by

z0 if t < T 1

xt =
Zi if t [T i , T i+1 [

z if t T .
Then the probability structure on the process is determined by either

joint distribution for (T i , Zi , i = 1, 2, . . .) or specifying
3 (i) distribution of (Z1 , T 1 )
(ii) for each k = 1, 2, . . . , conditional distribution of (S k , Zk | T ki , i =

1, 2, . . .), where S k = T k T ki is then k th inter-arrival time.
We will start studying the process (xt ) having a single jump, i.e.,

z0 if t < T ()
xt

Z() if t T ().
1. Martingale Theory for Jump Processes 5
If T = , let Z = z , a fixed point of X.
Figure 1.2:
Define the probability space (, F, P) as the canonical space for T ,

Z,
i.e., ((R+ X)U{(, z )}, B(R+ ) S , {(, z )}, )
where is a probability measure on
((R+ X)U{(, z )}, B(R+ ) S , {(, z )}).
The random function (xt ) generates the increasing family of fields 4

(Ft0 ), i.e.,
Ft0 = {xS , s t}.
We suppose
(([0, ] {z0 })U{0} X) = 0.
This assumption guarantees that the process xt does jump at its jump
time T , i.e.,
P(T > 0 and Z , z0 ) = 1.
Recall that an R+ - valued random variable is a stopping time of a
filtration Ft , if ( t )Ft , t. Let
Ft = Completion of Ft0 with all F

0
null sets .
Proposition 1.1. T is not an Ft0 stopping time, but T is an Ft stopping

time.
Proof. Let A = {Z = z0 } and K be any set in X. Then
x1
S (K) =

([s, ] X)U([o, s] {z0 }) if z0 K and Z(E A) K =

((s, ] X)U([o, s] K) if z0 K and Z(E A) K , .

[o, s] K if z0 < K.
where E = R+ X A.
Clearly [0, t] X cannot be in the - algebra generated by sets of
the above form. So T is not an Ft0 stopping time. Let B = X {z0 }. By
assumption, P(A) = 0; so A Ft .
0
x1
t (B) = [0, t] X A F t .
5 So
[0, t] X Ft .
But {T t} = [0, t] X. Hence T is an Ft stopping time.
It can be seen that

Ft = B[o, t] S U(]t, ] X)U null sets of F .
The stopped - field FT is given by
FT = {G F : G (T t) Ft , t}.
Clearly
FT = F .
Definition 1.2. A process (Mt ) is an Ft -martingale if E | Mt |< and

for s t
E[Mt | F s ] = M s a.s.
(Mt ) is a local Ft - martingale if there exists a sequence of stopping
times S n a.s. such that Mtn := MtS n is a uniformly integrable
martingale for each n; here tS n := Min(t, S n ).
Proposition 1.2. If Mt is a local martingale and S is a stopping time

such that S T a.s., then MS = MT a.s.
Proof. Let S n be stopping times such that S n a.s. Then MtS n is u.i.
martingale. Let Mtn = MtS n , n. Then by optional sampling theorem
E[MSn FT S n ] = MTn ;
but FT = F .
So 6
E[MSn | FT ] = MSn .
Also
lim MTn = MT
n
and lim MSn = MS .
n
So
MS = MT a.s.

Proposition 1.3. Suppose is an Ft -stopping time. Then there exist

t0 R+ such that T = t0 T a.s.
Proof. If is a stopping time, then (T t) Ft , t. But if T is

not constant a.s. on ( T ), then
(T t) (]t, ] X) ]t, [X for some t R+ .

,
But [t, ] X is an atom of Ft . This contradicts the fact that is a

stopping time. So
T = t0 T a.s.
The general definition of a stopped -field is that if U is a stopping
time. Then
FU = {A F | A (U t) Ft , t}.
But this is an implicit definition of the -field.
Exercise 1.1. Suppose = t0 T . Show that
(i) F = Fto
(ii) F = {xs , S 0}.

7
Definition 1.3. For A S , define
F A (t) = ([t, ] A)
and F(t) = F X (t) = P[T > t].
Note that F(0) = 1 and F(.) is monotone decreasing and right con-
tinuous. Define

inf {t : F(t) = 0}
c=
+ if {t : F(t) = 0} = .
Proposition 1.4. Suppose (Mt )t0 is an Ft local martingale. Then
(a) if c = or c < and F(c) = 0, then Mt is a martingale on [0, c[.
(b) if c < , F(c) > 0, then (Mt ) is a uniformly integrable martingale.

Here F(c) = lim F(t).
tc
Proof. (a) If k T a.s. for some k, then
Mtk = Mtk T = MtT = Mt .
So Mt is a u.i. martingale. Hence suppose P[k < T ] > 0 for all

k(); then by Proposition 1.3,
k T = tk T for some fixed tk
8 and tK < c because of (). Also tk c since k .

So
Mtk = Mtk T = MtT tk = Mttk .
Hence Mttk is a u.i. martingale. So (Mt )t<c is a martingale.

(b) c < , F(c ) > 0.

F(c ) = P(T = c)
P(T = c) > 0; so it must be the case that tk = c for some k. Oth-

erwise P(k < c) F(c) > 0; so k a.s. fails. For this
k,
Mttk = Mt
So (Mt ) is a u.i. martingale.
Our main objective is to show that all local martingales can be rep-
resented in the form of stochastic integrals. So we introduce some
elementary martingales associated with the process (xt ). For A S
and t R+ , define
p(t, A) = I(tT ) I(ZA)

Z
1
p(t, A) = dF A (s).
F(s )
]o,T t[
Proposition 1.5. Let q(t, A) = p(t, A) p(t, A). Then (q(t, A))t0 is an Ft
- martingale, i.e., p(t, A) the compensator of the point process p(t, A).
Proof. (Direct calculation). Take t > s, then 9
F A (s) F A (t)
E[p(t, A) p(s, A) | F s ] = I(s<T ) .
F(s)
E[ p(t, A) p(s, A) | F s ] = I s<T

Z Z Z

F(t) dF A (u) 1 dF A (u)

dF(r)

F(s)
F(u) F(s) F(u)

[s.t] [s.t] [s.r]
and
Z Z Z Z
dF A (u) 1
dF(r) = dF(r)dF A (u)
F(u) F(u)
[s,t] [s,r] [s,t] [u,t]
Z
1
= (F(t) F(u))dF A (u)
F(u)
[s.t]
Z
dF A (u)
= F(t) + F A (t) F A (s).
F(u)
[s.t]
So
E[q(t, A) q(s, A) | F s ] = 0

Another expression for p(t, A): We have F A (.) << F(.) (i.e. F A (.) is
absolutely continuous w.r.t. F(.)). So there exists a function (s, A) such
that Z
A A
F (0) F (t) = (s, A)dF(s).
]o.t]
In fact
(s, A) = P(Z A | T = s).
Suppose X is such that a regular version of this conditional probabil-
dF A (s)
ity exists (which is the case, since X is Borel space). Then =
F(s)
dF(s)
(s, A)d(s) where d(s) .
F(s)
Then Z
p(t, A) = (s, A)d(s).
]o,T T ]
10 Stochastic Integrals
Let I denote the set of measurable functions g : R such that
g(, z ) = 0.
(a) Integrals w.r.t. p(t, A): Suppose Nt is a counting process. Since

its sample functions are monotone increasing and there is a one-
to-one correspondence between monotone increasing functions
and measures, and since in this case, mass is concentrated at the
jump points and they are only countable; the function Nt defines
P
a random measure on (R, B(R)) say, = T i where X is the
i
Dirac measure at x. Similarly, the one jump process can be iden-
tified with the random measure (T,x R T ) on R+ X. So we can
define Stieltjes integrals of the form g(t, x)p(dt, dx) for suitable
integrands g I as
Z
g(t, x) p(dt, dx) = g(T, xT ).

We say g L1 (p) if
Z
E |g(t, x) | p(dt, dx) <

and denote
Z
|| g ||L1 (p) = E | g(t, x) | p(dt, dx)

Clearly g L1 (p) if and only if

Z
| g(t, x)d <
R+ X
(b) Integrals w.r.t. p(t, A): 11

R
Recall p(t, A) = (s, A)d(s).
]o,T t]
So we define
Z Z Z
g(t, x) p(dt, dx) = g(t, x)(t, dx)d(t)
[o,T ] X
Z
and say g L1 ( p) if |g(t, x) | p(dt, dx) <
Z
and || g ||L1 ( p) = | g(t, x) | p(dt, dx).

Proposition 1.6.
|| g ||L1 (p) =|| g ||L1 ( p)
and so
L1 (p) = L1 ( p).
Proof.
Z Z
1
|| g ||L1 ( p) = | g(s, x) | d(s, x)dF(T )
F(s )
R+ [o,T ]
Z Z
1
= | g(s, x) | ( dF(t))d(s, x)
F(s )
[s,]
Z
= | g(s, x) | d(s, x)

=|| g ||L1 (p) .
Define
Lloc
1 (p) = {g I | g(s, x)I st L1 (p), t < c}
Lloc
1 ( p) = {g I | g(s, x)I st L1 (p), t < c}. Clearly
Lloc loc
1 (p) = L1 ( p).
12 Following is the main result of this section, which gives an integral

representation for Ft local martingales.
Proposition 1.7. All Ft -local martingales are of the form

Z
Mt = (g(s, x)I(st) dq(s, x)

Z Z
= (g(s, x)Ist dp(s, x)
(g(s, x)I(st) dp(s, x).

for some g Lloc

1 (p).
We need the following result.

Lemma 1.1. Suppose (Mt )t>0 is u.i.Ft martingale with M0 = 0. Then

there exists a function h : R such that
E | h(T, Z) |< (1)
and Z
l
Mt = I(tT ) h(T, Z) I(t<T ) h(s, z)(ds, dz) (2)
F(t)
]0,t]X
Proof. If (Mt ) is a u.i. martingale, then Mt = E[ | Ft ] for some F -

measurable r.v. and from the definition of F , we have
= h(T, Z) a.s.
for some measurable h : R. Expression (1) is satisfied since Mt is 13

u.i., and Mo = 0 implies Z
h.d = 0. (3)

Now
Mt = E[h(T, Z)|Ft ]
Z
1
= I(tT ) h(T, Z) + I(t<T ) h(s, x)(ds, dx). (4)
F(t)
]t,]X
From (3) and (4), we have (2).
For g Lloc
I (p), the stochastic integral
Z
g
Mt = g(s, x)q(ds, dx)
]0,t]X
is defined by
Z Z
g
Mt = I(st) g(s, x)p(ds, dx) I(st)g(s,x) p(ds, dx).
R+ X R+ X
g
Then the question is whether Mt given by (2) is equal to Mt for
some g. As a motivation to the answer consider the following example.
Example 1.1. Let (X, S ) = (R, B(R)) and
(ds, dx) = (s, x) ds dx.
Then

ZT Z

g
1

Mt = I(tT )
g(T, Z) g(s, x)(s, x)dxds

F(s)

0 R
t

Z Z

1

I(tT )
g(s, x)(s, x)dxds
(5)

F(s)

0 R
g
14 If My given by (5) is equal to Mt given by (2), then the coefficients
of I(tT ) and I(t<T ) must agree. Comparing the coefficients of I(t>T ) , we
require
Zt Z
1
h(t, z) = g(t, z) g(s, x)(s, x)dxds.
F(s)
0 R
Let
(t) = h(t, z) g(t, z).
Define Z
(t) = hdx
R
and Z
f (t) = dx.
R
Then

ZT Z
1
(t) = h(s, z) + (s) (s, x)dx)ds
F(s)
0 R
Zt Zt
1 1
= (s)ds + (s) f (s)ds;
F(s) F(s)
0 0
that is
d f (t) 1
(t) = n(t) + (t)
dt F(t) F(t)
(o) = 0
which has a unique solution
Zt
1
(t) = (t, s) (s)ds,
F(s)
0
where
Zt
f (u)
(t, s) = exp du
F(u)
s
F(s) dF(t)
= , since f (t) = .
F(t) dt
So 15
Zt
1
(t) = (s)ds.
F(t)
0
Hence
Zt Z
1
g(t, z) = h(t, z) + h(s, x)(s, x)dxds (6)
F(t)
0 R
Now it can be checked that with this choice of g the coefficients of I(t<T )
in (2) coincides with that of (5). So Mt = Mtg .
Now we can prove the general case given in Proposition 1.7.
Proof of Proposition 1.7.

Case 1. c < , F(c ) > 0. Take a local martingale Mt with Mo = 0.
Then (Mt ) is u.i. So Mt = E[h(T, Z)|Ft ] for some measurable h such that
g
E|h| < , Eh = 0. Then we claim that Mt = Mt where
Z
1
g(t, z) = h(t, z) + I(t<c) h(s, z) (ds, dz).
F(s)
],t]x
But this can be seen algebraically following similar calculations as

of the example 1.1. Now to show that g Lloc
1 (p).
Z Z Z Z
1
|g|d |h|d |h|ddF(t)
F(t)
]o,c[ ]o,t]X
Z Z Z
1
|h|d |h|ddF(t)
F(c )
]o,c[ ]o,t]X
Z Z
1
|h|d + (F(t) F(c ))|h|d
F(c )
]o,c[X
!Z
1
1+ |h|d < .
F(c )
16 Case 2. c = , or c < and F(c ) = 0. Then from proposition 1.4, Mt

is a martingale on [0, c], and so it is u.i. on [o, t] for t < c. Therefore
M s = E[h(T, Z)|F ]
for some function h satisfying

Z
|g(s, x)|d(s, x) < for all t < c.
]o,t]xX
Define g(s, x) as in (6). Then calculations as in case 1. Show that

M s = M sg for s t < c. Now
Z Z Z Z
1
|g|d |h|d |h|d dF(s)
F(s)
[o,t]X ]o,tX ]o,t] ]o,s]X
Z Z
1
|h|d(1 dF(s))
F(s)
]o,t]X ]o,t]
< for t < c.
Hence g Llog
1 (p).
log g
Conversely, suppose g L1 (p). Then it can be checked that Mt is
a local martingale.
Remark 1.1. If g Llog g

1 (p) then Mt is a martingale. But the result does
not say Mt is a martingale if and only if M = M g for g L1 (p), it only
characterizes local martingales.
Remark 1.2. All preceding results hold if zo is a random variable; then

should be taken as conditional distribution of (T, Z) given zo
The multi-jump case: The process xt has jump times T 1 , T 2 , . . . with 17

corresponding states Z1 , Z2 , . . . Let (Y, y) denote the measurable space
(Y, y) = ((R+ X)U{(, z )}, {B(R+) S , {(, z )}}).
Define
Y
Y
K
= Yi , K = Yi
i=1 i=1

Y

o
F =
yi

i=1
where (Yi , yi ) denote a copy of (Y, y). Let

S k () = T k () T k1 ()
and wk () = (S (), Z1 (), . . . S k (), Zk ()).
Then
X
k
T k () = S i ()
i=1
T () = lim T k ().
k
As before, (xt (w))to is defined by

zo if t < T ()

xt () =
Zk if t [T k (), T k+1 ()]

z if t T ()
A probability measure on (, F o ) is specified by a family i :

i1 y [0, 1] (with o = ) satisfying
(i) i (.; ) is measurable for each fixed
(ii) i (wi1 (); .) is a probability measure on (Y, y) for each fixed

,
18 (iii) i (wi1 (); .({0} X) (R+ {zi1 ()})) = 0 for all ,
(iv) i (wi1 (); .{(, z )}) = 1 if S i1 () = .
Then for y and i1 , is defined by
[(T 1 , Z1 ) ] = 1 ()
[(S i , Zi ) |wi1 = ] = i ( : )i = 2, 3
Notice that as in the single jump case, here (iii) ensure that two
jump times T i1 , T do not occur at one and that the process xt does
effectively jump at its jump times (iv) ensures that [Zk = z |T k = ] =
1.
As before, Fto = {xs , s t} and
Ft = completion of Fto with all -null sets of F o .
Proposition 1.8. (i) F = F, the completion of F o

Q
n Q

(ii) FT n = yi yi .
i1 i=n+1
The idea here is to reduce everything to one jump case. That is, the
process restarts at each T k . We need the following result.
Proposition 1.9.

F(T kl +t) T k = FT kl V x(T kl +s) T k , o s t) .
Proof of this is an application of the Galmarino test (Dellacherie

and Mayer [12], theorem IV, pp. 100).
19 Recall that in one jump case Fto T = Fto . Now we conjecture that,
if U = (T k1 + to )T k , then
k1
Y Y
FU = yi ykto Yi ,
i1 i=k+1
where ykto = S B[o, to ] (x [to , ]) .
As an example, see the following exercise.

Exercise 1.2. Consider a point process with k = 2, and take the proba-
bility space as R2+ . Then
X
2
xt = I(tT i ) .
i=1
Then
(a) Show that
Ft = Borel sets in {S 1 + S 2 t}
\
+ (A R+ ) B + [t, ] R+ , A B(R)
where \
B = {S 1 + S 2 t} {S 2 t}.
(b) With U = (T 1 + to )T 2 , show that

\
FU = B(R+ [o, t]) + {(A R+ ) (R+ ]to , ]) : A B(R)}.
Elementary Martingales, Compensators Define

X
p(t, A) = I(tT i ) I(zi A)
i
which counts the jump of (xt ) ending in the set A. Define

Z
A 1
1 (s) = 1
dF 1A (u)
F (u)
[o,s]
where 20
F 1 = 1 ([t, ] X)
and Z
1
kA (wk1 ; s) = dF kA (u)
F k (u )
[o,s]
where
F kA (u) = k (wk1 ; [u, ] A).
Now define
p(t, A) = 1A (T 1 ) + 2A (w1 ; S 2 ) + Aj (w j1 ; t T j1 ())
for t ]T j1 , T j ].
Exercise 1.3. Consider a renewal process
X
xt = I(tT i )
i
and S i s are independent, P(S i > t) = F(t) is continuous. Then show

that the compensator for xt is
p(t) = n(F(S 1 )F(S 2 ) F(S k1 )F(t T k1 )) for t [T k1 , T k ],
and xt p(t) is a martingale.
Example 1. If F(t) = et , then p(t) = t.
21 Proposition 1.10. For fixed k, and A S ,

q(tT k , A) = p(tT k , A) p(tT k , A), t 0
is an Ft - martingale.
Proof. Calculation as in proposition 1.5.
The class of integrands I consists of measurable function g(t, x, )

such that

g1 (t, x), t T 1 ()

k
g(t, x, ) =
g (wk1 , t T kl , x), t ]T kl (), T k ()],

0, t T ()
for some function gk such that g1 (, x) = gk (wk ; , x) 0. Such g s

are F1 predictable processes. Now we define L1 (p), L1 ( p), etc. exactly
as in one jump case:
Z X
g dp = g(T i , Zi )
i

X

Li (p) =
g I : E |g(T i , Z i )| <

i
Z X Z
dF k (s)
g d p = g(kl , s, x)(kl , s, dx) k
k
F (s )
]o,T k T kl ]X
where
dF kA
(kl , s, A) = (s)
dF k
g loc
1 (p) if there exists a sequence of stopping times k T a.s. and
gI(tn L1 (p), n. For g L1oc
1 (p) we define
Z
g
Mt = g(s, x) q (ds, dx)
[o,t]X
Z Z
= g(s, x) p (ds, dx) g(s, x) p (ds, dx).
[o,t]X [o,t]X
Proposition 1.11. If g L1oc

1 then there exists a sequence of stopping 22
g
times T n < T such that n T and MtT n
is a u.i. martingale for
each n.
Proof. Take n = nT n n . Then the result follows by direct calcula-
tions using the optional sampling theorem.
Now let (Mt )t0 be a u.i. Ft martingale. Then
X

Mt = MtT 1 + (MtT k MT k1 ), ItT kl (7)
k=2
because this is an identity if t < T and the right-hand side is equal to
lim MT K is t T . Here we have MT = MT . Now we state the main
result.
Theorem 1.1. Let (Mt ) be a local martingale of Ft . Then there exists

g Lloc
1 (p) such that
Z
Mt Mo = g(s, x, ) q(ds, dx).
[o,t]x
Proof. Suppose first that Mt in a u.i. martingale. define
x1t = MtT 1
xkt M(t+T kl )T k MT k1 , k = 2, 3, . . .
Then from (7)

X

Mt = xk(tT kl )Vo .
k=l
We can now use proposition 1.7 to represent each xk . Fix k and

define for t 0.
Ht = F(t+T k1 )T k .
23 Then xkt is an Ht martingale. Then there exists a measurable function
hk such that
xkt = E(hk (k1 ; S k , Zk )|Ht ).
Then using proposition 1.7, there exists gk (k1 ; s, z) such that
Z
xkt = gk (k1 ; s, z)qk (ds, dz)
]o,t]X
where qk (t, A) = q((t + T k1 )T k , A) and gk Lloc k

1 (p ) for all
k1
a.s. Piecing these results together for k = 1, 2, 3, . . . gives the desired

representation with g = (gk ). It remains to prove that g Lloc 1 (p) as
defined, for which we refer to Davis [6].
If (Mt ) is only a local martingale with associated stopping time se-
quence n such that Mt n is a u.i. martingale, apply the above
arguments to Mt n to complete the proof.
Corollary 1.1. If T = a.s. then the result says (Mt ) is a local

g
martingale of Ft if and only if Mt = Mt for some g Lloc
1 (p).
2. Some Discontinuous Markov Processes 23
Remark 1.3. It would be useful to determine the exact class of inte-

grands g required to represent u.i. martingales (as opposed to local mar-
tingales) when the jump times T i are totally inaccessible, Boel, Varaiya
and Wong [4] show that {M g , g L1 (p)} coincides with the set of u.i. 24
martingales of integrable variation. It seems likely that this coincides
with the set u.i. martingales if E p(t, E) < for all t (a somewhat
stronger condition than T i a.s.) but no proof of this is available as
yet.
2 Some Discontinuous Markov Processes
Extended Generator of a Markov Process

Let the process xt (E, E), some measurable space. Then (xt , Ft ) is
a Markov process if for s t
E[ f (xt )|F s ] = E[ f (xt )|xs ]a.s.
A transition function p(s, x, t, ) is a function such that
p(s, xs , t, ) = P(xt |xs )

= E[I (xt )|xs ] a.s. f or t s.
p satisfies the Chapman-Kolmogorov equation

Z
p(s, x, t, ) = p(s, x, u, dy)p(u, y, t, ) for s u t.
E
Not every Markov process has a transition function, but usually one
wants to start with transition function and construct the corresponding
process. This is possible if (E, E) is a Borel space (required to apply
Kolmogorov extension theorem; refer Wentzel [27]). One constructs a
Markov family,
{P x,s , (x, s) E R+ }px,s
being the measure for the process starting at xs = x. All measures P s,x
have the same transition function p. Denote by E x,s integration w.r.t
P x,s . Let B(E) be the set of bounded measurable functions
f : E R with k f k = sup | f (x)|

xE
25 Define
T s,t f (x) = E x,s | f (xt )|], s t.
T s,t is an operator on B(E) such that
(i) it is contraction, kT s,t f k k f k, T s,t 1 = 1.
(ii) Semi group property: r s t,
T r,t = T r,s T s,t
for
T r,t (T s,t f ) (x) = E x,r [E xs ,s ( f (xt ))]

= E x,r [E( f (xt )|F s )]
= E x,r f (xt ))
= T r,t f (x).
T s,t is time invariant if T s+r,t+r = T s,t for all r s. Then T s,t =

T 0,ts T ts . So T is a one parameter family; this happens when the
transition function is time invariant i.e., p(s, x, t, ) = p(s + r, x, t +
r, ). Then get a one parameter family of measures (P x , x E) and the
connection is
T t f (x) = E x f (xt ); T 0 f = f.
Let
B0 (E) = { f B(E) : kT t f f k 0, t 0}.

An operator A with domain D(A) B (E) is the strong infinitesimal
generator of T t if

lim k(T t f f ) A f k = 0.
t0
26 So
d
Af = T t f (x)|t=0
dt

Take f D(A). Then
1 1
lim (T t T s f T s f ) = lim (T t+s f T s f )
t0 t t0 t
1
= lim T s (T t f f )
t0 t

= T s A f.

So f D(A) implies T s f D(A) and AT s f = T s A f . So we get
backward Kolmogorov equation
d
T s f = A(T s f ). (1)
ds
The main results of the analytic theory of Markov semigroups are the
following;
(i) Hille-Yosida theorem: Necessary and sufficient conditions for an

operator A to be the generator of some semigroup.

(ii) If A satisfies these conditions, then D(A) is dense in B0 (E) and

(A, D(A)) determines T t (via the so called resolvent operator).

NB: The domain DA provides essential information.
Integrating (1), we get Dynkins formula
Zt

T t f (x) f (x) = T s A f (x)ds
0
Zt

i.e., E x f (xt ) f (x0 ) = E x A f (xs )ds, f D(A).
0

Proposition 2.1. If f D(A) then the process
Zt

Ctf = f (xt ) f (x0 ) A f (xs )ds
0
is a martingale.
27 Proof. For t s

Zt

E|Ctf C sf |F s ] = E f (xt ) f (xs ) A f (xu )du|F s

s
Zt

= E xs f (xt ) f (xs ) E xs A f (xs )ds. 0.
s

Definition 2.1. Let M(E) be the set of measurable functions f : E R.

Then A, D(A) with D(A) M(E) is the extended generator of (xt ) if Ctf
is a local martingale, where
Zt
f
Ct = f (xt ) f (x0 ) A f (xs )ds.
0

This is an extension of (A, DA)) in that D(A) D(A) and A f = A f

for f D(A). We have uniqueness of A in the following sense. Write
Zt
f
f (xt ) = f (x0 ) + A f (xs )ds + Ct .
0
This shows that ( f (xt )) is a special semi-martingale (=local mar-

tingale + predictable bounded variation process). The decomposition is
unique. So, if B is another generator then
Zt
(A f (xs ) B f (xs ))ds = 0 P x a.s.t.
0
Thus A f (x) = B f (x) except on a set of potential zero. where a set of

has potential zero, where a set has potential zero if
Z
Ex I (xs )ds = 0 x.
0
Example 2.1. Suppose xt Rd satisfies

dxt = b(xt )dt + (xt ) dwt
with standard Ito conditions. If f C 2 , then 28
Zt
X f 1 X 2f

d f (xt ) = bi (xt ) (xt ) +
( )i j

dt + f dw.
i
x i 2 i, j
xi xi x j
0
So C 2 D(A) and
X Zt
f 1X 2 f
A f (x) = bi (x) + ( )i j Cf = f dw.
i
xi 2 i j xi x j t
0
NB: This is not a characterisation of D(A).

f
Remark 2.1. If we had required Ct to be a martingale rather than a
local martingale in definition 2.1, then not every f C 2 would be in
D(A) because of the properties of I to integrals.
Exercise 2.1. For i = 1, 2, . . . ,let Nti be a Poisson process with rate i

P
where i i < . Define
X

Xt = i Nti
i=1
where i 0 and
X

i i = r < .
i=1
Find the extended generator of xt
(This is also an example where jump times are not isolated).
Piecewise-linear Markov Process 29

Gnedenko-Kovalenko introduced the concept of piecewise linear
Markov process. Later, Vermes [24] simplified the definition as fol-
lows. A piecewise linear process is a two component Markov process
(xt ) = (t , t ) where vt is integer-valued and t takes values in an interval
[an , bn ] of the real line if t = n (bn may be +). Let E be the state
space, i.e., E = {(n, ) Z R : Z R : [an , bn ]} Then the prob-
abilistic description is that if the motion starts at (n, z) E and xt os
given by t = n, t = z + t for t < T 1 , the first jump time. Spontaneous
jumps happen at rate (xt ), i.e., probability jump occurs in (t, t + dt),
is (xt )dt, and process must jump if t = bn . Let the transition measure
be given by Q(A; x) for A B(E). Then xT 1 is selected from the prob-
ability distribution Q(A; xT 1 ). After a jump, motion restarts as before.
Thus the law of the process is determined by specifying the intervals
[an , bn ], the jump intensity (x) and the transition measure Q(A; x).
Example 2.2. Non-stationary countable state Markov Process ( )

( ) takes integer values with the-dependent transition rates ai j (t)
such that
Pt+h = i|t = j] = ai j (t) + (), i , j.
Then xt = (t , t) is a PL process with no barriers, i.e., an = 0, bn = +.
30 Example 2.3. Countable state process with non-exponential sojourn

times
Here, jump times of the process (xt ) form a renewal process with
inter arrival density b(.) and transition matrix qi j = P[xT k , = i, xT k = j].
This is a PL process with t = xt , and t the time since last arrival. The
jump rate is
b()
(, ) =
R
b(t)dt

Here again an = 0, bn = +.
Example 2.4. Virtual waiting times. (The M/ G/ 1 queue)

Customers arrive at a single-server queue according to Poisson pro-

cess with rate , and have i.i.d. service time requirements with distri-
butions F. The virtual waiting time t is the time a customer arriving at
time t would have to wait for service. Piecewise linear process structure
is :

1 if queue is not empty
vt =

0 if queue is empty
and a1 = 0, b1 = , a = b = 0. Here t moves to left with uniform

speed and transition to (0, 0) is certain if xt = (1, 0).
A more general definition of PL process of Gnedenko and Kova-

lenko allows t to move in an open subset 0t of Rd (t ) with uniform
speed in a fixed direction V(t ). Again transition must take place if
t , 0t , the boundary of 0t .
Example 2.5. VWT with renewal process arrivals. (The GI/G/I queue) 31
Suppose the inter arrivals times in Example 2.4 are not exponential,
but form a renewal process with inter arrival density b(.). Now the ap-
propriate structure is is 0 or 1 as before, d(1) = 2, d(0) = 1. (When
= 1 we have to remember both the value of VMT and the time since
the last arrival.)
We cannot accommodate this in previous framework, because there

[an , bn ] is fixed, whereas here the length of the interval is random.
Davis [7] introduced the piecewise deterministic (PD) process
which is a further generalization. It is similar to the piecewise linear pro-
cess, except that t satisfies some ordinary differential equation, rather
than moving in straight line.
Example 2.6. Shot noise. This has sample functions similar to the VWT
process except that decay between arrivals is exponential rather than
linear (fig. 2.1).
Figure 2.1:
32 Example 2.7. A model for capacity expansion. Suppose that the demand
for some utility is monotone increasing and follows a Poisson process
with rate . Each unit of supply provides q units of capacity. These
are built one at a time at a cost of Rs.p. Investment takes place at a rate
of Rs.u(t)/week and u(t) constant. When
Zt
u(s)ds = p,
0
then the project is finished, capacity is increased by q and investments

are channelled into next project.
Denote dt = demand; ct = capacity at time t; t = cumulative invest-

ment in current project
Zt
= u(s)ds,

where is the last time project was completed. Investment is determined
by some policy , i.e.,
d
u(t) = t = (ct , dt , t )
dt
where (ct , dt , t ) is the current situation. Define vt = (ct , d). Then the
process xt = (t , t ) evolves in the state space E = Z+2 [o, p] (Z+2
is the 2-dimensional positive integer lattice). Then for = (c, d) if
d
g () = (c, d, ), t satisfies t = gt (t ).
dt
The piecewise-deterministic process;

Let K be a countable set and d : K N (= natural numbers) be a
given function. For each K, M is an open subset of Rd() (M can be
a d()-dimensional manifold). Then the state space of the PD process is 33
E = U M = {(, ); K, M }.
K
Let
E = A ; A B(M)
K
Then (E, E) is a Borel space. Then the process is xt = (t , t ). The

probability law of (xt ) is specified by The probability law of (xt ) is spec-
ified by
(i) Vector fields (X , K)
(ii) A rate function : E R+
(iii) A transition measure Q : E E [0, 1]
Assume that corresponding to each X there is a unique integral

curve (t, z), i.e., (t, z) satisfies
d
f ( (t, z)) = X f ( (t, z))
dt
(o, z) = z
for every smooth function f , and (t, z) exists for all t o. Let M
be the boundary of M . M is those points in M at which integral
curves exit from M , i.e., M = {z M : (t, ) = z for some
(t, ) R M }.
Let
= {, z : : K, z M }.
So is the set of points on the boundary at which jumps may take
place. For x = (, z) E, denote
t (x) = inf{t > 0 : (t, z) M }.

Write Xh(x) for the function whose value at x = (, z) is X h(, .)(z).

For , we suppose that the function t (, (t, z)) is Lebesgue inte-
grable on [o, ] for some > 0, 0(., x) is a probability measure on (E, E) 34
for each x EU.
The motion of the process (xt ) starting from x = (n, z) E is de-
scribed as follows. Define
t
R
exp (n, n (s, z))ds , t < t (x)

F(t) = 0

0, t t (x).
This is the distributions of T 1 , the first jump time. More precisely,

F(t) is the survivor function
F(t) = P x [T 1 > t].
Now let Z1 be an E-valued random variable with distribution

Q(, ; n )(T 1 , z)). Then define

(n, n (t, z)) t < T 1
xt =

Z1 t = T1
and restart with (n, z) replaced by Z1 . Assume T k a.s. Then xt de-

fines a measurable mapping from (, a, P) (countable product of unit
interval probability spaces) into space of right continuous E- valued
functions. This defines a measure P x on the Canonical space.
NB: The condition on ensures that T 1 > 0 a.s and hence that T k
T k1 > 0 a.s.
Proposition 2.2. (xt , P x ) is a Markov process.
Proof. Suppose that T x t < T k+1 . The distributions of T k+1 T k is

given by
s

R

exp T k , T k du , s < t (xT k )

P[T k+1 T k > s] =
0

0, s t (xT k ).
35 Denote = T k , = T k , Then for s > t and s < t (xT k )
P[T k+l > s|T k , T k+l > t] = P[T k+l T k > s T k |T k , T k+l T k > t T k ]
sT
Z k

= exp (, (u, ))du

tT k
st
Z

= exp (t , t (u, t ))du

o
where we used the semigroup property of . Since the process restarts

at T k+1 , the law of the process for s > t given part upto t coincides with
the law given xt . Hence the Markov Property.
Let be the subset for which, if y = (, )
P[T 1 = T (x)] asx = (n, z) y.
Then is called the essential boundary
Exercise 2.2. Prove that y if and only if P x [T 1 = T (x)] > 0 for

some x = (, z).
So P x [xs ] for some s > 0] = 0, and Q(A, x) need not be

specified for x .
Example 2.8. Here has only a single value; so delete it. takes values

in M = [0.1] R+ , = 0 and X = . Then = {(l, y); y R+ }. Let
1
1 1
Q(., (l, y)) = (1 y, y).
2 2
Then starting at x0 = (0, 1) we have 36
X
n
1
Tn = so that lim T n < .
k=1
k1 n
The same effect could be achieved by the combined effect of and

Q, suitably chosen. So, we prefer to assume T = a.s. rather than
stating sufficient conditions on , , Q to ensure this. To illustrate the

1
difference between and , suppose that (1 , 1) = (This is
1 1
equivalent to saying T 1 is uniformly distributed on [0, 1] if the process
starts at (0, 1). Then (1,1) os never hit, whatever be the starting point.
So (1,1) .
Figure 2.2:
The Associated Jump Process.

Let (xt ) be an PD process. Definite the associated jump process (zt )
by
zt = xT k , t [T k , T k+1 [ (2)
37 This is an E-valued jump process such that ZT K = xT K . Let Ft =
{xs , s t} and Ftz = {zs , s t}.
Proposition 2.3. Ft = Ftz for each t
Proof. This follows form the fact that there is a one-to-one mapping
from x[o,t] to z[o,t] .x z is given by (2). Conversely, if z[o,t] is given
then x[o, t] can be constructed since the motion in the interval [T k , T k+1 [
is deterministic.
NB:
(1) xt and zt are not in one-to-one correspondence at each fixed time t.

(2) (zt ) is not a Markov process.
Since Ft = Ftz , we can apply jump process theory. Define

X
p(t, A) = I(xTi A)
T t
i
X
pt = I(xTi )
T i t
Zt Zt
p(t, A) = Q(A, xs )(xs )ds + Q(A, xs )dps (3)
0 0
Proposition 2.4. Suppose E(p(t, E)) < . Then for each
A E, q(t, A) = p(t, A) p(t, A) (4)
is an Ft -martingale.
Proof. From previous results, the compensator of p(tT 1 , A) is

Z
dF s
p(tT 1 , A) = Q(A, xs ) .
F s
]o,tT 1 ]
But t 38
R
exp (xs )ds

t < t1 (x)
Ft =
0

0 t t1 (x).
dFt
Thus - = (xt )dt for t < t1 (x) and
Ft
Ftt
= 1.
Ft1
This verifies the result for t T 1 . As before, we show by consider-

ing intervals [T k1 , T k ] that the compensator of p(tT n, A) is p(tT n , A)
given by (3). Since p(t, A) and p(t, A) are monotonic increasing func-
tions and T n a.s. E(p(t, E)) < , taking the limits, we have
q(t, A) = p(t, A) p(t, A)
is a martingale.
Exercise 2.3. Show that pt is an Ft predictable process.
Then (4) is the Doob-Meyer decomposition of the submartingale p.

The next step is to use stochastic integrals to calculate the extended
generator of xt . Choose the following integrands. For Measurable f :
E R, define
B f (x, s, ) = f (x) f (xs ())
Then Bf L1 (p) if
X
E | f (xT i ) f (xT i )| <
T i t
39 for each t 0. This certainly holds if f is bounded and E p (t, E) < .
Zt Z Z Z
B f (y, s, w) p(ds, dy) = ( f (y) f (xs ))Q(dy; xs )(xs )ds
o E [0,t] E
Z Z
+ ( f (y) f (xs ))Q(dy; xs )dss . (5)
[0,t] E
Suppose that f satisfies the boundary condition

Z
f (x) = f (y)Q(dy; x), x . (6)
E
Then the second integral in (5) is zero. The following result charac-
terizes the extended generator A of (xt ).
Theorem 2.1. The domain D(A) of the extended generator A of (xt )

consists of those functions f satisfying
(i) For each (n, z) E the function t f (n, n (n, z)) is absolutely
continuous for t [0, t (n, z)[.
(ii) The boundary condition (6) is satisfied.
(iii) Bf Lloc
1 (p).
Then for f D(A)

Z
A f (x) = X f (x) + (x) [ f (y) f (x)]Q(dy; x). (7)
E
R
Proof. Suppose that f satisfies (i)-(iii). Then Bf dq is a local martin-
gale, and
Zt X Zt Z
B f dq = f (xT i ) f (xT i ) [ f (y) f (x s )]Q(dy; x s)(x s )ds.
0 T i t 0 E
Now, 40

X X
f (xT i ) f (xT i ) = ( f (xT i ) f (XT i1 )) + f (xt ) f (xT n )
T i t T i t

X
( f (xT i ) f (xT i1 )) + f (xt ) f (xT n )
T i t
where T n is the last jump time before t. The first bracket is ( f (xt )
f (xo )). Note that
ZT i
f (xT i ) f (xT i1 ) = XTi1 f (T i1 Ti1 (T i1 , s)ds a.s).
T i1

Rt
So the second bracket is equal to x f (xs )ds and
0
Z
B f dq = f (xt ) f (x0 )
Zt Z Zt
( f (y) f (xs ))Q(dy, xs )(xs )ds X f (xs )ds.
o E o
Rt
So A f is given by (7) and Ctf = Bf dq. Conversely, suppose
0
f D(A). Then there exists a function h such that s h(xs ) is
Rt
Lebesgue integrable and Mt = f (xt ) f (x0 ) h(xs )ds is a local mar-
0
g
tingale. By the martingale representation theorem, Mt = Mt for some
g
g L1oc
1 (p). Now the jumps of Mt and Mt must agree, these only occur
when t = T i for some i and are then given by
Mt = Mt Mt = f (xT i ) f (xT i ).
g g g
Mt = Mt Mt
Z
= g(xt , t, w) g(y, t, w)Q(dy, xt )I(xt )
E
at t = T i . It follows that
g(x, t, w)I(xt </) = ( f (x) f (xt ))I( xt<)
41 except possibly on a set G E p such that
Z
Ey IG p(dt, dx) = 0 for all y E.
R+ E
Now suppose XT i = z ; then

Z
f (x) f (z) = g(x, t, ) g(y, t, )Q(dy; z)
E
for all x except a set A E such that Q(A, z) = 0. Since only the first
terms on the left and right involve x it must be the case that
f (x) = g(x, t, ) + f(t, )
Z
and f (z) = g(y, t, )Q(dy; z) + f(t, w)
E
for some predictable process f. Since g = f f,

Z
f (z) = f (y)Q(dy; z)
E
for z , i.e., f satisfies condition (ii). Hence
g(x, t, ) = f (x) f (xt ).
Hence we get
k(B f g)I(t<n ) kL1 (p) = 0.
So condition (iii) is satisfied. Fix and consider (Mt )ot<t1 () start-

ing at (o , o ). , then
Zt
Mt = f (o , o (t, o )) f (o , o ) h(xs )ds
o
Zt Z
g
Mt = ( f (y) f (xs ))Q(dy; xs )(xs )ds.
o E
Hence f (o , o (t, o )) is absolutely continuous for t < T 1 (). Since 42

(o , o ) is arbitrary and T 1 (w) > 0 a.s. this shows that (i) is satisfied.
A Feynman-Kac formula.
This is used to calculate expected values of functionals such as
t
Z

E x es c(s, xs )ds + et (xt ) .

o
There is no extra generality in allowing a P.D. Process to be time-

varying, because time can always be included as one component of t .
However, it is sometimes convenient to consider the joint process (t, xt )

with generator A = + A. Then for f D(A)
t
Zt ! Zt

f (t, xt ) f (o, xo ) = + A f (s, xs )ds + B f dq.
s
o o
!

If + A f (s, xs ) = o and B f L1 (p), then f (t, xt ) is a martin-
s
gale, so it has constant expectation
E xo f (t, xt ) = f (o, xo ).
Then
f (o, xo ) = E xo (xt )
where f (t, x) = (x) ( prescribed).
Proposition 2.5. Let t > o be fixed and : [o, t] E R+ , c : [o, t]
E R and : E R be measurable functions. Suppose f : [o, t]
E R satisfies:

(i) f (s, . D(A))

(ii) f (t, x) = (x), x E
(8)

(iii) B f Ll (p)
f (s, x)
+ A f (s, x) (s, x) f (s, x) + c(s, x) = 0 (9)
s
(s, x) [o, t[ E.
43 Then
t s
Z Z

f (o, x) = Eo,x exp (u, xu )du c(s, xs )ds

0 o
t
Z

+ exp (u, xu )du (xt ) (10)

o
Proof. Suppose f satisfies (8). Define

Z s !
es = exp (u, xu )du .
o
Then
d(es f (s, xs )) = es d f (s, xs ) + f (s, xs )des

!
f
= es + A f ds + es B f dq (s, xs )es f ds
s
= es c(s, xs )ds + es B f dq (by (9)).
Now by (iii), es B f L1 (p) since es 1. Thus the last term is a

martingale and
t
Z

E x [et f (t, xt ) f (o, x)] = E x es c(s, xs )ds .

o
This with (ii) gives (10).
Example 2.9. The Renewal Equation:

Let (Nt ) be a renewal process with inter arrival density f (.). Let
m(t) = ENt . Since the process restarts at renewal times,

0, s>t
E[Nt |T 1 = s] =

m(t s) + 1, s < t.
So
Z
m(t) = E[Nt |T 1 = s] f (s)ds
o
which gives the renewal equation 44
Zt
m(t) = (1 + m(t s)) f (s) ds. (11)
o
This can be solved by Laplace transforms. Defining
Z
f(p) = ept f (t)dt
o
etc., we get
1
m(p) = m(p) f(p) + f(p).
p
So
1
p f(p)
m(p) = .
1 f(p)
In particular, for the Poisson process f (t) = et ,

f =
+ p
will give

m(p) =
p2
to get
m(t) = t.
Exercise 2.4. Compute M (t) = E Nt , where the component in service

at time 0 has age (and is replaced by a new component when it fails).
(Nt ) is a PD process if we take xt = (t , t ) where t = Nt and t is

the time since last renewal. Then
f ()
X = , () =
R
f (u)du

45 and Q(.; , ) = (+1,0) , so that

A f (, ) = f (, ) + ()[ f ( + 1, 0) f (, 0)].

Use proposition 2.5 with = c = 0 and (x) = to get
f (0, , ) = E(,) t .
Clearly
f (s, + 1, ) = f (s, , ) + 1.
Define
f (s, 0, ) = h(s, ).
Then the equation for f (or h) becomes

h(s, ) + h(s, ) + ()[1 + h(s, 0) h(s, )] = 0 (12)
s
h(t, ) = 0.
Define
z(u) = h(u, u).
Then
d
z(u) = (u)[1 + h(u, 0) z(u)]
du
z(t) = 0.
Thus z(u) satisfies
z(u) = (u) z(u) (u)[1 + h(u, 0)] (13)
where
Z
F f
(u) = = , F(u) = f (s) ds.
F F
u
Equation (13) is a linear ODE satisfied by z(u). The transition function

corresponding to (u) is
F(u)
(u, v) =
F(v)
Hence (13) has the following solution at time 0: 46
Zt
z(o) = h(o, o) = f (u)[1 + h(u, o)]du (14)
o
Define
m(s) = h(t s, o).
Then (14) coincides with the renewal equation (11). Having deter-
mined h(u, o) o u t, h(s, ) for s , , o can be calculated from
(12). The result will be equivalent to that of Exercise 2.4.
Chapter 2
Optimal Control of pd
Processes
General formulations of stochastic control problems have been studied 47

using martingale theory, where the conditions for optimality, existence
of optimality are derived (E1 Karoui [15]). But this does not give ways
of computing optimal control. Control of Markov jump processes has
been studied using dynamic programming (Pliska [21]). In this Chap-
ter, we will be dealing with control theory for PD processes, following
Vermes [25].
Let Y be a compact metric space. Control arises when the system
functions X, , Q contain a parameter y Y i.e., for x = (, )
X f (, )
X y f (x) = b(, , y)
i
i
Q = Q(A, x, y).
A feedback policy (or strategy) is a function u : R+ E Y.

Let U denote the set of all strategies. u is stationary if there is no t-
dependence, i.e., u : E Y. Corresponding to policy u we get a PD
process with characteristics X u , u , Qu given by
X f
Xu f = b(, , u(x)) (x)
i
i
45
46 2. Optimal Control of pd Processes
u (x) = (x, u(x))

Qu (A; x) = Q(A; x, u(x)).
More conditions on u will be added when required. Then we get a

PD process xt with probability measure Pu determined by X u , u , Qu .
48 Given a cost function, say, for example,
t
Z

Jx (u) = E ux es c(xs , us )ds + et (xt )

o
where E ux is the expectation w.r.t. Pu starting at x and > 0. The control

problem is to choose u(.) to minimise Jx (u). The usual approach to
such problems is via dynamic programming. Let V(s, x) be a function
of (s, x). Introduce the Bellman-Hamilton-Jacobi equation
V(s.x)
+ min [Ay V(s, x) + c(x, y)] V(s, x) = 0 (B)
s yY
where Ay is the generator corresponding to X y , (., y), Q(., y).

If Y has one point, then this coincides with the equation for Jx as
before.
Proposition 1. Suppose (B) has a nice solution (i.e., satisfies bound-

ary condition etc.). Then
V(o, x) = min Jx (u)

uU
and the optimal strategy uo (s, x) satisfies

o (s,x)
Au V(s, x) + c(x, uo (s, x)) = min (Ay v + c).
yY
Proof. Same calculations arise as before. Let xt correspond to an arbi-

trary control policy u. Then
!
V
d(es V(s, xs )) = es V(s, xs )ds + es + Au V ds
s
+ e BV dq e c(xs , u(s, xs ))ds + es Bu dq (1)
s s
47
49 So
t
Z

V(o, x) E x es c(xs , us )ds + et (xt )

o
= Jx (u).
Now suppose u = u0 , then equality holds in place of inequality in

(1). So
V(0, x) = Jx (u0 ).
So uo is optimal.
Objections:
(1) There is no general theory under which (B) has solution.
(2) uo (x) constructed as above may fail to be an admissible control:

to make sense of it, we must be able to solve the ODE
d
(s) = buo ( s ) = b(, , uo (s, )).
ds
There is no guarantee that uo leads to a solvable ODE.

So we must redefine admissible controls so that this is avoided.
Remark 1. In control of diffusion processes, the equation is
dxt = b(xt , u(xt ))dt + (xt )dWt .
Here we handle nonsmooth u by using weak solutions.
Remark 2. In deterministic control, one uses open-loop controls de-

pending only on time. The equation here is of the form
x = b(xt , u(t)).
Then solution is well defined for the measurable u(.).

Special cases: 50
(1) Control only appears in Q. Then the problem reduces to a se-

quential decision problem where a decision is taken each time
a jump occurs. (Rosberg, Varaiya and Walrand [22]).
(2) X = 0. Here Markov jump process with piecewise constant paths

are considered. Control appears in and Q.
Then

Z

A f = (x, u(x)) ( f (z) f (x))Q(dz; x, u(x))
u

E
is a bounded operator on B(E). Regard (B) as an ODE in Banach

space B(E). Let V(s) := V(s, .), then
dV
= g(V(s)) = min(Ay V + c).
ds yY
So g is a nonlinear function, but it is Lipschitz continuous in V

[Pliska [21]].
(3) Piecewise linear processes (Vermes [25]).

Here t is on dimensional and X = . Control appears in and Q.
Consider a stationary control problem, where the Bellman equa-
tion takes the form
min(Ay V + c(x, y)) = 0

yY
V(x) = (x), xET .
This corresponds to minimising

Z

E x c(xs , us )ds + (x ) ,

0
where is the first hitting time of some target set ET . Then

Z

Ay V(x) = V(, ) + (x, y) (V(z) V(x))Q(dz, x, y).

E
49
51 Suppose {1, 2, . . . , n} and

!
V(l, )
V() =
V(N, )
Then Bellman equation takes the form
d
V() = g(V(.)).
d
This is an ordinary functional differential equation with non-stan-

dard boundary condition. Vermes showed existence of an optimal feed-
back strategy in special cases.
Generalised Dynamic Programming Conditions:
Let us consider next optimal control of the deterministic differential
system:
xt = f (xt , t, ut ), t[to , tl ]. (2)
Then the control problem is to
Ztl
minimize (xt , t, ut )dt
to
over admissible control/trajectory pairs ut , xt i.e., pairs of functions

for which
(i) (2) is satisfied,
(ii) x(tl ) = xl , x(to ) = xo with xo , xl given,
(iii) xt A, ut , where A, are compact and A = A [to , tl ].
This will be called the strong problem (S ).

We assume (a) an admissible pair (xt , ut ) exists, and also we make a
temporary assumption

f (x, t, )
(b) is convex. 52
(x, t, )
This enables relaxed controls to be avoided. Define

(S ) = value of S
Z
i.e., inf dt.
(xt ,ut )ad
Theorem 1. There exists an optimal admissible pair (xt , ut ) for the

strong problem.
This is a standard result in optimal control theory (Vinter and
Lewis [26]). It depends critically on the convexity assumption (b).
A Sufficient Condition for Optimality: (Standard Dynamic Program-
ming). Suppose (xt , ut ) is admissible and is in C 1 (A) such that
t (t, x) + max(x (x, t) f (x, t, u) (x, t, u)) = 0
u
(xl , tl ) A
and t (t, xt ) + (xt , t) f (xt , t, ut ) (xt , t, ut ) = 0 a.a.t.
then (xt , ut ) is optimal and (S ) = (xo , to ). The main result of Vinter
and Lewis is as follows.
Theorem 2. The strong problem has a solution (i.e., there exists an
optimal pair (xt , ut )). There exists a sequence {i } in C 1 (A) such that
it + max(x f ) o, (x, t)A
u
i
(x1 , t1 ) = 0
53 and (xt , ut ) is optimal if and only if
lim H i (t) = 0 in Ll [to , t1 ]
t
where H i (t) = it (xt , t) + ix (xt , t) f (xt , t, ut ) (xt , t, ut ).
The Weak Problem

For (xt , ut ) admissible, define x,u C (A ), the dual of C(A ),
by
Ztl
< g, x,u >= g(xt , t, ut ) dt
to
for arbitrary gC(A ). x,u satisfies
51
(i) x,u P+ (i.e., if g o then < g, x,u > o).
(ii) Take C 1 (A) and g(x, t, u) = (x, t) + x (x, t) f (x, t, u)
then
< t + x f, x,u >= (xl , tl ) (xo , to ).
Define
= {C (A ) : (i) and (ii) are satisfied}.
Proposition 2. is weak* compact and convex.
Note 1. The cost function for (xt , ut ) is < , x,u >.
Weak Problem (W): Minimise < , u > over . So 54
(W) (S ).
Theorem 3. (S ) = (W). There exists an optimal x, u for S , so x,u is

optimal for W.
Now we incorporate the constraints on into the cost function in the

following way. Define extended real valued functions p, q on C (A )
as follows:

< , > if || tl to , P+
p() =

+ otherwise.
Let M2 = {C : Condition (ii) is satisfied}. Then

o if M2
q() =

otherwise.
Proposition 3. p is . s.c. and convex,

q is u.s.c. and concave,
and
(W) = inf {p() q()}.
C
The Fenchel dual problem is as follows:
max (q () p ()) (D)

C(A)
where p , q are dual functionals defined by
p () = sup < , > p().

C
q () = inf < , > q().

C
Proposition 4.
p () = max ((t, x, u) (t, x, u))+ (tl to ).

(x,t,u)A
55 where a+ := max(a, 0).
Sketch of proof:
p () = sup [< , > < , >]

||tl to
P+
= sup [< , >].

||tl to
P+
If < , > < , > is negative, then the optimum is zero. If

< , > < , > o, then put Dirac measure x(tl to ) on maximum
point to get the result.
Proposition 5.
W = {C : = t + x f for some C l (A)}.
Then

if < W

q () =
i i

lim( (x1 , t1 ) (xo , to )) if W
i
where = lim i and i = it + ix f.
i
53
Proof. For W, by definition of q and q , we get q () = (x1 , t1 )

(x0 , to ).
A similar argument gives the result for u W. For < W, there
exists a separating hyperplane, i.e., C such that < , >, 0, W
and < , >= 0. If M2 , then + c M2 . So
q () = inf < , >= .

M2
Characterizing the solution of (D): 56

"
(D) = max lim(i (x1 , t1 ) i (xo , to ))
W i
#
max ((t, x, u) (t, x, u)+ (tl , to ))
(x,t)A
= sup ((x1 , t1 ) (xo , to ) max(t + x f )+ (tl to )).

C x,t,u
It is no restriction to assume (x1 , t1 ) = 0. Then Vinter and Lewis show

by an ingenious argument that
(D) = sup((xo , to ))
where the supremum over C l such that (x1 , t1 ) = 0 and (t + x f

) 0 (x, t, u).
Theorem 4.
(D) = (W) = (S ).
Proof. This follows from a standard result in duality theory: q is
finite at some point in its domain where p is continuous.
Proof of the main results now follow easily. The strong problem has
a solution since (S ) = (D).
(S ) = lim(i (xo , to ))
for some sequence of i s satisfying the Bellman inequality. The char-

acterization of optimal pairs (xt , ut ) follows.
Remark 3. If the set ! 57

f (x, t, )
(x, t, )
is not convex, them the results are still valid but relaxed controls must
be used.
A relaxed control t is a C ()-valued function

R on [to , t1 ] such that
t is a probability measure for every t and t g(t, u)t (du) is measur-
able for every continuous function g.
Interpretation: xt , t is an admissible pair, whenever
Z
dxt
= f (xt , t, u)t (du);
dt

Ztl Z
the cost is f (xt , t, u)t (du)dt.
to
Optimal Control PD Processes (Vermes [25])

In this section, we adopt a slightly modified definition of the PD
process (xt ). It will take values in E, a closed subset of Rd , and we
suppose that
E = Eo UE UET (disjoint)
where ET is a closed set, Eo is an open set and
E = (Eo Eo ) ET .
Let Eo , E , ET be compactification of Eo , E , ET respectively and

E be the disjoint union of Eo , E , and ET . Then a controlled PD process
is determined by functions
f : Eo Y Rd ;
: Eo Y R+ ,
and Q : (Eo UE ) Y m1 (Eo )
58 where ml (Eo ) is the set of probability measures on Eo and Y is a compact

55
(control) space. Function f gives the deterministic motion by
xt = f (xt , yt ).
We assume f satisfies a Lipschitz condition in x.

Admissible Controls: Feed back controls u(t) = u(xt ) are not the
right class of controls because the equation x = f (x, u(x)) only has
a unique solution under strict conditions on u(.). Let
t = last jump time before t.

n(t) = x(t)
z(t) = t (t).
Then n(t), z(t) determine xt ; in fact, for fixed yY,
xt = Xn(t),z(t)
Zz
where Xn,z = n + f (Xn,s , y)ds.
o
Then admissible controls are Y-valued measurable functions

u(n(t), z(t)). By Caratheodorys theorem, the equation
Zz
Xn,z = n + f (Xn,s , u(n, s))ds
o
has a unique solution, and PD process is well defined for such u. We 59

will consider the three component process (xt , zt , nt ) for notational con-
venience.
Relaxed
R controls are functions : E R+ m1 (Y) such that (n, z)
(n, z, y)(dy; n, z) is a measurable function for (n, z) for all continuous
. Corresponding to , define
Z
f (x, n, z) = f (x, y)(dy, n, z)
Z
(x, n, z) = (x, y)(dy; n, z)
Z

Q (A, x, n, z) = Q(A, x, y)(dy, n, z).
Then we construct a PD process (xt , nt , zt ) corresponding to f , ,

Q in the usual way.
The strong problem is to minimise Jxo () over admissible relaxed
controls , where xo = (x, x, o), and

Z Z

Jxo () = E xo o (xt , y)(dy; nt , zt )dt

o Y

X Z
+ o (xt , y)(dy; nt , zt ) + T (x ) .
{t:xt E }
Here is the first hitting time of set ET .

Main Results:
Theorem 5. There exists an optimal (relaxed) control.
Theorem 6. The value function (x) = sup (x, x, o) where the supre-
mum is over all functions C 1 (E) such that
z (x, n, z) + min x (x, n, z) f (x, y) + (x, y)

yY
Z ! !
(, , o)Q(d, x, y) (x, n, z) + o (x, y) (x, n, z) Eo (3)
0
(Z )
(x, n, z) min (, , o)Q(d, y, x) + o (x, y) (x, n, z) E (4)
yY
(x, n, z) T (x), xET (5)

60 and E is the space of triplets (x, n, z).
Theorem 7. There exists a sequence k satisfying (3), (4), (5) above
such that o is optimal if and only if
Z n o
k (x, n, z) + x k (x, n, z) f (x, y) + (x, y)
Y
57
"Z #
k k
(, , 0) Q(d, x, y) (x, n, z)
+ o (x, y)o (dy, n, z) o in Ll (Qoo ). (6)

Z

Z

k

o

(, , o)Q(d, x, y) + o (x, y)
(dy, n, z)

Y Eo
k (x, n, z) 0 in Ll (Qo ) (7)

k
(x, n, z) T (x) 0 in L1 (QoT ). (8)
The measures Qoo , Qo and QoT are defined as follows.

Denote xt = (xt , nt , zt ). For A Eo ,
Z

Qoo (A) = E xoo A ( xt )dt
o
which is a measure on Eo and is called potential measure of xt . 61

X

Qo (A) = E xoo A ( xt )
t
where A E .

QoT (A) = P xoo [ xT A]
for A ET .
Comparing with deterministic case, the necessary and sufficient con-
dition there was that (xt , t ) is optimal if and only if
Z n o
i
t (xt , t) + ix (xt , t) f (xt , t, u) (xt , t, u) t (du) o in Ll (t0 , t1 ).
The probability measure corresponding to t is Dirac measure on

x(.) and Qoo (A) is the time spent by x(.) in A. Thus the conditions stated
are a direct generalization of the deterministic ones.
Remark 4. Note that if we define

Z
Qoo (A) = E x A ( xs )ds
o
then for any positive measurable function g,

Z Z
E x g( xs )ds = g()Qoo (d);
o
E
o
X
for, if g( x) = ci Ai ( x)
i
62 then
Z X Z
E x g( xs )ds = ci E x Ai ( xs )ds
o i 0
X

= ci Q (Ai )
Zi
= g() Q(d).
Eo
The general case follows by monotone convergence.
Remark 5. The Qi are potentials of additive functionals t is an ad-

ditive functional if o and1
t+s = t + s ot
t, pt , I(t) are some example of additive functionals.
The potential of an additive functional is an operator

Z
U g( x) = E x g( xs )d s .
o
Here QoO , Qo , QoT correspond precisely to this with t = t, pt , I(t>T )

respectively.
The Weak Problem:
1
t is the shift operator on the space of right continuous functions: (t w)s = t+s
59
The deterministic weak problem involved the fact that
Zt1
(x1 , t1 ) (xo , to ) = (t + x f )ds
to
for any C 1 . The stochastic equivalent of this is Dynkins formula.

To get this in the appropriate form, define operators Ay , By as follows.
Ay (x, n, z) = z (x, n, z) + x (x, n, z) f (x, y)

Z
+ (x, y) ((, , o) (x, n, z))Q(d, x, y)
EO
Z
and By(x, n, z) = (, , o)Q(d, x, y) (x, n, z)
E
for (x, n, z) E . Then the Dynkin formula on the interval (o, ) is 63

E x (x , n , z ) ( x)

Z Z

= E x Ay (xt , nt , zt )(dy; nt , zt )dt

o Y

Z Z

+ By(xt , nt , zt )(dy, zt , nt )dpt

o Y
Z Z

= Ay (x, n, z)(dy; n, z)Qo (dx, dn, dz)
Eo Y
Z Z
+ By(x, n, z)(dy; x, z)Qo (dx, dn, dz)
Ed Y
Now Z

E x (x , n , z ) = (x, n, z)QT (dx, dn, dz).
ET
So we can express the Dynkin formula as follows:

Z
( xo ) = L( x, y)M (d x, dy)
EY
where L(x, n, z, y) = ET (x, n, z) + Eo Ay (x, n, z) + E BY (x, n, z).
\ Z Z

M (S 1 S 2 ) = QT S 1 ET + (dy; n, z)Qo (dx, dn, dz)
T
S1 Eo S 2
Z Z

+ (dy; n, z)Q (dx, dn, dz).
T
S1 E S 2
64 The cost for the relaxed control is

Z
Jxo () = ( x, y)M (d x, dy).
EY
The following supplementary assumption is required.
inf Jxo () = inf Jxo ()

uu uo
for some c > o and u is the set of relaxed controls,

uc = u : [ + p ] c
R
with this assumption the weak problem is to minimize dM over
EY
measures M m1+c (E Y) (where ma is the set positive measures of
total mass less than or equal to a) such that
1. M = Mo + M + MT
where MT m1 (ET ).
M m(E Y)
MO m(Eo Y).
61
R
2. ( xo ) = LdM, C 1 (E).
65 From this point on, the development follows the Vinter-Lewis argu-
ments closely. We reformulate the weak problem as a convex optimiza-
tion problem by incorporating the constrains in the cost function and
obtain the characterization of optimality by studying the dual problem.
The reader is referred to Vermes [25] for the details.
Remark 6. The optimality condition involves the measures QoO , Qo , QoT

corresponding to o . These can be computed from the following system
of equations.
o
A h( x) + T Eo = 0, x EO
o
B h( x) + T E = 0, x E
x ET .
h( x) + T E (x),
T
Then
Qo () = h( xo ).
Example 1. If Eo , then Dynkins formula says

Z
o
h( xo ) = E xO ( xs )ds
O
o
= QO ().
The results outlined above are the first general results on optimal
control of PD processes. Obviously much work remains to be done;
natural next steps would be to determine necessary conditions for op-
timality of Pontrjagin type; to develop computational methods; and to
study optimal stopping and impulse control for PD processes. For
some related work, see van der Duyn Schouton [29], Yushkevich [30]
and Rishel [31].
Part II
Filtering Theory
63
0. Introduction 65
0 Introduction
Suppose {xt } is a signal process which represents the state of a system, 66
but cannot be observed directly. We observe a related process {yt }. Our
aim is to get an expression for the best estimate of xt , given the history
of {yt } upto time t.
In Section 1, we give quick derivations of the Kalman filter for
the linear systems, and nonlinear filtering equations, that of Fujisaki,
Kallianpur and Kunita and Zakais equation for unnormalized condi-
tional density (Kallianpur [19], Davis and Marcus [8]). In section 2,
we will study pathwise solutions of differential equations. In section
3, we will study the Robust theory of filtering as developed by Clark
[5], Davis [10] and Pardoux [20]. Here the above filtering equations are
reduced to quasi-deterministic form and solved separately for each ob-
servation sample path. Also, we will look here into some more general
cases of filtering developed by Kunita [17], where the existence of con-
ditional density functions is proved using methods related the theory of
Stochastic flows.
1 Linear and Nonlinear Filtering Equations

Kalman Filter (Davis [11])
Suppose that the signal process xt satisfies the liner stochastic differ- 67
ential equation
dxt = Axt dt + cdVt (1)
where Vt is some Wiener process. Observation yt is given by
dyt = H xt dt + dWt (2)
where Wt is a Wiener process independent of Vt . Assume xo N(o, Po ).
To get a physical model, suppose we write (2) as
dyt dWt
= H xt +
dt dt
dWt dyt
then corresponds to white noise and is the physical obser-
dt dt
vation.
66
The filtering problem is to calculate best estimate of xt given

(ys , s t). There are two formulations for Kalman filter.
(a) Strict Sense: If (Vt , Wt ) are Brownian motions then (xt , yt ) is a
Guassian process. Then xt = E[xt |ys , s t] is the best estimate
in the sense of minimizing E(xt z)2 over all yt -measurable, square
integrable random variables z, where
yt = {Y s , s t}.
Because of normality, xt is a liner function of (ys , s t).
68 (b) Wide Sense Formulation: Do not suppose Vt , Wt are normally dis-

tributed. Just suppose that the ith coordinates Vti , Wti are uncorrelated
and EVti V si = ts; EWti W si = ts, i.e., V i , W i are orthogonal incre-
ment processes. Now look for the best linear estimate of xt given
(ys , s t). This will coincide with E(xt |yt ) in strict sense case.
Calculating xt is a Hilbert space projection problem. The random
variables we consider belong to Lo2 (, F, P) which is a Hilbert space
with inner product (X, Y) = EXY, where o denotes the elements are
of zero mean. For any process, say yt define H y = L(yt , t o), the
linear span of yt ; this is a linear subspace. Then if z denotes the
projection of z onto H y , then
kz zk = miny kz Uk.
UH
Let xt be projection of xt onto Hty = L(ys , s t). Then the Innova-

tions process t is defined by
dt = dyt H xt dt. (3)
The Innovations process t has the following properties:

(i) t is an orthogonal increments process.
y
(ii) Ht = Ht .
(t )
R

(iii) Ht = g(s)d s , g L2 [o, t] .
o
1. Linear and Nonlinear Filtering Equations.... 67
69 Then xt satisfies the linear equation
d xt = A xt dt + P(t)H dt
xo = 0 (4)
where the error covariance P(t) = E(xt xt )(xt xt ) , ( denotes the

transpose).
P(t) satisfies the Riccati equation

d
P(t) = AP(t) + P(t)A + CC P (t)HH P(t)
dt
P(o) = Po = Cov(xo ).
The above equation (4) is the Kalman Filter.

Derivation of Kalman Filter equation: From properties (ii), (iii) we
know
Zt
xt = g(t, s)d s
o
for some g such that

Zt
g2 (t, s)ds < .
o
Now using projection, xt xt s , s t. So
E xt s = E xt s
t
Z

= E g(t, u)dvu s

o
Zt
= g(t, u)du.
o
Hence
d
g(t, s) = E xt s .
ds
68
Write innovations process as 70
dt = H xt dt + dWt where xt = xt xt .
Zs

E xt s = E(xt x u ))H du.
o
Now
Zt
xt = (t, u)xu + (t, r)CdVr
u
where is the transition matrix of A. So
Zs
E xt s = (t, u)( xu x u )H du
o
Zs
= (t, u)P(u)H du
o
g(t, s) = (t, s)P(s)H .
So
Zt
xt = (t, s)P(s)H d s .
o
But this is the unique solutions of (4).
Important Points:
(1) It is a recursive estimator.
(2) In the strict sense version xt is a sufficient statistic for the condi-
tional distribution of xt given (ys , s t), since this distribution is
N( xt , P(t)) and P(t) is nonrandom.
Exercise 1.1 (Constant Signal). Let xt = with E() = 0, Var () = 2

and
dyt = dt + dWt
71 with independent of Wt . Show directly by projection that

1
t = yt .
t + 12
Now show that the Kalman filter gives the same result.
Nonlinear Filtering
Suppose signal xt is a Markov process and observation yt is
given by
dyt = h(xt )dt + dWt ,
generally h is a bounded measurable function (extra smoothness condi-
tion will be added later). Assume that for each t, xt and (Wu Wv ), u, v
t are independent, which allows for the feedback case. Our objective
is to calculate in recursive from the estimates of xt . to do this, it is
necessary to compute the condition of xt given
yy = {ys , s t}.
The Innovations Approach to Nonlinear Filtering

This approach was originally suggested by Kailath for the linear
case and by Kailath and Frost for nonlinear filtering. The definitive
formulation of the filtering problem from the innovations standpoint was
given by Fujiskki, Kallianpur and Kunita [18].
Innovations Processes: Consider process yt satisfying 72
dyt = zt dt + dWt , t [o, T ] (5)
where Wt is Brownian motion and assume
ZT
E z2s ds < (6)
o
and the feedback condition is satisfied. Let
zt = E[zt |yt ].
70
More precisely zt is the predictable projection of zt onto yt . The

innovations process is then
dt = dyt zy dt. (7)
Note (i): t is a Brownian motion w.r.t. yt , i.e., t is a yt martingale and

y
< >t . If Ft = { s , s t}, the question is whether Ft = Ft . It has
been shown that in general, this is not true. But if (i) holds and (zt ), (Wt )
y
are independent, then Allinger-Mitter proved that Ft = Ft .
Note (ii): All yt -martingales are stochastic integrals w.r.t. (t ), i.e., if
Mt is a yt -martingale, then there is a g such that
ZT
g2s ds < a.s.
o
and
Zt
Mt = gs dS .
o
y y
This is true even if Ft , Ft , but note that (gs ) is adapted to Ft , not
necessarily to Ft .
73 A General Filtering Formula: Take an Ft -martingale nt , process (t )
satisfying
ZT
E | s |2 ds <
o
and Fo measurable random variable o with Eo2 < .

Now define an Ft semi-martingale t by
Zt
t = o + s ds + nt . (8)
o
Since < W, W >t = t, we have

Zt
< n, W >t = s ds
o
for some t and for any martingale nt . Let
t = E[t |yt ].
Then t satisfies the following stochastic differential equation

Zt Zt
t = o + s ds + [d
s z s + s z s + s ]d s . (9)
o o
Proof. Define
Zt
t = t o = s ds
o

Then t is a yt - martingale. So there is some integrable function

such that
Zt
t = s d s . (10)
o
Now we will identity the form of nt , using ideas of Wong [28]. Using 74
(5) and (8) and I to formula,
Zt Zt
t yt = o yo + s (zs ds + dW s ) + ys (as ds + dns )
o o
Now calculate t yt using (7) and (10),

Zt Zt Zt
t yt = o yo + s (zs ds + dvs ) + ys (as ds + s dvs ) + s ds.
o o o
72
Now for t s, h i
E t yt t yt |ys = 0.
So t
Z

E (dz
u u ) z
u u + u u du|y s = 0.

s
Let
V(u) = d
u zu u zu + u u .
Then V(u) is predictable process and

"Z t #
E V(u)du|F s = 0.
s
This is, Z
V(u)dudP = 0s, t s, A F s .
A[s,t]
The class of sets A[s, t] generates P, the predictable -field. Hence

V(u, ) = 0 a.e. dt dP. Hence the result.
75 Formula (9) is not a recursive equation for t . Still we can use it to
obtain more explicit results for filtering of Markov processes. Let (xt )
be a Markov process and A, D(A) be generator, i.e., for f D(A) then
Zt
f
Ct = f (xt ) f (xs ) A f (xu )du
s
is a martingale. Suppose
Zt
f
< C , W >t = Z f (xs )ds
o
for some function Z f . Introduce the notation

Y
( f ) = E[ f (xt )|yt ]
t
f
Now apply (9) with t = f (xt ); A f (xs ) = s , Ct = nt and zt = h(xt )
to get Fujisaki-Kallianpur-Kunita filtering formula
Zt Zt

t ( f ) = o ( f ) + s (A f )ds + s (D f ) s (h) s ( f ) dvs (11)
o o
where
D f (x) = Z f (x) + h(x) f (x).
If we interpret t as the conditional distribution of xt given yt , so
that Z
t ( f ) = f (x)t (dx) = E[ f (xt )|yt ],
then (11) is a measure-valued stochastic differential equation, and gives

an infinite-dimensional recursive equation for filtering.
Exercise 1.2. Derive the Kalman filter from the Fujisaki-Kallianpur- 76

Kunita equation.
The Unnormalized (Zakai) Equations:

Introduce a new probability measure Po on (, F) with t [o, T ] by
T
Z ZT
dPo 1
= exp h(xs )dW s h2 (xs )ds .
dP 2
o o
Since h is bounded, Po is probability measure and (yt ) is a Po -

Brownian motion. Also
f f
< Ct , y >t =< Ct , w >t
Zt
= Z f (xs )ds.
o
f
Note that, in general, Ct is a semi-martingale under Po but < ., . > is
invariant under absolutely continuous change of measure. Also if Z = o,
74
then xt has the same distribution under either measure. Let

T
Z ZT
dP 1
T = = exp h(xs )dys h2 (xs )ds .
dPo 2
o o
Let Eo denote the expectation under Po . Then it can be calculated

that under measure Po ,
t ( f ) = E[ f (xt )|yt ]
Eo [ f (xt )At |yt ]
=
Eo [At |yt ]
t ( f )
=: .
t (1)
77 Then t ( f ) is an unnormalized conditional distribution since t (1)
does not depend on f . To obtain an equation satisfied by t , we need a
semi-martingale representation for t (1). First we have
dt = h(xt )t dyt (12)
i.e.,
Zt
t = 1 + h(xs ) s dys
o
also t is a (Ft , Po ) martingale. Then as before
t = Eo [t |yt ]
is a yt -martingale, so there exists some yt -adapted integrand t such that
Zt
t = 1 + s dys (13)
o
To identify t , we use the same technique as in deriving the FKK

equation. Calculate using (12) and I tos rule,
Zt Zt Zt
t yt = t dys + ys s h(xs )dys + s h(xs )ds.
o o o
Calculating using (13) and Itos rule,

Zt Zt Zt
t yt = s dys + ys s dys + s ds.
o o o
Now
Eo [t yt t yt |ys ] = o for t s,
so we get
d t ) := Eo [t h(xt )|yt ].
t = (t)h(x
So (13) becomes ag 78
Z
t = 1 + s s (h)dys (14)
This has a unique solution

t
Z Zt
1
t = exp s (h)dys 2s (h)ds
2
o o
= t (1)
Theorem 1.1. t ( f ) satisfies the Zakai equation
dt ( f ) = t (A f )dt + t (D f )dyt (15)
o ( f ) = o ( f ) = E[ f (xt )].
Proof. Direct calculation using (11), (14) and the fact that
t ( f ) = t t ( f ).

Corollary 1.1. There is a one-to-one relation between Zakai equation

and FKK equation, in that whenever t satisfies Zakai equation, f ( f )/
Q
t (1) satisfies (11), and whenever t ( f ) satisfies (11),
t
Z Zt
1
t ( f ) exp s (h)dys s (h)ds
2
o o
satisfies Zakai equation.

76
The Zakai Equation with Stratonovich Integrals: 79

Recall
Zt Zt
1
us dvs := us dvs + < u, v >t
2
o o
where o denotes a Stratonovich Stochastic integral and u and v are con-
tinuous semi-martingales. We have to calculate < .(D f ), y >t . From
Zakai equation
dt (D f ) = t (AD f )dt + t (D2 f )dyt .
So
d < .(D f ), y >t = t (D2 f )dt.
So the Stratonovich version of the Zakai equation is
1
dt ( f ) = t (A f )dt + t (D f )odyt t (D2 f )dt
2
= t (L f )dt + t (D f )odyt
where
1
L f (x) = A f (x) D2 f (x).
2
Application to Diffusion Process:
Consider a process xt Rd satisfying
j
d f (xt ) = Xo f (xt )dt + X j f (xt )odBt (16)
for arbitrary smooth f , where Xo , . . . , Xr are vector fields on Rd we sup-
pose that t = j t for some constants 1 , . . . , r
80 Note that A is the generator of xt under measure P (not Po ). This is
given by
1X 2
A f (x) = Xo f (x) + X f (x).
2 j j
Proof. Rewrite (16) in Ito form. Replace f by Xk f in (16).

dXk f (xt ) = Xo Xk f (xt )dt + Xi Xk f (xt )odBtj .

Then
d < Xk f, Bk >t = Xk2 f (xt )dt.
Then Ito version of (16) is
!
1X 2 j
d f (xt ) = Xo + X j f (xt )dt + X j f (xt )dBt .
2
So
1X 2
A = XO + Xj
2
and2
Zt
f j
Ct = X j f (xs )dB s .
o
Proposition 1.1. For Z given by

Z
< C f , W >t = Z f (xs )ds.
with Z f = j X j , Z is a vector field.
Proof.
Z
f
d < C ,W > = d < X j f dB j , W >
= X j f d 
= j X j f (xt )dt.
So 81
D=Z+h
= i Xi + h.

2
We sometimes use the convention of implied summation over repeated indices.
78
Proposition 1.2. There exist vector fields Yo Y1 , . . . Yr such that

1 1X 2 1
A D2 = Y j + Yo Dh.
2 2 j 2
Proof.
D2 f = (i Xi + h)( j X j f + h f )
= i j Xi X j f + i Xi (h f ) + j hX j f + h2 f
= i j Xi X j f + hZ f + Dh f.
Let = (1 2 , . . . r ) and suppose I is nonnegative definite.

Write (I ) = and let X = (X1 , . . . , Xr ) .
X
X X = Xi2 i j Xi X j .
So define Y = X. Then Yi s are vector fields and
1 1X 2 1
A D2 = Y hZ + Xo Dh
2 2 i i 2
1X 2 1
= Y + Yo Dh
2 i i 2
where Yo = Xo hZ. It remains to check that I o. Take Rr

with || = 1. Then
(I ) = 1 ( )2
1||2
0.
82 Since ||2 1, for

X X
 t = i t
i i
= ||2 t
P
So 2 t = EUWt , where U = i
i Bi .
||4 t2 EU 2 t = ||2 t2 .
So we have Zakai equation in Itos form
dt ( f ) = t (A f )dt + t (D f )dyt
With vector fields X0 , X1 , . . . Xr and in Stratonovich form
dt ( f ) = t (L f ) + t (D f )odyt
With vector Fields Y0 , Y1 , . . . Yr plus a 0th -order term. Now we

investigate what sort of process is xt under the measure Po .
Proposition 1.3. Under Po , xt satisfies the equation

j
d f (xt ) = Yo f (xt )dt + Z f (xt )odyt + Y j f (xt )odbt (17)
where b1 , . . . br are independent standard Brownian motions indepen-

dent of yt
Proof. Recall the Girsanov transformation. Under P, B1, . . . Br are inde-

pendent and t = j t. Now
!
dPo 1
= exp Mt < M, M >t
dP 2
j
where M is a P-martingale. Under Po , Bt t is a martingale 83
and hence a Brownian motion. Here
Zt
Mt = h(xs )dW s ;
o
d = j h(xt )dt.
j

80
So under Po ,
j j
dVt = dBt + j h(xt )dt
are independent Brownian motions, but V j is not independent of yt , in
fact < V j , y >t = j t.
j j
Now define bt = Vt j yt , then = 0, and this implies b j , y
are independent. But the b j are now not independent. In fact,

j k t for k , j
j k
t =

1 ( ) t for k = j
j 2
So
h i
t = t
= (I )t.

Let (I ) = as before and define bt = 1 bt .
Then bt = by and t = It. So
d f (xt ) = Xo f (xt )dt + X j f (xt )odBi
= Xo f (xt )dt + X j f (xt )o( j h(xt )dt + j dyt + dbtj )
j
= (Xo f (xt ) hZ f (xt ))dt + Z f (xt )odyt + Y j f (xt )odbt
j
= Yo f (xt )dt + Z f (xt )odyt + Y j f (xt )o d bt
84 where Yo f = Xo f hZ f and Y = X.
The so called Kallianpur- Striebel Formula gives the solution t of
the Zakai equation as a function space integral in the following form,
where xt is functional of (y, b1 , . . . br ).
t
Z Zt
1
t ( f ) = Eo f (xt ) exp h(xs )dys h (xs )ds yt
2
2
o o
i.e., as a function of y, we have

t
Z Z Zt
1 2 1 r
t ( f )(y) = f
t(x ) exp h(x s )dy s h (x s )ds w (db ) . . . w (db )
2
Cr [o,t] o o
where w (db) is the Wiener measure on C[, T ].

2. Pathwise Solutions of Differential Equations 81
2 Pathwise Solutions of Differential Equations

Consider the Doss- Sussman construction for the equation
x = b(x) + g(x)w (1)

x(o) = x
where w C (R+ ). Let (t, x) be the flow of g, i.e.,

(t, x) = g((t, x))
t
(o, x) = x.
If b = o, then it is immediate that the solution of (1) is 85
xt = (w(t), x).
If b , o, then the solution of (1) is of the form
xt = (w(t), (t)) (2)
where (t) satisfies some ODE. With x(t) defined by (2),
x(t) = g(x(t))w(t) + x (w(t), (t))(t)
and we require that
x (w(t), (t))(t) = b((w(t), (t))).
So x(t) satisfies (1) if (t) satisfies
= ( x (w(t), (t)))1 b((w(t), (t)))

(o) = x
Coordinate-free form: Let

df
Xo f (x) = b(x)
dx
df
X1 f (x) = g(x)
dx
82
t (x) = (w(t), x).
Define
1
(t Xo ) f (x) = Xo ( f ot1 )(t (x)).
Then the equation for (t) can be expressed as
d 1
f (t ) = (t Xo ) f (t )
dt
o = x,
86 for
1 d
(t Xo ) f (t ) = b(t (x)) f ( 1 (x))|t (x)
dx t
d d
= b(t (x)) f (x) (t1 (x))|t (x)
dx dx
d
= b(t (x)) f (x)( x (x))1 . ()
dx
Since 1 ((x)) = x and so
d 1 d
( ((x))) (x) = 1.
dx dx
When x Rd , then
X
d
f (x)
Xo f (x) = bi (x)
i=1
xi
X
d
f (x)
X1 f (x) = gi (x)
i=1
xi
Then () is of the form
X
d

1
(t Xo ) f (t ) = bi (x) { f t1 }|t (x)
i=1
xi
X
d X
d
f (x) ( 1 ) jt
= (x).
j=1 i=1
xi xi
So the same results apply for xt Rd , but generally not for more
than one input, i.e., for vector w(t).
Interpretation: xt defined by (2) makes sense for any w(.) C(R+ ).
In particular, if w(t) is a sample path of Brownian motion, then what 87
equation does xt satisfy?
Answer: the Stratonovich equation
dxt = b(xt )dt + g(xt )odwt (3)
Exercise 2.1. Expand xt given by (2) using Itos calculus and show that
it satisfies (3).
The following examples show that the pathwise solution idea cannot
generally be extended to multi-input equations.
Example 2.1. Let
x = g1 (x)w1 + g2 (x)w2
x(o) = x.
The solution should be of form
xt = h(w1t , w2t ). (4)

Then with h1 (w1 , w2 ) = h(w1 , w2 ) etc., we have
w1
xt = h1 w1 + h2 w2
h1 (w1t , w2t ) = g1 oh(w1t , w2t )

h2 (w1t , w2t ) = g2 oh(w1t , w2t )
and
h12 (w1t , w1t ) = g1x oh.h2 = (g1x oh)(g2 oh)

84
h21 (w1t , w2t ) = (g2x oh)(g1 oh).
So we must have
g1 g2x = g2 g1x .
88 Define the Lie bracket [X1 , X2 ] = X1 X2 X2 X1 . Now
!
1 d 2df
X1 X2 f = g g
dx dx
g1 g2x fx + g1 g2 fxx .
Therefore
[X1 , X2 ] f = (g1 g2x g2 g1x ) fx .
So a necessary condition for (4) to hold is that
[X1 , X2 ] = 0
i.e., X1 X2 = X2 X1 .
Exercise 2.2. Consider

X
n
X = gi (x)wi .
i=1
Let i (t, x) be the flow of gi and ti (x) = i (wit , x). Then show that
xt = t1 ot2 o . . . otn
if [X i , X j ] = 0 i, j.
With one input, ||wn w|| 0 implies xnt xt , where || || is the

sup norm. But with inputs w1 , w2 , the solution map generally is not
continuous.
89 Example 2.2 (Sussmann [23]). Let t [0, 1] and
xn = Axn w1,n + Bxn w2,n

x(o) = xo
where A, B are n n matrices with
[A, B] = AB BA , 0.
" #
n j1 j
Partition [0, 1] into n equal intervals I j = , , j = 1, 2, . . . , n.
n n
Partition each I nj into four equal intervals I nj,i , i = 1, 2, 3, 4. Define w1,n
to be equal to 4n1/2 for t I nj,1 to 4n1/2 for I nj,2 , and to zero for all other
t. Similarly, let w2,n be equal to 4n1/2 for t I nj,2 to 4n1/2 for t I nj,4 ,
and to zero for all other t.
Then Z t
i,n
w (t) = wi,n (s)ds, i. = 1, 2.
o
Clearly wi,n
converges to zero uniformly as n , i = 1, 2. Let
s=n1/2 , then
xn (1/n) = eBs eAs eBs eAs xo

= e xo .
We use the Baker-Campbell -Hausdorff formula eA eB = eC where
1 1
C = A + B + [A, B] + {[[B, A], A] + [[B, A], B]} +
2 12
we get
1
= [B, A] + o(1/n).
n
So 90
xn (1) = en xo
= e([B,A]+o(1/n)) xo
Hence
lim xn = et[B,A] xo .
n t
86
3 Pathwise Solution of the Filter Equation

Consider the equation
d f (xt ) = Yo f (xt )dt + Z f (xt )odyt + Yi f (xt )odbit .
To express this pathwise in (yt ), let (t, x) be the integral curve of Z

and t (x) = (y(t), x). Define t as follows:
j
d f (t ) = (t1 Yo ) f (t )dt + (t
1
Y j ) f (t ) dbt
o = x.
Then
xt = t ot
= (y(t), (t)).
The generator of is
1 X 1 2
At = t
1
Yo + Yj .
2 j t
91 The Kallianpur-Striebel Formula: Recall

" Z t Z !#
(b) 1 t 2
t ( f ) = E f (xt ) exp h(xs )dys h (xs )ds
o 2 o
" Z t Z !#
1 t
= E (b) f (xt ) exp h(xs )odys Dh(xs )ds
o 2 o
where D = Z + h.
Notation: For any diffeomorphism : M M, : C (M) C (M)
is given by
f (x) = f o(x) = f ((x)).
So
f (xt ) = t f (t ),
3. Pathwise Solution of the Filter Equation 87
" Z t Z !#
(b) 1
and t ( f ) = E t f (t ) exp s h( s )odys Dh( s )ds
o 2 o
The next step is to remove ody. Define

Z t
H(t, x) = s h(x)ds.
o
Calculate H(yt , t ) using Stratonovich calculus
dH(yt , t ) = t h(t )odyt + t

1 1
Yo Hys ( s )ds + t Yi Hys ( s )odbis .
Notation:
gs (x) = H(y(s), x)
yj = t
1
Yj
Z s !
B s f (x) = s f (x) exp u h(x)du .
o
Finally, we get 92
t ( f ) = E (b) [By(t) f (t )ot (y)]
where the multiplicative functional ts is given by

"Z t Z
j 1 t 2
ts (y) = exp Y j gu (u )dbu (Y ) gu (u )du
s 2 s j
Z t Z #
1 t
Yo gu (u )du Dh(nu )du
s 2 s u
So t ( f ) is now pathwise in y with t ( f ) : C[o, t] R and (t ( f )/

t (1)) is a version of E[ f (xt )|yt ]. Now we want to compute t ( f ) re-
cursively.
(a) Multiplicative Functional Approach:
Let (xt ) be a Markov process with extended generator (A, D(A)).
The associated semigroup on B(E) is
88
T s,t f (x) = E s,x [ f (xt )]

ts (s t) is a multiplicative functional (m. f.) of (xt ) if ts is {xu , s
u t}-measurable and for r s t,
rt = rs ts .
Corresponding to ts there is a semigroup defined by

T s,t f (x) = E s,x f (xt )ts .
In particular,

T s,t 1 = E s,x [ts ].
93 It is a Markov (or Sub-Markov) semigroup when
E s,x [ts ] = 1 ( 1).
If (xt ) is a homogeneous Markov process, ts is a homogeneous m.f.

if
ts = t+r
s+r
o r
where r xt = xt+r .
Then
ts = ots o s .
So denoting t = ot , the m. f . property is
t+s = t . s o t .
Now we want to find the generator of
T t f (x) = E x ( f (xt ) t ).
Suppose for the moment that t 1, t. Then t is monotone

decreasing. In this cases t corresponds to killing at rate (dt /t ).
It is possible to construct an -subprocess which is a Markov process
xt such that
E x [ f (xt )] = T t f (x).
See Blumenthal and Getoor [1]. Define the extended generator of

T to be the extended generator of xt , i.e.,
Zt
f (xt ) f (xo ) A f (xs )ds
o
is a local Martingale if f D(A ). This says (excluding stopping) 94

Zt

E f (xt ) f (xs ) A f (xu )du|F s = 0

s
Zt
or E xs [ts f (xt )] f (xs ) E xs us A f (xu )du = 0.
s
So equivalently, f D(A ) if
Zt
t ( f (xt ) f (x) s A f (xs )ds
o
is a local Martingale (P x ) for every x.

This characterizes A even when the condition t 1 is not satis-
fied, so we adopt it as our definition.
!
Rt
Example 3.1. Let t = exp V(xs )ds where V B(E). Take
o
f D(A) and compute
d(t f (xt )) = t A f (xt ) dt + t d Mt f V(xt ) f (xt ) t dt.

Zt Zt
t f (xt ) f (x) = s [A f (xs ) V(xs ) f (xs )] ds + s dM s f.
o o
So
A f (x) = A f (x) V(x) f (x).
90
a(xt )
Example 3.2. Let t = where a D(A) and a(x) > o x. Then 95
a(xo )
1
T t g(x) = T t (a f )(x).
a(x)
Exercise 3.1. Show that

1
A f (x) = A(a f )(x).
a(x)
Now suppose xt satisfies

j
d f (xt ) = Xo f (xt )dt + X j f (xt ) o dwt .
Take g Cb (E) and define

t
Z Zt X
1
t = exp X j g(xu )dwuj (X j g(xu ))2 du . (1)
2
o j o
If we define
dPx
= t ,
dP x
then
j j
dwt = dwt + X j g(xt )dt
is a Px - Brownian motion. Thus
X
j
d f (xt ) = (Xo f (xt ) X j g(xt )X j f (xt ))dt + X j f (xt )o dwt .
j
Now t is a m. f . of xt (as will be verified below) and
E x [ f (xt )t ] = E x [ f (xt )].
96 So
X 1X 2
A f (x) = Xo X j g(xt )X j f + X f.
j
2 j j
The three examples here are related by

Zt Zt
j
X j g(xs )dw s = g(xt ) g(x) Ag(xs )ds.
o o
Using this in (1), we see that t factors
t = t t
with
1 X
V(x) = Ag(x) + (X j g(x))2
2 j
a(x) = eg(x) .
So
1 X
A f (x) = eg A(eg f ) Ag (X j g)2 f (x).
2 j
So
X 1 X
eg A(eg f ) = A f (X j g)X j f Ag + (X j g)2 f.
2 j
Exercise 3.2. Verify that this result is correct by direct calculation of

eg A(eg f ).
We have the unnormalized solution of filtering problem as
t ( f ) = E[Byt f (t )ot (y)]
for y C[o, T ]. Here t is a diffusion, ot (y) is a m.f. of t . Now

t
Z Zt X
1 2
(Y j gu (u )) du
s j
t (y) = exp Y j g(u )dbu
2
s s j

Zt X Zt
1 2 1
exp
(Y j gu (u )) du (Y j )2 gu (u )du
2 2
s j s
92

Zt Zt
1
Yo gu (u )du u Dh(u )du .

2
s s
97 This factors ts
into product of a Girsanov m. f . and a Feynman-
Kac m. f . Hence the corresponding generator is

X 1 X 1
Y j gt Y j f + (Y j gt )2 A gt t Dh f.
y
At f = At f
j
2 j 2

Proposition 3.1. Ayt f = Byt A 21 D2 B1
yt .
Proof. This can be verified by a straightforward but somewhat lengthy

calculation, using the expansion for eg Aeg obtained previously, once
has obtained an expression for B1
t . Recall that Bt is defined by
Zt
Bt f (x) = f ((t, x)) exp h((u, x))du.
o

It is a group of operators with generator D = Z + h. The inverse B1

t
is given as follows. Let g(x) = Bt f (x), then
f (x) = B1
t g(x)
t
Z

= g((t, x)) exp h((u, (t, x)))du

o
t
Z

= g( 1 (t, x)) exp h( 1 (s, x))ds .

o
98
Example 3.3 (Independent signal and noise). Take Z = 0, then
(t, x) = x and
!
y h(x)y(t) 1 2 y(t)h(.)
At f (x) = e A h (e f (.))(x)
2
1
= ehy(t) Aehy(t) f h2 f.
2
It is easy to see that this must be the right formula. The calculations
y
have been carried out for arbitrary y C[o, T ] but At depends only on
y y
y(t). So At = At where y(s) y(t) (t fixed). Now
t
Z Zt
1
t ( f )(y) = E f (xt )eh(xt )yt exp y(s)dh(xs ) h2 (xs )ds
2
o o

Zt
1
= E f (xt )eh(xt )yt exp y(t)h(xs ) + y(t)h(xo ) h2 (xs )ds
2
o

Zt
exp (y(t)h(x )) 1
= E f (xt )eh(xt )yt . exp h2 (xs )ds .
t
exp(y(t)h(xo )) 2
o
So we have separated into two functionals and the result follows.

Direct Solution of Zakai Equation: We will consider a slight general-
ization from the original Zakai equation. Define
1X 2
L= Y + Yo + ho
2 j j
D=Z+h
where Yi , Z are smooth vector fields; h, ho are Cb functions (Previously, 99

1 R
we had ho = Dh). Write h f, i for f d and consider the measure-
2
valued equation
dh f, t i = hL f, t idt + hD f, i dyt (2)
where h f, .i = f (x) i.e., o = x . The solution can be expressed as

follows: Define xt by
j
d f (xt ) = Z f (xt ).dyt + Yo f (xt )dt + Y j (xt ) dbt , xo = x,
where b j are Brownian motion independent of y. Then the solution is

" Z t Z t !#
b
t ( f ) = E f (xt ) exp ho (xs )ds + h(xs ) dys .
o o
94
Kunita [17] show that this solution is unique if coefficients are

smooth and bounded. Now the question is whether t has a density.
Theorem 3.1.
h f, t i = hByt f, t i (3)
where
Byt f (x) = h f, t i
and t , t satisfy the equations
d < f, t > =< D f, t > odyt (4)

< f, 0 > = f (x)
d < f, t > =< Byt L B1
yt , t > dt. (5)
100 Proof. If L = o, then (4) is the same as (2); so the solution of (4) is
t
Z

< f, t >= f (xt ) exp h(xs ) dys

o
where xt satisfies
d f (xt ) = Z f (xt )odyt .

But this has pathwise solution xt = (yt , x). The previous definition
of B was yt
Z

Byt f (x) = f ((yt , x)) exp h((u, x))du .

o
Now, yt
Z

d h((u, x))du = h((yt , x))odyt .

o
So (3) holds with Bt defined as before. Now
d < Byt f, t > = d < Byt f, t > + < Byt f, t > dt

= Byt L Bl
yt Byt f, t > dt+ < Byt D f, t > odyt
by (5) and (4).
=< Byt L f, t > dt+ < Byt D f, t > odyt .
This verifies (2).

Proposition 3.2. Suppose t has a density function qt (z, x) 3 . Then for 101
t > o.t has density
yt
Z
1
t (V) = qt ( (yt , V)x) exp h( (s, V))ds |
1 1
(yt , V) | (6)
V
o
d 1
where | | is the Jacobian of the map V 1 (yt , V).
dV
Proof. If t has a density qt , then
Z
< f, t > = Byt f (z)qt (z, x)dz
Z Zyt
= f ((yt , z)) exp( h((u, z))du)qt (z, x)dz.
o
Changing the variable to V = (yt , z) gives (6).
Theorem 3.2 (Bismut [2]). t has C -density if the Yi are smooth

vector fields, i.e., coefficients are bounded with bounded derivatives of
all orders and Y1 , . . . , Yn satisfy the restricted Hormander condition
H :- Consider vector fields Yi , [Yi , Y j ], [[Yi , Y j ], Yk ] . . .. At each x the
restrictions of these vector fields to x span T x (M).
P
In local coordinates Yi = , . . . etc. So the condition says
bi (x)
i xi
the vectors b etc. span Rd at each x. Recall, Byt LB1 st
yt = At + (1 and
3
Here z is the dummy variable and x refers of the initial condition in (2), i.e.,
0 = x
96
0th order terms) X

A = 1
(t Yi )2 +
i
Now
t1 1 1
[Yi , Y j ] = [t Yi , t Y j ] etc .
102 So if Yi satisfy the Hormander condition, then (t 1 Y ) satisfies it.

i
Hormanders own result requires coefficients to be C in (t, x). Here
the coefficients are continuous (but not even C 1 ) in t. Bismuts version
of Malliavin calculus shows that the result still holds with this degree of
smoothness in t.
In the filtering problem, the signal process involved vector fields
X1 , X2 , . . . Xn , X0 and Y = X, where is nonsingular if | |< 1. Then
X = 1 Y. So
X h i
. . . [[Xi1 , Xi2 ]Xi3 ] . . . Xik = c j . . . [[Y1 , Y2 ], Y3 ] . . . Yin, j .
j
So if the X Lie brackets span Rd then there must be a collection of

Y brackets which also span Rd . The Hormander condition for X with
| |< 1 implies the existence of density.
The Case of Vector Observations: Let dyi = hi (xt )dt + dWt0,i , W 0,i are
independent Brownian motions. will now be a matrix,
i j t =< W 0,i , W 0, j > .
Consider the following cases. (a) Independent signal and noise:

Here i j = 0i, j. Then whole theory goes through unchanged.
103 Then (2) becomes
X
d < f, t >= < hi f, t > odyit
i
with solution

X
< f, t > = exp yi (t)h(x) f (x)
i
Y
= exp(yi (t)h(x)) f (x)
i
So this gives pathwise solution as before.

Another Point of View: The Kallianpur-Striebel formula is

X Zt X Zt
(x) 1
t ( f ) = E f (xt ) exp i
hi (xs )dys hi (xs )ds
2
2
i i i o

Y X Zt
(x)
= E f (xt ) eyi (t)h(xt )
exp yi (s)dhi (xs )

i i o

X Z t
1
= h2i (xs )ds .
2 i

o
(b) The General Case: Here we have no pathwise theory (except

under very artificial conditions) but the same theory goes through a.s.
(Wiener measure). There is no continuous extension to the whole of
C p [o, T ]. In this case, equation (2) becomes
X
d < f, t >= < Di f, t > odyit
i
where Di = Zi + h and Zi is a vector field .
A pathwise solution only exists if Di s commute, which is very arti-

ficial. But, as before, the solution can be expressed as

X Zt

< f, t >= f (xt ) exp hi (xs )odyis (7)

i 0
where xt satisfies 104

X
d f (xt ) = Di f (xt )odyit
i
98
xo = x.
Regard as the operator mapping f < f, t >. Then stochastic

flow theory (Elworthy [13], Kunita [17], Bismut [2] says that if Di s
have smooth coefficients then x xt (x, ) is a diffeomorphism a.a. ,
and so the inverse map x1 t (x) exists. We have to calculate t . Gener-
1
alize (7) slightly to

X Zt
i
< f, s,t >= f (xt , t) exp hi (xr , r) odyr

i s
X
d f (xt , t) = Di f (xt , t) o dyit t s
i
xs = x.
Proposition 3.3 (Kunita [17]).

X Zt
i
1 1
s,t ( f (x)) = f (xs,t (x)) exp hi (xr,t (x)) dyr
1
(8)

i s
where od means backwards Stratonovich integral. Here define

fields Fr,t r t, by
n o
Fr,t = yiu yiv , r u, v t, i = 1, 2, . . . , d .
Rt
Then r dyi is a well defined backward I to integral if r is a back-
s
105 ward semimartingale w.r.t. (Fr,t )rt . Then the Stratonovich integral is
defined as usual. If r is continuous, then
Z X n !
tkn + tk+1
i
r=
r ody yikn yitn
k
2 k+1 k
So 1
s,t (x) is well defined by (8). Now verify that
Zt
i
hi x1
r,t (p) dyr p=x s,t (x)
s
Zt
= hi (xs,r )odyir .
s
This checks that 1

s,t ( s,t (x)) = x.
Now all remaining calculations go through as before but only a.s.

(Wiener measure).
More general results on existence of densities have been obtained
by Bismut and Michel [3]
Bibliography
[1] R.M. BLUMENTHAL and R.K. GETOOR, Markov processes and 106
Potential theory, Academic press, New York 1968.
[2] J.M. BISMUT, Martingales, The Malliavin calculus and Hypoel-
licity under general Hormander conditions: Z. Wahrscheinlichkeit-
stheorie 56 (1981) 469-505.
[3] J.M. BISMUT and D. MICHEL, Diffusions conditionelles I, II, J.
Functional Analysis 44 (1981), 174-211, 45 (1982) 274-292.
[4] R. BOEL, P. VARAIYA and E. WONG, Martingales on jump pro-
cesses. I Representation results, SIAM J. Control 13 (1975) 999-
1021.
[5] J.M.C. CLARK, The design of robust approximations to the
Stochastic differentials equations of nonlinear filtering, Communi-
cation Systems and Random Process Theory, ed. J.K. Skwirzynski,
NATO Advanced Study Institute Series, Sijthoff and Noordhoff,
Alphen aan den Rijn, 1978.
[6] M.H.A. DAVIS, The representation of Martingales of jump pro-
cesses, SIAM J. Control and Optimization 14 (1976) 623-638.
[7] M.H.A. DAVIS, Piecewise-deterministic Markov Processes, a gen-
eral class of non-diffusion Stochastic models, to appear in J. Royal
Statist. Soc. (B).
[8] M.H.A. DAVIS and S.I. MARCUS, An Introduction to nonlin-
ear filtering, Stochastic Systems: The Mathematics of Filtering
101
102 BIBLIOGRAPHY
and Identification and Applications ed. M. Hazewinkel and J.C.

Willems, 1981
107 [9] M.H.A. Davis, Pathwise nonlinear filtering, Stochastic Systems,

The Mathematics of Filtering and Identification and applications
ed. M. Hazewinkel and J.C. Willems, Reidel (1981) 505-528.
[10] M.H.A. Davis, On multiplicative functional transformation arising

in nonlinear filtering theory, Z. Wahrscheinlichkeits theorie Verw.
Geb. 54, (1980) 125-139.
[11] M.H.A. DAVIS, Linear Estimation and Stochastic Control, Cham-

pan & Hall, London 1977.
[12] C. DELLACHERIE and P.A. MEYER, Probablity and Potentials,

North-Holland 1978.
[13] K.D.ELWORTHY, Stochastic dynamical systems and their flows,

Stochastic Analysis, ed. A. Friedman and M. Pinsky, Acadenic
Press, New York 1978.
[14] R.J. ELLOTT, Stochastic Calculus and Applications Springer-

Verlag 1982.
[15] N. EL Karovi, Les aspectes probabilistes du controle stochastique,

in Lecture Notes in Mathematics, 876, Springer-Verlag 1981
[16] B.V. GNEDENKO and I. I. KOVALENKO, Introduction to the the-

ory of Mass Service[Russian] Moscow, 1966.
[17] H. KUNITA, Densities of a measure-valued process governed by a

stochastic partial differential equation, Systems and Control letters
1 (1981) 100-104.
[18] H. KUNITA, Stochastic partial differential equations connected

with nonlinear filtering, Nonlinear filtering and Stochastic Con-
trol, Lecture Notes in Mathematics 972, Springer-Verlag, Berlin
1982.
BIBLIOGRAPHY 103
108 [19] G. KALLIANPUR, Stochastic Filtering theory, Springer-Verlag,

1980.
[20] E. PARDOUX, Equations du filtering non-linear, de la prediction

et du lissage, Stochastics 6 (1982) 193-231.
[21] S. R. PLISKA, Controlled jump processes, Stochastic processes

and their applications 3 (1979) 259-282.
[22] Z. ROSBERG, P.P. VARAIYA and J. C. WALRAND, Optimal

Control of Service in Tandem Queues, IEEE Trans. Automatic
Control 27 (1982) 600-610.
[23] H.J. SUSSMAN, On the gap between deterministic and Stochastic

ordinary differential equations, The Annals of Probability 6 (1978)
19-41.
[24] D. VERMES, Optimal dynamic control of a useful class of ran-

domly jumping processes, report pp-80-15. International Institute
for Applied Systems Analysis, Laxenburg, Australia, 1980.
[25] D. VERMES, Optiomal Control of Piecewise-deterministic Pro-

cesses, Preprint, Bolyai Institute, University of Szeged, Hungary
1983 (to appear in Stochastics)
[26] R.B. VINTER and R.M. LEWIS, A necessary and sufficient con-
dition for optimality of Dynamic Programming type, Making no
apriori assumption on the controls, SIAM J. Control and Optimiza-
tion 16 (1978) 571-583.
[27] A.D. WENTZELL, A course in the theory of Stochastic processes,

McGraw-Hill Inc. 1981.
[28] E. WONG, Recent Progress in Stochastic Processes-a survey.

IEEE Trans. Inform. Theory, IT-19, 1973, 262-275.
[29] F.A.VAN DER DUYA SCHOUTON, Markov Decision Drift Pro- 109
cesses, Mathematics Centre Tract, Amsterdam 1983.
104 BIBLIOGRAPHY
[30] A.A. YUSHKEVICH, Continuous-time Markov decision pro-

cesses with interventions, Stochastics 9 (1983) 235-274.
[31] R. RISHEL, Dynamic programming and minimum principles for

systems with jump Markov disturbances, SIAM J. Control 13
(1975) 338-371.

Lectures On Stochastic Control and Nonlinear Filtering: M. H. A. Davis

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Lectures On Stochastic Control and Nonlinear Filtering: M. H. A. Davis

Uploaded by

Copyright:

Available Formats

Lectures on

Stochastic Control and Nonlinear Filtering

Tata Institute of Fundamental Research

Lectures delivered at the

Published for the

ISBN 3-540-13343-7 Springer-Verlag, Berlin. Heidelberg.

No part of this book may be reproduced in any

Printed by M. N. Joshi at The Book Centre Limited,

differential equations can be dealt with by stochastic flow theory through

I Stochastic Jump Processes and Applications 1

1 Stochastic Jump Processes 3

2 Optimal Control of pd Processes 45

Stochastic Jump Processes

Stochastic Jump Processes

Stochastic jump processes are processes with piecewise constant paths. 1

Figure 0.1: Arrival time of customers

1 Martingale Theory for Jump Processes

Let z0 , z be fixed elements of X. Define the path (xt )t0 by

Then the probability structure on the process is determined by either

(ii) for each k = 1, 2, . . . , conditional distribution of (S k , Zk | T ki , i =

If T = , let Z = z , a fixed point of X.

Define the probability space (, F, P) as the canonical space for T ,

i.e., ((R+ X)U{(, z )}, B(R+ ) S , {(, z )}, )

where is a probability measure on

((R+ X)U{(, z )}, B(R+ ) S , {(, z )}).

The random function (xt ) generates the increasing family of fields 4

Ft = Completion of Ft0 with all F

Proposition 1.1. T is not an Ft0 stopping time, but T is an Ft stopping

Proof. Let A = {Z = z0 } and K be any set in X. Then

It can be seen that

The stopped - field FT is given by

Definition 1.2. A process (Mt ) is an Ft -martingale if E | Mt |< and

Proposition 1.2. If Mt is a local martingale and S is a stopping time

Proposition 1.3. Suppose is an Ft -stopping time. Then there exist

Proof. If is a stopping time, then (T t) Ft , t. But if T is

(T t) (]t, ] X) ]t, [X for some t R+ .

But [t, ] X is an atom of Ft . This contradicts the fact that is a

Exercise 1.1. Suppose = t0 T . Show that

(ii) F = {xs , S 0}.

Definition 1.3. For A S , define

Proposition 1.4. Suppose (Mt )t0 is an Ft local martingale. Then

(a) if c = or c < and F(c) = 0, then Mt is a martingale on [0, c[.

(b) if c < , F(c) > 0, then (Mt ) is a uniformly integrable martingale.

Proof. (a) If k T a.s. for some k, then

Mtk = Mtk T = MtT = Mt .

So Mt is a u.i. martingale. Hence suppose P[k < T ] > 0 for all

k T = tk T for some fixed tk

8 and tK < c because of (). Also tk c since k .

Hence Mttk is a u.i. martingale. So (Mt )t<c is a martingale.

(b) c < , F(c ) > 0.

P(T = c) > 0; so it must be the case that tk = c for some k. Oth-

p(t, A) = I(tT ) I(ZA)

(a) Integrals w.r.t. p(t, A): Suppose Nt is a counting process. Since

Clearly g L1 (p) if and only if

(b) Integrals w.r.t. p(t, A): 11

12 Following is the main result of this section, which gives an integral

Proposition 1.7. All Ft -local martingales are of the form

for some g Lloc

We need the following result.

Lemma 1.1. Suppose (Mt )t>0 is u.i.Ft martingale with M0 = 0. Then

E | h(T, Z) |< (1)

Proof. If (Mt ) is a u.i. martingale, then Mt = E[ | Ft ] for some F -