Mean field simulation for Monte Carlo integration

Part II : Feynman-Kac models

P. Del Moral
INRIA Bordeaux & Inst. Maths. Bordeaux & CMAP Polytechnique
Lectures, INLN CNRS & Nice Sophia Antipolis Univ. 2012
Some hyper-refs

Mean field simulation for Monte Carlo integration. Chapman & Hall - Maths & Stats [600p.] (May 2013).

Feynman-Kac formulae, Genealogical & Interacting Particle Systems with appl., Springer [573p.] (2004)

Particle approximations of Lyapunov exponents connected to Schr

odinger operators and Feynman-Kac
semigroups. ESAIM-P&S (2003) (joint work with L. Miclo).

Coalescent tree based functional representations for some Feynman-Kac particle models. Annals of
Applied Probability (2009) (joint work with F. Patras, S. Rubenthaler).

On the concentration of interacting processes. Foundations & Trends in Machine Learning [170p.]
(2012). (joint work with P. Hu & L.M. Wu)

More references on the websitehttp://www.math.u-bordeaux1.fr/delmoral/index.html [+ Links]

Contents
Introduction
Mean field simulation
Interacting jumps models
Feynman-Kac models
Path integration models
The 3 keys formulae
Stability properties
Some illustrations ( Part III)
Markov restrictions
Multilevel splitting
Absorbing Markov chains
Quasi-invariant measures
Gradient of Markov
semigroups

Some bad tempting ideas

Interacting particle
interpretations
Nonlinear evolution equation
Mean field particle models
Graphical illustration
Island particle models
Concentration inequalities
Current population models
Particle free
energy/Genealogical tree
models
Backward particle models

Introduction
Mean field simulation
Interacting jumps models
Feynman-Kac models
Some illustrations ( Part III)
Some bad tempting ideas
Interacting particle interpretations
Concentration inequalities

Introduction

Mean field simulation

=
Universal adaptive & interacting sampling technique

Part I
I

2 types of stochastic interacting particle models:

Diffusive particle models with mean field drifts

[McKean-Vlasov style]

Interacting jump particle models

[Boltzmann & Feynman-Kac style]

Part II Interacting jumps models

Interacting jumps = Recycling transitions =

Discrete generation models ( geometric jump times)

Equivalent particle algorithms

Sequential Monte Carlo

Particle Filters
Genetic Algorithms
Evolutionary Population
Diffusion Monte Carlo
Quantum Monte Carlo
Sampling Algorithms

Sampling
Prediction
Mutation
Exploration
Free evolutions
Walkers motions
Transition proposals

Resampling
Updating
Selection
Branching-selection
Absorption
Reconfiguration
Accept-reject-recycle

Equivalent particle algorithms

Sequential Monte Carlo

Particle Filters
Genetic Algorithms
Evolutionary Population
Diffusion Monte Carlo
Quantum Monte Carlo
Sampling Algorithms

Sampling
Prediction
Mutation
Exploration
Free evolutions
Walkers motions
Transition proposals

Resampling
Updating
Selection
Branching-selection
Absorption
Reconfiguration
Accept-reject-recycle

More lively buzzwords:

Bootstrapping, spawning, cloning, pruning, replenish, cloning, splitting,
condensation, resampled Monte Carlo, enrichment, go with the winner,
subset simulation, rejection and weighting, look-a-head sampling, pilot
exploration,..

A single stochastic model

Particle interpretation of Feynman-Kac path integrals

Introduction
Feynman-Kac models
Path integration models
The 3 keys formulae
Stability properties
Some illustrations ( Part III)
Some bad tempting ideas
Interacting particle interpretations
Concentration inequalities

Feynman-Kac models
FK models = Markov chain Xn En functions Gn : En [0, [

Y
1
Gp (Xp ) dPn with Pn = Law(X0 , . . . , Xn )
dQn :=

Zn
0p<n

Flow of n-marginals

n (f ) = n (f )/n (1)

with n (f ) := E f (Xn )

Y
0p<n

Gp (Xp )

Feynman-Kac models
FK models = Markov chain Xn En functions Gn : En [0, [

Y
1
Gp (Xp ) dPn with Pn = Law(X0 , . . . , Xn )
dQn :=

Zn
0p<n

Flow of n-marginals

n (f ) = n (f )/n (1)

with n (f ) := E f (Xn )

Gp (Xp )

0p<n

Evolution equations :
with Mn Markov trans. of Xn and Qn+1 (x, dy ) = Gn (x)Mn+1 (x, dy )
n+1 = n Qn+1

and n+1 = Gn (n )Mn+1

The 3 keys formulae

Time marginal measures = Path space measures:

n (fn ) = E fn (Xn )

Gp (Xp )

0p<n

[ Xn := (X0 , . . . , Xn ) & Gn (Xn ) = Gn (Xn )] = n = Qn

I

Normalizing constants (= Free energy models):

Y
Y
p (Gp )
Zn = E
Gp (Xp ) =
0p<n

0p<n

The last key

I

Backward Markov models

Qn (d(x0 , . . . , xn )) 0 (dx0 )Q1 (x0 , dx1 ) . . . Qn (xn1 , dxn )

with
Qn (xn1 , dxn )

:=
hyp

Gn1 (xn1 )Mn (xn1 , dxn )

Hn (xn1 , xn ) n (dxn )
1
n+1 (dx) =
n (Hn+1 ( , x)) n+1 (dx)
n (Gn )

If we set
Mn+1,n (xn+1 , dxn ) =

n (dxn ) Hn+1 (xn , xn+1 )

n (Hn+1 ( , xn+1 ))

then we find the backward equation

n+1 (dxn+1 ) Mn+1,n (xn+1 , dxn ) =

1
n (dxn ) Qn+1 (xn , dxn+1 )
n (Gn )

The last key (continued)

Qn (d(x0 , . . . , xn )) 0 (dx0 )Q1 (x0 , dx1 ) . . . Qn (xn1 , dxn )

n+1 (dxn+1 ) Mn+1,n (xn+1 , dxn ) n (dxn ) Qn+1 (xn , dxn+1 )

Backward Markov chain model :

Qn (d(x0 , . . . , xn )) = n (dxn ) Mn,n1 (xn , dxn1 ) . . . M1,0 (x1 , dx0 )
with the dual/backward Markov transitions
Mn+1,n (xn+1 , dxn ) n (dxn ) Hn+1 (xn , xn+1 )

Stability properties
I

Transition/Excursions/Path spaces
0
Xn = (Xn0 , Xn+1
)

0
Xn = X[T
n ,Tn+1 ]

Xn = (X00 , . . . , Xn0 )

Continuous time models Langevin diffusions

Z tn+1
Xn = X[t0 n ,tn+1 ] & Gn (Xn ) = exp
Vt (Xt0 )dt
tn

OR Euler schemes (Langevin diff.

Metropolis-Hasting moves)
OR Fully continuous time particle models
Schrodinger operators
d
t (f ) = t (LVt (f ))
dt

with LVt = L0t + Vt

Important observation:


Z t
Z t
0
t (1) = E exp
Vs (Xs )ds = exp
s (Vs )ds
0

with t = t /t (1)

Stability properties
I

Change of probability measures-Importance sampling (IS) Sequential Monte Carlo methods (SMC) :
For any target probability measures of the form
Qn+1 (d(x0 , . . . , xn+1 ))

Qn (d(x0 , . . . , xn )) Qn+1 (xn , dxn+1 )

0 (dx0 )Q1 (x0 , dx1 ) . . . Qn+1 (xn , dxn+1 )

0
0
s.t. Qn+1 (xn , )  Mn+1
(xn , )
and any Markov transition Mn+1

Gn (xn , xn+1 )

=
=

Target at time (n + 1)
Target at time (n) Twisted transition
dQn+1 (xn , )
(xn+1 )
0
dMn+1
(xn , )

.
.

Stability properties
I

Change of probability measures-Importance sampling (IS) Sequential Monte Carlo methods (SMC) :
For any target probability measures of the form
Qn+1 (d(x0 , . . . , xn+1 ))

Qn (d(x0 , . . . , xn )) Qn+1 (xn , dxn+1 )

0 (dx0 )Q1 (x0 , dx1 ) . . . Qn+1 (xn , dxn+1 )

0
0
s.t. Qn+1 (xn , )  Mn+1
(xn , )
and any Markov transition Mn+1

Gn (xn , xn+1 )

=
=

Target at time (n + 1)
Target at time (n) Twisted transition
dQn+1 (xn , )
(xn+1 )
0
dMn+1
(xn , )

.
.

0
Feynman-Kac model with Xn = Xn0 , Xn+1

Y
1
Qn =
Gp (Xp ) dPn with Pn = Law(X0 , . . . , Xn )

Zn
0p<n

Introduction
Feynman-Kac models
Some illustrations ( Part III)
Markov restrictions
Multilevel splitting
Absorbing Markov chains
Quasi-invariant measures
Gradient of Markov semigroups
Some bad tempting ideas
Interacting particle interpretations
Concentration inequalities

Markov restrictions
I

Confinements: Xn random walk Zd A

&

Gn := 1A

Qn = Law ((X0 , . . . , Xn ) | Xp A, 0 p < n)

Zn = Proba (Xp A, 0 p < n)

SAW :

Xn = (Xp0 )0pn

0
& Gn (Xn ) = 1Xn0 6{X00 ,...,Xn1
}

Qn = Law (X00 , . . . , Xn0 ) | Xp0 6= Xq0 , 0 p < q < n

Zn = Proba Xp0 6= Xq0 , 0 p < q < n

Multilevel splitting
Decreasing level sets An , with B non critical recurrent subset.
Tn := inf {t Tn1 : Xt0 (An B)}
Excursion valued Feynman-Kac model:
Xn = (Xt0 )t[Tn ,Tn+1 ]

& Gn (Xn ) = 1An+1 (XT0 n+1 )

Qn



0
= Law X[T
| X 0 hits An1 before B
0 ,Tn ]

Zn

= P(X 0 hits An1 before B)

Absorbing Markov chains

absorption (1Gn )

exploration Mn+1

b c X c
Xnc Enc X
n+1
n

Qn = Law((X0c , . . . , Xnc ) | T abs. n)

& Zn = Proba T abs. n

Quasi-invariant measures : (Gn , Mn ) = (G , M) & M -reversible

1
log P T abs. n 'n = top spect. of Q(x, dy ) = G (x)M(x, dy )
n
[Frobenius theo] Q(h) = h = eigenfunction (ground state)
P(Xnc dx | T abs. > n) 'n

1
h(x) (dx)
(h)

Doob h-processes X h
Qn (d(x0 , . . . , xn )) P((X0h , . . . , Xnh ) d(x0 , . . . , xn )) h1 (xn )
with
M h (x, dy ) =

M(x, dy )h(y )
1 1
h (x)Q(x, dy )h(y ) =

M(h)(x)

Invariant measure h = h M h & Additive functionals

F n (x0 , . . . , xn ) =

X
1
f (xp ) = Qn (F n ) 'n h (f )
n+1
0pn

If G = G depends on some R

f :=

log G

log 'n
log Zn+1
= Qn (F n )

n + 1

Gradient of Markov semigroups

Xn+1 (x) = Fn (Xn (x), Wn )

X0 (x) = x Rd

Pn (f )(x) := E (f (Xn (x)))

First variational equation

Xn
Xn+1
(x) = An (x, Wn )
(x) with
x
x

A(i,j)
n (x, w ) =

Fni ( , w )
(x)
x j

Random process on the sphere U0 = u0 Sd1

Xn
x (x) u0
Un+1 = An (Xn , Wn )Un /kAn (Xn , Wn )Un k = X
n (x) u0
x

Feynman-Kac model Xn = (Xn , Un , Wn ) & Gn (x, u, w ) = kAn (x, w ) uk

Pn+1 (f )(x) u0 = E
F
(X
)
G
(X
)
p
p
| {zn+1}

0pn
f (Xn+1 ) Un+1 |
{z
}
n
k X
x (x) u0 k

Introduction
Feynman-Kac models
Some illustrations ( Part III)
Some bad tempting ideas
Interacting particle interpretations
Concentration inequalities

Bad tempting ideas

I.i.d. weighted samples Xni

N
Y
1 X Y
Zn := E
Gp (Xp ) ' ZnN :=
Gp (Xpi )
N
0p<n

i=1 0p<n

or in terms of killing-absorption models

Zn = P (T n) ' ZnN :=

1 X
1T i n
N
1iN

Bad tempting ideas

I.i.d. weighted samples Xni

N
Y
1 X Y
Zn := E
Gp (Xp ) ' ZnN :=
Gp (Xpi )
N
i=1 0p<n

0p<n

or in terms of killing-absorption models

Zn = P (T n) ' ZnN :=

1 X
1T i n
N
1iN

Example : Xn simple RW Z , Gn = 1[10,10] (killed at the boundary)

n = n() : ZnN = 0

Bad tempting ideas

I.i.d. weighted samples Xni

N
Y
1 X Y
Zn := E
Gp (Xp ) ' ZnN :=
Gp (Xpi )
N
i=1 0p<n

0p<n

or in terms of killing-absorption models

Zn = P (T n) ' ZnN :=

1 X
1T i n
N
1iN

Example : Xn simple RW Z , Gn = 1[10,10] (killed at the boundary)

n = n() : ZnN = 0

and

N E

2 !
ZnN
1
Zn

1 Zn
Zn
1

' Proba(Xp A, 0 p < n)1 = P (T n)

Our objective
Find an unbiased estimate ZnN s.t.

N E

ZnN
1
Zn

2 !
c n

using the multiplicative formula

Y
Y
Gp (Xp ) =
p (Gp )
E
0p<n

0p<n

And estimating/Learning each (larger) terms in the product

p (Gp ) ' pN (Gp ) with

pN =

1 X
ni
N
1iN

Introduction
Feynman-Kac models
Some illustrations ( Part III)
Some bad tempting ideas
Interacting particle interpretations
Nonlinear evolution equation
Mean field particle models
Graphical illustration
Island particle models
Concentration inequalities

Flow of n-marginals [Xn Markov with transitions Mn ]

n (f ) = n (f )/n (1)

with n (f ) := E f (Xn )

Gp (Xp )

0p<n

m (n (1) = Zn )
Nonlinear evolution equation :
n+1

Zn+1

= n (Gn ) Zn

Gn (n )Mn+1

Nonlinear m.v.p. = Law of a Markov X n (perfect sampler)

n+1

n+1 (n )

n (Sn,n Mn+1 ) = n Kn+1,n = Law(X n+1 )

Examples related to product models

1
n (dx) :=
Zn

n
Y

)
hp (x)

(dx) with hp 0

p=0

2 illustrations:
hp (x) = e (p+1 p )V (x)

hp (x) = 1Ap+1 (x) Ap

1 n V (x)
e
(dx)
Zn
1
1An (x) (dx)
= n (dx) =
Zn
= n (dx) =

For any MCMC transitions Mn with target n , we have

n+1 = n+1 Mn+1 = hn+1 (n )Mn+1 Feynman-Kac model

Mean field particle model

McKean Markov chain model
n+1 = n Kn+1,n = Law(X n )

Markov chain n = (ni )1iN EnN

ni

i
n+1
Kn+1,nN (ni , dx) with nN =

1 X
ni
N
1iN

and the (unbiased) particle normalizing constants

Y
N
Zn+1
= nN (Gn ) ZnN =
pN (Gp )
0pn

Mean field particle model

Mean field FK simulation ni

i
n+1
Kn+1,nN = Sn,nN Mn+1

m
Sequential particle simulation technique (SMC)
Gn -acceptance-rejection with recycling Mn+1 -propositions

Mean field particle model

Mean field FK simulation ni

i
n+1
Kn+1,nN = Sn,nN Mn+1

m
Sequential particle simulation technique (SMC)
Gn -acceptance-rejection with recycling Mn+1 -propositions
Genetic type branching particle algorithm (GA)
G selection

M mutation

n
n
n
bn
n+1

Mean field particle model

Mean field FK simulation ni

i
n+1
Kn+1,nN = Sn,nN Mn+1

m
Sequential particle simulation technique (SMC)
Gn -acceptance-rejection with recycling Mn+1 -propositions
Genetic type branching particle algorithm (GA)
G selection

M mutation

n
n
n
bn
n+1

Reconfiguration Monte Carlo (particles

walkers) (QMC)

(Selection, Mutation) = (Reconfiguration, exploration)

Continuous time Feynman-Kac particle models

Master equation

t ( ) =

t ( )
= Law(X t )
t (1)

d
t (f ) = t (Lt,t (f ))
dt

Continuous time Feynman-Kac particle models

Master equation

t ( ) =

t ( )
= Law(X t )
t (1)

d
t (f ) = t (Lt,t (f ))
dt

(ex. : Vt = Ut 0)
Lt,t (f )(x) = L0t (f )(x) +
| {z }
free exploration

Z
Ut (x)
| {z }

(f (y ) f (x))

acceptance rate

t (dy )
| {z }

interacting jump law

m
Lt,t (f ) = L0t (f ) Ut f +
|
{z
}
Schr
odingers op.

Ut t (f )
| {z }

normalizing-stabilizing term

Particle model: Survival-acceptance rates Recycling jumps

Graphical illustration : n ' nN :=

1
N

1iN

ni

Graphical illustration : n ' nN :=

1
N

1iN

ni

Graphical illustration : n ' nN :=

1
N

1iN

ni

Graphical illustration : n ' nN :=

1
N

1iN

ni

Graphical illustration : n ' nN :=

1
N

1iN

ni

Graphical illustration : n ' nN :=

1
N

1iN

ni

Graphical illustration : n ' nN :=

1
N

1iN

ni

Graphical illustration : n ' nN :=

1
N

1iN

ni

Graphical illustration : n ' nN :=

1
N

1iN

ni

Graphical illustration : n ' nN :=

1
N

1iN

ni

Graphical illustration : n ' nN :=

1
N

1iN

ni

Graphical illustration : n ' nN :=

1
N

1iN

ni

Graphical illustration : n ' nN :=

1
N

1iN

ni

Graphical illustration : n ' nN :=

1
N

1iN

ni

Graphical illustration : n ' nN :=

1
N

1iN

ni

Graphical illustration : n ' nN :=

1
N

1iN

ni

Graphical illustration : n ' nN :=

1
N

1iN

ni

Graphical illustration : n ' nN :=

1
N

1iN

ni

Graphical illustration : n ' nN :=

1
N

1iN

ni

How to use the full ancestral tree model ?

hyp

Gn1 (xn1 )Mn (xn1 , dxn ) = Hn (xn1 , xn ) n (dxn )

Qn (d(x0 , . . . , xn )) = n (dxn )

Mn,n1 (xn , dxn1 ) . . . M1,0 (x1 , dx0 )

{z
}
|

n1 (dxn1 ) Hn (xn1 ,xn )

How to use the full ancestral tree model ?

hyp

Gn1 (xn1 )Mn (xn1 , dxn ) = Hn (xn1 , xn ) n (dxn )

Qn (d(x0 , . . . , xn )) = n (dxn )

Mn,n1 (xn , dxn1 ) . . . M1,0 (x1 , dx0 )

{z
}
|

n1 (dxn1 ) Hn (xn1 ,xn )

Particle approximation = Random stochastic matrices

N
QN
(xn , dxn1 ) . . . M1,0N (x1 , dx0 )
N
n (d(x0 , . . . , xn )) = n (dxn ) Mn,n1

How to use the full ancestral tree model ?

hyp

Gn1 (xn1 )Mn (xn1 , dxn ) = Hn (xn1 , xn ) n (dxn )

Qn (d(x0 , . . . , xn )) = n (dxn )

Mn,n1 (xn , dxn1 ) . . . M1,0 (x1 , dx0 )

{z
}
|

n1 (dxn1 ) Hn (xn1 ,xn )

Particle approximation = Random stochastic matrices

N
QN
(xn , dxn1 ) . . . M1,0N (x1 , dx0 )
N
n (d(x0 , . . . , xn )) = n (dxn ) Mn,n1

Ex.: Additive functionals

QN
n (fn ) :=

fn (x0 , . . . , xn ) =

1
n+1

0pn fp (xp )

X
1
nN Mn,n1
. . . Mp+1,pN (fp )
N
n+1
{z
}
|
0pn
matrix operations

4 particle estimates
I

Individuals ni almost iid with law

n ' nN =

1 X
ni
N
1iN

Path space models

Ancestral lines almost iid with law

Qn ' nN :=

1 X
Ancestral linen (i)
N
1iN

Backward particle model

N
QN
(xn , dxn1 ) . . . M1,0N (x1 , dx0 )
N
n (d(x0 , . . . , xn )) = n (dxn ) Mn,n1

Normalizing constants
Y
Y
N
pN (Gp )
Zn+1 =
p (Gp ) 'N Zn+1
=
0pn

0pn

(Unbiased)

Island models

Reminder : the unbiased property

Y
E fn (Xn )
Gp (Xp ) = E nN (fn )
0p<n

pN (Gp )

0p<n

= E Fn (Xn )

Gp (Xp )

0p<n

with the Island evolution Markov chain model

Xn := nN

and Gn (Xn ) = nN (Gn ) = Xn (Gn )

particle model with (Xn , Gn (Xn )) = Interacting Island particle model

Some key advantages

I

Mean field models=Stochastic linearization/perturbation technique

1
N
nN = n1
Kn,n1
+ VnN
N
N
with VnN ' Vn independent centered Gaussian fields .

n = n1 Kn,n1 stable Non propagation of local sampling errors

= Uniform control w.r.t. the time horizon

No burning, no need to study the stability of MCMC models.

Stochastic adaptive grid approximation

Nonlinear system

Simple and natural sampling algorithm.

Local conditional iid samples Stability of nonlinear sg

positive - beneficial interactions.

New concentration and empirical process theory

Introduction
Feynman-Kac models
Some illustrations ( Part III)
Some bad tempting ideas
Interacting particle interpretations
Concentration inequalities
Current population models
Particle free energy/Genealogical tree models
Backward particle models

Current population models

Constants (c1 , c2 ) related to (bias,variance), c finite constant
Test functions/observables kfn k 1, (x 0, n 0, N 1).
When En = Rd :
Fn (y ) := n 1(,y ]

and FnN (y ) := nN 1(,y ]

with y Rd

The probability of any of the following events is greater than 1 e x .

 N


n n  (fn ) c1 1 + x + x + c2
x
N
N
p



sup  pN p (fp ) c x log (n + e)/N

0pn

p
N

Fn Fn c d (x + 1)/N

Particle free energy/Genealogical tree models

Constants (c1 , c2 ) related to (bias,variance), c finite constant
(x 0, n 0, N 1).
The probability of any of the following events is greater than 1 e x



1
 log ZnN 1 log Zn  c1
 N
n
n

1+x +

 N


 n Qn (fn ) c1 (n + 1)
N

c2
x +
x
N

1 + x + x + c2

(n + 1)
x
N

with nN = Genealogical tree models := nN (in path space)

Backward particle models

Constants (c1 , c2 ) related to (bias,variance), c finite constant.

For any normalized additive functional fn with kfp k 1,
(x 0, n 0, N 1)
The probability of the following event is greater than 1 e x
r
 N



x
 Qn Qn (fn ) c1 1 (1 + (x + x)) + c2
N
N(n + 1)