Ehrhard Behrends

Introduction to Markov Chains

with Special Emphasis on Rapid Mixing

vieweg

Vll

Contents
Part I Finite Markov chains (the background)
1 Markov chains: how to start? Random walk, Markov chain, stochastic process, Markov process, Kolmogorov's theorem, Markov chains vs. Markov processes, exercises. 2 Examples of Markov chains Classical examples, Markov chains as graphs, shuffling, processes with a short memory, exercises. 3 How linear algebra conies into play &-step transitions and the fc'th power of the transition matrix, equilibrium distribution, the loss-of-memory phenomenon, exercises. 4 T h e fundamental notions in connection with Markov chains Closed subsets of a chain, irreducible chains, states which communicate, the period of a state, recurrent and transient states, exercises. 5 Transient states Fundamental matrix, running time until absorption, exercises. 6 A n analytical lemma The fundamental lemma of renewal theory, an example from renewal theory, exercises. 7 Irreducible Markov chains All states in an irreducible chain are positive recurrent, the equilibrium distribution, irreducible chains with an arbitrary period, the first passage time matrix, exercises. 8 Notes and remarks Summary of part I, history, Markov chains on countable state spaces and in continuous time, the ergodic theorem, exercises.

1
4

12

19

23

36

42

48

61

Part II Rapidly mixing chains

9 Perron-Frobenius theory Eigenvalues and eigenvectors of a stochastic matrix, the canonical form, exercises.

67
71

10 Rapid mixing: a first approach

The time reversal chain, reversible chains, an estimate of the mixing rate using eigenvalues, estimates which use the entries of the transition matrix directly, applications to renewal theory, exercises.

77

viii 11 Conductance Capacity, ergodic flow, conductance, chains defined by graphs, edge magnification, the method of canonical paths, applications, exercises. 12 Stopping times and t h e strong Markov property Partial information vs. sub-a-algebras, filtrations, stopping times, the strong Markov property, exercises. 13 Coupling methods The total variation distance, examples of couplings, the coupling inequality, coupled Markov chains, applications, the power of coupling methods, exercises. 14 Strong uniform times Stopping times which are compatible with the Markov property, strong uniform times, an application to shufflings, exercises. 15 Markov chains on finite groups I (commutative groups) Probability measures on groups give rise to Markov chains, the character group of a finite group, Fourier transform of functions and measures, the Fourier inversion formula, the Plancherel formula, convolutions, characterization of irreducible and aperiodic chains, rapid mixing on commutative groups, applications, exercises. 16 Markov chains on finite groups II (arbitrary groups) (Unitary) representations, irreducible representations, Schur's lemma, connecting matrices, duals of a group, properties of the coordinate functions, the Peter-Weyl theorem, examples of duals, Fourier transform of functions and measures, the Fourier inversion formula, the Plancherel formula, convolutions, characterization of irreducible and aperiodic chains, rapid mixing on arbirary groups, estimates which use matrix norms, selfadjointness or class measures, applications, exercises. 17 Notes and remarks Notes and remarks concerning part II, coupling from the past, exact simulation, cut-off-phenomenon.

Contents 91

102

108

121

128

142

168

Part III Rapidly mixing chains: applications

18 Random generation and counting Self-reducible problems, the solution tree, counting implies simulation and vice versa, simulation without counting, exercises. 19 Markov random fields Random fields, neighbourhood systems, Markov random fields, local characteristics, the chain associated with a family of local characteristics, a counterexample, exercises. 20 Potentials, Gibbs fields, and the Ising model Gibbs potentials and Gibbs fields, Gibbs fields are Markov random fields, the toric Ising model, calculation of the partition function (the transfer matrix method), Gibbs sampler, exercises.

171
174

183

195

IX

21 The Metropolis sampler and simulated annealing Proposal chain, Metropolis sampler, the spectral gap of the proposal chain vs. the gap of the Metropolis chain, simulated annealing, a logarithmic cooling schedule, exercises. 22 N o t e s and remarks Notes and remarks concerning part III, Monte-Carlo integration, volume estimation, image analysis.

209

220

Bibliography Index

223 229