Markov chains

Some parts of the lecture notes are borrowed

from Dr. Federico Ang, EEE 25, 1st Sem
2016-17
 Examples

EEE 25: Probability and Random Processes for EEE EEE Department, UP Diliman, QC
 Example

EEE 25: Probability and Random Processes for EEE EEE Department, UP Diliman, QC
 Examples
Communication Systems

Channel is Good Channel is Bad

EEE 25: Probability and Random Processes for EEE EEE Department, UP Diliman, QC
 Examples

EEE 25: Probability and Random Processes for EEE EEE Department, UP Diliman, QC
EEE 25 1st Sem AY 2016-2017 Federico M. Ang

Reference
• Chapter 7 of our Reference Textbook (Bertsekas
and Tsitsiklis)

• Sections 7.1-7.2

Lecture 18, Finite-state Markov chains 3

EEE 25 1st Sem AY 2016-2017 Federico M. Ang

Markov Processes Overview

• Markov chains: general class of RP
• Typically for modeling evolution of dynamical
systems: information sciences, queueing theory,
internet applications, games, physics, chemistry
• Basic concepts (temporal dependence: states)
• Short-term behavior
• Long-term behavior (prediction?)
• Steady-state
• Initial-state independence

Lecture 18, Finite-state Markov chains 8

EEE 25 1st Sem AY 2016-2017 Federico M. Ang

Checkout counter example

Markov process intro
• Discrete time

• Customer arrivals:
• Customer service times:
• “state” :

Lecture 18, Finite-state Markov chains 9

EEE 25 1st Sem AY 2016-2017 Federico M. Ang

Discrete-time Finite State

Markov Chains
• : state after transitions
- belongs to a finite set
- initial state either given or random
- transition probabilities:

• Markov property/assumption:

• Model: know states, transitions with probabilities

Lecture 18, Finite-state Markov chains 10
EEE 25 1st Sem AY 2016-2017 Federico M. Ang

Summary I

Lecture 18, Finite-state Markov chains 11

EEE 25 1st Sem AY 2016-2017 Federico M. Ang

-step transition probabilities

• State probabilities, given initial state :

• Key recursion:

• Random initial state:

Lecture 18, Finite-state Markov chains 12

EEE 25 1st Sem AY 2016-2017 Federico M. Ang

Summary II

Lecture 18, Finite-state Markov chains 13

EEE 25 1st Sem AY 2016-2017 Federico M. Ang

Example

Lecture 18, Finite-state Markov chains 14

EEE 25 1st Sem AY 2016-2017 Federico M. Ang

Convergence Questions
• Does converge to something?

• Does the limit depend on initial state?

Lecture 18, Finite-state Markov chains 15

EEE 25 1st Sem AY 2016-2017 Federico M. Ang

Recurrent, Transient States

• State is recurrent if “starting from , and from
wherever you can go, there is a way of returning
to “
• Otherwise, transient

• Recurrent class: recurrents only communicating

Lecture 18, Finite-state Markov chains 16
EEE 25 1st Sem AY 2016-2017 Federico M. Ang

Summary III

Lecture 18, Finite-state Markov chains 17

EEE 25 1st Sem AY 2016-2017 Federico M. Ang

Warm-Up Exercise

Lecture 19, Steady-state Behavior of Markov chains 6

EEE 25 1st Sem AY 2016-2017 Federico M. Ang

Periodic States in a
Recurrent Class
States in a recurrent class are periodic if they can
be grouped into groups where so that all
transitions from one group lead to the next group

Otherwise, aperiodic

Lecture 19, Steady-state Behavior of Markov chains 11

EEE 25 1st Sem AY 2016-2017 Federico M. Ang

Summary I

Lecture 19, Steady-state Behavior of Markov chains 12

EEE 25 1st Sem AY 2016-2017 Federico M. Ang

Steady-State Probabilities
• Do the converge to some ?
(independent of the initial state )
• Yes, if:
- recurrent states are all in a single class, and
- single recurrent class is not periodic
• Assuming “yes”, start from key recursion

- take the limit as the additional eq.:

Lecture 19, Steady-state Behavior of Markov chains 13

EEE 25 1st Sem AY 2016-2017 Federico M. Ang

Review: Steady-state Behavior

• Markov chain with a single class of recurrent
states and aperiodic (special term: ergodic); and
some transient states

• The ‘s can be found as the unique solution to

the balance equations

and
Lecture 20, Markov Chain Absorption, RP Stationarity 4
EEE 25 1st Sem AY 2016-2017 Federico M. Ang

Example

Lecture 19, Steady-state Behavior of Markov chains 14

EEE 25 1st Sem AY 2016-2017 Federico M. Ang

Summary II

Lecture 19, Steady-state Behavior of Markov chains 15

EEE 25 1st Sem AY 2016-2017 Federico M. Ang

Frequency Interpretations
• Balance equations

• (Long run) frequency being in :

• Frequency of transitions :

• Frequency of transitions into :

Lecture 19, Steady-state Behavior of Markov chains 16

EEE 25 1st Sem AY 2016-2017 Federico M. Ang

Consequential Interpretations

Lecture 19, Steady-state Behavior of Markov chains 17

EEE 25 1st Sem AY 2016-2017 Federico M. Ang

Birth-Death Processes

Lecture 19, Steady-state Behavior of Markov chains 18

EEE 25 1st Sem AY 2016-2017 Federico M. Ang

Birth-Death Processes

• Special case: and for all

- assume

- assume and

Lecture 19, Steady-state Behavior of Markov chains 19

EEE 25 1st Sem AY 2016-2017 Federico M. Ang

Absorbing States
• Recurrent state with

Lecture 20, Markov Chain Absorption, RP Stationarity 7

EEE 25 1st Sem AY 2016-2017 Federico M. Ang

Absorption Probabilities
• What is the probability that the chain
eventually settles in given it started in ?

• Unique solution from

Lecture 20, Markov Chain Absorption, RP Stationarity 8

EEE 25 1st Sem AY 2016-2017 Federico M. Ang

Summary I

If there are many absorbing states, the probability of eventually being in one
state GIVEN AN INITIAL STATE should sum to 1. For example, in the
previous slide, if you wish to solve for the case of s = 5, just subtract the
results for each initial state from 1.
Lecture 20, Markov Chain Absorption, RP Stationarity 9
EEE 25 1st Sem AY 2016-2017 Federico M. Ang

Expected Time to Absorption

• Find expected number of transitions until
reaching , given that the initial state is

• Unique solution from

Lecture 20, Markov Chain Absorption, RP Stationarity 10

EEE 25 1st Sem AY 2016-2017 Federico M. Ang

Summary II

Lecture 20, Markov Chain Absorption, RP Stationarity 11

EEE 25 1st Sem AY 2016-2017 Federico M. Ang

Mean First Passage Time,

Mean Recurrence Time
• Chain with one recurrent class; fix a recurrent
• Mean first passage time from to :

- unique solution to:

• Mean recurrence time of

- solution to:

Lecture 20, Markov Chain Absorption, RP Stationarity 12

EEE 25 1st Sem AY 2016-2017 Federico M. Ang

Summary III

Lecture 20, Markov Chain Absorption, RP Stationarity 13