EE266, Spring 2014-15

Professor S. Lall

EE266 Homework 3 Solutions

1. Second passage time. In this problem we will consider the following Markov chain.

Note that self-loops are omitted from this figure.

is

0.4 0.3 0 0.3

0 0.4 0 0.3

0.3 0 0.1 0
P =
0.3 0
0 0.4

0 0.3 0.3 0.3

0
0 0.3 0

The transition matrix for this chain

0
0
0.3 0

0.3 0.3
.
0 0.3

0.1 0
0.3 0.4

This matrix is given in passage_time_data.m so you dont have to type it into MATLAB.
We will compute several things about this Markov chain. Let the destination set be
the single state E = {1}, and let the initial state be i = 2. Recall that the first passage
time to the set E is
E = min{t > 0 | xt E}
The kth passage time for a Markov chain is defined recursively as
(k)

(k1)

E = min{t > E

| xt E},

(1)

where E = E .
(a) For the Markov chain above, compute and plot the distribution of the first passage
time conditioned on the initial state, i.e., compute
f (t) = Prob(E = t | x0 = i)
as a function of t.
1

(b) One may calculate the distribution of the second passage time by constructing a
new Markov chain, whose first passage time is equal to the second passage time
of the original chain. What is the transition graph of this chain?
(c) For t = 1, 2, . . ., use your construction to compute the distribution of the second
passage time, which is
(2)

s(t) = Prob(E = t | x0 = i).

Use your method to compute and plot this distribution also.
(d) Plot the sum of s(t) and f (t), and also plot Prob(xt E). Explain what you see.
(e) For a given state j, the recurrence time is the first passage time to the state j
given the initial condition x0 = j. This has distribution
r(t) = Prob({j} = t | x0 = j).
Let j = 1. Explain how to compute this, compute it, and plot it.
(f) Show that s = f r, where denotes convolution. Verify this numerically, and
interpret this result.
(g) Give a method to compute the distribution of the kth passage time, for any k.
How does your method scale with k?
Solution:
(a) To calculate the first passage time, we will make the nodes in E absorbing, with
the corresponding transition probability matrix denoted by Pa . Using distribution
propagation, i.e., t+1 = t Pa , we can calculate
p(t) = Prob(xt E|x0 = i)
for t = 0, 1, . . ., where 0 (i) = 1 for i = x0 and zero otherwise. Then, we can
calculate the first passage time of the original Markov chain as follows:

p(t)
t=0
f (t) = Prob(E = t|x0 = i) =
p(t) p(t 1) t > 0.
0.14

0.12

0.1

0.08

0.06

0.04

0.02

10

15

20

25
t

30

35

40

45

50

You can find the matlab code to calculate the first passage time below.
function fp = first_passage(P,E,x0,T)
n = size(P,1);
% make E absorbing
P_a = P;
P_a(E,:) = 0; P_a(E,E) = 1;
% first passage time using distribution propagation
pi = zeros(T,n);
prob = zeros(T,1);
pi(1,x0) = 1;
for t = 1 : T
prob(t) = pi(t,E);
pi(t+1,:) = pi(t,:)*P_a;
end
fp = prob - [0; prob(1:end-1)];
(b) We will construct a Markov chain whose first passage time gives the second passage
time of the original Markov chain as follows. We replicate the Markov chain, so
now we have 2n states. Then, link each state i E in the first copy to the
corresponding nodes j in the second copy, as dictated by Pij . We will denote the
second copy of E by E2 , i.e., E2 = {i + n|i E}. The first passage time through
E2 for the newly constructed Markov chain will give the second passage time of
the original Markov chain.
The transition probability matrix for this Markov chain is

Pi,j
if 1 i, j n

Pi,jn
if i E, n + 1 j 2n
(2)
Pij =
P
if
n + 1 i, j 2n

in,jn

0
otherwise
To calculate the first passage time of this Markov chain, we will need to use the
method of part (a).
(c) The distribution is plotted below.

0.06

0.05

0.04

0.03

0.02

0.01

10

20

30

40

50
t

60

70

80

90

100

(d) Below see the plots of f (t) + s(t), as well as Prob(xt E) for t = 0, . . . , 100. We
notice that f (t) + s(t) = Prob(xt E) for the first few time steps, after which it
seems to have converged to a value around 0.175.
0.18
0.16

fpluss
probofxine

0.14
0.12
0.1
0.08
0.06
0.04
0.02
0

10

20

30

40

50

60

70

80

90

100

(e) Lets denote the first passage time starting from x0 = i by fi (t). Then,
r(t) = 
Prob(E = t|x0 E)
0
t=0
= Pn
P
(E,
i)f
(t

1)
t
>0
i
i=1
The distribution of recurrence time, r(t), is shown below.

0.4

0.35

0.3

0.25

0.2

0.15

0.1

0.05

10

20

30

40

50
t

60

70

80

90

100

(f) We will show that s(t) = f (t) r(t) as follows:

P
f (m)r(t m)
f (t) r(t) =
Pm=
= tm=0 f (m)r(t m)
P
= tm=0 Prob(E = m|x0 = i) Prob(E = t m|x0 E)
P
= tm=0 Prob(E = m|x0 = i) Prob(E = t|xm E)
P
(2)
= tm=0 Prob(E = m, E = t|x0 = i)
(2)
= Prob(E = t|x0 = i)
= s(t).
Below you can see a plot of f (t) r(t) and s(t).
0.06

s(t)
f(t)*r(t)

0.05

0.04

0.03

0.02

0.01

10

20

30

40

50

60

70

80

90

100

(g) To compute the kth passage time, we would need to make k copies of the Markov
chain, so the number of states would be nk. We would connect the nodes in the
5

E for the ith copy to the corresponding nodes in the (i + 1)th copy. Calculating
the first passage time involves evaluating t P for t = 0, 1, . . ., which scales with
O(k 2 ) as k grows.
You can find the matlab implementation of parts (a)-(f) below.
clear all;
close all;
passage_time_data;
E = 1; % destination node
x0 = 2; % initial state
do_print = 1;
%% part a
T = 200;
fp = first_passage(P,E,x0,T);
plot([0:T-1], fp, b.-, LineWidth, 1, MarkerSize,16)
set(gca,xlim, [0 50])
xlabel(t)
ylabel(f)
if do_print
saveas(gcf,../figures/passage_1.eps, psc2);
end

%% part c
% second passage time
n = 6;
P2 = zeros(2*n,2*n);
P2(1:n,1:n) = P;
P2(n+1:2*n,n+1:2*n) = P;
P2(E,:) = [zeros(1,n) P(E,:)]; % make links from E to the second copy
E2 = E+n;
sp = first_passage(P2,E2,x0,T);
figure;
plot([0:T-1], sp, .-, LineWidth, 1, MarkerSize,16);
set(gca,xlim, [0 100])
xlabel(t)
ylabel(s)
if do_print
saveas(gcf,../figures/passage_2.eps, psc2);
end
%% part d
6

% Prob(x_t\in E | x_0)
pi = zeros(T,n);
pi(1,x0) = 1;
for t = 1:T-1
pi(t+1,:) = pi(t,:)*P;
end
figure;
plot([0:T-1], fp+sp, .-, LineWidth, 1, MarkerSize,16)
hold on
plot([0:T-1], pi(:,E), ro-, LineWidth, 1);
set(gca,xlim, [0 100]);
leg = legend(fpluss,probofxine);
set(leg,FontSize, 20);
if do_print
saveas(gcf,../figures/passage_fs_vs_prob.eps, psc2);
end
%% part e: recurrence time
%
loop over next states
r = zeros(T,1);
for i = 1 : n
if (P(E,i) ~= 0)
fp_i = first_passage(P, E, i, T);
r = r + P(E,i)*[0 ;fp_i(1:end-1)];
end
end

figure;
plot([0:T-1], r, .-, LineWidth, 1, MarkerSize,16);
hold on
set(gca,xlim, [0 100]);
xlabel(t)
ylabel(r)
if do_print
saveas(gcf,../figures/passage_recurrence.eps, psc2);
end
%% part f
figure;
plot([0:T-1], sp, b.-, LineWidth, 1, MarkerSize,16);
hold on
c= conv(fp,r);
7

plot([0:T-1], c(1:T), ro-, LineWidth, 1);

set(gca,xlim, [0 100]);
leg=legend(s(t),f(t)*r(t));
set(leg,FontSize, 24);
if do_print
saveas(gcf,../figures/passage_conv.eps, psc2);
end
2. Class decomposition of a Markov chain. In lecture we claimed that the states of a
Markov chain can be ordered so that the probability-transition matrix has the form


P11 P12
P =
,
0 P22
where P11 is block upper triangular with irreducible blocks on the diagonal, and P22
is block diagonal with irreducible blocks. Each of the blocks on the diagonal of P11
represents a transient class, while each of the blocks on the diagonal of P22 represents
a recurrent class. In this problem you will write code to find an ordering of the states
that puts P in this form. We will use standard graph-theory terminology throughout
the problem. Feel free to consult Wikipedia or other external sources if you encounter
any unfamiliar concepts. The file class_decomposition_data.m defines a specific
probability-transition matrix that you should use throughout this problem. However,
your code must work for any probability-transition matrix.
(a) Let G be a graph with adjacency matrix A Rnn . We write i j if there is
a path from node i to node j in G. We define the reachability matrix R Rnn
such that
(
1 i j,
Rij =
0 otherwise.
An algorithm for constructing R is given in algorithm 1.
R = A /* A is the adjacency matrix
repeat
for i = 1 to n do
/* iterate over all states k we know are reachable from i
for k such that Rik = 1 do
/* iterate over all states j we know are reachable from k
for j such that Rkj = 1 do
/* j is reachable from i through k
Rij = 1
end
end
end
until no changes are made to Rij
Algorithm 1: Computing the reachability matrix
Write a function that implements algorithm 1; use the following function header.
8

*/

*/
*/
*/

function R = reachable_states(A)
For reference, our implementation of reachable_states is less than twenty lines.
Hint. The function find if you are using MATLAB may be useful.
(b) We write i j if i j and j i, and we define the communication matrix
C Rnn such that
(
1 i = j or i j,
Cij =
0 otherwise.
Explain how to construct the communication matrix from the reachability matrix.
(c) We define the transience vector t Rn such that
(
1 i is a transient state,
ti =
0 otherwise.
Explain how to construct the transience vector from the communication and reachability matrices.
(d) Write a function that implements algorithm 2.
L Empty List /* L will contain sorted nodes
S Set of all nodes with no incoming edges
while S is non empty do
Remove a node n from S
Add n to tail of L
for each node m with edge e from n to m do
Remove edge e from graph
if m has no other incoming edges then
Insert m into S
end
end
end
if Graph has edges then
return error: graph has at least one cycle
else
return L /* L a topologically sorted order
end
Algorithm 2: Computing topological sort
Use the following function header.
function L = topological_sort(A)
For reference, our implementation of topological_sort is less than twenty lines.
(e) Suppose that duplicate rows of C have been removed, and the rows have been
sorted so that the first nt rows represent the transient classes. (You need to
write code to do this; the commands unique and sort may be useful.) Note
that there is now a unique row of C corresponding to each class. For two classes
9

*/

*/

Ci , Cj {1, . . . , n}, we write Ci Cj if x y for some x Ci and y Cj . The

adjacency matrix At Rnt nt for the set of transient classes is defined such that
(
1 Ci Cj ,
(At )ij =
0 otherwise.
Explain how to use the reachability matrix R of the Markov chain to find the
adjacency matrix At for the set of transient classes.
(f) You should now have all the tools you need to construct an ordering of the states
that puts the probability transition matrix in the desired form. Apply your method
to the matrix in class_decomposition.data.m. Attach a plot of calling spy on
your reordered matrix (the data file generates a plot of the original matrix). For
reference, our solution has thirty-five lines (not including reachable_states and
topological_sort).
Solution:
(a) An example implementation of reachable_states is given below.
function R = reachable_states(A)
n = size(A,1);
R = A;
no_changes = false;
while ~no_changes
no_changes = true;
for i = 1:n
for k = find(R(i,:))
for j = find(R(k,:))
if ~R(i,j)
no_changes = false;
end
R(i,j) = 1;
end
end
end
end
end
(b) Note that (RT )ij = 1 if j i and (RT )ij = 0 otherwise. Form the matrix R RT ,
where denotes entrywise and. Then, we have that (R RT )ij = 1 if i j,
and (R RT )ij = 0 otherwise. This is very close to the definition of C; the only
difference is that the diagonal entries of R RT may not be equal to one. We
can correct this deficiency using the formula C = (R RT ) I, where denotes
entrwise or, and I is the identity matrix.

10

(c) Observe that i is a transient state if and only if

n
X

Rij >

j=1

n
X

Cij .

j=1

This condition says that we can reach more states from i than there are states
that communicate with i; this is what it means for i to be transient. We can use
this condition to construct t from R and C.
(d) An example implementation of topological_sort is given below.
function L = topological_sort(A)
L = [];
S = find(sum(A) == 0);
while ~isempty(S)
n = S(1);
S = S(2:end);
L = [L n];
for m = find(A(n,:))
A(n,m) = 0;
if sum(A(:,m)) == 0
S = [S m];
end
end
end
if any(A(:))
L = [];
end
end
(e) Let x and y be fixed representatives of Ci and Cj , respectively. We claim that
(At )ij = 1 if and only if Rxy = 1. The definition of At says that (At )ij = 1 if and
only if there exist x Ci and y Cj such that x y. Then, we have that
x x y y,
so the properties of the communication relation imply that x y if and only if
x y. This proves the claim.
(f) The following script uses the tools and observations developed above to compute
the class decomposition of a probability-transition matrix.
clear all; close all; clc;
class_decomposition_data;
R = reachable_states(P > 0);
C = (R & R) | eye(n);
11

t = any(R & ~C,2);

[C , idx] = unique(C , rows);
t = t(idx);
[t , idx] = sort(t , descend);
C = C(idx,:);
nt = sum(t);
At = zeros(nt);
for i = 1:nt
x = find(C(i,:),1);
for j = 1:nt
y = find(C(j,:),1);
if i ~= j && R(x,y)
At(i,j) = 1;
end
end
end
L = topological_sort(At);
C(1:nt,:) = C(L,:);
idx = [];
for i = 1:size(C,1)
idx = [idx , find(C(i,:))];
end
P = P(idx,idx);
figure();
spy(P);
print -depsc class_decomposition.eps
The reordered probability-transition matrix is as follows.

12

20

40

60

80

100

120

20

40

60
nz = 1527

13

80

100

120

3. Markov web surfing model. A set of n web pages labeled 1, . . . , n contain (directed)
links to other pages. We define the link matrix L Rnn as

1 if page i links to page j
Lij =
0 otherwise.
We define o Rn as o = L1, which gives the number of outgoing links from each page.
A very crude model of a web surfer is a Markov chain on the pages, with transitions
described as follows. The model includes a parameter (0, 1), which (roughly) gives
the probability that the surfer follows a link from the current page. For a page with
oi > 0 (i.e., with at least one outgoing link) the surfer moves to each of the linkedto pages with probability /oi , and jumps to a page not linked to i with probability
(1 )/(n oi 1). For a page with no outgoing links (i.e., oi = 0) the surfer jumps
to a random page, chosen from a uniform distribution on (the other) pages.
We will assume that web surfer starts at a random page, uniformly distributed.
We earn a payment when the surfer follows (i.e., clicks on) a link, given by Rij 0.
This payment matrix satisfies Rij = 0 when Lij = 0 (i.e., we are not paid for random
jumps; only following links).
The following questions concern the specific instance of the problem with data given in
link_matrix_data.m.
(a) What is the most likely page the surfer is on, at time t = 10? at t = 100?
(b) Let J denote the expected total payment over t = 0, . . . , 50. Compute J three
ways:
Monte Carlo simulation (which gives an estimate of J, not the exact value).
Distribution propagation.
Value iteration.
Be sure to check that the values are consistent.
Remark. The Markov model described in this problem leads to Googles famous PageRank, which corresponds to the fraction of time spent at each site, when T . (The
current version of PageRank is based on far more than just the link topology, but the
first versions really did make heavy use of the Markov surfing model.)
Solution:
(a) To find the most likely page the surfer is on, we will calculate the state distributions
at times t = 10 and t = 100 by distribution propagation. The most likely page
at t = 10 is page 96, and the most likely page at t = 100 is page 83. Below you
can see a plot of the most likely page for all time t = 0, . . . , 100. Note that for
t = 1, all pages are equally likely to be visited, so the choice of page 1 in the plot
is arbitrary.

14

100

90

80

70

pmax

60

50

40

30

20

10

10

20

30

40

50
t

60

70

80

90

100

(b) We check the expected total payment using the three methods. We get J =
22.1335 using value iteration, J = 22.1335 using distribution propagation, and
J = 22.0188 using Monte Carlo simulation.
The code is shown below.
clear all; close all;
link_matrix_data;
% create the probability transition matrix
P = zeros(n,n);
for i = 1 : n
n_o = sum(L(i,:));
if n_o > 0
P(i,:) = (1-theta)/(n-n_o-1);
P(i,L(i,:)>0) = theta/n_o;
else
P(i,:) = 1/(n-1);
end
P(i,i) = 0;
end
% distribution propagation
T = 101;
pi = zeros(T,n);
pi(1,:) = 1/n;
exp_g = zeros(T,1);
for t = 1 : T
15

pi(t+1,:) = pi(t,:)*P;
exp_g(t) = sum(pi(t,:)*(R.*P));
end
% most likely page at t
max_prob = zeros(T,1);
for t = 1:T
[xx max_prob(t)] = max(pi(t,:));
end
stairs(0:T-1,max_prob,LineWidth, 2);
xlabel(t); ylabel(pmax);
set(gca,Ylim, [1 n]);
saveas(gcf,../figures/markovian_pagerank_ml_page.eps, psc2);
% cost over 50 time steps using distribution propagation
T = 51;
J_distprop = sum(exp_g(1:T));
% value iteration
V = zeros(n,T+1);
V(:,T+1) = 0;
for t = T:-1:1
V(:,t) = sum(P.*R,2) + P*V(:,t+1);
end
J_valiter = mean(V(:,1));
% monte carlo
N = 1000;
cost = zeros(N,1);
for iter = 1 : N
x = randsample(n,1,true,(1/n)*ones(n,1));
cost(iter) = 0;
for t = 1 : T
y = randsample(n,1,true,P(x,:));
cost(iter) = cost(iter) + R(x,y);
x = y;
end
end
J_mc = mean(cost);

16