A Primer on Probability Theory and Stochastic

Modelling

Mohamoud Dualeh
May 2004
 2

Markov Chain
Suppose there are two urns. John has one urn and Abdul has the other. The two urns each
contain n balls. Of the total of 2n balls, n are red and n are black. At each step of a random
process, one of the balls in each urn (John’s and Abdul’s) is chosen at random and John and
Abdul then exchange these two balls. So, that each urn continues to contain n balls. Let the
state of the system be indexed by the number1, r, of red balls in John’s urn.

Now, the transition probabilities for a Markov Chain model of this process is given by:

If there are X i red balls in John’s urn at step i,

Pr ,s = P ( X i +1 = s X i = r ) for r, s = 0, 1, ..., n.

Then P0,1 = 1; also Pn,n+1 = 1

When X i = r , John’s urn contains r red and (n-1) black, and Abdul’s (n-1) red and r black.

Pr ,r +1 = P (choose red in Rod's urn X i = r ). P (choose black in Abukar's urn X i =r )

r 
2

=  
n 

 r  r
Pr ,r = 2  1 −  by similar argument; and also
 n  n 

 r
2

Pr ,r +1 = 1 −  otherwise Pr,s = 0

 n

Now that we worked out an expression for transition probabilities, we can write a system of
equations that must be satisfied by the stationary distribution of this model.

Π = [ Π 0 , Π1 ,..., Π n ]

The probability πi is P (i red balls in Rod's urn ) .

Therefore,

1 1
2 2

π0 =   π1 ; and πn =   πn −1 . (A)

n  n 

1
You can choose your own index if you prefer.

©YieldCurve.com Mohamoud Dualeh 2004

 3

for r = 1, 2, n-1,

πr = Pr −1,r πr −1 + Pr ,r πr + Pr +1,r πr +1 ,

 r − 1  r  r  r + 1
2 2

i .e . πr = 1 −  πr −1 + 2  1 −  πr +  π . (B)

 n   n  n   n  r +1

n
Equations (A) and (B) define the process, with ∑ πr = 1 .
r =0

Example:

Suppose n = 3, we can solve above equations in order to find Π1 , Π 2 and Π3 .

From equation (A) we have

1
π0 = π1
9

1
π3 = π2
9

and π0 + π1 + π2 + π3 = 1

Also, from equation (B) we have

 1 − 1  1  1  1 + 1
2 2

π1 = 1 −  π0 + 2  1 −  π1 +  π ; substituting gives

 3   3  3   3  2

1 1 10 9
π1 + π1 + π2 + π2 = 1 = (π1 + π2 ) , so π1 + π2 = .
9 9 9 10

Now use

1
π0 = π1
9
9
π2 = − π1
10
1 1
π3 = − π1
10 9

©YieldCurve.com Mohamoud Dualeh 2004

 4

Therefore

4 4
π1 = π0 + π1 + π2
9 9

1 4 4 9  1 2
= π1 + π1 +  − π1  = π1 + .
9 9 9 10  9 5

8 2
∴ π1 = ,
9 5
or

9
π1 =
20
1
π0 =
20
9
π2 =
20
1
π3 = .
20

So it is complete. Whew!!

I have omitted couple of tedious calculations! As usual, all the typos and mistakes are
mine!

* * * *

©YieldCurve.com Mohamoud Dualeh 2004