Markov Process: Properties, Analysis and

Applications
AJAY KUMAR
Assistant Professor
Atal Bihari Vajpayee-
Indian Institute of Information Technology & Management, Gwalior
Morena Link Road, Gwalior-474010
MP, India
March 10, 2010
1 / 111
Stochastic Process: A stochastic process is a collection of
random variables, {X(t) : t T}, where X(t) is a random variable
for each t T.

The set T is called the index set of the process and t T is

called and index or parameter.

The values taken by X(t) are called as states of the

stochastic process, corresponding to the index t.

If T is countable, the stochastic process is said to be discrete

time process

If T in an interval, the stochastic process is called a

continuous time process
2 / 111

Chain: If the state space S of a stochastic process

{X(t) : t T} is countable, the process is said to be a chain.

Based on the countable or continuous nature of the index set

and the space state, we have the following types of stochastic
processes:

Discrete time, discrete state space processes

Discrete time, continuous state space processes

Continuous time, discrete state space processes

Continuous time, continuous state space processes

3 / 111
Example: Consider a single machine prone to failures and let us
examine its states at discrete points of time, t
0
, t
1
, t
2
, . Let us
dene its state as 0, if the machine is up and 1, if machine is
down. The machine will keep changing its state with time. Let, X
i
denote the state of the machine at time t
i
, i = 0, 1, 2, , then
X
i
: i N is a stochastic process.

Here X
i
is a discrete random variable with range {0, 1}, so the
state space is S = {0, 1}.

Index set T = {t
0
, t
1
, t
2
, }

The stochastic process, dened above is a discrete time chain

If the state of the machine is examined at any instant of time,

then the index set will be an interval [0, ] and the stochastic
process X(t) : t 0 will be a continuous time chain.
4 / 111
Markov Process: Markov process (or chain) is a stochastic
process which has the property that the probability of transition
from a given state to any future state depends only on the present
state and not on the manner in which it was reached. OR
Mathematically, a stochastic process is called a Markov process, if
the occurrence of the future state depends on the immediately
preceding state and only on it. Thus, if t
0
, t
1
, t
2
, , represents the
points in time scale, then the family of random variables {X(t
n
)}
is said to be a Markov process if it holds the following property:
P {X(t
n
) = x
n
|X(t
n1
) = x
n1
, X(t
n2
) = x
n2
, , X(t
0
) = x
0
}
= P {X(t
n
) = x
n
|X(t
n1
) = x
n1
} , X(t
0
), X(t
1
), X(t
2
),
5 / 111
Transition Probability: The probability of moving from one state
to another is said to be transition probability. Mathematically,
p
ij
= P{X(t
n
) = j |X(t
n1
= i )}
is called transition probability.
Transition Probability Matrix (TPM): The transition
probabilities can be arranged in a matrix form. Such a matrix is
said to be one step-transition probability matrix. It is denoted by P
and is given by:
P =
_

_
p
11
p
12
p
1m
p
21
p
22
p
2m
.
.
.
.
.
.
.
.
.
.
.
.
p
m1
p
m2
p
mm
_

_
6 / 111
Properties of TPM:

P is a square matrix

0 p
ij 1
, i , j

Sum along each row is unity, i.e.,

m

j =1
p
ij
= 1
7 / 111
Discrete Time Markov Chain (DTMC): A discrete time Markov
chain is a stochastic process, {X
n
: n N}, with countable state
space S, such that for all n N, and for all
x
i
S, i = 0, 1, 2, , n:
P {X(t
n
) = x
n
|X(t
n1
) = x
n1
, X(t
n2
) = x
n2
, , X(t
0
) = x
0
}
= P {X(t
n
) = x
n
|X(t
n1
) = x
n1
}
Which implies, that given the current state of the system, the
future is independent of the past.
8 / 111
Let, P[s
j
(0)] be the probability that the system be in state s
j
at
time t = t
0
, and
p
ij
= P{X(t
n
) = j |X(t
n1
) = i }
is a transition probability from state s
i
to s
j
. Then, the stochastic
matrix,
P =
_

_
p
01
p
02
p
0m

p
11
p
12
p
1m

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
_

_
called as TPM, along with P[s
j
(0)], completely denes the DTMC.
9 / 111
Question: What is the probability that the system will occupy
state s
j
after n transitions, if its state at n = 0, is known.
Let P[s
j
(n + 1)] = the probability that the system is in state s
j
at
time t = t
n+1
, Then:
P[s
j
(n + 1)] = P[s
1
(n)]p
1j
+ P[s
2
(n)]p
2j
+ + P[s
m
(n)]p
mj
Now, put j = 1, 2, , m, we get:
[P[s
1
(n + 1)], P[s
2
(n + 1)], , P[s
m
(n + 1)]]
= [[P[s
1
(n)], P[s
2
(n)], , P[s
m
(n)]]
_

_
p
11
p
12
p
1m
p
21
p
22
p
2m
.
.
.
.
.
.
.
.
.
.
.
.
p
m1
p
m2
p
mm
_

_
10 / 111
The above system can be written as:
P(n + 1) = P(n) P
where, P(n + 1) is a row vector. Now,
P(1) = P(0) P
P(2) = P(1) P = P(0) P
2
P(3) = P(2) P = P(0) P
3

P(k) = P(k 1) P = P(0) P
k
11 / 111
Question: Gwalior city is divided into three zones A, B and C.
From the records of drivers, the following information is available.
Out of passengers picked up in zone A (B) (C) 40% (30%) (30%)
go to a destination in zone A, 40% (40%) (50%) are taken to a
destination in zone B, 20% (30%) (20%) go to a destination in
zone C. Suppose in the beginning of the day 30% of the taxis are
in zone A, 30% are in zone B and 40% are in zone C. What is the
distribution of taxis after all have go one ride and two rides. Also
determine the long run distribution of taxis.
Answer:
P(1) = (0.33, 0.44, 0.23)
P(2) = (0.333, 0.423, 0.244)
= (11/33, 14/33, 8/33)
12 / 111
Limiting State Probabilities: As, if n , P(n) will be
constant. In this situation limiting state probabilities exist and are
independent of the initial conditions. The process in this case is
said to be ergodic process.
Let lim
n
P(n) = , then:
lim
n
P(n + 1) = lim
n
P(n) P
= P
Where, is a 1 m vector.
13 / 111

Transient State: If the system can leave the state but never
return to it, e.g., s
1

Trapping or Absorbing State: If the system can enter the

state and cannot leave it, e.g., s
2

Recurrent Chain: A recurrent chain is a collection of all

trapping states.
s
1
s
2
0.75
0.25
1
14 / 111
Continuous Time Markov Chain (CTMC): A discrete state
space, continuous time process {X(t) : t 0} with state space S
is called CTMC, if the following Markov or memoryless property is
satised, for all s 0, u 0, t s and i , j , x(u) S,
P {X(t) = j |X(s) = i , X(u) = x(u), for 0 u < s}
= P {X(t) = j |X(s) = i }
Transition Rate: a
ij
is the transition rate of a process from state i
to state j , i = j .
Transition Rate Matrix: A = [a
ij
] is the transition rate matrix.
Transition Probability: Given that the present state is s
i
, the
conditional probability that s
i
s
j
, during time interval t is
a
ij
t. The unit of a
ij
is sec
1
. So a
1
ij
= expected time between
transitions
15 / 111
Question: Let P[s
i
(t)]=probability that the system occupies state
i at time t. Find out the probability of being in state j at time
t +t.
Note that:
P[s
j
(t +t)] = P[s
j
(t)]P(no transition from j )
+ P[s
1
(t)]a
1j
t + P[s
2
(t)]a
2j
t
+ + P[s
j 1
(t)]a
j 1,j
t + P[s
j +1
(t)]a
j +1,j
t
+ + P[s
m
(t)]a
mj
t
Next, the probability of marking no transition from j =1-probability
of making transition from j
= 1
m

i =j
a
ji
t
16 / 111
Substituting,
P[s
j
(t +t)] = P[s
j
(t)]
_
_
1
m

i =j
a
ji
t
_
_
+
m

i =j
P[s
i
(t)]a
ij
t
j = 1, 2, 3, , m
Now dene the diagonal elements of the transition rate matrix A
by:
a
jj
=
m

i =j
a
ji
P[s
j
(t +t)] = P[s
j
(t)](1 + a
jj
t) +
m

i =j
P[s
i
(t)]a
ij
t
or
P[s
j
(t +t)] P[s
j
(t)] =
m

i =1
P[s
i
(t)]a
ij
t
17 / 111
Dividing by t and taking limit t 0, leads to:
d
dt
P[s
j
(t)] =
m

i =1
P[s
i
(t)]a
ij
; j = 1, 2, , m
Next, dene:
P(t) = [P[s
1
(t)], P[s
2
(t)], , P[s
m
(t)]]
Which results in:
d
dt
P(t) = P(t) A
Which has the solution:
P(t) = P(0)e
At
18 / 111
Steady State Solution: As time t , P(t) will be constant
and so from:
d
dt
P(t) = P(t) A
we have:
0 = P() A = A
= rate of leaving the state=rate of entering in it
19 / 111
Example: Consider a 2-component system, which has following
four states:
State Component-1 Component-2
1 operating operating
2 failed operating
3 operating failed
4 failed failed
Assumptions:

Each component will be in one of the two states, operating or

failed

The failures among the components are independent

Initially the system is in working state

The failure behavior is exponential

20 / 111

If the two components are in parallel conguration, only state

4 results in system failure, so reliability is given by:
R
p
(t) = R
1
(t) + R
2
(t) + R
3
(t)

If the two components are in series conguration, states 2, 3,

and 4, result is system failure, so reliability is given by:
R
s
(t) = R
1
(t)

The system must be in one of the given four states, so

P
1
(t) + P
2
(t) + P
3
(t) + P
4
(t) = 1

The failures rates are

1
and
2
, respectively.
21 / 111
The transition diagram is given below:
dP
1
(t)
dt
+ (
1
+
2
)P
1
(t) = 0
dP
2
(t)
dt
+
2
P
2
(t) =
1
P
1
(t)
dP
3
(t)
dt
+
1
P
3
(t) =
2
P
1
(t)
dP
4
(t)
dt
=
1
P
3
(t) +
2
P
2
(t)
22 / 111
The probabilities are given by:
P
1
(t) = e
(
1
+
2
)t
P
2
(t) = e

2
t
e
(
1
+
2
)t
P
3
(t) = e

1
t
e
(
1
+
2
)t
P
4
(t) = 1 P
1
(t) P
2
(t) P
3
(t)

Reliability for series system:

R
s
(t) = P
1
(t) = e
(
1
+
2
)t

Reliability for parallel system:

R
p
(t) = R
1
(t) + R
2
(t) + R
3
(t)
= e

1
t
+ e

2
t
e
(
1
+
2
)t
23 / 111
Load-Sharing System: A system has two components, working in
parallel conguration. There is a dependency between the
components. If one component fails, the failure rate of the other
component increases as a result of additional load, placed on it.
Assume that,
+
1
and
+
2
are the increased failure rates of
component-1 and component-2, respectively.
24 / 111
The transition diagram is shown below:
dP
1
(t)
dt
+ (
1
+
2
)P
1
(t) = 0
dP
2
(t)
dt
+
+
2
P
2
(t) =
1
P
1
(t)
dP
3
(t)
dt
+
+
1
P
3
(t) =
2
P
1
(t)
25 / 111
The solution of the system is given below:
P
1
(t) = e
(
1
+
2
)t
P
2
(t) =

1

1
+
2

+
2
_
e

+
2
te
(
1
+
2
)t
_
P
3
(t) =

2

1
+
2

+
1
_
e

+
1
te
(
1
+
2
)t
_
The reliability is given by:
R(t) = P
1
(t) + P
2
(t) + P
3
(t)
MTTF can be calculated as:
MTTF =

_
0
R(t)dt
26 / 111
Standby Systems: Two components, A & B, are working in
standby redundancy mode, i.e., if A fails, B starts functioning.
Such systems are said to be standby systems.
The failure rate of standby unit will depend on the state of primary
(on-line) unit. We may consider:

The standby unit will have no failure or reduced failure rate,

while in its standby mode

While, the standby unit active, two cases arise:

it may experience the same failure rate as the online system, if

they are identical

it may experience dierent failure rate, if they are not identical

The problem is to model the system and to nd out its:
(a) reliability, and (b) MTTF
27 / 111
The transition diagram is shown below:
dP
1
(t)
dt
+ (
1
+

2
)P
1
(t) = 0
dP
2
(t)
dt
+
2
P
2
(t) =
1
P
1
(t)
dP
3
(t)
dt
+
1
P
3
(t) =

2
P
1
(t)
28 / 111
The solution of the system is given below:
P
1
(t) = e
(
1
+

2
)t
P
2
(t) =

1

1
+

2

2
_
e

2
te
(
1
+

2
)t
_
P
3
(t) = e

1
t
e
(
1
+

2
)t
The reliability is given by:
R(t) = P
1
(t) + P
2
(t) + P
3
(t)
= e

1
t
+

1

1
+

2

2
_
e

2
te
(
1
+

2
)t
_
and
MTTF =
1

1
+

1

2
(
1
+

2
)
29 / 111
Degraded Systems: Some systems may continue to operate in a
degraded state.
Dene the states as fully operational (state 1), degraded (state 2)
and failed (state 3). The transition diagram in this case is shown
below:
30 / 111
The system of ODEs can be generated as:
dP
1
(t)
dt
+ (
1
+
2
)P
1
(t) = 0
dP
2
(t)
dt
+
3
P
2
(t) =
2
P
1
(t)
Solution:
P
1
(t) = e
(
1
+
2
)
P
2
(t) =

2

1
+
2

3
_
e

3
t
e
(
1
+
2
)t
_
and
MTTF =
1

1
+
2
+

2

1
+
2

3
_
1

1
+
2
_
31 / 111
Availability: Availability is the probability that a system or
component is performing its required function at a given point in
time when operated and maintained in a prescribed manner.
Mathematically,
Availability =
uptime
uptime+downtime
=
MTBF
MTBF+MTTR
Note: Repairs are not considered while observing reliability, but
they are taken into account while calculating availability.
32 / 111
Calculate point availability and steady state availability for:

A system having one component

A system has two components, working in parallel

33 / 111
Example: Consider the CTMC, which has three states
0: set up, 1: Processing, and 2: Down
The transition diagram is given below:
0
2
1
p
r
s
f
Find out the ODEs for the above transition diagram and nd out:

Transient/Instantaneous state probabilities

Steady state probabilities

34 / 111
For steady state, the values of
0
,
1
and
2
are given by:

0
=
pr
pr + rs + fs
,
1
=
rs
pr + rs + fs
,
2
=
fs
pr + rs + fs
Moreover, if s = 20 per hour, p = 4 per hour, f = 0.05 per hour,
and r = 1 per hour, then:

0
= 0.16,
1
= 0.80, and
2
= 0.04
35 / 111
Example: Consider a system with two machines, M
1
and M
2
,
working in parallel conguration. Machine M
1
is fast and M
2
is a
slow machine. Raw parts are always available and have to undergo
one operation each, which may be carried out either on M
1
or on
M
2
. Each machine will immediately start processing, after nishing
the processing of one part. Each machine can fail randomly, and
after possible repair, it will resume processing.
The objective is to nd CTMC for the above system, and to
analyze the system performance, with:

no repair facility

single repair facility

two identical repair facilities

36 / 111
The system can be modeled using CTMC with four states given
by:

0: M
1
and M
2
both working

1: M
1
working and M
2
down

2: M
1
down and M
2
working

3: M
1
and M
2
both down
Solution:

If no repair facility, state 3 will be absorbing state and 0, 1,

and 2 are transient states.
0
1
2
3
f
f
f
f
37 / 111

For one repair facility, assume that in state 3, where both the
machines are down, the repair facility will spend its time
equally between M
1
and M
2
.
0
1
2
3
r
r
f
f
f
f
r /2
r /2
The steady-state probabilities are given by:

0
=
r
2
r
2
+ 2fr + 2f
2

1
=
2
=
fr
r
2
+ 2fr + 2f
2

3
=
2f
2
r
2
+ 2fr + 2f
2
38 / 111

When there are two identical repair facilities, the CTMC

model will be similar to that in last gure, except the labels
on the directed arcs from state 3 to state 2, and from state 3
to state 1.
The steady-state probabilities are given by:

0
=
r
2
r
2
+ 2fr + f
2

1
=
2
=
fr
r
2
+ 2fr + f
2

3
=
2f
2
r
2
+ 2fr + f
2
39 / 111
Computational Techniques for CTMC

Transient or Instantaneous Analysis

Solution of system of linear ODEs

Steady State Analysis

Solution of A = 0
40 / 111

Methods for Steady-State Analysis

Power method

SOR

Gaussian elimination Method

Gauss-Seidel Method

Methods for Transient Analysis

Fully Symbolic Method (Laplas Transformation)

Eulers Method

Runge-Kutta forth order Method

41 / 111
Methods for Steady State
Analysis
42 / 111
Power Method: The problem is to solve A = 0

Step-1: choose, a max

i
|a
ii
|

Step-2: The new system will be:

=
_
I +
A
a
_
Note A

= I +
A
a
is a transition probability matrix of a DTMC.

Do iteration:
(i )
=
(i 1)
A

Initial value:
(0)
= (0)
43 / 111
Example: Consider the following transition rate matrix to solve
A = 0:
A =
_
_
_
_
2 2c 0 2(1 c)
0 c (1 c)
0 0 0 0
0 0 0 0
_
_
_
_
Choose a = 2, we get:
A

=
_
_
_
_
0 c 0 1 c
0
1
2
c
2
1c
2
0 0 1 0
0 0 0 1
_
_
_
_
Now, take
(0)
= [1, 0, 0, 0] and nd the rest.
44 / 111
SOR (Successive Over Relaxation) Method: Let A be a
symmetric matrix then the linear system Ax = b can be solved
using the following steps:

Start with an initial guess

Iterate until some criteria for convergence are satised:

x
(k+1)
= (D +L)
1
[(1 )D U]x
(k)
+(D +L)
1
b

where, x
(k+1)
is the solution vector at the k
th
iteration, L is a
lower triangular matrix, U is an upper triangular matrix, and
D is a diagonal matrix

Gauss-Seidel is a special case of SOR with = 1, i.e.,

x
(k+1)
= (D + L)
1
Ux
(k)
+ (D + L)
1
b
45 / 111
Gauss Elimination Method: To solve the system Ax = b

nd the augmented metrix, [A|b]

choose, a
(k)
jk
= max |a
(k)
ik
|, k i n, and interchange rows k
and j

The elements a
(k)
ij
with i , j k are given by:
a
(k+1)
ij
= a
(k)
ij

a
(k)
ik
a
(k)
kk
a
(k)
kj
i = k + 1, k + 2, , n
j = k + 1, k + 2, , n, n + 1
where, a
(1)
ij
= a
ij

this method will always give the exact solution

46 / 111
Example: Solve the system of equations:
_
_
_
_
2 1 1 2
4 0 2 1
3 2 2 0
1 3 2 1
_
_
_
_
_
_
_
_
x
1
x
2
x
3
x
4
_
_
_
_
=
_
_
_
_
10
8
7
5
_
_
_
_
using Gauss elimination method.
[A|b] =
_
_
_
_
2 1 1 2 10
4 0 2 1 8
3 2 2 0 7
1 3 2 1 5
_
_
_
_
R
2
R
1
=
_
_
_
_
4 0 2 1 8
2 1 1 2 10
3 2 2 0 7
1 3 2 1 5
_
_
_
_
R
2

1
2
R
1
, R
3

3
4
R
1
, R
4

1
4
R
1
47 / 111
=
_
_
_
_
4 0 2 1 8
0 1 0 5/2 14
0 2 1/2 3/4 1
0 3 3/2 5/4 7
_
_
_
_
R
4
R
2
=
_
_
_
_
4 0 2 1 8
0 3 3/2 5/4 7
0 2 1/2 3/4 1
0 1 0 5/2 14
_
_
_
_
R
3

2
3
R
2
, R
4

1
3
R
2
=
_
_
_
_
4 0 2 1 8
0 3 3/2 5/4 7
0 0 1/2 1/12 17/3
0 0 1/2 25/12 35/3
_
_
_
_
R
4
R
3
48 / 111
=
_
_
_
_
4 0 2 1 8
0 3 3/2 5/4 7
0 0 1/2 1/12 17/3
0 0 0 13/6 52/3
_
_
_
_
R
4
R
3
Finally, we get:
_
_
_
_
4 0 2 1
0 3 3/2 5/4
0 0 1/2 1/12
0 0 0 13/6
_
_
_
_
_
_
_
_
x
1
x
2
x
3
x
4
_
_
_
_
=
_
_
_
_
8
7
17/3
52/3
_
_
_
_
and so, x
4
= 8, x
3
= 10, x
2
= 6, x
1
= 5
49 / 111
Methods for Transient Analysis
50 / 111
Fully Symbolic Method

Taking the Laplace transform on both sides of the

Kolmogorov dierential equation and rearranging the terms:

P(s) = P(0)(sI A)
1

The transient state probability is obtained by computing the

inverse Laplace transform of

P(s)

Solutions obtained will be closed-form and full symbolic in

both the system parameters and time t
51 / 111
Using Matrix Theory: The solution of Kolmogorov dierential
equations is:
P(t) = P(0)e
At
Now, the problem is to compute e
At
:

Input matrix A

Find out eigenvalues and eigenvectors of matrix A

A = V D V
1
and e
A
= V e
D
V
1
, where V is collection
of eigenvectors and D is diagonal matrix whose diagonal
elements are eigenvalues of A

in MATLAB, e
A
can be directly obtained using:

Y = expm(A), or

[V,D] = EIG(A);
EXPM(A) = V*diag(exp(diag(D)))/V
52 / 111
Euler Method: Consider the following dierential equation:
u

= f (t, u), u(0) = u

0
then:
u
j +1
= u
j
+ hf
j
, j = 0, 1, 2, , N 1
53 / 111
Example: Solve the ode:
u

= 2tu
2
, u(0) = 1
We have:
u
j +1
= u
j
2ht
j
u
2
j
with h = 0.2 and u
0
= 1
u(0.2) = u
1
= 1.0
u(0.4) = u
2
= 0.92
u(0.6) = u
3
= 0.78458
u(0.8) = u
4
= 0.63684
u(1) = u
5
= 0.50706
54 / 111
Runge-Kutta Method of Fourth Order (RK4): Consider the
following dierential equation:
u

= f (t, u), u(0) = u

0
Then:
u
i +1
= u
i
+
1
6
[k
1
+ 2k
2
+ 2k
3
+ k
4
]
Where:
k
1
= hf (t
i
, u
i
)
k
2
= hf
_
t
i
+
h
2
, u
i
+
k
1
2
_
k
3
= hf
_
t
i
+
h
2
, u
i
+
k
2
2
_
k
4
= hf (t
i
+ h, u
i
+ k
3
)
55 / 111
RK4 Method for System of ODEs: Consider the system of n
equations:
dU
dt
= F(t, u
1
, u
2
, , u
n
)
U(t
0
) = U
0
Where, U = [u
1
, u
2
, , u
n
]
T
,
F = [f
1
, f
2
, , f
n
]
T
,
U
0
= [u
1
(0), u
2
(0), , u
n
(0)]
56 / 111
Then the solution is given by:
U
j +1
= U
j
+
1
6
(K
1
+ 2K
2
+ 2K
3
+ K
4
)
Where:
U
j +1
=
_
_
_
_
_
u
1,j +1
u
2,j +1
.
.
.
u
n,j +1
_
_
_
_
_
; K
i
=
_
_
_
_
_
k
1i
k
2i
.
.
.
k
ni
_
_
_
_
_
and:
k
i 1
= hf
i
(t
j
, u
1,j
, u
2,j
, , u
n,j
)
k
i 2
= hf
i
(t
j
+
h
2
, u
1,j
+
1
2
k
11
, u
2,j
+
1
2
k
21
, , u
n,j
+
1
2
k
n1
)
k
i 3
= hf
i
(t
j
+
h
2
, u
1,j
+
1
2
k
12
, u
2,j
+
1
2
k
22
, , u
n,j
+
1
2
k
n2
)
k
i 4
= hf
i
(t
j
, u
1,j
+ k
13
, u
2,j
+ k
23
, , u
n,j
+ k
n3
)
57 / 111
Solve the system of equations:
u

= 3u + 2v, u(0) = 0
v

= 3u 4v, v(0) = 0.5

with h = 0.2 on the interval [0, 0.4]. Use RK4 method.
58 / 111
Eulers Method for System of ODEs: Consider the system of n
equations:
dU
dt
= F(t, u
1
, u
2
, , u
n
)
U(t
0
) = U
0
Where,
U = [u
1
, u
2
, , u
n
]
T
,
F = [f
1
, f
2
, , f
n
]
T
,
U
0
= [u
1
(0), u
2
(0), , u
n
(0)]
The solution is given by:
U
j +1
= U
j
+ hU

j
; j = 0, 1, 2, , N 1
59 / 111
Computational Techniques for DTMC
1: Given a two state Markov Chain, with the transition probability
matrix:
P =
_
1 a a
b 1 b
_
; 0 a, b 1, |1 a b| < 1
then matrix P
n
is given by:
P
n
=
_
b+a(1ab)
n
a+b
aa(1ab)
n
a+b
bb(1ab)
n
a+b
a+b(1ab)
n
a+b
_
60 / 111
Assignments
1. Make a computer program to solve a given DTMC having
2-states.
Inputs: TPM, P and initial probability, P(0)
Output: nstep transition probabilities, P(n)
2. Extend the program to m m TPM, using
P(n + 1) = P(0) P
n
or P(n + 1) = P(n) P
61 / 111
Pulping System: The four major actions carried out in the system
are, (i) cooking of chips, (ii) separation of knots, (iii) washing of
pulp, and (iv) opening of bers. The pulping system consists of for
subsystems, namely:

Digester (A): One unit, used for cooking the chips whose
failure causes the complete failure of the cooking system.

Knotter (B): Two units, one working and other standby,

used to remove the knots from the cooked chips. Whose
complete failure occurs only if both the units fail.
62 / 111

Decker (C): Three units, arranged in series conguration,

used to remove liquor for the cooked chips. Failure of any one
causes the complete failure of the pulping system.

Opener (D): Two units, one working and other standby, used
to separate the bers. Whose complete failure occurs only if
both the units fail.
The interactions among the various components of pulping system
are shown by RBD in Fig. 1.
A
B
2
B
1
C
1
C
2
C
3
D
2
D
1
Digester
Knotter
Deckers
Opener
Figure: Reliability Block Diagram of Pulping System
63 / 111
Figure: CTMC for Pulping System
, , , : Failure rates of digester, knotter, decker and opener,
respectively.
, , , : Repair rates of digester, knotter, decker and opener,
respectively.
64 / 111
Assignments (Contd...)
Make computer programmes to nd out steady state probabilities
using:
3. Power Method
4. SOR/Gauss-Seidel Method
5. Gauss Elimination Method
65 / 111
Assignments (Contd...)
Make computer programmes to nd out transient probabilities
using:
6. Matrix Method
7. Eulers Method
8. RK4 Method
Also, simulate the system behavior for t = 100 hours.
66 / 111
Birth and Death Process (BD Process): A homogeneous
CTMC {X(t) : t 0}, with state space {0, 1, 2, } is called a
BD process, if there exist constants,

i
; i = 0, 1, 2, , and

i
; i = 1, 2, 3, such that the transition rates are given by:
p
i ,i +1
=
i
, i = 0, 1, 2,
p
i ,i 1
=
i
, i = 1, 2, 3,
p
ij
= 0, for |i j | > 1
67 / 111
In steady-state,we get:

0
=
1

1
(
k
+
k
)
k
=
k1

k1
+
k+1

k+1
, k 1
Now,

k

k+1

k+1
=
k1

k1

k
=
k2

k2

k1

k1
=
k3

k3

k2

k2
=
=
0

1
= 0
68 / 111
Thus, we have:

k
=
k+1

k+1
or

k+1
=

k

k+1

k
=
k
=

k1

k1
, for k 1
or

k
=

0

1

k1

1

2

k

0
=
k1

i =0

i +1
69 / 111
Using the fact that

j
= 1, we get:

0
+

k1

0
k1

i =0

i +1
= 1
i.e.,

0
=
1
1 +

k1
k1

i =0

i +1
The limiting probabilities will exist provided the series

k1
k1

i =0

i +1
converges.
70 / 111
Queues: A queue is a system into which customers arrive to
receive service. When the servers in the system are busy, incoming
customers wait for their turn. Upon completion of a service, the
customer to be served next is selected according to some queuing
discipline.
Queuing discipline: The queuing discipline is a rule for selecting
the customer for the next service among the set waiting customer.
This could be either:

First in rst out (FIFO),

Last in rst out (LIFO),

Service in random order (SIRO),

Shortest processing time (SPT), etc.

71 / 111
A queuing system can be completely described by:
(a). The input or arrival pattern
(b). The service mechanism or service pattern
(c). The number of service channels
(d). Customers behavior
(e). Maximum length of a queue
(f). The size of population
72 / 111
Notation for queues: Generally queuing model may be completely
specied in the following symbolic form: (A/B/C):(D/E/F), where

A: probability law for the arrival (or inter-arrival) time

B: probability law according to which the customers are being

served

C: number of service channels (service stations)

D: queue/service discipline

E: capacity of the system, i.e., the maximum number allowed

in the system (in service or waiting)

F: Population size
D. Kendall (1953) introduced the three characteristics (A/B/C).
Later, A. Lee (1966) added fourth, fth and sixth characteristics to
the notations.
73 / 111
Let C
1
, C
2
, . . . be any arbitrary arrival stream of customers with:

t
j
= arrival epoch of C
j
, (0 t
1
t
2
. . . < ), and

T
j
= t
j +1
t
j
is inter-arrival time between C
j
and C
j +1
A customer who has arrived but not yet departed is said to be in
the system. While in the system, a customer may be either in
service or in the queue. Let,

S
j
= service time of C
j

D
j
= delay in queue of C
j
, (waiting time in queue)

W
j
= D
j
+ S
j
= waiting time in system of C
j
74 / 111
The number of customers in the system at epoch t is given by
N(t), which can be written as
N(t) = N
q
(t) + N
s
(t)
where N
q
(t) = number of customers in the queue at time t, and
N
s
(t) = number of customers in the service at time t.
The number of arrivals by time t, n
A
(t) and number of departures
by time t, n
D
(t), are given by:
n
A
(t) = max{j : t
j
t}
n
D
(t) = n
A
(t) N(t)
The process {n
A
(t) : t 0} is called the arrival process and the
process {n
D
(t) : t 0} is called the departure process.
75 / 111
Performance Measures: The performance of a queue will be
dened in terms of properties of one or more of the following
stochastic processes: {N(t) : t 0}, {N
q
(t) : t 0},
{N
s
(t) : t 0}, {W
j
: j N}, and {D
j
: j N}.
Let us dene the following parameters rst:
= Arrival rate = lim
t
n
A
(t)
t
1

= Average time in service = lim

n
1
n
n

j =1
S
j
, where is called
the service rate. We now dene generic measures of performance
for a queue. Let, m be the number of identical servers in the
queuing system.
76 / 111
Average number of customers in the system:
L = lim
t
1
t
t
_
0
N(u)du (1)
Average number of customers in the queue (mean queue
length):
Q = lim
t
1
t
t
_
0
N
q
(u)du (2)
Average number of customers in the service:
B = lim
t
1
t
t
_
0
N
s
(u)du (3)
77 / 111
Average server utilization:
U =
B
m
(4)
Average waiting time in the system:
W = lim
n
1
n
n

j =1
W
j
(5)
Average waiting time in the queue:
D = lim
n
1
n
n

j =1
D
j
(6)
The measures L, Q, B and U are averages over time, whereas the
measures W and D are averages over customers. The values on
the right hand side are random variables.
78 / 111
Littles Result: The result states that, given a queue, if the limits:
= lim
t
n
A
(t)
t
and W = lim
n
1
n
n

j =1
W
j
exist, with < and
W < , then the following limit,
L = lim
t
1
t
t
_
0
N(u)du
exists, where L < and
L = W (7)
A consequence of Littles result is that if the limits dening and
D exist, with < , D < , then the limit dening Q exists,
with Q < and Q = D. Note that this enables the result to be
applied to parameters connected with the queue alone.
79 / 111
The M/M/1 Queue: In the M/M/1 queue, the arrivals are
following Poisson distribution with rate , the departures are
following Poisson distribution with rate , and there is only single
server.
Let N(t) be the number of customers in the M/M/1 queue at time
t, then obviously {N(t) : t 0} is a CTMC. Also it is a special
case of Birth and Death Process with:

k
= k 0

k
= k 1
The ratio =

is called as trac intensity of the system.

80 / 111
From the last equation of BD process, we have:

0
=
1

k0

k
= (1 ) (8)
That is, the probability of an empty system is given by (1 ).
Thus represents the proportion of the time, the server is busy.
This is called the utilization U of the server or trac intensity of
the queue.

k
=
k
(1 ), k 0 (9)
The probability
k
is to be described, in the following ways:

as the proportion of time in the steady state the system has

exactly k customers,

in the steady state, a randomly arriving customer nds exactly

k customers in the system,

a departing customer leaves behind exactly k customers in the

system.
81 / 111
The mean number of customers in the system, L is found as the
expected value of random variable N, where,
P{N = k} = lim
t
P{N(t) = k} =
k
Thus we have,
L = E[N] =

k=0
k
k
= (1 )

k=0
k
k
(10)
But, we have,

k=0
k
k
=

(1 )
2
(11)
So by (10) and (11), we have,
L =

1
=


(12)
82 / 111
The mean number of customers, Q in the queue is similarly given
as
Q =

k=1
(k 1)
k
=

k=1
k
k

k=1

k
= L (1
0
) = L
=

2
1
=

2
( )
(13)
We can also rewrite as,
L = Q +
Now, W : the mean waiting time in the system or mean response
time, is given by,
W = D +
1

(14)
where,
1

is the service time.

83 / 111
To nd out the explicit formula for W, we use Littles formula,
L = W (15)
Using (15), we obtain the expressions for W and D for the M/M/1
queue. From (12) and (15), we have,
W =
L

=
1
(1 )
=
1

(16)
and, from (14)
D =
1


1

=

( )
(17)
84 / 111
Example: Consider an NC machine center processing raw parts
one at a time in M/M/1 fashion. Let = 8 parts/h and = 10
parts/h. Then nd out:
a. Machine utilization,
b. Mean number of customers in the system,
c. Mean number of customers in the queue,
d. Mean waiting time in the system, and
e. Mean waiting time in queue.
The answers are 0.8, 4.0, 3.2, 0.5 h, and 0.4 h.
85 / 111
Stability of M/M/1 Queue: From (8), we see that:

0
=
1

k0

k
We can solve this equation for
0
if and only if the series in the
numerator is convergent. The series will be convergent if and only
if < 1. When 1, the series diverges, which means that the
queue grows without bound and the values of L, Q, W and D
would go to innity. We say that the queue is stable if < 1 and it
is unstable if 1. The stability condition physically means that
the arrival rate must be strictly less than the service rate.
When is close to 1, queues become very long and as 1, the
number in the system grows without bound. Transient analysis
reveals system behavior as 1 or short term behavior when
> 1.
86 / 111
Waiting Time Distributions: We have seen that W and D are
the mean values of the random variables w and d, whose
distributions can be dened by:
F
w
(t) = lim
n
1
n
n

j =1
P{W
j
t}
F
d
(t) = lim
n
1
n
n

j =1
P{D
j
t}
Now we will calculate F
w
(t) and F
d
(t), assuming that the
customers are served in FIFO (or FCFS) order.
87 / 111
First note that d is a continuous random variable except that there
is a nonzero probability,
0
that d = 0 and the customers enters
service immediately on arrival. We have:
F
d
(0) = P{d 0} = P{d = 0}
= P{System is empty on arrival}
So,
F
d
(0) =
0
= 1 (18)
88 / 111
Now we will nd F
d
(t) for t > 0. If there are k customers in the
system upon arrival, for the customer to get into the system
between 0 and t, all k units must be served by the time t. Since
the service time distribution is memoryless, the distribution of the
time required for k service times is independent of the time of the
current arrival, and is the convolution of k exponential random
variables, which is an Erlang-k distribution. The probability density
function of the Erlang distribution is:
f (x; k, ) =
(x)
k1
e
x
(k 1)!
for x, 0
The parameter k is called the shape parameter and the parameter
is called the rate parameter.
89 / 111
The probability that an arrival nds k units in the system in the
steady state is simply the probability
k
. Thus we have:
F
d
(t) = P(d t)
= F
d
(0) +

k=1
_
P(0 < d t|k units in the system)
k
_
P(0 < d t|k units in the system) = P(sum of k service times t)
=
t
_
0
(x)
k1
e
x
(k 1)!
dx
90 / 111
Therefore, we get for t > 0,
F
d
(t) = (1 ) + (1 )

k=1

k
t
_
0
(x)
k1
e
x
(k 1)!
dx
= (1 ) + (1 )
t
_
0
e
x
_

k=1
(x)
k1
(k 1)!
_
dx
= (1 ) + (1 )
t
_
0
e
x(1)
dx (19)
91 / 111
Simplifying (19) we get:
F
d
(t) = 1 e
(1)t
for t > 0
The distribution for waiting time in queue is then:
F
d
(t) =
_
1 , for t = 0
1 e
(1)t
for t > 0
(20)
From (20), we can compute the mean waiting time in queue as:
D = E[D] =

( )
(21)
92 / 111
Let us now compute the distribution of w, following the similar
arrangements. We have for t 0,
F
w
(t) =

k=0
_
P(w t|k units in the system)
k
_
=

k=0
P(sum of (k + 1) service times t)
k
=

k=0
_
t
_
0
(x)
k
e
x
k!
dx
_

k
(1 )
93 / 111
F
w
(t) = (1 )
t
_
0
e
x

k=0
(x)
k
k!
dx
= (1 )
t
_
0
e
x(1)
dx
= 1 e
(1)t
= 1 e
()t
94 / 111
Thus we have:
F
w
(t) =
_
0, for t < 0
1 e
()t
, for t 0.
(22)
Therefore, w is exponentially distributed with mean given by:
W =
1

It is noted that the mean values D and W are independent of the
queuing discipline, whereas the distributions of d and w depend on
the queuing discipline.
95 / 111
The M/M/1/N Queue: In an M/M/1/N queueing system, the
maximum number of jobs in the system is N, which implies a
maximum queue length of N 1. An arriving job enters the queue
if it nds fewer than N jobs in the system and is lost otherwise.
This behavior can be modeled by a birth-death process with:

k
=
_
for k = 0, 1, 2, . . . , N 1
0 for k N
The service rate
k
= for all k 1. The CTMC here is a nite
birth and death process, with N + 1 states.
96 / 111
The steady state probabilities are given by:

k
=
(1 )
k
1
N+1
, k = 0, 1, 2, . . . , N
= 0, k > N (23)
Note that a system with nite population such as above will always
be stable for all values of .
The performance measures can be veried to be given by:
U = 1
0
=
(1
N
)
1
N+1
(24)
97 / 111
Q =

2
[1
N
N
N1
(1 )]
(1 )(1
N+1
)
(25)
L =
[1
N
N
N
(1 )]
(1 )(1
N+1
)
(26)
W =
1
N
N
N
(1 )
(1 )(1
N
)
(27)
D =
[1
N
N
N1
(1 )]
(1 )(1
N
)
(28)
98 / 111
Example: It would be interesting to study the performance
measures of a single-server queuing system for various queue
capacities. First, consider the M/M/1 queue of Example 1, where
we had = 8 parts/h and = 10 parts/h. We have seen that
U = 0.8, L = 4, Q = 3.2, W = 0.5 h and D = 0.4 h.
At the other extreme, we have M/M/1/1 system in which no
queuing is allowed. In this case, there are only two states, 0 and 1,
and

0
=

+
= 0.556; and
1
= 0.444
Verify that U = 0.444, L = Q = 0.444, and W = D = 0.1h.
Can you calculate the performance measures for M/M/1/10
queue?
99 / 111
Example: Consider a machine center modeled as an M/M/1/N
queue with parameters and . Let C be the cost of operating
the machine at a rate of and let a prot of q accrue for every
nished part produced. Find out the service rate that maximizes
the total prot.
Solution: Production rate of the machine center=U
=
(1
n
)
1
N+1
=
(
N

N
)

N+1

N+1
The above is the number of parts produced in one time unit (for
example one hour, if is expressed in terms of parts per hour).
The corresponding total prot is given by:
P =
(
N

N
)

N+1

N+1
q C
The service rate that maximizes the prot can now be obtained
in the usual way.
100 / 111
The M/M/m Queue: Consider a queuing system with Poisson
arrival rate as before, but where m( 1) exponential servers,
with rate each, share a common queue. This is referred to as the
M/M/m queue and is illustrated in Fig. 3. If the number of
customers in the system is greater than the number of servers,
then a queue forms.
Figure: M/M/m queue and its state diagram
101 / 111
Steady State Analysis: The M/M/m queue is also a special case
of the birth and death model with rates:

k
= , k = 0, 1, 2, . . . (29)

k
= k, k = 0, 1, . . . , m
= m, k > m (30)
102 / 111
The state transition diagram is shown in Fig. 3. The steady state
probabilities could be obtained by using (29) and (30) in the
expressions of
0
and
k
of BD process. We nd that:

k
=

0
(m)
k
k!
; k = 0, 1, 2, . . . , m
=

0
m
m

k
m!
; k > m (31)
where is dened as:
=

m
(32)
Equation (31) is valid for determining the steady state probabilities
whenever < 1, i.e., the stability condition holds.
103 / 111
The expression for
0
can now be obtained using (31) and the fact
that the steady state probabilities sum up to 1. Thus:

0
=
_
(m)
m
m!(1 )
+
m1

k=0
(m)
k
k!
_
1
(33)
To compute the dierent performance measures, we rst consider
N
s
, the number of busy servers in the steady state. N
s
is a discrete
random variable with range {0, 1, 2, . . . , m} and with pmf given by:
P(N
s
= k) =
k
k = 0, 1, . . . , m 1
=

j =m

j
k = m
104 / 111
The average number of busy servers in the steady state, B can be
immediately obtained as:
B = E[N
s
] =
m

k=0
kP(N
s
= k) =

(34)
The mean utilization U of each server is simply the ration of B to
the total number of servers and hence:
U =

m
= (35)
105 / 111
We now compute Q, the average number of customers waiting in
queue:
Q =

k=m
(k m)
k
=
(m)
m

0
m!(1 )
2
(36)
The average number L in the system is given by:
L = Q + B =
(m)
m

0
m!(1 )
2
+

(37)
106 / 111
We now use Littles law to compute the average waiting time in
the system. We have:
W =
L

=
(m)
m

0
m!(1 )
2
+
1

(38)
The average waiting time in queue can now be obtained as:
D = W
1

=
(m)
m

0
m!(1 )
2
(39)
107 / 111
The M/M/ Queue: In an M/M/ queueing system we have a
Poisson arrival process with arrival rate and an innite number of
servers with service rate each.
If there are k jobs in the system, then the overall service rate is k
because each arriving job immediately gets a server and does not
have to wait. Once again, the underlying CTMC is a birth-death
process.
From the expressions of
0
and
k
of BD process, we obtain the
steady-state probability of k jobs in the system:

k
=
0
k1

i =0

(i + 1)
=
0
_

_
k
1
k!
(40)
108 / 111
The steady state probability of no jobs in the system:

0
=
1
1 +

k=1
_

_
k
1
k!
= e

(41)
and nally:

k
=
_

_
k
e

k!
(42)
This is the Poisson pmf, and the expected number of jobs in the
system is:
L =

(43)
With Littles theorem, the mean waiting time in the system is
given by:
W =
1

(44)
109 / 111
Queuing models for self study:
A. M/M/s queuing model with nite storage and innite
population, and
B. M/M/s queuing model with nite storage and nite population.
110 / 111
Thank You
111 / 111