IEOR 4106: Spring 2011, Professor Whitt

Guide to Reading: Basics of CTMCs

These questions provide a guide to your reading on the basics of continuous-time Markov
chains, covering material in Sections 1-4 in the long CTMC notes.
Here are SOME QUESTIONS:
1. What is a stochastic process?
2. What are the finite-dimensional distributions (f.d.d.s) of a stochastic process?
3. What is the probability law of a stochastic process?
4. When is a continuous-time stochastic process a Markov process? (See (2.1).)
5. What are the transition probabilities of a CTMC? (See (2.2)
6. How do you get from the transition probabilities of a CTMC to the finite-dimensional
distributions of a CTMC? (See (2.8.)
7. What are the Chapman-Kolmogorov equations of a CTMC? (See Lemma 2.1.)
8. What are the Kolmogorov ODEs of a CTMC, i.e., the Kolmogorov forward and
backward ordinary differential equations (ODEs) for a CTMC? (See Theorem 3.1)
9. What are the three main ways to construct a CTMC model? (See Sections 3.1, 3.2 and
3.3.)
10. How is the rate matrix Q defined? (See (3.1)-(3.3).)
11. Is the rate matrix Q a nonnegative matrix? Why or why not? (See (3.3)-(3.5).)
12. What are the Kolmogorov forward and backward ordinary differential equations (ODEs)
for a CTMC? (See Theorem 3.1.)
13. What is the solution of a usual one-dimensional ODE? (See the first display on p. 9.)
14. For an irreducible CTMC, is it possible to have Pi,j (t) = 0 for some i, j and t > 0?
(See Lemma 4.1.)
15. For a continuous-time stochastic process {X(t) : t 0}, what is a limiting probability vector? (See Section 4.)
16. For an irreducible finite-state CTMC, what is the limiting probability vector? (See
(4.1). We denote it by ; the component j is the limiting probability of being in state j.)
17. For a continuous-time stochastic process {X(t) : t 0}, what is a stationary probability vector? (See Section 4.)
18. For a continuous-time stochastic process {X(t) : t 0}, if the process has a unique
stationary vector, then does it necessarily also have a limiting a stationary probability vector?
(See Example 4.1.)
19. For a continuous-time stochastic process {X(t) : t 0}, if the process has a unique
limiting vector, then does it necessarily also have a stationary probability vector? (See Example
4.1.)

20. Does a finite-state CTMC necessarily have a unique stationary probability vector?
21. Does an irreducible finite-state CTMC necessarily have a unique stationary probability vector? (See Theorem 4.1.)
22. Given the rate matrix Q {Qi,j } of a finite-state irreducible CTMC, how can we
compute the limiting probability vector ? (See Theorem 4.2 (c).)
22. Given the matrix of transition probabilities P (t) {Pi,j (t)} of a finite-state irreducible
CTMC, for any fixed t > 0, how can the limiting probability vector be computed? (See
Theorem 4.2 (d).)
22. Given a finite-state irreducible CTMC characterized by an embedded DTMC with
one-step transition probabilities given by the matrix P {Pi,j } and exponential transition
times with means 1/i in state i, how can the limiting probability vector be computed? (See
Theorem 4.2 (b).) (See Theorem 4.2 (d).)
23. When is a CTMC a birth-and-death process? (See Section 5.)
24. How does the answer to question 21 simplify when the CTMC is a birth-and-death
process? (See Theorem 5.2.)