10 views

Uploaded by spitzersglare

X

- Continuous Time Markov Chains
- HW2Solutions.pdf
- Iterated Proportional Fitting Procedure and Infinite Products of Stochastic Matrices
- Foss Lecture4
- lecture3-LocationManagement
- Modeling Vehicle Safety in Vehicular Networks Using Markov Chain Model Based on Cooperative Awareness_388
- Stochastic Process
- A Markov Model for Human Resources Supply Forecast
- Selected Solutions
- Modified Modified Modified Page Page Rank Rank Model Model in Investigation InvestigationInvestigationInvestigation Investigation of Criminal Criminal Criminal Gang
- Chapter 1 Sections 1 and 2
- Assignment 3
- Notes
- PET SOE
- 09fin
- Regime Switching Liquidity Hedge Funds
- PERFORMANCE AUGEMENTATION of USER SEARCH HISTORIES
- 3
- Complexity Parondo
- fan2014.pdf

You are on page 1of 8

STAT 150 Spring 2006 Lecturer: Jim Pitman Scribe: Stephen Bianchi <>

(These notes also include material from the subsequent guest lecture given by Ani Adhikari.) Consider a continuous time stochastic process (Xt , t 0) taking on values in the nite state space S = {0, 1, 2, . . . , N }. Recall that in discrete time, given a transition matrix P = p(i, j ) with i, j S , the n-step transition matrix is simply P n = pn (i, j ), and that the following relationship holds: P n P m = P n+m . Where pn (i, j ) is the probability that the process moves from state i to state j in n transitions. That is, pn (i, j ) = Pi (Xn = j ), pn (i, j ) = P(Xn = j |X0 = i), pn (i, j ) = P(Xm+n = j |Xm = i). Now moving to continuous time, we say that the process (Xt , t 0) is a continuous time markov chain if the following properties hold for all i, j S , t, s 0: Pt (i, j ) 0,

N j =0

Pt (i, j ) = 1,

N k =0 N k =0

This last property can be written in matrix form as Ps+t = Ps Pt . This is known as the Chapman-Kolmogorov equation (the semi-group property). Example 21.1 (Poissonize a Discrete Time Markov Chain) Consider a pois , which son process (Nt , t 0) with rate , and a jump chain with transition matrix P 21-1

21-2

makes jumps at times (Nt , t 0). Let Xt = YNt (YNt is the value of the jump chain at time Nt ), where Xt takes values in the discrete state space S . Assume (Y0 , Y1 , . . .) and Nt are independent. Find Pt for the process Xt . By denition Pt (i, j ) = P(Xt = j |X0 = i). Condition on Nt to give,

Pt (i, j ) =

n=0

=

n=0

t n=0

)n (i, j ) (tP n! .

t tP

Pt = et(P I ) . Pt is a paradigm example of a continuous time semi-group of transition matrices. It is easy to check that Pt Ps = Pt+s , since eA+B = eA eB .

21.1

Generalization

Let A be a matrix of transition rates, where qij = A(i, j ) is the rate of transitions from state i to state j (i = j ), qij 0, and qi = A(i, i) is the total rate of transitions out of state i. Hence, qi = N j =0,j =i qij and each row of A sums to 0. Also, since Pt is continuous and dierentiable we have qij = lim qi Ph (i, j ) h0 h 1 Ph (i, i) . = lim h0 h

In words, qij is the probability per unit time of transitioning from state i to state j in a small time increment (h). The limit relations above can be expressed in matrix form, Pt+h Pt Pt Ph P t Pt [Ph I ] d Pt = lim = lim = lim h0+ h0+ h0+ dt h h h

21-3

hence

Ph I d d Pt = Pt lim = Pt P0 = Pt A = APt , h 0+ dt h dt

d P. dt 0

where A =

d d is Pt = eAt . dt Pt = Pt A is known as the Kolmogorov forward equations and dt Pt = APt is known as the Kolmogorov backward equations. We claim that (Pt , t 0) is a collection of transition matrices of some continuous time markov chain.

In the example given above we have, I) A = (P (i, i) 1) A(i, i) = (P (i, j ). A(i, j ) = P Example 21.2 Consider the two state markov chain with states {0, 1} and A= .

The sojourn times in state 0 and 1 are independent and exponentially distributed with parameters and , respectively. Find Pt for this chain. First note that d Pt = APt = dt Then Pt (0, 0) + Pt (1, 0) Pt (0, 1) + Pt (1, 1) Pt (0, 0) Pt (1, 0) Pt (0, 1) Pt (1, 1) .

d (Pt (0, 0) Pt (1, 0)) = ( + )(Pt (0, 0) Pt (1, 0)). dt This equation has the form f (t) = cf (t), and a solution is Pt (0, 0) Pt (1, 0) = ce(+)t . Since c evaluated at t = 0 is one (i.e., P0 (0, 0) P0 (1, 0) = 1)), we have Pt (0, 0) Pt (1, 0) = e(+)t , which implies d Pt (0, 0) = e(+)t . dt

21-4

Then

t

Pt (0, 0) = P0 (0, 0) +

0

e(+)s ds

(+)t Pt (0, 0) = 1 + e + + (+)t + e . Pt (0, 0) = + + Likewise (+)t e + + (+)t e Pt (0, 1) = + + (+)t Pt (1, 1) = + e . + + Pt (1, 0) = Notice that as t Pt (0, 0) = Pt (1, 0) = + Pt (0, 1) = Pt (1, 1) = . +

These limits are independent of the initial state. The stationary distribution of the chain is 0 = + . 1 = +

21.2

Let (Xt , t 0) be a markov chain with transition rate matrix A, such that A(i, j ) 0 for i = j and A(i, i) = i=j A(i, j ). Let Hi be the holding time in state i (i.e., the process holds in state i for an amount of time Hi and then jumps to another state). The markov property implies Pi (Hi > s + t|Hi > s) = Pi (Hi > t). We also recognize this as the memoryless property of the exponential distribution (fact: the only continuous distribution with the memoryless property is the exponential).

21-5

By the idea of competing risks we will see that Pi (Hi > t) = eA(i,i)t . Imagine that each state j = i has a random variable j exp(A(i, i)). Think of a bell that rings in state j at time j , and from state i the process moves to the state where the bell rings rst. Construct Hi = mini=j j , then Pi (Hi > t) = P(

i=j

j > t)

= i=j eA(i,j )t P = e( i=j A(i,j ))t = eA(i,i)t , where A(i, i) < 0. What is the probability is that the rst transition out of state i is to state k ? This is given by

P(min j = k ) =

i=j 0

P(min dt, min = k )dt P(k dt, all the others are > t)dt

0

= =

0

0 P

i=j

i=k,j

A(i,k ))t

dt

= =

A(i,j ))t

dt

Observe from this calculation that the following mechanisms are equivalent: 1. (a) competing random variables j exp(A(i, j )) for state j = i (b) Hi = minj =i j (c) jump to the state j which attains the minimum 2. (a) Hi exp(A(i, i) =

i=j

A(i, j ))

A(i,k ) A(i,i)

Example 21.3 (Hold Rates for a Poissonized Chain) Consider a general tran , where some of the diagonal entries maybe greater than zero. So we sition matrix P have Pt A A(i, i) A(i, j ) = = = = et(P I ) I) (P (i, i) 1) = (1 P (i, i)) (P (i, j )). (P

21-6

(i, i)) = Hence A(i, i) = (1 P i=j P (i, j ) = i=j A(i, j ) is the total rate of (i, j )) is the rate of i to j transitions. Let transitions out of state i, and A(i, j ) = (P Hi = S 1 + S 2 + + S G , where S1 , S2 , . . . exp() are independent. Dene G to be the number of steps of the chain until the process leaves state i. Then P (i, i)n1 (1 P (i, i)). P(G = n) = P (i, i) and Hi has an exponential So G has a geometric distribution with parameter 1 P distribution with parameter (1 P (i, i)). It is easy to check the expectations, since (i, i)), and E[Hi ] = 1/(1 P (i, i)). Then by Walds E[Si ] = 1/, E[G] = 1/(1 P identity 1 1 1 . = E[Hi ] = E[Si ]E[G] = (i, i) (i, i)) 1P (1 P

21.3

Theorem

Theorem 21.4 (for nite state irreducible chain) There is a unique probability distribution so that 1. limt Pt (i, j ) = j for all i. 2. solves the balance equations A = 0 (equivalently, Pt = for all t). 3. j is the long-run proportion of time spent in state j 4. the expected return time is Ej [Tj ] = Proof: [for part (2)] Assume (1) is true. Ps+t (i, j ) =

k

1 . q j j

Letting s we get j =

k

k Pt (k, j )

21-7

Look at the j th element of A (which we get by multiplying by the j th column of A). We see that i qij j qj = 0

i=j

i=j

This is the reason these equations are referred to as balance equations. Computations can often be simplied if there is also detailed balance, which simply means that for all pairs i, j (i = j ) we have i qij = j qji . This is the condition of reversibility of the markov chain. Example 21.5 (Birth-Death Process) Consider a process that, from state i, can only move to state i + 1 or state i 1. That is qi,i+1 = i qi,i1 = i qi,j = 0, j {i + 1, i 1}. So i is the birth rate and i is the death rate. We try to nd so that 0 q0,1 = 1 q1,0 1 q1,2 = 2 q2,1 . . . i1 qi1,i = i qi,i1 . Then 0 q0,1 = 1 q1,0 0 0 = 1 1 0 1 = 0 . 1 And 1 q1,2 = 2 q2,1 1 1 = 2 2 1 0 1 1 = 0 . 2 = 2 1 2

21-8

Hence k = and 0 =

0 1 k1 0 , 1 2 k 1

0 1 k1 k =1 1 2 k

1+

- Continuous Time Markov ChainsUploaded byapple_tree_fudge
- HW2Solutions.pdfUploaded byDavid Lee
- Iterated Proportional Fitting Procedure and Infinite Products of Stochastic MatricesUploaded byAnonymous PKVCsG
- Foss Lecture4Uploaded byJarsen21
- lecture3-LocationManagementUploaded byapi-3717973
- Modeling Vehicle Safety in Vehicular Networks Using Markov Chain Model Based on Cooperative Awareness_388Uploaded byAmjad Ullah Khattak
- Stochastic ProcessUploaded byhopanz
- A Markov Model for Human Resources Supply ForecastUploaded bysarbani
- Selected SolutionsUploaded byNghiep Huynh
- Modified Modified Modified Page Page Rank Rank Model Model in Investigation InvestigationInvestigationInvestigation Investigation of Criminal Criminal Criminal GangUploaded byATS
- Chapter 1 Sections 1 and 2Uploaded byWei Shi
- Assignment 3Uploaded bymarcus_boniface
- NotesUploaded bymasahi129
- PET SOEUploaded bykrismarie94
- 09finUploaded byspitzersglare
- Regime Switching Liquidity Hedge FundsUploaded byzyphie
- PERFORMANCE AUGEMENTATION of USER SEARCH HISTORIESUploaded byseventhsensegroup
- 3Uploaded byalisha Dhawan
- Complexity ParondoUploaded byNattapat Bunyuenyongsakul
- fan2014.pdfUploaded byAhmed Yousufzai
- hw4Uploaded bynilyu
- 3230160303_ftpUploaded byPawanKumar Barnwal
- 11MA4ICMAT[1]Uploaded byLeoThomas
- Non Linear Control: Magnetic Levitation SystemUploaded byWahyu Rahmaniar
- TBUploaded byrafal77
- Generalized martingale-residual processes for goodness-of-fit inference in Cox's type regression modelsUploaded byPaweł Marzec
- Binomial ContractingUploaded bytluo001
- Transfer of Scaled MQ-coherence MatricesUploaded byYash Nora
- QueueingUploaded bysuhas
- NetworksUploaded bySDS adad

- ps13Uploaded byspitzersglare
- Lec25Uploaded byspitzersglare
- sol04Uploaded byspitzersglare
- ps08Uploaded byspitzersglare
- ps12Uploaded byspitzersglare
- ps07Uploaded byspitzersglare
- sol01Uploaded byspitzersglare
- ps10Uploaded byspitzersglare
- ps10Uploaded byspitzersglare
- sol05.pdfUploaded byspitzersglare
- sol02Uploaded byspitzersglare
- ps11Uploaded byspitzersglare
- ps11Uploaded byspitzersglare
- ps09Uploaded bysniprip
- sol03Uploaded byspitzersglare
- Mqf Top 10 RanksUploaded byspitzersglare
- Parking Placards Reg 3164Uploaded byspitzersglare
- Parking Placards Reg 3164Uploaded byspitzersglare
- ps04Uploaded bysniprip
- ps06Uploaded byspitzersglare
- ps02Uploaded byspitzersglare
- Probability Jan 12Uploaded byspitzersglare
- ps05Uploaded byspitzersglare
- ps03Uploaded byspitzersglare
- ps03Uploaded byspitzersglare
- ps01Uploaded byspitzersglare
- UntitledUploaded byapi-61200414
- Proofs SolutionsUploaded byspitzersglare
- Midterm2 ConceptsUploaded byspitzersglare
- Polling News and Notes 04-26-12Uploaded byspitzersglare

- Copy of LM324 Data SheetUploaded byS.v. NagaRaj
- Strings & DTM's.labUploaded byAndry Depari
- Which Welding Process is BestUploaded byJohn Maldonado
- mod7Uploaded byMatthew Smith
- Tool to Calculate Project CO2 Emission From Fossil Fuel CombustionUploaded byFrancisco Espinheira Nobre
- GBM's purchase of the Kangaroo Flat mineUploaded byJoe Hinchliffe
- Drug Addiction BIOLOGYUploaded byTarsem Garg
- Jacob Lurie- On Infinity-TopoiUploaded bySwertyy
- Blondeau Ritual MeaningUploaded byMigellango
- Mariner Issue 170Uploaded byPat Reynolds
- Power Point Vietnamese CultureUploaded byNestor Balboa
- 720001733 EnUploaded byglobus999
- Refrigeration and cryogenicsUploaded byJunhyo Lee
- Labor and DeliveryUploaded bydexter
- Janowcyk Andrew Publication ListUploaded byJessicxMasenrp
- Final Case StudyUploaded byalihsohi
- Pneumatic Bar Feeding 3 Cylinders 2013 FinalUploaded bySelva Ganapathy
- B GL 332 006 Engineering Insert (2000)Uploaded byJustin
- Exercise IndexUploaded byalex_carnall6060
- Schlage L SeriesUploaded byb.silver3892
- Raised Floor MethodologyUploaded byMaSh Joaconcho Ybañez
- Bas, Van Geel Et Al. 2011Uploaded byMarla Saldanha
- Asus Eee PC 1215T Rev1.0Uploaded byOsman Efendi
- Addition and Subtraction of FractionsUploaded byAlano Jo
- Medical Devices Harmonized StandardUploaded byhitham shehata
- ADDMATH FORM 4Uploaded byNor Hanina
- Epoxy.pdfUploaded bymuti_khan20006003
- Bare_AAACUploaded byAjit Pillai
- Field and Galois Theory - J.S. MilneUploaded byYNN1
- naeniacornubiaed00borluoftUploaded byjorge naught