9 views

Uploaded by Songya Pan

Stochastic Models I (for first-year doctoral students)

- MDP.pdf
- On the Convergence of Parallel Simulated Annealing
- markov-chains
- Markov09(1)
- Markov Chain
- s to Chas Tic Columbia 4
- Multi-scale Representation of High Frequency
- Syllabus of M.a. M.sc Part II
- ProblemsMarkovChains
- Lec18
- IJCAI11-113
- (17)a High-resolution Stochastic Model of Domestic Activity Patterns
- pachet-11b
- E_bdpros
- Application to Matrix Algebra
- prelimwork.pdf
- aglect
- A We 762014 Bj Mcs 12415
- 08274909
- 2018-Novel binary encoding water cycle algorithm for solving Bayesian network structures learning problem.pdf

You are on page 1of 5

Problem 4.40 Consider a segment of a sample path beginning and ending in state i, with no visit to i in between, i.e, the vector (i, j1 , j2 , j3 , . . . , jn1 , jn = i), where jk = i for the non-end states jk . Going forward in time, the probability of this segment is i Pi,j1 Pj1 ,j2 Pj2 ,j3 Pjn1 ,i . The probability, say p, of the reversed sequence (i, jn1 , jn2 , jn3 , . . . , j1 , j0 = i) under the reverse DTMC with transition matrix j Pj,i P i,j i is p = i P i,jn1 P jn1 ,jn2 P jn2 ,jn3 P j1 ,i . However, successively substituting in the reverse-chain transition probabilities, we get p = i jn1 Pjn1 ,i P jn1 ,jn2 P jn2 ,jn3 P j1 ,i i = Pjn1 ,i jn1 P jn1 ,jn2 P jn2 ,jn3 P j1 ,i j Pj ,j = Pjn1 ,jn2 jn1 n2 n2 n1 P jn1 ,jn2 P jn2 ,jn3 P j1 ,i jn1 = Pjn1 ,i Pjn2 ,jn1 Pjn3 ,jn2 . . . Pj1 ,j2 j1 P j1 ,i = Pjn1 ,i Pjn2 ,jn1 Pjn3 ,jn2 . . . Pj1 ,j2 Pi,j1 i = i Pi,j1 Pj1 ,j2 Pj2 ,j3 Pjn1 ,i . Problem 4.41 (a) The reverse time chain has transition matrix j Pj,i P i,j i To nd it, we need to rst nd the stationary vector . By symmetry (or by noting that the chain is doubly stochastic), j = 1/n, j = 1, , n. Hence,

Pij = j Pji /i = Pji =

p 1p

if j = i 1 if j = i + 1

(b) In general, the DTMC is not time reversible. It is in the special case p = 1/2. Otherwise, the probabilities of clockwise and counterclockwise motion are reversed.

Problem 4.42 Imagine that there are edges between each of the pair of nodes i and i + 1, i = 0, , n 1, and let the weight on edge (i, i + 1) be wi , where w0 = 1

i

wi =

j =1

pj , qj

i1

where qj = 1 pj . As a check, note that with these weights Pi,i+1 = pi /qi wi = = pi , wi1 + wi 1 + pi /qi 0<i<n.

Since the sum of the weights on edges out of node i is wi1 + wi , i = 1, , n 1, it follows that i = c

j =1

i1

pj + qj

i j =1

0 = c pj c = qj qi pj qj

i1 j =1

pj , 0<i<n qj

n1

n = c

j =1

n j =0 j

= 1.

Problem 4.46 (a) Yes, it is a Markov chain. It suces to construct the transition matrix and verify that the process has the Markov property. Let P be the new transition matrix. Then we have, for 0 i N and 0 j N ,

Pi,j = Pi,j + k=N +1

Pi,k Bk,j ,

(N )

where Bk,j is the probability of absorption into the absorbing state j in the absorbing Markov chain, where the states N + 1, N + 2, . . . are the transient states, while the state (N ) 1, 2, . . . N are the N absorbing states. In other words, Bk,j is the probability that the next state with index in the set {1, 2, . . . , N } visited by the Markov chain, starting with k > N is in fact j . It is easy to see that the markov property is still present. (b) The proportion of time in j is j /

N i=1 i .

(N )

(c) Let i (N ) be the steady-state probabilities for the chain, only counting to visits among the states in the subset {1, 2, . . . , N }. (This chain is necessarily positive recurrent.) By renewal theory, i (N ) = (E [Number of Y transitions between Y visits to i)1 2

and j (N ) = = E [No. Y -transitions to j between Y visits to i] E [No. Y -transitions to i between Y visits to i] E [No. X -transitions to j between X visits to i] 1/i (N )

(d) For the symmetric random walk, the new MC is doubly stochastic, so i (N ) = 1/(N +1) for all i. By part (c), we have the conclusion. (e) It suces to show that

i (N )Pi,j = j (N )Pj,i

i (N )Pi,j

= i (N )Pi,j + i (N )

k=N +1

Pi,k Bk,j ,

(N )

and

j (N )Pj,i

= j (N )Pj,i + j (N )

k=N +1

Pj,k Bk,i ,

(N )

The two terms on the right are equal in these two displays. First, by the original reversibility, we have i (N )Pi,j = j (N )Pj,i . Second, by Theorem 4.7.2, we have

j (N )

k=N +1

Pj,k Bk,i = i (N )

k=N +1

(N )

Pi,k Bk,j .

(N )

We see that by expanding into the individual paths, and seeing that there is a reverse path. Problem 4.47 Intuitively, in steady state each ball is equally likely to be in any of the urns and the positions of the balls are independent. Hence it seems intuitive that (n) = M! n1 ! nm ! 1 m

M

To check the above and simultaneously establish time reversibility let n = (n1 , , ni1 , ni 1, ni+1 , , nj 1 , nj + 1, nj +1 , , nm )

M

ni 1 M m1 1 m

M

nj + 1 1 M m1

Problem 4.48 (a) Each transition into i begins a new cycle. A reward of 1 is earned if state visited from i is j . Hence average reward per unit time is Pij /ii . (b) Follows from (a) since 1/jj is the rate at which transitions into j occur. (c) Suppose a reward rate of 1 per unit time when in i and heading for j . New cycle whenever enter i. Hence, average reward per unit time is Pij ij /ii . (d) Consider (c) but now only give a reward at rate 1 per unit time when the transition time from i to j is within x time units. Average reward is E[Reward per cycle] E[Time of cycle] = = = where Xij Fij . Problem 4.49

t

limt P(S (t) = j, X (t) = i) P(X (t) = i) Pij 0 Fij (y )dy/ii by Theorem 4.8.4 Pi Pij ij i

(b) P(heading for 2) = P1 (c) fraction of time from 2 to 3 = P2 P23 t23 8 60 3 = = 2 38 80 19 3 P12 t12 15 10 = = 1 38 25 19

- MDP.pdfUploaded byHrishikesh Khandeparkar
- On the Convergence of Parallel Simulated AnnealingUploaded byDomingo Ben
- markov-chainsUploaded byelgianka
- Markov09(1)Uploaded byمحمد فرقان
- Markov ChainUploaded byactivation2007a
- s to Chas Tic Columbia 4Uploaded byRyan Davis
- Multi-scale Representation of High FrequencyUploaded bydoc_oz3298
- Syllabus of M.a. M.sc Part IIUploaded bynav00007777
- ProblemsMarkovChainsUploaded byJuan Carlos Fdz
- Lec18Uploaded byspitzersglare
- IJCAI11-113Uploaded bystef
- (17)a High-resolution Stochastic Model of Domestic Activity PatternsUploaded byEva Viskadouraki
- pachet-11bUploaded byjuan perez arrikitaun
- E_bdprosUploaded bySaurav Karanjai
- Application to Matrix AlgebraUploaded byAqma Najihah
- prelimwork.pdfUploaded byAnnabel Amor Antig
- aglectUploaded byFuzzylatex
- A We 762014 Bj Mcs 12415Uploaded byhc87
- 08274909Uploaded bydan_intel6735
- 2018-Novel binary encoding water cycle algorithm for solving Bayesian network structures learning problem.pdfUploaded byShahid Younas
- EE-684 System ReliabilityUploaded byNamrajit Dey
- Stochastic ProcessUploaded byCarlos de la Calleja
- MIT15_450F10_lec06Uploaded byseanwu95
- M347-201806Uploaded byapolaz
- Bourke(2001)Uploaded byanon_413169311
- Basic Elements of Queueing Theory lec notes Philippe NAINUploaded byjoystick2in

- midtermTwo6711F12solsUploaded bySongya Pan
- midtermOne6711F10solsUploaded bySongya Pan
- midtermOne6711F08solsUploaded bySongya Pan
- midtermOne6711F12solsUploaded bySongya Pan
- Midterm Two 6711 f 10Uploaded bySongya Pan
- Midterm One 6711 f 10Uploaded bytabrupt
- Midterm One 6711 f 08Uploaded bySongya Pan
- Midterm Two 6711 f 12Uploaded bySongya Pan
- Lect 0910Uploaded bySongya Pan
- Midterm Two f 07Uploaded bySongya Pan
- Midterm One 6711 f 12Uploaded bySongya Pan
- Lect 0903Uploaded bySongya Pan
- Lect 0905Uploaded bySongya Pan
- midtermTwoF07solsUploaded bySongya Pan
- midtermTwo6711F10solsUploaded bySongya Pan
- homewkLTUploaded bySongya Pan
- homewk13Uploaded bySongya Pan
- homewk10Uploaded bySongya Pan
- homewk12Uploaded bySongya Pan
- homewk10SolsUploaded bySongya Pan
- homewk8Uploaded bySongya Pan
- homewk11Uploaded bySongya Pan
- homewk13SolsUploaded bySongya Pan
- homewk8SolsUploaded bySongya Pan
- homewk7SolsUploaded bySongya Pan
- homewk7Uploaded bySongya Pan
- homewk9SolsUploaded bySongya Pan
- homewk9Uploaded bySongya Pan
- homewk12SolsUploaded bySongya Pan

- fbla 276Uploaded byapi-324719969
- Aluminium Foil Af546 SwUploaded byfhsn84
- Moral Distress Among Nursing and Non-nursing StudentsUploaded byRomielle Grazel Joi Ramos
- moline-roberts articleUploaded byapi-244230664
- Guidelines - Inferential Questions AssessmentUploaded byDavid Woo
- ALP - Adverb and Adverbial Phrases Improving our skills with exercisesUploaded bySandra Velásquez
- Algebra.docxUploaded byTahajan Hadjirul
- POM- 1st AssignmentUploaded byKaleem Sami
- Andrew Arato - Part IV IntroductionUploaded bylmzds
- Eel 4309Uploaded byMarvin Sinues
- Constructing Norms of Humanitarian InterventionUploaded byMauludina Fernanda Putri Gusman
- 1-2018 Editorial WritingUploaded byIan Jumalin
- ethno bank.pdfUploaded byMuhammad Nasrum
- Lecture #1 - Modern Programming Platforms (CSHTP3e_03)Uploaded byOrçan Yedal Öksüz
- PDF ENGLISHUploaded bypaucaste
- works citedUploaded byapi-316945711
- Ucsb Personal StatementUploaded byJohnLin
- Multiple Intelligences Theory[1][1]Uploaded bySameera Dalvi
- Hrm Project on NIB Bank PakistanUploaded byarzoo786
- Impact Analysis Checklist for Requirements ChangesUploaded bysumitkalra3962
- Di Zi Gui :Standards for Being a Good Student and Child - Seven Subjects and 113 RulesUploaded byYau Yen
- Studies in Soviet Thought Volume 20 Issue 3 1979 [Doi 10.2307%2F20098903] David M. Rasmussen -- Marx- On Labor, Praxis and Instrumental ReasonUploaded byAndrés Serrano
- Just War Theory SummaryUploaded byDženan Hakalović
- Inherent Powers of the StateUploaded byMarko Lopez
- Valenzuela v. Maricopa, Ariz. Ct. App. (2015)Uploaded byScribd Government Docs
- English Grammar - Cases of Nouns and PronounsUploaded byChiby Mirikitufu
- Turkish Vocabulary WIKIGBUploaded byJean-pierre Negre
- 02 SynopsisUploaded byFatma Fabigha
- r Cheat SheetUploaded byHaritha Atluri
- Lukes Methodological Individualism ReconsideredUploaded bymihai_valentinlaurentiu83