EC744 Lecture Note 7 Stochastic Dynamic Programming

Prof. Jianjun Miao

Principle of Optimality Measurable spaces (X, X ), (Z, Z) Transition function Q : Z Z [0, 1] Return function F : X X Z R Constraint correspondence : X Z X. Let the state space be S = X Z.

A plan is a value 0 X and a sequence of measurable functions t : Z t X, t = 1, 2, ... A plan is feasible from s0 S if (a) 0 (s0), (b) t z t t1 z t1 , zt , all z t Z t, t = 1, 2, ... The set of feasible plans from s0 is denoted by (s0) The transition function Q and the initial state s0 = (x0, z0) induce a probability measure t (z0, ) : Z t [0, 1].

Sequence Problem v (s0) = sup +

(s0) XZ

F (x0, 0, z0)
tF t1 , z t , z t z , dz t . t t t1 z 0 t

t=1 Z

Dynamic Programming Problem (Bellman Equation) v(x, z) = sup

x0(x,z)

F (x, x0, z) +

v(x0, z 0)Q(z, dz 0)

The policy correspondence G : S X is dened as G (x, z) = y (x, z) : v(x, z) = F (x, y, z) +

v(y, z 0)Q(z, dz 0)

If the policy correspondence is single valued, then it is called a policy function, and denoted by g(x, z)

Impose Assumption 9.1 such that (s0) is nonempty. Impose Assumption 9.2 such that the sequence problem is well dened. Dene un (, s0) : (s0) R as follows: u0 (, s0) = F (x0, 0, z0) , un (, s0) = F (x0, 0, z0) +
n XZ

t=1

h i tF t1 , z t , z t z , dz t t t t1 z 0

Dene u (, s0) = limn un (, s0) . Then v (s) = sup u (, s) .

(s)

Key issue of stochastic models: Measurability

Theorem. Let (X, X ) , (Z, Z) , Q, , F and be given. Let Assumptions 9.1 and 9.2 hold, and let v be dened by (2). Let v be a measurable function satisfying the Bellman equation, and such that
t Z

lim

all (s0) , all (x0, z0) = s0 S. Let G be the policy correspondence, and suppose that G is nonempty and permits a measurable selection. Then v = v , and any plan generated by G attains the supremum in the sequence problem. Proof: Repeated substitution

tv t1 , z t z , dz t = 0 t t1 z 0 t

Theorem: Let (X, X ) , (Z, Z) , Q, , F, and be given. Let Assumptions 9.1-9.3 hold. Assume that v is measurable and satises the Bellman equation. Assume that G is nonempty and permits a measurable selection. Let (x0, z0) = s0 S, and let (s0) be a plan that attains the supremum in the sequence problem. Then there exists a plan G generated by G from s0 such that G = , and 0 0 G z t = z t , t (z0, ) -a.e., t = 1, 2, ... t t Proof: Recursive relation

Bounded Returns Assumption 9.4. X is a convex Borel set in Rl , with its Borel subsets X . Assumption 9.5. One of the following conditions holds: a. Z is a countable set and Z is the -algebra containing all subsets of Z; or b. Z is a compact (Borel) set in Rk , with its Borel subsets Z, and the transition function Q on (Z, Z) has the Feller property.

Lemma. Let (X, X ) , (Z, Z) and Q satisfy Assumptions 9.4 and 9.5. If f : X Z R is bounded and continuous, then Mf dened by (M f ) (y, z) =
Z 0 Q z, dz 0 , all (y, z) X Z, f y, z

is also; that is, M : C (S) C (S) . If f is increasing (strictly increasing) in each of its rst l arguments, then so is Mf ; and if f is concave (strictly concave) jointly in its rst l arguments, then so is M f. What if Z is not compact? Need Z to be convex. See Lemma 12.14.

Assumption 9.6. The correspondence : X Z X is nonempty, compactvalued, and continuous. Assumption 9.7. The function F : A R is bounded and continuous, and (0, 1). Theorem (Existence). Let (X, X ) , (Z, Z) , Q, , F, and satisfy Assumptions 9.4-9.7, and dene the operator T on C (S) by T f (x, z) = sup
y(x,z)

F (x, y, z) +

0 Q z, dz 0 . f y, z

Then T : C (S) C (S) ; T has a unique xed point v in C (S) and for any v0 C (S) . Moreover, the correspondence G : S X dened by G (x, z) = y (x, z) : v (x, z) = F (x, y, z) + is nonempty, compact-valued, and u.h.c.
Z

v y, z 0 Q z, dz 0

Assumption 9.8. For each (y, z) X Z, F (, y, z) : X R is strictly increasing. Assumption 9.9. For each z Z, (, z) : X X is increasing in the sense that x x0 implies that (x, z) x0, z . Theorem (Monotonicity). Let (X, X ) , (Z, Z) , Q, , F and satisfy Assumptions 9.4-9.9, and let v be the unique xed point of the operator T . Then for each z Z, v (, z) : X R is strictly increasing.

Assumption 9.10. For each z Z, F (, , z) : X X R satises all (0, 1) , and all (x, y) , x , y x 6= x0.

0, y 0 , z F (x, y, z) + (1 ) F x0, y 0, z F (x, y) + (1 ) x 0 0

X X and the inequality is strict if

Assumption 9.11. For all z Z and all x, x0 X,

0 x + (1 ) x0, z all [0, 1] . y + (1 ) y

0 x0, z implies y (x, z) and y

Theorem (Concavity). Let (X, X ) , (Z, Z) , Q, , F, and satisfy Assumptions 9.4-9.7 and 9.10-9.11; let v be the unique xed point of the operator T ; and let G be the policy correspondence. The for each z Z, v (, z) : X R is strictly concave and G (, z) : X X is a continuous (single-valued) function.

Theorem (Convergence). Let (X, X ) , (Z, Z) , Q, , F, and satisfy Assumptions 9.4-9.7 and 9.10-9.11; let C 0 (S) C (S) be the set of bounded continuous functions on S that are weakly concave jointly in their rst l arguments; let v C 0 (S) be the unique xed point of the operator T ; and let g = G be the (single-valued) policy function. Let v0 C 0 (S) and dene {(vn, gn)} by vn = T vn1 and gn (x, z) = arg max F (x, y, z) +
y(x,z)

0 Q z, dz 0 . vn y, z

Then gn g pointwise. If X and Z are both compact, then the convergence is uniform.

Assumption 9.12. For each xed z Z, F (, , z) is continuously dierentiable in (x, y) . Theorem (Dierentiability). Let (X, X ) , (Z, Z) , Q, , F, and satisfy Assumptions 9.4-9.7 and 9.10-9.11; let v be the unique xed point of the operator T ; and let G be the policy correspondence. If x0 int X and g (x0, z0) int (x0, z0), then v (, z0) is continuously dierentiable in x at x0, with derivatives given by vi (x0, z0) = Fi (x0, g (x0, z0) , z0) , i = 1, ..., l.

Assumption 9.13 For each (x, y) X X, F (x, y, ) is strictly increasing. Assumption 9.14 For each x X, (x, ) : Z X is increasing in the sense that z z 0 implies (x, z) x, z 0 . Assumption 9.15 Q is monotone. Theorem (Monotonicity). Let (X, X ) , (Z, Z) , Q, , F, and satisfy Assumptions 9.4-9.7 and 9.13-9.15; let v be the unique xed point of the operator T . Then for each x X, v (x, ) : Z R is strictly increasing.

Another Formulation

v (x, z) = subject to

sup
y(x,z)

F (x, y, z) +

0, z 0 Q z, dz 0 v x

We need to impose additional assumptions on . The analysis is similar.

0 = x, y, z 0 . x

Unbounded Returns v (x, z) = sup

y(x,z)

F (x, y, z) +

0 Q z, dz 0 v y, z

Boundedness condition may be voilated,

t t1 , z t z , dz t = 0 lim v t1 z t 0 t t Z Z

all (s0) , all (x0, z0) = s0 S.

b Dene (s0) to be the subset of (s0) on which the above condition holds. Then dene b v (s) = sup u (, s) . b (s)

Theorem. Let (X, X ) , (Z, Z) , Q, , F and be given. Let Assumptions b b 9.1 and 9.2 hold, and let v , , and v be dened as before. Let v be a measurable function satisfying the Bellman equation, and the associated policy correspondence G is nonempty and permits a measurable selection. For each s S, let (, s) be a plan generated by G from s. Suppose in addition that
b a. (; s) (s) , all s S; and

b b b b. for any s S and (s) , there exists (s) such that u (, s) u (, s) .

Then
b v (s) = v (s) = v (s) = u ( (; s) , s)

Example 1. Stochastic Growth max E

t=0

subject to

t ln (ct)

ct + kt+1 = ztkt , zt iid

DP v (k, z) = Solution k0 = g (k, z) = zk, 1 v (k, z) = A0 + ln (k) + ln (z) . 1 1

k0 [0,zk]

max

ln zk k0 +

0, z 0 Q dz 0 v k

Verify (a): satises the boundedness condition.

b Verify (b): We show that for any (s0) , either (s0) or u (, s0) = , so is dominated by .

Plug the value function in the boundedness condition: lim tEt t But
(

1 A0 + ln (kt) + ln (zt) = 0. 1 1
X

u (, s0)

tEt [ ln kt + ln zt]

t=0

If boundedness condition is violated, RHS diverge to .

Example 2. Investment with Quadratic Adjustment Cost max E where

X

1 2 1 t ztkt bkt c (kt+1 kt)2 , 2 2 t=0 zt+1 = zt + ut+1, ut+1 iid

DP
2 1 2 1 0 v (k, z) = max zk bk c k k + 2 2 k0 Solution Z 0, z 0 Q z, dz 0 v k

v (k, z) = v0 + v1z + v2k + v3zk v4k2, k0 = g (k, z) = g0 + g1k + g2z.

Constant Returns to Scale Assumption 9.18. X Rl is a convex cone, with its Borel subsets X . Assumption 9.19. The correspondence : X X X is nonempty, compact-valued, and continuous; for any (x, z) X Z,
1 , and for some 0,

y (x, z) implies y (x, z) , all 0;

||y|| ||x|| , all y (x, z) , all (x, z) X Z.

Assumption 9.20. The function F : A R is continuous; for each z Z, the function F (, , z) : Az R is homogeneous of degree one; for some B > 0, |F (x, y, z)| B (||x|| + ||y||) , all (x, y, z) A; and the discount factor is (0, 1) .

Take the space H (S) of continuous functions f : S = X Z R that are homogeneous of degree one in x for each xed z, and such that ||f || = sup H (S) is a Banach space. Apply contraction mapping theorem to T on the space H (S) . sup
zZ xX,||x||1

|f (x, z)| < .

Stochastic Euler Equation. max F (x0, 0, z0) + FOC

t1 , z t , z 0 = Fy t1 z t t Z t , t+1 , z + Fx t z t+1 z t+1 Q (zt, dzt+1) i tF z t , t+1 , z t = 0. lim Et x t t+1 z t+1 t z t h
X

t=1

t1 , z t , z t z , dz t F t1 z t t 0 t Z

Transversality condition