Paulo Brito Ecomat Discreto

Mathematical Economics
Deterministic dynamic optimization

Discrete time
Paulo Brito
pbrito@iseg.utl.pt
4.12.2012
1
Paulo Brito Mathematical Economics, 2012/13 0
Contents
1 Introduction 1
1.1 Deterministic and optimal sequences . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Some history . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Types of problems studied next . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.4 Some economic applications . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2 Calculus of Variations 6
2.1 The simplest problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2 Free terminal state problem . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.3 Free terminal state problem with a terminal constraint . . . . . . . . . . . . 16
2.4 Innite horizon problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3 Optimal Control and the Pontriyagins principle 20
3.1 The simplest problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.2 Free terminal state . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.3 Free terminal state with terminal constraint . . . . . . . . . . . . . . . . . . 26
3.4 The discounted innite horizon problem . . . . . . . . . . . . . . . . . . . . 29
4 Optimal control and the dynamic programming principle 37
4.1 The nite horizon problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.2 The innite horizon problem . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
5 Bibliographic references 44
A Second order linear dierence equations 45
A.1 Autonomous problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
A.2 Non-autonomous problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
1 Introduction
We introduce deterministic dynamic optimization problems and three methods for solving
them.
Deterministic dynamic programming deals with nding deterministic sequences which
verify some given conditions and which maximize (or minimise) a given intertemporal cri-
terium.
1.1 Deterministic and optimal sequences
Consider the time set T = {0, 1, . . . , t, . . . , T} where T can be nite or T = . We denote
the value of variable at time t, by x
t
. That is x
t
is a mapping x : T R.
The timing of the variables dier: if x
t
can be measured at instant t we call it a state
variable, if u
t
takes values in period t, which takes place between instants t and t + 1 we
call it a control variable.
Usually, stock variables (both prices and quantities) refer to instants and ow variables
(prices and quantities) refer to periods.
A dynamic model is characterised by the fact that sequences have some form of in-
tertemporal time-interaction. We distinguish intratemporal from intertemporal relations.
Intratemporal, or period, relations take place within a single period and intertemporal rela-
tions involve trajectories.
A trajectory or path for state variables starting at t = 0 with the horizon t = T is denoted
by x = {x
0
, x
1
, . . . , x
T
}. We denote the trajectory starting at t > 0 by x
t
= {x
t
, x
t+1
, . . . , x
T
}
and the trajectory up until time
t
x = {x
0
, x
1
, . . . , x
t
}.
We consider two types of problems:
1. calculus of variations problems: feature sequences of state variables and evaluate these
sequences by an intertemporal objective function directly
J(x) =
T1
t=0
F(t, x
t
, x
t+1
)
2. optimal control problems: feature sequences of state and control variables, which are
related by a sequence of intratemporal relations
x
t+1
= g(x
t
, u
t
, t) (1)
and evaluate these sequences by an intertemporal objective function over sequences
(x, u)
J(x, u) =
T1
t=0
f(t, x
t
, u
t
)
From equation (1) and the value of the state x
t
at some points in time we could also determine
an intertemporal relation
1
.
In a deterministic dynamic model there is full information over the state x
t
or the path
x
t
for any t > s if we consider information at time s.
In general we have some conditions over the value of the state at time t = 0, x
0
and
we may have other restrictions as well. The set of all trajectories x verifying some given
conditions is denoted by X. In optimal control problems the restrictions may involve both
state and control sequence, x and u. In this case we denote the domain of all trajectories by
D
Usually X, or D, have innite number of elements. Deterministic dynamic optimisation
problems consist in nding the optimal sequences x
X (or (x
, u
) D).
1.2 Some history
The calculus of variations problem is very old: Didos problem, brachistochrone problem
(Galileo), catenary problem and has been solved in some versions by Euler and Lagrange
1
In economics the concept of sustainability is associated to meeting those types of intertemporal relations.
(XVII century) (see Liberzon (2012). The solution of the optimal control problem is due to
Pontryagin et al. (1962). The dynamic programming method for solving the optimal control
problem has been rst presented by Bellman (1957).
1.3 Types of problems studied next
The problems we will study involve maximizing an intertemporal objective function (which
is mathematically a functional) subject to some restrictions:
1. the simplest calculus of variations problem: we want to nd a path {x
t
}
T
t=0
,
such that T is known, such that both the initial and the terminal values of the state
variable are known , x
0
=
0
and x
T
=
T
such that it maximizes the functional
T1
t=0
F(x
t+1
, x
t
, t). Formally, the problem is: nd a trajectory for the state of the
system, {x
t
}
T
t=0
, that solves the problem
max
{xt}
T
t=0
T1
t=0
F(x
t+1
, x
t
, t), s.t. x
0
=
0
, x
T
=
T
where
0
,
T
and T are given;
2. calculus of variations problem with a free endpoint: this is similar to the
previous problem with the dierence that the terminal state x
T
is free. Formally:
max
{xt}
T
t=0
T1
t=0
F(x
t+1
, x
t
, t), s.t. x
0
=
0
, x
T
free
where
0
and T are given;
3. the optimal control problem with given terminal state: we assume there are
two types of variables, control and state variables, represented by u and x which are
related by the dierence equation x
t+1
= g(x
t
, u
t
). We assume that the initial and
the terminal values of the state variable are known x
0
=
0
and x
T
=
T
and we
want to nd an optimal trajectory joining those two states such that the functional
T1
t=0
F(u
t
, x
t
, t) is maximized by choosing an optimal path for the control.
Formally, the problem is: nd a trajectories for the control and the state of the system,
{u
t
}
T1
t=0
and {x
t
}
T
t=0
, which solve the problem
max
{ut}
T
t=0
T1
t=0
F(u
t
, x
t
, t), s.t. x
t+1
= g(x
t
, u
t
), t = 0, . . . T 1, x
0
=
0
, x
T
=
T
where
0
,
T
and T are given;
4. the optimal control problem with free terminal state: nd a trajectories for the
control and the state of the system, {u
t
}
T1
t=0
and {x
t
}
T
t=0
max
{ut}
T
t=0
T1
t=0
F(u
t
, x
t
, t), s.t. x
t+1
= g(x
t
, u
t
), t = 0, . . . T 1, x
0
=
0
, x
T
=
T
where
0
and T are given.
5. in macroeconomics the innite time discounted optimal control problem is the
most common: nd a trajectories for the control and the state of the system, {u
t
}
t=0
and {x
t
}
t=0
max
{ut}
t=0
T1
t=0
t
F(u
t
, x
t
), s.t. x
t+1
= g(x
t
, u
t
), t = 0, . . . , x
0
=
0
,
where (0, 1) is a discount factor and
0
is given. The terminal condition
lim
t
t
x
t
0 is also frequently introduced, where 0 < < 1.
There are three methods for nding the solutions: (1) calculus of variations, for the rst
two problems, which is the reason why they are called calculus of variations problems, and
(2) maximum principle of Pontriyagin and (3) dynamic programming, which can be used for
all the ve types of problems.
1.4 Some economic applications
The cake eating problem : let W
t
be the size of a cake at instant t. If we eat C
t
in period
t, the size of the cake at instant t + 1 will be W
t+1
= W
t
C
t
. We assume we know that
the cake will last up until instant T. We evaluate the bites in the case by the intertemporal
utility function featuring impatience, positive but decreasing marginal utility
T1
t=0
t
u(C
t
)
If the initial size of the cake is
0
and we want to consume it all until the end of period
T 1 what will be the best eating strategy ?
The consumption-investment problem : let W
t
be the nancial wealth of a consumer
at instant t. The intratemporal budget constraint in period t is
W
t+1
= Y
t
+ (1 + r)W
t
C
t
, t = 0, 1, . . . , T 1
where Y
t
is the labour income in period t and r is the asset rate of return. The consumer has
nancial wealth W
0
initially. The consumer wants to determine the optimal consumption
and wealth sequences {C
t
}
T1
t=0
and {W
t
}
T
t=0
that maximises his intertemporal utility function
T1
t=0
t
u(C
t
)
where T can be nite or innite.
The AK model growth model: let K
t
be the stock of capital of an economy at time
and consider the intratemporal aggregate constraint of the economy in period t
K
t+1
= (1 + A)K
t
C
t
where F(K
t
) = AK
t
is the production function displaying constant marginal returns. Given
the initial capital stock K
0
the optimal growth problem consists in nding the trajectory
{K
t
}
t=0
that maximises the intertemporal utility function
t=0
t
u(C
t
)
subject to a boundedness constraint for capital. The Ramsey (1928) model is a related
model in which the production function displays decreasing marginal returns to capital.
2 Calculus of Variations
Calculus of variations problems were the rst dynamic optimisation problems involving nd-
ing trajectories that maximize functionals given some restrictions. A functional is a function
of functions, roughly. There are several types of problems. We will consider nite horizon
(known terminal state and free terminal state) and innite horizon problems.
2.1 The simplest problem
The simplest calculus of variations problem consists in nding a sequence that max-
imizes or minimizes a functional over the set of all trajectories {x} {x
t
}
T
t=0
, given initial
and a terminal value for the state variable, x
0
and x
T
.
Assume that F(x
, x) is continuous and dierentiable in (x
, x). The simplest problem

of the calculus of variations is to nd one (or more) optimal trajectory that maximizes the
value functional
max
{x}
T1
t=0
F(x
t+1
, x
t
, t) (2)
where the function F(.) is called objective function
subject to x
0
=
0
and x
T
=
T
(3)
where
0
and
T
are given.
Observe that the the upper limit of the sum should be consistent with the horizon of the
problem T. In equation (2) the value functional is
V ({x}) =
T1
t=0
F(x
t+1
, x
t
, t)
= F(x
1
, x
0
, 0) + F(x
2
, x
1
, 1) + . . . + F(x
t
, x
t1
, t 1) + F(x
t+1
, x
t
, t) + . . .
. . . + F(x
T
, x
T1
, T 1)
because x
T
is the terminal value of the state variable.
We denote the solution of the calculus of variations problem by {x
t
}
T
t=0
.
The optimal value functional is a number
V
V ({x
}) =
T1
t=0
F(x
t+1
, x
t
, t) = max
{x}
T1
t=0
F(x
t+1
, x
t
, t).
Proposition 1. (First order necessary condition for optimality)
Let {x
t
}
T
t=0
be a solution for the problem dened by equations (2) and (3). Then it veries
the Euler-Lagrange condition
F(x
t
, x
t1
, t 1)
x
t
+
F(x
t+1
, x
t
, t)
x
t
= 0, t = 1, 2, . . . , T 1 (4)
and the initial and the terminal conditions
_
_
x
0
=
0
, t = 0
x
T
=
T
, t = T.
Proof. Assume that we know the optimal solution {x
t
}
T
t=0
. Therefore, we also know the
optimal value functional V ({x
}) =
T1
t=0
F(x
t+1
, x
t
, t). Consider an alternative candidate
path as a solution of the problem, {x
t
}
T1
t=0
such that x
t
= x
t
+
t
. In order to be admissible, it
has to verify the restrictions of the problem. Then, we may choose
t
= 0 for t = 1, . . . , T 1
and
0
=
T
= 0. That is, the alternative candidate solution has the same initial and
terminal values as the optimal solution, although following a dierent path. In this case the
value function is
V ({x}) =
T1
t=0
F(x
t+1
+
t+1
, x
t
+
t
, t).
where
0
=
T
= 0. The variation of the value functional introduced by the perturbation
{}
T1
t=1
is
V ({x}) V ({x
}) =
T1
t=0
F(x
t+1
+
t+1
, x
t
+
t
, t) F(x
t+1
, x
t
, t).
If F(.) is dierentiable, we can use a rst order Taylor approximation, evaluated along the
trajectory {x
t
}
T
t=0
,
V ({x}) V ({x
}) =
F(x
0
, x
1
, 0)
x
0
(x
0
x
0
) +
_
F(x
0
, x
1
, 0)
x
1
+
F(x
2
, x
1
, 1)
x
1
_
(x
1
x
1
) + . . .
. . . +
_
F(x
T1
, x
T2
, T 2)
x
T1
+
F(x
T
, x
T1
, T 1)
x
T1
_
(x
T1
x
T1
) +
+
F(x
T
, x
T1
, T 1)
x
T
(x
T
x
T
) =
=
_
F(x
0
, x
1
, 0)
x
1
+
F(x
2
, x
1
, 1)
x
1
_
1
+ . . .
. . . +
_
F(x
T1
, x
T2
, T 2)
x
T1
+
F(x
T
, x
T1
, T 1)
x
T1
_
T1
because x
t
x
t
=
t
and
0
=
T
= 0 Then
V (x) V (x
) =
T1
t=1
_
F(x
t
, x
t1
, t 1)
x
t
+
F(x
t+1
, x
t
, t)
x
t
_
t
. (5)
If {x
t
}
T1
t=0
is an optimal solution then V ({x})V ({x
}) = 0, which holds if (4) is veried.

Interpretation: equation (4) is an intratemporal arbitrage condition for period t. The
optimal sequence has the property that at every period marginal benets (from increasing
one unit of x
t
) are equal to the marginal costs (from sacricing one unit of x
t+1
):
Observations
equation (4) is a non-linear dierence equation of the second order: if we set
_
_
y
1,t
= x
t
y
2,t
= x
t+1
= y
1,t+1
.
then the Euler Lagrange equation can be written as a planar equation in y
t
= (y
1,t
, y
2,t
)
_
_
y
1,t+1
= y
2,t
y
2,t
F(y
2,t
, y
1,t
, t 1) +

y
2t
F(y
2,t+1
, y
2,t
, t) = 0
if we have a minimum problem we have just to consider the symmetric of the value
function
min
y
T1
t=0
F(y
t+1
, y
t
, t) = max
y
T1
t=0
F(y
t+1
, y
t
, t)
If F(x, y) is concave then the necessary conditions are also sucient.
Example 1: Let F(x
t+1
, x
t
) = (x
t+1
x
t
/22)
2
, the terminal time T = 4, and the state
constraints x
0
= x
4
= 1. Solve the calculus of variations problem.
Solution: If we apply the Euler-Lagrange equation we get a second order dierence
equation which is veried by the optimal solution
x
t
_
_
x
t
x
t1
2
2
_
2
_
+

x
t
_
_
x
t+1
x
t
2
2
_
2
_
= 0,
evaluated along {x
t
}
4
t=0
.
Then, we get
2x
t
+ x
t1
+ 4 + x
t+1
t
2
2 = 0
If we introduce a time-shift, we get the equivalent Euler equation
x
t+2
= 5/2x
t+1
x
t
2, t = 0, . . . , T 2
which together with the initial condition and the terminal conditions constitutes a mixed
initial-terminal value problem,
_
_
x
t+2
= 5/2x
t+1
x
t
2, t = 0, . . . , 2
x
0
= 1
x
4
= 1.
(6)
In order to solve problem (6) we follow the method:
1. First, solve the Euler equation, whose solution is a function of two unknown constants
(k
1
and k
2
next)
2. Second, we determine the two constants (k
1
, k
2
) by using the initial and terminal
conditions.
First step: solving the Euler equation Next, we apply two methods for solving the
Euler equation: (1) by direct methods, using equation (60) in the Appendix, or (2) solve it
generally by transforming it to a rst order dierence equation system.
Method 1: applying the solution for the second order dierence equation (60)
Applying the results we derived for the second order dierence equations we get:
x
t
= 4 +
_
1
3
2
t
+
4
3
_
1
2
_
t
_
(k
1
4) +
_
2
3
2
t
2
3
_
1
2
_
t
_
(k
2
4). (7)
Method 2: general solution for the second order dierence equation We follow
the method:
1. First, we transform the second order equation into a planar equation by using the
transformation y
1,t
= x
t
, y
2,t
= x
t+1
. The solution will be a known function of two
arbitrary constants, that is y
1,t
=
t
(k
1
, k
2
).
2. Second, we apply the transformation back the transformation x
t
= y
1,t
=
t
(k
1
, k
2
)
which is function of two constants (k
1
, k
2
)
The equivalent planar system in y
1,t
and y
2,t
is
_
_
y
1,t+1
= y
2,t
y
2,t+1
=
5
2
y
2,t
y
1,t
2
which is equivalent to a planar system of type y
t+1
= Ay
t
+B where
y
t
=
_
_
y
1,t
y
2,t
_
_
, A =
_
_
0 1
1 5/2
_
_
, B =
_
_
0
2
_
_
.
The solution of the planar system is y
t
= y +P
t
P
1
(k y) where y = (I A)
1
B that is
y =
_
_
4
4
_
_
.
and
=
_
_
2 0
0 1/2
_
_
, P =
_
_
1/2 2
1 1
_
_
, P
1
=
_
_
2/3 4/3
2/3 1/3
_
_
.
Then
_
_
y
1,t
y
2,t
_
_
=
_
_
4
4
_
_
+
_
_
1
2
2
t
2
_
1
2
_
t
2
t
_
1
2
_
t
_
_
_
_
2
3
(k
1
4) +
4
3
(k
2
4)
2
3
(k
1
4)
1
3
(k
2
4)
_
_
If we substitute in the equation for x
t
= y
1,t
and take the rst element we have, again, the
general solution of the Euler equation (7).
Second step: particular solution In order to determine the (particular) solution of the
CV problem we take the general solution of the Euler equation (7), and determine k
1
and
k
2
by solving the system x
t
|
t=0
= 1 and x
t
|
t=4
= 1:
4 + 1 (k
1
4) + 0 (k
2
4) = 1 (8)
4 +
_
1
3
2
4
+
4
3
_
1
2
_
4
_
(k
1
4) +
_
2
3
2
4
2
3
_
1
2
_
4
_
(k
2
4) = 1 (9)
Then we get k
1
= 1 and k
2
= 38/17. If we substitute in the solution for x
t
, we get
x
t
= 4
3
17
2
t
48
17
(1/2)
t
Therefore, the solution for the calculus of variations problem is the sequence
{x
}
4
t=0
= {1, 38/17, 44/17, 38/17, 1}.
Example 2: The cake eating problem Assume that there is a cake whose size at the
beginning of period t is denoted by W
t
and there is a muncher who wants to eat it until the
beginning of period T. The initial size of the cake is W
0
= and, o course, W
T
= 0 and the
eater takes bites of size C
t
at period t. The eater evaluates the utility of its bites through a
logarithmic utility function and has a psychological discount factor 0 < < 1. What is the
optimal eating strategy ?
Formally, the problem is to nd the optimal paths {C
} = {C
t
}
T1
t=0
and {W
} = {W
t
}
T
t=0
that solve the problem
max
{C}
T
t=0
t
ln(C
t
), subject to W
t+1
= W
t
C
t
, W
0
= , W
T
= 0. (10)
This problem can be transformed into the calculus of variations problem, because C
t
=
W
t
W
t+1
,
max
{W}
T
t=0
t
ln(W
t
W
t+1
), subject to W
0
= , W
T
= 0.
The Euler-Lagrange condition is:

t1
W
t1
W
t
+

t
W
t
W
t+1
= 0.
Then, the rst order conditions are:
_
_
W
t+2
= (1 + )W
t+1
W
t
, t = 0, . . . T 2
W
0
=
W
T
= 0
In the appendix we nd the solution of this linear scone order dierence equation (see
equation (56))
W
t
=
1
1
_
k
1
+ k
2
+ (k
1
k
2
)
t
_
, t = 0, 1 . . . , T (11)
Figure 1: Solution to the cake eating problem with T = 10,
0
= 1,
T
= 0 and = 1/1.03
which depends on two arbitrary constants, k
1
and k
2
. We can evaluate them by using the
initial and terminal conditions
_
_
W
0
=
1
1
(k
1
+ k
2
+ (k
1
k
2
)) =
W
T
= k
1
+ k
2
+ (k
1
k
2
)
T
= 0.
Solving this linear system for k
1
and k
2
, we get:
k
1
= , k
2
=

T
1
T

Therefore, the solution for the cake-eating problem {C
}, {W
} is generated by
W
t
=
_
T
1
T
_
, t = 0, 1, . . . T (12)
and, as C
t
= W
t
W
t+1
C
t
=
_
1
1
T
_
t
, t = 0, 1, . . . T 1. (13)
2.2 Free terminal state problem
Now let us consider the problem
max
x
T1
t=0
F(x
t+1
, x
t
, t)
subject to x
0
=
0
and x
T
free (14)
where
0
and T are given.
Proposition 2. (Necessary condition for optimality for the free end point problem)
Let {x
t
}
T
t=0
F(x
t
, x
t1
, t 1)
x
t
+
F(x
t+1
, x
t
, t)
x
t
= 0, t = 1, 2, . . . , T 1 (15)
and the initial and the transversality conditions
x
0
=
0
, t = 0
F(x
T
, x
T1
, T 1)
x
T
= 0, t = T. (16)
Proof. Again we assume that we know {x
t
}
T
t=0
and V ({x
}), and we use the same method

as in the proof for the simplest problem. However, instead of equation (5) the variation
introduced by the perturbation {
t
}
T
t=0
is
V (x) V (x
) =
T1
t=1
_
F(x
t
, x
t1
, t 1)
x
t
+
F(x
t+1
, x
t
, t)
x
t
_
t
+
F(x
T
, x
T1
, T 1)
x
T
T
because x
T
= x
T
+
T
and
T
= 0 because the terminal state is not given. Now Then V (x)
V (x
) = 0 if and only if the Euler and the transversality (16) conditions are veried.
Condition (16) is called the transversality condition. Its meaning is the following: if
the terminal state of the system is free, it would be optimal if there is no gain in changing
the solution trajectory as regards the horizon of the program. If
F(x
T
,x
T1
,T1)
x
T
> 0 then we
could improve the solution by increasing x
T
(remember that the utiity functional is additive
along time) and if
F(x
T
,x
T1
,T1)
x
T
< 0 we have an non-optimal terminal state by excess.
Example 1 (bis) Consider Example 1 and take the same objective function and initial
state but assume instead that x
4
is free. In this case we have the terminal condition associated
to the optimal terminal state,
2x
4
x
3
4 = 0.
If we substitute the values of x
4
and x
3
, from equation (7), we get the equivalent condi-
tion 32 + 8k
1
+ 16k
2
= 0. This condition together with the initial condition, equation
equation (8), allow us to determine the constants k
1
and k
2
as k
1
= 1 and k
2
= 5/2. If
we substitute in the general solution, equation (7), we get x
t
= 4 3(1/2)
t
. Therefore,
the solution for the problem is {1, 5/2, 13/4, 29/8, 61/16}, which is dierent from the path
{1, 38/17, 44/17, 38/17, 1} that we have determined for the xed terminal state problem.
However, in free endpoint problems we need sometimes an additional terminal condition
in order to have a meaningful solution. To convince oneself, consider the following problem.
Cake eating problem with free terminal size . Consider the previous cake eating
example where T is known but assume instead that W
T
is free. The rst order conditions
from proposition (18) are
_
_
W
t+2
= (1 + )W
t+1
W
t
, t = 0, 1, . . . , T 2
W
0
=
T1
W
T
W
T1
= 0.
If we substitute the solution of the Euler-Lagrange condition, equation (11), the transver-
sality condition becomes
T1
W
T
W
T1
=

T1
T1
1
k
1
k
2
=
1
k
1
k
2
which can only be zero if k
2
k
1
= . If we look at the transversality condition, the last
condition only holds if W
T
W
T1
= , which does not make sense.
The former problem is mispecied: the way we posed it it does not have a solution for
bounded values of the cake.
One way to solve this, and which is very important in applications to economics is to
introduce a terminal constraint.
2.3 Free terminal state problem with a terminal constraint
Consider the problem
max
{x}
T1
t=0
F(x
t+1
, x
t
, t)
subject to x
0
=
0
and x
T

T
(17)
where
0
,
T
and T are given.
Proposition 3. (Necessary condition for optimality for the free end point problem with
terminal constraints)
Let {x
t
}
T
t=0
F(x
t
, x
t1
, t 1)
x
t
+
F(x
t+1
, x
t
, t)
x
t
= 0, t = 1, 2, . . . , T 1 (18)
and the initial and the transversality condition
_
_
x
0
=
0
, t = 0
F(x
T
,x
T1
,T1)
x
T
(
T
x
T
) = 0, t = T.
Proof. Now we write V ({x}) as a Lagrangean
V ({x}) =
T1
t=0
F(x
t+1
, x
t
, t) + (
T
x
T
)
where is a Lagrange multiplier. Using again the variational method with
0
= 0 and
T
= 0 the dierent between the perturbed candidate solution and the solution becomes
V ({x}) V ({x
}) =
T1
t=1
_
F(x
t
, x
t1
, t 1)
x
t
+
F(x
t+1
, x
t
, t)
x
t
_
t
+
+
F(x
T
, x
T1
, T 1)
x
T
T
+ (
T
x
T

T
)
From the Kuhn-Tucker conditions, we have the conditions, regarding the terminal state,
F(x
T
, x
T1
, T 1)
x
T
= 0, (
T
x
T
) = 0.
The cake eating problem again Now, if we introduce the terminal condition W
T
0,
the rst order conditions are
_
_
W
t+2
= (1 + )W
t+1
W
t
, t = 0, 1, . . . , T 2
W
0
=
T1
W
T
W
T
W
T1
= 0.
If T is nite, the last condition only holds if W
T
= 0, which means that it is optimal to eat
all the cake in nite time. The solution is, thus formally, but not conceptually, the same as
in the xed endpoint case.
2.4 Innite horizon problems
The most common problems in macroeconomics is the discounted innite horizon problem.
We consider two problems, without or with terminal conditions.
No terminal condition
max
{x}
t=0
t
F(x
t+1
, x
t
) (19)
where, 0 < < 1, x
0
=
0
where
0
is given.
Proposition 4. (Necessary condition for optimality for the innite horizon problem)
Let {x
t
}
t=0
be a solution for the problem dened by equation (19). Then it veries the
Euler-Lagrange condition
F(x
t
, x
t1
)
x
t
+
F(x
t+1
, x
t
)
x
t
= 0, t = 0, 1, . . .
and
_
_
x
0
= x
0
,
lim
t
t1
F(x
t
,x
t1
)
xt
= 0,
Proof We can see this problem as a particular case of the free terminal state problem
when T = . Therefore the rst order conditions were already derived.
With terminal conditions If we assume that lim
t
x
t
= 0 then the transversality
condition becomes
lim
t
t
F(x
t
, x
t1
)
x
t
x
t
= 0.
Exercise: the discounted innite horizon cake eating problem The solution of the
Euler-Lagrange condition was already derived as
W
t
=
1
1
_
k
1
+ k
2
+ (k
1
k
2
)
t
_
, t = 0, 1 . . . ,
If we substitute in the transversality condition for the innite horizon problem without
terminal conditions, we get
lim
t
t1
ln(W
t1
W
t
)
W
t
= lim
t
t1
(W
t
W
t1
)
1
= lim
t
t1
t1
1
k
1
k
2
=
1
k
2
k
1
which again ill-specied because the last equation is only equal to zero if k
2
k
1
= .
If we consider the innite horizon problem with a terminal constraint lim
t
x
t
0 and
substitute, in the transversality condition for the innite horizon problem without terminal
conditions, we get
lim
t
t1
ln(W
t1
W
t
)
W
t
W
t
= lim
t
W
t
k
1
k
2
=
k
1
+ k
2
(1 )(k
2
k
1
)
because lim
t
t
= 0 as 0 < < 1. The transversality condition holds if and only if
k
2
= k
1
. If we substitute in the solution for W
t
, we get
W
t
=
k
1
(1 )
1

t
= k
1
t
, t = 0, 1 . . . , .
The solution verie the initial condition W
0
=
0
if and only if k
1
=
0
. Therefore the
solution for the innite horizon problem is {W
t
}
t=0
where
W
t
=
0
t
.
Figure 2: Solution for the cake eating problem with T = , = 1/1.03 and
0
= 1
3 Optimal Control and the Pontriyagins principle
The optimal control problem is a generalization of the calculus of variations problem. It
involves two variables, the control and the state variables and consists in maximizing a
functional over functions of the state and control variables subject to a dierence equation
over the state variable, which characterizes the system we want to control. Usually the initial
state is known and there could exist or not additional terminal conditions over the state.
The trajectory (or orbit) of the state variable, {x} {x
t
}
T
t=0
, characterizes the state of
a system, and the control variable path u {u
t
}
T
t=0
allows us to control its evolution.
3.1 The simplest problem
Let T be nite. The simplest optimal control problem consist in nding the optimal paths
({u
}, {x
}) such that the value functional is maximized by choosing an optimal control,

max
{u}
T1
t=0
f(x
t
, u
t
, t), (20)
subject to the constraints of the problem
_
_
x
t+1
= g(x
t
, u
t
, t) t = 0, 1, . . . , T 1
x
0
=
0
t = 0
x
T
=
T
t = T
(21)
where
0
,
T
and T are given.
We assume that certain conditions hold: (1) dierentiability of f; (2) concavity of g and
f; (3) regularity
2
Dene the Hamiltonian function
H
t
= H(
t
, x
t
, u
t
, t) = f(x
t
, u
t
, t) +
t
g(x
t
, u
t
, t)
2
That is, existence of sequences of x = {x
1
, x
2
, ..., x
T
} and of u = {u
1
, u
2
, ..., u
T
} satisfying x
t+1
=
g
x
(x
0
t
, u
0
t
)x
t
+ g(x
0
t
, u
t
) g(x
0
t
, u
0
t
).
where
t
is called the co-state variable and {} = {
t
}
T1
t=0
is the co-state variable path.
The maximized Hamiltonian
H
t
(
t
, x
t
) = max
u
H
t
(
t
, x
t
, u
t
)
is obtained by substituting in H
t
the optimal control, u
t
= u
(x
t
,
t
).
Proposition 5. (Maximum principle)
If {x
} and {u
} are solutions of the optimal control problem (20)-(21) and if the former
dierentiability and regularity conditions hold, then there is a sequence {} = {
t
}
T1
t=0
such
that the following conditions hold
H
t
u
t
= 0, t = 0, 1, . . . , T 1 (22)
t
=
H
t+1
x
t+1
, t = 0, . . . , T 1 (23)
x
t+1
= g(x
t
, u
t
, t) (24)
x
T
=
T
(25)
x
0
=
0
(26)
Proof. Assume that we know the solution ({u
}, {x
}) for the problem. Then the optimal

value of value functional is V
= V ({x
}) =
T1
t=0
f(x
t
, u
t
, t).
Consider the Lagrangean
L =
T1
t=0
f(x
t
, u
t
, t) +
t
(g(x
t
, u
t
, t) x
t+1
)
=
T1
t=0
H
t
(
t
, x
t
, u
t
, t)
t
x
t+1
where Hamiltonian function is
H
t
= H(
t
, x
t
, u
t
, t) f(x
t
, u
t
, t) +
t
(g(x
t
, u
t
, t). (27)
Dene
G
t
= G(x
t+1
, x
t
, u
t
,
t
, t) H(
t
, , x
t
, u
t
, t)
t
x
t+1
.
Then
L =
T1
t=0
G(x
t+1
, x
t
, u
t
,
t
, t)
If we introduce again a variation as regards the solution {u
, x
}
T
t=0
, x
t
= x
t
+
x
t
, u
t
= u
t
+
u
t
and form the variation in the value function and apply a rst order Taylor approximation,
as in the calculus of variations problem,
L V
=
T1
t=1
_
G
t1
x
t
+
G
t
x
t
_
x
t
+
T1
t=0
G
t
u
t
u
t
+
T1
t=0
G
t
t
.
Then, get the optimality conditions
G
t
u
t
= 0, t = 0, 1, . . . , T 1
G
t
t
= 0, t = 0, 1, . . . , T 1
G
t1
x
t
+
G
t
x
t
= 0, t = 1, . . . , T 1
where all the variables are evaluated at the optimal path.
Evaluating these expressions for the same time period t = 0, . . . , T 1, we get
G
t
u
t
=
H
t
u
t
=
f(x
t
, u
t
, t)
u
+
t
g(x
t
, u
t
, t)
u
= 0,
G
t
t
=
H
t
t
x
t+1
= g(x
t
, u
t
, t) x
t+1
= 0,
which is an admissibility condition
G
t
x
t+1
+
G
t+1
x
t+1
=
(H
t
t
x
t+1
)
x
t+1
+
H
t+1
x
t+1
=
t
+
f(x
t+1
, u
t+1
, t + 1)
x
+
t+1
g(x
t+1
, u
t+1
, t + 1)
x
= 0.
Then, setting the expressions to zero, we get, equivalently, equations (22)-(26)
This is a version of the Pontriyagins maximum principle. The rst order conditions
dene a mixed initial-terminal value problem involving a planar dierence equation.
If
2
H
t
/u
2
t
= 0 then we can use the inverse function theorem on the static optimality
condition
H
t
u
t
=
f(x
t
, u
t
, t)
u
t
+
t
g(x
t
, u
t
, t)
u
t
= 0
to get the optimal control as a function of the state and the co-state variables as
u
t
= h(x
t
,
t
, t)
if we substitute in equations (23) and (24) we get a non-linear planar ode in (, x), called
the canonical system,
_
t
=
H
t+1
x
t+1
(x
t+1
, h(x
t+1
,
t+1
, t + 1), t + 1),
t+1
, t + 1)
x
t+1
= g(x
t
, h(x
t
,
t
, t), t)
(28)
where
H
t+1
x
t+1
=
f(x
t+1
, h(x
t+1
,
t+1
, t + 1), t + 1)
x
t+1
+
t+1
g(x
t+1
, h(x
t+1
,
t+1
, t + 1), t + 1)
x
t+1
The rst order conditions, according to the Pontriyagin principle, are then constituted by
the canonical system (29) plus the initial and the terminal conditions (25) and (26).
Alternatively, if the relationship between u and is monotonic, we could solve condition
H
t
/u
t
= 0 for
t
to get
t
= q
t
(u
t
, x
t
, t) =
f(x
t
,u
t
,t)
ut
g(x
t
,u
t
,t)
ut
and we would get an equivalent (implicit or explicit) canonical system in (u, x)
_
_
q
t
(u
t
, x
t
, t) =
H
t+1
x
t+1
(x
t+1
, u
t+1
, q
t+1
(u
t+1
, x
t+1
, t + 1), t + 1)
x
t+1
= g(x
t
, u
t
, t)
(29)
which is an useful representation if we could isolate u
t+1
, which is the case in the next
example.
Exercise: cake eating Consider again problem (10) and solve it using the maximum
principle of Pontriyagin. The present value Hamiltonian is
H
t
=
t
ln(C
t
) +
t
(W
t
C
t
)
and from rst order conditions from the maximum principle
_
_
H
t
Ct
=
t
(C
t
)
1
t
= 0, t = 0, 1, . . . , T 1
t
=
H
t+1
W
t+1
=
t+1
, t = 0, . . . , T 1
W
t+1
= W
t
C
t
, t = 0, . . . , T 1
W
T
= 0
W
0
= .
From the rst two equations we get an equation over C, C
t+1
t
=
t+1
C
t
, which is sometimes
called the Euler equation. This equation together with the admissibility conditions, lead to
the canonical dynamic system
_
_
C
t+1
= C
t
W
t+1
= W
t
C
t
, t = 0, . . . , T 1
W
T
= 0
W
0
= .
There are two methods to solve this mixed initial-terminal value problem: recursively or
jointly.
First method: we can solve the problem recursively. First,we solve the Euler equation
to get
C
t
= k
1
t
.
Then the second equation becomes
W
t+1
= W
t
k
1
t
which has solution
W
t
= k
2
k
1
t1
s=0
s
= k
2
k
1
1
t
1
.
In order to determine the arbitrary constants, we consider again the initial and terminal
conditions W
0
= and W
T
= 0 and get
k
1
=
1
1
T
, k
2
=
and if we substitute in the expressions for C
t
and W
t
we get the same result as in the
calculus of variations problem, equations (13)-(12).
Second method: we can solve the canonical system as a planar dierence equation
system. The rst two equations have the form y
t+1
= Ay
t
where
A =
_
_
0
1 1
_
_
which has eigenvalues
1
= 1 and
2
= and the associated eigenvector matrix is
P =
_
_
0 1
1 1
_
_
.
The solution of the planar equation is of type y
t
= P
t
P
1
k
_
_
C
t
W
t
_
_
=
1
1
_
_
0 1
1 1
_
_
_
_
1 0
0
t
_
_
_
_
1 1
1 0
_
_
_
_
k
1
k
2
_
_
=
=
_
_
k
1
t
k
2
k
1
1
t
1
_
_
.
3.2 Free terminal state
Again, let T be nite. This is a slight modication of the simplest optimal control
problem which has the objective functional (20) subject to
_
_
x
t+1
= g(x
t
, u
t
, t) t = 0, 1, . . . , T 1
x
0
=
0
t = 0
(30)
where
0
is given.
The Hamiltonian is the same as in the former problem and the rst order necessary
conditions for optimality are:
If {x
}
T
t=0
and {u
}
T
t=0
are solutions of the optimal control problem (20)-(30) and if the
former assumptions on f and g hold, then there is a sequence {} = {
t
}
T1
t=0
such that for
t = 0, 1, ..., T 1
H
t
u
t
= 0, t = 0, 1, . . . , T 1 (31)
t
=
H
t+1
x
t+1
, t = 0, . . . , T 1 (32)
x
t+1
= g(x
t
, u
t
, t) (33)
x
0
=
0
(34)
T1
= 0 (35)
Proof. The proof is similar to the previous case, but now we have for t = T
G
T1
x
T
=
T1
= 0.
3.3 Free terminal state with terminal constraint
Again let T be nite and assume that the terminal value for the state variable is non-
negative. This is another slight modication of the simplest e simplest optimal control
problem which has the objective functional (20) subject to
_
_
x
t+1
= g(x
t
, u
t
, t) t = 0, 1, . . . , T 1
x
0
=
0
t = 0
x
T
0 t = T
(36)
where
0
is given.
The Hamiltonian is the same as in the former problem and the rst order necessary
conditions for optimality are
If {x
}
T
t=0
and {u
}
T
t=0
are solutions of the optimal control problem (20)-(36) and if the
former conditions hold, then there is a sequence = {
t
}
T1
t=0
such that for t = 0, 1, ..., T 1
satisfying equations (31)-(34) and
T1
x
T
= 0 (37)
The cake eating problem Using the previous result, the necessary conditions according
to the Pontryiagins maximum principle are
_
_
C
t
=
t
/
t
t
=
t+1
W
t+1
= W
t
C
t
W
0
=
0
T1
= 0
This is equivalent to the problem involving the canonical planar dierence equation system
_
_
C
t+1
= C
t
W
t+1
= W
t
C
t
W
0
=
0
T1
C
T1
= 0
whose general solution was already found. The terminal condition becomes
T1
C
T1
=

T1
T1
k
1
=
1
k
1
which can only be zero if k
1
= , which does not make sense.
If we solve instead the problem with the terminal condition W
T
0, then the transver-
sality condition is
T1
W
T
=
T1
W
T
C
T1
= 0
If we substitute the general solutions for C
t
and W
t
we get
T1
W
T
C
T1
=
1
1
_
k
1
+ (1 )k
2
k
1
+
k
1
k
1
T
_
which is equal to zero if and only if
k
2
= k
1
1
T
1
.
We still have one unknown k
1
. In order to determine it, we substitute in the expression for
W
t
W
t
= k
1
T
1
,
evaluate it at t = 0, and use the initial condition W
0
= and get
k
1
=
1
1
T
.
Therefore, the solution for the problem is the same as we got before, equations (13)-(12).
3.4 The discounted innite horizon problem
The discounted innite horizon optimal control problem consist on nding (u
, x
) such that
max
u
t=0
t
f(x
t
, u
t
), 0 < < 1 (38)
subject to
_
_
x
t+1
= g(x
t
, u
t
) t = 0, 1, . . .
x
0
=
0
t = 0
(39)
where
0
is given.
Observe that the functions f(.) and g(.) are now autonomous, in the sense that time does
not enter directly as an argument, but there is a discount factor
t
which weights the value
of f(.) along time.
The discounted Hamiltonian is
h
t
= h(x
t
,
t
, u
t
) f(u
t
, y
t
) +
t
g(y
t
, u
t
) (40)
where
t
is the discounted co-state variable.
It is obtained from the current value Hamiltonian as follows:
H
t
=
t
f(u
t
, x
t
) +
t
g(x
t
, u
t
)
=
t
(f(u
t
, y
t
) +
t
g(y
t
, u
t
))

t
h
t
where the co-state variable () relates with the actualized co-state variable () as
t
=
t
. The Hamiltonian h
t
is independent of time in discounted autonomous optimal control
problems. The maximized current value Hamiltonian is
h
t
= max
u
h
t
(x
t
,
t
, u
t
).
If {x
t=0
and {u
t=0
is a solution of the optimal control problem (38)-(39) and if the former
regularity and continuity conditions hold, then there is a sequence {} = {
t
}
t=0
such that
the optimal paths verify
h
t
u
t
= 0, t = 0, 1, . . . , (41)
t
=
h
t+1
x
t+1
, t = 0, . . . , (42)
x
t+1
= g(x
t
, u
t
, t) (43)
lim
t
t
= 0 (44)
x
0
=
0
(45)
Proof. Exercise.
Again, if we have the terminal condition
lim
t
x
t
0
the transversality condition is
lim
t
t
x
t
= 0 (46)
instead of (44).
The necessary rst-order conditions are again represented by the system of dierence
equations. If
2
h
t
/u
2
t
= 0 then we can use the inverse function theorem on the static
optimality condition
h
t
u
t
=
f(x
t
, u
t
, t)
u
t
+
t
g(x
t
, u
t
, t)
u
t
= 0
to get the optimal control as a function of the state and the co-state variables as
u
t
= h(x
t
,
t
)
if we substitute in equations (42) and (43) we get a non-linear autonomous planar dierence
equation in (, x) (or (u, x), if the relationship between u and is monotonic)
_
t
=
_
f(x
t+1
,h(x
t+1
,
t+1
))
x
t+1
+
t+1
g(x
t+1
,h(x
t+1
,
t+1
))
x
t+1
_
x
t+1
= g(x
t
, h(x
t
,
t
))
plus the initial and the transversality conditions (44) and (45) or (46).
Exercise: the cake eating problem with an innite horizon The discounted Hamil-
tonian is
h
t
= ln (C
t
) +
t
(W
t
C
t
)
and the f.o.c are
_
_
C
t
= 1/
t
t
=
t+1
W
t+1
= W
t
C
t
W
0
=
0
lim
t
t
W
t
= 0
This is equivalent to the planar dierence equation problem
_
_
C
t+1
= C
t
W
t+1
= W
t
C
t
W
0
=
0
lim
t
t Wt
Ct
= 0
If we substitute the solutions for C
t
and W
t
in the transversality condition, we get
lim
t
t
W
t
C
t
= lim
t
k
1
+ (1 )k
2
+ k
1
t
(1 )k
1
=
k
1
+ (1 )k
2
(1 )k
1
= 0
if and only if k
1
= (1 )k
2
. Using the same method we used before, we nally reach again
the optimal solution
C
t
= (1 )
t
, W
t
=
t
, t = 0, 1, . . . , .
Exercise: the consumption-savings problem with an innite horizon Assume that
a consumer has an initial stock of nancial wealth given by > 0 and gets a nancial return
if s/he has savings. The intratemporal budget constraint is
W
t+1
= (1 + r)W
t
C
t
, t = 0, 1, . . .
where r > 0 is the constant rate of return. Assume s/he has the intertemporal utility
functional
J(C) =
t=0
t
ln (C
t
), 0 < =
1
1 +
< 1, > 0
and that the non-Ponzi game condition holds: lim
t
W
t
0. What are the optimal
sequences for consumption and the stock of nancial wealth ?
We next solve the problem by using the Pontriyagins maximum principle. The discounted
Hamiltonian is
h
t
= ln (C
t
) +
t
((1 + r)W
t
C
t
)
where
t
is the discounted co-state variable. The f.o.c. are
_
_
C
t
= 1/
t
t
= (1 + r)
t+1
W
t+1
= (1 + r)W
t
C
t
W
0
=
0
lim
t
t
W
t
= 0
which is equivalent to
_
_
C
t+1
= C
t
W
t+1
= W
t
C
t
W
0
=
0
lim
t
t Wt
Ct
= 0
If we dene and use the rst two and the last equation
z
t

W
t
C
t
we get a boundary value problem
_
_
z
t+1
=
1
_
z
t
1
1+r
_
lim
t
t
z
t
= 0.
The dierence equation for z
t
has the general solution
3
z
t
=
_
k
1
(1 + r)(1 )
_
t
+
1
(1 + r)(1 )
.
We can determine the arbitrary constant k by using the transversality condition:
lim
t
t
z
t
= lim
t
t
_ _
k
1
(1 + r)(1 )
_
t
+
1
(1 + r)(1 )
_
= k
1
(1 + r)(1 )
+ lim
t
t
_
1
(1 + r)(1 )
_
=
= k
1
(1 + r)(1 )
= 0
which is equal to zero if and only if
k =
1
(1 + r)(1 )
.
3
The dierence equation is of type x
t+1
= ax
t
+ b, where a = 1 and has solution
x
t
=
_
k
b
1 a
_
a
t
+
b
1 a
where k is an arbitrary constant.
Then, z
t
= 1/ ((1 + r)(1 )) is a constant. Therefore, as C
t
= W
t
/z
t
the average and
marginal propensity to consume out of wealth is also constant, and
C
t
= (1 + r)(1 )W
t
.
If we substitute in the intratemporal budget constraint and use the initial condition
_
_
W
t+1
= (1 + r)W
t
C
t
W
0
=
we can determine explicitly the optimal stock of wealth for every instant
W
t
= ((1 + r))
t
=
_
1 + r
1 +
_
t
, t = 0, 1, . . . ,
and
C
t
= (1 + r)(1 )
_
1 + r
1 +
_
t
, t = 0, 1, . . . , .
We readily see that the solution depends crucially upon the relationship between the rate
of return on nancial assets, r and the rate of time preference :
1. if r > then lim
t
W
t
= : if the consumer is more patient than the market s/he
optimally tends to have an abounded level of wealth asymptotically;
2. if r = then lim
t
W
t
= : if the consumer is as patient as the market it is optimal
to keep the level of nancial wealth constant. Therefore: C
t
= rW
t
= r;
3. if r < then lim
t
W
t
= 0: if the consumer is less patient than the market s/he
optimally tends to end up with zero net wealth asymptotically.
The next gures illustrate the three cases
Figure 3: Phase diagram for the case in which > r
Figure 4: Phase diagram for the case in which = r
Figure 5: Phase diagram for the case in which < r
Observe that although s/he may have an innite level of wealth and consumption, asymp-
totically, the optimal value of the problem is bounded
J
t=0
t
ln (C
t
=
=
t=0
t
ln
_
(1 + r)(1 ) ((1 + r))
t
,
_
=
=
t=0
t
ln ((1 + r)(1 )) +
t=0
t
ln
_
((1 + r))
t
_
=
=
1
1
ln ((1 + r)(1 )) + ln ((1 + r))
t=0
t
t
=
=
1
1
ln ((1 + r)(1 )) +

(1 )
2
ln ((1 + r))
then
J
=
1
1
ln
_
_
(1 + r)(1 )
1
1/(1)
_
which is always bounded.
4 Optimal control and the dynamic programming prin-
ciple
Consider the discounted nite horizon optimal control problem which consists in nding
(u
, x
) such that
max
u
T
t=0
t
f(x
t
, u
t
), 0 < < 1 (47)
subject to
_
_
x
t+1
= g(x
t
, u
t
) t = 0, 1, . . . , T 1
x
0
=
0
t = 0
(48)
where
0
is given.
The principle of dynamic programming allows for an alternative method of solution.
According to the Principle of the dynamic programming (Bellman (1957)) an op-
timal trajectory has the following property: in the beginning of any period, take as given
values of the state variable and of the control variables, and choose the control variables
optimally for the rest of period. Apply this methods for every period.
4.1 The nite horizon problem
We start by the nite horizon problem, i.e. T nite.
Proposition 9. Consider problem (47)-(48) with T nite. Then given an optimal solution
the problem (x
, u
) satises the Hamilton-Jacobi-Bellman equation

V
Tt
(x
t
) = max
ut
{f(x
t
, u
t
) + V
Tt1
(x
t+1
)} , t = 0, . . . , T 1. (49)
Proof. Dene value function at time
V
T
(x
) =
T
t=
t
f(u
t
, x
t
) = max
{ut}
T
t=
T
t=
t
f(u
t
, x
t
)
Then, for time = 0 we have
V
T
(x
0
) = max
{ut}
T
t=0
T
t=0
t
f(u
t
, x
t
) =
= max
{ut}
T
t=0
_
f(x
0
, u
0
) + f(x
1
, u
1
) +
2
f(x
2
, u
2
) + . . .
_
=
= max
{ut}
T
t=0
_
f(x
0
, u
0
) +
T
t=1
t1
f(x
t
, u
t
)
_
=
= max
u
0
_
f(x
0
, u
0
) + max
{ut}
T
t=1
T
t=1
t1
f(x
t
, u
t
)
_
by the principle of dynamic programming. Then
V
T
(x
0
) = max
u
0
{f(x
0
, u
0
) + V
T1
(x
1
)}
We can apply the same idea for the value function for any time 0 t T to get the equation
(49) which holds for feasible solutions, i.e., verifying x
t+1
= g(x
t
, u
t
) and given x
0
.
Intuition: we transform the maximization of a functional into a recursive two-period
problem. We solve the control problem by solving the HJB equation. To do this we have to
nd {V
T
, . . . , V
0
}, through the recursion
V
t+1
(x) = max
u
{f(x, u) + V
t
(g(x, u))} (50)
Exercise: cake eating In order to solve the cake eating problem by using dynamic pro-
gramming we have to determine a particular version of the Hamilton-Jacobi-Bellman equa-
tion (49). In this case, we get
V
Tt
(W
t
) = max
Ct
{ ln(C
t
) + V
Tt1
(W
t+1
)} , t = 0, 1, . . . , T 1,
To solve it, we should take into account the restriction W
t+1
= W
t
C
t
and the initial and
terminal conditions.
We get the optimal policy function for consumption by deriving the right hand side for
C
t
and setting it to zero
C
t
{ ln(C
t
) + V
Tt1
(W
t+1
)} = 0
From this, we get the optimal policy function for consumption
C
t
=
_
V

Tt1
(W
t+1
)
_
1
= C
t
(W
t+1
).
Then the HJB equation becomes
V
Tt
(W
t
) = ln(C
t
(W
t+1
)) + V
Tt1
(W
t+1
), t = 0, 1, . . . , T 1 (51)
which is a partial dierence equation.
In order to solve it we make the conjecture that the solution is of the type
V
Tt
(W
t
) = A
Tt
+
_
1
Tt
1
_
ln(W
t
), t = 0, 1, . . . , T 1
where A
Tt
is arbitrary. We apply the method of the undetermined coecients in order to
determine A
Tt
.
With that trial function we have
C
t
=
_
V

Tt1
(W
t+1
)
_
1
=
_
1
(1
Tt1
)
_
W
t+1
, t = 0, 1, . . . , T 1
which implies. As the optimal cake size evolves according to W
t+1
= W
t
C
t
then
W
t+1
=
_

Tt
1
Tt
_
W
t
. (52)
which implies
C
t
=
_
1
1
Tt
_
W
t
, t = 0, 1, . . . , T 1.
This is the same optimal policy for consumption as the one we got when we solve the problem
by the calculus of variations technique. If we substitute back into the equation (51) we get
an equivalent HJB equation
A
Tt
+
_
1
Tt
1
_
ln W
t
=
= ln
_
1
1
Tt
_
+ ln W
t
+
_
A
Tt1
+
_
1
Tt1
1
__
ln
_

Tt
1
Tt
_
+ ln W
t
__
As the terms in ln W
t
cancel out, this indicates (partially) that our conjecture was right.
Then, the HJB equation reduces to the dierence equation on A
t
, the unknown term:
A
Tt
= A
Tt1
+ ln
_
1
1
Tt
_
+
_

Tt
1
_
ln
_

Tt
1
Tt
_
which can be written as a non-homogeneous dierence equation, after some algebra,
A
Tt
= A
Tt1
+ z
Tt
(53)
where
z
Tt
ln
_
_
_
1
1
Tt
_
1
Tt
1
_

Tt
1
_
Tt
1
_
_
In order to solve equation (53), we perform the change of coordinates = T t and oberve
that A
TT
= A
0
= 0 because the terminal value of the cake should be zero. Then, operating
by recursion, we have
A
= A
1
+ z
=
= (A
2
+ z
1
) + z
=
2
A
2
+ z
+ z
1
=
= . . .
=
A
0
+ z
+ z
1
+ . . . +
z
0
=
s=0
s
z
s
.
Then
A
Tt
=
Tt
s=0
s
ln
_
_
_
1
1
Tts
_
1
Tts
1
_

Tts
1
_
Tts
1
_
_
.
If we use terminal condition A
0
= 0, then the solution of the HJB equation is, nally,
V
Tt
(W
t
) = ln
_
_
Tt
s=0
_
1
1
Tts
_
Tt
1
_

Tts
1
_
s+1
Tt
1
_
_
+
+
_
1
Tt
1
_
ln(W
t
), t = 0, 1, . . . , T 1 (54)
We already determined the optimal policy for consumption (we really do not need to deter-
mine the term A
Tt
if we only need to determine the optimal consumption)
C
t
=
_
1
1
Tt
_
W
t
=
_
1
1
T
_
t
, t = 0, 1, . . . , T 1,
because, in equation (52) we get
W
t
=
_
1
Tt
1
T(t1)
_
W
t1
=
=
_
1
Tt
1
T(t1)
_
_
1
T(t1)
1
T(t2)
_
W
t2
=
2
_
1
Tt
1
T(t2)
_
W
t2
=
= . . .
=
t
_
1
Tt
1
T
_
W
0
and W
0
= .
4.2 The innite horizon problem
For the innite horizon discounted optimal control problem, the limit function V = lim
j
V
j
is independent of j so the Hamilton Jacobi Bellman equation becomes
V (x) = max
u
{f(x, u) + V [g(x, u)]} = max
u
H(x, u)
Properties of the value function: it usually hard to get the properties of V (.). In
general continuity is assured but not dierentiability (this is a subject for advanced courses
on DP, see Stokey and Lucas (1989)).
If some regularity conditions hold, we may determine the optimal control through the
optimality condition
H(x, u)
u
= 0
if H(.) is C
2
then we get the policy function
u
= h(x)
which gives an optimal rule for changing the optimal control, given the state of the economy.
If we can determine (or prove that there exists such a relationship) then we say that our
problem is recursive.
In this case the HJB equation becomes a non-linear functional equation
V (x) = f(x, h(x)) + V [g(x, h(x))].
Solving the HJB: means nding the value function V (x). Methods: analytical (in some
cases exact) and mostly numerical (value function iteration).
Exercise: the cake eating problem with innite horizon Now the HJB equation is
V (W) = max
C
_
ln (C) + V (

W)
_
,
where

W = W C. We say we solve the problem if we can nd the unknown function
V (W).
In order to do this, rst, we nd the policy function C
= C(W), from the optimality

condition
{ln (C) + V (W C)}
C
=
1
C
V

(W C) = 0.
Then
C
=
1
V

(W (C))
,
which, if V is dierentiable, yields C
= C(W)).
Then

W = W C
(W) =

W(W) and the HJB becomes a functional equation
V (W) = ln (C
(W)) + V [

W(W)].
Next, we try to solve the HJB equation by introducing a trial solution
V (W) = a + b ln(W)
where the coecients a and b are unknown, but we try to nd them by using the method
of the undetermined coecients.
First, observe that
C =
1
1 + b
W
W =
b
1 + b
W
Substituting in the HJB equation, we get
a + b ln (W) = ln (W) ln (1 + b) +
_
a + b ln
_
b
1 + b
_
+ b ln (W)
_
,
which is equivalent to
(b(1 ) 1) ln (W) = a( 1) ln (1 + b) + b ln
_
b
1 + b
_
.
We can eliminate the coecients of ln(W) if we set
b =
1
1
.
Then the HJB equation becomes
0 = a( 1) ln
_
1
1
_
+

1
ln ()
then
a = ln (1 ) +

1
ln ().
Then the value function is
V (W) =
1
1
ln (W), where
_
(1 )
1
_
1/(1)
.
and C
= (1 )W, that is
C
t
= (1 )W
t
,
which yields the optimal cake size dynamics as
W
t+1
= W
t
C
t
= W
t
which has the solution, again, W
t
=
t
.
5 Bibliographic references
(Ljungqvist and Sargent, 2004, ch. 3, 4) (de la Fuente, 2000, ch. 12, 13)
References
Richard Bellman. Dynamic Programming. Princeton University Press, 1957.
Angel de la Fuente. Mathematical Methods and Models for Economists. Cambridge University
Press, 2000.
Daniel Liberzon. Calculus of Variations and Optimal Control Theory: A Concise Introduc-
tion. Princeton UP, 2012.
Lars Ljungqvist and Thomas J. Sargent. Recursive Macroeconomic Theory. MIT Press,
Cambridge and London, second edition edition, 2004.
L. S. Pontryagin, V. G. Boltyanskii, R. V. Gamkrelidze, and E. F. Mishchenko. The Math-
ematical Theory of Optimal Processes. Interscience Publishers, 1962.
Frank P. Ramsey. A mathematical theory of saving. Economic Journal, 38(Dec):54359,
1928.
Nancy Stokey and Robert Lucas. Recursive Methods in Economic Dynamics. Harvard
University Press, 1989.
A Second order linear dierence equations
A.1 Autonomous problem
Consider the homogeneous linear second order dierence equation
x
t+2
= a
1
x
t+1
+ a
0
x
t
, (55)
where a
0
and a
1
are real constants and a
0
= 0.
The solution is
x
t
=
_
1
a
1
t
1
+
a
1
t
2
_
k
1
(
1
a
1
)(
2
a
1
)
a
0
(
1
2
)
_
t
1
t
2
_
k
2
(56)
where k
1
and k
2
are arbitrary constants and
1
=
a
1
2

_
_
a
1
2
_
2
+ a
0
_
1/2
(57)
2
=
a
1
2
+
_
_
a
1
2
_
2
+ a
0
_
1/2
(58)
Proof: We can transform equation 55 into an equivalent linear planar dierence equation
of the rst order, If we set y
1,t
x
t
and y
2,t
x
t+1
, and observe that y
1,t+1
= y
2,t
and
equation 55 can be written as y
2,t+1
= a
0
y
1,t
+ a
1
y
2,t
.
Setting
y
t

_
_
y
1,t
y
2,t
_
_
, A
_
_
0 1
a
0
a
1
_
_
we have, equivalently the autonomous rst order system
y
t+1
= Ay
t
,
which has the unique solution
y
t
= P
t
P
1
k
where P and are the eigenvector and Jordan form associated to A, k = (k
1
, k
2
)
is a
vector of arbitrary constants.
The eigenvalue matrix is
=
_
_
1
0
0
2
_
_
and, because a
0
= 0 implies that there are no zero eigenvalues,
P =
_
_
(
1
a
1
)/a
0
(
2
a
1
)/a
0
1 1
_
_
.
As x
t
= y
1,t
then we get equation (56).
A.2 Non-autonomous problem
Now consider the homogeneous linear second order dierence equation
x
t+2
= a
1
x
t+1
+ a
0
x
t
+ b (59)
where a
0
, a
1
and b are real constants and a
0
= 0 and 1 a
1
a
0
= 0.
The solution is
x
t
= x +
_
1
a
1
t
1
+
a
1
t
2
_
(k
1
x)
(
1
a
1
)(
2
a
1
)
a
0
(
1
2
)
_
t
1
t
2
_
(k
2
x) (60)
where
x =
b
1 a
0
a
1
is the steady state of equation (59).
Proof: If we dene z
t
x
t
x then we get an equivalent system y
t+1
y = A(y
t
y),
where y = ( x, x)
which has solution y

t
y = P
t
P
1
(k y).

Paulo Brito Ecomat Discreto

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Paulo Brito Ecomat Discreto

Uploaded by

Copyright:

Available Formats

Mathematical Economics

Deterministic dynamic optimization

, x) is continuous and dierentiable in (x

, x). The simplest problem

}) = 0, which holds if (4) is veried.

}), and we use the same method

}) such that the value functional is maximized by choosing an optimal control,

}) for the problem. Then the optimal

) satises the Hamilton-Jacobi-Bellman equation

= C(W), from the optimality

which has solution y

You might also like