Optimal Control With Discounting

Elphego Torres

OPTIMIZACIÓN DINÁMICA

Profesor(a):
YAÑEZ JIMENEZ CARLOS ALBERTO

2018

Elphego Torres (IPN) Escuela Superior de Economı́a 2018 1 / 27

 Current Value Hamiltonian Function

We have noted that a major feature of the control problem is to maximise

the objective function F (x, u). However, for many economic problems
F (x, u) would represent such things as profits or net benefits. The
economist would not simply maximise such an income stream without first
discounting it to the present. Thus, if δ were the rate of discount then the
aim of the control would be to
Z T
máx J = e−δt F (x, u) dt (1.1)
{u(t)} 0

subject to various conditions which are unaffected by the discounting.

Thus, the typical continuous time maximization principle problem with
discounting is the control problem

Elphego Torres (IPN) Escuela Superior de Economı́a 2018 2 / 27

 Current Value Hamiltonian Function

Z T
máx J = e−δt F (x, u) dt (1.2)
{u(t)} 0
ẋ = g(x, u)
x(0) = x0
x(T ) = xT

Consider now discounting under a continuous time model. Consider the

control problem (1.2). The Lagrangian is
Z T n o
L= e−δt F (x, u) + λ[g(x, u) − ẋ] dt (1.3)
0

and the Hamiltonian is

H(x, u) = e−δt F (x, u) + λg(x, u) (1.4)

Elphego Torres (IPN) Escuela Superior de Economı́a 2018 3 / 27

 Current Value Hamiltonian Function

Define the current value Hamiltonian function, Hc , by

Hc (x, u) = F (x, u) + µg(x, u) (1.5)

then

Hc = Heδt or H = Hc e−δt
µ = λeδt or λ = µe−δt

Now reconsider our five optimality conditions. Since eδt is a constant for a
change in the control variable, then condition (i) is simply ∂Hc /∂u = 0.
The second condition is less straightforward. We have

∂H ∂Hc −δt
λ̇ = − =− e
∂x ∂x

Elphego Torres (IPN) Escuela Superior de Economı́a 2018 4 / 27

 Current Value Hamiltonian Function

From λ = µe−δt

λ̇ = µ̇e−δt − δµe−δt
Equating these we have

∂Hc −δt
− e = µ̇e−δt − δµe−δt
∂x
∂Hc
or µ̇ = − + δµ
∂x
Condition (iii) is

∂H ∂Hc −δt ∂Hc

ẋ = = e = = g(x, u)
∂λ ∂λ ∂µ
while condition (iv) becomes

λ(T ) = µ(T )e−δt = 0

Elphego Torres (IPN) Escuela Superior de Economı́a 2018 5 / 27

 Current Value Hamiltonian Function

and condition (v) remains 2

unchanged. To summarise, define

the current value Hamiltonian and ∂Hc
µ̇ = − + δµ 0≤t≤T
current value Lagrangian multiplier, ∂x
i.e. 3

∂Hc
δt ẋ = = g(x, u)
Hc (x, u) = H(x, u)e ∂µ
= F (x, u) + µg(x, u) 4

when λ = µe−δt .
Then the x(0) = x0
optimality conditions are:
5
1

∂Hc
=0 0≤t≤T µ(T )e−δt = 0 (or x(T ) = xT )
∂u

Elphego Torres (IPN) Escuela Superior de Economı́a 2018 6 / 27

 Current Value Hamiltonian Function

Current Value Formulation

These optimality conditions allow us to eliminate the control variable u
using condition (i) and to obtain two differential equations:one for the
state variable, x, and the other for the current value costate variable, µ.
Many control problems in economics literarture have the following
structure:

Z t1
máx F (t, x, u)e−δt dt
u∈U ⊆R t0

ẋ = g(t, x, u)
x(t0 ) = x0 (1.6)

(a) x(t1 ) = x1

(b) x(t1 ) ≥ x1

(c) x(t1 ) free


Elphego Torres (IPN) Escuela Superior de Economı́a 2018 7 / 27

 Current Value Hamiltonian Function

The new feature is the explicit appearance of the discount e−δt . For such
problems it often convenient to formulate the maximum principle in a
slightly different form. The ordinary Hamiltonian is
H = p0 F (t, x, u)e−δt + pg(t, x, u). Multiply it by eδt to obtain the current
value Hamiltonian Hc = Heδt = p0 F (t, x, u) + eδt pg(x, t, u). Introducing
µ = eδt p as the current value shadow price for the problem, one can
write Hc in the form (where we put p0 = µ0 )

Hc (t, x, u, µ) = µ0 F (t, x, u) + µg(t, x, u) (1.7)

Note that if µ = eδt p, then µ̇ = δeδt p + eδt ṗ and so ṗ = e−δt (µ̇ − δµ).
Also, Hc = Heδt implies that ∂Hc /∂x = eδt (∂H/∂x). So ṗ = −∂H/∂x
takes the form µ̇ − δµ = −∂Hc /∂c. In fact, one can prove the following:

Elphego Torres (IPN) Escuela Superior de Economı́a 2018 8 / 27

 Current Value Hamiltonian Function

Theorem (1)

The Maximum principle Current Value Formulation.

Suppose that the admissible pair (x∗ (t), u∗ (t)) solves problem (1.6) and let Hc
be the current value Hamiltonian (1.7). Then there exists a continuous function
µ(t) and a number µ0 , either 0 or 1, such for all t ∈ [t0 , t1 ] we have (µ0 , µ(t))
6= (0, 0) and:

u = u∗ (t) maximizes Hc (t, x∗ (t), u, µ(t)) for u ∈ U

∂Hc (t, x∗ (t), u∗ (t), µ(t))
µ̇(t) − δµ(t) = −
∂x
Finally, the transversality conditions are:
µ(t1 ) no conditions
µ(t1 ) ≥ 0 (µ(t1 ) = 0 if x∗ (t1 ) > x1 )
µ(t1 ) = 0

Elphego Torres (IPN) Escuela Superior de Economı́a 2018 9 / 27

 Current Value Hamiltonian Function

The Mangasarian and Arrow sufficiency, have immediate extensions to

problem (1.6). The conditions in Theorem (1) are sufficient for optimality
if µ0 = 1 and

Hc (t, x, u, µ(t)) is concave in (x, u) (Mangasarian)

or (more generally)

H
b c (t, x, µ(t)) = máx Hc (t, x, u, µ(t)) is concave in x (Arrow)
u∈U

RONALD SHONE. Economic Dynamics Phase Diagrams and Their

Economic Application, Second Edition, Cambridge University Press, New
York 2002.
Knut Sydsæter, Peter Hammond, Atle Seierstad and Arne
Strøm. Further Mathematics for Economic Analysis. Pearson Education
Limited 2005.
Elphego Torres (IPN) Escuela Superior de Economı́a 2018 10 / 27
 Exercises

Z 10
x − u2 e−t dt


1 opt
0
ẋ = x + u
(2.1)
1
x(0) =
2
x(10) = 0

The current value Hamiltonian is

Hc = x − u2 + µ(x + u)

with optimality conditions

∂Hc
1
∂u = −2u + µ = 0
2 µ̇ = −1
3 ẋ = x + u
Elphego Torres (IPN) Escuela Superior de Economı́a 2018 11 / 27
 Exercises

From (1) we have u = µ/2, which when substituting (2.4) in (2.2)

substituted into (3) gives ẋ = x + µ/2.
Thus we have two differential equations 10 − t
ẋ = x + (2.5)
2
µ solving (2.5)
ẋ = x + (2.2)
2
t 9
µ̇ = −1 (2.3) x(t) = − + k2 et (2.6)
2 2
solve (2.3) applying condition in (2.6)

µ(t) = −t + k1 t 9
x∗ (t) = − + 5et
2 2
transversality condition:
and
µ∗ (t) = 10 − t (2.4) u∗ (t) = 5 − 0.5t

Elphego Torres (IPN) Escuela Superior de Economı́a 2018 12 / 27

 Exercises

From (1) we have u = µ, which when

Z 2 substituted into (3) gives ẋ = 3x + 2µ.
x2 − u2 e−2t dt


2 opt Thus we have two differential equations
0
ẋ = 3x + 2u (2.7)
x(0) = 3
ẋ = 3x + 2µ (2.8)
µ(2) = 0 µ̇ = −2x − µ (2.9)
The current value Hamiltonian is
Hc = x2 − u2 + µ(3x + 2u) from (2.8): µ = ẋ/2 − 3x/2, we make
with optimality conditions
∂Hc
1
∂u = −2u + 2µ = 0  
d ẋ 3x
2 µ̇ = −2x − µ µ= −
dt 2 2
3 ẋ = 3x + 2u ẍ 3ẋ
µ̇ = − (2.10)
2 2

Elphego Torres (IPN) Escuela Superior de Economı́a 2018 13 / 27

 Exercises

substituting (2.10) in (2.9) The roots are: repeated eigenvalues,

r1 = 1 = r2 = 1, so the general
ẍ 3ẋ ẋ 3x solution
− = −2x − −
2 2 2 2
simplifying: x(t) = k1 tet + k2 et (2.12)
ẍ − 2ẋ + x = 0 (2.11) from (2.12)
The equation (2.11) is a homogeneous
differential equation of second order. ẋ(t) = k1 et (t + 1) + k2 et .
We can solve the characteristic We know that
equation
ẋ 3x ẋ(t) 3x(t)
r2 − 2r + 1 = 0 µ= − ∴ µ(t) = −
2 2 2 2

so

k1 et (t + 1) + k2 et 3 k1 tet + k2 et


µ(t) = −
2 2
Elphego Torres (IPN) Escuela Superior de Economı́a 2018 14 / 27
 Exercises

simplifying then, the general solution

1
µ(t) = −k1 tet + k1 et − k2 et (2.13)
2
applying the conditions: x(0) = 3, µ(2) = 0 in (2.12) and (2.13), so we
have

k1 = −2 k2 = 3
Finally

x∗ (t) = 3et − 2tet

µ∗ (t) = 2tet − 4et

Elphego Torres (IPN) Escuela Superior de Economı́a 2018 15 / 27

 Exercises

5
u2 x2
Z 
3 opt − + xu − e−3t dt
0 2 2
ẋ = u (2.14)
x(0) = 1
µ(5) = 0

The current value Hamiltonian is From (1) we have u = x + µ, which

u 2
x 2 when substituted into (2) and (3),
Hc = − + xu − + µu gives ẋ = x + µ and µ̇ = 2µ. Thus we
2 2
have two differential equations
with optimality conditions
∂Hc
1
∂u = −u + x + µ = 0 ẋ = x + µ (2.15)
2 µ̇ = −u + x + 3µ µ̇ = 2µ (2.16)
3 ẋ = u

Elphego Torres (IPN) Escuela Superior de Economı́a 2018 16 / 27

 Exercises

from (2.16), we have therefore

µ(t) = k1 e2t x(t) = k2 et with x(0) = 1

transversality condition
x∗ (t) = et
∗
µ (t) = 0
then from (2.15) for u

ẋ − x = 0 u∗ (t) = et

Elphego Torres (IPN) Escuela Superior de Economı́a 2018 17 / 27

 Exercises

Z2 " 2 #
1
4 opt x− + u2 e−3t dt
2
0
(2.17)
ẋ = 3x − 2u
x(0) = 2
µ(2) = 0
The current value Hamiltonian is
 2
1
Hc = x − + u2 + µ(3x − 2u)
2
with optimality conditions From (1) we have u = µ, which when
substituted into (3) gives ẋ = 3x + 2µ.
∂Hc Thus we have two differential equations
1 = 2u − 2µ = 0
∂u
2 µ̇ = 1 − 2x
ẋ = 3x − 2µ (2.18)
3 ẋ = 3x − 2u µ̇ = 1 − 2x (2.19)

Elphego Torres (IPN) Escuela Superior de Economı́a 2018 18 / 27

 Exercises

0
from (2.18): µ = 3x x
2 − 2 : factoring:
3x x0
 
d
µ= − (r − 1)(r + 4) = 0
dt 2 2

3x0 x00 .
µ0 = − (2.20) The roots r1 = −1 and r2 = 4, are real
2 2
substituting (2.20) in (2.19) and distinct, thus our particular
solution is −2/ − 4 = 1/2, so the
3x0 x00 general solution
− = 1 − 2x
2 2
normalizing
1
x(t) = k1 e−t + k2 e4t + (2.22)
2
x00 − 3x0 − 4x = −2 (2.21)
We can solve the characteristic 3x x0
we know that µ = 2 − 2 ∴
equation
3x(t) x0 (t)
r2 − 3r − 4 = 0 µ= −
2 2

Elphego Torres (IPN) Escuela Superior de Economı́a 2018 19 / 27

 Exercises

that is to say
 
3 1 1
k1 e−t + k2 e4t + −k1 e−t + 4k2 e4t


µ(t) = −
2 2 2
simplifying

k2 e4t 3
µ(t) = 2k1 e−t − +
2 4
applying the conditions, we have

3(e8 − 1)e2 3(e2 + 4)

k1 = k2 =
2(e10 + 4) 2(e10 + 4)
finally

" #
e2 + 4 e4t + e10 − e2 e−t



∗ 3 1
x (t) = +
2 e10 + 4 3
" #
3 −e2 − 4 e4t + 4 e10 − e2 e−t



∗ ∗
u (t) = µ (t) = +1
4 e10 + 4

Elphego Torres (IPN) Escuela Superior de Economı́a 2018 20 / 27

 Exercises

Solving (2.17) through matricial methods.

Matricial form (2.18) and (2.19)
      
ẋ 3 −2 x 0
= +
µ̇ −2 0 µ 1
Define:


3 −2
A= .
−2 0
The characteristic equation of A is

 
A − δI  = 3 − δ −2
 
−2 −δ
= δ(δ − 3) − 4
= δ 2 − 3δ − 4 = 0
= (δ + 1)(δ − 4) = 0,

Elphego Torres (IPN) Escuela Superior de Economı́a 2018 21 / 27

 Exercises

the two eigenvalues of A are

δ1 = 4 δ2 = −1.
The eigenvector for δ1 :

    
−1 −2 a 0
A − δ1 v1 = 0 = = ,
−2 −4 b 0

then:
   
−1 −2 0 Gauss/Jordan 1 2 0
−−−−−−−−−→ .
−2 −4 0 0 0 0
The eigenvector of δ1 is:
 
−2
v1 = .
1

Elphego Torres (IPN) Escuela Superior de Economı́a 2018 22 / 27

 Exercises

The eigenvector for δ2 :

    
4 −2 c 0
A − δ 2 v2 = 0 = = ,
−2 1 d 0

then:

− 12
   
4 −2 0 Gauss/Jordan 1 0
−−−−−−−−−→ .
−2 1 0 0 0 0
The eigenvector of δ2 is:
 
1
v2 = .
2
The homogeneous solution is:

     
x(t) −2 4t 1 −t
= k1 e + k2 e .
µ(t) h 1 2

Elphego Torres (IPN) Escuela Superior de Economı́a 2018 23 / 27

 Exercises

The non-homogeneous solution is:

   
ẋ 0
= ,
µ̇ nh 0
then:

3x − 2µ = 0 (2.23)
− 2x + 1 = 0 (2.24)

from (2.24):
1
x= ,
2
substituting the value of x in (2.23):
3
µ= .
4

Elphego Torres (IPN) Escuela Superior de Economı́a 2018 24 / 27

 Exercises

The general solution is:

" # " # " # " #

1
x(t) −2 4t 1 −t
= k1 e + k2 e + 23 ,
µ(t) 1 2 4

applying the conditions.

" # "
# " # "
#" # " #

x∗ (t) −3 e2 + 4 −2 4t 3 e8 − 1 e2 1 −t 1
2
= e + e + 3 .
µ∗ (t) 4 (e10 + 4) 1 2 (e10 + 4) 2 4

Finally

" #
e2 + 4 e4t + e10 − e2 e−t



∗ 3 1
x (t) = +
2 e10 + 4 3
" #
3 −e2 − 4 e4t + 4 e10 − e2 e−t



∗
µ (t) = +1 .
4 e10 + 4

Elphego Torres (IPN) Escuela Superior de Economı́a 2018 25 / 27

 Exercises

Z20
4K − u2 e−0.25t dt


5 máx
u≥0
0
(2.25)
K̇ = −0.25K + u
K(0) = K0
K(20) is free

Economic interpretation: K(t) is the value of machine, u(t) is the repair effort,
4K − u2 is the instantaneous net profit at time t, and e−0.25t is the discount
factor
Solution: The current value Hamiltonian is: Hc = 4K − u2 + µ(−0.25K + u)
(with µ0 = 1), and so ∂H ∂Hc
∂u = −2u + µ and ∂K = 4
c
− 0.25µ. Assuming that
∂Hc∗
u (t) > 0 (we thy assumption in the following), ∂u = 0, so u∗ (t) = 0.5µ(t).
∗

The adjoint function µ satisfie

∂Hc∗
µ̇ − 0.25µ = − = −4 + 0.25µ, µ(20) = 0
∂K

Elphego Torres (IPN) Escuela Superior de Economı́a 2018 26 / 27

 Exercises

It follows that

µ(t) = 8 1 − e0.5t−10 u∗ (t) = 0.5µ = 4 1 − e0.5t−10

and
Note that u∗ (t) > 0 in [0, 20). The time path of K ∗
(t) is found from
K̇ ∗ = −0.25K ∗ + u∗ = −0.25K ∗ + 4 1 − e0.5t−10 . Solving this linear
differential equation with k ∗ (0) = K0 , we get
 
∗ 16 −10 −0.25t 16
K (t) = K0 − 16 + e e + 16 − e0.5t−10
3 3
2


Here Hc = 4K − u + µ(−0.25K + u) is concave in (K, u), so we have found
the complete solution.

Elphego Torres (IPN) Escuela Superior de Economı́a 2018 27 / 27