ECO5011F QUANTITATIVE METHODS FOR ECONOMISTS

MODULE 2: DYNAMIC OPTIMISATION

CHAPTER 1: THE NATURE OF DYNAMIC OPTIMISATION

Lecture Notes: “Chiang, A. C. (1992). Elements of Dynamic Optimization, McGraw-Hill

Publishing”.

1.0 What is dynamic optimisation?

1.1 The salient features of dynamic optimisation problems
1.2 Variable endpoints and transversality conditions
1.3 The objective functional
1.4 Alternative approaches to dynamic optimisation

1.0 What is dynamic optimisation?

Optimisation is a predominant theme in economic analysis. In economic theory we talk

about firms maximising profits, firms minimising costs, consumers maximising utility, social
planners maximising social welfare – they are all trying to achieve the best in their
circumstances. This all constitutes optimisation of some kind. If present decisions do not
affect future events then one only need make the best decision for the present i.e. static
optimisation techniques are sufficient. In a static (timeless) problem, one seeks an optimal
number or finite set of numbers. For instance, a firm may seek the production level x* that
maximises the profit F(x) generated by producing and selling x units:
max x≥0 F ( x)
The answer to this problem is a number! How do you get this number? By using calculus.
The classical calculus methods are applicable only to static optimisation problems, where
the solution is usually a single optimal magnitude for every choice variable. Static
optimisation problems do not call for a schedule of optimal sequential action.

A dynamic optimisation problem poses the question of what is the optimal magnitude of a
choice variable in each period of time within the planning horizon, y*(t) i.e. an optimal time
path for every choice variable, detailing the best value of the variable today, tomorrow and
so forth, till the end of the planning period.

1.1 The salient features of dynamic optimisation problems

It is also possible to envisage the planning horizon as a sequence of stages in an economic

process, rather than time. In that case, dynamic optimisation can be viewed as a problem of
multistage decision making.

Multistage decision making

Suppose that a firm engages in transforming a certain substance from an initial state A
(wood pulp) into a terminal state Z (textbook on dynamic optimisation) through a five-stage
production process. In every stage, the firm faces the problem of choosing among several
1
possible alternative sub-processes, each entailing a specific cost. How should the firm select
the sequence of sub-processes through the five stages in order to minimise total cost?

Our problem is to choose a connected sequence of arcs going from left to right, starting at A
and terminating at Z, such that the sum of the values of the component arcs is minimized.
Such a sequence of arcs will constitute an optimal path. Discussion question: What
path/solution would you get with a myopic, one-stage-at-a-time optimisation procedure?
What would be the associated cost of production? What is the optimal path/solution? What
is the minimum cost of production? The major lesson from this exercise is that a method
that takes into account the entire planning horizon needs to be used. Instead of casting the
discussion in terms of a discrete number of stages i.e. where the state variable takes values
belonging to a small finite set {A, B, C, …, Z}, we could consider a case where the state
variable is continuous.

The continuous-variable version

We have drawn only five possible paths. Each possible path is now seen to travel through an
infinite number of stages in the interval [0,T]. We can visualise figure 1.2 to be an open
terrain. The stage variable represents the longitude and the state variable represents the
latitude. The task is to transport a load of cargo from A to Z at minimum cost by selecting an
appropriate travel path. The costs will also depend on the topography of the path.
Discussion question: Characterise the cost per unit of distance if the terrain is homogenous?
With homogenous terrain, what travel path do you need to choose in order to minimise
costs? Why?

2
For most problems discussed, the stage variable will represent time, in which case the
curves will depict time paths. For example, consider a firm with an initial capital stock equal
to A at time 0 and a predetermined target capital stock equal to Z at time T. Many
alternative investment plans over the time period [0.T] are capable of achieving the target
capital at time T. And each investment plan implies a specific capital path and entails a
specific potential profit for the firm. The problem of the firm is to identify the investment
plan – hence the capital path – that yields the maximum potential profit. As we showed in
the previous section, there is no guarantee that the path/solution that you get with a
myopic, one-stage-at-a-time optimisation procedure will be optimal. The optimal
path/solution would have to be obtained taking into account the entire planning horizon.
Discussion question: What political, social and economic problems can you think of which
require determining optimality by taking into account the entire planning horizon? What is
the common characteristic in these problems?

From the wood pulp and capital stock examples, it is clear that a simple type of dynamic
optimisation problem would contain the following basic ingredients:
1. A given initial point and a given terminal point
2. A set of admissible paths from the initial point to the terminal point
3. A set of path values serving as performance indices (cost, profit, etc) associated with
the various paths
4. A specified objective – either to maximise or to minimise the path value or
performance index by choosing the optimal path

The concept of a functional

The relationship between paths and path values represents a special sort of mapping – not a
mapping from real numbers to real numbers as in the usual function, but a mapping from
paths (curves) to real numbers (performance indices).

Think of the paths in question as time paths and denote them by y I (t ) , y II (t ) , y III (t ) , and
so on. Then the mapping is as shown in figure 1.3 where VI , VII , V III , etc represent the
associated path values.
3
The general notation for the mapping should therefore be V[ y (t ) ], which differs from the
composite-function symbol g[ f (x) ] which can be reduced to g(x). In V[ y (t ) ], the y (t )
component comes as an integral unit indicating time paths. So V should be understood to be
a function of “ y (t ) ” as such, and not as V[t]. This type of mapping has a distinct name:
functional. Many writers write the functional as V[ y ] thereby underscoring the fact that it is
the change in the position of the entire y path – the variation in the y path – that results in a
change in the path value V.

When the symbol y is used to indicate a certain value, it is suffixed, and it appears as, say,
y (0) for the initial state or y (T ) for the terminal state. The notation for a specific time
interval is y (0, T ) or y (0,τ ) . Otherwise we shall simply use the y (t ) symbol in path
connotation where the optimal time path is then denoted by y * (t ) ,

1.2 Variable endpoints and transversality conditions

Earlier, we simplified matters by assuming a given initial point and a given terminal point. A
given initial point may not be unduly restrictive, because in the usual problem, the
optimising plan must start from some specific initial position, say, the current position.

The terminal position, on the other hand, may be with no inherent need for it to be
predetermined.
• We may, for instance, face only a fixed terminal time, but have complete freedom to
choose the terminal state (e.g. capital stock).
• We may also be assigned a rigidly specified terminal state (e.g. inflation target) but
are free to select the terminal time.
4
In such cases, the terminal point becomes a part of the optimal choice.

Types of variable terminal points

The fixed-time-horizon/fixed-time/vertical-terminal-line problem: in this first type of

variable terminal point problem we may be given a fixed terminal time T, but free terminal
state (see fig 1.5a). The planner enjoys much greater freedom in the choice of the optimal
path and, as a consequence, will be able to achieve a better – or at least no worse – optimal
path value, V*, than if the terminal point is rigidly specified.

To give an example of this kind of problem, suppose that a monopolistic firm is seeking to
establish a (smooth) optimal price path over a given planning period, say, 12 months, for the
purpose of profit maximisation. The current price enters into the problem as the initial
state. If there is no legal price restriction in force, the terminal price will be completely up to
the firm to decide. So in this case, we are given a fixed terminal time T, which is 12 months,
5
but free terminal state, which is the price at time T. Discussion question: Does fig 1.5a depict
the possibility of negative prices? Why or why not? If at all you do, how would you redraw
fig 1.5a if negative prices are admissible? If an official positive price ceiling is expected to be
in force at the terminal time t=T, redraw fig 1.5a to depict the appropriate terminal line?

The fixed-endpoint/horizontal-terminal-line problem: in this second type of variable

terminal point problem the terminal state Z (quality) is stipulated, but the terminal time is
free. y = Z in fig 1.5b constitutes the set of admissible terminal points. Each of these may
be associated with a different terminal time, T1, T2 and T3. Again, there is greater freedom of
choice if the firm is producing a good with a particular quality characteristic at minimum
cost. This permits the rise of a lengthier, but less expensive production method that might
not be feasible under a rushed production schedule.

The terminal-curve/terminal surface problem: in this third type of variable terminal point
problem, neither the terminal time T nor the terminal state Z is individually preset, but the
two are tied together via a constraint equation of the form Z = φ (T ) . In fig 1.5c such an
equation associates a particular terminal time (say, T1) with a corresponding terminal state
(say, Z1). The planner actually has only one degree of freedom in the choice of the terminal
point.

To give an example of this kind of problem, consider a custom order by the chemistry
department for glass beakers having (1) an early date of completion, and (2) low
breakability. The customer accepts a tradeoff as the curve Z = φ (T ) between the two
considerations: breakability and early completion, where y denotes breakability. Discussion
question: In fig 1.5c, let y denote heat resistance in a product (tyre, balloon, fireman’s suit),
a quality which takes longer production time to improve. How would you redraw the
terminal curve to depict the tradeoff between early completion date and high heat
resistance?

Transversality condition

Let us compare the boundary conditions for the optimal path in the fixed-terminal-point
versus variable-terminal-point cases. In the former, the optimal path must satisfy the
boundary conditions:
y (0) = A and y (T ) = Z where A, T and Z are all given

In the latter case:

y (0) = A still applies but the terminal condition y (T ) = Z no longer points the optimal
path for us since T and/or Z are now variable.

All admissible paths, ending at Z1, Z2 or other terminal positions equally satisfy the condition
y (T ) = Z . What is needed, therefore, is a terminal condition that can conclusively
distinguish the optimal path from the other admissible paths – a transversality condition.
The transversality condition is a description of how the optimal path crosses/transverses the
terminal line/curve.

6
Variable initial point

Although we have assumed that only the terminal point can vary, this discussion can be
adapted to a variable initial point as well. If the initial point is variable, the characterisation
of the optimal path must also include another transversality condition in place of the
equation y (0) = A , to describe how the optimal path crosses the initial line/curve.
Discussion question: Sketch suitable diagrams similar to fig 1.5 for the case of a variable
initial point?

1.3 The objective functional

An optimal path is one that maximises or minimises that path value V[ y ]. Any y path total
value would naturally be a sum i.e. the path value is the sum of the values of its component
arcs. In the discrete-time setting, we are merely thinking about the sum of values on the
arcs [A, B ,C ,…, Z]. The continuous-time counterpart of such a sum is a definite integral,
T
∫0
(arc value)dt . But how do we express the arc value for the continuous case?

To identify an arc on a continuous-time path three pieces of information are needed: (i) the
starting stage (time), (ii) the starting state, and (iii) the direction in which the arc proceeds.
These three items are represented by, respectively: (i) t, (ii) y (t ) , and (iii) y ′(t ) ≡ dy dt .

If there exists some function, F, that assigns arc values to arcs, then the value of the said arc
can be written as F [t , y (t ), y ′(t )] and the path value functional – the sum of arc values – can
generally be written as the definite integral:

T
V [ y ] = ∫ F [t , y (t ), y ′(t )]dt
0

Remember that it is the variation in the y path ( y I (t ) versus y II (t ) ) that alters the
magnitude of V. Each different y path consists of a different set of arcs in the time interval
[0,T], which through the arc-value-assigning function F, takes a different set of arc values.
The definite integral sums those arc values on each y path into a path value. Discussion
question: What challenges does one encounter in assigning values on arcs in a continuous
variable setting that arise due to differences from the multistage framework presented
earlier? What is the arc value of the arc associated with a specific point of time t0 on a given
path y I (t ) ?

If there are two state variables, y and z, in the problem, the arc values on both the y and z
paths must be taken into account. The objective functional should then appear as:

T
V [ y, z ] = ∫ F [t , y (t ), z (t ), y ′(t ), z ′(t )]dt
0

7
A problem with the objective functional of the form V[y] or V[y,z] constitutes the standard
problem. For simplicity, we shall often suppress the time argument for state variables and
write F [t , y, y ′] or F [t , y, z , y ′, z ′] .

A microeconomic example

A profit maximising, long-range planning monopolistic firm has a dynamic demand function:
Qs = Q
Qd = D ( P, P ′) where P ′ = dP dt
Setting Qd = Qs to allow no building of stocks
Q = D( P, P ′)
The total revenue function is
R ≡ PQ = P * D ( P, P ′) = R ( P, P ′)
Assume the following total cost function
C = C (Q ) = C[ D( P, P ′)]
Total profit is
π ≡ R − C = R ( P, P ′) − C[ D( P, P ′)] = π ( P, P ′)
Summing over, say, a five year period
5
∫ π ( P, P′)dt
0
If either the revenue function or the cost function shifted over time, then the objective
functional would contain t as an argument i.e.
5
∫ π (t , P, P′)dt
0
Another possibility of generating a t in the objective functional would have been to bring it
via a discount factor, e − ρt .

To each price path in the time interval [0,5], there must correspond a particular five-year
profit figure, and the objective of the firm is to find the optimal price path P*[0,5] that
maximises the five year profit figure. Discussion question: Connect the discussion in the
above example to the relevant objects/paths/variables in fig 1.3? What is V in the above
example? And y? Tell how you would have described the firm’s problem in order to end up
with an argument t as a variable in the objective functional?

A macroeconomic example

Let the social welfare of an economy at any time be measured by the utility from
consumption, U = U (C ) . The production function is Q = Q ( K , L) . Define consumption:
C = Q ( K , L) − I = Q( K , L ) − K ′
The utility function can be rewritten as:
U (C ) = U [Q ( K , L) − K ′]
If the societal goal is to maximise the sum of utility over a period [0,T], then its objective
functional takes the form
T
∫ U [Q( K , L) − K ′]dt
0

8
This exemplifies the functional where there are two state variables K and L i.e. F [ L, K , K ′] .

Other forms of the objective functional

The optimisation criterion in a problem may rely exclusively on the position of the terminal
point attained. The objective functional appears as:

V [ y ] = G[T , y (T )] (this is called the problem of Mayer or the terminal control problem)
where the G function is based on what happens at the terminal time T only.

With two state variables, the objective functional is:

V [ y, z ] = G[T , y (T ), z (T )]

It may also happen that both the definite integral and the terminal point criterion enter
simultaneously in the objective functional. Then we have:

T
V [ y ] = ∫ F [t , y (t ), y ′(t )]dt + G[T , y (T )] (this is called the problem of Bolza)
0
where the G function may represent for instance, the scrap value of some capital
equipment.

So we have encountered the standard problem, the problem of Mayer and the problem of
Bolza. These three types of problems are all convertible into one another. In view of this
convertibility we shall deem it sufficient to couch our discussion in the form of an integral.
Discussion question: Suppose we are given a function D(t) which gives the desired level of
the state variable at every point of time in the interval [0,T]. All deviations from D(t),
positive or negative, are undesirable, because they inflict a negative payoff (cost, pain, or
disappointment). To formulate an appropriate dynamic minimisation problem, which of the
following objective functional would be acceptable? Why?
T
(a) ∫0
[ y (t ) − D (t )]dt
T
(b) ∫0
[ y (t ) − D (t )]2 dt
T
(c) ∫0
[ y (t ) − D (t )]3 dt
T
(d) ∫0
y (t ) − D (t ) dt

1.4 Alternative approaches to dynamic optimisation

To tackle the previously stated problem of dynamic optimisation there are three major
approaches: (i) the calculus of variations, (ii) optimal control theory, and (iii) dynamic
programming.

The calculus of variations

9
Dating back to the late 17th century, the calculus of variations is the classical approach to the
problem of dynamic optimisation. The calculus of variations is analogous to classical calculus
in its arena of applicability. This is usually used when functions are smooth and the optimum
is inside the feasible region i.e. for internal solutions. Some popular names associated with
this approach are Isaac Newton, John and James Bernoulli, and Leonhard Euler. These
problems can be represented by the following general formulation:

T
Maximise or Minimise V [ y ] = ∫ F [t , y (t ), y ′(t )]dt
0

subject to y (0) = A ( A given)

and y (T ) = Z (T , Z given)

Such a problem, with an integral functional in a single state variable, with completely initial
and terminal points, and with no constraints, is known as the fundamental problem (or
simplest problem) of calculus of variations.

The basic methodology underlying the calculus of variations closely parallels that of the
classical differential calculus. The main difference is that, instead of dealing with the
differential dx that changes the value of y = f ( x) , we will now deal with the “variation” of
an entire curve y (t ) that affects the value of the functional V[ y ].

The single most significant development in the calculus of variations is the Euler equation.

Although the advent of optimal control theory has caused the calculus of variations to be
overshadowed, a knowledge of the latter is indispensible for understanding many classic
economic papers written in the calculus of variations mould. The method is used even in
recent writings and a background in the calculus of variations facilitates a better and fuller
understanding of optimal control theory.

Optimal control theory

The continued study of variational problems has led to the development of the more
modern method of optimal control theory. The dynamic optimisation problem is viewed as
consisting of three (rather than two) types of variables: (i) the time variable t, (ii) the state
variable y (t ) , and (iii) a control variable u (t ) . The latter type of variable gives optimal
control theory its name and occupies the center of stage in this new approach.

The decision on a control path u (t ) will, once given an initial condition on y ,

unambiguously determine a state-variable path y (t ) as a by-product. For this reason an
optimal control problem must contain an equation that relates y to u :

dy
= f [t , y (t ), u (t )]
dt

10
Such an equation of motion (transition equation or state equation) shows how, at any
moment of time, given the value of the state variable, the planner’s choice of u will drive
the state variable y over time. Once we have found the optimal control-variable path
u * (t ) , the equation of motion would make it possible to construct the related optimal state-
variable path y * (t ) .

The optimal control problem corresponding to the calculus of variations problem given
earlier is as follows:

T
Maximise or Minimise V [u ] = ∫ F [t , y (t ), u (t )]dt
0

subject to y ′(t ) = f [t , y (t ), u (t )]
y (0) = A ( A given)
and y (T ) = Z (T , Z given)

Not only does the objective functional contain u as an argument, it has also been changed
from V[ y ] to V[ u ]. This reflects the fact that u is now the ultimate instrument of
optimisation.

The single most significant development in optimal control theory is known as the maximum
principle – commonly associated with the Russian mathematician Pontryagin. The
powerfulness of that principle lies in its ability to deal directly with certain constraints on
the control variable e.g. where the control variable u are confined to some closed, bounded
convex set µ . The set µ may be the closed interval [0,1], requiring 0 ≤ u (t ) ≤ 1 throughout
the planning period. If the marginal propensity to save is the control variable, for instance,
then such a constraint may very well be appropriate.

Dynamic programming

This is an optional approach for this course. If we still have time after covering the first two
approaches we will give a brief overview of this approach.

11