J Glob Optim (2012) 52:305322

DOI 10.1007/s10898-011-9667-4
An adaptive least-squares collocation radial basis
function method for the HJB equation
H. Alwardi S. Wang L. S. Jennings S. Richardson
Received: 20 January 2011 / Accepted: 25 January 2011 / Published online: 6 February 2011
Springer Science+Business Media, LLC. 2011
Abstract We present a novel numerical method for the HamiltonJacobiBellman equa-
tion governing a class of optimal feedback control problems. The spatial discretization is
based on a least-squares collocation Radial Basis Function method and the time discretization
is the backward Euler nite difference. Astability analysis is performed for the discretization
method. An adaptive algorithmis proposed so that at each time step, the approximate solution
can be constructed recursively and optimally. Numerical results are presented to demonstrate
the efciency and accuracy of the method.
Keywords HJB equation Optimal feedback control Radial basis functions
Adaptive method
H. Alwardi
Department of Mathematics, Nizwa College of Applied Sciences, PO Box 699, Nizwa 611,
Sultanate of Oman
e-mail: alwardy.niz@cas.edu.om
S. Wang (B) L. S. Jennings
School of Mathematics and Statistics, The University of Western Australia, 35 Stirling Hwy, Crawley,
WA 6009, Australia
e-mail: swang@maths.uwa.edu.au
L. S. Jennings
e-mail: les@maths.uwa.edu.au
S. Richardson
School of Engineering, Edith Cowan University, 270 Joondalup Drive, Joondalup, WA 6027, Australia
e-mail: s.richardson@ecu.edu.au
1 3
306 J Glob Optim (2012) 52:305322
1 Introduction
Consider the following optimal control problem arising from many real-world problems:
min
uU
J(y, s, u) =
T
_
s
L (t, x(t ), u(t )) dt + h (x(T)), (1)
subject to
_
x(t ) = f (t, x(t ), u(t )) , a.e. t [s, T],
x(s) = y,
(2)
where x = (x
1
, . . . , x
n
)
T
and u = (u
1
, . . . , u
m
)
T
are respectively state and control vari-
ables with positive integers n and m, f = ( f
1
, . . . , f
n
)
T
is a given vector valued function,
(s, y) [0, T] R
n
, and U R
m
is the set of admissible controls. When the initial time
s and the initial state y is xed, the above problem can be solved easily as an open-loop
optimal control problem. However, this solution is neither robust nor stable with respect to
perturbations in the state x. This is because the solution to the open-loop problem provides
an optimal control only along the optimal trajectory x starting from the initial state y. There-
fore, a feedback optimal control solution which is dened in a region containing the optimal
trajectory is much preferred in practice. This feedback solution gives a global optimal con-
trol dened over a time-space region so that if the state is away from the open-loop optimal
trajectory due to disturbance, a corresponding optimal control can be found for the system.
To achieve this, we assume that s and y in (1)(2) are not xed. In this case, it is well known
that, via the dynamic programming approach, (1) and (2) can be formulated as the following
HamiltonJacobiBellman (HJB) nal value problem:

v
t
+ sup
u U
[
x
v f (t, x, u) L(t, x, u)] = 0, (t, x) [0, T) R
n
(3)
v(T, x) = h(x), (4)
where v is the vale function and
x
denotes the gradient operator with respect to x. The
problem (3)(4) is a nal value problem in two unknowns, the value function v and the
control u. This problem has been discussed in the open literature for many years. For more
details we refer the reader to the monograph [1]. There are some numerical methods in the
open literature for the solution of (3)(4) such as those in [1, 6] and in our previous works
[1315, 20, 26]. Most of these methods are mesh-basedandthus suffer severelyfromthe curse
of dimensionality. On the other hand, HJB equations of practical importance are normally
high dimensional which may not be solved efciently by the existing methods. Therefore,
efcient and accurate methods are in urgent need for solving high-dimensional problems.
Recently, Huang et al. [15] proposed a mesh-less numerical method for the HJB equa-
tion, based on a collocation Radial Basis Function scheme in space and an explicit time
stepping scheme. The numerical results in [15] showed that the method is superior to an
existing method. In fact, to achieve a comparable accuracy as that in [25], it only requires
about one quarter of the number of basis functions used by the method in [25].
RBFs have been used very successively in function interpolations in high dimensions (cf.,
for example, [4]) as well as numerical solution of PDEs and numerical optimization (cf.,
for example, [8, 10, 12, 16, 27]). One of the merits of a RBF method is that, unlike a nite
difference or element method, it does not need any mesh, though a set of points are needed
to dene the centers of the RBFs. Therefore, a RBF can be dened anywhere in a given
domain, independently of other RBFs. This exibility allows us to construct a RBF approxi-
mation to a given function by recursively (or adaptively) adding RBFs so as to minimize the
1 3
J Glob Optim (2012) 52:305322 307
approximation error at each step. Also, RBFs are innitely smooth and so they can easily be
used to approximate a function and its derivatives. Furthermore, existing numerical results
such as those in [15] show that RBF methods have convergence rates higher than those of
piecewise linear nite elements and central nite difference method. Because of the above
merits, RBFs are ideal for solving high-dimensional PDEs, though they may not be able to
be completely free from the curse of dimensionality.
In this paper, we present a new numerical method for (3)(4). In this method, the back-
ward Eulers implicit nite difference method is used for the time discretization and RBFs
are used for approximating v in space. The main motivation of this work is to use as few
RBFs as necessary selected according to the criteria of the least mean squared residual over a
given set of data points to achieve a desired accuracy. To achieve this, we propose an adaptive
technique to recursively increase the number of radial basis functions so as to minimize the
least-squares error at each step. A formula for selecting the shape parameter c of a RBF
based proportionally on the average distance between centers of RBFs is proposed. We also
prove that the time stepping method is unconditionally stable. The numerical results will be
presented to show the efciency and accuracy of the method.
2 Discretization method
RBFs have been praised for their exibility and ease of implementation in the approximation
of multivariate functions [4], and they are becoming a viable choice as a method for the
numerical solution of conventional PDEs [8, 16]. In this work, we use the following inverse
multiquadratics:

j
(x) =
1
_
||x y
j
||
2
2
+c
2
j
, (5)
where
2
denotes the Euclidean norm on R
n
, y
j
is a given point dening the center of the
RBF and c
j
> 0 is the shape parameter.
This nal/terminal value problem (3)(4) is dened on R
n
. However, in practice, we are
only interested in solutions in a nite region in x as follows:

v
t
+ sup
u U
[
x
v f (t, x, u) L(t, x, u)] = 0, (t, x) [0, T) , (6)
v(T, x) = h(x), , (7)
where R
n
denotes a bounded region. We assume that f (t, x, u) and L(t, x, u) are
continuously differentiable with respect to u.
In what follows we shall present a novel method for (6)(7) based on an adaptive least-
squares RBF technique and a time stepping discretization. In this approach, it is not necessary
to dene articial boundary conditions as in [13, 14, 20, 26], though an extension of the solu-
tion region is still needed to have a proper transit layer to maintain the quality of solutions
within (cf. [21]).
2.1 Spatial discretization
We now consider the spatial discretization of (6) on the region

=

n
h=1
(a
h
, b
h
) contain-
ing the region of interest, , as a proper subset, where a
h
, b
h
R are two real constants
satisfying a
h
< b
h
for h = 1, 2, . . . , n. Let
1 3
308 J Glob Optim (2012) 52:305322
v(x, t ) =
N

j =1

j
(t )
j
(x) (8)
with
j
(x) given in (5). Replacing v with v, (6) can be written as

j =1

j
(x)
d
j
(t )
dt

N

j =1

j
(t )
j
(x) f (t, x, u

(x, t )) L(t, x, u

(x, t )) = 0, (9)
u

(x, t ) = arg sup

uU
_
_

j =1

j
(t )
j
(x) f (t, x, u) L(t, x, u)
_
_
, (10)
Note that at time t , the above system contains N unknowns {
j
}
N
j =1
. To determine them, we
choose a set of M distinct points x
i


, i = 1, 2, . . . , M and require that (9) and (10) hold
at these points, yielding the following semi-discrete system:

j =1

j
(x
i
)
d
j
(t )
dt

N

j =1

j
(t )
j
(x
i
) f (t, x
i
, u
i
(t )) L(t, x
i
, u
i
(t )) = 0, (11)
u
i
(t ) = arg sup
uU
_
_

j =1

j
(t )
j
(x
i
) f (t, x
i
, u) L(t, x
i
, u)
_
_
, (12)
for i = 1, 2, . . . , M. This is a set of collocation equations for the unknowns {
j
}
N
j =1
. When
M < N, this system is under-determined and the solutions to (11)(12) may not be unique.
In the case that M = N, it has been shown in [15] that (11)(12) is uniquely solvable when
the RBFs are properly chosen. However, for both of these situations, the number of sampling
points is boundedabove bythe number of RBFs. Thus, the approximationis exact onlyat these
M( N) collocation points which is not satisfactory. This is because the number of RBFs,
N, is normally small to moderate. A more appropriate way to determine the coefcients of v
is to choose an M such that M > N. In this case, (11) becomes an over-determined system
for {
j
}
N
j =1
and thus it is satised only in the least-squares sense. This denes a least-squares
collocation method which has been used for conventional partial differential equations
(cf., for example, [19] and the references therein).
Note that (11) is a system of ordinary differential equations in the unknowns {
j
(t )}
N
j =1
and thus a time discretization is required as given in the next subsection.
2.2 Time discretization
We nowconsider the time discretization of (11) and (12). Let T = t
0
> t
1
> > t
K
= 0 be
a mesh with t
k
= t
k
t
k1
< 0, where K > 1 is a positive integer. There are several implicit
schemes we can use, for example, the rst order Backward Euler method and the second-order
CrankNicolson method. Both of these are unconditionally stable for conventional problems.
For discussion simplicity, we apply the Backward Euler scheme to (11)(12) and dene the
following difference equations for
k
= (
k
1
,
k
2
, . . . ,
k
N
)
T
and u
k
= (u
k
1
, u
k
2
, . . . , u
k
M
)
T
:

j =1

j
(x
i
)

k1
j

k
j
t
k

j =1

k
j

j
(x
i
) f (t
k
, x
i
, u
k
i
) L(t
k
, x
i
, u
k
i
) = 0, (13)
1 3
J Glob Optim (2012) 52:305322 309
u
k
i
= arg sup
uU
_
_

j =1

k
j

j
(x
i
) f (t
k
, x
i
, u) L(t
k
, x
i
, u)
_
_
(14)
for i = 1, 2, . . . , M and k = 1, 2, . . . , K.
Note that the time-stepping scheme in (13) is implicit. In this case, (13) and (14) form a
coupled system for
k
and u
k
. To overcome this difculty, we decouple the two sub-systems
by replacing (13) with
N

j =1

j
(x
i
)

k1
j

k
j
t
k

j =1

k
j

j
(x
i
) f (t
k
, x
i
, u
k1
i
) L(t
k
, x
i
, u
k1
i
) = 0, (15)
for i = 1, 2, . . . , M and k = 1, 2, . . . , K. From (15) we have the following matrix equation
for
k
:
(A + B
k
)
k
= A
k1
t L
k1
(16)
where A = (A
i j
) and B
k
= (B
k
i j
) are two M N matrices dened respectively by
A
i j
=
j
(x
i
), B
k
i j
= t
k

j
(x
i
) f (t
k
, x
i
, u
k1
i
) (17)
and L
k1
= (L(t
k
, x
1
, u
k1
1
), . . . , L(t
k
, x
M
, u
k1
M
))
T
for i = 1, 2, . . . , M, j = 1, 2, . . . , N
and k = 1, 2, . . . , K.
For convenience, we further write (16) as
E
k

k
= p
k1
(18)
in the least-squares sense, where E
k
= (A + B
k
) and p
k1
= A
k1
t L
k1
. This
over-determined system can be solved by left-multiplying both side of (18) by (E
k
)
T
.
3 The adaptive RBF algorithm
Adaptive mesh renement has been used extensively with mesh-based numerical methods.
Behrens et al have formulated an adaptive strategy which has been used for nonlinear trans-
port equations [2, 3]. Other techniques, mainly in one dimension, have also been developed
for various RBFs [11, 17, 23].
The RBF in (5) contains a free shape parameter c that can be chosen by the user. The choice
of the value of c can greatly affect both the accuracy of the approximation and the conditioning
of the system matrix. In general, when c becomes large, the multiquadric coefcients matrix
becomes ill-conditioned and the condition number becomes an important factor in choosing
the shape parameter. In practice, the shape parameter c must be adjusted proportionally with
the distance between centers in order to produce a well-conditioned interpolation matrix to
be inverted.
How to choose a shape parameter has drawn much attention in recent years. Convention-
ally, c can be chosen by a rather costly trial and error approach in numerical experiments
(cf. for example, [15, 18]). Several methods for selecting the shape parameter c for both the
multiquadrics and inverse multiquadrics in R
2
have been proposed in the literature such as
those in [5, 7, 22]. However, in general, the optimal choice of c is still an open question.
1 3
310 J Glob Optim (2012) 52:305322
In this work, we present another formula for the shape parameter c. This choice depends
on the average distance between the j t h basis function and its nearest neighbors, given by
c
j
=
d
j
_

2
1
, j = 1, 2, . . . , N, (19)
where 1 < < 2 is an input parameter and d
j
is the average distance between the center of
the j t h basis function and its 2n nearest neighbors, dened by the following algorithm.
Algorithm A
1. Let Z
0
= {y
1
, y
2
, . . . , y
N
} and y
j
Z
0
be a given point. Set z

= y
j
.
2. For l = 1, 2, . . . , 2n,
set Z
l
= Z
l1
\ z

,
nd any z

= arg min
yZ
l
y
j
y
2
and set d
l
j
= ||z

y
j
||
2
.
3. Set d
j
=
1
2n

2n
l=1
d
l
j
.
The c
j
values are chosen as a compromise between smooth tting and ill-conditioning.
If c
j
is small the basis function is peaked while if c
j
is large the basis function is atter over
the space scale. Note that at a distance d
j
from y
j
,
j
falls to a value of (1/ )th of the value
at y
j
, namely
j
(y
j
) = 1/c
j
, and
j
(y
j
+ d
j
z) = 1/( c
j
), where z is a unit vector. The
closer is to 1 the smoother the basis function relative to the space scale. Note also that
each basis function
j
may have a different value of shape parameter c
j
which is taken to be
proportional to the average distance between the basis functions and its nearest neighbours
(AlgorithmA).
We now describe a simple, easy to implement scheme for recursively constructing RBF
approximation in (8) to the solution of the HJB equation based on the residual of the least
squares t over the data points. The idea is that, at each time step, solve (18) and (14) with
a small N, the number of RBFs. Next, calculate the point-wise residuals at the sampling
points x
i
, i = 1, . . . , M. If the maximum residual exceeds a prescribed criterion (or toler-
ance) and N < M, add a basis function centered at the point where the maximum residual
occurs. When each basis function is added, we also adjust the shape parameters using (19)
and AlgorithmA. Finally, we resolve (18) using the new set of RBFs. In short, the adaptation
process follows the familiar paradigm of solve-estimate-rene until a stopping criterion is
satised. Based on this idea, we propose the following algorithm for the numerical solution
of (6)(7).
Algorithm B
1. Choose a tolerance and a set of distinct points X = {x
1
, x
2
, . . . , x
M
}

. Initialize
{v
0
i
}
M
i =1
using the terminal condition (7).
2. Solve A
0
= (h(x
1
), . . . , h(x
M
))
T
for
0
and evaluate u
0
using (10) with k = 0 for
i = 1, 2, . . . , M, where A is dened in (17) and h() is the terminal condition given in
(7).
3. Choose Y
0
= {y
1
, y
2
, . . . , y
N
0 } X, where N
0
is the initial number of basis functions.
4. For k = 1, 2, . . . , K, perform the following steps:
(a) Solve (18) for
k
.
(b) Find an I = arg max
1i M
|( p
k1
E
k

k
)
i
| and R =

( p
k1
E
k

k
)
I

.
(c) If R > , set N
k
= N
k1
+ 1 and Y
k
= Y
k1
{x
I
}, adjust c
j
for all y
j
Y
k
using (19) and AlgorithmA, and go to (a). Otherwise, go to (d).
(d) Evaluate u
k
using (14).
1 3
J Glob Optim (2012) 52:305322 311
4 Stability of the scheme
We now discuss the stability of the scheme (16) in time. For ease of discussion, we assume
that t
k
= t = 1/K for k = 1, 2, . . . , K and assume c in (5) is independent of j . We rst
consider that case that M = N i.e. A is a square matrix. In this case, (16) can be rewritten as

k
= (A + B
k
)
1
A
k1
t (A + B
k
)
1
L
k1
(20)
for k = 1, 2, . . . , K. The following lemma is used to establish the stability results.
Lemma 1 Suppose X is an N N matrix. If X < 1, then (1 + X)
1

1
1X
for any
subordinate matrix norm || ||.
The proof of Lemma 1 can be found, for example, in [9]. The stability of the scheme when
M = N is contained in the following theorem.
Theorem 1 Let
k
be the solution to (20) with M = N for k = 1, 2, . . . , K and
0
be the
terminal condition given in AlgorithmB. Then, when K is sufciently large,

exp
_
k
K
_
_

+
A
1

K
k

n=1
L
n1

_
(21)
where = f
2,
(A)/2c,

denotes the -norm, c is the parameter used in (5),

(A) = A
1

(the condition number of A in

) and f
2,
= max
i,k
f
k
i

2
.
Proof Let G
n
= A
1
B
n
for any feasible n. From (20) we have

k
= (I + G
k
)
1

k1
t (I + G
k
)
1
A
1
L
k1
= (I + G
k
)
1
_
(I + G
k1
)
1

k2
t (I + G
k1
)
1
A
1
L
k2
_
t (I + G
k
)
1
A
1
L
k1
= (I + G
k
)
1
(I + G
k1
)
1

k2
t (I + G
k
)(I + G
k1
)
1
A
1
L
k2
t (I + G
k
)
1
A
1
L
k1
.
.
.
=
k

n=1
(I + G
n
)
1

0
t
k

n=1
_
_
kn+1

j =1
(I + G
kj +1
)
1
_
_
A
1
L
n1
for any k = 1, 2, . . . , K. Taking the norm || ||

on both sides of the above equation and

using the convention that

0
j =1
() = 1, we have
1 3
312 J Glob Optim (2012) 52:305322

n=1
(I + G
n
)
1

+|t |
k

n=1
_
_
kn+1

j =1
(I + G
kj +1
)
1

_
_
A
1

L
n1

n=1
_
1
1 G
n

+|t |
k

n=1
_
_
kn+1

j =1
_
1
1 G
kj +1

_
_
_
A
1

L
n1

(Lemma 1)

n=1
_
1
1 A
1

B
n

+|t |
k

n=1
_
_
kn+1

j =1
_
1
1 A
1

B
kj +1

_
_
_
A
1

L
n1

. (22)
In the above we used the fact that, when |t | is sufciently small, A
1
B
n

A
1

B
n

< 1 for any feasible n by (17).

From (5) and (17) we see that
B
k
i j
= t
j
(x
i
) f
k
i
=
t (x
i
x
j
) f
k
i
(x
i
x
j

2
2
+c
2
)
3
2
=
t (x
i
x
j
) f
k
i
x
i
x
j

2
2
+c
2
A
i j
. (23)
Therefore, taking absolute value on both sides, we obtain
|B
k
i j
|
|t |x
i
x
j

2
f
k
i

2
x
i
x
j

2
2
+c
2
| A
i j
|
|t | f
k
i

2
2c
| A
i j
|,
because y/(y
2
+c
2
) 1/2c for all y 0. Using this estimate we have
B
k

= max
i
N

j =1
|B
k
i j
|
|t |
2c
max
i
_
_
f
k
i

2
N

j =1
| A
i j
|
_
_
=
|t | f
2,
2c
A

. (24)
Substituting this into (22) we obtain

n=1
_
1
1 |t |A
1

f
2,
2c
A

+|t |
k

n=1
_
_
kn+1

j =1
_
1
1 |t |A
1

f
2,
2c
A

_
_
_
A
1

L
n1

_
1
1 |t |
f
2,
2c
(A)
_
k

+|t |
_
1
1 |t |
f
2,
2c
(A)
_
k
A
1

n=1
L
n1

1 3
J Glob Optim (2012) 52:305322 313
Let = f
2,
(A)/2c. Since, |t | = 1/K, the above estimate then becomes

_
1
1

K
_
k
_

+
A
1

K
k

n=1
L
n1

_
(25)
for k = 1, 2, . . . , K. It is easy to show that the function (1+1/y)
y
is monotonically increas-
ing for y > 0, lim
y
(1 +a/y)
y
= e
a
, and 1/(1 y) = 1 + y/(1 y). Using these, we
have from (25)

_
1 +

K
1

K
_
k
K
1

K

1

K

K
_

+
A
1

K
k

n=1
L
n1

_
exp
_
k
K
1

K
__

+
A
1

K
k

n=1
L
n1

_
= exp
_
k
K
_
_

+
A
1

K
k

n=1
L
n1

_
(26)
for all k = 1, 2, . . . , K. Thus, we have proved this theorem.
Using Theorem 1, we can establish the stability of the method when M > N, as given in
the following corollary.
Corollary 1 Let
k
be the solution to (20) with M > N for k = 1, 2, . . . , K and let
0
be
the terminal condition given in AlgorithmB. Then, when t is sufciently small,
k
satises

exp
_
k
K
_
_

+
(A
T
A)
1

|| A
T
||

K
k

n=1
L
n1

_
, (27)
where = f
2,
(A
T
A)/2c.
Proof Multiplying both sides of (16) by A
T
from the left gives
A
T
(A + B
k
)
k
= A
T
A
k1
t A
T
L
k1
.
Since B
k
depends linearly on t , when t is sufciently small, A
T
(A + B
k
) is invertible
because A
T
A is non-singular. Therefore, we have

k
= (

A +

B
k
)
1

A
k1
t (

A +

B
k
)
1

L
k1
for k = 1, 2, . . . , K, where

A = A
T
A,

B
k
= A
T
B
k
and

L
k1
= A
T
L
k1
. This is of the
same form as (20). Therefore, most of the proof for Theorem 1 holds for this case, except for
the estimate of the upper bound of ||

B
k
||

. We now estimate the upper bound for ||

B
k
||

.
From the definition of

B
k
= (

B
k
i j
) and (23) we have

B
k
i j
=
M

l=1
A
il
B
k
l j
=
M

l=1
A
il
A
l j
t (x
l
x
j
) f
k
l
x
l
x
j

2
2
+c
2
.
Using the same argument as that for (24) we obtain
||

B
k
||

|t | f
2,
2c


A

.
1 3
314 J Glob Optim (2012) 52:305322
Using this upper bound and following the proof of (26) we have (21) with A and L replaced
by

A and

L, respectively. Finally, noting that ||

L
k
||

|| A
T
||

||L
k
||

for any feasible k,

we see that (27) follows from the resulting estimate by replacing A and L in (21) with

A and

L respectively.
Remark 1 In the proofs of the above stability results we assumed that the shape parameter c
in (5) is constant for all j = 1, . . . , N, However, it is easy to see from the proofs that, when
the shape parameter depends on j such as the one in (19), Theorem 1 and Corollary 1 still
hold if we dene c = min
1j N
c
j
.
Remark 2 Since k/K 1, the estimate (21) or (27) provides an upper bound for
k

which is essentially independent of K and k, though (A) or (A

T
A) depends on {d
j
}
N
j =1
.
This implies that any initial error in
0
will not grow in the time stepping procedure whereas
the condition number will grow as we add more basis functions in space.
5 Numerical experiments
To verify the usefulness and accuracy of the method discussed in the previous sections, some
numerical experiments were carried out on test problems in 1, 2, and 3 space dimensions.
There are a few issues that need to be claried before we present our numerical experiments.
For the test problems given below, we choose [0, T] = [0, 1] [1, 1]
n
in
(6)(7). Our basis functions are centered at y
1
, . . . , y
N
that are located in an extended region

= [a, a]
n
[1, 1]
n
. Note that the basis functions are time invariant, and a needs to
be sufciently large so that we have a proper transit layer to maintain the quality of solutions
within [1, 1]
n
as reported in [15]. The parameter a is chosen to be a = 3 in all tests below.
Table 1 Computed errors using the collocation method in [15] for Test 1
M

| M 6 | 11 10 | 21 15 | 41 27 | 81
N

| N 6 | 11 10 | 21 15 | 41 27 | 81
L

([0, 1] [1, 1]) 9.45e 2 5.34e 2 3.27e 2 1.78e 2

L

([0, 1], L
2
([1, 1])) 3.45e 2 1.65e 2 8.09e 3 3.66e 3
c 2 1 0.5 0.3
K 41 81 161 321
Table 2 Computed errors using AlgorithmB with a constant c for Test 1
M

| M 6 | 16 10 | 28 16 | 46 28 | 82
N

| N 2 | 6 2 | 10 5 | 22 8 | 33
L

([0, 1] [1, 1]) 8.06e 2 4.52e 2 2.84e 2 1.60e 2

L

([0, 1], L
2
([1, 1])) 8.14e 3 3.86e 3 1.23e 3 4.83e 4
8e 2 8e 3 1e 3 8e 3
c 0.75 2 0.75 0.3
K 41 81 161 321
1 3
J Glob Optim (2012) 52:305322 315
Table 3 Computed errors using Algorithms A and B for Test 1
M

| M 6 | 16 10 | 28 16 | 46 28 | 82
N

| N 2 | 7 2 | 12 7 | 18 8 | 20
L

([0, 1] [1, 1]) 6.67e 2 4.19e 2 2.63e 2 1.23e 2

L

([0, 1], L
2
([1, 1])) 7.71e 3 3.6e 3 1.18e 3 2.1e 4
5e 3 1e 2 5e 4 1e 4
1.016 1.05 1.015 1.005
K 41 81 161 321
Table 4 Computed errors using the collocation method from [15] for Test 2
M

| M 6 | 11 11 | 21 18 | 41 33 | 81
N

| N 6 | 11 11 | 21 18 | 41 33 | 81
L

([0, 1] [1, 1]) 1.16e 1 6.74e 2 3.48e 2 2.11e 2

L

([0, 1], L
2
([1, 1])) 8.47e 2 4.00e 2 2.15e 2 1.36e 2
c 2 0.8 0.25 0.1
K 41 81 161 321
Table 5 Computed errors using AlgorithmB with a xed c for Test 2
M

| M 6 | 16 10 | 28 18 | 52 30 | 88
N

| N 3 | 9 3 | 13 11 | 29 19 | 62
L

([0, 1] [1, 1]) 1.13e 2 5.74e 2 3.00e 2 1.89e 2

L

([0, 1], L
2
([1, 1])) 1.96e 2 6.5e 3 1.87e 3 6.1e 4
2e 2 5e 2 1e 2 1e 3
c 1.45 0.35 0.25 0.1
K 41 81 161 321
Also, the initial number of RBFs for each run is chosen to be 1020% percent of the total
number of collocation points used.
Test 1 Consider the following control problem:
min
0u1
x(1)
subject to
_
x(t ) = x(t )u(t ), a.e. t [0, 1]
x(0) = y.
The corresponding HJB equation is
v
t
+ sup
0u1
[xuv
x
] = 0,
where (t, x) [0, 1) [1, 1] and v(1, x) = x. The exact value function v is given by
v(t, x) =
_
xe
1t
, if x > 0
x, if x 0.
1 3
316 J Glob Optim (2012) 52:305322
Table 6 Computed errors using Algorithms A and B for Test 2
M

| M 6 | 16 10 | 28 18 | 52 30 | 88
N

| N 3 | 9 3 | 13 7 | 23 15 | 47
L

([0, 1] [1, 1]) 9.15e 2 4.68e 2 2.89e 2 1.84e 2

L

([0, 1], L
2
([1, 1])) 1.8e 2 4.2e 3 1.6e 3 9.8e 4
2e 2 8e 2 5e 2 5e 2
1.1 1.9 1.9 1.5
K 41 81 161 321
Table 7 Computed errors using the collocation method from [15] for Test 3
M

| M 9 9|21 21 14 14|41 41 24 24|81 81

N

| N 9 9|21 21 14 14|41 41 24 24|81 81

L

([0, 1] [1, 1]) 6.73e 2 3.67e 2 2.05e 2

L

([0, 1], L
2
([1, 1])) 2.91e 2 1.25e 2 6.43e 3
c 1 0.6 0.25
K 81 161 321
Table 8 Computed errors using AlgorithmB with a xed c for Test 3
M

| M 5 5|13 13 9 9|25 25 15 15|42 42

N

| N 2 2|10 10 3 3|15 15 6 6|30 30

L

([0, 1] [1, 1]) 1.00e 2 6.00e 2 2.79e 2

L

([0, 1], L
2
([1, 1])) 1.44e 2 6.87e 3 1.52e 3
3e 2 5e 2 5e 3
c 2.5 1.5 0.75
K 41 81 161
Table 9 Computed errors using Algorithms A and B for Test 3
M

| M 5 5|13 13 9 9|25 25 15 15|42 42

N

| N 2 2|10 10 4 4|15 15 7 7|28 28

L

([0, 1] [1, 1]) 9.17e 2 6.25e 2 2.45e 2

L

([0, 1], L
2
([1, 1])) 1.6e 2 7.1e 3 1.57e 3
1e 2 2e 2 5e 3
1.045 1.06 1.04
K 41 81 161
We now use the method presented in the previous sections to solve the problem in the re-
gion [0, 1] [3, 3]. The basis functions are distributed in [3, 3]. The error comparisons
are given in Tables 1, 2 and 3. Results in Table 1 were obtained using the method from
[15]. Tables 2 and 3 contain the computed errors in two different discrete norms using our
1 3
J Glob Optim (2012) 52:305322 317
Table 10 Parameters for solving
Test 4 M

| M 11 11 11|33 33 33
N

| N 4 4 4|11 11 11
1.1
5 10
2
K 81
Fig. 1 Computed u
1
, u
2
, u
2
1
+u
2
2
and v at x
2
= 0, t = 0 for Test 4
new scheme with respectively a xed value of c and the variable values of c using (19).
In the tables, M

denotes the number of data points within [1, 1], M is the number of
data points on [3, 3], N

and N are respectively the numbers of basis functions within

[1, 1] and [3, 3], is the input parameter used in (19) and K is the number of time steps.
Also, L

([0, 1] [1, 1]) denotes the discrete maximum error in space and time and
L

([0, 1], L
2
([1, 1])) the maximumof the the discrete L
2
-errors in space at all time steps.
From Tables 1 and 2, we observe that for a xed c, our new scheme needs only about half
the number of basis functions of that required for the method in [15] to obtain a comparable
1 3
318 J Glob Optim (2012) 52:305322
Fig. 2 Computed u
1
, u
2
, u
2
1
+u
2
2
and v at x
2
= 0, t = 0.5 for Test 4
accuracy. Comparing Tables 2 and 3 we also see that, the error is reduced when c is determined
by (19).
Test 2 Consider the following HJB initial value problem:
v
t
+ sup
1u1
[uv
x
] = 0, (t, x) [0, 1) [1, 1]
v(1, x) = x
2
, x [1, 1].
The exact value function v is given by
v(t, x) = [|x| +(1 t )]
2
.
The computed errors from our method and that of [15] are listed in Tables 4, 5 and 6.
Similar to the previous example, results in these tables show that to achieve a given accuracy,
our method with variable c needs about half of the basis functions required by the method in
[15].
1 3
J Glob Optim (2012) 52:305322 319
Fig. 3 Computed u
1
, u
2
, u
2
1
+u
2
2
and v at x
2
= 0, t = 1 for Test 4
Test 3 Consider the following 2D HJB equation:
v
t
+ sup
0u1
_
xv
x
yv
y
_
= 0,
for (t, x, y) [0, 1) [1, 1]
2
with the terminal condition v(1, x, y) = x y. The exact
value function v is given by
v(t, x) =
_
(x + y)e
1t
, if x + y > 0
(x + y), if x + y 0.
This problem is a natural extension of Test 1 to two dimensions, and is considered in [15].
To solve this problem we choose = 10
2
. Tables 7, 8 and 9 show the mesh renement
studies of this 2D example. From the tables we see that, again, the number of basis functions
required by our present method is about half of that by the method in [15] in order for both
to achieve comparable accuracies. This, along with Test 1, suggests that the new scheme
is dimension-independent which is especially important as realistic control problems are
usually in high dimensions.
1 3
320 J Glob Optim (2012) 52:305322
Fig. 4 Computed u
1
, u
2
, u
2
1
+u
2
2
and v at x
3
= x
2
= 0 for Test 4
Test 4 Consider the following HJBequation which arises froma multivariate optimal control
problem:
v
t
+ sup
0 u
1
1 ,
0 u
2
1 ,
u
2
1
+ u
2
2
1.1
_
u
1
u
2
v
x
1
(u
2
u
1
)v
x
2
(u
1
+u
2
)v
x
3
u
1
(u
1
+u
2
)
_
= 0,
for (t, x
1
, x
2
, x
3
) [0, 1][1, 1]
3
with the terminal condition v(1, x
1
, x
2
, x
3
) = x
2
1
x
2
3
.
This is a problem with three state and two control variables and non-trivial nonlinear con-
straints and its exact solution is unknown.
The extended region is chosen to be

= [3, 3]
3
. This problem was solved by our
method using the set of the parameters listed in Table 10. At each time step, the constrained
nonlinear optimization problem is solved by the optimizer NLPQL ([24]). To visualize the
numerical results, we plot the approximate controls u
1
and u
2
, the constraint u
2
1
+u
2
2
and the
value function v at different cross-sections in Figs. 1, 2, 3, 4 and 5. These plots coincide
1 3
J Glob Optim (2012) 52:305322 321
Fig. 5 Computed u
1
, u
2
, u
2
1
+u
2
2
and v at x
1
= x
2
= 0 for Test 4
qualitatively well with those in [26] obtained by a nite volume method. From Figs. 1, 2, 3,
4 and 5 we also see that the value function is non-smooth and the nonlinear constraint is
always satised.
6 Conclusion
This paper presented a numerical method for the HJB equation governing a class of optimal
feedback control problems. The method is based on the implicit backward Euler scheme
in time and an adaptive least-squares collocation radial basis function method in space. The
stability of this method is analyzed showing that the method is unconditionally stable. Numer-
ical experiments showed that our adaptive method is superior to the standard RBF method in
[15] in the sense that the former requires about half of the number of RBFs required by the
latter in order to achieve a comparable accuracy. Thus, the proposed method is promising for
solving real-world optimal control problems via HJB equations.
1 3
322 J Glob Optim (2012) 52:305322
References
1. Bardi, M., Capuzzo-Dolcetta, I.: Optimal Control and Viscosity Solutions of HamiltonJacobiBellman
equations. Birlhauser, Boston (1997)
2. Behrens, J., Iske, A.: Grid-free adaptive semi-Lagrangian advection using radial basis functions. Comput.
Math. Appl. 43(35), 319327 (2002)
3. Behrens, J., Iske, A., Kaser, M.: Adaptive meshfree method of backward characteristics for nonlinear
transport equations, Meshfree methods for partial differential equations (Bonn, 2001). Lect. Notes Com-
put. Sci. Eng. 26, 2136 (2003)
4. Buhman, M.D.: Radial Basis Functions: Theory and Implementations, Cambridge Monographs on
Applied and Computational Mathematics. Vol. 12, Cambridge University Press, Cambridge (2003)
5. Carlson, R.E., Foley, T.A.: The parameter R
2
in multiquadric interpolation. Comput. Math. Appl. 21,
2942 (1991)
6. Falcone, M., Giorgo, T. : An approximation scheme for evolutive HamiltonJacobi equations. In: McEne-
ancy, W.M., Yin, G., Zhang, Q. (eds.) Stochastic Analysis, Control, Optimization and Applica-
tions, pp. 289303. Birkhuser, Boston (1999)
7. Foley, T.A.: Near optimal parameter selection for multiquadric interpolation. J. Appl. Sci. Comput. 1,
5469 (1991)
8. Franke, C., Schaback, R.: Solving partial differential equations by collocation using radial basis func-
tions. Appl. Math. Comput. 93, 7382 (1998)
9. Golub, G.H., Van Loan, C.F.: Matrix Computation. 3rd edn. The Johns Hopkins University Press, Balti-
more (1996)
10. Holmstrom, K.: An adaptive radial basis algorithm (ARBF) for expensive black-box global optimiza-
tion. J. Glob. Optim. 41, 17184 (2008)
11. Hon, Y.C.: Multiquadric collocation method with adaptive technique for problems with boundary
layer. Int. J. Appl. Sci. Comput. 6, 447464 (1999)
12. Horst, R., Pardalos, P.M.: Handbook of Global Optimization. Kluwer, Dordrecht (1995)
13. Huang, C.-S., Wang, S., Teo, K.L.: Solving HamiltonJacobiBellman equations by a Modied Method
of Characteristics. Nonlinear Anal. TMA 40, 279293 (2000)
14. Huang, C.-S., Wang, S., Teo, K.L.: On application of an alternating direction method to HamiltonJacobi
Bellman equations. J. Comp. Appl. Math. 166, 153166 (2004)
15. Huang, C.-S., Wang, S., Chen, C.S., Li, Z.-C.: A radial basis collocation method for HamiltonJacobi
Bellman equations. Automatica 42, 22012207 (2006)
16. Larsson, E., Fornberg, B.: A numerical study of some radial basis function based solution methods for
elliptic PDEs. Comput. Math. Appl. 46, 891902 (2003)
17. Ling, L., Trummer, M.R.: Adaptive multiquadric collocation for boundary layer problem. J. Comput.
Appl. Math. 188, 265282 (2006)
18. Lorentz, R.A., Narcowich, F.J., Ward, J.D.: Collocation discretizations of the transport equation with
radial basis functions. Appl. Math. Comput. 145, 97116 (2003)
19. Pereyra, V., Scherer, G.: Least squares collocation solution of elliptic problems in general regions. Math.
Comput. Simul. 73, 226230 (2006)
20. Richardson, S., Wang, S.: Numerical solution of HamiltonJacobiBellman equations by an exponentially
tted nite volume method. Optimization 55, 121140 (2006)
21. Richardson, S., Wang, S.: The viscosity approximation to the HamiltonJacobiBellman equation in
optimal feedback control: upper bounds for extended domains. J. Ind. Manag. Optim. 6, 161175 (2010)
22. Rippa, S.: An algorithm for selecting a good value for the parameter c in radial basis function interpola-
tion. Adv. Comput. Math. 11, 193210 (1999)
23. Sarra, S.A.: Adaptive radial basis function methods for time dependent partial differential equations. Appl.
Numer. Math. 54, 7994 (2005)
24. Schittkowski, K.: NLPQL: a Fortran subroutines for solving constrained nonlinear programming prob-
lems. Ann. Oper. Res. 5, 485500 (1986)
25. Wang, S., Gao, F., Teo, K.L.: An upwind nite difference method for the approximation of viscosity
solutions to HamiltonJacobiBellman equations. IMA J. Math. Control Inf. 17, 169178 (2000)
26. Wang, S., Jennings, L.S., Teo, K.L.: Numerical solution of HamiltonJacobiBellman equations by an
upwind nite volume method. J. Glob. Optim. 27, 177192 (2003)
27. Zilinskas, A.: On similarities between two models of global optimization: statistical models and radial
basis functions. J. Glob. Optim. 48, 173182 (2010)
1 3