Gradient-based mathematical optimization:

new challenges in solving real-world modelling

problems
Jan A. Snyman

Multidisciplinary Design Optimization Group (MDOG)

Department of Mechanical and Aeronautical Engineering,
University of Pretoria
SAMS Subject Classification: 16

Decision-making based on the mathematical modelling of real-world phenomena

often leads to the solution of a non-linear mathematical optimization problem
of the general form:
minimize f (x), x = [x1 , x2 , . . . , xn ]T ∈ Rn (1)
w.r.t. x
subject to the constraints

gj (x) ≤ 0, j = 1, 2, . . . , m
hj (x) = 0, j = 1, 2, . . . , r

where f (x), gj (x) and hj (x) are scalar functions of the real column vector x.
The continuous components xi of x = [x1 , x2 , . . . , xn ]T are called the (design)
variables, f (x) is the objective function, gj (x) denotes the inequality constraint
functions and hj (x) the equality constraint functions. The optimum vector x
that solves problem (1) is denoted by x∗ with corresponding optimum function
value f (x∗ ). If no constraints are specified, the problem is called an uncon-
strained minimization problem.
Here “decision-making” is interpreted in its broadest possible sense. It could
be deciding on the design dimensions of a vehicle to ensure optimum perfor-
mance, the determination of the coordinates of atoms in a molecular structure
that corresponds to a minimum energy configuration, or the composition of an
investment portfolio to give maximum return on investment.
The formulation of the appropriate optimization problem of the form (1), usually
arises from what is generally known as a mathematical modelling process. In
brief, in attempting to make the optimum (correct) decision with regard to

1
a real-world problem, the mathematical modelling exercise requires the cyclic
performance of the following four steps: 1) the observation and study of the
real-world situation associated with a practical problem, 2) the abstraction of
the problem by the construction of a mathematical model that is described in
terms of preliminary fixed model parameters p, and variables x that have to
be determined such that the model performs in an acceptable manner, 3) the
solution of a resulting purely mathematical problem that requires an analytical
or numerical solution to yield the optimum (correct) solution x∗ , and 4) the
evaluation of the solution x∗ and its practical implications. After step 4) it may
be necessary to adjust the parameters p and refine the model, which will result
in a new mathematical problem to be solved and the corresponding new solution
to be evaluated. It may be required to perform the modelling cycle a number
of times, before an acceptable optimum solution is obtained. More often than
not, the mathematical problem to be solved in 3) is a constrained optimization
problem, of the form (1).
Many methods are available for solving the general problem (1), often also called
a Nonlinear Programming, Mathematical Programming or Numerical Optimiza-
tion problem. Classical analytical methods include Lagrangian methods where,
for example, necessary conditions known as the Karush-Kuhn-Tucker (KKT)
conditions are used to identify candidate solutions. For n large, these meth-
ods, because of their combinatorial nature, become impractical, and solutions
are rather obtained numerically by means of suitable numerical algorithms. The
most important class of these methods are the so-called gradient-based methods.
This paper is restricted to methods of this kind.
Apart from the method of steepest descent due to Cauchy and Newton’s method,
the history of Numerical Optimization, where functions of many variables are
considered, is relatively short, spanning roughly only 55 years. At the end of
the 1940s the very important simplex method for solving the special class of
linear programming problems was developed. Since then numerous methods
for solving the general optimization problem have been developed, tested, and
successfully applied to many important problems of scientific, technological and
economic interest. There is no doubt that the advent of the computer was es-
sential for the development of these numerical optimization methods. The most
well known of these methods, in addition to the older ones mentioned above, are
various quasi-Newton and conjugate gradient methods for unconstrained prob-
lems, and the penalty function, gradient projection, augmented Lagrangian and
sequential quadratic programming (SQP) methods for constrained problems.
In spite of the proliferation of numerical gradient-based methods, there is to
date no universal method for solving all problems, even if restricted to cases
where all the functions are analytically known, continuous and smooth. Many
of the algorithms are tailored to a particular type of optimization problem and
it is often the user’s responsibility to choose an algorithm that is appropriate
for the specific application. In spite of the mathematical sophistication of exist-
ing gradient-based algorithms, certain inhibiting difficulties remain when these

2
algorithms are applied to real-world problems. This is particularly true in the
field of engineering, where unique difficulties occur that have prevented the gen-
eral application of mathematical optimization techniques to design problems.
Typical optimization difficulties that arise are that the functions are often very
expensive to evaluate (requiring, for example, the time-consuming finite ele-
ment analysis of a structure, the simulation of the dynamics of a multi-body
system, or a Computational Fluid Dynamics simulation). The existence of noise
(numerical or experimental) in the objective and constraint functions, as well
as the presence of discontinuities in the functions, constitute further obstacles
in the application of standard and established methods. Multiple local minima
(requiring global optimization techniques), the existence of regions in the design
space where the functions are not defined, and the occurrence of an extremely
large number of design variables, disqualifying the SQP method (for example),
are further factors complicating the use of classical gradient-based methods.
All the above stated difficulties have been addressed in research done at the
University of Pretoria over the past twenty years (Snyman, 2004). This research
has led to new gradient-based algorithms that enable the user to overcome most
of the above stated inhibiting factors. The new methods have the common
and unique property, for gradient-based methods, that no explicit objective
function line searches are required. This paper reviews these novel methods,
highlighting the unique characteristics of each algorithm by comparison with
classical gradient-based line search descent methods.
In conclusion, some examples of the application of the new methods to mechani-
cal engineering design problems will be presented. In these cases the mathemat-
ical models used are typically finite element models of structures (requiring the
iterative solution of large systems of simultaneous equations), dynamic simula-
tions of the motion of multi-body systems (via numerical solutions of systems of
ODE’s), or the modelling of the flow of gasses/fluids via computational fluid dy-
namics (CFD) (involving the numerical solution of the Navier-Stokes equations
over large and fine computational grids). These models often give rise to the
difficulties mentioned above and which have inhibited the general application
of traditional gradient-based optimization techniques to mechanical engineering
design problems. The examples given in the current paper will show how these
difficulties can be overcome by the use of the novel optimization methods.

Reference

Snyman J.A., 2004, Practical Mathematical Optimization: An introduction to

basic optimization theory and classical and new gradient-based algorithms, Kluwer
Academic Publishers, Dordrect, The Netherlands.