NELDER-MEAD ALGORITHM

The Nelder-Mead simplex algorithm nds a minimum of a function of several

variables without dierentiation. It is widely used, even though too little is known
about its convergence properties. See Nelder, J.A. and Mead, R., "A Simplex
Method for Function Minimization", Computer Journal, Vol. 7, Issue 4 (1965),
308-313 for the original article, and see Lagarias, Jerey C., James A. Reeds, Margaret H. Wright, and Paul E. Wright, "Convergence Properties of the Nelder-Mead
Simplex Method in Low Dimensions," SIAM Journal on Optimization, Vol. 9,
No. 1 (1998), 112-147.for a more contemporary view of the algorithm. A Google
Scholar (scholar.google.com) search on "Nelder-Mead" will demonstrate the level of
current activity investigating this algorithm. The Nelder-Mead algorithm is one of
those great ideas that turns out to be widely eective and very dicult to analyze
mathematically.
To begin, we need a denition.

Denition 1. A simplex in Rn is a set of n + 1 points x1 , . . . , xn in Rn such that

the set of vectors {xi x0 :: i = 1 . . . n} is linearly independent in Rn .

(1) Any permutation of a simplex is a simplex.

(2) A simplex is any set of n + 1 points not contained in any n 1-dimensional
hyperplane..
(3) In R2 , a simplex is a triangle.
(4) In R3 , a simplex is a tetrahedron.

The Nelder-Mead algorithm starts with a simplex in domain of the function to

be minimized, then modies the simplex ve dierent ways until the simplex is
very at (function value is almost the same at all the vertices), at which point
the minimum is the vertex with the smallest function value. Usually when the
minumum is found the simplex is very small.
The ve transformations of the simplex are:
1

NELDER-M EAD ALGORITHM

Reflection

Outward
Contraction

Expansion

Inward
Contraction

Shrink
Sometimes the simplex is called an amoeba because of the way it crawls around
looking for the minimum. A web site that shows this behavior is at
http://www.cse.uiuc.edu/eot/modules/optimization/NelderMead/.
Here is how the Nelder-Mead algorithm works to nd the minimum point for
a function f (x):Rn R. The algorithm is taken from Press, et.al., Numerical
Recipes, Cambridge U. Press, Cambridge GB, 1992.
(1) Pick a starting point x0 Rn and step size s R about 1/2 distance or
less from starting point to minimum

NELDER-M EAD ALGORITHM

(2) Dene xi = x0 + sei , where ei are the standard basis vectors in Rn . The
set S = {x0 , . . . , xn } is our starting simplex.
(3) Throughout the algorithm, starting now, xhi is the point in the simplex
where f is maximum, xsec is the point where f takes its second highest
value, and xlo is the point where f is minimum. Dene y as the centroid of
x0 + + xc
hi + + xn
.
the face of the simplex opposite xhi . Thus y =
n
(4) Now heres the algorithm, assuming the initial simplex is dened.

NELDER-M EAD ALGORITHM

// assume given f and x[0] and stepsize

Calculate x[1],...,x[n]
determine lo, sec, hi, y for starting values of x[0],...,x[n]
While (f(x[hi])-f(x[lo]))/(abs(f(x[hi]))+abs(f(x[lo]))+1) < 1e-15 do
// stop when simplex is horizontal
z = y + (y - x[hi]) // reflection
if f(z) < f(x[lo]) then
w = y + 2(y - x[hi]) // reflection with expansion
If f(w) < f(z) then
x[hi] = w // at this point x_hi may no longer be highest
else
x[hi] = z // at this point x_hi may no longer be highest
endif
else
if f(z) < f(x[hi]) then // contraction will be outward
x[hi] = z // at this point x_hi may no longer be highest
endif // else contraction will be inward
if f(x[hi]) > f(x[sec]) then
z = x[hi] + 0.5(y - x[hi])
if f(z) < f(x[hi]) then
x[hi] = z
// contraction
else
for i=0..n do // shrink
if i != lo then
x[i] = 0.5(x[i] + x[lo])
endif
endfor
endif
endif
endif
determine lo, sec, hi, y for current values of x[0],...,x[n]
endwhile
Return(x[lo])

Press, et.al. recommend running the algorithm, then restarting at the minimum
point xlo . If the algorithm got stuck, it will go to a new minumum; if you found
a true minimum that point will be returned again. Keep repeating the algorithm
until you get the same minimum point (essentially) twice in a row.
Heres a sample run:

NELDER-M EAD ALGORITHM

-0.5
-1
-1.5
-2
-2.5
-3
-3.5
1