Neural Comput & Applic (1999)8:265–272

 1999 Springer-Verlag London Limited

Injecting Knowledge into the Solution of the Two-Spiral

Problem
J.R. Álvarez-Sánchez
Departamento Inteligencia Artificial, UNED, Madrid, Spain

Wieland’s two-spiral problem is often used as a test
for comparing the quality of different supervised-
learning algorithms and architectures. In this paper,
we use this two-spiral problem to illustrate the
advantages obtained from using all the additional
knowledge about the problem domain in designing
the neural net which solves a given problem. The
characteristics of the knowledge-based net, with
regard to complexity, number of elements, training
speed and generalisation quality, make it appreci-
ably better than alternative nets which make no use
of this knowledge.
Keywords: Backpropagation; Benchmark; Generalis-
ation; Knowledge based design; Neural networks;
Two spirals of Wieland
collection of public-domain benchmark problems [6],
consists of learning to discriminate between two sets
of training points which are distributed along two
spirals in the plane. These spirals make three com-
plete turns round the origin and around each other.
The set of training points (input, output) are
represented graphically in Fig. 1; the positive output
values are marked with a cross and the negative
ones with a circle. The values of the coordinates,
for the 194 points which form the training set, have
been generated using the program provided by Alex
Wieland [6]. Points are placed on 32 angles and
correspond to three complete turns for each spiral
(2·(32 · 3 + 1) = 194 points), the radius varying from

1. The Two-Spiral Task

The two-spiral problem was originally used to dem-
onstrate that multi-layered structures of neurons with
a threshold function could solve non-linearly separ-
able problems, as well as to check the generalisation
capabilities of these neuron structures [1,2]. Many
authors now include it in the benchmarks for speed
and quality of learning for new algorithms and
architecture types [3,4]. In this paper, we apply the
technique of domain-knowledge based neural net
design to this problem [5]. Being a problem of
artificial origin, the knowledge which can be injected
is of a formal nature.
The task, as it appears in the documentation
included in the University of Carnegie Mellon’s

Correspondence and offprint requests to: J.R. Álvarez-Sánchez, Fig. 1. Cartesian plane representation of the data used in training
Dpto. Inteligencia Artificial, Facultad de Ciencias-UNED, Senda a neural net to solve the two-spiral problem. The crosses represent
del Rey, s/n, E-28040 Madrid, Spain. E-mail: jras얀dia.uned.es one output class and the circles the other.
266
6.5 to 0.5 on one spiral and from −6.5 to −0.5 on
the other. The values of the training set are the
pairs of Cartesian coordinates of each point and a
value indicating to which spiral they belong.
The net is trained on the 194 pairs of input/output
(point/class) until it can produce the correct output
for each input of the example points. The time
required for this training is a measure of the quality
of the net, understood as a Problem Solving
Method (PSM).
The choice of which output values are to represent
the two classes is left up to the experimenter’s
judgement. In the documentation for this example,
it is suggested that, to compare results, a correctness
criterion of 40-20-40 be used for the representation
of bipolar outputs across a range of values, assigning
each of the top and bottom 40% of the range,
respectively, to the two classes, and leaving the
central 20% as undefined (incorrect) output. This
criterion is chosen to make it easier for neurons
with sigmoid output to give a correct response, but
is not very precise. In the work reported on in this
paper, we have adopted a much more restrictive
criterion, equivalent to 10-80-10 with a range of −1
to 1; that is, we consider an output between 0.8
and 1 as an indication of belonging to class ‘+’, an
output between −1 and −0.8 as an indication of
belonging to class ‘−’ and the intermediate values,
those between −0.8 and 0.8, as incorrect.
Formulated in this way, the problem is a difficult
one for classical backpropagation nets, and is
impossible to solve using a linear separator. For this
reason, it is used to compare how the training time
varies for different algorithms and according to the
number of weights (or their equivalent) in the net.
The rest of the paper is divided into six sections.
The following section summarises the most signifi-
cant reference results for this problem. In Section 3,
an analysis is made of the notion of adding previous
knowledge about the solution to the problem, and
of the method used to incorporate this knowledge
in the design of the net.
Based on the information obtained, we then
present the design of the net which we use to
solve this problem in Section 4. In Section 5, the
information about the problem obtained in Section 3
is used to calculate adequate initial values for the
weights of the proposed net. In Section 6 the simul-
ation results for this net are presented.
2. Summary of Previous Results
Here, we summarise the main aspects of the results
obtained by different authors and contained in the
J.R. Álvarez-Sánchez
University of Carnegie Mellon’s repository of
examples. All the reports of this repository deal
with neural nets with feedforward architectures, lin-
ear-type units (neurons), non-linear sigmoid transfer
and learning through error backpropagation by gradi-
ent descent. The main difference between the differ-
ent reports lies in the variations in the learning
algorithm, the layering structure and the number of
elements per layer. Table 1 gives a comparison of
the results of these reports, together with the results
obtained in our work (bottom row), these latter
results being detailed in later sections.
The comparison table includes data about the type
of learning algorithm: ‘BP’ refers to the standard
gradient-descent backpropagation algorithm, ‘quick
prop’ refers to an improvement introduced in Fahl-
man [1], ‘cascade correlation’ refers to an algorithm
which generates the structure of the net (in cascade)
during the training [7], and ‘resilient propagation’
[3] refers to a modification of the backpropagation
algorithm in which the weights are incremented
or decremented as a function of the sign of the
error gradient.
‘Cascade correlation’ algorithms use a technique
in which the weights in hidden layers are frozen so
that the effective number of weights modified in
each example of a given epoch are reduced, on
average, (from a total of n(n+5)
2
weights for n neurons
in cascade, one for each layer, all of which receive
the inputs to the net) to a value which can be
estimated as a function of the total number of
changes (referred to as ‘cross connections’ in Fahl-
man and Lebiere [7]) divided by the total number
of examples presented in all the epochs (that is,
divided by the number of epochs and by the number
of examples), and divided by two modification
phases (forwards and backwards). This estimated
equivalent number of modifiable weights is the one
given in the table (the values in italics) for the two
cases of type ‘cascade correlation’ which appear
there. To calculate the equivalent values of the
weights, we have used the ‘cross connections’ data
from Treadgold and Gedeon [4]. In the case of GCS
from Fritzke [8], the number of weights was counted
by hand in the connections figure.
In the last column of the table the average number
of epochs for the training is indicated and, in some
cases, also the corresponding number of epochs for
the same problem when the point density is quad-
rupled (in brackets). The indications of the technique
quality are the lowest number of epochs and the
lowest number of weights together.
To illustrate the type of generalisation usually
obtained in the results, Fig. 2 shows the response
of a net, in the region of the plane used in the
Injecting Knowledge into the Solution of the Two-Spiral Problem 267

Table 1. Summary of the results for the set of examples available in the C.M.U. repository.

Authors Learn Err.Func. Transf. Connect. N #w tol #t epoch

Lang and BP ␮ = 0.5, SSE sigmoid 2=5=5=5=1 16 138 – 3 20000

Witbrock [2] ␭ = 0.001 [−1,+1] shortcut (64000)
id. cross-entr. id. id. 16 138 – 3 11000
quickprop argtanh id. id. 16 138 – 3 7900
Walker, D. BP ␭ = 0.1, cut at 0.15 sigmoid 2-20-10-1 31 281 30% – 13900
(unpublished) ␮ = 0.7 [−.,+.]
id. id. id. 2-16-8-1 25 193 30% – 300000
Leighton, R. BP + noise in argtanh, cut sigmoid 2=5=5=5=1 16 138 40% 3 4000
(unpublished) deriv. 0.001 [−.,+.] shortcut
Fahlman and cascade argtanh sigmoid cascaded, 16 31 – 100 1700
Lebiere [7] correlation [−1,+1] shortcut (2262)
Treadgold and cascor + argtanh sigmoid cascaded, 15 101 20% 100 2437
Gedeon [4] RPROP [−.,+.] shortcut
Fritzke [8] supervised GCS class err. 0, 1 RBF adapt. 2-145-1 146 530 – 1 180
This paper BP modif. SSE sin [−1,+1] 2-(2)-1 3 3 10% 100 96
Section 6 (MATLAB) (102)

BP = Backpropagation, ␭ = learning rate, ␮ = momentum coefficient, cascor = cascade correlation, RPROP = Resilient Backpropagation,
GCS = Growing Cell Structures, shortcut = neurons in a layer receive inputs from all previous layers, SSE = Sum Squared Error.
N = Number of neurons, #w = number of adjustable weights, tol = error tolerance (in % of range) for correct classification, #t = number
of tests.

black and white zones do not divide the plane into

separate clearly-defined spirals.
Better generalisation results have been obtained
with some other architectures, for example, by
Fritzke [8] or by Augusteijn and Steck [10].

3. Analysis of the Problem

Fig. 2. Representation of the response, over the whole plane, of
a net (with 15 hidden units) trained using Fahlman’s ‘cascade
correlation’ algorithm (in black, the positive zone and in white,
the negative zone).
training, obtained from simulation using Fahlman’s
‘cascade correlation’ algorithm [9]. In this figure,
the training examples shown in Fig. 1 have been
superimposed to show that the results coincide for
practically all the points. Note, however, that the
The knowledge we possess about the geometric
nature of the discrimination problem which the net
has to solve is very useful for deciding which
transformations of the input and output data of the
net make the learning process more efficient. We
first examine a graphical representation of the data
to be used in the training, that presented in Fig. 1,
which highlights the central symmetry and the per-
iodic character of the alternation between points of
the two output classes. This central symmetry sug-
gests the use of polar coordinates. The relation
between the distance of the points from the origin
(radius) and the angle with respect to the x-axis
is shown in Fig. 3. The distribution in (periodic)
alternating bands can be readily observed.
The points of a spiral obey the equation
r = p · (␪ + 2␲n) + ro relating the radius r and the
angle ␪ (with n an integer indicating the number of
the cycle); the parameters p and ro define the parti-
cular spiral. The geometric interpretation of these
parameters on the polar coordinate representation of
the data is also shown in Fig. 3. In this case they
268
Fig. 3. Representation of the radius in function of the angle
(between −␲ and ␲) for the training points of the two spirals.
The parameters of the spiral, the initial radius, ro, and the step
of the spiral, 2␲p, are indicated.
correspond to 2␲p ⬵ 2, and therefore p ⬵ ␲1 , and
ro ⬵ 0.5. This information will prove useful for the
initialisation of the weights. Note that there is a
generalisation implicit in the design, namely that
not only does the net assign the correct values to
the training examples, but it also divides the whole
plane into two spiral regions of opposite polarity,
i.e. including the points not given to the net as
examples.
4. Selection of Pre-processing and
Transference Function
To take advantage of the central symmetry property
discussed in the previous section, we pre-process
the inputs by transforming them into polar co-
ordinates. The results of this transformation are then
used as the terms of a weighted sum (adjustable
weights). Due to the periodic alternation of the two
classes of points, we use a differentiable transference
function of periodic type, such as sine.
The transformation from Cartesian to polar co-
ordinates can be performed in two separate units
(without learning) constituting the first layer,
which generates the values r = √x2 + y2 and
␪ = arctan ( xy ) + ␲ · sign(y). This latter function for
␪, which is expressible in many programming langu-
ages as atan2(y,x), gives the angle (between −␲
and ␲), but preserving the information of the quad-
rant, information which is lost with the standard
arctangent function. The second (output) layer has
only one unit, which generates the output of the net
in the form Z = sin(w1␪ + w2r + b), where the
weights w1 and w2, together with the displacement
b, are modifiable through learning.
J.R. Álvarez-Sánchez
Since only the output layer has adjustable weights,
we can use simple learning algorithms of the per-
ceptron type (though it is also possible to use more
elaborate algorithms which modify the weights in
proportion to the derivative of the error with respect
to the weights). In this case, we can apply, for
example, the weight-modification formula ⌬wi =
␣␾icos(w1␪ + w2r + b) · (D − Z), where ␾1 is ␪, ␾2
is r and D is the desired output.
5. Initialising the Weights
Making good use of the knowledge of the problem
domain, we can also calculate approximate values
for the weights, and in this way reduce the training
time. We have already seen that the relation between
the radius r and the angle ␪ for the spirals is
r = p · (␪ + 2␲n) + ro, with n an integer, where the
characteristic constants p and ro define each spiral.
We want the net output Z = sin(w1␪ + w2r +b) to
be equal to 1 for the spiral, so that the argument
of the sine function should be w1␪ + w2r + b =
2 + 2␲m, for m any integer. Substituting r of the
␲
spiral equation in this latter equation and grouping
terms, we obtain the expression
(w1 + w2p)␪ + w2(2␲np + ro) + b
␲
= + 2␲m; n,m 苸 Z
2
Since this equation must be valid for any angle,
we can separate it into two parts: the ␪ component
and the rest.
冦
␪(w1 + w2p) = 0 ∀␪ ⇒ w1 = −w2p
␲
2␲nw2p + w2ro + b = + 2␲m n,m 苸 Z (1)
2
The second part of Eq. (1) can be separated into
terms relative to the integers and all other terms.
This then leaves us with equations which relate the
weights in function of the values measured on the
training data pⴱ and rⴱo, which will be approximately
Fig. 4. Scheme of blocks and connections in the solution pro-
posed for 2-dimensional classification problems with central or
spiral symmetry.
Injecting Knowledge into the Solution of the Two-Spiral Problem
equal to p and ro, the parameters which represent
the spiral (though the values p and ro are not unique
for each theoretical spiral, if negative values are
considered). We assign initial values to the weights
w2 = npⴱ
m
; w1 = −w2pⴱ and to the displacement
b = 2 − w2rⴱ0. The values of the integer variables m
␲
and n are chosen not to be divisible by 2, so that
an odd number of spirals of each class alternate in
the solution.
This type of initialisation relation allows the rep-
resentative values of the spiral to which the example
points belong to be deduced, after the adjustment
through learning, the value of p and ro being
obtained from the previous formulas by putting n
and m equal to 1 (canonical representative).
6. Experimental Results
To confirm the previous theoretical studies, we car-
ried out experiments using the commercial tool for
simulating traditional neural nets contained in the
MATLAB ‘neural net toolbox’. More concretely, we
used the standard algorithm for learning by acceler-
ated backpropagation. This algorithm employs a
momentum term in the modification of the weights,
and can increase or decrease the learning coefficient
in the course of the training if the quadratic error
does not diminish. An example of the result of
training the net is shown in Fig. 5, where its
response in the region of the plane where it was
Fig. 5. Example of the result after adjustment of the parameters.
After 109 epochs, in this case, the values of the weights were
(−1.016, −3.294) and the displacement (bias), −2.511. The training
examples are superimposed.
269
given examples (shown superimposed) is illustrated.
Comparing this with the response typical of other
nets, such as that shown in Fig. 2, it can be readily
observed that it is more generic. Furthermore, the
net can be seen to really represent the concept
of a spiral, which is implicit in the generalisation
assumptions that are part of the objectives, but
which only exists from the observer’s point of view
and not in the specifications themselves.
To implement the transfer function as a sine
function, as required, it was only necessary to write
it and its derivative (the cosine) in the form used
by the simulation tool. We also needed to write a
small program for generating the training examples,
the initial values of the weights (as described in the
last section), and the calls to the subroutines of the
tool for the training and the visualising of the
results. Figure 6 shows an example of the response
of the net for the initial values of the weights before
the training. It can be seen that this first solution
already approximates the desired one, though train-
ing is needed to tune the response completely.
To generate different values of the ratio between
the integers m and n, which is used to calculate the
initial values of the weights, mn was assigned to the
value of the formula ‘1+0.2ⴱrandn+2ⴱfloor
(randⴱ2)’. This generates a distribution lying
between 1 and 5 while avoiding multiples of 2 (or
values close to this), giving rise to solutions with
interleaved multiple spirals. Such a solution is shown
Fig. 6. Response obtained, before training, with initial values of
the weights (−0.7366, −2.143) and displacement (bias) 2.446.
The training examples are shown superimposed (only the visible
ones coincide correctly). This response is from the same simul-
ation run as the one shown in Fig. 5.
270
in Fig. 7, where a response with two interleaved
triple spirals was obtained after training and with
an initial ratio of mn close to 3.
For the initial value of the learning coefficient,
an approximation was made, ensuring that the coef-
ficient multiplied by the maximum absolute value
of the lower-range input (−␲ ⬍ ␪ ⬍ ␲) and by the
maximum accumulable error (twice the number of
examples) is smaller than half the value of the
average weight. Using this approximation, the initial
1
value of the learning coefficient is less than 2␲·2·194
−4
⬇ 4 · 10 . The learning algorithm modifies this
value, increasing or decreasing it as necessary, in
order to reduce the error. Figure 8 shows an example
of the evolution of the sum of the total error and
the learning coefficient, through a series of train-
ing epochs.
The criterion used to terminate the training in the
algorithm employed is that the sum of the quadratic
error of the examples in an epoch falls below a
certain limit. If we assume that the errors follow a
normal distribution for all the training examples,
then the standard deviation of the error is the square
root of the average quadratic error. To ensure that
the probability of the error remaining within a given
interval is 99.9%, the maximum error (as a percent-
age of the range) must be 3.5 times the standard
deviation. From this condition, the quadratic error
sum criterion as a function of the maximum per-
centage of the error range for each example, can be
obtained. In our case, we have chosen to consider
Fig. 7. Example of convergence to values which represent inter-
leaved triple spirals, obtained after 229 training epochs, with
final weights (−2.995, −9.375) and displacement (bias) 6.083.
J.R. Álvarez-Sánchez
Fig. 8. Evolution of the parameters on a 229-epoch simulation
run with training. The upper half of the figure shows the variation
of the sum of the quadratic error for all the examples (on a
logarithmic scale) until a maximum error (dotted line) is reached.
The lower half of the figure shows the variation in the learning
coefficient on the same simulation run.
the training valid if the maximum error is 10% of
the range, and therefore a criterion of
194(0.13.5 ) ⬇ 0.63 must be established for the
2 2
maximum value of the sum of the quadratic error
in the examples.
100 training tests were carried out for the net we
designed, with different initial values for the
weights, according to the ratios deduced in the
previous section. An average of 96 epochs was
obtained (with deviation 120, minimum of 0 and
maximum of 438 epochs). The maximum error in
each example after each simulation run was 9.8%
and therefore less than the 10% proposed as the
admissible limit.
Simulations were also carried out with a quad-
ruple density of training points. An average of 102
epochs was obtained over 100 runs (with deviation
120, minimum 0 and maximum 543 epochs). The
maximum error for each example after each simul-
ation run was 9.6% (in this case the criterion for
the maximum value of the sum of the quadratic
error was 2.5). With the proposed architecture, there
is no substantial increase in the number of training
epochs when the number of examples is quadrupled
(unlike other architectures). This is due to the
capacity of this architecture to directly represent the
structure, with no dependence on the number of
examples if these all belong to the same spirals.
7. Conclusions
In this article we present the design of a neural net
to solve the well-known two-spiral problem. In the
Injecting Knowledge into the Solution of the Two-Spiral Problem
Fig. 9. Output of the net (Z) in response to the input of the points in the zone of interest of the XY plane. (a) Output of a correctly-
trained net in the case of homogeneous, centred, symmetric spirals; (b) output of a net with non-unitary coefficients in the layer
implementing the conversion to polar (elliptic) coordinates, concretely: r = √0.3x2 + y2; ␪ = arctan (−0.2y,x); and weights in the output
layer w1 = −2, w2 = 1, b = 0.
design phase, we have used knowledge about the
problem to improve the response of the net. The
net obtained is simpler (smaller number of weights)
and needs fewer training epochs than other neural
nets commonly used to solve the same problem. We
have also verified that the type of response generated
by the net designed in this article, is of better
quality from the point of view of the generalisation
of the two-spiral problem.
The architecture proposed and developed in the
work reported on here is, apparently, specific to the
two-spiral problem. However, this is also the case
for the architectures with which it is compared
(Section 2). These architectures have also been
designed specifically for this problem, selecting the
algorithm, the number of neurons, the number of
layers (except in the ‘cascade’ algorithms case), the
type of activation function, the error function, the
initialisation of the weights, etc.
The essential characteristic of a neural net is its
parametric nature. Starting from a skeletal model
with a multilayered architecture, we select the pre-
processing to apply to the inputs, the connectivity,
the local function of each neuron and the learning
algorithm. The design of this architecture is knowl-
edge injected a priori into the net. The rest, the
supervised learning, is responsible for the adjustment
of the parameters, thus projecting the generic net
and its pre-processing onto the specific net (with
already-adjusted parameter values) which best solves
the problem.
The net designed here for the two-spiral problem
can be easily modified to solve other problems of
the same family, i.e. problems involving points with
central symmetry and alternation in bands (not
271
necessarily equidistant or homogeneous). This modi-
fication consists in also allowing the parameters of
the pre-processing layer (conversion to polar
coordinates) to be modifiable (non-homogeneous
elliptic coordinates, in the case where the coef-
ficients are not equal to 1). This change enables
other problems, like the example shown in Fig. 9(b),
to be solved. This figure can be compared to
Fig. 9(a), in which the typical response for the con-
crete example of the spirals presented in this paper
is shown.
Acknowledgements. This work has been partially
supported by the project TIC-97-0604 of the Comi-
sión Interministerial de Ciencia y Tecnologı́a
(CICYT) of Spain.
References
1. Fahlman SE. Faster-learning variations on back-propa-
gation: an empirical study. In: Touretzky DS, Hinton
GE, Sejnowski TJ (eds), Proceedings Connectionist
Models Summer School, Morgan Kaufmann, 1988
2. Lang KJ, Witbrock MJ. Learning to tell two spirals
apart. In: Touretzky DS, Hinton GE, Sejnowski TJ
(eds), Proceedings Connectionist Models Summer
School, Morgan Kaufmann, 1988
3. Riedmiller M, Braun H. A direct adaptive method for
faster backpropagation learning: the RPROP algorithm.
Proceedings IEEE International Conference on Neural
Networks, IEEE, San Francisco, 1993
4. Treadgold NK, Gedeon TD. A cascade network algor-
ithm employing progressive RPROP. In: Mira J, Mor-
eno-Dı́az R, Cabestany J (eds), Biological and Arti-
ficial Computation: From Neuroscience to Technology.
Lecture Notes in Computer Science 1240, Springer-
Verlag, 1997
272
5. Russell S, Norvig P. Artificial Intelligence: A Modern
Approach. Prentice-Hall, 1995; 625–648
6. Wieland A. Two spirals. In: White M (ed), http:
//www.boltz.cs.cmu.edu/benchmarks/two-spirals.html.
CMU Repository of Neural Network Benchmarks,
1988
7. Fahlman SE, Lebiere C. The cascade-correlation learn-
ing architecture. In: Touretzky DS (ed), Advances in
Neural Information Processing Systems 2, Morgan
Kaufmann, 1990
J.R. Álvarez-Sánchez
8. Fritzke B. Growing cell structures – a self-organizing
network for unsupervised and supervised learning.
Neural Networks 1994; 7: 1441–1460
9. Fahlman SE. Cascade neural network simulator v1.0
(public domain software). In: White M (ed), http:
//www.boltz.cs.cmu.edu/software.html#Cascade. CMU
Repository of Neural Network Benchmarks, 1988
10. Augusteijn MF, Steck UJ. Supervised adaptive clus-
tering: A hybrid neural network architecture. Neural
Computing & Applications 1994; 78–89