Analysis of Support Vector

Stella M. Clarke
Department of Industrial & Manufacturing
Engineering,
The Pennsylvania State University,
University Park, PA 16802
Jan H. Griebsch
Doctoral Candidate
Lehrstuhl fr Effiziente Algorithmen,
The Technical University of Munich,
Munich, Germany
Timothy W. Simpson1
e-mail: tws8@psu.edu
Departments of Mechanical & Nuclear
and Industrial & Manufacturing Engineering,
The Pennsylvania State University,
University Park, PA 16802
Regression for Approximation of
Complex Engineering Analyses
A variety of metamodeling techniques have been developed in the past decade to reduce
the computational expense of computer-based analysis and simulation codes. Metamod-
eling is the process of building a model of a model to provide a fast surrogate for a
computationally expensive computer code. Common metamodeling techniques include
response surface methodology, kriging, radial basis functions, and multivariate adaptive
regression splines. In this paper, we investigate support vector regression (SVR) as an
alternative technique for approximating complex engineering analyses. The computation-
ally efficient theory behind SVR is reviewed, and SVR approximations are compared
against the aforementioned four metamodeling techniques using a test bed of 26 engi-
neering analysis functions. SVR achieves more accurate and more robust function ap-
proximations than the four metamodeling techniques, and shows great potential for meta-
modeling applications, adding to the growing body of promising empirical performance
of SVR. DOI: 10.1115/1.1897403

1 Introduction In this paper, support vector regression SVR is investigated as

Much of todays engineering analysis requires running complex
and computationally expensive analysis and simulation codes,
such as finite element analysis and computational fluid dynamics
models. Despite continuing increases in computer processor
speeds and capabilities, the huge time and computational costs of
running complex engineering codes maintains pace. A way to
overcome this problem is to generate an approximation of the
complex analysis code that describes the process accurately
enough, but at a much lower cost. Such approximations are often
called metamodels in that they provide a model of the model
1. Mathematically, if the inputs to the actual computer analysis
are supplied in vector x, and the outputs from the analysis in
vector y, then the true computer analysis code evaluates
y = fx
where fx is a complex engineering analysis function. The com-
putationally efficient metamodel approximation is
y = gx
such that
y = y +
where includes both approximation and random errors.
There currently exist a number of metamodeling techniques to
approximate fx with gx, such as polynomial response surface
models, multivariate adaptive regression splines, radial basis func-
tions, kriging, and neural networks, and a recent comparison of
the first four of these metamodeling techniques can be found in
Ref. 2. All of these techniques are capable of function approxi-
mation, but they vary in their accuracy, robustness, computational
efficiency, and transparency. For example, although neural net-
works are able to approximate very complex models well, they
have the two disadvantages of i being a black box approach
and ii having a computationally expensive training process 3.
Black box means that little can be seen and understood about the
model, because an exact function is not generated, only a trained
box that accepts inputs and returns outputs.
1
Corresponding author.
Contributed by the Design and Automation Committee for publication in the
JOURNAL OF MECHANICAL DESIGN. Manuscript received October 7, 2003; revised
August 13, 2004. Associate Editor: Wei Chen.
an alternative technique for approximating complex engineering
analyses. SVR is a particular implementation of support vector
machines SVM, a principled and very powerful method that in
the few years since its introduction has already outperformed most
other systems in a wide variety of applications 4. In many
applications, SVMs are known to produce equally good, if not
better, results than neural networks, while being computationally
cheaper and producing an actual mathematical function i.e., no
black box. Hearst 5 comments that SV learning is based on
some beautifully simple ideas and provides a clear intuition of
what learning from examples is about and that SVMs can lead
to high performances in practical applications. In addition,
Takeuchi and Collier 6 state that SVMs have been applied very
successfully in the past to several traditional classification tasks
1 such as text classification. Other successful applications of
SVMs have included handwritten character and digit recognition,
face detection, text categorization, and object detection in ma-
chine vision 7; SVR has also been used to estimate minimum
2 zone straightness and flatness tolerances 8.
The SVM algorithm is a nonlinear generalization of the gener-
alized portrait algorithm developed in Russia in the 1960s 9. In
its present form, SVM was developed at AT&T Bell Laboratories
3
by Vapnik and co-workers in the early 1990s 10. Smola and
Schlkopf 11 acknowledge the success of SVMs since this time
and also add that in regression and time series prediction appli-
cations, excellent performances were soon obtained. The appli-
cation of the support vector approach to regression retains much
of the elegance of SVMs, but adds the capability to approximate
linear and nonlinear functions. The resulting support vector re-
gression SVR is showing promising empirical performance
12,13, and in this paper we seek to provide additional empirical
evidence by investigating the performance of SVR in comparison
to four metamodeling techniques commonly used in engineering
design: kriging, response surfaces, radial basis functions, and mul-
tivariate adaptive regression splines.
The remainder of this paper is as follows. An overview of the
four metamodeling techniques is given next. In Section 3, support
vector regression is discussed, beginning with linear function ap-
proximations and then nonlinear approximations. A simple one-
dimensional function approximation is presented for illustration
purposes in Section 3.3. A more thorough analysis is documented
in Section 4, wherein a test bed of 26 different engineering analy-
ses is used to compare the five metamodeling techniques, includ-

Journal of Mechanical Design Copyright 2005 by ASME NOVEMBER 2005, Vol. 127 / 1077

Downloaded From: http://mechanicaldesign.asmedigitalcollection.asme.org/ on 05/26/2015 Terms of Use: http://asme.org/terms

 ing SVR. In Section 4, the test bed is introduced, and the method
for comparing the techniques is explained, followed by a discus-
sion of the results. Section 5 contains closing remarks, and out-
lines future research in using SVR for function approximation.

2 Existing Metamodeling Techniques

This section briefly presents an overview of the four metamod-
eling techniques against which SVR is compared: i response
surface methodology, ii radial basis functions, iii kriging, and
iv multivariate adaptive regression splines. These techniques
were chosen based on their widespread use in engineering design
14 and availability of algorithms.
2.1 Response Surface Methodology. Response surface
methodology RSM approximates functions by using the least
squares method on a series of points in the design variable space. Fig. 1 Accounting for slack variables
Low-order polynomials are the most widely used response surface
approximating functions 14. Equation 4 is a first-order polyno-
mial that can be used for approximating functions with little to no tional relationship between the dependent and independent vari-
curvature. Equation 5 is a second-order polynomial that includes ables; instead, MARS constructs this relation from a set of coef-
quadratic terms and all two-factor interactions. ficients and basis functions that are determined from regression
data 30. The input space is divided into regions containing their
k
own regression equation; thus, MARS is suitable for problems
y = b0 + bx
i=1
i i 4 with high input dimensions, where the curse of dimensionality
would likely create problems for other techniques. For this work,
k k
we use the MARS algorithm developed by Chen et al. 31, Chen
32, and her colleagues.
y = b0 +
i=1
b ix i + b x +b xx
i=1
2
ii i
ij
ij i j 5
3 Support Vector Regression
The constants b0 , bi , bii , bij are determined by least squares re-
3.1 Linear Regression Using SVR. There are two basic aims
gression; more information can be found in Ref. 15. Response
in SVR. The first is to find a function fx that has at most
surfaces have become popular in mechanical design, having been
used in a variety of applications including compliant mechanism deviations from each of the targets of the training inputs. For the
design 16, robust design 17, multidisciplinary optimization linear case, f is given by
18, adaptive strategies for global optimization 19, manufactur- fx = w x + b 8
ing analysis 20, and game theoretic approaches to collaborative
design 21. where a b is the dot product between a and b.
At the same time, we would like this function to be as flat as
2.2 Radial Basis Functions. Radial basis functions RBF at- possible. Smola et al. 33 showed that choosing the flattest func-
tempt approximation by using a linear combination of radially tion in the feature space leads to a smooth function in the input
symmetric functions 22,23. Mathematically, the model can be space. Flatness in this sense means a small w in Eq. 8. This
expressed as second aim is not as immediately intuitive as the first, but is
nevertheless important in the formulation of the optimization
y = a X X
i
i 0i 6 problem used to construct the SVR approximation
1 2
where ai is a real-valued weight, and X0i is the input vector. Ra- Minimize w
dial basis functions have produced good approximations to arbi- 2
trary contours. For example, they have been successfully applied
to electronic circuit simulation models 24 and the construction
of metamodels for a desk lamp example 25.
subject to y i w xi b
w xi + b y i
9

2.3 Kriging. The kriging model postulates a combination of a
known function and departures of the form
yx = fx + Zx
where yx is the unknown function of interest, fx is a known
polynomial function, which is often taken as a constant, and Zx
is the correlation function, which is a realization of a stochastic
A key assumption in this formulation is that there exists a func-
tion fx that can approximate all input pairs xi , y i with preci-
sion; however, this may not be the case or perhaps some error
7
allowance is desired. Thus, the slack variables, i and *i , can be
incorporated into the optimization problem to yield the following
formulation 10:

process with mean zero, variance 2, and nonzero covariance 1 2
Minimize w + C i + i*
26. Flexibility in kriging is achieved through a variety of spatial 2 i=1
correlation functions 27, and in this work, we use a Gaussian

correlation function 26 and the fitting procedures described in y i w xi b + i
Ref. 28. Comparisons between kriging and response surfaces
can be found in Refs. 2,16,28, and a recent investigation of subject to w xi + b y i + i* 10
kriging models to support conceptual design can be found in Ref. i, i* 0
29.
where the constant C 0 determines the tradeoff between flatness
2.4 Multivariate Adaptive Regression Splines. Multivariate small w and the degree to which deviations larger than are
adaptive regression splines MARS is a nonparametric regression tolerated see Fig. 1, and is the number of samples. This is
procedure that makes no assumption about the underlying func- referred to as the -insensitive loss function proposed by Vapnik

1078 / Vol. 127, NOVEMBER 2005 Transactions of the ASME

Downloaded From: http://mechanicaldesign.asmedigitalcollection.asme.org/ on 05/26/2015 Terms of Use: http://asme.org/terms

 et al. 12, which enables a sparse set of support vectors to be Table 1 Common kernel functions 34
obtained for regression.
The optimization function and linear constraints in Eq. 10 can
be written as the Lagrangian function

+ y + w x + b
1
L = w2 + C i + i* i i i i
2 i=1 i=1

i=1
i* + i* + y i w xi b +
i=1
i i
* *
i i 11

Through Lagrangian theory, necessary conditions for to be a

solution to the original optimization problem are

bL =
i=1
*
i i = 0 12
nonlinear transformation on the input vectors. This transformation
is referred to as the kernel function and is represented by kx , x,
wL = w
i=1
*
i ixi = 0 13 where x and x are each input vectors. Table 1 lists common
kernel functions where the kernel function substitution maintains
the elegance of the optimization method used for linear SVR 34.
Applying the kernel function to the dot product of input vec-
iL = C i i = 0 14
tors, we obtain

*L = C i* i* = 0 15

1
i i i* j *j kxi,x j
2 i,j=1
Substituting Eqs. 1215 into Eq. 11 yields the optimization Maximize
problem in dual form
+ + y * *

i i i i i

1 i=1 i=1
i i* j *j xi x j

2 i,j=1
Maximize
= 0 i
*

i
i + i* + y ii i* Subject to i=1 19
i=1 i=1 i, i* 0,C
Replacing the dot product in Eq. 18, the SVR approximation

subject to

i=1
i i* =0
16
becomes

i i*
From Eq. 13,
w=
i=1
*
i ixi
and so the linear regression in Eq. 8 becomes
fx = x x + b
i=1
i
*
i i
Thus, the training algorithm and the regression function fx can
be expressed in terms of the dot product xi x.
Transforming the optimization problem into dual form yields
two advantages. First, the optimization problem is now a qua-
dratic programming problem with linear constraints and a positive
definite Hessian matrix, ensuring a unique global optimum. For
such problems, highly efficient and thoroughly tested quadratic
solvers exist. Second, as can be seen in Eq. 16, the input vectors
only appear inside the dot product. The dot product of each pair of
input vectors is a scalar and can be preprocessed and stored in the
quadratic matrix M ij = xix jij. In this way, the dimensionality of
the input space is hidden from the remaining computations, pro-
viding means for addressing the curse of dimensionality 13.
0,C fx = kx ,x + b
i=1
i
*
i i 20
The kernel function kxi , x can be preprocessed, and the results
are stored in the kernel matrix, K = kxi , xji,j=1
n
. The kernel ma-
17 trix must be positive definite in order to guarantee a unique opti-
mal solution to the quadratic optimization problem. The kernel
functions presented in Table 1 yield positive definite kernel ma-
trices 4. Thus, by using the kernel function and corresponding
kernel matrix, nonlinear function approximations can be achieved
18 with SVR while maintaining the simplicity and computational ef-
ficiency of linear SVR approximations.
Now that the theory behind SVR has been reviewed, we can
begin to investigate the application of SVR to engineering analy-
ses. First, a simple one-dimensional example is presented next for
illustration purposes, and then a test bed of more realistic engi-
neering problems is discussed in Section 4.
3.3 A Simple One-Dimensional Example. This section illus-
trates the application of SVR to a one-dimensional example,
which means that the length of each input vector is only one. The
function to be approximated comes from Su and Renaud 35 and
is shown in Fig. 2. The five training points noted in the figure are
used to fit the SVR.
The eighth-order function shown in Fig. 2 is given by
9

3.2 Nonlinear Regression Using SVR. Another benefit of the fx = a x 900

i=1
i i
i1
21
dual form is that nonlinear function approximations can be
achieved by replacing the dot product of input vectors with a where

Journal of Mechanical Design NOVEMBER 2005, Vol. 127 / 1079

Downloaded From: http://mechanicaldesign.asmedigitalcollection.asme.org/ on 05/26/2015 Terms of Use: http://asme.org/terms

 Fig. 2 Eighth-order one-dimensional function

a1 = 659.23, a2 = 190.22, a3 = 17.802, a4 = 0.82691

a5 = 0.021885, a6 = 0.0003463, a7 = 3.2446 106

a8 = 1.6606 108, a9 = 3.5757 1011

The Matlab code developed by Gunn 13 was used to execute the

SVR algorithm. Figure 3 shows a flowchart of the implementa-
tion, which is discussed as follows.
As seen in Fig. 3, the kernel matrix is first calculated from the
training points. Using the Gaussian kernel function presented in
Table 1 for each combination of input vectors, the following ker-
nel matrix is obtained:

1.0000 0.8749 0.5858 0.3003 0.1178
0.8749 1.0000 0.8749 0.5858 0.3003
K = 0.5858 0.8749 1.0000 0.8749 0.5858
0.3003 0.5858 0.8749 1.0000 0.8749
0.1178 0.3003 0.5858 0.8749 1.0000
Fig. 3 Flowchart of the SVR algorithm
This matrix is used in the quadratic optimization problem stated
in Eq. 19. The resulting solution yields all the variables required
to calculate the approximating function. The vector of differences
of Lagrange multipliers is

433.8411
924.0233
i* i = 999.9999
647.1798
229.4055
The difference of Lagrange multipliers is then used with the

training points to calculate the weight vector, w = i=1 *i ixi,
and the offset b. The offset b is calculated using Karush-Kuhn-
Tucker conditions see Ref. 36 for more details, and all of these
variables are then substituted into Eq. 20 to yield the SVR
approximation.
The default value of 1 104 was used for , and the radius of
the Gaussian kernel was manually optimized to a value of 9.7 for
the given training data. This involved continually updating the
Gaussian radius and running the SVR algorithm until the root
mean square error RMSE was close to minimum. Figure 4
shows the SVR function approximation compared to the actual Fig. 4 Fit of one-dimensional function using SVR

1080 / Vol. 127, NOVEMBER 2005 Transactions of the ASME

Downloaded From: http://mechanicaldesign.asmedigitalcollection.asme.org/ on 05/26/2015 Terms of Use: http://asme.org/terms

 Table 2 Error measures for accuracy assessment
function. We can see a very close fit between the SVR approxi-
mation and the actual function within the range of the training
data, but the fit starts to deteriorate outside of this range, as one
might expect.
To assess the accuracy of these results, the three error equations
in Table 2 are calculated, where nerror is the number of random test
points used, y i is the actual function value and y i is the predicted
value from the function approximation method. For comparison,
the response surface and kriging models constructed in Ref. 37
are included in this assessment. The resulting errors for SVR, the
second-order RSM model, and the kriging model are listed in
Table 3; these errors are based on 16 evenly spaced test points
between x = 920 and x = 945. The SVR method has achieved the
lowest error values in all three categories; hence, SVR provides a
very accurate approximation for this one-dimensional example.
4 Approximation Test Bed and Comparison
The test bed of engineering analyses used to benchmark SVR
against other approximation methods is listed in the Appendix.
This test bed was initially created in Ref. 37 to compare the
predictive capability of kriging models. The 26 functions to be
approximated are derived from six engineering problems typically
used to test optimization algorithms: a two-bar truss 38, a three-
bar truss 38, a two-member frame 39, a helical compression cation of how accurate the approximation was overall, while the
spring 40, a welded beam 41, and a pressure vessel 40. maximum error can reveal the presence of regional areas of poor
4.1 Training and Testing Procedure. The procedure used to
train and test each approximation technique is summarized as
follows.
Step 1Generate training data and test data. To minimize
interactions between metamodel type and the choice of experi-
mental design, six different sets of training data were generated
for each function using six different experimental designs as listed
in Table 4. A review and detailed comparison of these experimen-
tal designs can be found in Ref. 42. The number of training
points generated for each problem n1 is also shown. Since there
are six sets of training data for each of the 26 functions, a total of
6 26= 156 function approximations are constructed using each
approximation technique.
Additional test data is generated as shown in Table 4 using a
random Latin hypercube design 4346. The number of test
points is very large to ensure a thorough analysis of the accuracy
of the resulting approximation throughout the performance space.
Table 3 Error comparisons between approximation methods
for one-dimensional example
Journal of Mechanical Design
Downloaded From: http://mechanicaldesign.asmedigitalcollection.asme.org/ on 05/26/2015 Terms of Use: http://asme.org/terms
Table 4 Generation of training data and test data
Step 2Construct metamodels using training data. Every
training data set is submitted to each metamodeling algorithm to
generate a function approximation. For SVR, Gunns 13 support
vector machine Matlab code is implemented as in the one-
dimensional example see Fig. 3. The Gaussian kernel function
was selected due to its good features and strong learning capa-
bility 47, and the radius of the Gaussian kernel function is
manually optimized for each function for each data set. The de-
fault value of 1 104 was used for as we found that this pa-
rameter had little impact on the resulting SVR model. For kriging,
RSM, RBF, and MARS, Fortran algorithms from Ref. 37 were
utilized.
Step 3Compare the accuracy and robustness of each
metamodel. Additional test data is used to test the accuracy of the
function approximation generated by each of the five metamodel-
ing techniques. For a given function and a given metamodel, the
inputs of the test data are submitted to the function approximation
generated in Step 2. The outputs are the predicted values of the
original function according to the corresponding metamodeling
technique. The difference between each predicted value y i and the
actual function value y i is calculated as the error for that test point.
As in the one-dimensional example, the three error measures
listed in Table 2 are calculated to assess accuracy: i RMSE, ii
maximum error, and iii average error. The RMSE gives an indi-
approximation; however, due to the different magnitudes of the 26
test functions, these error statistics enable comparisons of algo-
rithms only within each function. It is desirable to compare the
effectiveness of each algorithm across all of the functions so that
conclusions can be drawn; however, normalizing the errors
against their nominal values does not provide a realistic compari-
son since many nominal values are close to zero, which exacer-
bates the relative value of the error. A common normalized statis-
tic is the correlation coefficient R2, but it is only suited for linear
approximations 48. Although correlation coefficients have been
devised for nonlinear approximations see Ref. 48, they apply
only to simple functions and not to the complex equations in the
test bed. Thus, to enable comparisons of SVR to the four other
metamodeling techniques, the average percentage differences in
error values between each algorithm and SVR is computed. In this
manner, the error in the SVR approximations becomes the bench-
mark to which all other metamodels are compared. Positive per-
centage values imply that the corresponding approximation had
larger errors than SVR, while negative values imply that the ap-
proximation had smaller errors. These results are presented and
discussed in Section 4.2.1.
The robustness of an approximation method is indicated by the
variance between its error values across different sample sets 2.
The capability to continually repeat a function approximation with
similar accuracies i.e., similar errors increases the reliability of
the results and the robustness of the approximation method. To
test for robustness, SVR is again used as a benchmark to examine
the robustness of each approximation technique, and the standard
NOVEMBER 2005, Vol. 127 / 1081
 Fig. 5 Comparison of errors between metamodeling techniques relative to SVR
deviation is computed to indicate robustness using the following
steps:
1. For a given approximation technique, find the standard de-
viation of each error i.e., RMSE, MAE, AAE across the six
different training sets for each function.
2. Normalize each standard deviation against the standard de-
viation for SVR for the same function. This normalized stan-
dard deviation NormStdDev is calculated as:
Norm Std Dev
Std. Dev. for Given Technique Std. Dev. for SVR
=
Std. Dev. for SVR
22
3. Average the NormStdDev across all functions for a given
metamodel.
The normalized standard deviation reflects the variance in the er-
ror for a given approximation technique, relative to SVR. A posi-
tive value indicates a greater variance than SVR and hence less
robustness, while a negative value indicates a lower variance than
SVR and a more robust approximation method. These results are
presented and discussed in Section 4.2.2.
Step 4Compare the computational efficiency and trans-
parency of each metamodeling technique. Following Jin et al.
2, the computational efficiency refers to the computational effort
required to construct the metamodel and predict response values
for a set of new points using the metamodel, and the transparency
is the capability of providing information concerning contribu-
tions of different variables and interactions among variables. The
five metamodeling techniques are compared on these two aspects
in Section 4.2.3.
4.2 Analysis of Results.
4.2.1 Accuracy Results. Figure 5 shows the percentages by
which average error values were higher than the corresponding
error value for SVR. These percentage differences have been av-
eraged over all 26 approximated functions. Except for the overall
maximum error for kriging, SVR has achieved lower average er-
ror values i.e., RMSE, average error, and maximum error com-
1082 / Vol. 127, NOVEMBER 2005
Downloaded From: http://mechanicaldesign.asmedigitalcollection.asme.org/ on 05/26/2015 Terms of Use: http://asme.org/terms
pared to the four other approximation techniques. As previously
mentioned, lower overall RMSE values represent good global
function approximation across the performance space, while lower
overall maximum error values reflect the absence of poorly ap-
proximated regions in the performance space. Thus, the results
imply that kriging models avoid areas of poor approximation, but
they do not perform as well globally as the SVR approximations.
In general, SVR has outperformed the four other approximation
techniques, giving lower overall error values. Kriging achieved
the next best overall performance, followed by MARS, RSM, and
finally RBF. These trends are consistent with those observed in
Ref. 37, but we note that RBF has performed much better in
other studies 2,25; we are still investigating why this has
occurred.
To further compare the performance of each algorithm accord-
ing to the type of function being modeled, the three error statistics
for linear and nonlinear function approximations were considered
separately. Table 5 shows the number of times each metamodel
type performed the best i.e., achieved the lowest corresponding
error statistic for nonlinear and linear function approximations.
SVR achieved the highest frequency of best performances for the
nonlinear functions by a significant margin, whereas kriging out-
performs more often for the linear functions by a smaller margin.
These results are further confirmed in Figs. 6 and 7, which show
Table 5 Frequency in which each metamodel performed best
for each error measure
Transactions of the ASME
 Fig. 6 Comparison of errors for linear functions relative to SVR

the comparison of errors relative to SVR for the linear and non-
linear functions, respectively.
This separation into linear and nonlinear functions reveals the
stronger performance of kriging in approximating linear functions,
especially in the maximum error category, indicating a good gen-
eral performance over the entire input space. However, the perfor-
mance of RSM for linear functions is not a significant improve-
ment over its performance with nonlinear functions, as would be
expected. It has been noticed that the errors produced by RSM for
the linear functions within the spring test bed problem were par-
ticularly large, which greatly influenced this poor result when the
errors were averaged. Further investigation is needed to under-
stand why RSM did not perform as well as expected for the linear
function approximations.
4.2.2 Robustness Results. Figure 8 presents the normalized
standard deviations of the errors for each approximation tech-
nique, relative to SVR. Hence, the results indicate that SVR is the
most robust of the five approximation techniques, reinforcing the
validity of the error values obtained in Fig. 5. Kriging is the sec-
ond most robust metamodeling technique. The variance between
errors for RSM is very large, when measured relative to SVR,

Fig. 7 Comparison of errors for nonlinear functions relative to SVR

Journal of Mechanical Design NOVEMBER 2005, Vol. 127 / 1083

Downloaded From: http://mechanicaldesign.asmedigitalcollection.asme.org/ on 05/26/2015 Terms of Use: http://asme.org/terms

 Fig. 8 Comparison of normalized standard deviations of errors between metamodels relative to SVR
indicating very inconsistent performance. This could be reflective
of the fact that RSM is most suitable for linear approximations
only, performing well in these cases and poorly in nonlinear cases.
The relatively high robustness of SVR is followed by kriging,
MARS, RBF, and finally RSM.
4.2.3 Computational Efficiency and Transparency. We end our
analysis with a comparison of the computational efficiency and
transparency of the five metamodeling techniques. The time to fit
an SVR approximation was on the order of a few seconds using
Gunns 13 Matlab code running on a 1.8 GHz desktop PC, even
with the optimization process, and prediction with a fitted SVR
approximation took less than a second. The fitting and prediction
times for SVR were very comparable to that for MARS, while
RSM and RBF were much faster, taking less than a second each.
The least computationally efficient were the kriging models,
which could take several minutes to fit depending on the dimen-
sionality of the problem and number of samplesfortunately, pre-
diction with the kriging models took less than a second each. We
refer the reader to Ref. 2 for a more detailed investigation of the
effect of problem size and sample size on the computational effi-
ciency of kriging, RSM, RBF, and MARS.
RSM provides the most transparency in terms of the function
relationship and the factor contributions, see Eqs. 4 and 5.
MARS provides some transparency since the MARS models can
be recast into the form 30
y = a0 + f x + f
i i ij xi,x j + f ijkxi,x j ,xk
Km=1 Km=2 Km=3
where the first sum is over all basis functions that involve only a
single variable; the second is over all basis functions that involve
two variables, representing two-variable interactions if present;
the third is over three variables, etc. RBF and SVR have explicit
function equations, Eqs. 6 and 20, respectively, but the indi-
vidual factor contributions are not clear, making them even less
1084 / Vol. 127, NOVEMBER 2005
Downloaded From: http://mechanicaldesign.asmedigitalcollection.asme.org/ on 05/26/2015 Terms of Use: http://asme.org/terms
transparent. The kriging models are the least transparent; however,
the theta values can be interpreted with some practice 2,27,49.
5 Conclusions and Future Work
In conclusion, the theory behind SVR has been presented and
shown to possess the desirable qualities of mathematical and al-
gorithmic elegance, producing an actual approximating function
as opposed to a trained black box. In comparison to four common
approximating techniques, SVR had the best overall performance
for the test bed of 26 engineering analysis functions. Only kriging
outperformed SVR in the category of average lowest maximum
error. The strong performance of the SVR approximations was
reinforced through relatively small variances between error val-
ues, indicating that SVR also yields a more robust approximation.
SVR provides a good compromise between prediction accuracy
and robustness of a kriging model, with the computational effi-
ciency and transparency near that of a RSM or RBF approxima-
tion. These results add to the growing body of promising empiri-
cal performance of SVR, and we are currently investigating the
theoretical foundations of all five metamodeling techniques in
more detail to better understand where SVR obtains its strength.
The SVR implementation employed produced successful re-
sults; however, better results using SVR are anticipated through
increased attention to the SVR algorithm, itself, and the model
parameters selected. The Matlab algorithm with which the SVR
results were obtained is not as efficient as other available algo-
+ rithms, such as SVMlight and mySVM http://www-ai.cs.uni-
dortmund.de/, which are available in C and more efficient than
23 the relatively slower Matlab implementation; however, users are
limited in the amount of tweaking they can do to the software. A
better solution would be to code an entire SVR algorithm in C and
employ any of a number of commercially available quadratic
solvers to give users more flexibility in fitting and using SVR.
In addition, the SVR implemented in our experiments consis-
tently used a Gaussian kernel function that was manually opti-
mized for each set of training data. Automatically optimizing the
Transactions of the ASME
 radius of the Gaussian kernel function during training could fur-
ther improve the results. For example, data suspected to be poly-
nomial will probably be modeled more accurately with the use of
a polynomial kernel function instead of a Gaussian one; however,
the optimal choice for the kernel function is still an area of active
research and will be investigated in future work. Theoretically,
any symmetric function that results in a positive definite kernel
matrix can be used. The radius of the Gaussian kernel function
was also manually optimized in this study for each function.
Adopting a method to automatically optimize this radius such as
Appendix: Approximation Test Bed for Comparing Metamodeling Techniques
Journal of Mechanical Design
Downloaded From: http://mechanicaldesign.asmedigitalcollection.asme.org/ on 05/26/2015 Terms of Use: http://asme.org/terms
the simulated annealing algorithm in kriging 28 could further
improve the results. Alternatively, cross-validation methods for
fitting the SVR approximation could be investigated.
Acknowledgments
We gratefully acknowledge support from the U.S. Department
of Transportation and the Federal Transit Administration. This
work was conducted as part of the development of a web-based
information management system for storing, displaying, and ana-
lyzing bus test data.
NOVEMBER 2005, Vol. 127 / 1085
 References
1 Kleijnen, J. P. C., 1987, Statistical Tools for Simulation Practitioners, Marcel
Dekker, New York.
2 Jin, R., Chen, W., and Simpson, T. W., 2001, Comparative Studies of Meta-
modeling Techniques Under Multiple Modeling Criteria, Struct. Multidiscip.
Optim., 231, pp. 113.
3 Haykin, S., 1999, Neural Networks: A Comprehensive Foundation, 2nd Edi-
tion, Prentice Hall, Upper Saddle River, NJ.
4 Cristianni, N., and Shawe-Taylor, J., 2000, An Introduction to Support Vector
Machines and Other Kernel-Based Learning Methods, Cambridge University
Press, Cambridge, UK.
5 Hearst, M. A., 1998, Trends Controversies: Support Vector Machines, IEEE
Intell. Syst., 134, pp. 1828.
6 Takeuchi, K., and Collier, N., 2002, Use of Support Vector Machines in
Extended Named Entity, Proc. of Sixth Conference on Natural Language
Learning (CoNLL-2002), D. Roth and A. van den Bosch, eds., Taipei, Taiwan,
Association for Computational Linguistics, New Brunswick, NJ, pp. 119125.
7 Dumais, S. T., Platt, J., Heckerman, D., and Saharni, M., 1998, Inductive
Learning Algorithms and Representations for Text Categorization, Proc. of
7th Int. Conference on Information and Knowledge Management, Bethesda,
MD, ACM, New York, pp. 148155.
8 Prakasvudhisarn, C., Trafalis, T. B., and Raman, S., 2003, Support Vector
Regression for Determination of Minimum Zone, ASME J. Manuf. Sci. Eng.,
1254, pp. 736739.
9 Vapnik, V., and Lerner, A., 1963, Pattern Recognition Using Generalized
Portrait Method, Autom. Remote Control Engl. Transl., 246, pp. 774
780.
10 Vapnik, V., 1995, The Nature of Statistical Learning Theory, Springer, New
York.
11 Smola, A. J., and Schlkopf, B., 1998, A Tutorial on Support Vector Regres-
sion, NeuroCOLT2 Technical Report Series, NC2-TR-1998-030, Berlin, Ger-
many.
12 Vapnik, V., Golowich, S., and Smola, A., 1997, Support Vector Method for
Function Approximation, Regression Estimation, and Signal Processing, Ad-
vances in Neural Information Processing Systems, M. Mozer, M. Jordan, and
T. Petsche, eds., MIT Press, Cambridge, MA, pp. 281287.
13 Gunn, S. R., 1997, Support Vector Machines for Classification and Regres-
sion, Technical Report, Image Speech and Intelligent Systems Research
Group, University of Southampton, UK.
14 Simpson, T. W., Peplinski, J., Koch, P. N., and Allen, J. K., 2001, Metamod-
els for Computer-Based Engineering Design: Survey and Recommendations,
Eng. Comput., 172, pp. 129150.
15 Myers, R. H., and Montgomery, D. C., 1995, Response Surface Methodology:
Process and Product Optimization Using Designed Experiments, Wiley, New
York.
16 Cappelleri, D. J., Frecker, M. I., Simpson, T. W., and Snyder, A., 2002, A
Metamodel-Based Approach for Optimal Design of a PZT Bimorph Actuator
for Minimally Invasive Surgery, J. Mech. Des., 1242, pp. 354357.
17 Chen, W., Allen, J. K., Tsui, K.-L., and Mistree, F., 1996, A Procedure for
Robust Design: Minimizing Variations Caused by Noise and Control Factors,
J. Mech. Des., 1184, pp. 478485.
1086 / Vol. 127, NOVEMBER 2005
Downloaded From: http://mechanicaldesign.asmedigitalcollection.asme.org/ on 05/26/2015 Terms of Use: http://asme.org/terms
18 Korngold, J. C., and Gabriele, G. A., 1997, Multidisciplinary Analysis and
Optimization of Discrete Problems Using Response Surface Methods, J.
Mech. Des., 1194, pp. 427433.
19 Wang, G., 2003, Adaptive Response Surface Method Using Inherited Latin
Hypercube Designs, J. Mech. Des., 1252, pp. 210220.
20 Hernandez, G., Simpson, T. W., Allen, J. K., Bascaran, E., Avila, L. F., and
Salinas, F., 2001, Robust Design of Families of Products With Production
Modeling and Evaluation, J. Mech. Des., 1232, pp. 183190.
21 Lewis, K., and Mistree, F., 1998, Collaborative, Sequential, and Isolated De-
cisions in Design, J. Mech. Des., 1204, pp. 643652.
22 Dyn, N., Levin, D., and Rippa, S., 1986, Numerical Procedures for Surface
Fitting of Scattered Data by Radial Basis Functions, SIAM Soc. Ind. Appl.
Math. J. Sci. Stat. Comput., 72, pp. 639659.
23 Powell, M. J. D., 1987, Radial Basis Functions for Multivariable Interpola-
tion: A Review, Algorithms for Approximation, J. C. Mason and M. G. Cox,
eds., Oxford University Press, London, pp. 143167.
24 Tu, C., and Barton, R. R., 1997, Production Yield Estimation by the Meta-
model Method with a Boundary-Focused Experiment Design, ASME Design
Engineering Technical Conferences-Design Theory and Methodology, Sacra-
mento, CA, ASME, Paper No. DETC97/DTM-3870.
25 Meckesheimer, M., Barton, R. R., Simpson, T. W., Limayem, F., and Yannou,
B., 2001, Metamodeling of Combined Discrete/Continuous Responses,
AIAA J., 3910, pp. 19551959.
26 Sacks, J., Welch, W. J., Mitchell, T. J., and Wynn, H. P., 1989, Design and
Analysis of Computer Experiments, Stat. Sci., 44, pp. 409435.
27 Koehler, J. R., and Owen, A. B., 1996, Computer Experiments, Handbook of
Statistics, S. Ghosh and C. R. Rao, eds., Elsevier Science, New York, pp.
261308.
28 Simpson, T. W., Mauery, T. M., Korte, J. J., and Mistree, F., 2001, Kriging
Metamodels for Global Approximation in Simulation-Based Multidisciplinary
Design Optimization, AIAA J., 3912, pp. 22332241.
29 Pacheco, J. E., Amon, C. H., and Finger, S., 2003, Bayesian Surrogates Ap-
plied to Conceptual Stages of the Engineering Design Process, J. Mech. Des.,
1254, pp. 664672.
30 Friedman, J. H., 1991, Multivariate Adaptive Regression Splines, Ann. Stat.,
191, pp. 167.
31 Chen, V. C. P., Ruppert, D., and Shoemaker, C. A., 1999, Applying Experi-
mental Design and Regression Splines to High-Dimensional Continuous-State
Stochastic Dynamic Programming, Oper. Res., 47, pp. 3853.
32 Chen, V. C. P., 1999, Application of MARS and Orthogonal Arrays to Inven-
tory Forecasting Stochastic Dynamic Programs, Comput. Stat. Data Anal.,
30, pp. 317341.
33 Smola, A. J., Schlkopf, B., and Mller, K. R., 1998, The Connection Be-
tween Regularization Operators and Support Vector Kernels, Neural Net-
works, 114, pp. 637649.
34 Schlkopf, B., and Smola, A. J., 2002, Learning with Kernels: Support Vector
Machines, Regularization, Optimization, and Beyond, MIT Press, Cambridge,
MA.
35 Su, J., and Renaud, J. E., 1997, Automatic Differentiation in Robust Optimi-
zation, AIAA J., 356, pp. 10721079.
36 Markowetz, F., 2001, Support Vector Machines in Bioinformatics, Diploma
Thesis in Mathematics, University of Heidelberg, Germany.
Transactions of the ASME
 37 Simpson, T. W., 1998, A Concept Exploration Method for Product Family
Design, Ph.D. Dissertation, G.W. Woodruff School of Mechanical Engineer-
ing, Georgia Institute of Technology, Atlanta, GA.
38 Schmit, L. A., 1981, Structural SynthesisIts Genesis and Development,
AIAA J., 1910, pp. 12491263.
39 Arora, J. S., 1989, Introduction to Optimum Design, McGraw-Hill, New York.
40 Sandgren, E., 1990, Nonlinear Integer and Discrete Programming in Me-
chanical Design Optimization, J. Mech. Des., 1122, pp. 223229.
41 Ragsdell, K. M., and Phillips, D. T., 1976, Optimal Design of a Class of
Welded Structures Using Geometric Programming, ASME J. Eng. Ind.,
983, pp. 10211025.
42 Simpson, T. W., Lin, D. K. J., and Chen, W., 2001, Sampling Strategies for
Computer Experiments: Design and Analysis, Int. J. Reliab. Appl., 23, pp.
209240.
43 McKay, M. D., Beckman, R. J., and Conover, W. J., 1979, A Comparison of
Three Methods for Selecting Values of Input Variables in the Analysis of
Output From a Computer Code, Technometrics, 212, pp. 239245.
Journal of Mechanical Design
Downloaded From: http://mechanicaldesign.asmedigitalcollection.asme.org/ on 05/26/2015 Terms of Use: http://asme.org/terms
44 Kalagnanam, J. R., and Diwekar, U. M., 1997, An Efficient Sampling Tech-
nique for Off-Line Quality Control, Technometrics, 393, pp. 308319.
45 Tang, B., 1993, Orthogonal Array-Based Latin Hypercubes, J. Am. Stat.
Assoc., 88424, pp. 13921397.
46 Fang, K.-T., Lin, D. K. J., Winker, P., and Zhang, Y., 2000, Uniform Design:
Theory and Application, Technometrics, 42, pp. 237248.
47 Wang, W. J., Xu, Z. B., Lu, W. Z., and Zhang, X. Y., 2003, Determination of
the Spread Parameter in the Gaussian Kernel for Classification and Regres-
sion, Neurocomputing, 551, pp. 643663.
48 Cameron, A. C., and Windmeijer, F. A. G., 1997, An R-Squared Measure of
Goodness of Fit for Some Common Nonlinear Regression Models, J.
Econometr., 772, pp. 329342.
49 Martin, J. D., and Simpson, T. W., 2004, On the Use of Kriging Models to
Approximate Deterministic Computer Models, ASME Design Engineering
Technical Conferences-Design Automation Conference, Salt Lake City, W.
Chen, ed., ASME, New York, ASME Paper No. DETC2004/DAC-57300.
NOVEMBER 2005, Vol. 127 / 1087