10th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing

A comparative landscape analysis of tness functions for search-based testing

Raluca Lefticaru, Florentin Ipate Department of Computer Science and Mathematics University of Pitesti Str. Targu din Vale 1, 110040 Pitesti, Romania raluca.lefticaru@gmail.com, orentin.ipate@ifsoft.ro

AbstractLandscape analysis of tness functions is an important topic. This paper makes an attempt to characterize the search problems associated with the tness functions used in search-based testing, employing the following measures: diameter, autocorrelation and tness distance correlation. In a previous work, a general form of objective functions for structural search-based software testing was tailored for state-based testing. A comparison is performed in this paper between the general tness functions and some problem-specic tness functions, taking into account their performance with different search methods. Keywords: search-based testing, nite state machines, metaheuristic search techniques, tness functions, landscape analysis.

I. I NTRODUCTION Software testing is a very expensive, tedious and time consuming task, which was estimated to require around 50% of the total cost of software development. Therefore, different techniques have been employed to automate test generation, among them random testing [1], symbolic execution [2], domain reduction [3] and constraint-based testing [4]. An approach with a great potential for the automatic generation of test data is to model the testing task as a search problem, which can be solved by applying search techniques like genetic algorithms, simulated annealing, particle swarm optimization or tabu search. Search-based testing is characterized by the usage of search techniques for test generation. Whatever the test aim is, it is rst transformed into an optimization problem with respect to some tness (cost or objective) function. The search space, in which the test data that fulls a given test aim is searched for, is the input domain of the test object (program, function). The search spaces obtained are usually complex, discontinuous, and non-linear, due to the non-linearity of software (if-statements, loops etc.) [5]. Therefore the use of neighborhood search methods, such as hill climbing, is not recommended; instead, metaheuristic search methods are employed [5]. Many papers have been published, which present applications of metaheuristic algorithms for test data generation. Consequently, a comprehensive survey [6] on the search-based test data generation followed them. Most studies have concentrated on the application of such techniques in structural (program-based or white-box) testing [7][9]. In structural testing, the program is represented as a

directed graph, in which each node corresponds to a statement or a sequence of statements and each edge (branch) to a transfer of control between two statements. Search-based techniques are then used to generate test data to cover the desired graph elements (nodes, branches or paths). Functional search-based testing has been less investigated. Some papers have concentrated on generating test data from Z specications [10], [11]. This idea of using the functional specication of a program was extended to conformance testing [11], an objective function of the form pre-condition post-condition, that measures the closeness of the test data to uncovering a fault in the implementation was employed. The application of evolutionary functional testing to some industrial software (an automated parking system) is presented in [12], [13]. Other approaches tackle the problem of generating realistic signals and propose an innovative solution, by building the overall signal from a series of simple signal types, for example sine, spline and linear curves [14]. Evolutionary methods for testing state-based programs have been studied, mainly from a structural testing point of view, regarding the ags (loop-assigned ags), enumerations and counters used to manage the internal state of the objects [15], [16]. In this paper we discuss the use of search based-methods for testing from state-based specications. A general tness function was tailored in [17] to nd input parameters for a given path in a state machine, which satisfy some given constraints for each transition, and preliminary experiments showed that its usage with genetic algorithms and particle swarm optimization is promising [17][19]. The use of this general tness function, based on state machines, is further investigated in this paper. Furthermore, the investigation is also extended to other search techniques: simulated annealing and particle swarm optimization. Experimental results show that this general tness function may produce results comparable to those produced by tness functions designed especially for a particular situation, which assume a much richer knowledge of the problem in hand. Metrics that help the assessment of the tness functions are also presented. The paper is structured as follows. Section II provides a brief overview of metaheuristic search techniques and section III some measures used for the landscape characterization of the search dynamics. Section IV presents the tness functions

978-0-7695-3523-4/08 $25.00 2008 IEEE DOI 10.1109/SYNASC.2008.69

201

for search-based testing from state machines, some experimental results are resumed in section V and the proposed approach is compared with previous work in section VI. Finally, conclusions and further work and are drawn in section VII.

III. M EASURES TO C HARACTERIZE S EARCH P ROBLEMS This section presents some measures used for characterizing the search problems and the metaheuristics applied to solve them. Let S be a solution space, f a cost (objective or tness) function assigning a real number to each point in S and N a neighbourhood operator connecting each point in S to a nonempty set of neighbours. The diameter D of a search landscape is the maximal distance between two points in S, in terms of successive applications of N . Intuitively, small diameters should be more favorable than large ones, because any point can potentially be reached quicker [21]. For a successful SA process, the diameter D should be signicantly lower than M AX IT ER, the maximum number of iterations per run. This criterion is intended to ensure that an optimal solution has a chance to be reached before the end of the run, whatever the starting point of the search is [21]. The autocorrelation d of a series measures the variation of tness for points that are at distance d. The most important value is 1 and it is estimated by means of a random walk, whose steps consist in moving to a new point chosen randomly among the neighbors of the current point (for example, using N , the neighbourhood operator of a SA process, as in Fig. 1). Select a starting point x1 f1 = f (x1 ) FOR i = 1 to n-1 DO Randomly select xi+1 N (xi ) fi+1 = f (xi+1 ) END FOR
Fig. 1. Random walk algorithm

II. M ETAHEURISTIC S EARCH T ECHNIQUES Simulated Annealing (SA) is a local search technique, proposed by Kirkpatrick et al. in 1983 [20]. In the original algorithm, the optimization starts with an initial candidate solution; a random generated neighbour replaces the current solution whether it has a better objective value, or it has a worse objective value, with a probability that depends on the difference between the corresponding function values and a global parameter t (temperature). The dependency on the current temperature t is such that the current solution changes almost randomly when t is large, but increasingly downhill as t goes to zero. The probability of acceptance of a worse candidate is higher at the beginning of the algorithm, when the temperature t is high and it is gradually decreased according to a cooling schedule that controls the parameter t. Genetic Algorithms (GA) are global search heuristics, that use techniques inspired from biology, such as selection, recombination (crossover) and mutation, applied on a population of potential solutions, called chromosomes (or individuals). GA are a particular class of evolutionary algorithms; they are closely related to Evolutionary Strategies (ES), the main difference being that for GA recombination is the main operator that drives the search, whereas ES principally use mutation. High level descriptions of these algorithms can be found in [6], [20], [21]. Particle Swarm Optimization (PSO) is a relatively recent search technique introduced by Eberhart and Kennedy in 1995, inspired by social metaphors of behavior and swarm theory. The system is initialized with a population of random solutions (particles) and searches for optimum by updating generations. Each particle maintains its current position, velocity and best position explored so far. It ies through the problem space by following the current optimum particles, with a velocity, dynamically adjusted according to its own previous best position and also to the previous best position of its companions [22]. Unlike in GA, in PSO the selection and crossover operators are not employed: all the particles are kept as members of the population. In the basic PSO algorithm, the particles are manipulated according to the following equations [22]: vid = wvid +c1 rand()(pid xid )+ c2 Rand()(pgd xid ) and xid = xid + vid , where: Xi = (xi1 , xi2 , . . . , xiD ) represents the ith particle, Vi = (vi1 , vi2 , . . . , viD ) the rate of change (velocity) for the ith particle and Pi = (pi1 , pi2 , . . . , piD ) its best previous position. The index of the best particle among all the population is g, c1 , c2 > 0 are the acceleration coefcients, w the initial inertia weight and rand(), Rand() random functions in the range [0, 1].

An estimation of 1 is: 1 =
n1 i=1 (fi f )(fi+1 n 2 i=1 (fi f )

f)

where n is the number of samples, fi the tness values and f their mean [23]. If |1 | is close to one, the landscape is smooth. If |1 | is close to zero, the neighboring cost values are unrelated and neighborhood search techniques are not expected to be effective. Another measure, proposed for GA, which indicates the problem hardness by characterizing the joint variation of distances and objective values is the tness distance correlation (FDC). Given a set F = {f1 , f2 , . . . , fn } of n individual tness values and a corresponding set D = {d1 , d2 , . . . , dn } of the n distances to the nearest global optimum, the correlation coefcient r is: cF D r= sF sD where cF D = 1 n
n

(fi f )(di d)
i=1

202

is the covariance of F and D, and sF , sD , f and d are the standard deviations and means of F and D respectively [24]. The FDC denition involves the notion of distance to the nearest global optimum and consequently this restricts its applicability to problems with known optima. The ideal tness function will have r = 1 when minimizing (the tness decreases as distance to global minima decreases) and r = 1 when maximizing (the tness increases as the distance to a global maxima decreases). In the rest of the paper we are interested in minimizing the tness function. As shown in [24], GA problems can be classied, when minimizing, according to the value of the FDC coefcient in: misleading (r 0.15), in which the tness tends to increase as the global minimum approaches, difcult (0.15 < r < 0.15), in which there is virtually no correlation between tness and distance and straight-forward (r 0.15), in which tness decreases as global minimum approaches. The second class corresponds to problems for which the difculty cannot be estimated because r doesnt bring any information and in this case the tness-distance scatterplot may give information on problem difculty. IV. F ITNESS F UNCTION FOR S TATE - BASED T ESTING Extended forms of nite state machines (e.g. statecharts [25], X-machines [26]) have received extensive study in recent years. They are formally described by nite state machines (FSMs) which have the transitions between the states labelled by function (method) names. One of the great benets of FSMs are the associated testing techniques, e.g. the W-method proposed by Chow [27]. These methods were generalized and used for testing extended nite state machines [26], [28], [29]. However, the transformation of a function sequence, obtained with a certain testing method for extended state machines, into methods call sequences is hindered by the difculty of choosing the input values to be provided to the methods. In this paper we investigate the applicability of metaheuristic search techniques to the following type of tness function, which is tailored in [17] and further analysed in [18], [19]. The purpose of this type of tness function is to nd input data for state machine constrained paths, as described in the following. Given a particular path in a state machine, p =
fm f1 [g1 ] f3 [g3 ] / S1 f2 [g2 ] / S2 / . . . Sm1 [gm ] / Sm , S0 where fi represent methods, Si states and gi some constraints (for example guards in the state machine diagram), an individual (possible solution) is a list of input values, x = (x1 , x2 , . . . , xn ), corresponding to all parameters of the methods f1 , f2 , . . . , fm (in the order they appear). If the sequence of method calls, having the parameter values x = (x1 , x2 , . . . , xn ), determines the transitions between the states specied by the path p and validates the predicate gi of each transition, then x is a solution for p [17], [18]. The transition constraints g1 , g2 , . . . , gm are not necessary the guards from the state machine diagram; they can be predicates obtained from stronger variants of state-based criteria, such as full predicate [28], in which all individual

clauses in a decision are exercised, or disjunct coverage [29], in which guard conditions are transformed into disjunctive normal form and a separate transition is dened for each disjunct. Furthermore, these predicates can be arbitrary chosen, for example to realize a partitioning into equivalence classes of the parameters provided to the methods f1 , f2 , . . . , fm . The tness value of one individual will be computed as f itness = approach level + normalized branch level [17], [18], as described in Fig. 2. The rst component of the tness function, namely the approach (approximation) level, has a similar metric in evolutionary structural test data generation. It is calculated by subtracting one from the number of critical branches lying between the node from which the individual diverged away from the target node, and the target itself. In the structural approach, a critical branch is a program branch which leads to a miss of the current structural target for which test data is sought [16]. For state-based testing we consider all the transitions to be critical and for the transitions without a guard, the true predicate as pre-condition [17], [18]. An individual has missed the target path if at some point a pre-condition is not satised (this could cause a transition to another state). The approach level will be 0 for the individuals which follow the target path; for those which diverge from the path, it will be calculated as described in Fig. 2.
[not g1]

S0

f1 [g1]

TARGET MISSED approach level = m-1 branch level = obj(g1)

[not g2]

S1

f2 [g2]

TARGET MISSED approach level = m-2 branch level = obj(g2)

S2

[not gm]
TARGET MISSED approach level = 0 branch level = obj(gm)

Sm-1

fm [gm]

Sm

Fig. 2.

Calculating the tness function

A tness function employing just the approach level has plateaux (for each value 0, 1, . . . , m 1) and it does not offer enough guidance to the search. Consequently a second component of the tness function will compute, for the place where the actual path diverges from the required one, how close was the pre-condition predicate to being true. This second metric is called branch level and can be derived from the guard predicates using the transformations from Table I [6],

203

[10], [11], where K is a positive constant. The branch level is then mapped onto [0, 1) or normalized. Thus the tness value is: f itness = approach level + normalized branch level, for short al + nbl.
Predicate a=b a=b a<b ab Boolean ab ab Objective function obj if abs(a b) = 0 then 0 else abs(a b) + K if abs(a b) = 0 then 0 else K if a b < 0 then 0 else (a b) + K if a b 0 then 0 else (a b) + K if TRUE then 0 else K obj(a) + obj(b) min(obj(a), obj(b)) TABLE I T RACEY S OBJECTIVE FUNCTIONS

the Generation Rate of Better Solutions (GRBS), was introduced to monitor convergence of the SA search process and implement a stopping criteria. B. Extending the Motivating Example However several cost functions can be built for Cal1 problem and we consider other 6 tness functions: cost ydm, cost ymd, cost mdy, cost myd, cost dmy and cost dym. All of them have the form approach level + normalized branch level (al + nbl), discussed previously. Fig. 3 shows how cost dmy is computed, checking rst the condition day = 31, then the constraint month = 12 and nally year = 2000. The other 5 functions are calculated similarly, the only difference is in the order in which the constraints are checked, suggested by their name. To calculate branch level we employed the formulas from Table I, where the constant considered was K = 1 and the normalization function was norm(d) = 1 1.001d, d > 0 (just for better visualizing the scatterplots in Fig. 4 we replaced this with norm(d) = 1 1.01d, d > 0).
D
d) y( da ] s_ =31 i n [d

V. E MPIRICAL E VALUATION A. Motivating Example A measurement approach is experimented in [21] on various landscapes of search problems used for test generation, in order to perform a better tuning of SA. One of the examples, Cal 1 (calendar problem), is an articial problem, inspired from software testing [11] and has exactly one optimum to be found, arbitrarily taken as December 31, 2000. The solution space is the set of dates from January 1, 1900 up to December 31, 3000 and each date is encoded by a triplet (day, month, year), for short (d, m, y). Waeselynck et al. considered 24 landscapes for Cal 1 problem, corresponding to the combinations of 4 cost functions and 6 neighborhood operators, described below [21]. cost1 =0 if (d, m, y) = (31, 12, 2000), else cost1 is randomly chosen in [1, 1000] cost2 =|31d|+|12m|+|2000y|+Kd +Km +Ky Kd = 0 if d = 31, else Kd = 10; Km = 0 if m = 12, else Km = 10; Ky = 0 if y = 2000, else Ky = 100 cost3 =|31 d| + 31 |12 m| + 365 |2000 y| cost4 =Exact number of days between date (d, m, y) and date (31, 12, 2000) N 1 Any date obtained by letting one, two or three triplet parameters (d, m, y) vary 1, modulo the maximal parameter value N 2 Same as N 1, without modulo (e.g., a December date is no more connected to a January one) N 3 Any date no more than 15 days apart from the current date, modulo the solution space boundaries, that is, (1, 1, 1,900) and (31, 12, 3,000) are connected N 4 Same as N 3, without modulo N 5 Any date no more than 400 days apart from the current date, modulo the solution space boundaries N 6 Same as N 5, without modulo The measures retained by the authors are: diameter of the search landscape and autocorrelation, for which the experimental values obtained are provided. Also, a new measure,

ins_month(m) [m=12]
) y(d da s_ 31] in d= [

DM
ins [y= _ye 20 ar(y 00 ) ]

) (y ar ] ye 0 s_ 00 i n y =2 [

S0

ins_month(m) [m=12]
in s [y _ye =2 a 0 0 r(y 0] )

ins_year(y) [y=2000]
( th on 2] m s_ m = 1 in [ ins_day(d) [d=31] m)

MY

ins_day(d) [d=31]
m h( nt mo 2] s _ m= 1 in [ )

DMY

DY

(a) State machine diagram

[d <> 31]

S0
[m <> 12]

ins_day(d) [d == 31]

TARGET MISSED approach level = 2 branch level = abs(d-31)

ins_month(m) [m == 12]

TARGET MISSED approach level = 1 branch level = abs(m-12)

[y <> 2000]

DM

ins_year(y) [y = 2000]

TARGET MISSED approach level = 0 branch level = abs(y-2000)

DMY

(b) Corresponding path: S0 D DM DM Y

Fig. 3. Computing the tness for cost dmy

Having the previous 4 cost functions [21], of which cost3 and cost4 are the most adequate for the Cal 1 search problem, we compared the other 6 tness functions with them (note that cost3 is a good metric, counting the approximate number of days to the solution of the problem and cost4 returns the exact number of days). The latter 6 cost functions, of the

204

form al + nbl have a general form and they incorporate no additional information, like the number of days to the target date (31,12,2000) or other properties, which could give a better guidance and transform them in perfect metrics for Cal1. The diameter criterion was used in [21] to ensure that an optimal solution has a chance to be reached before the end of the run of a SA process, whatever the starting point of the search. The diameter values for each neighbourhood operator N 1, . . . , N 6 are presented in Table II and show that for M AX IT ER = 2000, N 3 and N 4 do not pass
cost1 1000 800 Fitness Fitness 600 400 200 0 0
5

cost2 1500

1000

500

4 3 Fitness 2 1 0

x 10

2 Distance cost3

3 x 10

4
5

0
5

4 3 Fitness 2 1 0

x 10

2 Distance cost4

3 x 10

4
5

2 Distance cost_ydm

3 x 10

4
5

2 Distance cost_ymd

3 x 10

4
5

3 2.5 Fitness Fitness

3 2.5 2 1.5 1

2 1.5

0.5 1 0 1 2 Distance cost_mdy 3 x 10 4

5

2 Distance cost_myd

3 x 10

4
5

2.5 2 Fitness Fitness 1.5 1 0.5 0 0 1 2 Distance cost_dmy 3 x 10 4

5

2.5

1.5

this criterion (the diameter should be signicantly lower than M AX IT ER, the maximum number of iterations per run). In the Tables II and III, bold values indicate that a criterion is passed, i.e. for the autocorrelation that |1 | is close to 1 and for FDC that r 0.15. Another measure employed for comparison was the autocorrelation of each tness function, using random walks with the neighbourhood operators N 1, . . . , N 6. The results, given in Table II, which were calculated after 100 random walks, each one of 1000 samples, show that cost3, cost4, cost ymd, cost ydm pass the criterion for N 1, . . . , N 6; cost2 passes the criterion just for N 1, N 2; the remaining combinations do not satisfy it. The last measure used was the FDC, computed for 1000 random samples; the results were averaged over 100 runs. To measure the distance between a point and the global optimum we used 2 metrics: d1. the exact number of days between the two dates; d2. the euclidian distance between 2 real points from R3 , as the solutions were coded as triplets (d, m, y). The mean and standard deviations of the FDC coefcients obtained for both distances are presented in Table III. These results show that for cost2, cost3, cost4 the tness values vary according to the distance to optimum and they are ideal for a search. Among the other 6 functions only cost ydm and cost ymd have a high FDC coefcient (they also have a high autocorrelation) and we would expect them to have a better behaviour in guiding the search, than cost dmy for example (the FDC criterion would classify it as difcult). The scatterplots from Fig. 4 correspond to the tness functions presented and the distance d1 (exact number of days). It is obvious from Fig. 4 that cost1 (the random function) is the worst; cost2, cost3 and cost4 are straightforward; cost ymd and cost ydm are, in a certain way, guiding the search to the optimum. However, the last 4 functions are somehow leveled and not so appropriate for guiding the search. The same tness values from the intervals [1, 1 + ], [2, 2 + ], where 0 < , < 1, might correspond to points in the search space that are very close or very far away from the optimum. For example, x1 = (31, 11, 2000) is just 1 month away from the target date (31, 12, 2000); x2 = (31, 11, 3000) is more than 999 years away from the target and cost mdy(x1 ) = cost mdy(x2 ) = 2 + , 0 < < 1.
Diameter cost1 cost2 cost3 cost4 cost ydm cost ymd cost mdy cost myd cost dmy cost dym N1 550 0.00 0.88 0.99 0.99 0.95 0.96 0.38 0.39 0.19 0.22 N2 1100 0.00 0.90 0.99 0.99 0.97 0.97 0.38 0.40 0.24 0.25 N3 6700 -0.01 0.21 0.90 0.99 0.91 0.90 0.67 0.71 0.01 0.02 N4 13400 0.00 0.20 0.90 0.99 0.88 0.91 0.70 0.71 0.01 0.02 N5 250 0.00 0.34 0.98 0.99 0.98 0.95 0.03 0.04 0.00 0.00 N6 500 0.00 0.32 0.98 0.99 0.97 0.97 0.03 0.04 0.00 0.00

2 Distance cost_dym

3 x 10

4
5

2.5 2 Fitness Fitness 0 1 2 Distance 3 x 10 4

5

2.5

1.5 1 0.5 0

1.5

2 Distance

3 x 10

4
5

TABLE II D IAMETER AND AUTOCORRELATION ( MEAN )

Fig. 4.

Fitness distance scatterplots

205

cost1 cost2 cost3 cost4 cost ydm cost ymd cost mdy cost myd cost dmy cost dym

Mean1 (Std1) 0.000 (0.032) 0.998 (<0.001) 0.999 (<0.001) 0.999 (<0.001) 0.978 (0.014) 0.978 (0.016) 0.005 (0.029) 0.086 (0.028) 0.005 (0.032) 0.037 (0.029)

Mean2 -0.004 0.998 0.999 0.999 0.978 0.977 0.006 0.082 0.001 0.037

(Std2) (0.031) (<0.001) (<0.001) (<0.001) (0.013) (0.017) (0.035) (0.031) (0.029) (0.029)

TABLE III FDC: MEAN ( STANDARD DEVIATION )

and GA5 scattered crossover) and the results in Table V show that heuristic crossover provided for this problem the best success rates among the other recombination operators. PSO (last column in Table V) achieved also high success rates. The conguration for the PSO experiment was: number of particles 20, maximum particle velocity = (15, 6, 550), i.e. (ubd lbd)/2, acceleration coefcients c1 = 2, c2 = 2 , maximum number of iterations (epochs) 100, inertia weight w linearly decreased from 0.9 to 0.4. The results from Table V show also that the al+nbl functions might obtain success rates close to the ones of problem specic functions, in particular for GA with heuristic crossover and PSO.
cost1 cost2 cost3 cost4 cost ydm cost ymd cost mdy cost myd cost dmy cost dym N1 0.0% 13.6% 46.8% 56.6% 2.6% 48.7% 20.1% 31.2% 1.8% 0.1% N2 0.0% 14.5% 48.7% 59.6% 2.5% 47.9% 18.6% 27.9% 0.9% 0.1% N3 0.0% 0.0% 1.8% 3.1% 0.1% 0.2% 0.1% 0.1% 0.0% 0.0% N4 0.0% 0.0% 2.4% 2.7% 0.1% 0.2% 0.1% 0.0% 0.0% 0.0% N5 0.0% 0.6% 15.4% 36.9% 27.1% 29.8% 0.9% 7.3% 0.6% 1.3% N6 0.0% 1.0% 15.7% 41.0% 27.0% 28.6% 0.8% 8.0% 0.5% 0.7%

The second step of our investigation consisted in evaluating the success rates obtained for each cost function, employing different search algorithms. The Matlab toolbox for genetic algorithms and direct search [30] was used for SA and GA implementations; for PSO, the experiments were performed with the particle swarm optimization toolbox (PSOt) [31], presented in [32]. The settings for SA were: problem with lower and upper bound constraints, no hybridization, no re-annealing; maximum iteration number 2000, maximum tness evaluations 3000, initial temperature 100, cooling schedule t = t0.95, acceptance function acceptancesa (based on a Boltzmann probability density, i.e. 1/(1 + exp(/t))). Table IV presents the results obtained for these parameters, varying the neighbour operators and cost functions specied before. As explained previously [21], N 3 and N 4 neighborhood operators are considered as bad choices, according to the diameter measure, when SA has maximum iteration number 2000, so the results on their columns are not surprising. The difference of success rates for cost2, cost3, cost4, cost ydm and cost ymd (which had the best statistical coefcients, but success rates between 2.5% and 59.6%), show that FDC and the autocorrelation measures have a limited discriminating power. On the other hand, these measurements show that the general tness functions (e.g. cost ymd) may produce results comparable to those produced by tness functions designed especially for a particular situation (cost3, cost4). The functions of the form al + nbl are used often in structural testing with genetic algorithms and consequently it is worth investigating the GA dynamics when applied to them. The GA parameter settings were: population size 20, maximum number of generations allowed 100, elite count 2, crossover fraction 0.8, mutation mutationadaptf easible (mutation operator for linearly constrained/bounded problems, which creates the mutated children using adaptive mutation such as the mutated genes satisfy linear constraints)1 , selection function selectionremainder, tness scaling function f itscalingrank. The crossover function was varied (GA1 heuristic, GA2 single point, GA3 two points, GA4 arithmetic
1 The three search techniques were restricted to the same space bounds and to take into account just integer values, for each function was considered f itness(f loor(x)) instead of f itness(x); consequently a solution to the problem was f loor(x).

TABLE IV S UCCESS RATES OF SA

VI. R ELATED W ORK In the last years search-based testing has been widely studied and has proved its effectiveness as a technique for automating the process of test case generation. However, there exist only a few theoretical investigations which complement the experimental work performed so far. These kind of investigations are necessary to show when, why and which search algorithms work well for a certain problem. A theoretical analysis of the scenarios in which evolutionary algorithms are suitable for structural test case generation is performed in [33]. The empirical study considers real world programs, where the search spaces are several orders of magnitude larger than those previously considered. One conclusion of the study is that Royal Road property is found in the branches of real world programs and evolutionary testing succeeds in nding test data, whilst hill climbing always fails. On
GA1 0.0% 84.7% 86.9% 89.1% 76.1% 84.3% 78.2% 84.9% 76.6% 73.8% GA2 0.0% 80.6% 82.6% 41.0% 29.4% 28.9% 51.2% 42.1% 48.8% 34.5% GA3 0.1% 70.4% 62.1% 27.9% 20.6% 19.8% 37.4% 23.2% 42.2% 30.2% GA4 0.0% 52.1% 49.1% 22.9% 9.9% 10.6% 30.3% 16.3% 24.8% 16.2% GA5 0.1% 79.7% 77.2% 38.6% 24.6% 26.0% 48.9% 38.5% 42.9% 34.7% PSO 0.1% 91.8% 90.1% 96.2% 58.6% 78.1% 88.7% 71.4% 83.1% 40.6%

cost1 cost2 cost3 cost4 cost ydm cost ymd cost mdy cost myd cost dmy cost dym

TABLE V GA AND PSO SUCCESS RATES

206

the other hand, the same study provides some examples where the simple hill climbing approach signicantly outperforms evolutionary testing. Arcuri et al. formally analyse in [34] the expected runtime of three different search algorithms on the problem of test data generation for an instance of the triangle classication program. Intuitively, the relief of the landscape should have a strong impact on the dynamics of exploration strategies [21] and some papers tackle this issue. The properties of tness functions employed in search-based testing, more precisely the landscape characterizations, are analyzed in [6], [12], [21]. McMinn presents different types of tness functions, with their associated landscapes, and they are characterized to be deceptive, or to have plateaux (which do not offer enough guidance to the search) [6]. An evaluation of different tness functions for the evolutionary testing of an autonomous parking system, by performing a number of experiments to analyze the tness landscapes for the tness functions, is performed in [12]. Waeselynck et al. investigated a measurement approach to nd an adequate setting of simulated annealing parameters, applied to test generation [21]. VII. C ONCLUSIONS AND F UTURE W ORK This paper investigates the usage of general tness functions for specication based-testing and performs an empirical measurement approach (employing the diameter, the autocorrelation and the FDC as characterizing measures). It also realizes an evaluation on various examples, which show that this general tness function may produce results comparable to those produced by tness functions designed especially for a particular situation. Metaheuristic are general search techniques and have to be tuned for each cost function considered; consequently nding adequate parameter settings that will give performance to the search is a difcult problem. The experiments performed suggest some tuning choices for SA, GA and PSO. Future work concerns analyzing other variants of tness function for state-based testing, experimentation on a larger benchmark of real world objects and extending the approach to the case in which the method parameters have complex types. R EFERENCES
[1] R. Hamlet, Random testing, Encyclopedia of Software Engineering. Wiley, 1994, pp. 970978. [2] J. King, Symbolic execution and program testing, Communications of the ACM, vol. 19, no. 7, pp. 385394, 1976. [3] A. J. Offutt, Z. Jin, and J. Pan, The dynamic domain reduction procedure for test data generation, Software - Practice and Experience, vol. 22, no. 2, pp. 167193, 1999. [4] R. A. DeMillo and A. J. Offutt, Constraint-based automatic test data generation, IEEE Trans. Softw. Eng., vol. 17, no. 9, pp. 900909, 1991. [5] A. Baresel, H. Sthamer, and M. Schmidt, Fitness function design to improve evolutionary structural testing, Proc. of the Genetic and Evolutionary Computation Conference (GECCO02), 2002, pp. 1329 1336. [6] P. McMinn, Search-based software test data generation: a survey. Softw. Test., Verif. Reliab., vol. 14, no. 2, pp. 105156, 2004.

[7] R. P. Pargas, M. J. Harrold, and R. Peck, Test-data generation using genetic algorithms. Softw. Test., Verif. Reliab., vol. 9, no. 4, pp. 263 282, 1999. [8] P. Tonella, Evolutionary testing of classes, Proc. of the 2004 ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA04), 2004, pp. 119128. [9] P. McMinn, M. Harman, D. Binkley, and P. Tonella, The species per path approach to search-based test data generation, Proc. of the 2006 International Symposium on Software Testing and Analysis (ISSTA06), 2006, pp. 1324. [10] N. Tracey, J. Clark, and K. Mander, Automated program aw nding using simulated annealing, Proc. of the 1998 ACM SIGSOFT International symposium on Software testing and analysis (ISSTA98), ACM Press, 1998, pp. 7381. [11] N. J. Tracey, A search-based automated test-data generation framework for safety-critical software, Ph.D. dissertation, University of York, 2000. [12] J. Wegener and O. B hler, Evaluation of different tness functions u for the evolutionary testing of an autonomous parking system, Proc. of the Genetic and Evolutionary Computation Conference (GECCO04)(2), 2004, pp. 14001412. [13] O. B hler and J. Wegener, Evolutionary functional testing, Computers u and Operations Research, 2007. [14] A. Baresel, H. Pohlheim, and S. Sadeghipour, Structural and functional sequence test of dynamic and state-based software with evolutionary algorithms. Proc. of the Genetic and Evolutionary Computation Conference (GECCO03), 2003, pp. 24282441. [15] P. McMinn and M. Holcombe, The state problem for evolutionary testing. Proc. of the Genetic and Evolutionary Computation Conference (GECCO03), 2003, pp. 24882498. [16] P. McMinn and M. Holcombe, Evolutionary testing of state-based programs. Proc. Genetic and Evolutionary Computation Conference (GECCO05), 2005, pp. 10131020. [17] R. Lefticaru and F. Ipate, Automatic state-based test generation using genetic algorithms, Proc. of the Ninth International Symposium on Symbolic and Numeric Algorithms For Scientic Computing (SYNASC 07). IEEE Computer Society, 2007, pp. 188195. [18] R. Lefticaru and F. Ipate, Functional search-based testing from state machines, Proc. of the 2008 International Conference on Software Testing, Verication, and Validation. IEEE Computer Society, 2008, pp. 525528. [19] R. Lefticaru and F. Ipate, Search-based testing using state-based tness, in Proc. of the 2008 IEEE International Conference on Software Testing Verication and Validation Workshop . IEEE Computer Society, 2008, p. 210. [20] S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi, Optimization by simulated annealing, Science, vol. 220, no. 4598, pp. 671680, 1983. [21] H. Waeselynck, P. Th venod-Fosse, and O. Abdellatif-Kaddour, Sime ulated annealing applied to test generation: landscape characterization and stopping criteria, Empirical Software Engineering, vol. 12, no. 1, pp. 3563, 2007. [22] Y. Shi and R. Eberhart, Empirical study of particle swarm optimization, Proc. of the 1999 Congress on Evolutionary Computation (CEC99), 1999, pp. 19451950. [23] W. Hordijk, A measure of landscapes, Evolutionary Computation, vol. 4, no. 4, pp. 335360, 1996. [24] T. Jones and S. Forrest, Fitness distance correlation as a measure of problem difculty for genetic algorithms, Proc. of the 6th International Conference on Genetic Algorithms, 1995, pp. 184192. [25] D. Drusinsky, Modeling and Verication Using UML Statecharts: A Working Guide to Reactive System Design, Runtime Monitoring and Execution-based Model Checking. Newnes, 2006. [26] F. Ipate, Testing against a non-controllable stream X-machine using state counting. Theoretical Comput. Sci., vol. 353, no. 1-3, pp. 291 316, 2006. [27] T. S. Chow, Testing software design modeled by nite-state machines. IEEE Trans. Softw. Eng., vol. 4, no. 3, pp. 178187, 1978. [28] A. J. Offutt, S. Liu, A. Abdurazik, and P. Ammann, Generating test data from state-based specications. Softw. Test., Verif. Reliab., vol. 13, no. 1, pp. 2553, 2003. [29] L. C. Briand, M. D. Penta, and Y. Labiche, Assessing and improving state-based class testing: A series of experiments. IEEE Trans. Softw. Eng., vol. 30, no. 11, pp. 770793, 2004. [30] Genetic Algorithm and Direct Search Toolbox 2.2, http://www.mathworks.com/products/gads/.

207

[31] PSOtoolbox, http://psotoolbox.sourceforge.net/. [32] B. Birge, PSOt - a particle swarm optimization toolbox for use with Matlab, Proc. IEEE Swarm Intelligence Symposium, 2003, pp. 182 186. [33] M. Harman and P. McMinn, A theoretical & empirical analysis of evolutionary testing and hill climbing for structural test data generation, Proc. of the 2007 International Symposium on Software Testing and Analysis (ISSTA07), 2007, pp. 7383. [34] A. Arcuri, P. K. Lehre, and X. Yao, Theoretical runtime analyses of search algorithms on the test data generation for the triangle classication problem, Proc. of the 2008 IEEE International Conference on Software Testing Verication and Validation Workshop. IEEE Computer Society, 2008, pp. 161169.

208