Professional Documents
Culture Documents
Dr Simon Spacey
B.Sc. Hons. (York), M.Sc. (Lancaster), M.B.A. (Cass), J.L.P. (Keio), D.E.A. (Montpellier), D.CSC. (Cambridge), D.I.U. (Imperial), Ph.D. (Imperial)
Simon Spacey.
20/05/2013
<1>
Types of Testing
!! Main Types of Pass-Fail Testing:
1.! Unit Testing Development Unit Level e.g. Class Methods 2.! Integration Testing Interfaces Between Developed Units 3.! System Testing Performance, Scalability, Reliability, Security... 4.! User Acceptance Testing (UAT) Functional and Non-Functional
Design
3 1
Unit/ Int
5-10 10 1
System
10 15 10
UAT
10-100 25-100 10-25
[1] T. Hammell, Test-Driven Development: A J2EE Example, Apress, 2004
Simon Spacey.
18/04/2013
<2>
Predicting Regression Test Failures Using Genetic Algorithm-Selected Dynamic Performance Analysis Metrics
Michael Mayo and Simon Spacey
Waikato University, Hamilton, New Zealand {mmayo,sspacey}@waikato.ac.nz http://cs.waikato.ac.nz/
Abstract. A novel framework for predicting regression test failures is proposed. The basic principle embodied in the framework is to use performance analysis tools to capture the runtime behaviour of a program as it executes each test in a regression suite. The performance information is then used to build a dynamically predictive model of test outcomes. Our framework is evaluated using a genetic algorithm for dynamic metric selection in combination with state-of-the-art machine learning classiers. We show that if a program is modied and some tests subsequently fail, then it is possible to predict with considerable accuracy which of the remaining tests will also fail which can be used to help prioritise tests in time constrained testing environments. Keywords: regression testing, test failure prediction, program analysis, machine learning, genetic metric selection.
Introduction
Regression testing is a software engineering activity in which a suite of tests covering the expected behaviour of a software system are executed to verify a systems integrity after modication. As new features are added to a system, regression tests are re-run and outputs compared against expected results to ensure new feature code and system changes have not introduced bugs into old feature sets. Ideally, we would like to run all regression tests as part of the normal development process when each new feature is committed. However, regression testing the large number of tests required to cover (an ever expanding) previous feature set can take considerable time. For example, the regression test suite [14, 15] used in Section 4 of this work takes approximately 12 hours to execute fully in our environment which makes on-line regression testing dicult. Recently authors concerned with regression testing have began looking to performance analysis and machine learning to aid software engineering [1] and in this paper we propose a method that joins performance analysis [2], machine
G. Ruhe and Y. Zhang (Eds.): SSBSE 2013, LNCS 8084, pp. 158171, 2013. c Springer-Verlag Berlin Heidelberg 2013
Use OpenPAT
! Assumes Quality: Who Tests the Tester? Are Tests Reliable? How do we Know Code Coverage?
Can be done by different person than the developer
JUnit: Overview
!! JUnit Overview:
!! A Test Harness to Test Java Functional Outputs and State Changes with Defined Inputs !! Typically Used for Bottom-Up Testing before Code is Developed (Ideal) or After Code Review !! Test Cases Can be Written by Technical User/ BA/ Experienced Tester/ PM (Ideal), Interface User (without source = Black Box) or Pair Programmer (with source = White Box) !! Historic Tests Used for Regression Testing So Know New Features Dont Break Old Features !! Can be Incorporated into Smoke Tests A Subset of Tests to Quickly Confirm Main Features !! To Access Privates Need to use Reflection, Getter/Setters, Nested Classes or Package Scope
JUnit: Details
!! Structure:
!! Separate Class Called <Class Being Tested>Test! !! Import org.junit.* and org.junit.Assert.*! !! Test Methods for Each Class Method usually prefixed as test<Method>!
Tests run in definition order!
!! Key Annotations:
@Test a test method to run for each test Design Decision @Before/@After run before/after each test @BeforeClass/AfterClass run once each test set (must be static) ! @RunWith(Suite.class) @Suite.SuiteClasses({<class>[,...]}) run suite of test classes
JUnit = instance
Simon Spacey.
20/05/2013
<6>