You are on page 1of 7

9/27/2012

The Test Plan


The test plan is a mandatory document. You cant test without one. For simple, straight-forward projects the plan doesnt have to be elaborate but

Test Plan

it must address certain items

Planning may be documented in a master test plan and in separate test plans for test levels such as system testing and acceptance testing

The outline of a test planning document is covered by the Standard for

Sept Dec, 2012


2

SoftwareTest Documentation (IEEE Std 829-1998)


Software Testing 05/02/2011

The Test Plan


Planning is influenced by:
o The test policy of the organization o The scope of testing o Objectives o Risks o Constraints o Criticality o Testability o The availability of resources
3 Software Testing 05/02/2011 4

The Test Plan


As the project and test planning progress, more information becomes available and more detail can be included in the plan Test planning is a continuous activity and is performed in all life cycle processes and activities Feedback from test activities is used to recognize changing risks so that planning can be adjusted As identified by the American National Standards Institute and Institute for Electrical and Electronic Engineers Standard 829/1983 for Software Test Documentation, the following components (next slide) should be covered in a software test plan
Software Testing 05/02/2011

9/27/2012

The Test Plan

Test Planning Activities


Test planning activities for an entire system or part of a system may include:
o Determining the scope and risks and identifying the objectives of testing o Defining the overall approach of testing, including the definition of the test

levels and entry and exit criteria


o Integrating and coordinating the testing activities into the software life cycle

activities (acquisition, supply, development, operation and maintenance)


o Making decisions about what to test, what roles will perform the test

activities, how the test activities should be done, and how the test results will be evaluated

Software Testing

05/02/2011

Software Testing

05/02/2011

Test Planning Activities


o Scheduling test analysis and design activities o Scheduling test implementation, execution and evaluation o Assigning resources for the different activities defined o Defining the amount, level of detail, structure and templates for the test

Reduce Risk With a Test Plan


The release of a new application or an upgrade inherently carries a certain amount of risk that it will fail to do what its supposed to do. A good test plan goes a long way towards reducing this risk. By identifying areas that are riskier than others we can concentrate our testing efforts there. These areas include not only the must-have features but also areas in which the technical staff is less experienced, perhaps such as the real-time loading of a web forms contents into a database using complex ETL logic.

documentation
o Selecting metrics for monitoring and controlling test preparation and

execution, defect resolution and risk issues


o Setting the level of detail for test procedures in order to provide enough

information to support reproducible test preparation and execution.

Software Testing

05/02/2011

Software Testing

05/02/2011

9/27/2012

Reduce Risk With a Test Plan


Because riskier areas require more certainty that they work properly, failing to correctly identify those risky areas leads to a misallocated testing effort. How do we identify risky areas? Ask everyone for their opinion! Gather information from developers, sales and marketing staff, technical writers, customer support people, and of course any users who are available. Historical data and bug and testing reports from similar products or previous releases will identify areas to explore.

Reduce Risk With a Test Plan


Bug reports from customers are important, but also look at bugs reported by the developers themselves. These will provide insight to the technical areas they may be having trouble in. When the problems are inevitably found, its important that both the IT side and the business users have previously agreed on how to respond. This includes having a method for rating the importance of defects so that repair work effort can be focused on the most important problems.

Software Testing

05/02/2011

10

Software Testing

05/02/2011

Reduce Risk With a Test Plan


It is very common to use a set of rating categories that represent decreasing relative severity in terms of business/commercial impact. In one system, '1' is the most severe and 6' has the least impact. Keep in mind that an ordinal system doesnt allow an average score to be calculated, but you shouldnt need to do that anywaya defects category should be pretty obvious. 1. Show Stopper It is impossible to continue testing because of the severity of the defect. 2. Critical - Testing can continue but the application cannot be released into production until this defect is fixed.
11 Software Testing 05/02/2011 12

Reduce Risk With a Test Plan


3. Major - Testing can continue but this defect will result in a severe departure from the business requirements if released for production. 4. Medium - Testing can continue and the defect will cause only minimal departure from the business requirements when in production. 5. Minor Testing can continue and the defect will not affect the release into production. The defect should be corrected but little or no changes to business requirements are envisaged. 6. Cosmetic Minor cosmetic issues like colors, fonts, and pitch size that do not affect testing or production release. If, however, these features are important business requirements then they will receive a higher severity level.
Software Testing 05/02/2011

9/27/2012

What Should a Test Plan Test?


Testing experts generally agree that test plans are often biased towards functional testing during which each feature is tested alone in a unit test, and that the systems integration test is just a series of unit tests strung together. The problem that this approach causes is that if we test each feature alone and then string a bunch of these tests together, we might never find that a series of steps such as
o open a document, edit the document, print the document, save the document, edit one

What Should a Test Plan Test?...


Admittedly, testing every combination of keystrokes or commands is difficult at best and may well be impossible (this is where unstructured testing comes in), but we must remember that features dont function in isolation from each other. Users have a task orientation. To find the defects that they will find, the ones that are important to them, test plans need to exercise the application across functional areas by mimicking both typical and atypical user tasks. A test like the sequence shown above is called scenario testing, task-based testing, or use-case testing.
05/02/2011 14 Software Testing 05/02/2011

page, print one page, save as a new document doesnt work. But a user will find out and probably quickly.

13

Software Testing

What Should a Test Plan Test?...


An incomplete test plan can result in a failure to check how the application works on different hardware and operating systems or when combined with different third-party software. This is not always needed but you will want to think about the equipment your customers use. There may be more than a few possible system combinations that need to be tested, and that can require a possibly expensive computer lab stocked with hardware and spending much time setting up tests.

What Should a Test Plan Test?...


Configuration testing isn't cheap, but its worth it when you discover that the application running on your standard in-house platform which "entirely conforms to industry standards" behaves differently when it runs on the boxes your customers are using. Development and testing done on 386-class machines may result in an application just working fine but customers may complain about performance if they use a 286-class machine with slow hard drives.

15

Software Testing

05/02/2011

16

Software Testing

05/02/2011

9/27/2012

What Should a Test Plan Test?...


A crucial test is to see how the application behaves when its under a normal load and then under stress. The definition of stress, of course, will be derived from your business requirements, but for a web-enabled application stress could be caused by a spike in the number of transactions, a few very large transactions at the same time, or a large number of almost identical simultaneous transactions. The goal is to see what happens when the application is pushed to substantially more than the basic requirements.
17 Software Testing 05/02/2011 18

What Should a Test Plan Test?...


Stress testing is often put off until the end of testing, after everything else thats going to be fixed has been. Unfortunately that leaves little time for repairs when the requirements specify 40 simultaneous users and you find that performance becomes unacceptable at 50. Marick (1997) points out two common omissions in many test plans - the installation procedures and the documentation are ignored. Everyone has tried to follow installation instructions that were missing a key step or two, and weve all paged through incomprehensible documentation.

Software Testing

05/02/2011

What Should a Test Plan Test?...


Although those documents may have been written by a professional technical writer, they probably werent tested by a real user. Bad installation instructions immediately cause lowered expectations of what to expect from the product, and poorly organized or written documentation certainly doesnt help a confused or irritated customer feel better. Testing installation procedures and documentation is a good way to avoid making a bad first impression or making a bad situation worse.

The Test Plan Terminology

19

Software Testing

05/02/2011

20

Software Testing

05/02/2011

9/27/2012

The Entry Criteria


Entry criteria define when to start testing such as at the beginning of a test level or when a set of tests is ready for execution. Typically entry criteria may cover the following:
o Test environment availability and readiness o Test tool readiness in the test environment o Testable code availability o Test data availability

The Exit Criteria


Exit criteria define when to stop testing such as at the end of a test level or when a set of tests has achieved specific goal. Typically exit criteria may cover the following:
o Thoroughness measures, such as coverage of code, functionality or risk o Estimates of defect density or reliability measures o Cost o Residual risks, such as defects not fixed or lack of test coverage in certain

areas
o Schedules such as those based on time to market

21

Software Testing

05/02/2011

22

Software Testing

05/02/2011

Test Development Process


The test development process can be done in different ways, from very informal with little or no documentation, to very formal. The level of formality depends on the context of the testing including:
o The maturity of testing o Development processes o Time constraints o Safety or regulatory requirements o The people involved

Test Development Process


A test condition is defined as an item or event that could be verified by one or more test cases (e.g. a function, transaction, quality characteristic or structural element). Establishing traceability from test conditions back to the specifications and requirements enables both effective impact analysis when requirements change, and determining requirements coverage for a set of tests. During test analysis the detailed test approach is implemented to select the test design techniques to use based on, among other considerations, the identified risks.
05/02/2011 24 Software Testing 05/02/2011

During test analysis, the test basis documentation is analyzed in order to determine what to test, i.e. to identify the test conditions.
23 Software Testing

9/27/2012

Test Development Process


During test design the test cases and test data are created and specified. A test case consists of a set of input values, execution preconditions, expected results and execution post-conditions, developed to cover a certain test objective(s) or test condition(s). The Standard for Software Test Documentation (IEEE STD 829-1998) describes the content of test design specifications (containing test conditions) and test case specifications.

Test Development Process


Expected results should be produced as part of the specification of a test case and include outputs, changes to data and states, and any other consequences of the test. If expected results have not been defined, then a plausible, but erroneous, result may be interpreted as the correct one. Expected results should ideally be defined prior to test execution. During test implementation the test cases are developed, implemented, prioritized and organized in the test procedure specification (IEEE STD 829-1998). The test procedure specifies the sequence of actions for the execution of a test.

25

Software Testing

05/02/2011

26

Software Testing

05/02/2011

Test Development Process


If tests are run using a test execution tool, the sequence of actions is specified in a test script (which is an automated test procedure). The various test procedures and automated test scripts are subsequently formed into a test execution schedule that defines the order in which the various test procedures, and possibly automated test scripts, are executed. The test execution schedule will take into account such factors as regression tests, prioritization, and technical and logical dependencies.

END

27

Software Testing

05/02/2011

28

Software Testing

05/02/2011

You might also like