Black Box Software Testing: Introduction To Test Documentation

Black Box Software Testing
Part 12.
Introduction to Test Documentation

Co-authors: Cem Kaner & James Bach
Acknowledgement
Some of this material is from the Third Los Altos Workshop on Software Testing (LAWST), February 7 and 8, 1998. I founded the LAWST and was co-organizer of LAWST 3. At LAWST, we discussed test documentation (test planning strategies and materials). The agenda item was:
How do we know what test cases we have? How do we know which areas of the program are well covered and which are not? How do we develop this documentation EFFICIENTLY? As many of you know, I despise thick test plans and I begrudge every millisecond that I spend on test case documentation. Unfortunately, some work is necessary. My question is, how little can we get away with, while still preserving the value of our asset?
The following people attended LAWST 3: Chris Agruss, James Bach, Karla Fisher, David Gelperin, Kenneth Groder, Elisabeth Hendrickson, Doug Hoffman, III, Bob Johnson, Cem Kaner, Brian Lawrence, Brian Marick, Thanga Meenakshi, Noel Nyman, Jeffery E. Payne, Bret Pettichord, Johanna Rothman, Jane Stepak, Melora Svoboda, Jeremy White, and Rodney Wilson.
Copyright
2003
Cem Kaner & James Bach
499
Copyright Notice
These slides are distributed under the Creative Commons License. In brief summary, you may make and distribute copies of these slides so long as you give the original author credit and, if you alter, transform or build upon this work, you distribute the resulting work only under a license identical to this one. For the rest of the details of the license, see http://creativecommons.org/licenses/by-sa/2.0/legalcode.
Copyright
2003
500
Definitions
The test planning documents include: The Testing Project Plan, which identifies classes of tasks and broadly allocates people and resources to them; High-level designs for test cases (individual tests) and test suites (collections of related tests); Detailed lists or descriptions of test cases; Anything else that you would put in a hard copy or virtual binder that documents the collection of test cases that you will develop and run. This collection of materials is sometimes called the test plan and sometimes called the test documentation.
Copyright
2003
501
Levels of Analysis
1-Variable Basic user interface verification Filters Many testers stop here, at the most superficial level of testing 2-Variable Shared or related filters Logical relationships among variables Multi-Variable The core problem is the vastness of the space of test cases. We need a sampling strategy.
Copyright
2003
502
Basic Test Documentation Components

Lists: Such as lists of fields, error messages, DLLs Outlines: An outline organizes information into a hierarchy of
lists and sublists
Such as the testing objectives list later in the course notes Tables: A table organizes information in two dimensions
showing relationships between variables.
Such as boundary tables, decision tables, combination test tables Matrices: A matrix is a special type of table used for data
collection.
Such as the numeric input field matrix, configuration matrices

Refer to Testing Computer Software, pages 217-241. For more examples, see page Testing Computer Software, page 218.
Copyright
2003
503
Configuration Planning Table

Var 1 Config 1 Config 2 Config 3 Config 4 Config 5 Config 6 V1-1 V1-2 V1-3 V1-4 V1-5 V1-6 Var 2 V2-1 V2-2 V2-3 V2-4 V2-5 V2-6 Var 3 V3-1 V3-2 V3-3 V3-4 V3-5 V3-6 Var 4 V4-1 V4-2 V4-3 V4-4 V4-5 V4-6 Var 5 V5-1 V5-2 V5-3 V5-4 V5-5 V5-6
This table defines 6 standard configurations for testing. In later tests, the lab will set up a Config-1 system, a Config-2 system, etc., and will do its compatibility testing on these systems. The variables might be software or hardware choices. For example, Var 1 could be the operating system (V1-1 is Win 2000, V1-2 is Win ME, etc.) Var 2 could be how much RAM on the computer under test (V2-1 is 128 meg, V2-2 is 32 meg., etc.). Var 3 could be the type of printer, Var 4 the machines language setting (French, German, etc.). Config planning tables are often filled in using the All Pairs algorithm. 504 Black Box Software Testing Copyright 2003 Cem Kaner & James Bach
Configuration Test Matrix

Config 1 Test 1 Test 2 Test 3 Test 4 Test 5 Pass Pass Fail Pass Config 2 Pass Fail Pass Fail Pass Config 3 Pass Pass Pass Fail Fail Pass Config 4 Pass Pass Pass Config 5 Pass Pass Pass Pass Pass Config 6
This matrix records the results of a series of tests against the 6 standard configurations that we defined in the Configuration Planning Table.
In this table, Config 1 has passed 3 tests, failed 1, and hasnt yet been tested with Test 2. Config 6 is still untested.
Copyright
2003
505

Part 13.
Scripting: An Industry Worst Practice
Copyright
2003
506
Scripting
COMPLETE SCRIPTING is favored by people who believe that repeatability is everything and who believe that with repeatable scripts, we can delegate to cheap labor. 1 ____ Pull down the Task menu 2 ____ Select First Number 3 ____ Enter 3 4 ____ Enter 2 5 ____ Press return 6 ____ The program displays 5
Copyright
2003
507
Scripting: Another script format

Step #
1.
Check What to ? do
____ Pull down task menu
What to see
Task menu down
Design notes
This starts the blah blah test, with the blah blah goal
Observation notes
Copyright
2003
508
Scripting: The Bus of Testing

Scripting is the Greyhound Bus of software testing: Just relax and leave the thinking to us.
To the novice, the test script is the whole tour. The tester goes through the script, start to finish, and thinks hes seen what there is to see. To the experienced tester, the test script is a tour bus. When she sees something interesting, she stops the bus and takes a closer look. One problem with a bus trip. Its often pretty boring, and you might spend a lot of time sleeping.
Copyright
2003
509
Scripting (Manual Test Cases)

Worst of all worlds:
Not automated Not maintainable No communication of analysis NOT REPEATABLE!
Copyright
2003
510
Scripting
Should we create scripts as pseudocode, in preparation for test automation?
Suggestion made at STAR 97 But most (88%) of bugs were found during creation of the scripts (my experience too).
Maybe the benefit lies in structured exploration, rather than regression.
Copyright
2003
511
Never Do ANY kind of Scripting?

Checklists (as distinct from scripts) have their place. For example, think of releasing a product: Many very different tasks All of them must be done The task is rarely done, so many steps may be forgotten. I prefer to tell testers what to test (what issues to cover), not how to do the tests. Teaching people how is a matter of training, not something that I record time and time again in the test plan. A checklist will sometimes be the right way to present the list of issues. I think the level of detail is sufficient if I can successfully pass the section to a reasonably experienced tester who is a little familiar with the program and be confident that she can figure out what the test cases are and how to run them.
512
Copyright
2003

Part 14.
Requirements Analysis for Test Documentation
Copyright
2003
513
Problems with the (allegedly) standard approach

It is essential to understand your requirements for test documentation. Unless following a standard helps you meet your requirements, it is empty at best, anti-productive at worst.
Copyright
2003
514

IEEE Standard 829 for Software Test Documentation Test plan Test-design specification Test-case specification
Test-case specification identifier Test items Input specifications Output specifications Environmental needs Special procedural requirements Intercase dependencies
We often see one or more pages per test case.
Test-procedure specification Test-item transmittal report Test-log
Copyright
2003
515

What is the documentation cost per test case? What is the maintenance cost of the documentation, per test case? If software design changes create documentation maintenance costs, how much inertia do we build into our system? How much does extensive test documentation add to the cost of late improvement of the software? How much should we add? What inertia is created in favor of invariant regression testing? Is this incompatible with exploratory testing? Do we always want to discourage exploration?
Copyright
2003
516

What is the impact on high-volume test automation? How often do project teams start to follow 829 but then give it up mid-project? What does this do to the net quality of the test documentation and test planning effort? WHAT REQUIREMENTS DOES A STANDARD LIKE THIS FULFILL? WHICH STAKEHOLDERS GAIN A NET BENEFIT FROM IEEE STANDARD DOCUMENTATION? WHAT BENEFITS DO THEY GAIN, AND WHY ARE THOSE BENEFITS IMPORTANT TO THEM?
Copyright
2003
517
Standards: ISO 9000-3

The basic thinking underlying ISO 9000 goes like this: Decide what youre going to do (create and document a process). Do what you said youd do. Document what youve done in a way that lets an auditor check what youve done against your process. Risks It can be difficult to build in room for exploratory testing. To some degree (perhaps a large degree), this depends on the auditor. The style and extent of documentation needed to clue in a third party (auditor) might go beyond your internal needs. On finite budgets, where do we draw the line? What is the best tradeoff between process development and test execution? This must be a practical question, not a religious question.
Black Box Software Testing Copyright
2003
518
Requirements
There are many different notions of what a good set of test documentation would include. Before spending a substantial amount of time and resources, its worth asking what documentation should be developed (and why?) Test documentation is expensive and it takes a long time to produce. If you figure out some of your main requirements first, you might be able to do your work in a way that achieves them.
Copyright
2003
519
Defining documentation requirements

Stakeholders, interests, actions, objects
Who would use or be affected by test documentation? What interests of theirs does documentation serve or disserve? What will they do with the documentation? What types of documents are of high or low value?
Asking questions Context-free questions Context-free questions specific to test planning Evaluating a plan
Copyright
2003
520
Discovering Requirements
Requirements Anything that drives or constrains design Stakeholders Favored, disfavored, and neutral stakeholders Stakeholders interests Favored, disfavored, and neutral interests Actions Actions support or interfere with interests Objects
Copyright
2003
521
What is your groups mission?

Find important problems Assess quality Certify to standard Fulfill process mandates Satisfy stakeholders Assure accountability Advise about QA Advise about testing Advise about quality Maximize efficiency Minimize time Minimize cost
The quality of testing depends on which of these possible missions matter and how they relate.
Many debates about the goodness of testing are really debates over missions and givens.
Copyright
2003
522
Test Docs Requirements Questions

Is test documentation a product or tool? Is software quality driven by legal issues or by market forces? How quickly is the design changing? How quickly does the specification change to reflect design change? Is testing approach oriented toward proving conformance to specs or nonconformance with customer expectations? Does your testing style rely more on already-defined tests or on exploration? Should test docs focus on what to test (objectives) or on how to test for it (procedures)? Should the docs ever control the testing project? Black Box Software Testing Copyright 2003 Cem Kaner & James Bach
523

If the docs control parts of the testing project, should that control come early or late in the project?
Who are the primary readers of these test documents and how important are they?
How much traceability do you need? What docs are you tracing back to and who controls them?
To what extent should test docs support tracking and reporting of project status and testing progress?
How well should docs support delegation of work to new testers?
What are your assumptions about the skills and knowledge of new testers?
Is test doc set a process model, a product model, or a defect finder? 524 Black Box Software Testing Copyright 2003 Cem Kaner & James Bach

A test suite should provide prevention, detection, and prediction. Which is the most important for this project? How maintainable are the test docs (and their test cases)? And, how well do they ensure that test changes will follow code changes? Will the test docs help us identify (and revise/restructure in face of) a permanent shift in the risk profile of the program? Are (should) docs (be) automatically created as a byproduct of the test automation code?
Copyright
2003
525
Example: Early vs. Late Planning

Benefits of early planning Scheduling and staffing (unless things change) Early opportunity to review (if people can understand your work) Costs of early planning Any delays in finding bugs are expensive. You should start finding bugs ASAP. Early investment in test case design is risky if the product design is subject to change. You will keep learning over time. Planning early sometimes precludes groups from inventing new tests later. Depth of testing should reflect risks, many of which are unclear until mid-project. MY USUAL GOAL: Do just-in-time test planning without losing key benefits of early work.
Copyright
2003
526
Ultimately, write a mission statement

Try to describe your core documentation requirements in one sentence that doesnt have more than three components. Examples: The test documentation set will primarily support our efforts to find bugs in this version, to delegate work, and to track status. The test documentation set will support ongoing product and test maintenance over at least 10 years, will provide training material for new group members, and will create archives suitable for regulatory or litigation use.
Copyright
2003
527

Black Box Software Testing: Introduction To Test Documentation

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Black Box Software Testing: Introduction To Test Documentation

Uploaded by

Copyright:

Available Formats

Black Box Software Testing

Introduction to Test Documentation