CTFL Chapter 1 Fundamentals

1
Chapter
Software Testing Foundations

MSTB-GTB
Certified Tester
Fundamentals of Testing
2011
Version 1.0
After this lecture you should.
Be able to use examples to describe the way in which a software defect can cause harm to
people, to the environment, or to a company;
Know what is meant by the terms deficiency, defect, failure, defect condition / defect and error
and explain this with examples and compare;
Be able to distinguish between the cause of a defect and its effects;
Derive by examples why testing is necessary;
Be able to describe why testing is part of quality assurance and give examples of how testing
contributes to a higher quality;
Know the quality characteristic according to ISO 9126;
Recall the general objectives and principles of testing;
Differentiate between testing and debugging;
Know how the fundamental test process looks like;
Describe how expected values are calculated during testing;
Know and characterize the psychological problems during the testing;
Differentiate between the different mindset of a tester and a developer;
Know the code of ethics of a software tester.
Chapter 1
Page 2
Software Testing Foundations Certified Tester
Copyright 2011 to MSTB/GTB

V 1.0
Learning Objectives for Fundamentals of Testing

(according to the Certified Tester Foundation Level Syllabus, Version 2011)
Chapter 1
1.1 Why is Testing Necessary? (K2)

LO-1.1.1
Describe, with examples, the way in which a defect in software can cause
harm to a person, to the environment or to a company (K2)
LO-1.1.2
Distinguish between the root cause of a defect and its effects (K2)
LO-1.1.3
Give reasons why testing is necessary by giving examples (K2)
LO-1.1.4
Describe why testing is part of quality assurance and give examples of how
testing contributes to higher quality (K2)
LO-1.1.5
Explain and compare the terms error, defect, fault, failure and the
corresponding terms mistake and bug, using examples (K2)
1.2 What is Testing? (K2)

LO-1.2.1
Recall the common objectives of testing (K1)
LO-1.2.2
Provide examples for the objectives of testing in different phases of the

software life cycle (K2)
LO-1.2.3
Differentiate testing from debugging (K2)
Page 3

V 1.0
Learning Objectives for Fundamentals of Testing
(according to the Certified Tester Foundation Level Syllabus, Version 2011)
1.3 Seven Testing Principles (K2)

LO-1.3.1
1.4 Fundamental Test Process (K1)

LO-1.4.1
Chapter 1
Explain the seven principles in testing (K2)
Recall the five fundamental test activities and respective tasks from planning
to closure (K1)
1.5 The Psychology of Testing (K2)

LO-1.5.1
Recall the psychological factors that influence the success of testing (K1)
LO-1.5.2
Contrast the mindset of a tester and of a developer (K2)
Page 4

V 1.0
Incident
Review
Test Condition
Independence
Terms you should be familiar with
Chapter 1
Page 5
Test Basis
Quality
Risk
Test Objective
Test Plan
Testware
Master Test Plan
Test Design
Test Run Log
Quality Assurance
Error cause
Confirmation Test
Exhaustive Test
Test Coverage
Software Test
Debugging
Fundamental Test Process
Test Summary Report

Exit Criteria
Test Data
Test Policy
Failure
Error
Regression Test Requirement
Error Test Execution
Test Case Error Guessing
Defect Testing Test Suite

V 1.0
Chapter
Fundamentals
of Testing
Chapter 1
Page 6
Terms and Motivation

Principles of Testingda
Fundamental Test Process Test Process
Test Cases, Expected Results and Test Oracles
Psychology of Testing
Ethics of Testing

V 1.0
In General: What is a Defect or a Deficiency?

A situation can only be classified as defective if it is pre-defined, how
the expected, correct and therefore the non-defective situation should
look like.
Defect
Is the inability to fulfill a specific requirement. It is a discrepancy
between the actual behaviour (during the execution of the tests or
identified operation) and the expected behaviour (in the test
specification or the stated requirements).
Deficiency
Non-fulfilment of a requirement related to an intended or specified
use. A deficiency is for example the impairment of usability, while
meeting performance or failure to perform any reasonable
expectation.
Based on: DIN EN ISO 9000:2005
Quality Management Systems
Fundamentals and Terms (German)
Beuth Verlag, Berlin, 2005
Chapter 1
Page 7

V 1.0
Connection:
Error - Fault/Defect - Failure
People make mistakes; they commit errors.
Example:
A programmer commits
an error by re-using a
a piece of software which is
not intended in the context
of the current project
(the Ariane 5 satellite
launching rocket)
Error
ISTQB Glossary:
Error: Human action that produces an incorrect result [after IEEE 610].
Chapter 1
Page 8

V 1.0
Error Other Definitions

Other defintions:
1. Human action (of the developer) that
produces an error condition in the software
2. Human action (of the user) that produces
an unwanted result (failure) as a
consequence (misoperation)
3. Unknowingly, inadvertently or intentionally
performed act, or omission which leads under given
circumstances (task, environment) to an impairment of the
required function of a product
Chapter 1
Page 9

V 1.0
Connection:
An error made by a person
(e.g. a developer) can result
in a defect in the sofware.
(Defect and Fault are
synonymous terms)
Example: After the
programmer has
re-used the piece of
code in the wrong
context, the software
contains a defect/fault.
Defect/Fault
Error
in a program
ISTQB Glossary:
Defect: A flaw in a component or system that can cause the component
or system to fail to perform its required function, e.g. an incorrect statement or
data definition A(the
defect,
if encountered
execution,
may cause a
full definition
followsduring
on page
12)
failure of the component or system.
Chapter 1
Page 10

V 1.0
Connection:
A failure is the effect of
a defect when executing
a program (test object)
which appears to the
outside.
Failure
Defect/Fault
Error
Example:
Boom
The Ariane 5
rocket crashes
on its first
mission.
Cost ~ $7 billion
ISTQB Glossary:
Deviation of the component or
system from its expected delivery,
service or result [after Fenton].
Chapter 1
Page 11
in a program
which appears

V 1.0
Fault/Defect Failure
Additional comments
Defect (full definition from ISTQB glossary): A flaw in a
component or system that can cause the component or system to
fail to perform its required function, e.g. an incorrect statement or
data definition. A defect, if encountered during execution, may
cause a failure of the component or system.
Defect masking: An occurrence in which one defect prevents the
detection of another [after IEEE 610]
A failure
can also be called a malfunction or an external defect
can also (but rarely) be caused by cosmic radiation, electromagnetic fields or hardware failure. Such failures can cause
slow execution, incorrect output or the termination of execution.
Debugging is the development activity that finds, analyzes and
removes the cause of the failure.
Chapter 1
Page 12

V 1.0
Testing Definition and Goals

The process consisting of all life cycle activities, both static and
dynamic, concerned with planning, preparation and evaluation of
software products and related work products to
determine that they satisfy specified requirements
demonstrate that they are fit for purpose
detect defects
Source: [ISTQB Glossary]
Chapter 1
Page 13

V 1.0
Validation and Verification - Definitions

Validation
Confirmation by examination and through provision of
objective evidence that the requirements for a specific
intended use or application have been fulfilled .
[ISO 9000] and [ISTQB Glossary]
Did we implement the right system?
Verification
Confirmation by examination and through the provision
of objective evidence that specified requirements have
been fulfilled [ISO 9000] and [ISTQB Glossary]
Did we implement the system right?
Chapter 1
Page 14

V 1.0
What is Software Quality?

A feature or characteristic that affects an item's quality [IEEE 610].
A set of attributes of a software product by which its quality is
described and evaluated. A software quality characteristic may be
refined into multiple levels of sub-characteristics [ISO 9126]*.
Quality attributes relate to requirements
Functional requirements (capabilities, interfaces,
professionalism, ...)
Non-functional requirements (quality and implementation
requirements, project-specific requirements, ...)
*Remark:
The description of product quality attributes provided in ISO9126 is
used as a guide to describing software quality attributes. Other
standards, such as the ISO/IEC 25000 series, may also be of use.
Chapter 1
Page 15

V 1.0
Software Quality according to ISO/IEC 9126 (1)

Software Quality
External and
Internal Quality
Quality in Use
Effectiveness
Productivity
Security
Satisfaction
Fulfillment of tasks
within the
accuracy and
completeness
limits
within the
expenditure limits
for users (time,
costs, ...)
within the limits of
risk (life and
health,
business...)
Subjective user
satisfaction within
the context of use
ISO/IEC 9126: Rate of software products, quality characteristics

and guidance on usage
Chapter 1
Page 16

V 1.0
Software Quality according to ISO/IEC 9126 (2)

Software Quality
Quality in Use
Functionality
Reliability
Suitability
Maturity
Accuracy
Interoperability
Fault tolerance
(Data) Security
Compliance
Recoverability
Compliance
Usability
Understandability
Learnability
Operability
Attractiveness
Compliance
External and
Internal Quality
Efficency
Time behavior /
Performance
Resource
utilization
Compliance
Maintainability
Portability
Analyzability
Adaptivity
Changeability
Installability
Stability
Co-Existence
Testability
Replaceability
Compliance
Compliance
ISO/IEC 9126: Evaluation of software products, quality characteristics and guidance on usage
Chapter 1
Page 17

V 1.0
External Quality Attribute: Functionality (1)

Existence of functions with specified attributes.
These functions meet the specified or implied requirements.
sub-attributes of the quality attribute functionality are described
below and on the next page.
Suitability
The capability of the software product to provide an appropriate set
of functions for specified tasks and user objectives [ISO 9126].
Accuracy
The capability of the software product to provide the right or agreed
results or effects with the needed degree of precision [ISO 9126].
(Further sub-attributes of the quality attribute functionality follow)
Chapter 1
Page 18

V 1.0
External Quality Attribute: Functionality (2)

(Further sub-attributes of the quality attribute functionality)
Interoperability
The capability of the software to interact with one or more specified
components or systems [after ISO 9126].
(Data) Security
Attributes of software that bear on its ability to prevent
unauthorised access, whether accidental or deliberate, to programs
and data [after ISO 9126].
Compliance
Attributes of the software which demonstrate that the software
fulfills application-specific standards or agreements or statutory
directives or similar regulations. [after ISO 9126].
Note: this also applies to all other quality attributes (e.g. reliability)
Chapter 1
Page 19

V 1.0
External Quality Attribute: Reliability

Attributes that relate to the ability of the software to maintain a specified
level of performance under given conditions and for a specified time period.
Maturity
The capability of the software product to avoid failure as a result of
defects in the software [ISO 9126].
Fault tolerance
The capability of the software product to maintain a specified level of
performance in cases of software faults (defects) or of infringement
of its specified interface [ISO 9126].
Recoverability
The capability of the software product to re-establish a specified level
of performance and recover the data directly affected in case of failure
[ISO 9126].
Chapter 1
Page 20

V 1.0
External Quality Attribute Usability

Attributes that relate to the effort required to use, and on the individual
assessment of such use by a specific or implied group of users.
Understandability
The capability of the software product to enable the user to
understand whether the software is suitable, and how it can be used
for particular tasks and conditions of use [ISO 9126].
Learnability
The capability of the software product to enable the user to learn its
application [ISO 9126].
Operability
The capability of the software product to enable the user to operate
and control it [ISO 9126].
Chapter 1
Page 21

V 1.0
External Quality Attribute: Efficiency

Attributes that relate to the relationship between the performance
levels of the software and the amount of equipment used under
specified conditions.
Time behaviour / Performance
The degree to which a system or component accomplishes its
designated functions within given constraints regarding processing
time and throughput rate [after IEEE 610].
Resource utilization
The capability of the software product to use appropriate amounts
and types of resources, for example the amounts of main and
secondary memory used by the program and the sizes of required
temporary or overflow files, when the software performs its function
under stated conditions [after ISO 9126].
Chapter 1
Page 22

V 1.0
Internal Quality Attribute: Maintainability (1)

Attributes that relate to the effort that is necessary to carry out software
changes.
Analyzability
The capability of the software product to be diagnosed for deficiencies
or causes of failures in the software, or for the parts to be modified to
be identified [ISO 9126].
Changeability
The capability of the software product to enable specified modifications
to be implemented [ISO 9126].
(Further sub-attributes of the quality attribute maintainability follow)
Chapter 1
Page 23

V 1.0
Internal Quality Attribute: Maintainability (2)

(Further sub-attributes of the quality attribute maintainability)
Stability
The capability of the software product to avoid unexpected effects
from modifications in the software [ISO 9126].
Testability
The capability of the software product to enable modified software to
be tested [ISO 9126].
Chapter 1
Page 24

V 1.0
Internal Quality Attribute Portability (1)

Attributes that relate to the ability of software to be transferred from
one environment to another.
Adaptability
The capability of the software product to be adapted for different
specified environments without applying actions or means other
than those provided for this purpose for the software considered
[ISO 9126].
Installability
The capability of the software product to be installed in a specified
environment [ISO 9126].
(Further sub-attributes of the quality attribute portability follow)
Chapter 1
Page 25

V 1.0
Internal Quality Attribute: Portability(2)

(Further sub-attributes of the quality attribute portability)
Co-Existence
The capability of the software product to co-exist with other
independent software in a common environment sharing common
resources [ISO 9126].
Replaceability
The capability of the software product to be used in place of
another specified software product for the same purpose in the
same environment [ISO 9126].
Chapter 1
Page 26

V 1.0
Quality Requirements
Quality requirements state what quality attributes the product
should have at what quality level.
The sum of all quality attributes and the specific variants
Not all quality attributes can be achieved equally well.
E.g. efficiency may come at the expense of maintainability
Set priorities
In close consultation with clients and users.
Quality requirements (with the exception of functionality) are
part of the non-functional requirements in the specifications.
Chapter 1
Page 27

V 1.0
Testing and Quality

Testing measures the quality e.g. based on the number of located
failures and defects.
Testing increases the quality indirectly, as defects are detected
which can be corrected prior to delivery.
Testing increases the process quality indirectly, as defects are
documented which can then be analysed and help prevent similar
errors from taking place in the future.
Testing increases the confidence in the quality of the system when
few or no failures and defects are found.
Chapter 1
Page 28

V 1.0
Quality Assurance - QA
Quality Assurance
Analytic QA
Artefacts
Constructive QA
Processes
Audits
Software, etc.
Dynamic
Testing:
White-Box,
Black-Box
Chapter 1
Page 29
Software, Documentation,
etc.
Norms, standards,
project management,
software techniques,
exchange of experiences,
training
...
Static Testing:
Reviews,
Static Analysis

V 1.0
Chapter
Fundamentals
of Testing
Chapter 1
Page 30

Principles of Testing
Fundamental Test Process Test Process
Ethics of Testing

V 1.0
Seven Principles of Testing (1)

In the previous 40 years, the following seven testing principles have
emerged and can serve as guidelines:
Principle 1: Testing shows presence of defects
Testing can show that defects are present, but cannot prove that
there are no defects.
Program testing can be used to show the presence of bugs,
but never to show their absence! Edger W. Dijkstra, 1970
Principle 2: Exhaustive testing is impossible
Complete / exhaustive testing is not feasible except for trivial
programs (see slides in chapter 1 on exhaustive testing)
(continued)
Chapter 1
Page 31

V 1.0

Principle 3: Early testing
Testing is not a phase in the software development that happens at
the end; it shall be started as early as possible. Through early testing
(e.g. reviews) which take place parallel to the development activities,
defects are detected earlier and thus cost less to fix.
Principle 4: Defect clustering
Testing effort shall be focused proportionally to the expected and the
observed defect density of modules. A small number of modules often
contains most of the defects discovered during pre-release testing, or
is responsible for most of the operational failures.
(continued)
Chapter 1
Page 32

V 1.0

Principle 5: Repetitions have no effects - Pesticide Paradox
Just repeating tests brings no new insights. Test cases need to be
regularly reviewed, revised and modified.
Principle 6: Testing is context dependent
Testing is depending on context. For example, safety-critical software
systems are tested differently (more intensely and using other
techniques) from commercial applications.
Principle 7: Absence-of-errors fallacy
A system without failures does not imply that the system will meet the
users expectations.
Chapter 1
Page 33

V 1.0
Testing Effort?
Testing is economically useful, as long as the costs of finding and
fixing a defect in testing are lower than the costs that are
associated with the occurrence of a defect when used.
M. Pol, T. Koomen, A. Spillner:
Management and Optimization of the Test Process
(German). dpunkt-Verlag, 2. Edition, 2002
A good test is like a liability insurance: it costs big money, but

allows the project manager and the customer to sleep peacefully. A
good sleep includes a good insurance policy that covers all
possible risks. The trust in a software product includes a good test
that fully covers the reality of production.
Siedersleben, J. (Hrsg):
Software methodology: Practical Knowledge for
Software Engineers, (German). 2. Edition, Hanser,
2003
Successful testing (detection of failures) reduces costs

Chapter 1
Page 34

V 1.0
More regarding Testing (1)

Testing effort in practice:
25% to 50% of the development effort
Test intensity and scope depends on risks and the criticality of the
project, the system and the domain
Defects can result in huge costs:
Estimated losses due to software defects in medium and large
companies in Germany are approximately 84.4 billion Euros per
annum
Productivity losses due to computer failures caused by software
defects are approximately 2.6% of sales - 70 billion Euros per
annum
Study by LOT Consulting, Karlsruhe

IT-Services 3/2001, P. 31 (German)
Chapter 1
Page 35

V 1.0
More regarding Testing (2)

Testing is always subject to limited resources
(e.g. time, test personal)
Particularly important:
Select appropriate test procedures which are compatible with
the test objectives and quality objectives
Avoid unnecessary tests which provide no new information
Take positive and negative test conditions into account
It can also be useful to test for functionality that has not been
requested
Chapter 1
Page 36

V 1.0
Prioritization of Tests
Limited resources (time and personnel) require that the most important
test cases are carried out first!
Criteria for prioritization:
Expected impact (the severity of the damage a defect would cause)
Probability of occurrence of a failure
Combined consideration of impact and likelihood of occurrence
( Risk = Likelihood * Impact)
Perception of the failure
Prioritization of requirements by the customer
Importance of quality characteristics by the customer
Priority of test cases from the perspective of development (serious
consequences and / or complex cases first)
High project risk
Where many defects are found, more defects are likely to be
discovered! A change in the priority must be possible!
Chapter 1
Page 37

V 1.0
Chapter
Fundamentals
of Testing
Chapter 1
Page 38

Ethics of Testing

V 1.0
SW-Development Models: Waterfall-Model - 1970
Royce, W. W.:
Managing the Development of Large Software
Systems: Concepts and Techniques
Proc. WESCON, IEEE Computer Society Press, Los
Alamitos, CA, 1970.
Reprinted at the ICSE'87, Monterey, California, USA.
March 30 - April 2, 1987
Chapter 1
Page 39

V 1.0
SW-Development Models: Waterfall-Model - 1981
Boehm, B.:
Software Engineering Economics
Prentice-Hall Inc., London, 1981
Chapter 1
Page 40

V 1.0
V-Model (B. Boehm, 1979)
Barry W. Boehm:
Guidelines for Verifying and Validating Software
Requirements and Design Specification.
EURO IFIP 79, P.A. Samet (eds.) North-Holland, IFIP
1979, 711-719
Chapter 1
Page 41

V 1.0
General V-Model
Definition of Requirements
Acceptance Testing
Functional
System Design
System Testing
Technical
System Design
Integration Testing
Component
Specification
Component Testing
Programming
Key
Test cases based on
corresponding documents
Chapter 1
Page 42

V 1.0
Test Process
Is closely linked with software development
Is, however, a separate, independent process
A test plan is necessary for every test level (level test plan)
Testing cannot be considerd as a single activity (e.g. test execution)
Many individual tasks are performed within testing
The test process represents these individual testing activities in a coherent

process
Chapter 1
Page 43

V 1.0
Activities of the Fundamental Test Process

Start
Test Planning and Control

Test Analysis and Design
Planning and
Test Implementation and Execution

Evaluating Exit Criteria and
Reporting
Test Closure Activities
These activities may overlap or take
place concurrently.
The Fundamental Test Process is to
be tailored to each test level (e.g.
module test, system test)
Chapter 1
Page 44
Analysis and Design
Implementation and
Execution
Control
Evaluation and
Report
Closure
Finish

V 1.0
Start
Testing Tasks: Test Planning and Control (1)

Specification of Test Plan
Determine test objectives, amount and
risks of testing.
Specify test approach (techniques*,
test objects, coverage, teams who
participate in testing, testware).
Planning and
Analysis and
Design
Implementation
and Execution
Control
Evaluation and
Report
Test Closure
Details of Test plan

Finish
Test resources (e.g. staff, test environment, PCs).
How intensive should particular parts of the system be tested?
Which test techniques should be used?
Determine exit criteria. What level coverage should be reached?
Prioritizing of test (taking defect risks into consideration).
Planning for tool support (use, acquisition, training).
Test schedule and test strategy are to be recorded in the test plan
*refer to chapters 4 and 5
Chapter 1
Page 45

V 1.0
Start
Testing Tasks: Test Planning and Control (2)

Drawing up a rough schedule
Schedule tasks for test analysis and test
specification.
Schedule test implementation, test
execution and test evaluation.
Test Control
Measuring and analysing results.
Monitoring and recording of test progress,
achieved test coverage and exit criteria.
Initiate corrective actions.
Adapting time and resource planning.
Making decisions.
Refer to chapter 6 for more details
Chapter 1
Page 46
Planning and
Analysis and
Design
Implementation
and Execution
Control
Evaluation and
Report
Test Closure
Finish

V 1.0
Start
Testing Tasks: Test Analysis and Design (1)
Planning and
Analysis and
Design
General testing objectives are detailled to specific

Implementation
test requirements and criteria.
Control
and Execution
Basis are all documents showing the
Evaluation and
Report
requirements for the software (test basis).
Reviewing the test basis (requirements,
Test Closure
architecture, design, interface, ...).
Finish
Evaluating testability of requirements and the system.
Identifying test conditions / test requirements.
Designing the test environment (infrastructure, tools, ...).
Designing test cases using the chosen test design techniques.
Distinction to be made between high level test cases and detailed
test cases.
Creating bi-directional traceability between test basis and test
cases
Chapter 1
Page 47

V 1.0
Start
Testing Tasks: Test Analysis and Design (2)
Planning and
Analysis and
Design
Test cases contain more that just test data!

Implementation
Criteria and constraints that have to be met
and Execution
before execution (e.g. condition of test
Evaluation and
object, data, network status)
Report
Before test execution determine which result
Test Closure
or behavior is expected and the post-conditions,
that have to be fulfilled after the test, e.g. condition
Finish
of the test object and the database.
Refer to the next section in this chapter for further details
Control
Test cases are partly linked together to create test suites (test
procedure specification) for test execution
Each test case should only contain one or a few elementary
process steps or system calls
Pay attention in the test design to the post-conditions of the
test cases. These are the pre-conditions of next test cases.
Chapter 1
Page 48

V 1.0
Start
Testing Tasks: Test Implementation
Planning and
Analysis and
Design
Specifying and prioritizing test cases.

Creating test data and test scenarios.
Creating test suites.
Preparing test harnesses and (possibly) writing
automated test scripts.
Implementation
and Execution
Control
Evaluation and
Report
Test Closure
Finish
Verifying that the test system and test environment has been set
up correctly.
Before test execution: Verifying completeness of parts to be tested.
Chapter 1
Page 49

V 1.0
Start
Testing Tasks: Test Execution (1)
Planning and
Analysis and
Design
First check main functions. If any incidents

are found here, continuing with deeper tests
may not make much sense!
Execute test cases either manually or
by using test execution tools according to the
test procedure specification
Implementation
and Execution
Control
Evaluation and
Report
Test Closure
Finish
Record the test execution accurately and completely (versions of

test object, test tools and testware ) in the test log.
Chapter 1
Page 50

V 1.0
Start
Planning and
Analysis and
Design
Compare actual results with expected results.

Report discrepancies as incidents and analyze
them. If a discrepancy of actual and expected
result is detected, it may not always mean
a software defect has been detected.
Check if any of the following aspects is true:
Implementation
and Execution
Control
Evaluation and
Report
Test Closure
Finish
incorrect expected result

test environment not set up properly
the test case specification is incorrect
Determine defect severity and priority for fixing
Repeat test activities as a result of action taken to resolve each
incident/defect (Re-Testing).
Chapter 1
Page 51

V 1.0
Start
Planning and
Analysis and
Design
Create a test log
Implementation
and Execution
The test log must contain details on which

parts were tested, when, by whom,
how intensive and with what result.
On the one hand the description of test
execution in the test log has to be
comprehensive for those not directly involved
(e.g. clients)
Control
Evaluation and
Report
Test Closure
Finish
On the other hand it has to be traceable if the planned test strategy

has really been applied
Chapter 1
Page 52

V 1.0
Start
Testing Tasks: Evaluating Exit Criteria

Compare results of test execution with the
defined objectives of the test.
Checking test logs against the exit criteria
specified in test planning.
Reached test end?
Achieved coverage?
(Problem: unreachable program code)
Planning and
Analysis and
Design
Implementation
and Execution
Control
Evaluation and
Report
Test Closure
Finish
Deciding if more tests are needed or if the specified exit criteria

should be changed (only after consulting stakeholders!)
Further effort justified?
Addition practical factors for ending the test:
No more time
No more money
Chapter 1
Page 53

V 1.0
Start
Testing Tasks: Test Reporting (1)
Planning and
Analysis and
Design
Write a test summary report for

stakeholders.
Implementation
and Execution
The test summary report may have two

purposes:
Control
Evaluation and
Report
Test Closure
Communicate test status
Finish
Report on test completion

Refer to chapter 6 for more details on test reporting
Chapter 1
Page 54

V 1.0
Start
Planning and
Analysis and
Design
Example for exit criteria:

Found defects in order of failure severity
Implementation
and Execution
Control
Evaluation and
Report
New / Testing Hour

3
Test Closure
2,5
Finish
Failure Severity
1,5
low
medium
high
0,5
critical
0
W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 W11
Chapter 1
Page 55
Wx: Calendar Week

V 1.0
Start
Planning and
Analysis and
Design
Example for exit criteria:

Comparison of found failures and corrected defects
Implementation
and Execution
Control
Evaluation and
Report
60
Test Closure
50
Finish
40
30
New failures
20
Corrected defects
10
In Process
0
W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 W11
Chapter 1
Page 56
Wx: Calendar Week

V 1.0
Start
Testing Tasks: Test Closure Activities (1)
Planning and
Analysis and
Design
Collect data from completed test phases and

consolidate experience, testware, facts, metrics
etc.
Check which planned deliverables have been
delivered
Close incident reports or raise change requests
for any that remain open
Implementation
and Execution
Control
Evaluation and
Report
Test Closure
Finish
Document the acceptance of the system
Chapter 1
Page 57

V 1.0
Start
Testing Tasks: Test Closure Activities (2)
Planning and
Analysis and
Design
Finalize and archive testware, the

test environment and the test infrastructure
Preserve and hand over the testware to the
maintenance organization everything should
be reusable during maintenance, so it has to be
portable and updatable
Implementation
and Execution
Control
Evaluation and
Report
Test Closure
Finish
Analyse and document any lessons learned

for later projects and to improve test maturity
Evaluation of test process
Assessment of test process
Recognizing potential for improvement
Improve the test process by applying the
findings
Chapter 1
Page 58

V 1.0
Chapter
Fundamentals
of Testing
Chapter 1
Page 59

Test Cases, Expected Results, Test Oraclesch\
Ethics of Testing

V 1.0
Criteria for Test Cases

Test cases for verification of specified results and of test object
delivered results and reactions
Positive test; expected input / operation (normal)).
Test cases that verify the specified handling of exceptional
situations and defects also need to be considered
Negative test; expected false input / operation (exceptional).
Remark: It is often difficult to create the conditions necessary
for execution of negative test cases (e.g. the overload of a
network connection).
Test cases for verification of reactions of the test object for invalid
and unexpected inputs or constraints, for which there was no
exception handling specified
Negative test / robustness test; unexpected erroneous inputs /
operations (catastrophic).
Chapter 1
Page 60

V 1.0
Test Specification
High Level and Specific Test Cases (1)
Example:
A company has ordered a program that should calculate the
Christmas bonus of the staff in relation to the time they have been
working there. In the description of the requirements you find the
following text:
Staff who have been with the company for more that three years will
receive 50 % of the monthly salary as Christmas bonus. Staff who
have been with the company for more than five years will receive 75
%. Staff who have been with the company for more than eight years
will receive 100 % of their monthly salary.
How do the test cases look like?
Chapter 1
Page 61

V 1.0
Test Specification
From the text you can set up the following relationship between the
allowance of the bonus and the time working for the company:
Chapter 1
years with the company <= 3
equals bonus = 0 %
3<
equals bonus = 50 %
5<
equals bonus = 75 %
years with the company
equals bonus = 100 %
Page 62
>8

V 1.0
Test Specification
High Level (logical)
Test Case
x<=3
3<x<=5
5<x<=8
x>8
Expected result
(bonus in %)
50
75
100
Specific Test Case
Input value x
(years with the company)
12
Expected result
(bonus in %)
50
75
100
Input value x
(years with the company)
Remarks:
no pre- and post-conditions or constraints are considered
the test cases were not derived systematically
only positive tests with expected results
Chapter 1
Page 63

V 1.0
Expected Results and Test Oracle

After each excecuted test case it must be decided whether there is
a failure or not.
For this decision the monitored result (actual result/behavior) has
to be compared with the expected result (expected
result/behavior).
The expected behavior of the test object has to be determined in
advance for each test case.
The tester must obtain this information from appropriate sources
when specifying a test case.
In this connection we also speak of an oracle or test oracle that
can be consulted and that predicts the expected results.
Chapter 1
Page 64

V 1.0
Expected Results and Test Oracle

The expected results are to be deducted of the specification.
Following three possibilities:
The tester derives the expected date from the input date on the basis
of the specification of the test object. This is the common practise.
If a formal specification is available, an executable prototype can be
created with tool support. The results of the prototype can be used
as a test oracle for testing the actual program.
Parallel designs of the software program may be created by different
development groups. Each version of the program will then be tested
against another version using the same test data. If there are
different results in the two versions there must be a failure in one of
the versions. Only those failures that have the same effect in all the
versions remain undetected. This procedure is called Back-to-back
test.
Chapter 1
Page 65

V 1.0
Other Test Oracles

Making use of user manuals
The system itself (create operational profile)
This is the worst possibility
(Tested!) previous versions
Programs with similar functionality
Exact values cannot always be predicted or calculated. Determine
range of tolerance, verify plausibility
Experience is important
Chapter 1
Page 66

V 1.0
Chapter
Fundamentals
of Testing
Chapter 1
Page 67

Test cases, Expected Results, Test Oracle
Ethics of Testing

V 1.0
Making mistakes is human
- but who likes admitting that?
Development = constructive
Test = destructive?
Testing is an extremely creative
and intellectually challenging task
Myers, Glenford J.:

Systematic Testing of Programs
Oldenbourg, 2001 (7. Edition)
Is it possible for developer to test

his own program?
What to you think?
Chapter 1
Page 68

V 1.0
Psychology of Testing Developer Test

Being blind to see own mistakes
If the developer has implemented a basic design error, e.g.
because he has misunderdstood the task, he is not able to
detect this through his own tests.
He will not be able to specify appropriate test cases.
No familiarisation
The developer knows his test object
Chapter 1
Page 69

V 1.0
Psycholoy of Testing - Independent Test Team
An independent test team is unbiased

It his not his/their product
The possible assumptions and misunderstandings of the developer
are not necessarily the assumptions and misunderstandings of the
tester.
Need for familiarization

To create test cases the tester needs to gain knowledge about the test
object. This requires time.
Test Know-how
The tester should have test know-how
A developer probably does not have this or first has to gain it (often
with not enough time available)
Even better, the know-how has already been acquired at university
Chapter 1
Page 70

V 1.0
Psychology of Testing Possible Levels

Tests are carried out with different levels of independence
By the developer himself
By colleagues of the developer in the same project
By persons of other departments
By persons of other organizations.
Tools can be used to support all possibilities.
The selection depends on the product as well as the project.
It is important to get the right mix and balance between
independent testing and tests performed by the developer.
Chapter 1
Page 71

V 1.0
Psychology of Testing Communication of Defects

Communication of identified defects or incidents to the developer
and/or the management requires
neutral, fact-focused and constructive way of communication
Undisturbed, open communication.
Proving another persons error is not an easy task. It requires good
interpersonal skills.
Reproduceability of defects is important!
Recording the test environment
Differences to the development environment
Clear requirements, precise specification
Its not a bug, its a feature!
Mutual understanding between testers and developers
Collaboration rather than battles!
Chapter 1
Page 72

V 1.0
Chapter
Fundamentals
of Testing
Chapter 1
Page 73

Test cases, Expected Results, Test Oracle
Ethics of Testing

V 1.0
Ethics of Testing (1)
Software testers often have access to confidential and legally privileged

information. This must not be used inappropriately.
Recognizing the ACM and IEEE code of ethics for engineers, the ISTQB
states the following code of ethics:
1. PUBLIC
Certified software testers shall act consistently with the public interest.
2. CLIENT AND EMPLOYER
Certified software testers shall act in a manner that is in the best interests
of their client and employer, consistent with the public interest.
Chapter 1
Page 74

V 1.0

3. PRODUCT
Certified software testers shall ensure that the deliverables they provide
(on the products and systems they test) meet the highest professional
standards possible.
4. JUDGMENT
Certified software testers shall maintain integrity and independence in
their professional judgment.
5. MANAGEMENT
Certified software test managers and leaders shall subscribe to and
promote an ethical approach to the management of software testing.
Chapter 1
Page 75

V 1.0

6. PROFESSION
Certified software testers shall advance the integrity and reputation of
the profession consistent with the public interest.
7. COLLEAGUES
Certified software testers shall be fair to and supportive of their
colleagues, and promote cooperation with software developers.
8. SELF
Certified software testers shall participate in lifelong learning regarding
the practise of their profession and shall promote an ethical approach to
the practice of the profession.
Chapter 1
Page 76

V 1.0
Summary
Failure Not fulfilling the requirements!
Describing this situation as failure
cause: fault/defect in the software
that has been caused by an error made by a person
Tests are important measures for securing the quality criteria
Functionality, reliability, usability, efficiency, maintainability und
portability [according to ISO 9126]
Exhaustive testing is not possible. Tests are always random
checks, therefore they can detect failures only with a certain
probability
The intensity and amount of testing depends on the expected risk
of faulty behavior of the program
Tests should be prioritized
Chapter 1
Page 77

V 1.0
Summary
The fundamental test process consists of the main activities

Test Planning and Control
Test Analysis and Design
Test Implementation and Execution
Evaluating Exit Criteria and Reporting
Test Closure Activities
A test case consists of

an input value and
an expected result as well as
preconditions and constraints for the execution of the test case, and
postconditions that have to be fulfilled after the test case.
While executing the test case, the test object shows its actual behavior. If there is a discrepancy
between expected and actual behavior there is a failure.
A test oracle determines the expected values for each test case before test execution
Humans make mistakes but they do not like admitting them!
Chapter 1
Page 78

V 1.0
By now, you should be able to answer the following

??? questions
Define the terms software failure, defect, error.
What is defect masking?
Explain the difference between testing and debugging.
Define the terms verification and validation.
Explain why tests cannot be exhaustive.
Name the main software quality attributes according to ISO 9126.
Define the term reliability of a system.
Explain the main activities of the fundamental test process.
What is a test oracle?
Why should developers not test their own programs?
Chapter 1
Page 79

V 1.0
Finally
The process cannot be stopped. The process cannot be stopped.
Installation
Copy file:
Cancel
Sort results
1 entry was found.

Would you like to sort the results?
Yes
Chapter 1
Page 80
No

V 1.0

CTFL Chapter 1 Fundamentals

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

CTFL Chapter 1 Fundamentals

Uploaded by

Copyright:

Available Formats

1

Software Testing Foundations

After this lecture you should.

Be able to distinguish between the cause of a defect and its effects;

Derive by examples why testing is necessary;

Know the quality characteristic according to ISO 9126;

Recall the general objectives and principles of testing;

Differentiate between testing and debugging;

Know how the fundamental test process looks like;

Describe how expected values are calculated during testing;

Know and characterize the psychological problems during the testing;

Differentiate between the different mindset of a tester and a developer;

Know the code of ethics of a software tester.

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB

Learning Objectives for Fundamentals of Testing

1.1 Why is Testing Necessary? (K2)

Give reasons why testing is necessary by giving examples (K2)

1.2 What is Testing? (K2)

Recall the common objectives of testing (K1)

Provide examples for the objectives of testing in different phases of the

Differentiate testing from debugging (K2)

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB

Learning Objectives for Fundamentals of Testing

(according to the Certified Tester Foundation Level Syllabus, Version 2011)

1.3 Seven Testing Principles (K2)

1.4 Fundamental Test Process (K1)

Explain the seven principles in testing (K2)

1.5 The Psychology of Testing (K2)

Contrast the mindset of a tester and of a developer (K2)

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB

Terms you should be familiar with

Software Testing Foundations Certified Tester

Master Test Plan

Fundamental Test Process

Test Summary Report

Copyright 2011 to MSTB/GTB

Terms and Motivation

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB

In General: What is a Defect or a Deficiency?

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB

Error Other Definitions

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB

Testing Definition and Goals

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB

Validation and Verification - Definitions

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB

What is Software Quality?

Software Testing Foundations Certified Tester