Software Faults and Failures

Page 1
Ivica Crnkovic
Mlardalen University
Department of Computer Engineering
ivica.crnkovic@mdh.se
Mdh/IDt, Ivica Crnkovic
SE course, Testing
Page 1, 2001-01-14
Software faults and failures

? What does it mean that the software has failed?
?
The software does not do what the requirements describe
? Reasons to the failures

?
Requirements incomplete, inconsistent, impossible to implement
Wrong interpretation of the requirements
The system design may contain a fault (misunderstanding

requirements, wrong architecture, incomplete design)
Program design may contain a fault - wrong algorithm
The program code may be wrong - improper, incomplete,

inconsistent implementation
Documentation can be wrong - can describe incorrectly the

behavior of the system
SE course, Testing
Page 2, 2001-01-14
We reserve all rights in this document and in the information contained therein. Reproduction, use or disclosure to third parties without express
authority is strictly forbidden. MdH/IDt Ivica Crnkovic
Page 2
Some other reasons to the failures

? Integration problems
?
The components integrated into a systems are not compatible

(wrong components, wrong component versions)
? The performance of the entire system does not correspond to the

requirements (expectations)
? The system requires too much resources
? Not failures but still unsuccessful
?
Bad Graphical User Interface
Complicated process
Missed points with requirements (users actually wanted something

else)
SE course, Testing
Page 3, 2001-01-14
Defect classifications
? What type of defect?

? When it occurred (when failure, when fault?)
? Severity level?
? What is the foot cause of the defect?
? .
SE course, Testing
Page 4, 2001-01-14
Page 3
Fault classification (HP)

ORIGIN: WHERE?
TYPE: WHAT?
Specification/
requirements
Requirements
or
specifications
Design
HW interface
SW interface
User interface
Functionality
Environment/
support
Code
(Inter-)Process
communications
Functional
description
Other
Logic
Test HW
Computation
Test SW
Data handling
Integration SW
Module
interface/
implementation
Development
tools
Data definition
Module design
Logic
description
Documentation
Error checking
Standards
Standards
MODE:
Missing
WHY?
Unclear
Wrong
Changed
Better way
SE course, Testing
Page 5, 2001-01-14
What are the most frequent faults
Other code
11%
Example:
Faults statistics (HP)
Data handling
6%
Documentation
19%
Computation
18%
Requirements
5%
Logic
32%
SE course, Testing
Hardware
4%
Process/interpr
ocess
5%
Page 6, 2001-01-14
Page 4
Costs to repair the failures

$$$ to fix the problems
Origin of problems
Specification/
requirements
Design
Code
SE course, Testing
Integration
Page 7, 2001-01-14
How to avoid a late detection of the faults?
? Use the development process that suits the best

? Do proper planning
? Ensure the quality in each process step
?
perform reviews
perform tests
SE course, Testing
Page 8, 2001-01-14
Page 5
V Model - document produced

Acceptance test
description
REQUIREMENTS
ANALYSIS
RS
OPERATION
& MAINTENANCE
Validate requirements
SYSTEM
DESIGN
ACCEPTANCE
TESTING
Verify design
System design
test description
SD, FS
SYSTEM
TESTING
PROGRAM
DESIGN
UNIT & INTE-
DS, Code
GRATION TESTING
Module test
description & code
Test analysis report.
CODING
SE course, Testing
Page 9, 2001-01-14
The testing process
Uni t
testing
Module
testing
Sub-system
t esting
System
testing
Acceptance
testing
Component
testing
Integration test ing
SE course, Testing
User
testing
Page 10, 2001-01-14
Page 6
Other
software
requirements
Customer
requirements
specification
Performance
test
Acceptance
test
User
environment
ponen
Unit
test
Design
System
specifications functional
requirements
d com
Unit
test
Teste
Component code
The testing process - details
Function
test
Installation
test
t
ponen
d com
test
Integration
test
Integrated
modules
Functioning
system
Verified,
validated
software
Teste
Component code
.
.
.
Unit
SE course, Testing
Accepted
system
SYSTEM
IN USE!
Page 11, 2001-01-14
Who performs the tests?

? Program authors (module tests)
? The developer group (module tests, component tests)
? Test teams
?
Integration tests, subsystem tests
System tests
Big configuration test
? Special customers (running alpha, beta releases)

Development
Alpha releases
Beta releases
Official
release
New functions
Fixing defects
? Customers with support people (acceptance tests)

SE course, Testing
Page 12, 2001-01-14
Page 7
Unit Testing
Different methods to test the code
? Code review - go trough the code
?
Code author, or a review group in the developer team (peer

review)
? Code inspection
?
Inspect the code against a prepared list of concerns (data-types,

style guide)
? Proving code correct

?
Formal verification
? Choosing test cases

?
Inputs to test the system and the predicted outputs from these
inputs if the system operates according to its specification
SE course, Testing
Page 13, 2001-01-14
Black-box testing
Input test data
Inputs causing
anomalous
behaviour
? Approach to testing where the

program is considered as a blackbox
? The program test cases are based
on the system specification
System
? Test planning can begin early in

the software process
Output test results
Oe
Outputs which reveal

the presence of
defects
SE course, Testing
Page 14, 2001-01-14
Page 8
Interface testing
? Takes place when modules or sub-systems are integrated to
create larger systems
? Objectives are to detect faults due to interface errors or invalid
assumptions about interfaces
? Particularly important for object-oriented development as objects
are defined by their interfaces
Test
ca se s
The test is applied to sub-systems,

not to individual modules
SE course, Testing
Page 15, 2001-01-14
Interfaces types
? Parameter interfaces
?
Data passed from one procedure to another
? Shared memory interfaces

?
Block of memory is shared between procedures
? Procedural interfaces
?
Sub-system encapsulates a set of procedures to be called by

other sub-systems
? Message passing interfaces

?
Sub-systems request services from other sub-systems
SE course, Testing
Page 16, 2001-01-14
Page 9
White-box testing (structural testing)

? Derivation of test cases according to program structure.
Knowledge of the program is used to identify additional test
cases
? Objective is to exercise all program statements (not all path
combinations)
Test d at a
Test s
Deri ves
Com po nent
cod e
Test
o ut puts
SE course, Testing
Page 17, 2001-01-14
Path testing
? A white-box testing
? Testing different combinations of program paths
? Measuring the coverage percentile of the test
i f-th en -e ls e
l oo p-wh ile
SE course, Testing
case -o f
Page 18, 2001-01-14
Page 10
Integration testing
? Test on the sub-systems levels
? Testing how the components work together
SE course, Testing
Page 19, 2001-01-14
Integration test - bottom-up principle
Test
E
Test
F
Test
G
Test
B,E,F
Test
C
Test
A,B,C,D,
E,F,G
A
Test
D,G
B
E
SE course, Testing
C
F
D
G
Page 20, 2001-01-14
Page 11
Top-down testing
? Test from the top level
? all components are not necessary completed (and tested)
?
writing STUBS (which simulates the component)
Test
A
Test
A,B,C,D,
E,F,G
Test
A,B,C,D
A
B
E
SE course, Testing
Page 21, 2001-01-14
Combination of Top-down and Down-top model

Test
B
Test
A
Test
C
Test
E
Test
A,B,C,D
Test
F
Test
A,B,C,D,
E,F,G
A
Test
D
Test
G
B
E
SE course, Testing
C
F
D
G
Page 22, 2001-01-14
Page 12
Sandwich Integration
Test
E
Test
B,E,F
Test
F
Test
A,B,C,D,
E,F,G
Test
D,G
Test
G
A
B
Test
A
SE course, Testing
C
F
D
G
Page 23, 2001-01-14
Microsoft synch-and stabilize approach

Milestone 1: Most critical features
and shared components
Design, code, prototype
Usability testing
Daily builds
Feature integration
Eliminate severe faults
Milestone 2: Desirable features
Usability testing
Daily builds
Feature integration
SE course, Testing
Milestone 3: Least critical features

Usability testing
Daily builds
Feature integration and completion
Page 24, 2001-01-14
Release to manufacturing
Page 13
Object-oriented systems testing

? Less closely coupled systems. Objects are not necessarily
integrated into sub-systems
? Cluster testing. Test a group of cooperating objects
? Thread testing. Test a processing thread as it weaves from object
to object. Discussed later in real-time system testing
? Complication with inheritance
SE course, Testing
Page 25, 2001-01-14
Object-oriented systems testing

Harder parts of testing
Easier parts of testing
Modularity
Inheritance
Polymorphism
Small methods
Reuse
Encapsulation
Dynamic binding
Complex Interfaces
Interfaces
identified earlier
More integration
SE course, Testing
Page 26, 2001-01-14
Page 14
Automated testing tools

? Static Analyzers
?
Code analysis
? Show some potential program errors and low quality code (too many
branches, complicated structure, bad naming policy
? Generate different graphs showing the program structure
? Data analyzer (initialization, usage, etc.)
? Dynamic Analyzers
?
Checking a running program
Memory leakage
non proper use of variables
Measuring timing
Measuring execution coverage
SE course, Testing
Page 27, 2001-01-14
When to stop with testing?
Probability
of existence
of additional
faults
Number of faults found to date

SE course, Testing
Page 28, 2001-01-14
Page 15
Incorrect, missing or unclear requirements

Incorrect or unclear translation
Requirements
analysis
Incorrect or unclear design specification
System
design
Incorrect or unclear design specification

Misinterpretation of system design
Program
design
Misinterpretation of program design

Incorrect documentation
Incorrect syntax or semantics
Program
implementation
Incomplete test procedures

New faults introduced when old ones corrected
Unit/integration testing
System testing
Incomplete test procedures

Incorrect user documentation
Poor human factors
New faults introduced when old one corrected
Changes in requirements
Maintenance
Integrated
modules
SE course, Testing
Page 29, 2001-01-14
System
functional
requirements
Other
software
requirements
Customer
requirements
specification
Function
test
Performance
test
Acceptance
test
Functioning
system
Verified,
validated
software
SE course, Testing
User
environment
Installation
test
SYSTEM
IN USE!
Accepted
system
Page 30, 2001-01-14
Page 16
TEST SPECIFICATION
TEST 1
Requirements tested
TEST PLAN
TEST ANALYSIS
REPORT
Test procedures
1.
Methods
2.
Conditions
1
3,4
TEST 1
Results
Perform
test 1
Functions tested
SYSTEM TEST
FUNCTION
Function 1
Function 2
.
.
TEST DESCRIPTION
TEST 1
Test data
TEST SPECIFICATION
TEST 2
Requirements tested
TEST DESCRIPTION
TEST 2
Test data
TEST ANALYSIS
REPORT
Test procedures
1.
Methods
TEST 2
Results
Perform
test 2
Functions tested
2.
Conditions
SE course, Testing
Page 31, 2001-01-14
FAULT REPORT
S.P0204.6.10.3016
ORIGINATOR:
Joe Bloggs
BRIEF TITLE:
Exception 1 in dps_c.c line 620 raised by NAS
FULL DESCRIPTION
Started NAS endurance and allowed it to run for a few minutes. Disabled the active NAS
link (emulator switched to standby link), then re-enabled the disabled link and CDIS exceptioned as above. (I
think the re-enabling is a red herring.)
(during database load)
ASSIGNED FOR EVALUATION TO:
DATE:
CATEGORISATION:
0 1 2 3 Design Spec Docn
SEND COPIES FOR INFORMATION TO:
EVALUATOR:
CONFIGURATION ID
dpo_s.c
DATE:
8/7/92
ASSIGNED TO
PART
COMMENTS: dpo_s.c appears to try to use an invalid CID, instead of rejecting the message. AWJ
ITEMS CHANGED
CONFIGURATION ID
dpo_s.c v.10
IMPLEMENTOR/DATE
AWJ 8/7/92
REVIEWER/DATE
MAR 8/7/92
BUILD/ISSUE NUM
6.120
INTEGRATOR/DATE
RA 8-7-92
COMMENTS:

FAULT CONTROLLER:
CLOSED
SE course, Testing
Page 32, 2001-01-14
DATE: 9/7/92
Page 17
DISCREPANCY REPORT FORM

DRF Number:__________________________________________
Tester name:____________________________
Date: ___________________________________
Time: ________________________________
Test Number: ______________________________
Script step executed when failure occurred: __________________________________________________________
Descripton of failure: ________________________________________________________________________
_______________________________________________________________________________________
_______________________________________________________________________________________
_______________________________________________________________________________________
Activities before occurrence of failure:
_______________________________________________________________________________________
_______________________________________________________________________________________
Expected results:
_______________________________________________________________________________________
_______________________________________________________________________________________
Requirements affected:
_______________________________________________________________________________________
_______________________________________________________________________________________
Effect of failure on test:
_______________________________________________________________________________________
_______________________________________________________________________________________
Effect of failure on system:
_______________________________________________________________________________________
_______________________________________________________________________________________
Severity level:
(LOW)
1
2
3
4
5 (HIGH)
SE course, Testing
Page 33, 2001-01-14

Software Faults and Failures

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Software Faults and Failures

Uploaded by

Copyright:

Available Formats

Page 1

Mdh/IDt, Ivica Crnkovic