Software Quality Assurance

For
Software Engineering
&&
Architecture and Design

Software Quality Assurance

What is quality?

Software Quality Assurance

What is quality?
IEEE Glossary: Degree to which a system,
component, or process meets (1) specified
requirements, and (2) customer or user needs
or expectations

Software Quality Assurance

What is quality?
IEEE Glossary: Degree to which a system,
component, or process meets (1) specified
requirements, and (2) customer or user needs
or expectations
ISO: the totality of features and
characteristics of a product or service that
bear on its ability to satisfy specified or
implied needs

Software Quality Assurance

An alternate view of Quality:

is not absolute
is multidimensional, can be difficult to quantify
has aspects that are not easy to measure
assessment is subject to constraints (e.g., cost)
is about acceptable compromises
criteria are not independent, can conflict

Software Quality Assurance

Quality Criteria include:

correctness
efficiency
flexibility
integrity
interoperability
maintainability
portability
reliability
reusability
testability
usability

What is Software Quality Assurance

(SQA)?
Set of systematic activities providing
evidence of the ability of the software
process to produce a software product that
is fit to use
G. Schulmeyer and J. McManus, Software
Quality Handbook, Prentice Hall, 1998.

What is SQA?
Monitoring processes and products throughout
the software development lifecycle to ensure
the quality of the delivered product(s)
Monitoring the processes
Provides management with objective feedback
regarding process compliance to approved plans,
procedures, standards, and analyses

Monitoring the products

Focus on the quality of product within each phase
of the SDLC
e.g., requirements, test plan, architecture, etc.

Objective: identify and remove defects throughout

the lifecycle, as early as possible

Quality of Software developed in-house

& COTS components
SQA processes apply when integrating
purchased or customer-supplied software
products into the developed product
Question. How do you determine the
quality of COTS components?
Current research problem

Process Assessment
Use of standards and process models has a positive
impact on the quality of the software product
Disciplined, controlled development process

Examples include:
ISO 9001
CMM
CMU SEI, 5 levels

SPICE
Developing a standard for software process assessment
ISO joint committee, Europe, Australia

IEEE 1074, IEEE 12207,

Product Assessment
Reviews, inspections, walkthroughs
Specialized techniques available:
How to review/assess requirements, architecture,
detailed designs, code

Testing
Simulation
Protoyping
Formal verification
Model checking, theorem proving

Product Assessment
Reviews, inspections, walkthroughs of
Plans, reports, models, standards
Project management, quality assurance, training,
test plan(s)
Requirements, analysis, architecture, detailed
design model, test cases
Issue or problem reports
Metric reports
Traceability reports
Documentation, coding standards

Software Reviews
They may include managerial reviews, acquirer-supplier reviews,
technical reviews, inspections, walkthroughs, and audits.
Inspection:
A formal evaluation technique in which an artifact (e.g., software
requirements, design, or code) is examined in detail by a person or group
other than the originator
detect faults, violations of development standards, and other problems.
review members are peers (equals) of the designer or programmer.
data is collected during inspections for later analysis and to assist in future
inspections.
Note
Introduced by Fagan, 1976.
Fagan, M., Design and Code Inspections to Reduce Errors in Program
Development, IBM Systems Journal, 15, 3 (1976), pp. 182-211
Fagan, M., Advances in Software Inspections, IEEE Transactions on Software
Engineering, 12, 7(July 1986), pp. 744-751

Picture from Inspections presentation

http://www.math.uaa.alaska.edu/~afkjm/cs470/handouts/inspections.pdf

Defect Checklists
Useful to support reviews, inspections, walkthroughs
Expertise is captured in a list format
Less experienced people can use
Straightforward to use (each check should be clear, simple to assess/apply)

Improve consistency of assessments

Example architecture checklist used in undergrad./grad. courses

for OO
spreadsheet in in the course materials subdirectory

One or more architectural styles are selected.

Capabilities and interfaces are defined for subsystems.
Capabilities of and interfaces among subsystems support all of the use cases.
Concurrency defined.
Distribution defined.
Error handling defined.
Start up and shut down defined.
Data persistency defined.
Rationale for the model is provided.
Other

Verifying Formal Specifications

Formal specifications may be verified in a number of
different ways:
More
Syntax, typechecking
straightforwar
If the notation is typed
d
Simulated
Model checked (e.g., SPIN)
Less
Proven correct (e.g., HOL, PVS)
straightforwar
d
More straightforward? Less assurance of correctness; fully
automated
Less straightforward? Higher assurance of correctness; not
fully automated

Problem Reporting, Tracking, and Resolving

Describe the practices and procedures to be followed
for reporting, tracking, and resolving problems

Who can report a problem?

How is it reported?
How is is tracked?
Who determines if it is a problem that going to be resolved?
How is it assigned for resolution?
How does the person indicate it has been corrected?
Who reviews it to determine if it can be closed?

Problems can be product or process related

e.g. incorrect requirement, incomplete class definition, code
defect, ambiguous description in user documentation,
process to review detailed design is not clearly defined, etc.

Metrics
Metrics for each artifact
e.g., Requirements
Number of requirements
Number of changes per requirement
Called churn rate

Characterization of defects
Not testable, ambiguous, inconsistent, incorrect,
incomplete redundant, infeasible,
Major or minor defect
Phase defect detected
Cost to fix

Tools, techniques, training

What tools?
e.g., CVS for CM, excel spreadsheet for
problem reporting/tracking, ...

What techniques?
e.g., formal peer review for deliverables,
checklists for defect detection, ...

What training is needed on tools,

techniques?

Media Control
Identify the media for each intermediate and
deliverable artifact
Documentation required to store the media, including
the backup and restore process
Protect computer program physical media from:
unauthorized access
inadvertent damage
degradation

Architecture Analysis Methods

Why evaluate an architecture?
http://www.slideshare.net/kevinjew/evaluating-softwarearchitectures-presentation

Specialized techniques available:

http://www.slideshare.net/timmenzies/architecturetradeoff-analysis-method-presentation
SEI presentation and technical report on ATAM are in the
course subdirectory