Quantitative and qualitative methods in process improvement

and product quality assessment.

Anna Bobkowska

Abstract
Successful improvement of the development process and product quality assurance should
take advantages of complementary use of both quantitative and qualitative methods. In the
paper, experience of such integrated activities during students quality lab is presented.

1. Introduction
One of the recommendations of the TQM (Total Quality Management) says: Measure
your success! [4] However things defined by business case or external problems such as
cancelled projects, late deliveries, exceeded budget, and useless software are just a tip of the
iceberg containing issues of the complex project reality. In order to make improvements
effectively good understanding of those subtitles is needed. One can argue that quantitative
methods are not sufficient for the purpose of successful software quality assurance. Intentional
and systematic use of qualitative methods can provide missing part of the solution.

2. Complementary applications of quantitative and qualitative methods

Quantitative and qualitative issues are strictly related, but in software engineering
qualitative methods are treated rather informally. Comparison of both approaches in aspects of
techniques and applications is presented in the figure 1.

Quantitative methods
Measurement
Quantification
Computations

Applications
Statistical results
Control and management
Comparison of similar products
Integration with other indicators

Qualitative methods
Comments, notes and schemas
Discussions and observations,
Questionnaires, interviews
Analysis, reasoning
Applications
Problem identification and removal
Theory generation and improvement
Understanding of project reality
Enhancement of the measurement structure
Cross-case analysis
Focus on details, complexity
Non-technical aspects

Figure 1. Comparison of quantitative and qualitative methods in aspects of techniques

and applications.

2.1. Quantitative methods

Quantitative methods use numbers. Measures are easy to take when there are countable
objects e.g. number of persons, number of elements of given type on the diagram, or there are
standard devices or agreements for objective measuring a given feature, e.g. time, cost. In the
case when those conditions are not satisfied but a number that represents a state according to
the defined goal is needed, it is necessary to quantify that feature. Examples include: ease of
use, understandability, and cover all the cases when it is very difficult to find objective metrics.
Quantification is concerned with assigning numbers to the states of reality with appropriate
rules of representation [3]. During quantification there is a problem of subjectivity and
imprecision of the achieved data although they are represented by numbers.
Quantitative methods are necessary for comparison of similar products, control and
management. Quantitative data easily deliver statistical results and can be processed
automatically. Another advantage of their use is ease of integration with systems covering
other indicators. It is important to define goals of measurement and prepare an appropriate
measurement plan. This can be done with GQM (Goal-Question-Metric) [2] approach or FCM
(Factor-Criteria-Metrics) structure [3] or Goal-Subgoal-Metrics structure [6].
In application of software quality defect density metric is used. Although defects are
countable ideas, it is not fully objective metric since defect classifications are made by humans.
Apart from the problem of not discovered defects, importance of the defect can differ in
cases when it has occurred a serious defect, a mistake or just a misunderstanding.
Another approach is concerned with FCM structure and making computation in order to
achieve numbers that represent higher level quality indicators. This is usually more total
approach which allows to involve non-technical (e.g. human) aspects of quality. This approach
is used and extended in the presented research.

2.2. Qualitative methods

Qualitative methods use objects different then numbers e.g. words, pictures. The techniques
include interviews, discussions, observations, comments, notes, questionnaires, and schemas.
Usually the results are more subjective and more difficult to process, and thus require more
work during analysis. But they give richer and more informative results.
Quantitative methods can be applied to theory generation and improvement, problem
identification and removal, and cross-case analysis [8]. They are necessary in the situations
when better reality or system understanding is required, and complexity, not reductions and
quick processing are a value. Searching for reasons of things and non-technical aspects are
good fields for them. Although they seem to be intuitive their application also requires
infrastructure preparation and special methods of results analysis.
Qualitative methods can be helpful in optimising a structure of measurements. On the other
hand, metrics can provide assessments or problem indication to focus on with qualitative
methods. After identification of the reasons of problems and their removal quantitative
methods can be used again to track the project.

3. Development process improvement

There is a relationship between development process parameters and software quality.
While attempting to achieve high quality products one should provide also optimal
development process. Optimal process being goal of this research is characterised by:
An appropriate schedule, in which deadlines are set to avoid late deliveries, and there is
unified weekly workload during all the semester.
Infrastructure for efficient co-operation between participant with different roles in the
project is provided, and there are some facilitators for internal organisation in the group,
e.g. task sequencing and work distribution.
Product templates for each phase are defined to fit the type of the system and avoid overdocumenting.
Before the changes, OMT [7] method without measurements and reviews was used. There
were problems with late deliveries, poor quality products, and students complaining about
workload. But there was no base to verify the workload and find reasons of problems. So, the
development process was exchanged to a more controlled process described in brief in section
3.1, and then it was optimised with use of combination of quantitative and qualitative methods
in the three following years.

3.1. Making a difference

Any set of improvement activities must be based on the development process definition.
Software development process, in this case, was tailored for Internet applications and covered
the phases of requirements specification, object-oriented analysis, design, implementation and
testing. The products of the analysis and design were documented with UML [5] and
structured text. Implementation technology was a design decision. Additionally the method of
playing roles by the groups of project participants [1] was introduced. There were the
following roles:
Developer - responsible for the project and the delivery of products,
Customer responsible for the problem domain, and
Quality expert - responsible for quality assurance.
It was assumed that groups playing the roles co-operate in partnership. The process included
reviews after each phase, and technical meeting of all participants after requirements
specification review phase.
Measurement plan was established. Collected data contained information about: time of
work spent on project activities by the participants, late deliveries and evaluation of products
quality delivered by the developer (self-evaluation), the quality expert, and teacher. Apart from
that, participants described way of work distribution, internal communication, and main
problems concerned with the phase. Those comments as well as observations and discussions
were used to find problems and look for solutions.
Introduction of the measurements and comments revealed author from relaying on opinions
and beliefs, and allowed to rely on facts. Metrics easily verified concepts, for example, real
time of work, quality of products, and scale of late deliveries (if those were problems of all
groups or just an exception!). On the other hand, descriptions gave an understanding of
activities made by students and their problems. A lot of problems were concerned with poor
organisation, misunderstanding, and had simple solutions. But there were also serious
problems that needed more advanced activities. In the task of searching for suggestions that
facilitate organisation or explanations that allow to better understand project issues, metrics
and descriptions made complementary work.

3.2. Process optimisation

Optimisation activity aimed to change process parameters in these places where problems
occurred. Collected data enabled tracking results of the new development process. Number of
weeks set for the development tasks, average time of work per group and deviation that
indicates how the results differ among groups are presented in the table 1. Symbols of *
indicate different development tasks made during the same period of time, e.g. in the first
iteration there was phase of design that covered system design and component design. Data
are taken from four best projects for each iteration.
Table 1. Number of weeks for a development task (#weeks), average time of work (ATW)
and deviation (d) in the iterations.
Development tasks
#weeks ATW d
#weeks ATW d #weeks ATW d
in iter.1
in iter.2
in iter.3
Vision
1
10.3 2.1
*3
Requirements
1
15.6 4.7
1
15.8 6.3
2
28.5 14.7
specification
Review
1
6.7 1.45
Analysis
3
23.8 6.3
2
32.6 10.4
2 *3
Review
1
12
2
1
8.3 1.4
1
11
3.5
*1
*2
*4
System design
3
24.9 14.5
2
24.5 8.8
2
17
7
User interface
2 *2
2 *4
prototype
Review
1
6.6 1.4
1
7.7
2
Component design
3 *1
2
37.5 14.3
2 **
27.3 8.8
Review
1
10.3 3.1
1
4.2 1.1
1
5.6 1.2
**
Implementation
3
52.3 19
3
119.4 33.9
3
22.2 8.8
Tests
1
8.2
4
1
6
1.6
1
7.9 1.6
Total
14
161.8 29.9
14
251.8 55.1
14
133.7 24.3
Time for review was not limited. Quality experts task was to find as many defects as
possible and then make evaluations and quality predictions. Just looking through
documentation one could find it was more precise and reader friendly then before the change.
Also late deliveries (except from design phase) did not happen in the scale of the students
group. However some problems appeared. Results of the revision of the first iteration and
improvement activities undertaken to avoid the problems in the next iteration are presented in
the table 2.
Table 2. Problems collected in the first iteration and improvements in the second iteration
Problems
Improvement activities
Large time of work, especially in the Establish groups of 3 persons instead of 2.
implementation phase (it is co-related Support internal organisation of the group by
with the measured time of work)
delivering suggestions of sequence of task and work
distribution.
Simplify questionnaires.
Late deliveries in the design phase Split design (as one task lasting a long time) into
(each group was late 1 or 2 weeks)
two: system design and component design.
Medium level of the customer Introduce prototyping user interface task with
satisfaction at the end of project.
review made by the customer.

The improvements gave satisfactory results. In the second iteration the customers
satisfaction level was higher, there was no late deliveries of system design, but there were still
problems with component design. It was possible to decrease the time of work (see large
deviation in the second iteration), but average results are so large because of a very good
group that worked a lot, but also delivered excellent products. The list of problems found in
the second iteration and improvement activities for the next iteration are presented in table 3.
Table 3. Problems collected in the second iteration and improvements in the third
iteration
Problems
Improvement activities
Is there possibility to make the Simplify of Requirements Specification document: Vision of
process more efficient? Still the system is made during the first week and non-functional
large time of work was requirements selected to fit best this type of system are
reported.
delivered together with the analysis phase.
Late deliveries in component Introduce two phases of component design together with
design phase
component prototyping, implementation and writing
technical documentation (the second one includes also
system integration) instead of the component design phase
and implementation phase.
Different understanding of the Introduce meeting of all participants after system design
system by the quality expert, review and user interface prototype review to make
the customer and the developer compromise for the design and implementation.
after system design review and
user interface prototype review.
After those improvements there were no late deliveries, time of work decreased, and level
of customers satisfaction was good. This optimal process enabled to make observations on
human factors, e.g. adequacy of assigning persons with certain competencies and personality
to the development tasks.
Presented work describes activities taken to achieve the optimal process with no late
deliveries, unified time of work during all the semester, good infrastructure supporting
development process and optimal product templates. Each iteration allowed to remove more
detailed problems. Removal of the most visible problems resulted in appearance of more
hidden ones, e.g. different views of an expert and customer would not be possible to find
without review of user interface prototype, or without split for system and component design
it would be difficult to locate the problem in component design. It is also interesting that
problems indicated by measurements usually have their explanations, e.g. late deliveries in
component design are caused by poor knowledge of implementation environment and
difficulty to imagine how the system works.

4. Product quality assessment

Inspections and tests are main techniques of quality assurance with focus on product.
Inspections can be applied to the documentation in early phases to avoid defect propagation
and thus minimise time and cost of improvements. In the final version of this work, reviews of
analysis and requirements specification, system design, and detailed design with component
implementation were made by the quality experts, and user interface prototype was verified by
the customer. Questionnaire templates with metrics and comments collection as well as

reasoning about quality were designed in order to support reviews. Tests were made by both
the quality experts and the customers at the end of the project.

4.1. Questionnaires for reviews

First activity in product quality assurance, and especially review design, is quality definition.
The FCM structure was used for this purpose, and for this type of application desired quality
was defined with the following factors:
Functionality with criteria of completeness, adequacy and coherency,
Satisfaction which was verified by checking ease of use, productivity and general
aesthetics,
Maintainability which was decomposed to documentation quality, portability and
modifiability,
Dependability in aspects of security and error-tolerance.
It was assumed that quality metrics assigned to criteria and factors were in scale [1..5], where
1 meant very low quality, and 5 very high.
In order to achieve those quality metrics data collection according to the following
elements were used:
Diagram metrics - they covered counting of elements on the diagrams, and allowed to
compare those numbers with expected values, which could be derived from historical data
about similar projects. Those expected values were not given in all projects, and thus
intuitive estimation of them had to be made. Diagram metrics were also a basis for local
size calculation which was used for defect metrics calculation. There was possibility to
comment on those values. This is a potential place for automation, but in the authors
opinion these kind of calculations bring very uncertain information. Diagram metrics are
objective, but quite a lot of subjectivity is associated with the expected values application.
Evaluations with comments - evaluations are numbers that represent subjective feelings
about some aspects of the work, e.g. ease of understanding, aesthetics, precision, etc.
Sometimes they are more efficient then metrics, e.g. instead of counting all the attributes
and compare it with fuzzy expected values, one can evaluate that (always, usually, ...,
never) attributes are complete and adequate and thus description is detailed enough (or
not). They could be given in the scale [1..5] described above or described as comments.
Defect collection according to the defect classification - those defects can be then counted
and combined with the diagram metrics. Such metrics are good quality indicators.
The templates were prepared to integrate quality issues with documentation fragments in order
to focus attention of the reviewer on limited fragment of documentation, e.g. one type of
diagram with descriptions, with their relevant metrics, evaluation and potential defects. Then
those elements were integrated into quality metrics that represented factors and
recommendations of improvement were generated. During the lab, intuitive reasoning
supported with questionnaires was used, but in this part quantitative calculations with
formulas that model quality relationships mathematically can be applied to partially automate
the process.
To summarise use of qualitative and quantitative methods: comments had application
during the project to data collection, reasoning about quality factors and criteria, and
generation of recommendations for improvement by the quality experts. Also questionnaire
analysis and optimisation over the iterations was possible thanks to those comments. Metrics
of diagrams and number of defects were used for calculation of the defect metrics. Quantified
values of criteria and factors allowed to compare products and gave an idea about the time
needed to make corrections.

4.2. Results
One of the constraints of this research is that participants are students, and not real
customers, developers and quality experts. In order to achieve possibly the most reliable
results ten best projects are analysed.
In the table 4 metrics of diagrams and defects are presented. Then average value (AV) and
deviation (d) are calculated. Since analysis consists of the use case model, class diagram and
sequence diagrams with descriptions, the metrics and defects are concerned with them. For the
use case diagram, there are the following metrics:
number of use cases (# use cases),
number of actors (# actors),
number of elements including use cases, actors and relationships (# elements),
number of elements incoherent with Software Requirements Specification (# incoherent),
number of inadequate elements (# inadequate), which includes missing, wrong scope
elements, and not needed elements,
number of elements difficult to understand (# dif. to understand), which includes ambiguous
and surprising elements and statements.
Table 4. Metrics of diagrams and defects of the analysis phase for ten projects (P1-P10)
Description
P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 AV d
# use cases
8
11 13
5
6
14 14 12 17
9 10.9 3.12
# actors
2
2
3
4
1
4
2
3
2
3 2.60 0.80
# elements
20 30 37 16 17 38 36 31 36 34 29.5 7.10
# incoherent
2
7
7
6
3
8
1
3
2
2 4.10 2.32
# inadequate
3
8
4
1
2
9
3
2
2
6 4.00 2.20
# dif. To understand
2
7
3
2
3
4
1
3
3
0 2.80 1.24
% inadequate
15% 27% 11% 6% 12% 24% 8% 6% 6% 18% 13% 6%
% dif. to understand
10% 23% 8% 13% 18% 11% 3% 10% 8% 0% 10% 5%
# classes
13 11
4
14 10 24 16 13
9
9 12.3 3.70
Size of the class diagr. 30 24
8
35 19 48 41 24 19 17 26.5 9.60
# imprecision
3
8
4
3
4
4
1
4
3
1 3.50 1.30
Precision evaluation
3
3 3,5 5
4
4 3,5 4 3,5 3 3.65 0.48
# inadequate
2
6
5
2
2
6
4
1
2
3 3.30 1.56
# dif. to understand
2
4
2
0
2
2
1
1
0
2 1.60 0.88
% imprecision
10% 33% 50% 9% 21% 8% 2% 17% 16% 6% 17% 11%
% inadequate
7% 25% 63% 6% 11% 13% 10% 4% 11% 18% 17% 11%
% dif. to understand
7% 17% 25% 0% 11% 4% 2% 4% 0% 12% 8% 6%
# sequence diagr.
8
11
8
5
2
14 10 11 11
1 8.10 3.30
Size (# interactions)
48 49 134 36
6
84 37 20 33 17 46.4 25.9
# incorrect interactions 4
3
3
0
2
3
3
2
2
2 2.40 0.80
# m_exceptions
x
3
5
1
4
4
2
x
x
x
x
x
%incorrect interactions 8% 6% 2% 0% 33% 4% 8% 10% 6% 12% 9% 6%
# inco A-SRS
3
4
3
0
2
1
0
2
0
0 1.50 1.30
# inco CD-UCD
0
3
5
0
1
0
0
0
2
3 1.40 1.48
# inco CD-SD
0
3
3
1
0
0
2
1
0
1 1.10 0.94
# inco UCD-SD
2
6
6
1
4
1
3
2
6
5 3.60 1.80

Defect metrics are calculated by division of the number of inadequate elements by the number
of elements (% inadequate), and the number of elements difficult to understand by the number
of elements (% dif. to understand). The number of elements stand here for the size of use case
diagram. Similar metrics are collected for the class diagram, but in this case precision is also
important and it is difficult to be captured by metrics, so apart from discovering imprecise
elements (# imprecise) there is precision evaluation in the scale of [1..5]. The size of the class
diagram is here defined as a sum of classes and relationships. And for sequence diagrams there
are the following metrics:
number of sequence diagrams (# sequence diagr.),
size counted as a number of interactions,
missing exceptions (# m_exceptions) - x is used when there is a lot of exceptions missing.
And finally coherency between the diagrams is checked:
incoherence between analysis diagrams and requirements specification (#inco A-SRS),
incoherence between class diagram and use case diagram (# inco CD-UCD),
incoherence between class diagram and sequence diagrams (# inco CD-SD), and
incoherence between use case diagram and sequence diagrams (# inco UCD-SD).
In the table 5, quality metrics for the same projects being result of the reasoning on the basis
of the metrics described above together with evaluations and comments are presented.

Description
Completeness
Adequacy
Precision
Functionality

Table 5. Quality metrics for the criteria in ten projects.

P1 P2 P3 P4 P5 P6 P7 P8
3
3
3
3
5
4
5
4
3,5 2
4
4
4
3 4,5 3
3,5 3
3
5
4
4 3,5 4,5
4
3
4
5
4
5
4
4

P9
5
3
4
4

P10
4
3
2
3

AV
3,9
3,4
3,7
4,0

The questionnaires support finding defects and help to integrate information taken from
different sources. For more precise reasoning it would be useful to introduce some standard
rules. Concrete defects are closely connected with the domain of application and they are used
during the project to improve quality, so other considerations then classified defect metrics
have no sense in this analysis. Statistics of the diagram metrics can be applied for gathering
expected values. Let us assume that average value is an expected value, and deviation sets the
range of accepted values. In this case, range of expected values for the number of classes is
[8..16], and it is possible to find automatically anomalies in project P3 with too little classes
(4), and project P6 with too many (24).

5. Conclusions
In the paper, experience of process improvement and product quality assessment in early
phases of software development with use of quantitative and qualitative methods is presented.
Defined process and collected data plan were the basis for making improvements. They
enabled to rely on facts, not opinions during revisions. Quantitative methods were used to
verify concepts, and qualitative ones for problem identification and removal, and deeper
analysis of project issues. After the change concerned with introduction of the new method
with reviews and measurements, the development process was optimised over three iterations.
Every iteration allowed to identify and remove more and more detailed problems.
In product quality assessment qualitative and quantitative methods were used by the quality
experts to data collection, reasoning about quality factors and criteria, and generation of
recommendations for improvement. They were also used in questionnaire optimisation.

6. References
[1]
[2]

[3]
[4]
[5]
[6]
[7]
[8]

Bobkowska A., Training on High Quality Software Development with the 3RolesPlaying
method, in SCI98/ISAS98 conference proceedings, Orlando, USA, July 1998.
Briand, L. C., Differding C. M., Rombach D. H., Practical Guidelines for Measurement Based Process Improvement, Technical Report of the International Software Engineering
Network (ISERN-96-05), 1996.
Fenton N.E., Software Metrics. A Rigorous Approach, Chapman&Hall, 1993.
Grudowski P., Kolman R., Meller A., Preihs J., Zarzdzanie jakoci (Quality Management),
Wydawnictwo Politechniki Gdaskiej, Gdask, 1996.
Rational Software Corporation, UML Notation Guide, www.rational.com
Rational Software Corporation, Rational Unified Process, www.rational.com
Rumbaugh J., Blaha M., Premerliani W., Eddy F., Lorensen W., Object-oriented modelling and
design, Prentice-Hall, 1991.
Seaman C. B., Qualitative methods in Empirical Studies of Software Engineering.,
Transactions on Software Engineering volume 25, 1999, pages 557-572.