Part 1

S o f t w a r e T e s t i n g
Pr i n ci ple s a n d Pr a ct i ce s
Srinivasan Desikan
Director of Quality Assurance,
AgileSoftware Enterprise, Pvt. Ltd.
Bangalore, India
Gopalaswamy Ramesh
Consultant and Adjunct Professor,
International Institute of Information Technology,
Bangalore, India
PEARSON
Education
Editor in Chief: Prof H N Mahabala
Titles under the series include
Software Maintainence
Gopalaswamy Ramesh and Ramesh Bhattiprolu
Software Testing - Principles &Practices
Gopalaswamy Ramesh and Srinivasan Desikan
Managing Global Software Projects
Gopalaswamy Ramesh
Mobile Computing - Technology Applications &Service Creation
Asoke K Talukder and Roopa Yavagal
Copyright @ 2006 Dorling Kindersley (India) Pvt. Ltd.
This book is sold subject to the condition that it shall not, by way of trade or otherwise, be lent, resold,
hired out, or otherwise circulated without the publisher's prior written consent in any form of binding or
cover other than that in which it is published and without asimilar condition including this condition being
imposed on the subsequent purchaser and without limiting the rights under copyright reserved above, no
part of this publication may be reproduced, stored inor introduced into aretrieval system, or transmitted in
any form or by any means (electronic, mechanical, photocopying, recording or otherwise), without the prior
written permission of both the copyright owner and the above-mentioned publisher of this book.
ISBN 81-7758-295-X
First Impression, 2006
Second Impression, 2007
Published by Dorling Kindersley (India) Pvt. Ltd., licensees of Pearson Education in South Asia.
Head Office: 482, F.I.E., Patparganj, Delhi 110092, India.
Registered Office: 14Local Shopping Centre, Panchsheel Park, New Delhi 110017, India.
Laser typeset by Sigma Publishing Services, Chennai.
Printed in India by Baba Barkha Nath Printers.
,
j
Parents and Teachers who taught me the basics of
life and science
-Srinivasan Desikan
To the millions of differently-abled children who are
wrongly labeled as "disabled" because of the
inappropriate testing practices of our society
-Gopalaswamy Ramesh
I
The area of Software Testing has acquired wider horizon and significance. Customers demand
defect-free products; regulatory authorities go into the nitty-gritties of the tests to which apiece
of software issubjected, just aspharmaceutical companies aremandated to declarehow they test
the medicines before release. Most of thebooks available inthe market cover theoretical aspects
but very few address practical aspects. Toprepare students for acareer in testing and to be of
value to practitioners, one needs to provide apragmatic and practical view of testing, together
with theright balance of people, process and technology. Thisforms thebasis of thebook Software
Testing-Principles and Practices.
Asthetitleindicates, theemphasis isonprinciples andpractices. Thematerial inthisbook has
already been "Betatested" byseveral universities inIndiasuchasAnnaUniversity inChennai and
International Institute of Information Technology, Bangalore(iiit-b).iiit-bisalsoadopting thebook
for itsCenter of ExcellenceinSoftwareEngineering andfor itsM. Techstudents. Someconcepts in
thebook havealready beenpresented topractitioners through international conferences and guest
lectures by theauthors of thisbook.
Wehave added chapters that focus on managing geographically distributed teams. This is
especially relevant for multinational companies, where distributed teams in different continents
work together inunison-developing, testing and delivering products to aglobal customer base.
Wehaveincluded emerging trends intesting likeextremetesting, adhoctesting and soon.
Thecontents of this book can beused by practitioners to understand the state-of-practice in
thetesting industry. Thechapters onMetrics and Measurements, TestPlanning, TestManagement
andAutomation havebeen added tohelp thepractitioners adopt theseconcepts at work. Wehave
provided sufficient academic rigor tosatisfy thecontents of thesyllabi of thevarious universities.
Wewould liketothank Ms. Gayathri Chandrasekar, Testmanager at eFunds, Chennai for her
contributions tosomeof thechapters inthebook.
Srinivasan Desikan
Gopalaswamy Ramesh
Acknowledgements
Mygratitude areduetoall theprevious organizations (Wipro, Novell andTalisma) I worked with,
for the opportunities and infrastructure provided for meto learn thepractical aspects of testing
on thejob. My thanks are due to the testing professionals across the globe for interacting with
meand providing insights and information for developing this book. Last but not least, I would
liketo thank my wife, son and daughter for their sacrificesand support. I can be contacted at
srinivasan.desikan@gmail.com.
Srinivasan Desikan
I want tothank Prof. Mahabala, my Guru, who hasbeen achampion insoftware testing and has
motivated metowards it. Hehasbeenaconstant sourceof inspiration for meover thelast twenty
years. I alsowant to thank Department of Computer Science,Anna University and International
Institute of Information Technology,Bangalbreforenablingmetoteachcoursesof SoftwareTesting,
whichhasbeenasourceof learning. Myspecial thanks arealsodueto Prof. Sadagopan, for theiiit-
bPress initiative. Finally,I want toacknowledge thesupport of myfamily,without which I would
not havebeen abletogivethekindof effort that went intomaking thisbook. I canbecontacted at
gopalaswamyJamesh@yahoo.com.
Gopalaswamy Ramesh
_ _ _ _ - - - - - - - - - " ' I
Software is becoming ubiquitous in today's world. Customers' expectations have increased
dramatically and the old fate of resignation that "it is OK for software to fail, we will have to live
with it" issimply no longer applicable. Software isexpected to work, meeting customers' changing
demands, first time and every time, consistently and predictably. Earlier, the software systems
were used for back-office and non-critical operations of organizations. Now, more and more critical
applications are implemented globally. This increased expectation for error-free functioning of
software has increased the demand of quality output fromsoftware vendors. Intum, wehave seen
over the last decade, the focus has shifted from mere programming and development to a more
holistic goal of producing software that works first time and all the time, thus increasing the focus
on testing.
Testing, in particular, has attracted significant interest over the past decade. Consider the
following which tell the whole story:
~ the number of testing jobs has increased several-fold providing bright career opportunities;
~ the compensation levels of testing functions are on the rise;
~ testing has become amajor opportunity for outsourcing;
~ thenumber of conferences and similar events dedicated totesting has increased significantly
over the last fiveyears; and
~ more and more professionals are considering software testing as acareer.
Themodus operandi of testing has alsoundergone aradical change, inkeeping with theincreased
demand. Firstly, globalization is here to stay. Organizations today exploit the geographic time
difference and global talent by distributing development and testing teams in different continents,
working seamlessly together. Inorder tobesuccessful inthis new world order, anorganization must
master the art of working in geographically distributed teams. Secondly, testing has transformed
from an ad hoc and haphazard hit-and-run attempt to a systematic, planned activity, complete
with all processes and measured by scientific metrics. Thirdly, success in testing today requires
careful exploitation of various technologies to meet product's time to market requirements. Test
automation-automation of all parts of testing lifecycle-has become anecessity and isno longer
aluxury. Finally,people's aspirations intesting careers have alsoundergone aseachange-a successful
organization should show career paths for the testing professionals and nurture their talent to ensure
their longevity within the organization and within the testing profession.
This book has come in atimely fashion to address the needs of the practitioners and aspiring
testing professionals and students. Both Ramesh and Srinivasan, have brought to life their forty
Foreword
Wi::~I'~~I::;~I:_
~ 't:d_:!LibALwX@~~; -.' - - < ::;::-}'=
years of combined practical experience. Thisbook covers some of the aspects which areessential
for theindustry's success, but seldomtouched upon by other books, suchas:
ffi Balancebetween theory and practice: Most books, while trying to give theoretical rigor,
try tooversimplify thereal-lifeproblems. Theauthors haveconsistently started fromwhat
existsinthereal world and haveprovided thetheoretical foundation toaddress thereal life
situations.
ffi Balancebetween people, process, and technology issues: Successful organizations work
systematically, exploit technologies and leverage people. The authors have successfully
resisted the temptation of only discussing "cool technology issues" at the expense of
practicalities. As an example, you will find coverage of topics like automation (which is
technology-intensive), people and organizational structures (which ispeople-centric) and
test organization and reporting (whichisprocess-focused).
ffi A solid exposure to foundations, from a practical viewpoint of the practitioners: The
authors have covered all the different types of testing extensively. As an example, topics
likeInternationalization-which areconsidered esoteric-are discussed at length.
ffi Thisbook isthefirst that I know which has recognized and articulated theimportance of
globalization and explicitly discussed thevarious global teamstructures possible and the
issues thereof. Thisclearly demonstrates theauthors experience of leading global software
testing teams.
ffi This book covers some of the recognized methodologies for testing which have been
accepted and presented inseveral international testing conferences throughout theworld.
Inaddition toall of theabove, thepractical exercisesgivenat theend of thechapters will expose
the reader to therealities of testing. Theauthors' experience of teaching full semester courses on
software testing isevident invarious exercises. I amconfident that with the advent of this book,
such courses will spread tovarious collegesand universities, thereby formalizing software testing
asadiscipline and widening thenet for developing morecompetent testing professionals.
I wishthebook andtheauthors all successintheir continuing endeavor tocreateanenvironment
where softwaretesting isconsidered acritical phaseof anysoftwaredevelopment lifecycle,careers
intesting arevalued significantly, and testing develops asanengineering discipline.
Vikram Shah
VikramShahiscurrentlyservingintheBoardofDirectorsinSilverSoftware,IT-Peopleand
isanactivememberof BiTES(BoardforITeducationStandardsbyKarnatakagovernment).
Vikrammentorsstart-upInformationTechnologycompanies.VikramShahis an industry
veteranwithmorethan30yearsof experienceandiscurrentlyservingasMDof Independent
Technologies(INTEC).Inthepast heservedasMDof several companiessuchasMahindra
BritishTelecom,Novell,AndiamoSoftware,andTalismaSoftware.VikramdidBSinElectronics
fromBITS,Pilani andMSinComputerSciencefromUniversityof California,Berkeley.
"'$! ; C o n t e A t ! j s
\ ' , , ; . : : : " r - ' : , " , : '< ; : " " ~ : : : ) " : : " : !- ' . . f. " , : , : , ~ " , . : ~ : w. :
Pr eface
Acknowledgements
For ewor d
v
vi
Vll
Part I
IIPrin ciple s o f Te s t in g
.-s o ft ware De ve lo pme n t Life C ycle Mo de ls
1 . 1
1.2
1.3
1.4
1 . 5
1.6
1.7
1.8
1.9
1.10
1.11
1.12
1.13
1.14
2.1
2.2
2 . 3
2.4
2.5
Context of Testing in Producing Software
About this Chapter
The Incomplete Car
Dijkstra's Doctrine
A Test in Time!
The Cat and the Saint
Test the Tests First!
The Pesticide Paradox
The Convoy and the Rags
The Policemen on the Bridge
The Ends of the Pendulum
Men in Black
Automation Syndrome
Putting it All Together
References
Problems and Exercises
Phases of Software Project
2.1.1 Requirements Gathering and Analysis
2.1.2 Planning
2.1.3 Design
2.1.4 Development or Coding
2.1.5 Testing
2.1.6 Deployment and Maintenance
Quality, Quality Assurance, and Quality Control
Testing, Verification, and Validation
Process Model to Represent Different Phases
Life Cycle Models
2.5.1 Waterfall Model
2.5.2 Prototyping and Rapid Application Development Models
3
4
6
7
8
9
11
12
1 2
14
1 5
1 6
1 9
2 0
2 2
2 3
23
25
2 6
26
26
2 6
27
27
27
2 7
2 9
31
3 2
3 2
34
"- - - - - - - - - - - - - - - - - - - - -
x Software Testing
2.5.3 Spiral or Iterative Model
2.5.4 The V Model
2.5.5 Modified V Model
2.5.6 Comparison of Various Life Cycle Models
References
36
37
40
43
43
43
Part II
IIWhite Box Testing
3.1 What is White Box Testing?
3.2 Static Testing
3.2.1 Static Testing by Humans
3.2.2 Static Analysis Tools
3.3 Structural Testing
3.3.1 Unit/Code Functional Testing
3.3.2 Code Coverage Testing
3.3.3 Code Complexity Testing
3.4 Challenges in White Box Testing
References
IIBlack Box Testing
47
48
48
49
53
56
56
57
63
67
68
68
73
4.1
4.2
4.3
4.4
4.5
What is Black Box Testing?
Why Black Box Testing?
When to do Black Box Testing?
How to do Black Box Testing?
4.4.1 Requirements Based Testing
4.4.2 Positive and Negative Testing
4.4.3 Boundary Value Analysis
4.4.4 Decision Tables
4.4.5 Equivalence Partitioning
4.4.6 State Based or Graph Based Testing
4.4.7 Compatibility Testing
4.4.8 User Documentation Testing
4.4.9 Domain Testing
Conclusion
References
74
75
76
76
76
82
84
87
90
93
96
99
101
104
104
105
IIIIntegration Testing
5.1 What isIntegration Testing?
5.2 Integration TestingasaTypeof Testing
5.2.1 Top-DownIntegration
5.2.2 Bottom-UpIntegration
5.2.3 Bi-Directional Integration
5.2.4 SystemIntegration
5.2.5 ChoosingIntegration Method
5.3 Integration TestingasaPhaseof Testing
5.4 ScenarioTesting
5.4.1 SystemScenarios
5.4.2 UseCaseScenarios
5.5 DefectBash
5.5.1 ChoosingtheFrequency andDuration of DefectBash
5.5.2 SelectingtheRightProduct Build
5.5.3 Communicating theObjectiveof DefectBash
5.5.4 SettingupandMonitoringtheLab
5.5.5 TakingActionsandFixingIssues
5.5.6 Optimizing theEffortInvolvedinDefectBash
5.6 Conclusion
References
Problems andExercises
107
108
108
111
113
114
115
116
117
118
118
120
122
123
123
123
123
124
124
125
126
126
II' System and Acceptance Testing
6.1
6.2
6.3
6.4
6.5
SystemTestingOverview
WhyisSystemTestingDone?
Functional VersusNon-Functional Testing
Functional SystemTesting
6.4.1 Design/Architecture Verification
6.4.2 BusinessVertical Testing
6.4.3 Deployment Testing
6.4.4 BetaTesting
6.4.5 Certification, Standards andTestingfor Compliance
Non-Functional Testing
6.5.1 SettinguptheConfiguration
6.5.2 ComingupwithEntry/Exit Criteria
6.5.3 BalancingKeyResources
6.5.4 ScalabilityTesting
6.5.5 ReliabilityTesting
6.5.6 StressTesting
6.5.7 Interoperability Testing
127
128
130
131
133
134
135
136
137
140
141
142
143
143
145
149
153
156
6.6
6.7
Acceptance Testing
6.6.1 Acceptance Criteria
6.6.2 Selecting Test Cases for Acceptance Testing
6.6.3 Executing Acceptance Tests
Summary of Testing Phases
6.7.1 Multiphase Testing Model
6.7.2 Working Across Multiple Releases
6.7.3 Who Does What and When
References
158
159
160
161
162
162
164
165
166
166
Performance Testing
169
7.1
7.2
7.3
7 . 4
7.5
7 . 6
Introduction
Factors Governing Performance Testing
Methodology for Performance Testing
7.3.1 Collecting Requirements
7.3.2 Writing Test Cases
7.3.3 Automating Performance Test Cases
7.3.4 Executing Performance Test Cases
7.3.5 Analyzing the Performance Test Results
7.3.6 Performance Tuning
7.3.7 Performance Benchmarking
7.3.8 Capacity Planning
Tools for Perform'ance Testing
Process for Performance Testing
Challenges
References
170
170
173
174
176
177
177
179
182
184
186
187
188
190
191
192
Regression Testing
8.1 What is Regression Testing?
8.2 Types of Regression Testing
8.3 When to do Regression Testing?
8.4 How to do Regression Testing?
8.4.1 Performing an Initial "Smoke" or "Sanity" Test
8.4.2 Understanding the Criteria for Selecting the Test Cases
8.4.3 Classifying Test Cases
8.4.4 Methodology for Selecting Test Cases
8.4.5 Resetting t1).eTest Cases for Regression Testing
8.4.6 Concluding the Results of Regression Testing
193
194
195
196
197
198
199
200
201
202
205
III'Internationalization (IlSn)Testing
8 . 5
9.1
9.2
9.3
9 . 4
9.5
9.6
9.7
9.8
9.9
9.10
9.11
Best Practices in Regression Testing
References
Introduction
Primer on Internationalization
9.2.1 Definition of Language
9.2.2 Character Set
9.2.3 Locale
9.2.4 Terms Used in This Chapter
Test Phases for Internationalization Testing
Enabling Testing
Locale Testing
Internationalization Validation
Fake Language Testing
Language Testing
Localization Testing
Tools Used for Internationalization
Challenges and Issues
References
206
208
208
211
212
212
212
213
214
214
215
216
218
219
221
222
223
225
225
226
227
IIAd hoc Testing
10.1 Overview of Ad Hoc Testing
10.2 Buddy Testing
10.3 Pair Testing
10.3.1 Situations When Pair Testing Becomes Ineffective
10.4 Exploratory Testing
10.4.1 Exploratory Testing Techniques
10.5 Iterative Testing
10.6 Agile and Extreme Testing
10.6.1 XP Work Flow
10.6.2 Summary with an Example
10.7 Defect Seeding
10.8 Conclusion
References
228
229
233
234
236
237
237
239
241
242
245
246
248
248
248
300
300
302
275
276
278
281
283
284
285
285
288
291
292
293
295
295
295
299
274
253
254
254
261
261
267
268
269
269
272
273
273
Usability and Accessibility Testing
12.1 What isUsability Testing?
12.2 Approach toUsability
12.3 WhentodoUsability Testing?
12.4 HowtoAchieveUsability?
12.5 Quality Factorsfor Usability
12.6 Aesthetics Testing
12.7 Accessibility Testing
12.7.1 BasicAccessibility
12.7.2 Product Accessibility
Toolsfor Usability
Usability LabSetup
TestRolesfor Usability
Summary
References
Testing of Object-Oriented Systems
11.1 Introduction
11.2 Primer onObject-Oriented Software
11.3 Differencesin00Testing
11.3.1 Unit Testingaset of Classes
11.3.2 Putting ClassestoWorkTogether-Integration Testing
11.3.3 SystemTestingandInteroperability of 00Systems
11.3.4 RegressionTestingof 00Systems
11.3.5 Toolsfor Testingof 00Systems
11.3.6 Summary
References
12.8
12.9
12.10
12.11
Part IV
xiv Software Testing
IIICommon People Issues
13.1 Perceptions andMisconceptions About Testing
13.1.1 "Testingisnot TechnicallyChallenging"
13.1.2 "TestingDoesNot ProvidemeaCareer Pathor Growth"
Part III
13.2
13.3
13.4
13.1.3 "I AmPut inTesting- What isWrongWithMe?!"
13.1.4 "TheseFolksAreMyAdversaries"
13.1.5 "TestingisWhatI CanDointheEndif I GetTime"
13.1.6 "ThereisnoSenseof Ownership inTesting"
13.1.7 "TestingisonlyDestructive"
ComparisonbetweenTestingandDevelopment Functions
Providing Career PathsforTestingProfessionals
TheRoleof theEcosystemandaCall forAction
13.4.1 Roleof EducationSystem
13.4.2 Roleof SeniorManagement
13.4.3 Roleof theCommunity
References
ProblemsandExercises
303
304
304
306
306
306
307
314
314
315
316
318
318
IIIOrganization Structuresfor Testing Teams 320
14.1 Dimensionsof Organization Structures 321
14.2 Structures inSingle-Product Companies 321
14.2.1 TestingTeamStructuresforSingle-Product Companies 322
14.2.2 Component-WiseTestingTeams 325
14.3 Structures forMulti-Product Companies 325
14.3.1 TestingTeamsasPart of "CTO'sOffice" 327
14.3.2 SingleTestTeamforAll Products 328
14.3.3 TestingTeamsOrganizedbyProduct 329
14.3.4 SeparateTestingTeamsforDifferentPhasesof Testing 329
14.3.5 HybridModels 331
14.4 Effectsof GlobalizationandGeographicallyDistributed Teams
onProduct Testing 331
14.4.1 BusinessImpact of Globalization 331
14.4.2 RoundtheClockDevelopment/TestingModel 332
14.4.3 TestingCompetency Center Model 334
14.4.4 ChallengesinGlobal Teams 336
14.5 TestingServicesOrganizations 338
14.5.1 BusinessNeedforTestingServices 338
14.5.2 DifferencesbetweenTestingasaServiceandProduct-
TestingOrganizations 338
14.5.3 Typical RolesandResponsibilitiesof TestingServicesOrganization 339
14.5.4 ChallengesandIssuesinTestingServicesOrganizations 342
14.6 SuccessFactorsforTestingOrganizations 344
References 346
ProblemsandExercises 346
Part V
Test Planning, Management, Execution, andReporting 351
15.1 Introduction 352
15.2 TestPlanning 352
15.2.1 PreparingaTestPlan 352
15.2.2 ScopeManagement: DecidingFeaturestobeTested/NotTested 352
15.2.3 DecidingTestApproach/Strategy 354
15.2.4 SettingupCriteriaforTesting 355
15.2.5 IdentifyingResponsibilities, Staffing,andTrainingNeeds 355
15.2.6 IdentifyingResourceRequirements 356
15.2.7 IdentifyingTestDeliverables 357
15.2.8 TestingTasks:SizeandEffortEstimation 357
15.2.9 ActivityBreakdownandScheduling 360
15.2.10CommunicationsManagement 361
15.2.11RiskManagement 362
15.3 TestManagement 366
15.3.1 Choiceof Standards 366
15.3.2 TestInfrastructureManagement 369
15.3.3 TestPeopleManagement 372
15.3.4 IntegratingwithProduct Release 373
15.4 TestProcess 374
15.4.1 PuttingTogetherandBaseliningaTestPlan 374
15.4.2 TestCaseSpecification 374
15.4.3 Updateof TraceabilityMatrix 375
15.4.4 IdentifyingPossibleCandidatesforAutomation 375
15.4.5 DevelopingandBaseliningTestCases 376
15.4.6 ExecutingTestCasesandKeepingTraceabilityMatrixCurrent 376
15.4.7 CollectingandAnalyzingMetrics 377
15.4.8 PreparingTestSummaryReport 377
15.4.9 RecommendingProduct ReleaseCriteria 377
15.5 TestReporting 378
15.5.1 RecommendingProduct Release 379
15.6 BestPractices 379
15.6.1 ProcessRelatedBestPractices 380
15.6.2 PeopleRelatedBestPractices 380
15.6.3 TechnologyRelatedBestPractices 380
AppendixA:TestPlanningChecklist 381
Appendix B:TestPlanTemplate 384
References 385
xVii
IIISoftware Test Automation 387
16.1 WhatisTestAutomation? 388
16.2 TermsUsedinAutomation 390
16.3 SkillsNeededforAutomation 392
16.4 What toAutomate, ScopeofAutomation 394
16.4.1 IdentifyingtheTypesof TestingAmenabletoAutomation 394
16.4.2 AutomatingAreasLessPronetoChange 395
16.4.3 AutomateTeststhat PertaintoStandards 395
16.4.4 Management AspectsinAutomation 396
16.5 DesignandArchitectureforAutomation 396
16.5.1 External Modules 397
16.5.2 ScenarioandConfigurationFileModules 398
16.5.3 TestCasesandTestFrameworkModules 398
16.5.4 ToolsandResultsModules 399
16.5.5 Report Generator andReports/MetricsModules 399
16.6 GenericRequirementsforTestTool/Framework 399
16.7 ProcessModel forAutomation 408
16.8 SelectingaTestTool 411
16.8.1 CriteriaforSelectingTestTools 412
16.8.2 StepsforToolSelectionandDeployment 415
16.9 Automationfor ExtremeProgrammingModel 415
16.10 ChallengesinAutomation 416
16.11 Summary 416
References 418
II' Test Metrics and Measurements 420
17.1
17.2
17.3
17.4
17.5
17.6
What areMetricsandMeasurements?
WhyMetricsinTesting?
Typesof Metrics
ProjectMetrics
17.4.1 EffortVariance(PlannedvsActual)
17.4.2 ScheduleVariance(PlannedvsActual)
17.4.3 EffortDistributionAcrossPhases
ProgressMetrics
17.5.1 TestDefectMetrics
17.5.2 Development DefectMetrics
Productivity Metrics
17.6.1 Defectsper 100Hoursof Testing
17.6.2 TestCasesExecutedper 100Hours of Testing
421
425
427
428
429
430
432
433
434
443
448
450
450
17.7
17.8
17.6.3 Test Cases Developed per 100Hours of Testing
17.6.4 Defects per 100Test Cases
17.6.5 Defects per 100Failed Test Cases
17.6.6 Test Phase Effectiveness
17.6.7 Closed Defect Distribution
Release metrics
Summary
References
450
450
451
452
452
453
455
456
456
IIIllustrations
IIReferences and Bibliography
IIIndex
457
481
483
aQIOulllas
I
This part of thebook sets the context for theentire book.
Westart Chapter I, Principles of Testing, by discussing the
changes inthesoftwarescenarioover thelast fewyearsand
thedemands that thesechangesmakeonsoftwarequality.We
translate thesedemands into elevenbasic principles which
provideanchorstorest of thechapters. InChapter 2,Software
Development Life Cycles, wedefinekeytermslikeverification,
validation, quality assurance, andqualitycontrol,gointothe
details of thevarious lifecyclemodels and theimplication
of thesemodels onverificationandvalidation activities. We
alsointroducetheconceptsof entryandexitcriteriathat will
benecessary to understand the different phases of testing
later inthebook.
Inthis chapter-
./ Context of testing in producing software
./ About this chapter
./ Theincomplete car
./ Dijkstra's doctrine
./ A test in time!
./ Thecat and the saint
./ Test the tests first!
./ Thepesticide paradox
./ Theconvoy and the rags
./ Thepolicemen on the bridge
./ Theends of the pendulum
./ Men in black
./ Automation syndrome
./ Putting it all together
4 Software Testing
1.1 CONTEXT OF TESTING IN PRODUCING SOFTWARE
Almost everything weuse today has anelement of software init. Inthe
early days of evolution of software, theusers of softwareformed asmall
number compared to the total strength of an organization. Today, in a
typical workplace (andathome), just about everyoneuses acomputer and
software. Administrative staff useofficeproductivity software (replacing
the typewriters of yesteryears). Accountants and finance people use
spreadsheets and other financial packages tohelp themdo much faster
what theyused todowith calculators (orevenmanually). Everyoneinan
organization and at homeuses e-mail and theInternet for entertainment,
education, communication, interaction, and for getting any information
theywant. Inaddition, of course, the"technical" peopleuseprogramming
languages, modeling tools, simulation tools, and database management
systems for tasks that they were mostly executing manually a few
years earlier.
Theaboveexamplesarejust someinstanceswheretheuseof softwareis
"obvious" totheusers. However, softwareismoreubiquitous andpervasive
thanseenintheseexamples. Softwaretoday isascommonaselectricitywas
intheearlypart of thelastcentury.Almosteverygadget anddevicewehave
at home and at work isembedded with asignificant amount of software.
Mobilephones, televisions, wrist watches, andrefrigerators or any kitchen
equipment all haveembedded software.
Another interesting dimension is that software is being used now in
missioncritical situations wherefailureissimplyunacceptable. Thereisno
wayonecansuggest asolutionof "pleaseshutdown andreboot thesystem"
for a software that is in someone's pacemaker! Almost every servicewe
havetakenfor granted has software. Hanks,air trafficcontrols, carsareall
powered by softwarethat simply cannot affordtofail. Thesesystems have
torunreliably,predictably, all thetime, everytime.
This pervasiveness, ubiquity, and mission criticality places certain
demands onthewaythesoftwareisdevelopedanddeployed.
First, an organization that develops any form of software product
or servicemust put inevery effort to drastically reduce and, preferably,
eliminate any defects in each delivered product or service. Users are
increasingly intolerant of the hit-and-miss approach that characterized
software products. Fromthe point of view of a software development
organization also, it may not beeconomically viable to deliver products
with defects. For instance, imagine finding a defect in the software
embedded in atelevision after it is shipped to thousands of customers.
How isit possibletosend "patches" tothesecustomers and ask themto
"install thepatch?" Thus, theonly solution istodoit right thefirst time,
beforesending aproduct tothecustomer.
Second, defectsareunlikely toremainlatent forlong. Whenthenumber
of userswaslimitedandthewaytheyusedtheproduct wasalsopredictable
Principles of Testing 5
(andhighly restricted), it was quite possible that there could bedefects in
thesoftwareproduct that would never get detected or uncovered for avery
longtime. However, with thenumber of users increasing, thechances of a
defectgoingundetected arebecomingincreasingly slim. If adefectispresent
intheproduct, someone will hit upon itsooner than later.
Third, thenatureofusageofaproduct oraserviceisbecomingincreasingly
unpredictable. When bespoke software isdeveloped for aspecificfunction
for aspecificorganization (for example, apayroll package), the nature of
usageof theproduct canbepredictable. Forexample, users canonlyexercise
thespecificfunctionality provided inthebespoke software. Inaddition, the
developers of thesoftwareknowtheusers, their business functions, andthe
user operations. On the other hand, consider ageneric application hosted
ontheInternet. Thedevelopers of theapplication havenocOI],trol over how
someonewill usetheapplication. Theymay exerciseuntested functionality;
they may haveimproper hardware or softwareenvironments; or they may
not befullytrained ontheapplication andthus simply usetheproduct inan
incorrect orunintended manner. Despiteall this"mishandling," theproduct
should work correctly.
Finally,theconsequenceandimpact of everysingledefectneeds analysis,
especially for mission critical applications. It may beacceptabletosay that
99.9%of defects arefixedinaproduct for arelease, and only 0.1% defects
are outstanding. It appears to be an excellent statistics to go ahead and
releasetheproduct. However, if wemap the0.1%failureinmission critical
applications, thedatawill look likethis.
~ Atotal of 10,000incorrect surgery operations per week.
~ Threeairplane crashes every day.
~ Noelectricity for fivehours every week.
For sure, theabovedataisunacceptable for anyindividual, organization,
or government. Providing aworkaround, suchas"Incaseof fire, wear this
dress," or documenting afailure, such as "Youmay loseonly body parts
in caseof awrong airplane landing" would not be acceptable in cases of
mission critical applications.
This book focuses on software testing. Traditionally, testing is defined
as being narrowly confined to testing the program code. Wewould like
to consider testing in abroader context asencompassing all activities that
address the implications of producing quality products discussed above.
Producing asoftwareproduct entails several phases (suchasrequirements
gathering, design, andcoding) inaddition totesting(inthetraditional sense
of the term). Whiletesting is definitely oneof the factors (and one of the
phases) that contributes toahighquality product, italonecannot add quality
toaproduct. Proper interaction of testing with other phases isessential for
agoodproduct. Theseinteractions andtheir impact arecaptured inthegrid
inFigure1.1.
Figure 1.1
Relationship of
effectiveness of
testing to quality
of other phases.
Poor upstream activities
Right balance,
Implies heavy
Promotes teamwork,
dependence on testing
Delivers a quality
to detect and correct
product to customers
defects
HIGH REWORK COST
IDEAL STATE
Poor upstream activities AND
May lead to unexpected
poor testing
defects due to poor testing
NOT SUSTAINABLE
RISKY
Quality of Other Phases
If thequality of theother phases islowandtheeffectivenessof testing is
low(lower left-hand corner of thegrid), thesituation isnot sustainable. The
product will most likelygoout of business verysoon. Tryingtocompensate
for poor quality in other phases with increased emphasis on the testing
phase (upper left-hand corner of thegrid) islikelytoput high pressure on
everyone asthedefects get detected closer tothetimetheproduct isabout
tobereleased. Similarly,blindly believingother phases tobeof highquality
and having apoor testing phase (lower right-hand side of the grid) will
lead to the risky situation of unforeseen defects being detected at the last
minute. Theideal stateof courseiswhen high quality ispresent inall the
phases including testing (upper right-hand corner of thegrid). Inthis state,
thecustomers feel thebenefitsof quality andthispromotes better teamwork
andsuccessinanorganization. .
1.2 ABOUTTHIS CHAPTER
In this chapter, we discuss some of the basic principles of testing. We
believe that these principles are fundamental to the objective of testing,
namely, to provide quality products to customers. These principles also
formthe motivation for the rest of thebook. Thus this chapter acts as an
anchor for therest of thebook.
Thefundamental principles of testing areasfollows.
1. Thegoal of testingistofinddefectsbeforecustomersfindthemout.
2. Exhaustive testing isnot possible; program testing can only show
thepresence of defects, never their absence.
3. Testingapplies all through thesoftwarelifecycleand isnot anend-
of-cycleactivity.
4. Understand thereason behind thetest.
Principles of Testing
5. Testthetestsfirst.
6. Testsdevelop immunity and havetoberevised constantly.
7. Defects occur in convoys or clusters, and testing should focus on
theseconvoys.
8. Testingencompasses defect prevention.
9. Testingisafinebalance of defect prevention and defect detection.
10. Intelligent and well-planned automation is key to realizing the
benefits of testing.
11. Testingrequirestalented, committedpeoplewhobelieveinthemselves
andworkinteams.
Wewill takeupeachoftheseprinciplesinthesubsequent sections.Where
appropriate, wewill illustratetheprinciplewithasimplestoryfromoutside
thearenaof information technology todrivehomethepoint.
1.3 THE INCOMPLETECAR
Eventually, whatever asoftwareorganization develops should meet the
needs of thecustomer. Everything elseissecondary. Testingis ameans of
making surethat theproduct meets theneeds of thecustomer.
Wewould liketo assign abroader meaning to theterm"customer." It
does not mean just external customers. There are also internal customers.
For example, if aproduct isbuilt using different components fromdifferent
groups within an organization, the users of these different components
shouldbeconsideredcustomers, eveniftheyarefromthesameorganization.
Having this customer perspective enhances thequality of all theactivities
including testing.
Wecan take the internal customer concept a step further where the
development teamconsiders thetesting teamasitsinternal customer. This
waywecanensurethat theproduct isbuilt not onlyforusagerequirements
8 Software Testing
but also for testing requirements. This concept improves "testability" of
the product and improves interactions between the development and
testingteams.
Wewould liketo urge the reader to retain these two perspectives-
customer perspective and perspective of quality not being an add-on
in the end, but built in every activity and component right fromthe
beginning-throughout thebook.
If our jobis to giveacompletecar to the customer (and not ask the
customers topaint thecar) andif our intent istomakesurethecar works
as expected, without any (major) problems, then we should ensure
that wecatchand correct all the defectsinthecar ourselves. This is the
fundamental objectiveof testing.Anythingwedointesting, itbehovesusto
remember that.
1.4 DIJKSTRA'S DOCTRINE
Consider aprogram that issupposed to accept asix-character codeand
ensure that thefirst character isnumeric and rests of the characters are
alphanumeric. Howmany combinations of input data should wetest, if
our goal istotest theprogramexhaustively?
Thefirst character canbefilledup inoneof 10ways (thedigits 0-9).
Thesecondthrough sixthcharacterscaneachbefilledupin62ways(digits
0-9, lower caselettersa-z andcapital lettersA-Z). Thismeansthat wehave
atotal of 10x (62
5
) or 9,161,328,320valid combinations of values to test.
Assumingthat eachcombinationtakes 10secondstotest, testingall these
validcombinationswill takeapproximately 2,905years!
Therefore, after 2,905years, wemay concludethat all valid inputs are
accepted. But that is not the end of the story-what will happen to the
programwhenwegiveinvalid data? Continuing theaboveexample, if we
assume there are 10punctuation characters, then wewill have to spend
a total of 44,176years to test all the valid and invalid combinations of
input data.
All this just to accept one field and test it exhaustively.,Obviously,
exhaustivetestingof areal lifeprogramisnever possible.
All theabovemeanthat wecanchoosetoexecuteonly asubset of the
tests. Tobeeffective,weshould chooseasubset of tests that canuncover
the maximumnumber of errors. Wewill discuss in Chapter 4, on Black
BoxTesting, and Chapter 3, onWhiteBoxTesting, sometechniques such
as equivalencepartitioning, boundary value analysis, codepath analysis,
andsoonwhichhelpinidentifyingsubsetsof test casesthat haveahigher
likelihoodof uncoveringdefects.
Nevertheless, regardless of whichsubset of testcaseswechoose, wecan
never be100%surethat therearenodefectsleftout. Butthen, toextendan
oldcliche,nothingcanbecertainother thandeathandtaxes,yetweliveand
doother thingsbyjudiciouslymanagingtheuncertainties.
1.5 A TEST IN TIME!
Defects in aproduct can comefromany phase. Therecould have been
errors while gathering initial requirements. If a wrong or incomplete
requirement formsthebasisfor thedesignanddevelopment of aproduct,
then that functionality can never be realized correctly in the eventual
product. Similarly, when a product design-which forms the basis for
the product development (a La coding)-is faulty, then the code that
realizes the faulty design will alsonot meet the requirements. Thus, an
essential condition should bethat every phase of software development
(requirements, design, coding, andsoon) should catchandcorrect defects
at that phase, without lettingthedefectsseeptothenext stage.
Let us look at thecost implications of lettingdefects seepthrough. If,
during requirements capture, somerequirements areerroneously captured
andtheerror isnot detecteduntil theproduct isdeliveredtothecustomer,
theorganizationincursextraexpensesfor
ffi performing awrong designbased onthewrong requirements;
ffi transforming thewrong designintowrongcodeduring thecoding
phase;
ffi testing to make sure the product complies with the (wrong)
requirement; and
ffi releasingtheproduct withthewrongfunctionality.
.InFigure1.2thedefectsinrequirementsareshowningray.Thecoloured
figureisavailableonpage457.Asyoucansee, thesegrayboxesarecarried
forwardthroughthreeofthesubsequentstages-design, codin~andtesting.
I
10 Software. Testing
Figure 1.2
How defects from early
phases add to the
costs.
Requirement
Phase
Design
Phase
Coding
Phase
Testing
Phase
When this erroneous product reaches the customer after the testing
phase, thecustomer may incur apotential downtime that canresult inloss
of productivity or business. Thisintumwould reflect asalossof goodwill
to the software product organization. On top of this loss of goodwill, the
softwareproduct organization would havetoredo all thesteps listedabove,
inorder torectifytheproblem.
Similarly,when adefect isencountered during thedesign phase (though
therequirements werecaptured correctly, depicted by yellow), thecostsof
all of thesubsequent phases (coding, testing, andsoon) havetobeincurred
multiple times. However, presumably, thecostswould belower than inthe
first case, where eventherequirements werenot captured properly. Thisis
becausethedesignerrors (represented byyellowboxes) arecarriedforward
onlytothecodingandtestingphases. Similarly,adefectinthecodingphase
iscarriedforward tothetestingphase(greenboxes). Again, asfewer phases
are affected by this defect (compared to requirements defects or design
defects), wecanexpect that thecost of defectsincodingshould belessthan
theearlier defects. Ascanbeinferred fromtheabovediscussion, thecostof
adefect iscompounded depending onthedelay indetecting thedefect.
Hence, smaller thelagtimebetween defectinjection(i.e.,when thedefect
wasintroduced) anddefect detection (i.e.,whenthedefectwasencountered
and corrected), lesser aretheunnecessary costs. Thus, it becomes essential
Figure 1.3
Compounding effect
of defects on software
costs.
10x
Reqmts Design Coding Testing Post release
Principles afTeSting
to catchthedefects asearly aspossible. Industry data has reaffirmed these
findings. Whilethere isno consensus about thecostsincurred due to delay
in defect detection, adefect introduced during the requirement phase that
makes ittothefinal releasemay costasmuch asathousand times thecost of
detecting and correcting thedefect during requirements gathering itself.
1.6 THECATANDTHESAINT
Testingrequires asking about and understanding what you aretrying to
test, knowing what thecorrect outcome is, andwhy youareperforming any
test. If wecarry out tests without understanding why wearerunning them,
wewill endup inrunning inappropriate tests that donot address what the
product should do. Infact, it may eventurn out that theproduct ismodified
tomakesurethetestsarerun successfully, evenif theproduct doesnot meet
theintended customer needs!
Understanding therationale of why wearetesting certain functionality
leadstodifferent types of tests, whichwewill coverinPart II of thebook. We
do white box testing to check thevarious paths inthe codeand make sure
they areexercisedcorrectly. Knowing which codepaths should beexercised
foragiventest enablesmaking necessary changes toensure that appropriate
paths arecovered. Knowing theexternal functionality of what theproduct
should do, wedesignblackboxtests. Integration testsareused tomakesure
that thedifferent components fittogether. Internationalization testingisused
toensure that theproduct works with multiple languages found indifferent
parts of theworld. Regression testing isdone to ensure that changes work
asdesigned and donot haveany unintended side-effects.
12 $oftware Testing
1.7 TESTTHETESTSFIRST!
Fromtheaboveexample, it isclear that it istheaudiologist who has a
hearingproblem, notthepatient!Imagineifthedoctor prescribedatreatment
for thepatient assuming that thelatter couldnot hear at20feetand30feet.
Testsarealsoartifacts produced by human beings, much as programs
and documents are. Wecannot assumethat thetestswill beperfect either!
It isimportant tomakesurethat thetests themselves arenot faulty before
westart using them. Oneway of making surethat thetests aretested isto
document theinputs and expected outputs for agiven test and have this
description validated byanexpert or getitcounter-checkedbysomemeans
outside thetests themselves. For example, by givingaknown input value
and separately tracing out thepath tobefollowedby theprogram or the
process, one can manually ascertain the output that should be obtained.
Bycomparing this "known correct result" with theresult produced by the
product, theconfidencelevel ofthetestandtheproduct canbeincreased. The
practices of reviews andinspection andmeticulous test planning discussed
inChapter 3andChapter 15provide means totest thetest.
1.8 THEPESTICIDE PARADOX
Defectsarelikepests; testing islikedesigning theright pesticides tocatch
andkill thepests; andthetest casesthat arewritten arelikepesticides. Just
likepests, defects develop immunity against test cases! As and when we
writenewtest casesanduncover newdefectsintheproduct, other defects
that were"hiding" underneath showup.
Principle~ of Testing 13
There are two possible ways to explain how products develop this
"immunity" against test cases. Oneexplanation is that theinitial tests go
acertain distanceinto the codeand arestopped fromproceeding further
becauseof thedefectstheyencounter. Oncethesedefectsarefixed,thetests
proceedfurther, encounter newer parts of thecodethat havenot beendealt
withbefore, and uncover new defects. Thistakes a"whitebox" or acode
approach toexplainwhynewdefectsgetunearthed withnewer tests.
A second explanation for immunity is that when users (testers) start
using (exercising) aproduct, the initial defects prevent themfromusing
thefull external functionality. Astests arerun, defectsareuncovered, and
problems arefixed, users gettoexplorenewfunctionality that hasnot been
usedbeforeand thiscausesnewer defectstobeexposed. This"blackbox"
viewtakesafunctionality approach toexplainthecausefor this "morewe
test moredefectscorneup" phenomenon.
An alternative way of looking at this problemis not that the defects
developimmunity but thetestsgodeeper tofurther diagnoseaproblemand
thus eventually "kill thedefects." Unfortunately, giventhecomplexnature
of softwareandtheinteractions amongmultiple components, thisfinal kill
happens veryrarely.Defectsstill survivethetests, haunt thecustomers, and
causeuntold havoc.
Software Testing
Theneed for constantly revising the tests to be run, with the intent
of identifying new strains of the defects, will take us to test planning
and different types of tests, especially regression tests. Regression tests
acknowledge that new fixes (pesticides) can cause new "side-effects"
(new strains of pests) and can also cause some older defects to appear.
Thechallenge in designing and running regression tests centers around
designing theright teststocombatnewdefectsintroduced bytheimmunity
acquired by aprogram against oldtest cases. Wewill discuss regression
testsinChapter 8.
1.9 THE CONVOYAND THE RAGS
Defectsin aprogram alsotypically display this convoy phenomenon.
They occur inclusters. Glenford Myers, inhis seminal work on software
testing[MYER-79],proposed that theprobability of the existence of more errors
in a section of a program is proportional to the number of errors already found in
that section.
Thismaysoundcounter-intuitive, but canbelogicallyreasonedout. Afix
foronedefectgenerallyintroduces someinstabilityandnecessitatesanother
fix.All thesefixesproduce side-effectsthat eventually causetheconvoyof
defectsincertainparts of theproduct.
Fromatest planning perspective, this means that if wefind defectsin
a particular part of product, more-not less-effort should be spent on
testingthat part. Thiswill increasethereturn oninvestments intesting as
thepurpose of testingisfindthedefects. Thisalsomeans that whenever a
product undergoes anychange, theseerror-proneareasneedtobetestedas
theymayget affected. Wewill cover theseaspectsinChapter 8,Regression
Testing.
Figure 1.4
The number of defects
yet to be found
increases with the
number of defects
uncovered.
"0
c:
~
Q
Q)
.c
,g
~
e n
~
o
Number of defects found
A fix for a defect is made around certain lines of code. This fix can
produce side-effects around the same pieceof code. This sets in spiraling
changes totheprogram, all localized tocertain select portions of thecode.
When welook at the codethat got thefixesfor the convoy 6f defects, it is
likely tolook likeapieceof rag! Fixingatear inoneplaceinashirt would
most likely causedamage inanother place. Theonly long-termsolution in
suchacaseistothrowaway theshirt andcreateanewone.Thisamounts to
are-architecting thedesign andrewriting thecode.
1.10 THE POLICEMEN ON THE BRIDGE
Testersare probably best equipped to know the problems customers
mayencounter. Likethesecondpoliceofficerintheabovestory,theyknow
peoplefall andtheyknowwhypeoplefall.Rather thansimplycatchpeople
whofall (andtherebybeexposedtotheriskof amissedcatch), theyshould
alsolook at theroot causefor fallingand advisepreventive action. It may
not bepossiblefor testersthemselvestocarryout preventiveaction. Just as
thesecondpoliceofficerhad toenlist thehelp of anengineer toplug the
hole, testerswouldhavetoworkwithdevelopment engineers tomakesure
theroot causeof thedefectsareaddressed. Thetestersshould not feel that
by eliminating theproblems totally their jobsareat stake. Likethesecond
policeman, their careerscanbeenrichingandbeneficial totheorganization
if theyharness their defectdetectionexperienceandtransformsomeof itto
defectprevention initiatives.
Defectprevention isapart of atester's job. A career as atester canbe
enriching and rewarding, if wecanbalancedefect prevention and defect
detectionactivities. Someof thesecareer pathpossibilitiesareencapsulated
inathree-stagemodel inChapter 13,CommonPeopleIssues. Wewill now
visitthequestionof what istherightbalancebetweendefectprevention and
defectdetection.
1.11 THEENDSOFTHEPENDULUM
The eventual goal of any software organization is to ensure that the
customers get products that arereasonably freeof defects. Therearetwo
approaches toachievingthisgoal. Oneistofocusondefect detection and
correction and thesecondistofocusondefect prevention. Thesearealso
calledquality control focusandquality assurance focus.
Testing is traditionally considered as a quality control activity, with
anemphasis on defect detection and correction. Wewould liketo take a
broader viewof testingandbelievethat thereareaspectsof testingthat are
also defect prevention oriented. For example, oneof theaspects of white
box testing, discussed in Chapter 3, is static testing, that involves desk
checking, codewalkthroughs, codereviews, and inspection. Eventhough
thesearetraditionally consideredas"qualityassurance" activities, planning
for overall testing activities with anintent to deliver quality products to
customers, cannotbedoneeffectivelyunlesswetakeaholisticviewof what
canbedone using quality assurance and what canbedone with quality
control (orthetraditional definitionof testing).
Quality assuranceis normally associatedwith process models such as
CMM, CMMI, ISO9001,and soon. Quality control, on the other hand, is
associatedwith testing(that formthebulk of thediscussionsinthisbook).
This has caused an unnatural dichotomy between these two functions.
Unfortunately,organizationsviewthesetwofunctionsasmutually exclusive,
"either-or" choices. Wehave even heard statements such as "with good
processes,testingbecomesredundant" or"processesaremereoverheads-we
Principles of Testing
Figure 1.5
Quality control and
quality assurance as
two methods to achieve
quality.
Dp.fect Detection "
,
,
,
,
, ,
. .
. .
. .
. .
, ,
.......
Defect Prevention
.... \,
: ;
. ,
, ,
........
canfindout everything by testing." It isalmost asif therearetwo schools
of thought at either extremes of a pendulum-one rooting for defect
prevention (quality assurance) focus and the other rooting for the defect
detection(quality control) focus. Itisalsocommontofindanorganization
swinging from one extreme to another over time, like a pendulum
(Figure1.5).
Rather than view defect prevention and defect detection as mutually
exclusivefunctions or ends of apendulum, webelieveit isworthwhile to
view thesetwo as supplementary activities, being done in the right mix.
Figure 1.6gives adefect prevention-defect detection grid, which views
the two functions as two dimensions. Theright mix of the two activities
corresponds tochoosingtheright quadrant inthisgrid.
Whenthefocusondefectprevention islow, theemphasis ontheuseof
appropriate standards, reviews, andprocesses areverylow. Thisactsasan
ideal "breeding ground" for defects. Most of theeffortinensuring quality
of aproduct isleftinthehands of thetestingand defect detectionteam. If
thefocusondefectdetectionisalsolow(representedbythelower left-hand
quadrant), thisisabadstateforanorganization tobein. Lackof testingand
Figure 1.6
Relationship between
defect detection focus
and defect prevention
focus.
Last minute rushes
Higher people dependency
Testers as "heroes" and
"adversaries"
Not a healthy state!
Lack of standards foster
"defect breeding"
Lack of testing makes
defects reach the customers
May be resource intensive but
gives better payback
Institutionalizes quality
Makes quality visible to
customers
Double-edged sword!
Excessive process
orientation
Lack of testing makes
defects reach the customers
Defect prevention focus

18 SoAware Te;~ing"
defect detection activities does not "kill" these defects in time; hence the
defectsreachthecustomers. Thisisobviously not ahealthy statetobein.
Evenwhenthedefectdetectionfocusincreases,withcontinuedlowdefect
prevention focus(upper lefthand quadrant), thetestingfunctions becomea
high-adrenalin rush, high-pressure job. Most defectsaredetectedinthelast
minute-before theproduct release. Testersthus becomesuperheroes who
"savetheday" byfindingall thedefectsjust intime. Theymayalsobecome
adversaries to developers as they always seemto find problems in what
thedevelopers do. Thisquadrant isbetter than theprevious one, but ends
up being difficult tosustain becausethelast-minute adrenalin rush burns
peopleout faster.
Preventing anillnessismoreeffectivethancuringit. Peoplewhoprevent
defects usually do not get much attention. They are usually the unsung
heroesof anorganization. Thosewhoput out thefiresaretheoneswho get
visibility, not necessarily thosewho make surefiresdo not happen inthe
first place. This, however, should not deter themotivation of people from
defect prevention.
Aswesawintheprevious section,defectprevention anddefectdetection
arenotmutually exclusive.Theyneedtobebalancedproperly forproducing
aquality product. Defect prevention improves the quality of the process
producing theproducts whiledefectdetectionandtestingisneeded tocatch
andcorrectdefectsthat escapetheprocess. Defectprevention isthusprocess
focusedwhiledefect detection isproduct focused. Defect detection actsas
anextrachecktoaugment theeffectivenessof defectprevention.
Anincreaseindefect prevention focusenables putting inplacereview
mechanisms, upfront standards tobefollowed, and documented processes
for performing thejob. Thisupfront and proactive focus on doing things
right to start with causes the testing (or defect detection) function to add
morevalue, andenablescatchinganyresidual defects(thatescapethedefect
prevention activities) before the defects reach the customers. Quality is
institutionalized withthisconsistentlyhighfocusonbothdefect prevention
and defect detection. An organization may have to allocate sufficient
resources for sustaining ahigh level of both defect prevention and defect
detectionactivities(upper right-hand quadrant inFigure1.6).
However, anorganization should becareful about not relyingtoomuch
ondefectprevention andreducingthefocusondefectdetection(lowerright-
hand quadrant inFigure1.6).Suchahighfocusondefectprevention andlow
focusondefect detectionwouldnot createafeelingof comfort amongst the
management onthequality of product released sincetherearelikely tobe
minimal internal defectsfound. Thisfeelingwill giverisetointroduction of
newprocessestoimprovetheeffectivenessof defect detection. Toomuchof
processesandsuchdefectprevention initiativesmayendupbeingperceived
asabureaucraticexercise,notflexibleoradaptabletodifferentscenarios.While
processesbringindisciplineandreducedependency onspecificindividuals,
they-when not implemented in spirit-could also end up being double-
edged swords, actingasadamper topeople'sdriveandinitiative. Whenan
organization pays equally high emphasis to defect prevention and defect
detection(upper right comer inthegrid), it may appear that it isexpensive
but thisinvestment isbound tohavearichpayback byinstitutional quality
internallyandmakingthebenefitsvisibleexternallytothecustomers.
Anorganization should choosethe right placeon eachof these two-
defect detection and defect prevention- dimensions and thus choose
theright placeinthegrid. Therelativeemphasis tobeplaced on thetwo
dimensions will varywiththetypeof product, closenesstothereleasedate,
and the resources available. Making aconscious choiceof thebalance by
consideringthevariousfactorswill enableanorganization toproducebetter
quality products. Itisimportant for anorganization not toover-emphasize
oneof theseattheexpenseof theother, asthenext sectionwill show.
1.12 MEN IN BLACK
As wecanseefromall the above discussions, testing requires abundant
talent inmultiple dimensions. Peopleinthetestingprofession should have
a customer focus, understanding the implications from the customer's
perspective. Theyshouldhaveadequate analytical skillstobeabletochoose
theright subset of testsandbeabletocounter thepesticide paradox. They
should think ahead interms of defect prevention and yet beableto spot
and rectify errors that cropup. Finally (aswewill seeinthenext section),
they must beabletoperformautomation functions.
Despiteall thesechallengingtechnical andinter-personal skillsrequired,
testing still remains a not-much-sought-after function. There was an
interesting experiment that was described by DeMarcoand Lister intheir
book, Peopleware [DEMA-1987].Thetestingteamwasseededwithmotivated
peoplewhowere"freefromcognitivedissonancethat hampers developers '.
when testing their own programs." Theteamwas given anidentity (by a
blackdress, amidst thetraditionally dressed remainder of theorganization)
20 Software Testing
and tremendous importance. All this increased their pride in work and
made their performance growby leaps and bounds, "almost likemagic."
Longaftertheindividual founding members leftandwerereplacedbynew
people, the"BlackTeam" continued itsexistenceandreputation.
Thebiggest bottleneck intakingup testingasaprofessionisthelackof
self-belief. Thislackof self-belief and apparent distrust of theexistenceof
career options intestingmakes peopleviewtheprofession asalaunching
pad todoother softwarefunctions (notably, "development," aeuphemism
forcoding).Asaresult, testersdonotnecessarilyseekacareerpathintesting
anddevelopskepticismtowards theprofession.
Wehave devoted an entire chapter in Part III of the book to career
aspirations andother similar issuesthat peopleface.Apart of thechallenge
that is faced is the context of globalization-the need to harness global
resources tostaycompetitive. Weaddress theorganizational issues arising
out of thisinanother chapter inPartIII.
1.13 AUTOMATIONSYNDROME
Principles ofTeiiting 21
If you gothrough thestory closelythereappear tobeseveral reasons
forthecropfailuresthat arenot todowiththeautomationintent atall. The
frustration of thefarmer should not bedirected at automation but onthe
processfollowedforautomationandtheinappropriate choicesmade. Inthe
secondcropcycle,thereasonfor failurewaslackof skillsandinthethird
cycleitisduetoimproper tool implementation.
Inthefirst cropcycle,thefarmer laidoff hisworkers immediately after
thepurchaseof motorcyclesandexpectedcostandtimetocornedown. He
repeated thesamemistakefor thethird cropcycle.Automation does not
yieldresultsimmediately.
Themoral of theabovestory asit applies totestingisthat automation
requires careful planning, evaluation, and training. Automation may
not produce immediate returns. An organization that expects immediate
returns fromautomation may end up being disappointed and wrongly
blameautomation for their failures, instead of objectivelylookingat their
level of preparedness for automation in terms of planning, evaluation,
andtraining.
Alargenumber of organizations fail intheir automation initiativesand
revert to manual testing. Unfortunately, they conclude-wrongly-that
automationwill never work.
Testing,bynature, involvesrepetitivework. Thus,itlendsitselfnaturally
to automation. However, automation is adouble-edged sword. Someof
thepoints that should bekept inmind whileharping onautomation are
asfollows.
~ Know first why you want to automate and what you want to
automate, beforerecommendingautomationforautomation's sake.
~ Evaluate multiple tools. before choosing one as being most
appropriate foryour need.
~ Trytochoosetoolstomatchyour needs, rather thanchangingyour
needs tomatchthetool'scapabilities.
~ Trainpeople firstbeforeexpecting themtobeproductive.
~ Donot expect overnight returns fromautomation.
1.14 PUTTING IT ALL TOGETHER
We have discussed several basic principles of testing in this chapter.
These principles provide an anchor to the other chapters that we have
in rest of the book. Wehave organized the book into five parts. The
first part (which includes this chapter) is Setting the Context, which sets
the context for therest of thebook. Inthechapter that follows, wecover
SoftwareDevelopment UfeCycle(SDLC)Models inthecontext of testing,
verification and validation activities.
In Part It Types of Testing, we cover the common types of testing.
Chapters 3through 10coverwhiteboxtesting, blackboxtesting, integration
testing, system and acceptance testing, performance testing, regression
testing, internationalization testing, andadhoctesting.
Part III, Select Topics in Specialized Testing, addresses two specific and
somewhat esoterictestingtopics-object oriented testinginChapter 11and
usability andaccessibilitytestinginChapter 12.
Part IV,People and Organizational Issues inTesting, provides anoftignored
perspective. Chapter 13 addresses the common people issues like
misconceptions, career path concerns and so on. Chapter 14address the
different organizational structures invoguetosetupeffectivetestingteams,
especiallyinthecontextof globalization.
The final part, Part V, Test Management and Automation, addresses the
process, management, andautomationissuestoensureeffectivetestinginan
organization. Chapter 16discussestest planningmanagement andexecution.
Thisdiscussesvariousaspectsofputtingtogetheratestplan, trackingatesting
projectandrelatedissues.Chapter17goesintodetailsofthebenefits,challenges,
and approaches in test automation-an area of emerging and increasing
importancein thetest community. Thefinal chapter, Chapter 18, goes into
details of what data arerequired to be captured and what analysis is to
beperformed for measuring effectiveness of testing, quality of aproduct
and similar perspectives and howthis information canbeused to achieve
quantifiablecontinuous improvement.
Whilewehaveprovided thenecessarytheoretical foundation indifferent
parts of thebook, our emphasis throughout thebookhasbeenonthestateof
practice. Thissectionshould set thecontext for what thereader canexpect
inrest of thebook.
One of the early seminal works on testing is [MYER-79].In particular,
theexample of trying to write test casesfor verifying three numbers to
bethesides of avalidtriangle still remains oneof thebest ways tobring
forth the principles of testing. [DEMA-87]provides several interesting
perspectives of the entire software engineering discipline. Theconcept
of black teamhasbeenillustrated inthat work. Theemphasis required for
processand quality assurancemethodologies andthebalancetobestruck
betweenquality assuranceandquality control arebrought out in[HUMP-
86].Someof theuniversally applicablequality principles arediscussed in
theclassics[CROS-80]and [DEMI-86].[DIJK-72],aTuringAward lecture
brings out thedoctrineof programtestingcannever provetheabsenceof
defects. [BEIZ-90]discussesthepesticideparadox.
1. Wehave talked about the pervasiveness of software as areason
why defectsleftinaproduct would get detected sooner thanlater.
Assume that televisions with embedded software were able to
download, install and self-correct patches over thecablenetwork
automatically and theTV manufacturer told you that this would
just takefiveminutes everyweek"atnocosttoyou, theconsumer."
Wouldyouagree?Givesomereasons why thisisnot acceptable.
2. Yourorganization hasbeensuccessful indeveloping aclient-server
application that isinstalled at several customer locations. Youare
changingtheapplicationtobeahosted, web-basedapplication that
anyonecanuseafter asimpleregistration process. Outlinesomeof
thechallenges that you should expect fromaquality and testing
perspectiveof thechanged application.
3. Thefollowing were someof thestatements made by people in a
product development organization. Identify the fallacies if any
inthe statements and relate it to theprinciples discussed in this
chapter.
a. "Thecodefor this product is generated automatically by a
CASEtool- itisthereforedefect- free."
b. "Wearecertifiedaccordingtothelatest process models - we
donot needtesting."
c. "Weneedtotestthesoftwarewithdot matrixprinters because
wehavenever released aproduct without testing with adot
matrix printer."
d. "I haverun all thetests that I havebeenrunning for thelast
tworeleasesandI don't needtorunanymoretests."
e. "This automation tool is being used by our competitors -
henceweshouldalsousethesametool."
4. Assumethat eachdefect ingathering requirements allowedtogo
to customers costs $10,000,and that thecorresponding costsfor
designdefectsandcodingdefectsare$1,000and$100,respectively.
Also, assume that current statistics indicate that on average ten
newdefectscomefromeachof thephases. Inaddition, eachphase
alsoletsthedefectsfromtheprevious phase seepthrough. What
is thetotal cost of thedefectsunder thecurrent scenario? If you
put aquality assuranceprocess to catch50%of thedefects from
eachphasenot togotothenext phase, what aretheexpectedcost
savings?
5. Youaretowriteaprogramthat adds twotwo-digit integers. Can
you test this program exhaustively? If so, how many test cases
arerequired? Assuming that eachtest casecanbeexecuted and
analyzed inonesecond, howlongwould it takefor youtorun all
thetests?
6. We argued that the number of defects left in a program is
proportional tothenumber of defectsdetected. Givereasons why
this argument lookscounterintuitive. Also, givepractical reasons
why thisphenomenon causesproblems intesting.
Inthis chapter-
,/ Phases of software project
,/ Quality, quality assurance, and quality control
,/ Testing, verification, and validation
,/ Process model to represent different phases
,/ Lifecyclemodels
26 Software jesting
2.1 PHASESOF SOFTWARE PROJECT
Asoftwareproject ismadeupof aseriesof phases. Broadly,most software
projectscomprisethefollowingphases.
~ Requirements gathering andanalysis
~ Planning
~ Design
~ Development or coding
~ Testing
~ Deployment andmaintenance
2.1.1
Requirements Gathering and Analysis
During requirements gathering, thespecificrequirements of thesoftware
to be built are gathered and documented. If the software is bespoke
software, thenthereisasinglecustomer whocangivetheserequirements.
If the product is ageneral-purpose software, then aproduct marketing
teamwithin thesoftwareproduct organization specifiestherequirements
by aggregating the requirements of multiple potential customers. In
either case, it is important to ensure that the right requirements are
captured at every stage. Therequirements get documented in the form
of aSystemRequirements Specification(SRS)document. This document
actsasabridgebetweenthecustomer andthedesigners chartered tobuild
theproduct.
2.1.2 Planning
Thepurpose of the planning phase is to comeup with aschedule, the
scope, and resourcerequirements for arelease. A plan explains how the
requirements will bemet andbywhichtime. Itneeds totakeintoaccount
the requirements-what will bemet and what will not bemet-for the
current release to decide on the scope for the project, look at resource
availability, and to comeout with set of milestones and release date for
theproject. Theplanning phase is applicable for both development and
testing activities. At theendof thisphase, bothproject plan and test plan
documents aredelivered.
2.1.3 Design
The purpose of the design phase is to figure out how to satisfy the
requirements enumerated in the System Requirements Specification
document. Thedesign phaseproduces arepresentation that will beused
bythefollowingphase, thedevelopment phase. Thisrepresentation should
servetwopurposes. First, fromthis representation, it should bepossible
toverifythat all therequirements aresatisfied. Second, thisrepresentation
Sojtware pevelgpmeftt L i f e C y c le, Models 27
should givesufficient information for thedevelopment phase toproceed
withthecodingandimplementation of thesystem. Designisusually split
intotwolevels-high-Ievel designandlow-level or adetailed design. The
designstepproduces thesystemdesigndescription (SDD)document that
will beusedby development teams toproduce theprograms that realize
thedesign.
2.1.4 Development or Coding
Designactsasablueprint fortheactual codingtoproceed. Thisdevelopment
orcodingphasecomprisescodingtheprograms inthechosenprogramming
language. Itproduces thesoftwarethat meetstherequirements thedesign
wasmeant tosatisfy.Inaddition toprogramming, thisphasealsoinvolves
thecreationof product documentation.
2.1.5 Testing
Astheprograms arecoded (inthechosenprogramming language), they
are also tested. In addition, after the coding is (deemed) complete, the
product is subjected to testing. Testingis the process of exercising the
softwareproduct inpre-defined waystocheckif thebehavior isthesame
as expected behavior. Bytesting the product, an organization identifies
andremoves asmany defectsaspossiblebeforeshipping it out.
2.1.6 Deployment and Maintenance
Onceaproduct istested, itisgiventothecustomers whodeploy itintheir
environments. Astheusers start usingtheproduct intheir environments,
theymayobservediscrepanciesbetweentheactual behavior oftheproduct
and what they were given to expect (either by the marketing people or
through theproduct documentation). Suchdiscrepancies couldendup as
product defects, whichneed tobecorrected. Theproduct nowenters the
maintenancephase, whereintheproduct ismaintained orchangedtosatisfy
thechangesthat arisefromcustomer expectations, environmental changes,
etc. Maintenance ismadeup of c orrec tive maintenanc e (for example, fixing
customer-reported problems), adaptive maintenanc e (for example, making
thesoftware run onanew version of an operating systemor database),
andpreventive maintenanc e (forexample, changingtheapplication program
codetoavoidapotential security holeinanoperating systemcode).
2.2 QUALITY, QUALITY ASSURANCE, AND QUALITY CONTROL
A software product is designed to satisfy certain requirements of a
given customer (or set of customers). How can we characterize this
phrase-"satisfying requirements"? Requirements get translated into
softwarefeatures, eachfeaturebeingdesigned tomeet oneor moreof the
28 Software Testing
requirements. For eachsuchfeature, theexpected behavior ischaracterized
byasetof test cases. Eachtest caseisfurther characterizedby
1. Theenvironment under whichthetest caseistobeexecuted;
2. Inputs that shouldbeprovided forthat test case;
3. Howtheseinputs should getprocessed;
4. What changes should be produced in the internal state or
environment; and
5. What outputs shouldbeproduced.
Theactual behavior of agivensoftwarefor agiventest case, under a
givensetof inputs, inagivenenvironment, andinagiveninternal stateis
characterizedby
1. Howtheseinputs actuallygetprocessed;
2. What changes are actually produced in the internal state or
environment; and
3. What outputs areactuallyproduced.
If the actual behavior and the expectedbehavior are identical in all
their characteristics,thenthat testcaseissaidtobepassed. Ifnot, thegiven
softwareissaidtohaveadefect onthat testcase.
Howdoweincreasethechancesof aproduct meetingtherequirements
expected of it, consistently and predictably? There are two types of
methods-quality control andqualityassurance.
Qualitycontrol attemptstobuildaproduct, testitforexpectedbehavior
after it isbuilt, andif theexpectedbehavior isnot thesameastheactual
behavior of theproduct, fixestheproduct asisnecessaryandrebuilds the
product. Thisiterationisrepeatedtill theexpectedbehavior of theproduct
matchestheactual behavior forthescenariostested. Thusqualitycontrol is
defect-detectionanddefect-correctionoriented, andworksontheproduct
rather thanontheprocess.
Quality assurance, on the other hand, attempts defect prevention by
concentratingontheprocessof producingtheproduct rather thanworking
on defect detection/correction after the product is built. For example,
insteadof producing andthentestingaprogramcodeforproper behavior
byexercisingthebuilt product, aqualityassuranceapproachwouldbeto
first reviewthedesignbeforetheproduct isbuilt and correct thedesign
errorsinthefirstplace. Similarly,toensuretheproductionof abetter code,
aqualityassuranceprocessmaymandatecodingstandards tobefollowed
by all programmers. As canbe seen fromthe aboveexamples, quality
assurancenormally tends toapply toall theproducts that useaprocess.
Also, sincequality assurancecontinuesthroughout thelifeof theproduct
it iseverybody'sresponsibility; henceit isastaff function. Incontrast, the
responsibility for quality control isusually localizedto aquality control
team. Table2.1summarizesthekeydistinctionsbetweenqualitycontrol and
qualityassurance.
Software Development Life Cycle lVfodeis 29
Table 2.1. Difference between quality assurance and quality control.
Wewill seemore details of quality assurance methods such as reviews
and audits inChapter 3. Butthefocusof therest of thisbook isonsoftware
testing, which is essentially aquality control activity. Let us discuss more
about testinginthenext section.
2.3 TESTING, VERIFICATION, AND VALIDATION
Thenarrow definition of theterm"testing" isthephase that followscoding
and precedes deployment. Testingistraditionally used tomean testing of
the program code. However, coding is adownstream activity, as against
requirements and design that occur much earlier in aproject life cycle.
Given that the objectiveof asoftware project is to minimize and prevent
defects, testing of program code alone is not sufficient. As wesaw in the
lastchapter, defectscancreepinduringanyphaseandthesedefectsshouldbe
detectedasclosetothepoint ofinjectionaspossibleandnotwaittill thetesting
of programs. Henceagainst this, if eachphaseis "tested" separately asand
when thephaseiscompleted (or,better still, asthephaseisbeingexecuted),
thendefectscanbedetectedearly,thereby reducingtheoverall costs.
Timelytesting increases thechances of aproduct or servicemeeting the
customer's requirements. When aproduct is tested with appropriate and
realistic tests that reflect typical usage patterns by theintended users, the
chancesof theproduct satisfyingthecustomer's requirement ismuchhigher.
While testing does not guarantee zero defects, effectivetesting certainly
increases thechancesof customer acceptanceof thesoftware.
Thepurpose of testing is touncover defects inthesystem(and tohave
someonefixthedefects). Testingisdonebyasetof peoplewithin asoftware
product (or service) organization whose goal and charter istouncover the
defectsintheproduct beforeitreaches thecustomer (seeSection1.3).Aswe
sawintheprevious chapter, thepurpose of testingisNOT toprovethat the
product hasnodefects. Thepurpose of softwaretestingistofinddefectsina
softwareproduct. Aswewill seeinthechapters onpeopleandorganizational
Sofrwar~< Iesting
issues(Chapters 13,14),thereward systemsandtheorganization structures
should create and foster an environment that encourages this purpose
of testing.
Testingis NOT meant to replace other ways of ensuring quality (like
reviews). It isoneof themethods todetect defects inasoftwareproduct.
Thereareother methods that achievethesamefunction. For example, we
will seelater that followingwell-defined processes and standards reduces
thechancesof defectscreepinginto asoftware. Wewill alsodiscuss other
methods likereviews and inspections, which actually attempt to prevent
defectscomingintotheproduct. Tobeeffective,testingshouldcomplement,
supplement, andaugment suchqualityassurancemethods discussedinthe
previous section.
Theidea of catching defects within each phase, without letting them
reach the testing phase, leads us to definetwo more terms-verification
andvalidation.
Duringtherequirements gatheringphase, therequirements arefaithfully
captured. TheSRSdocument istheproduct of therequirements phase. To
ensurethat requirements arefaithfully captured, thecustomer verifiesthis
document. Thedesign phase takes theSRSdocument as input and maps
therequirements toadesignthat candrivethecoding. TheSDDdocument
istheproduct of thedesignphase. TheSDDisverifiedbytherequirements
teamtoensurethat thedesignfaithfullyreflectstheSRS,whichimposedthe
conditions at thebeginningof thedesignphase.
Verificationtakes care of activities to focus on the question "Are we
building the product right?" andvalidation takes careof aset of activitiesto
address thequestion"Are we building the right product?"
Tobuild theproduct right, certainactivities/conditions/procedures are
imposed at thebeginning of thelifecycle. Theseactivities areconsidered
"proactive" astheir purpose istoprevent thedefectsbeforetheytakeshape.
Theprocess activities carried out during various phases for each of the
product releasescanbetermedasverification.Requirements review, design
review, andcodereviewaresomeexamplesof verificationactivities.
Tobuild the right product, certain activities are carried out during
variousphasestovalidatewhether theproduct isbuilt asper specifications.
Theseactivitiesareconsidered "reactive" astheir purpose istofinddefects
that affecttheproduct and fixthemassoonasthey areintroduced. Some
examples of validation includeunit testingperformed toverifyif thecode
logicworks, integration testingperformed toverifythedesign, andsystem
testingperformed toverifythat therequirements aremet.
Tosummarize, therearedifferent terminologies that may stand for the
sameor similar concepts. For all practical purposes in this book, wecan
assumeverificationandqualityassurancetobeoneandthesame. Similarly
quality control, validation, andtestingmeanthesame.
Software Development Life
2.4 PROCESSMODEL TO REPRESENTDIFFERENT PHASES
A process model is a way to represent any given phase of software
development that effectively builds in the concepts of validation and
verification to prevent and minimize the delay between defect injection
and defect detection (andeventual correction). Inthis model, eachphase
of asoftwareproject ischaracterized bythefollowing.
~ Entry criteria, which specifywhen that phase canbestarted. Also
included aretheinputs for thephase.
~ Tasks,or steps that needtobecarriedout inthat phase, alongwith
measurements that characterizethetasks.
~ Verification, which specifies methods of checking that the tasks
havebeencarriedout correctly.
~ Exit criteria, which stipulate the conditions under which one can
consider thephaseasdone. Alsoincluded aretheoutputs for only
thephase.
Thismodel, known astheEntryTaskVerificationeXitor ETVXmodel,
offersseveral advantages foreffectiveverificationandvalidation.
1. Clear entry criteria make sure that agiven phase does not start
prematurely.
2. The verification for each phase (or each activity in each phase)
helpsprevent defects, or atleast, minimizes thetimedelaybetween
defect injectionanddefect detection.
3. Documentation of the detailed tasks that comprise each phase
reduces theambiguity ininterpretation of theinstructions andthus
minimizes thevariations that cancomefromrepeated executions
of thesetasksbydifferent individuals.
4. Clear exit criteriaprovide ameans of validation of thephase, after
thephaseisdonebut beforehanding over tothenext phase.
Anexampleof applyingtheETVXmodel tothedesignphaseispresented
inFigure2.1.
Figure 2.1
ETVX model applied
to design. Entry criteria:
Approval of SRS by customer
0 8
Input:
Approved SRS
Exit criteria:
Complete traceability between
design and SRS
Development team ready to start
programming
D8
*
; ti
,: : >:
Output:
Architecture documents
Design documents
Program specifications
32 Software Testing
2.5 LIFE CYCLEMODELS
TheETVXmodel characterizes aphase of aproject. A LifeCyclemodel
describes howthephases combinetogether toformacompleteproject or
lifecycle.Suchamodel ischaracterized bythefollowingattributes.
The activities performed Inany givensoftwareproject, apart fromthe
most common activities or phases-requirements gathering, design,
development, testing, andmaintenance-there couldbeother activitiesas
well. Someof these activities could betechnical activities (for example,
porting) andsomecouldbenon-technical (forexample, hiring).
The deliverables from each activity Each activity produces a set of
deliverables, whicharetheendproducts of that activity. For example, the
requirements gathering phase produces the SRSdocument, the design
phaseproduces theSODdocument, andsoon.
Methods of validation of the deliverables Theoutputs produced by a
givenactivityrepresent thegoal tobesatisfiedbythat activity. Henceit is
necessary tohaveproper validation criteriafor eachoutput.
The sequence of activities The different activities work together in
unison inacertainsequenceof steps toachieveoverall project goals. For
example, theprocessof requirements gathering mayinvolvesteps suchas
interviews with customers, documentation of requirements, validation of
documented requirements withcustomers, andfreezing of requirements.
These steps may be repeated as many times as needed to get the final
frozenrequirements.
Methods of verification of each activity, including the mechanism of
communication amongst the activities Thedifferent activities interact
with one another by means of communication methods. For example,
when adefect isfound inoneactivity and istracedback tothecauses in
anearlier activity, proper verificationmethods areneeded toretracesteps
fromthepoint of defecttothecauseof thedefect.
Wewill nowlookatsomeof thecommonlifecyclemodelsthat areused
insoftwareprojects.Foreachmodel, wewill lookat:
1. abrief description of themodel;
2. the relationship of the model to verification and validation
activities;and
3. typical scenarioswherethat lifecyclemodel isuseful.
2.5.1 Waterfall Model
IntheWaterfall model, aprojectisdividedintoasetofphases(oractivities).
Eachphaseisdistinct, that is,thereareclearlinesofseparationbetweenthe
phases, withverycleardemarcation of thefunctions of eachof thephases.
Cycle Models
Figure 2.2
Waterfall model.
Aproject starts with aninitial phase, and upon completion of thephase,
moves ontothenext phase. Onthecompletion of this phase, theproject
moves to the subsequent phase and so on. Thus the phases are strictly
timesequenced.
Wedepict oneexampleof aprojectintheWaterfall model inFigure2.2.
Theprojectgoesthrough aphaseof requirements gathering. At theend of
requirements gathering, aSystemRequirements Specificationdocument is
produced.Thisbecomestheinputto thedesignphase.Duringthedesignphase,
adetaileddesignisproduced intheformof aSystemDesignDescription.
WiththeSODasinput, theprojectproceeds tothedevelopment or coding
phase, whereinprogrammers developtheprograms required tosatisfythe
design. Oncetheprogrammers completetheir codingtasks, they hand the
product tothetestingteam, whotesttheproduct beforeitisreleased.
If there is no problemin agivenphase, then this method canwork,
goinginonedirection (likeawaterfall). But what would happen if there
areproblems after goingto aparticular phase? For example, you gointo
thedesignphaseandfindthat itisnot possibletosatisfytherequirements,
34 Software Testing
going by the current design approach being used. What could be the
possiblecausesandremedies? Youmay tryanalternative designif possible
andseeif that cansatisfytherequirements. If therearenoalternativedesign
approaches possible, thentheremust befeedbacktotherequirements phase
tocorrect therequirements.
Letustaketheexampleonestepfurther. Supposeadesignwascreatedfor
agivenset of requirements andtheproject passed ontotheprogramming/
development phase. Atthispoint oftime, itwasfoundthatitwasnotpossible
todeveloptheprograms becauseof somelimitations. What would youdo?
Oneapproach wouldbetotryout alternativestrategies inthedevelopment
phasesothat thedesigncouldstill besatisfied. Another possibility couldbe
that thereareflawsindesign that causeconflictsduring d.evelopment and
hencethedesign has toberevisited. Whenthedesign phase isrevisited-
likeintheprevious case-it may happen that theproblemmay havetobe
addressed intherequirements phaseitself.So,aprobleminonephasecould
potentially betracedbacktoany of theprevious phases.
Sinceeachphasehasanoutput, thelatter canbevalidated against asetof
criteria. Toincreasetheeffectiveness,thecompletioncriteriaforeachoutput
canbepublished apriori. Beforeaphasestarts, thecompletioncriteriaforthe
previous phasecanbecheckedandthiscanactasaverificationmechanism
for the phase. This can minimize the kind of delays wediscussed in the
exampleabove.
Themain strength of theWaterfall Model is its simplicity. Themodel
is very useful when a project can actually be divided into watertight
compartments. But very few software projects can be divided thus. The
major drawback intheWaterfall model arises fromthedelay infeedback
amongthephases, andthustheineffectivenessof verificationandvalidation
activities. Anerror inonephase isnot detected till at least thenext phase.
When a given phase detects an error, the communication is only to the
immediately preceding phase. This sequential nature of communication
amongthephases canintroduceinordinate delaysinresolvingtheproblem.
Thereduced responsiveness that isinherent inthemodel and thefact that
thesegregation of phases isunrealistic severelyrestrictstheapplicability of
thismodel.
2.5.2 Prototyping and Rapid Application Development Models
Prototyping andRapidApplication Development (RAD)models recognize
and address thefollowing issues.
1. Early and frequent user feedback will increase the chances of a
softwareproject meeting thecustomers' requirements.
2. Changes are unavoidable and the software development process
must beabletoadapt itself torapid changes.
Software Development Life Cycle Models 35
ThePrototyping model comprises thefollowingactivities.
1. Thesoftware development organization interacts with customers
tounderstand their requirements.
2. Thesoftwaredevelopment organizationproduces aprototypetoshow
how theeventual softwaresystemwould look like. Thisprototype
would havethemodels of howtheinput screensandoutput reports
wouldlooklike,inadditiontohavingsome"emptycanfunctionality"
todemonstratetheworkflowandprocessinglogic.
3. The customer and the development organization review the
prototype frequently sothat thecustomer's feedback is taken very
earlyinthecycle(that is, during therequirements gathering phase).
4. Basedonthefeedbackandtheprototypethatisproduced, thesoftware
development organization produces theSystemRequirements Spe-
cificationdocument.
5. Once the SRSdocument is produced, the prototype can be dis-
carded.
6. The SRS document is used as the basis for further design and
development.
Thus, theprototype issimply used asameans of quicklygathering (the
right) requirements. This model has built-in mechanisms for verification
andvalidationof therequirements. Astheprototype isbeingdeveloped, the
customer'sfrequent feedbackactsasavalidationmechanism. OncetheSRSis
produced, itactsastheverificationmechanismforthedesignandsubsequent
steps. But theverificationand validationactivitiesof thesubsequent phases
areactuallydictatedby thelifecyclemodel that isfollowedafter theSRSis
obtained.
Thismodel isobviously advantageous when acustomer canparticipate
by givingfeedback. Thismodel isalsouseful incaseswhere thefeedback
canbeeasily quantified and incorporated, for example, determining user
interface, predicting performance, andsoon.
For a general-purpose product, which is meant for many customers,
there isno singlecustomer whose feedback canbetaken asfinal. Inthese
cases, aproduct manager in the marketing group of the product vendor
usually plays theroleof theeventual customer. Hencetheapplicability of
thismodel issomewhat limited togeneral-purpose products. Furthermore,
the prototype is used as a means of capturing requirements and is not
necessarilymeant tobeusedafterwards. Oftentimes, theprototype (orparts
of the prototype) makes its way to becoming the product itself. This can
have undesirable effects as the prototype usually employs several short
cuts, unstructured methods, and toolstoachieveaquick turnaround. Such
short cutsarepotential sourcesof defectsinliveenvironments andthus can
placeaheavy burden onmaintenance andtesting.
The Rapid Application Development model is a variation of the
Prototyping Model. LikethePrototyping Model, theRADModel relies on
feedbackandinteractionbythecustomers togather theinitial requirements.
36 Software Testing
However, the Prototyping model differs from the RAD Madelon
twocounts.
First, intheRADModel, it isnot aprototype that isbuilt but theactual
product itself.Thatis,thebuilt application(prototype, inthepreviousmodel)
isnot discarded. Hence, itisnamed RapidApplicationDevelopment model.
Second, inorder toensure formalismincapturing therequirements and
proper reflectionoftherequirements inthedesignandsubsequent phases, a
Computer Aided SoftwareEngineering (CASE)tool isused throughout the
lifecycle,right fromrequirements gathering. SuchCASEtoolshave
ffi methodologies toelicitrequirements;
ffi repositories tostorethegathered requirements andall downstream
entities suchasdesign objects; and
ffi mechanisms to automatically translate the requirements stored
in the repositories to design and generate the codein the chosen
programming environment.
Themethodologies provided byaCASEtool canprovideinbuilt meansof
verificationandvalidation. Forexample, thetool maybeabletoautomatically
detect and resolveinconsistencies indatatypes or dependencies. Sincethe
design(and, perhaps, eventheprogramcode)canbeautomatically generated
fromtherequirements, thevalidation canbeverycomplete, extending toall
thedownstream phases, unlikethePrototyping model.
This method can have wider applicability for even general-purpose
products. Theautomatic generation of thedesign and programs produced
by aCASEtool makes this model more attractive. Thecost of such CASE
toolsisafactor that anorganization wouldhavetoconsider beforedeciding
ontheuseof thismodel for agivenproject. In addition, CASEtoolsandthis
model isgenerallymoresuitedfor applications projectsrather thansystems
typeprojects.
2.5.3 Spiral or Iterative Model
TheSpiral or Iterative model follows aprocess inwhich therequirements
gathering, design, coding, and testing are performed iteratively till all
requirements aremet. Thereisalsoagood amount of overlap among the
activities of requirements gathering, design, coding, andtesting following
this model. What phase the product is in is difficult to conclude as each
requirement canbeat adifferent phase. Theonly conclusion that canbe
madeisatwhat phaseeach of therequirements isin. If adefect isproduced
inanyphaseof agivenrequirement, itmaycausethat requirement torevisit
an earlier phase. This model enables incremental development whereby
the product evolves, with requirements getting added to it dynamically.
Thisenables theproduct tobedemonstrated, atanypoint of time, withthe
functionality availableatthat point oftime. Italsoenablesthe"increments"
tobesent tothecustomer for approval. Theprogress of theproduct canbe
Table 2.2 Some product requirements and phases.
Figure 2.3
Spiral model.
seenfromthebeginning of theproject asthemodel delivers "increments"
at regular intervals. Eventhough it will bevery difficult toplan arelease
date following this model, it allows the progress to be tracked and the
customer approvals tobeobtained at regular intervals, thereby reducing
therisk of finding major defects at alater point of time. Table2.2givesan
example of phases for someof therequirements intheproduct.
Figure2.3(thecolouredfigureisavailableonpage457)depictstheSpiral
model andthephasesinvolvedinthemodel, fortheexampleonTable2.2.As
canbeseen, eachrequirement is"spiraling outwards" through thedifferent
phases astheentireproject evolves.
2.5.4 The V Model
TheWaterfall Model viewed testing asapost-development (that is, post-
coding) activity. The Spiral Model took this one step further and tried
to break up the product into increments each of which can be tested
38
separately. TheV Model starts off being similar to theWaterfall Model
inthat it envisages product development tobemade up of anumber of
phases or levels. However, thenewperspective that theV Model brings
inisthat different types of testing apply at different levels. Thus, froma
testingperspective, thetypeof teststhat needtobedoneateachlevel vary
significantly.
Forinstance,consideratypical product development activityrepresented
as a Waterfall Model earlier in Figure 2.2. The systemstarts with the
overall business requirements fromthepoint of viewof customers. These
requirements cover hardware, software, and operational requirements.
Sinceour focus is on the software, moving fromoverall requirements
to software requirements becomes the next step. In order to realizethe
softwarerequirements, theproposed softwaresystemisenvisaged asaset
of subsystems that work together. Thishigh-level design (of breaking the
systeminto subsystems with identified interfaces) then gets translated to
amoredetailed or low-level design. Thisdetailed design goesintoissues
like data structures, algorithm choices, table layouts, processing logic,
exceptionconditions, andsoon. Itresultsintheidentificationof anumber
of components, each component realized by program code written in
appropriate programming languages.
Giventheselevels, what kind of tests apply ineachof theselevels?To
beginwith, foroverall businessrequirements, eventuallywhatever software
isdevelopedshouldfitintoandwork inthisoverall contextandshouldbe
acceptedbytheendusers, intheir environment. Thistesting, thefinal proof
of thepudding, isacceptance testing. But,beforetheproduct isdeployed in
thecustomer's environment, theproduct vendor shouldtest it asanentire
unit to makesurethat all the softwarerequirements aresatisfiedby the
product that isdeveloped. Thistestingof theentiresoftwaresystemcanbe
calledsystem testing. Sincehigh-level designviewsthesystemasbeingmade
up of interoperating and integrated (software) subsystems, theindividual
subsystems should beintegrated and tested together beforeafull blown
systemtest canbedone. Thistesting of high-level design corresponds to
integration testing. Thecomponents that arethe outputs of the low-level
designhavetobetestedindependently beforebeingintegrated. Thus, the
testing corresponding to the low-level design phase is component testing.
Finally,sincecodingproduces several programunits, eachof thesesmaller
programunits havetobetested independently beforetrying to combine
themtogether toformcomponents. Thistestingof theprogramunitsforms
. unit testing.
Figure2.4depictsthedifferent typesof testingthat apply toeachof the
steps. For simplicity,wehavenot showntheplanning phaseasaseparate
entity sinceit iscommonfor all testing phases. But, it isnot possibleto
execute anyof thesetestsuntil theproduct isactuallybuilt. Inother words,
thestepcalled"testing" isnowbrokendownintodifferent sub-stepscalled
acceptancetesting, systemtesting, andsoonasshowninFigure2.4.So,itis
still thecasethat all thetestingexecution relatedactivitiesaredoneonlyat
theendof thelifecycle.
Figure 2.4
Phases of testing for
different development
phases.
,/
,/
,/
,/
,/
,/
,/
,/
,/
,/
,/
,/
Even though the execution of the tests cannot be done till the product is
built, the design of tests can be carried out much earlier. In fact, if we look at
theaspect of skill sets required for designing each type of tests, thepeople best
suited to design each of these tests arethose who areactually performing the
function of creating the corresponding artifact. For example, the best people
to articulate what the acceptance tests should beare the ones who formulate
the overall business requirements (and, of course, the customers, where
possible). Similarly, the people best equipped to design the integration tests
arethose who know how the system isbroken into subsystems and what the
interfaces between the subsystems are-that is, those who perform thehigh-
level design. Again, the people doing development know the innards of the
program code and thus arebest equipped to design the unit tests.
Not only are the skill sets required for designing these different types of
tests different, but also, there is no reason to defer the designing of the tests
till the very end. As and when each activity on the left-hand side of the "V"
is being carried out, the design of the corresponding type of tests can be
carried out. By performing an early design of the tests and deferring only
the test execution till the end, we achieve three important gains.
~ First, we achieve more parallelism and reduce the end-of-cycle time
taken for testing.
~ Second, by designing tests for each activity upfront, we arebuilding
in better upfront validation, thus again reducing last-minute
surprises.
~ Third, tests are designed by people with appropriate skill sets.
40 softwar( Testing
.................. " 'M_"M.~~WJ ._
'J ; ~ , * , i; 0\ , @ ? % i i c W > " ' . . '/ - ) :::7" ;-
; ; ~ : : : @ ~ 2 : : : ; : ~ t' (, ' : " " " : ; , , ' - ,.','.+',{ /: ,'H'
Figure 2. 5
V Model.
Acceptance
test design
Unit test design
Thi si sthebasi sfortheVModel, whi c hpresents exc ellentadvantages for
veri fi c ati onandvali dati on. Asshowni nFi gure2. 5,for eac htypeof test, we
movethedesi gnof testsupstream, alongwi ththeac tual ac ti vi ti esandretai n
thetest exec uti ondownstream, after theproduc t i sbui lt.
2.5.5 Modified V Model
TheV Model spli t the desi gn and exec uti on porti on of thevari ous types
of tests and attac hed thetest desi gn porti on tothe c orrespondi ng earli er
phases of thesoftwareli fec yc le.
An assumpti on made there was that even though the ac ti vi ty of test
exec uti onwas spli t i ntoexec uti onof tests of di fferent types, theexec uti on
c annot happen unti l the enti re produc t i s bui lt. For agi ven produc t, the
di fferent uni ts and c omponents c anbei ndi fferent stages of evoluti on. For
example, oneuni t c ouldbesti ll under development andthusbei ntheuni t-
testi ng phase whereas another uni t c ouldbeready for c omponent testi ng
whi lethec omponent i tself may not beready for i ntegrati on testi ng. There
maybec omponents that areready (thati s,c omponent tested) fori ntegrati on
and bei ng subjec ted toi ntegrati on tests (alongwi th other modules whi c h
arealsoready for i ntegrati on, provi ded thosemodules c anbei ntegrated) .
TheVModel doesnot expli c i tlyaddress thi snatural paralleli smc ommonly
found i nproduc t development.
Inthemodi fi ed V Model, thi s paralleli smi sexploi ted. W hen eac huni t
or c omponent or module i s gi ven expli c i t exi t c ri teri a to pass on to the
subsequent stage, theuni ts or c omponents or modules that sati sfy agi ven
phase of testi ng movetothenext phase of testi ng where possi ble, wi thout
Figure 2.6
Modified V Model.
necessarily waiting for all theunits or components or modules tomovein
unison fromonephaseof testingtoanother, asshowninFigure2.6.
Just as the V Model introduced various types of testing, the modified
V model introduces various phases of testing. Aphase of testinghas aone-
to-onemapping tothetypes of testing, that is, thereisaunit-testing phase,
component-testing phase, and soon. Onceaunit has completed theunit-
testing phase, it becomes part of acomponent and enters thecomponent-
testing phase. It then moves tointegration-testing phase and soon. Rather
than view the product as going through different types of tests (as the
V model does), the modified V Model views each part of the product to
gothrough different phases of testing. Theseareactually two sides of the
samecoinandthus provide complimentary views. Themainadvantage the
modified V model brings tothetableistherecognition of theparallelism
present indifferent parts of theproduct andassigningeachpart tothemost
appropriate phase of testing that ispossible. InFigure2.6, thecolumns of
thetablerepresents onesideofV,androws(whicharetestphases) represent
theother sideof V.
InFigure2.6,noticethat different phases of testingaredoneinparallel.
While starting a phase of testing it is important to look at whether the
product isready for testing. It isdetermined by aset of entry criteria. The
earliest possiblequality tostart thenext phaseof testingisdenotedbyentry
criteria, andtostart thenext phaseof testingtheearlier phaseneednot have
completed. Thetesting phases arealsoassociatedwith aset of exit criteria
to completethetest activities for eachphase. They aredetermined by exit
criteria. Theentry and exit criteriafor eachof thephases ensure that right
quality of product deliveredfor starting thetest andright amount of testing
iscompleted for therelease. Eventhough it isindicated inthepictureall of
thetest phases finishat thesametime, practically it canhavedifferent time
lines. Thelongest phasedetermines thereleasedate.
InFigure2.6,therearetwoadditional activitiesthathavenotbeendiscussed
before. Thecolouredfigureisavailableonpage458.Theseare"Component
(1,2...) Complete" and "Components Complete"; thesearenot additional
Design Unit test Components Components
IT complete
system
complete complete (1,2...) complete
complete
complete
Unit testing
,
l~ l
Component
testing
Entry criteria ______________
Integration
testing
~
System testing
~
Acceptance
Exit criteria
testing
42 SQftwateTesting
phases in alifecycle. They have been introduced just to denote that
integration testingcanstart after twocomponentshavebeencompleted,
and when all components areintegrated and tested, thenext phase of
testing, that is, systemtesting canstart.
Table 2.3 Model applicability and relevance to verification and validation.
2.5.6 Comparison of Various Life Cycle Models
As can be seen fromthe above discussion, each of the models has its
advantages anddisadvantages. Eachof themhasapplicability inaspecific
scenario. Each of them also provides different issues, challenges, and
opportunities for verification and validation. Wesummarize inTable2.3
the salient points about applicability and relevance to verification and
validation for eachof themodels.
TheWaterfall Model wasinitiallycoveredin[ROYC-70].Theorigins of the
Prototyping Model comefrom[BROO-75].TheSpiral Model wasoriginally
proposed in[BOEH-88].[GRAO-97]provides somevariations totheSpiral
Model. [RAME-2002],[PRES-97]and [HUMP-86]provide overviews toall
themodels.
1. WhichSOLCmodel wouldbemostsuitableforeachof thefollowing
scenarios?
a. Theproduct isabespokeproduct for aspecificcustomer, who
isalways availabletogivefeedback.
b. Thesameasabove, except that wealsohaveaccesstoaCASE
tool that generates programcodeautomatically.
c. A general purpose product, but with avery strong product
marketing teamwho understand and articulate the overall
customer requirements very well.
d. A product that ismade of anumber of features that become
availablesequentially andincrementally.
2. Which of the following products would you say is of "high
quality," based on the criteria we discussed in the book? Justify
your answer.
a. Threesuccessiveversions of theproduct had respectively 0,
79,and21defectsreported.
b. Threesuccessiveversions of theproduct had respectively 85,
90,and79defectsreported.
3. Listthreeor morechallenges fromthetesting perspective for each
of thefollowingmodels:
a. Spiral Model.
b. VModel.
c. ModifiedVModel.
I
4. What are some of the challenges that you should expect when
movingfromtheVModel totheModifiedVModel?
5. Figure2.1gavetheETVXDiagramforthedesignphaseofasoftware
project. Drawasimilar ETVXDiagramfor thecodingphase.
6. In thebook wehave discussed the Spiral Model asbeing ideally
suited for aproduct that evolves inincrements. Discusshow the
VModel isapplicablefor suchanincrementally evolvingproduct.
T v p e s o f T e
a Chap te r 6
System and Acceptance Testing
a Chap te r 7
Performance Testing
a Chap te r 8
Regression Testing
a Chap te r 9
Internationalization (1
18
n) Testing
a Chap te r 10
Ad hoc Testing
Thispart of thebook discusses various types of tests. Thechapters
progress fromthetypesoftestsclosertocodetothoseclosertousers.
Whiteboxtesting, whichtests theprograms byhaving aninternal
knowledge of programcode, isdiscussed inChapter 3. Blackbox
testing, which tests the product behavior by only knowing the
external behavior asdictatedbytherequirements specifications, is
discussed inChapter 4. Assoftware gets developed inamodular
fashionandthemodules havetobeintegrated together, integration
testing is covered in Chapter 5. Systemand acceptance testing,
which tests a product completely from a user's perspective in
environments similar to customer deployments, is discussed
in Chapter 6. Performance testing, which tests the ability of the
systemtowithstand typical andexcessivework loads, isdiscussed
inChapter 7.Sincesoftwareisalwayscharacterized bychangeand
sincechangesshouldnotbreak what isworkingalready, regression
testing becomes very important and isdiscussed inChapter 8.As
softwarehastobedeployedinmultiplelanguages acrosstheworld,
internationalization testing, thetopicof Chapter 9,comesintoplay.
Finally, adhoc testing, in Chapter 10, addresses the methods of
testingaproduct intypical unpredictable waysthat endusers may
subject theproduct to.
Inthis chapter-
./ What iswhite box testing?
./ Static testing
./ Structural testing
./ Challenges in white box testing
3.1 WHAT IS WHITE BOX TESTING?
Everysoftwareproduct isrealizedbymeansof aprogramcode. Whitebox
testingisawayoftestingtheexternal functionalityofthecodebyexamining
and testingtheprogramcodethat realizestheexternal functionality. This
isalsoknown asclear box, orglass box or open box testing.
Whiteboxtestingtakesintoaccount theprogramcode, codestructure,
andinternal designflow.Incontrast, blackboxtesting, tobediscussed in
Chapter 4,doesnot lookattheprogramcodebut looksattheproduct from
anexternal perspective.
A number of defects comeabout because of incorrect translation of
requirements anddesignintoprogramcode.Someother defectsarecreated
by programming errors and programming language idiosyncrasies. The
different methods of whiteboxtesting discussed inthis chapter canhelp
reduce the delay between the injection of adefect intheprogram code
and its detection. Furthermore, sincetheprogram coderepresents what
the product actually does (rather than what theproduct isintended to
do), testing by looking at theprogram codemakes us get closer towhat
theproduct isactually doing.
AsshowninFigure3.1,whiteboxtestingisclassifiedinto "static" and
"structural" testing. Thecorresponding colouredversion of Figure 3.1is
availableon page 458. Wewill look into the details of static testing in
Section3.2andtakeupstructural testinginSection3.3.
Figure 3.1
Classification of white
box testing.
3.2 STATIC TESTING
Static testing isatypeof testingwhichrequires onlythesourcecodeof the
product, not thebinaries or executables. Static testing does not involve
Whfte Box Testing 49
executing the programs on computers but involves select people going
through thecodetofindout whether
> I < thecodeworks accordingtothefunctional requirement;
> I < thecodehasbeenwritteninaccordancewiththedesigndeveloped
earlier intheproject lifecycle;
> I < thecodefor anyfunctionality hasbeenmissedout;
> I < thecodehandles errors properly.
Statictestingcanbedonebyhumansorwiththehelpof specializedtools.
3.2.1 Static Testing by Humans
Thesemethods relyontheprinciple of humans reading theprogramcode
to detect errors rather than computers executing thecodeto finderrors.
Thisprocesshasseveral advantages.
1. Sometimes humans can find errors that computers cannot. For
example, whentherearetwovariables withsimilar names andthe
programmer useda"wrong" variablebymistakeinanexpression,
the computer will not detect the error but executethe statement
and produce incorrect results, whereas ahuman being can spot
suchanerror.
2. Bymaking multiple humans read and evaluate the program, we
canget multiple perspectives and therefore have more problems
identifiedupfront thanacomputer could.
3. A human evaluation of the code can compare it against the
specifications or design and thus ensure that it does what is
intended todo. Thismaynot alwaysbepossiblewhen acomputer
runs atest.
4. Ahuman evaluation candetect many problems at onegoand can
eventrytoidentifytherootcausesof theproblems. Moreoftenthan
not, multiple problems canget fixedbyattending tothesameroot
cause. Typically,inareactivetesting, atest uncovers oneproblem
(or, at best, afewproblems) at atime. Often, such testing only
revealsthesymptoms rather thantheroot causes. Thus, theoverall
timerequired tofixall the-problemscanbereduced substantially
byahuman evaluation.
5. By making humans test the code before execution, computer
resources canbe saved. Of course, this comes at the expense of
human resources.
6. A proactive method of testing like static testing minimizes the
delayinidentificationof theproblems. AswehaveseeninChapter
1, thesooner adefectisidentifiedandcorrected, lesser isthecostof
fixingthedefect.
50 .Software Testing
7. Fromapsychological point of view, findingdefectslater inthecycle
(forexample, after thecodeiscompiled andthesystemisbeingput
together) createsimmensepressure onprogrammers. Theyhaveto
fixdefectswith lesstimetospare. Withthiskindof pressure, there
arehigher chancesof other defectscreeping in.
Therearemultiplemethods toachievestatictestingbyhumans. Theyare(in
theincreasingorder of formalism) asfollows.
1. Desk checkingof thecode
2. Codewalkthrough
3. Codereview
4. Codeinspection
Sincestatic testing by humans is done before the code is compiled and
executed, some of these methods can be viewed as process-oriented or
defect prevention-oriented or quality assurance-oriented activities rather
thanpuretestingactivities. Especiallyasthemethods becomeincreasingly
formal (for example, Fagan Inspection), thesetraditionally fall under the
"process" domain. They find a place in formal process models such as
ISO9001,CMMI,andsoonandareseldomtreated aspart of the"testing"
domain. Nevertheless, asmentioned earlier inthisbook, wetakeaholistic
viewof "testing" asanything that furthers thequality of aproduct. These
methods havebeenincludedinthischapter becausetheyhavevisibilityinto
theprogramcode.
Wewill nowlookintoeachof thesemethods inmoredetail.
3.2.1.1 Desk checking Normally done manually by the author of
the code, desk checking is amethod to verify the portions of the code
for correctness. Suchverificationisdoneby comparing thecodewith the
designorspecificationstomakesurethat thecodedoeswhat itissupposed
to do and effectively. This is the desk checking that most programmers
dobeforecompiling and executing thecode. Whenever errors arefound,
the author applies the corrections for errors onthespot. This method of
catchingandcorrectingerrors ischaracterized by:
1. Nostructured method or formalismtoensure completeness and
2. Nomaintaining of alogor checklist.
In effect, this method relies completely on the author's thoroughness,
diligence, and skills. Thereis no process or structure that guarantees or
verifies the effectiveness of desk checking. This method is effectivefor
correcting" obvious" codingerrorsbutwill notbeeffectiveindetectingerrors
that arise due to incorrect understanding of requirements or incomplete
requirements. Thisisbecausedevelopers (or,moreprecisely, programmers
who are doing the desk checking) may not have the domain knowledge
required tounderstand therequirements fully.
White Box Testing
Themain advantage offered by this method is that the programmer
who knows the code and the programming language very well is well
equipped to read and understand his or her own code. Also, sincethis
is doneby oneindividual, there arefewer scheduling and logistics over-
heads. Furthermore, thedefectsaredetectedand correctedwith minimum
timedelay.
Someof thedisadvantages of thismethod of testingareasfollows.
1. Adeveloper isnot thebest person todetect problems inhis or her
owncode. Heor shemay betunnel visioned and haveblind spots
tocertaintypes of problems.
2. Developers generally prefer towritenewcoderather than doany
formof testing! (Wewill seemoredetails of this syndrome later in
thesectiononchallenges aswell aswhen wediscuss people issues
inChapter 13.)
3. This method is essentially person-dependent and informal and
thus may not work consistently acrossall developers.
Owingtothesedisadvantages, thenext twotypesof proactivemethods are
introduced. Thebasicprinciple of walkthroughs and formal inspections is
toinvolvemultiplepeopleinthereviewprocess.
3.2.1.2 Codewalkthrough Thismethodandformalinspection(described
inthenextsection)aregroup-orientedmethods.Walkthroughsarelessformal
than inspections. The line drawn in formalism between walkthroughs
andinspections isvery thinandvaries fromorganization toorganization.
Theadvantage that walkthrough has over desk checkingisthat it brings
multipleperspectives. Inwalkthroughs, asetof peoplelookattheprogram
codeandraisequestions fortheauthor. Theauthor explainsthelogicof the
code, and answers the questions. If theauthor isunable toanswer some
questions, heor shethen takes those questions and finds their answers.
Completeness is limited to the area where questions are raised by the
team.
3.2.1.3 Formal inspection Codeinspection-also calledFaganInspection
(namedaftertheoriginalformulator)- isamethod,normallywithahighdegree
of formalism. Thefocus of this method is to detect all faults, violations,
and other side-effects. This method increases the number of defects
detected by
1. demanding thorough preparation beforeaninspection/review;
2. enlisting multiple diverseviews;
3. assigning specificrolestothemultiple participants; and
4. goingsequentially through thecodeinastructured manner.
Aformal inspectionshould takeplaceonlywhentheauthor hasmadesure
thecodeisready for inspection by performing somebasic desk checking
andwalkthroughs. Whenthecodeisinsuchareasonablestateof readiness,
52 Software Testing
aninspection meeting isarranged. Therearefour roles ininspection. First
istheauthor of thecode. Secondisamoderator who isexpected toformally
run theinspection according totheprocess. Third aretheinspectors. These
arethepeoplewho actually provides, reviewcomments for thecode. There
aretypicallymultiple inspectors. Finally,thereisascribe, whotakesdetailed
notes during theinspection meeting and circulates themtotheinspection
teamafter themeeting.
The author or the moderator selects the review team. The chosen
members havetheskill sets touncover asmany defects aspossible. In an
introductory meeting, theinspectors get copies (Thesecanbehard copies
or soft copies) of the code to be inspected along with other supporting
documents suchasthedesigndocument, requirements document, and any
documentation of applicablestandards. Theauthor alsopresents his or her
perspective of what theprogramisintended todoalongwith any specific
issuesthat heor shemaywant theinspectionteamtoput extrafocuson. The
moderator informstheteamabout thedate, time, andvenueoftheinspection
meeting. Theinspectors get adequate time to go through the documents
and program and ascertain their compliance to the requirements, design,
andstandards.
The inspection team assembles at the agreed time for the inspection
meeting(alsocalledthedefect logging meeting). Themoderator takestheteam
sequentially through theprogram code, asking eachinspector if there are
any defectsinthat part of thecode. If any of theinspectors raises adefect,
then the inspection teamdeliberates on the defect and, when agreed that
thereisadefect,classifiesitintwodimensions-minor/major andsystemic/mis-
execution. Amis-executiondefectisonewhich, asthenamesuggests, happens
because of an error or slip on the part of the author. It is unlikely to be
repeated later, either in this work product or in other work products. An
exampleof this isusing awrong variableinastatement. Systemic defects,
ontheother hand, canrequire correctionat adifferent level. For example,
an error such as using somemachine-specific idiosyncrasies may have to
removed by changing the coding standards. Similarly, minor defects are
defectsthat may not substantially affectaprogram, whereas major defects
needimmediate attention.
Ascribeformally documents thedefectsfoundintheinspection meeting
and the author takes care of fixing these defects. In casethe defects are
severe, theteammayoptionally call forareviewmeetingtoinspect thefixes
toensurethat theyaddress theproblems. Inanycase,defectsfoundthrough
inspectionneedtobetrackedtill completionandsomeoneintheteamhasto
verifythat theproblems havebeenfixedproperly.
3.2.1.4 Combining various methods Themethods discussed above
arenot mutually exclusive. Theyneedtobeusedinajudicious combination
tobeeffectiveinachieving thegoal of finding defects early.
White Box Testini 53
Formal inspections havebeen found very effectivein catching defects
early. Some of the challenges to watch out for in conducting formal
inspections areasfollows.
1. Thesearetimeconsuming. Sincetheprocess callsfor preparation
aswell asformal meetings, thesecantaketime.
2. Thelogistics and scheduling canbecome an issue sincemultiple
people areinvolved.
3. It is not always possible to go through every line of code, with
several parameters and their combinations inmind to ensure the
correctnessof thelogic,side-effectsandappropriate error handling.
It may also not benecessary to subject the entire code to formal
inspection.
Inorder to overcomethe abovechallenges, it isnecessary to identify,
duringtheplanning stages, whichparts of thecodewill besubjecttoformal
inspections. Portionsof codecanbeclassifiedonthebasisof their criticality
or complexity as"high," "medium," and "low." Highor mediumcomplex
critical codeshould besubject toformal inspections, whilethoseclassified
as"low" canbesubjecttoeither walkthroughs or evendesk checking.
Desk checking, walkthrough, reviewand inspection arenot only used
for codebut canbeused for all other deliverables intheproject lifecycle
suchasdocuments, binaries, andmedia.
3.2.2 Static Analysis Tools
Thereviewandinspectionmechanisms describedaboveinvolvesignificant
amount of manual work. Thereareseveral staticanalysis toolsavailablein
themarket that canreduce themanual work and performanalysis of the
codetofindout errors suchasthoselistedbelow.
1. whether there are unreachable codes (usage of GOTO statements
sometimes createsthissituation; therecouldbeother reasons too)
2. variables declaredbut not used
3. mismatch indefinition andassignment of values tovariables
4. illegal or error pronetypecasting of variables
5. use of non-portable or architecture-dependent programming
constructs
6. memory allocated but not having corresponding statements for
freeingthemupmemory
7. calculation of cyclomaticcomplexity (coveredintheSection3.3)
Thesestaticanalysistoolscanalsobeconsideredasanextensionofcompilers
astheyusethesameconceptsandimplementation tolocateerrors. Agood
compiler isalsoastaticanalysistool.Forexample, mostCcompilersprovide
different "levels" of code checkingwhich will catchthe various types of
programming errorsgivenabove.
54 Software Testing
Some of the static analysis tools can also check compliance for coding
standards as prescribed by standards such as POSIX. These tools can also
check for consistency incoding guidelines (forexample, naming conventions,
allowed data types, permissible programming constructs, and soon).
While following any of themethods of human checking-desk checking,
walkthroughs, or formal inspections-it is useful to have a code review
checklist. Given below is checklist that covers some of the common issues.
Every organization should develop its own code review checklist. The
checklist should bekept current with new learning asthey come about.
Inamulti-product organization, thechecklist may beat two levels-first,
anorganization-wide checklist that will include issues suchasorganizational
coding standards, documentation standards, and soon; second, aproduct-
or project-specific checklist that addresses issues specific to the product
or project.
CODE REVIEW CHECKLIST
56 Software Testing
3.3 STRUCTURALTESTING
Structural testing takes into account the code, code structure, internal
design, and how they are coded. The fundamental difference between
structural testing and static testing is that in structural testing tests are
actuallyrunbythecomputer onthebuilt product, whereas instatictesting,
the product is tested by humans using just the source code and not the
executables or binaries.
Structural testingentails running theactual product against somepre-
designedtest casestoexerciseasmuchof thecodeaspossibleor necessary.
Agivenportion of thecodeisexercisedif atest casecausestheprogramto
executethat portionof thecodewhenrunning thetest.
Asdiscussed at thebeginning of this chapter, structural testing canbe
further classifiedintounit/codefunctional testing, codecoverage, andcode
complexitytesting.
3.3.1 Unit/Code Functional Testing
Thisinitial part of structural testingcorresponds tosomequickchecksthat
adeveloper performs before subjecting the codetomore extensive code
coveragetesting or codecomplexity testing. This canhappen by several
methods.
1. Initially, thedeveloper canperformcertainobvious tests, knowing
the input variables and the corresponding expected output
variables. This can be a quick test that checks out any obvious
mistakes. By repeating these tests for multiple values of input
variables, theconfidencelevel ofthedeveloper togotothenextlevel
_1~, .. ~~-h. k
:!jfr:1W!t"f\W J4,o$ <i, *
0/.~,. ~< .- ,: ,: ,: ,: ,: ,. ,.' ,..... .' . " ;~.-';.;h~
White Box Testing 57
increases. Thiscanevenbedoneprior toformal reviews of static
testingsothat thereviewmechanismdoesnot wastetimecatching
obvious errors.
2. For modules withcomplex logicor conditions, thedeveloper can
builda/I debugversion" oftheproduct byputtingintermediate print
statements and making sure the program ispassing throughthe
right loopsanditerations theright number of times.Itisimportant
to remove the intermediate print statements after the defects
arefixed.
3. Another approachtodotheinitial test istoruntheproduct under a
debugger or anIntegrated Development Environment (IDE).These
toolsallowsinglestepping of instructions (allowingthedeveloper
tostopat theendof eachinstruction, viewor modify thecontents
of variables, and soon), setting break points at any function or
instruction, andviewingthevarioussystemparameters orprogram
variablevalues.
All theabovefall moreunder the"debugging" category of activitiesthan
under the"testing" categoryof activities.All thesame,theseareintimately
related tothe knowledge of codestructure and hencewehave included
theseunder the"whiteboxtesting" head. Thisisconsistent withour view
that testingencompasses whatever it takestodetect and correct defectsin
aproduct.
3.3.2 CodeCoverage Testing
Sinceaproduct isrealized interms of program code, if wecanrun test
casestoexercisethedifferent partsof thecode,thenthat part oftheproduct
realizedbythecodegetstested. Codecoveragetestinginvolvesdesigning
and executing test cases and finding out the percentage of codethat is
coveredby testing. Thepercentage of codecoveredby atest isfound by
adopting atechnique calledinstrumentation of code.Therearespecialized
tools available toachieveinstrumentation. Instrumentation rebuilds the
product, linking theproduct withaset of libraries provided by thetool
vendors. Thisinstrumented codecanmonitor and keep anaudit of what
portions of codearecovered.Thetoolsalsoallowreporting ontheportions
of thecodethat arecovered frequently, sothat thecritical or most-often
portions of codecanbeidentified.
Codecoveragetestingismadeupof thefollowingtypesof coverage.
1. Statement coverage
2. Pathcoverage
3. Condition coverage
4. Functioncoverage
..~i ~::;;):I ~;I i :~
3.3.2.1 Statement coverage Program constructs i n most conven-
ti onal programmi ng languages canbeclassi fi edas
1. Sequenti al control flow
2. Two-waydeci si onstatements li kei f then else
3. Multi -way deci si onstatements li keSwitch
4. Loopsli kewhi le do,repeat unti l and for
Object-ori ented languages have all of the above and, i n addi ti on, a
number of other constructsandconcepts. Wewi ll takeupi ssuespertai ni ng
to object ori ented languages together i n Chapter 11. Wewi ll confi neour
di scussi onsheretoconventi onal languages.
Statement coveragereferstowri ti ng test casesthat executeeachof the
programstatements. Onecanstart wi ththeassumpti on that morethecode
covered,thebetter i sthetesti ngof thefuncti onali ty,asthecodereali zesthe
functi onali ty. Basedonthi sassumpti on, codecoveragecanbeachi evedby
provi di ng coveragetoeachof theabovetypesof statements.
For asecti onof codethat consi sts of statements that are sequenti ally
executed(that i s,wi thnocondi ti onal branches), test casescanbedesi gned
torun throughfromtoptobottom. Atest casethat starts at thetopwould
generally havetogothroughthefull secti onti ll thebottomof thesecti on.
However, thi s may not always be true. Fi rst, i f there are asynchronous
excepti ons that thecodeencounters (for example, adi vi deby zero), then,
eveni f westart atestcaseatthebegi nni ngof asecti on,thetestcasemaynot
cover all thestatements i nthat secti on. Thus,eveni nthecaseof sequenti al
statements, coverage for all statements may not be achi eved. Second, a
secti on of code may be entered frommulti ple poi nts. Even though thi s
poi ntstonot followi ngstructured programmi ng gui deli nes, i ti sacommon
scenari oi nsomeof theearli er programmi ng languages.
Whenweconsi der atwo-way deci si onconstruct li kethe i f statement,
thentocover all thestatements, weshould alsocover thethen and else
parts of the i f statement. Thi smeans weshould have, for eachi f then
else, (atleast) onetest casetotest theThenpart and(atleast) onetest case
totest thee1separt.
Themulti -way deci si onconstruct suchas a Swi tchstatement canbe
reduced to multi ple two-way i f statements. Thus, to cover all possi ble
swi tchcases,therewouldbemulti pletestcases. (Weleavei tasanexerci se
for thereader todevelopthi sfurther.)
Loop constructs present more vari ati ons to take care of. A loop-i n
vari ous forms suchas for, whi le, repeat, and soon-i s characteri zed
byexecuti ngasetof statements repeatedly unti l orwhi lecertai ncondi ti ons
aremet. Agoodpercentageof thedefectsi nprograms comeabout because
of loops that donot functi on properly. Moreoften, loops fai l i nwhat are
called"boundary condi ti ons." Oneof thecommonloopi ngerrorsi sthat the
termi nati on condi ti on of theloopi snot properly stated. I norder tomake
,'~ite Box '{estin$
sure that there is better statement coveragefor statements within aloop,
thereshould betest casesthat
1. Skipthe loop completely, so that the situation of the termination
condition beingtruebeforestarting theloopistested.
2. Exercisetheloopbetween onceandthemaximumnumber of times,
tocheckall possible "normal" operations of theloop.
3. Try covering the loop, around the "boundary" of n-that is, just
belown, n, andjust aboven.
The statement coverage for aprogram, which is an indication of the
percentageof statements actuallyexecutedinasetof tests, canbecalculated
bytheformulagivenalongsideinthemargin.
It is clear from the above discussion that as the type of statement
progresses fromasimple sequential statement to if then else and
through to loops, the number of test cases required to achieve statement
coverageincreases.TakingacuefromtheDijkstra'sDoctrinein Chapter I, just
asexhaustivetestingof all possibleinput dataonaprogramisnot possible,so
alsoexhaustivecoverageof all statementsinaprogramwill alsobeimpossible
forall practical purposes.
Evenifweweretoachieveaveryhighlevel of statement coverage, itdoes
not meanthat theprogramisdefect-free. First, consider ahypothetical case
when weachieved 100percent codecoverage. If theprogram implements
wrong requirements and this wrongly implemented codeis"fully tested,"
with 100percent codecoverage, it still isawrong program and hencethe
100percent codecoveragedoesnot meananything.
Next, consider thefollowingprogram.
Total = O i /* set total to zero */
if (code == " M" ) {
stmtli
stmt2i
Stmt3i
stmt4i
Stmt5i
stmt6i
Stmt7i
else percent = value/Total*ldO
i
/* divide by zero */
In the above program, when we test with code ="M," we will get
80percent codecoverage. But if the data distribution in thereal world is
such that 90percent of thetime, thevalue of codeisnot ="M," then, the
program will fail 90percent of thetime (becauseof the divide by zero in
thehighlighted line). Thus, evenwithacodecoverageof 80percent, weare
left with adefect that hits theusers 90percent of thetime. Pathcoverage,
discussed inSection3.3.2.2,overcomesthisproblem.
60 Software Testing
3.3.2.2 Path coverage Inpathcoverage,wesplitaprogramintoanumber
of distinct paths. A program (or apart of aprogram) canstart fromthebe-
ginning and takeany of thepaths toitscompletion.
Let us takeanexample of adatevalidation routine. Thedateisaccepted
asthreefieldsrom, ddand yyyy. Wehaveassumed that prior toentering this
routine, thevalues arecheckedtobenumeric. Tosimplify thediscussion, we
haveassumed theexistenceof afunction calledleapyear whichwill return
TRUEif thegivenyear isaleap year. Thereisanarray called DayofMonth
which contains thenumber of days ineachmonth. A simplified flowchart
for this isgiveninFigure3.2below.
Ascanbeseenfromthefigure, therearedifferent paths that canbetaken
through the program. Eachpart of thepath isshown inred. Thecoloured
representation of Figure3.2isavailableonpage459.Someof thepaths are
> I<
A
> I< B-D-G
> I< B-D-H
> I< B-C-E-G
> I<
B-C-E-H
> I<
B-C-F-G
> I<
B-C-F-H
Regardless of the number of statements in each of these paths, if we
can execute these paths, then wewould have covered most of the typical
scenarios.
False
'1J
'J I False
(j
True
511 True
Figure 3.2
Flowchartforadate
validationroutine.

Part 1

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Part 1

Uploaded by

Copyright:

Available Formats

S o f t w a r e T e s t i n g

Defect prevention focus

You might also like