A COMPARATIVE STUDY OF USER

ACCEPTANCE TESTING BETWEEN MODIFIED

WATERFALL MODEL AND EXTREME
PROGRAMMING IN SMALL-SCALE PROJECT

IDA ARYANIE BAHRUDIN

UNIVERSITI TUN HUSSEIN ONN MALAYSIA

 UNIVERSITI TUN HUSSEIN ONN MALAYSIA
STATUS CONFIRMATION FOR MASTERS DISSERTATION

A COMPARATIVE STUDY OF USER ACCEPTANCE TESTING BETWEEN

MODIFIED WATERFALL MODEL AND EXTREME PROGRAMMING IN
SMALL-SCALE PROJECT

ACADEMIC SESSION: 2015/2016

I, IDA ARYANIE BINTI BAHRUDIN, agree to allow this Masters Dissertation to be kept at the
library under the following terms:

1. This Masters Dissertation is the property of the Universiti Tun Hussein Onn Malaysia.
2. The library has the right to make copies for educational purposes only.
3. The library is allowed to make copies of this report for educational exchange between higher
educational institutions.
4. ** Please Mark ()

(Contains information of high security or great

CONFIDENTIAL importance to Malaysia STIPULATED under the
OFFICIAL SECRET ACT 1972)

RESTRICTED (Contains restricted information as determined by the

Organization/institution where research was
conducted)

FREE ACCESS

Approved by,

(WRITERS SIGNATURE) (SUPERVISORS SIGNATURE)

Permanent Address:

NO 1C, JALAN LAMA, DR. NORAINI BINTI IBRAHIM

83700 YONG PENG, JOHOR

Date: ____________________ Date: ____________________

NOTE:
** If this Masters Dissertation classified as CONFIDENTIAL or
RESTRICTED, please attach the letter from the relevant
authority/organization stating reasons and duration for such classifications.
 th
This dissertation has been examined on date 12 June 2016 and is sufficient in
.........................
fulfilling the scope and the quality for the purpose of awarding the Degree of Master
of Computer Science (Software Engineering).

Chairperson:

PROF. DR. ISMAIL BIN ABDUL RAHMAN

Faculty of Civil and Environmental Engineering
DR. NOOR AZAH BINTI SAMSUDIN
Faculty of Computer Science and Information Technology
Universiti Tun Hussein Onn Malaysia

Examiners:

DR. HJ. MOHD ZAINURI BIN SARINGAT

Faculty of Computer Science and Information Technology
Universiti Tun Hussein Onn Malaysia

MDM. RUHAYA BINTI AB. AZIZ

Faculty of Computer Science and Information Technology
Universiti Tun Hussein Onn Malaysia
 ii

A COMPARATIVE STUDY OF USER ACCEPTANCE TESTING BETWEEN

MODIFIED WATERFALL MODEL AND EXTREME PROGRAMMING IN
SMALL-SCALE PROJECT

IDA ARYANIE BAHRUDIN

A dissertation submitted in partial

fulfillment of the requirement for the award of the
Degree of Master of Computer Science (Software Engineering)

Faculty of Computer Science and Information Technology

Universiti Tun Hussein Onn Malaysia

JULY 2016
 iii

STUDENT DECLARATION

I hereby declare that the work in this dissertation is my own except for quotations
and summaries which have been duly acknowledged

Student :
IDA ARYANIE BAHRUDIN
Date :

Supervisor :
DR. NORAINI IBRAHIM
 iv

To Ezree, for the understanding, endless love and care

 v

ACKNOWLEDGEMENT

In the Name of Allah, the Most Gracious, the Most Merciful, praise to Allah, the
Almighty, for granting me the opportunity to complete this study successfully. I
believe that I would not have been able to complete this journey without the help and
support of countless people over the past years. Hereby, I would like to express my
sincere gratitude to my supervisor, Dr Noraini Ibrahim, Faculty of Computer Science
and Information Technology, Universiti Tun Hussein Onn Malaysia for their excellent
guidance, enthusiasm, patience, and providing me with an excellent atmosphere for
doing this research. I have learned tremendously from her and her guidance on
practising excellent skills of writing articles, analysing and interpreting test results, as
well as polishing my presentation skills. I would also like to appreciate the member of
my research team, Mrs. Rafizah Mohd Hanifa for her helpful assistance and advice.
Thanks are extended to Mr. Muhammad Firdaus Kamarudin, Mr. Muhammad Aqil
Abdul Razak and Miss Nor Syafeeqa Abdullah Syukor for their assistance throughout
the research work. Most importantly, thank you to my dear husband, Mohd Ezree
Abdullah, for showing me the beauty and opposite side of life and calming me during
the hard times of the research work. Special thanks to all my beloved children, Durrah
Hannani, Farhah Amni, Nusaybah Auni and Ukasyah, for their patience and
understanding. Finally, I sincere wish to my beloved mother and father for their
endless love and prayers.
 vi

ABSTRACT

The successfulness of software development relies on many factors but the selection
of software development methodology plays a crucial role in order to complete the
project hence to get high quality software. Since the advent of traditional waterfall
model, software development methodology is still evolving. There are two categories
of software development methodologies that had been implemented recently
considered as traditional and Agile methodology. At present, many researchers had
conducted comparative studies on different agile software development
methodologies. However, literature review had shown few studies investigating the
comparison of software development methodologies especially in software testing
between traditional methods and Agile modeling. Therefore, this research focused in
details on the comparison of user acceptance testing (UAT) in Modified Waterfall
model with eXtreme Programming (XP) methodology. The objectives of the research
are to perform UAT to the system modules in two case studies, to identify project size
and defects in two case studies and to carry out comparative analysis of the error
discovery rate from UAT result of Modified Waterfall and XP for selected case studies.
A Mann-Whitney U test was performed on XP and Modified Waterfall to compare the
difference between error discovery rate means. The analysis shows that there is no
significant difference of error discovery rate between both methodologies in the
selected case studies. The main contribution of this work is to bring empirical evidence
from real life cases that contribute understanding the adoption of both methodologies
in small scale project. This study concluded that one of the factor of choosing the
software methodology that best fit to the organization is depending on the individual
project and circumstances.
 vii

ABSTRAK

Kejayaan pembangunan perisian bergantung kepada pelbagai faktor. Namun begitu,

pemilihan metodologi pembangunan perisian memainkan peranan yang sangat penting
untuk menyelesaikan projek seterusnya dapat menghasilkan perisian yang berkualiti.
Semenjak kemunculan model tradisional Air Terjun, metodologi pembangunan
perisian sentiasa berkembang. Terdapat dua kategori metodologi pembangunan
perisian yang telah dilaksanakan hingga kini iaitu metodologi tradisional dan Agil.
Pada masa kini, kebanyakan penyelidik melaksanakan kajian perbandingan terhadap
metodologi pembangunan Agil yang berlainan. Walaubagaimanapun, kajian literatur
menunjukkan bahawa masih kurang kajian berkaitan perbandingan metodologi
pembangunan sistem antara kaedah tradisional dengan permodelan Agil khususnya
dalam pengujian perisian. Oleh itu, kajian ini memfokus secara terperinci
perbandingan ujian penerimaan pengguna (UAT) antara model Air Terjun
Terubahsuai dengan Pengaturcaraan eXtreme (XP). Objektif penyelidikan ini adalah
untuk melaksanakan UAT terhadap modul sistem dalam dua kajian kes, untuk
mengenalpasti saiz projek dan kesilapan dalam kajian kes yang dipilih serta membuat
analisis perbandingan kadar penemuan ralat daripada keputusan UAT bagi kajian kes
model pembangunan Air Terjun Terubahsuai dan XP untuk kajian kes yang dipilih.
Ujian Mann-Whitney U telah dilaksanakan terhadap model Air Terjun Terubahsuai
dan XP untuk membandingkan perbezaan purata kadar penemuan ralat. Analisis
menunjukkan tiada perbezaan yang signifikan antara purata kadar penemuan ralat bagi
kedua-dua metodologi pembangunan untuk kajian kes yang dipilih. Sumbangan utama
kajian ini adalah untuk menunjukkan bukti empirikal daripada kes sebenar yang
menyumbang kepada pemahaman berhubung penggunaan kedua-dua metodologi
dalam projek berskala kecil. Kajian ini merumuskan bahawa salah satu faktor
pemilihan metodologi pembangunan sistem yang sesuai untuk organisasi adalah
bergantung kepada sesuatu projek itu sendiri dan kekangan dalam pembangunannya.
 viii

CONTENTS

TITLE i

STUDENT DECLARATION iii

ACKNOWLEDGEMENTS v

ABSTRACT vi

ABSTRAK vii

CONTENTS viii

LIST OF TABLES xii

LIST OF FIGURES xiv

LIST OF SYMBOLS AND ABBREVIATIONS xvi

LIST OF APPENDICES xvii

LIST OF AWARDS AND PUBLICATIONS xviii

CHAPTER 1 INTRODUCTION 1

1.1 Research Background 1

1.2 Problem Statement 3

1.3 Research Objectives 4

1.4 Research Scope 4

1.5 Research Hypothesis 4

1.6 Significance of Research 5

1.7 Dissertation Outline 5

1.8 Chapter Summary 6

 ix

CHAPTER 2 LITERATURE REVIEW 7

2.1 Introduction 7

2.2 Software Development Methodology 8

2.2.1 Heavyweight Methodology 9

2.2.2 Lightweight Methodology 13

2.2.3 Comparison between Heavyweight and

Lightweight Methodology 18

2.3 Software Testing 20

2.3.1 Unit Testing 21

2.3.2 Integration Testing 22

2.3.3 System Testing 22

2.3.4 User Acceptance Testing (UAT) 22

2.4 Designing User Acceptance Testing 23

2.4.1 The Importance of UAT 24

2.4.2 Testing Team 26

2.4.3 Designing Test Plan 27

2.4.4 Test Cases Design 28

2.4.5 Test Execution 31

2.4.6 Software Defects 31

2.5 User Acceptance Testing in Modified Waterfall

and Agile 33

2.5.1 UAT in Agile 34

2.5.2 UAT in Modified Waterfall Model 39

2.6 Comparison of UAT in Modified Waterfall and

eXtreme Programming 42

2.7 Chapter Summary 45

CHAPTER 3 RESEARCH METHODOLOGY 46

3.1 Introduction 46
 x

3.2 Research Plan 47

3.2.1 Phase 1 49

3.2.2 Phase 2 51

3.2.3 Phase 3 53

3.2.4 Phase 4 55

3.2.5 Phase 5 56

3.3 Chapter Summary 57

CHAPTER 4 ANALYSIS AND DESIGN 58

4.1 Description of Modified Waterfall Model and

XP Project 58

4.2 Case Study 1 58

4.2.1 Analyzing Functional Requirements

of eDoc (Modified Waterfall model) 60

4.2.2 Test Case Design for eDoc (Modified

Waterfall Model) 66

4.2.3 Analyzing User Stories of eDoc (XP

Methodology) 69

4.2.4 Test Case Design for eDoc (XP

Methodology) 76

4.3 Case Study 2 85

4.3.1 Functional Requirements of PiTOS

(Modified Waterfall Model) 87

4.3.2 Test Case Design for PiTOS (Modified

Waterfall Model) 93

4.3.3 User Stories for PiTOS (XP

Methodology) 98

4.3.4 Test Case Design for PiTOS

(XP Methodology) 104

4.4 Chapter Summary 111

 xi

CHAPTER 5 IMPLEMENTATION AND RESULT 112

5.1 Criteria for Case Selection 112

5.1.1 Project Size 112

5.1.2 Team Size 114

5.1.3 Duration 114

5.2 Test Case Execution for Modified Waterfall

Project 115

5.3 Analysis of Requirements and Defects in

Modified Waterfall Project 119

5.4 Test Case Execution for XP Project 121

5.5 Analysis of User Stories and Defects in XP Project 124

5.6 Comparison Between Modified Waterfall and XP

Projects 126

5.6.1 Rank Table 127

5.6.2 Test Statistic Table 127

5.7 Chapter Summary 129

CHAPTER 6 CONCLUSION AND RECOMMENDATIONS 130

6.1 Objectives Achievement 130

6.2 Conclusion 130

6.3 Future Works 132

6.4 Summary 132

REFERENCES 132
APPENDICES 142
 xii

LIST OF TABLES

Table 2.1 Advantages and disadvantages of Modified Waterfall model 13

Table 2.2 Description of the Agile Methodology 15
Table 2.3 Advantages and disadvantages of XP 18
Table 2.4 Comparison of heavyweight and lightweight methodologies 19
Table 2.5 Summary of comparative study from different authors 20
Table 2.6 Traceability in user story 39
Table 2.7 Requirement traceability 42
Table 4.1 Functional requirements of eDoc 65
Table 4.2 Test case TCCS1MW100 description 66
Table 4.3 Test case TCCS1MW200 description 67
Table 4.4 Test case TCCS1MW300 description 68
Table 4.5 Test Case TCCS1MW400 description 68
Table 4.6 Test case TCCS1MW500 description 69
Table 4.7 User authentication user story and acceptance criteria 70
Table 4.8 Add new folder user story with acceptance criteria 70
Table 4.9 Insert file user story with acceptance criteria 71
Table 4.10 View report user story with acceptance criteria 72
Table 4.11 View file user story with acceptance criteria 73
Table 4.12 Create folder or subfolder user story with acceptance criteria 75
Table 4.13 User story of additional requirement for user authentication 75
Table 4.14 Test case for user authentication 77
Table 4.15 Test case for add new folder 78
Table 4.16 Test case for insert file 79
Table 4.17 Test case for view report 80
Table 4.18 Test case for view files 81
Table 4.19 Test case for Iteration 1 83
Table 4.20 Test case for Iteration 2 84
 xiii

Table 4.21 Summary of functional requirements of PiTOS 92

Table 4.22 Test case TCCS2MW100 description 93
Table 4.23 Test case TCCS2MW200 description 94
Table 4.24 Test case TCCS2MW300 description 96
Table 4.25 Test case TCCS2MW400 description 96
Table 4.26 Test case TCCS2MW500 description 97
Table 4.27 Test case TCCS2MW600 description 97
Table 4.28 User authentication user story 98
Table 4.29 User story for create industrial trainee profile 99
Table 4.30 View all results user story 100
Table 4.31 Modify personal information user story 100
Table 4.32 View result user story 101
Table 4.33 User story of submit evaluation feature 102
Table 4.34 User story and acceptance criteria for Iteration 1 103
Table 4.35 User story and acceptance criteria for Iteration 1 103
Table 4.36 Test case for user authentication 105
Table 4.37 Test case for creating industrial trainee profile 106
Table 4.38 Test case for view all results 107
Table 4.39 Test case for modify personal information 108
Table 4.40 Test case for view result 108
Table 4.41 Test case for submit evaluation 109
Table 4.42 Test case for Iteration 1 110
Table 4.43 Test case for Iteration 2 110
Table 5.1 Project size 114
Table 5.2 Result summary of Test case TCCS1MW100-TCCS1MW500 115
Table 5.3 Result summary of Test case TCCS2MW100-TCCS2MW600 117
Table 5.4 Result summary for test case TCCS1XP000 to TCCS1XP200 121
Table 5.5 Result summary for test case TCCS2XP000 to TCCS2XP200 122
Table 5.6 Project size and defects number for both Modified
Waterfall and XP 126
Table 5.7 Error discovery rate for each project 127
Table 5.8 Rank table 127
Table 5.9 Test statistic table 128
 xiv

LIST OF FIGURES

Figure 2.1 Linear Sequential Model 10

Figure 2.2 Waterfall model 11
Figure 2.3 Modified Waterfall model 12
Figure 2.4 XP flow chart 16
Figure 2.5 Iterative project with possible documentation artifacts in Agile 17
Figure 2.6 Software testing strategy 21
Figure 2.7 UAT steps 24
Figure 2.8 Test case sample Error!
Bookmark not defined.
Figure 2.9 Software defects categories 32
Figure 2.10 Types of error and defect 33
Figure 2.11 AgileUAT framework 35
Figure 2.12 Sample of user story 36
Figure 2.13 Important consideration of writing user stories 37
Figure 2.14 Sample of user story with acceptance criteria 38
Figure 2.15 Requirement gathering techniques 40
Figure 2.16 SRS content 41
Figure 2.17 Comparison of Modified Waterfall and XP methodologies 44
Figure 3.1 Research activities 48
Figure 3.2 Use case for eDoc 50
Figure 3.3 Use case diagram of PiTOS 52
Figure 3.4 Index card 54
Figure 4.1 eDoc Framework 59
Figure 4.2 Process of using eDoc 60
Figure 4.3 User authentication use case 61
Figure 4.4 Use case for add new folder 62
Figure 4.5 Insert file use case 63
 xv

Figure 4.6 View report use case 64

Figure 4.7 View file use case 65
Figure 4.8 PiTOS Workflow 86
Figure 4.9 PiTOS architecture 86
Figure 4.10 Login page use case 87
Figure 4.11 Create industrial trainee profile list use case 88
Figure 4.12 View all results use case 90
Figure 4.13 Modify personal information use case 90
Figure 4.14 View result use case 91
Figure 4.15 Submit evaluation use case 91
Figure 5.1 Size of selected project using Modified Waterfall model 113
Figure 5.2 Size of selected project using XP methodology 113
Figure 5.3 Project duration of selected Case Study 115
Figure 5.4 Proportions of functional requirement defects in eDoc 119
Figure 5.5 Proportions of functional requirement defects in PiTOS 120
Figure 5.6 Proportions of functional requirement defects in eDoc
and PiTOS 120
Figure 5.7 Proportions of functional requirement defects for each
iteration in eDoc 124
Figure 5.8 Proportions of functional requirement defects for each
iteration in PiTOS 125
Figure 5.9 Proportion of defects and user stories for eDoc and PiTOS 125
Figure 5.10 Mean number of defects for Modified Waterfall and XP 126
 xvi

LIST OF SYMBOLS AND ABBREVIATIONS

XP - eXtreme Programming
UAT - User Acceptance Testing
ID - Identification
CeDS - Centre for Diploma Studies
eDoc - Electronic Document Management System
PiTOS - CeDS Industrial Online System
SDLC - Software Development Life Cycle
IS - Information system
SDM - Software development methodology
- Constant
p - Probability Value
Ho - Null Hypothesis
 xvii

LIST OF APPENDICES

APPENDIX TITLE PAGE

A Computer Software Configuration Item (CSCI) 142

Capability Requirement for Electronic Document
Online System (eDoc)
B Computer Software Configuration Item (CSCI) 151
Capability Requirement for Centre for Diploma Studies
Industrial Training Online System (PiTOS)
C Test Cases Result Details for Modified Waterfall 161
Case Study
D Test Cases Result Details for XP Case Study 171
 xviii

LIST OF AWARDS AND PUBLICATIONS

Awards

1. 1st Prize of Technical Category for the product Electronic Document

Management System (e-Doc) at Innovative & Creative Circle Project
Presentation, UTHM 2014.

2. Silver Award Category for the product Electronic Document Management

System (e-Doc) at 10th Innovative & Creative Circle Convention, Universiti
Malaysia Sarawak, 2014.

3. Copyright for Electronic Document Management System (e-Doc).

4. Silver Medal Awarded at Research and Innovation Festival for the product
Centre for Diploma Studies Industrial Training Online System (PiTOS)
organized by Pusat Penyelidikan dan Inovasi, Universiti Tun Hussein Onn
Malaysia, 19-20 Disember 2013.

Journal Papers

1. Rafizah Mohd Hanifa, Shamsul Mohamad, Ida Aryanie Bahrudin and

Muhammad Firdaus Kamarudin Development of Pusat Pengajian Diploma
Industrial Training Online System (PiTOS): A Step Forward Applied
Mechanics and Materials Vols. 321- 324 (2013) pp 2528-2531 (Scopus
index)

2. Ida Aryanie Bahrudin, Rafizah Mohd Hanifa, Mohd Ezree Abdullah,

Muhammad Firdaus Kamarudin, Adapting eXtreme Programming
Approach in Developing Electronic Document Online System (eDoc),
Applied Mechanics and Materials Vols. 321-324 (2013) pp 2938-2941
(2013) Trans Tech Publications, Switzerland
doi:10.4028/www.scientific.net/AMM.321-324.2938 (Scopus index)
 CHAPTER 1

INTRODUCTION

The aim of this chapter is to emphasize the key components of this dissertation. It
provides a clear statement of the research topic which includes the research
background, problem statement, research objectives, scope, hypothesis and also the
significance of this research.

1.1 Research Background

The emergence of information system (IS) development history began in the year
1940. It was a shift from scientific or mathematical application to business application
where the viewpoint of software was more on general view [1]. IS development was
based on information technology (IT) pioneers individual knowledge up to the 1960s
thus the period is referred as pioneer era or heroic age [2]. A systematic software
development approach makes the first appearance when the software crisis occurred.
It was known earlier as approach to problem solving in other disciplines which was
applied to systems development at the moment. In 1970s, the structured approach
based on waterfall cycle was introduced [3]. Waterfall model is a method which the
software can be developed in a systematic manner by improving the probability of
completing the software project within the deadline while maintaining the quality of
the software product to conform certain standard [4].
Recently, there are two categories of software development methodologies
that had been implemented which considered as heavyweight and lightweight
methodologies. Heavyweight methodologies are known as the traditional methodology
of software development while lightweight methodologies can be considered as agile
 2

modeling [5]. Heavyweight methodologies are basically based on a sequential series

of steps like requirements definition, solution building, testing and deployment [6]. It
is also requires defining and documenting a stable set of requirements at the beginning
of a project. The examples of software development approach that fall in heavyweight
category are waterfall model, prototyping and spiral model while the lightweight
methodologies consist of eXtreme Programming (XP) and Scrum [5]. Agile modeling
are stressed on two concepts which are the intolerant honesty of working code and the
effectiveness of people working together with goodwill [7]. Agile methodologies are
also focusing on the ability to embrace changes which are very likely to happen in
environments that lack predictability.
Software development methodologies are considered as key tools for all
software developers and also project manager in order to facilitate them in delivering
projects on time, within budget constraints while at the same time accomplish
customers requirements. The emergence of various methodologies provides the
developer the numerous choices to develop software that is able to produce satisfactory
results. A successful software development depends sturdily on the delivery of high
quality software attributes such as customer satisfaction [8]. Accordingly, one of the
first decisions that faced by the software developers for each of their project
implementations is which development methodology should they use because every
type of methodologies has their own uniqueness. This is a topic that gets a lot of
discussion yet open into ongoing debate regarding what methodologies suit to what
types of software development [9]. Both traditional and agile methodologies are
usable and established methodologies that have been involved in software
development projects for many years ago. Therefore, the decision of which
methodology to be utilized can refine the overall process that best suit to the project
goals. Eventually, although there are slight distinction between both methodologies in
terms of the software developing process, delivering software that satisfies customer
needs is what really takes into account.
During software development, there are number of phases involved throughout
the processes which begin with requirements capture and ending with maintenance
activities. Among all the phases involved, 50% of the development time is spent on
testing [10, 11]. According to [12], software testing is a resource-hungry, time-
consuming and also labor intensive work to be done. In other words, software testing
is a process which is used to measure the quality of software developed. There are
 3

many types of software testing such as unit testing, functional testing, system testing,
integration testing, regression testing, usability testing and also user acceptance testing
(UAT) [13].
Among all the types of software testing, researchers [14, 15] found that the
UAT is a kind of software testing that can identify the successfulness of the developed
software because the risk of software failure will increase without UAT. It can be said
that UAT need to be conducted in order to test whether the developed software had
met the user requirement [16]. However, the UAT in waterfall model is slightly
different as compared to the agile method in terms of the way to perform the testing.
The discovery on the result of UAT in selected both traditional and agile can contribute
to a better consideration factors of choosing the software methodology to employ.
Therefore, explorations in detail on UAT in both methodologies can improve better
understanding and guarantee something beneficial findings in the future.

1.2 Problem Statement

At present, many researchers had conducted comparative studies on different agile

software development methodologies [8, 17, 18]. Moniruzzaman [18] stated that Agile
development approach improves software development process to meet the rapid
changing business environments while, traditional plan-driven developments fail to
meet up these requirements. According to Mahalakshmi [19], Scrum Methodology
have the great potential to meet the new requirements of the software development
companies. On the other hand, researchers [20] believed that waterfall model is best
suit for large scale software project development while XP is fit for small scale project.
Despite a number of previous studies, literature review had shown few studies
investigating the comparison of software development methodologies especially in
software testing between traditional methods and Agile modeling [18, 21]. Therefore,
this research focused in details on the comparison of user acceptance testing in
Modified Waterfall model with XP methodology.
 4

1.3 Research Objectives

The aim of this study is to conduct the comparative analysis of UAT between Modified
Waterfall model and XP in small-scale project. In order to achieve this aim, the
objectives of the research are to:
(i) perform user acceptance testing to the system modules in selected case studies;
(ii) identify project size and defects in selected case studies;
(iii) carry out comparative analysis of the error discovery rate from UAT result of
Modified Waterfall and XP for selected case studies.

1.4 Research Scope

This research aims to analyze the output of UAT for the software developed using XP
as compared to the software developed using Modified Waterfall model. This study
involved two software that developed using Modified Waterfall which later
redeveloped using XP. The UAT result of each case study had been analyzed in order
to calculate its error discovery rate. This study focused on the XP practices mainly in
pair programming and on-site customer involvement for developing the software.

1.5 Research Hypothesis

The followings are the hypothesis of this research:

(i) H1: Error discovery rate in Modified Waterfall project is higher than the XP.
H1: ModifiedWaterfall Error Discovery Rate > XP Error Discovery Rate
(ii) H0: Error Discovery Rate in Modified Waterfall projects is the same or lower
than the XP.
H0: ModifiedWaterfall Error Discovery Rate XP Error Discovery Rate

where XP Error Discovery Rate is the ratio of defects per test cases on XP
projects and ModifiedWaterfall No. of Defects is the ratio of defects per test
cases on Modified Waterfall projects.
 5

1.6 Significance of Research

XP has recently become one of the major interests among researchers, software
developer and programmers in terms of new software development approach. Many
studies have been conducted on XP and most studies suggested XP as one of the best
agile methods that focusing on customer requirement [22] by bringing customer into
the development process. Studied by Woit [23] shown that XP project is built for the
customers, in accordance with their needs that stated during requirement gathering
process. Despite the importance of the on-site customer requirement, few XP
researches stated that XP's most problematic feature is the amount of on-site customer
involvement it requires [24]. There were also many studies regarding the advantages
of traditional method as compared to agile. According to Balaji [25], requirement
definition is clearly stated before development starts and each phases is properly
documented in waterfall model.
In order to measure how compliant the software with the business
requirements, UAT were performed. Since the issues regarding the factor of selecting
the suitable software development methodology that will increase the probability of
achieving software that conform the requirement has long been debated and is still
being discussed, more research in this area is required. Therefore, comparative study
between traditional methodology and agile methodology is required because
challenges in software development are somewhat different from other organizations
due to the cost, duration and also the involvement of stakeholder. Moreover, this study
provides the comparison of UAT between Modified Waterfall model and XP
especially in small-scale project in terms of its error discovery rate.

1.7 Dissertation Outline

This dissertation is organised into three chapters and each chapter consists of several
subsections, beginning with a brief of introductory passage and ended with the
research methodology. The dissertation structures begin with introduction in Chapter
1. Chapter 1 presents the overall perspectives related to this research, reviews of the
research background and description of the objectives. Chapter 2 gives a
comprehensive review from prior researches and practical experienced among
 6

researchers and software developer while implementing both traditional and agile
software development methodology. Inside this chapter, the advantages and
disadvantages of traditional methodology especially in Modified Waterfall model and
agile modelling that mainly focus in XP were identified clearly. In addition, the
exploration regarding UAT in both methodologies were explained in detail including
the similarities, distinction and also the way of performing the test. In Chapter 3, there
were two case studies involved where UAT were performed to both case studies in
order to accomplish the objectives throughout the research. This chapter explains the
activities of each phases of research work plan including materials, supporting tools,
standards, and the software testing procedures applied in this research. Chapter 4
consists the analysis and design of all the test cases in the case studies. The
implementation and results of the UAT were described in Chapter 5 while the last
chapter concludes the research work along with the future recommendation for this
study.

1.8 Chapter Summary

This chapter consists of explanations about research background and also addresses
the problem, objectives and scope of this research as well as the importance of
performing comparative analysis of error discovery rate from UAT between XP and
Modified Waterfall model.
 CHAPTER 2

LITERATURE REVIEW

The overall goals of this chapter were to elaborate a comprehensive summary of the
research particularly on UAT in both Modified Waterfall model and XP methodology.
This chapter described the comparative study of software development approach
between Agile and traditional methodology that performed by prior researchers. This
chapter was also highlighted the framework that used by Modified Waterfall Model
and XP in performing the UAT.

2.1 Introduction

Nowadays software has been an important part of the modern society. Since the
computer technology is rapidly increased, a great attention has been given to ensure
that the software program will perform as best as the computer hardware that it runs
on. Software has gone through the continuous redesign and also upgrading process to
adapt with the technology and also specification changes. Over the years, software
expertise or even software developer had introduced the numbers of rules, best
practices and also guidelines for developing the software. Research had shown that
software development can be considered as a complicated task [26-29]. According to
Yu [6], selecting software development methodology (SDM) is the most important
element in software development as it portrays the necessary phases in software
development. It can be said that the selection of software development methodology
is very important in order to deliver the required quality for business success.
There are number of phases involved throughout the software development
processes which begin with requirements capture and ending with maintenance
7
 8

activities. Among all the phases involved in SDM, software testing is the most crucial
part. Basically, it is more than a half of the whole development process time is spent
on testing [10, 11]. There is a bit different way of performing software testing in
different type of SDM. This chapter provided a detailed explanation of software testing
particularly in UAT for Modified Waterfall model and XP.

2.2 Software Development Methodology

Basically, SDM is a process of developing or maintaining the software. It is also

considered as creation of a program by generating code along with various tools. It is
a challenge process when choosing the suitable SDM that really suits with the software
development situation because we have to ensure that the development is completed
on time and within reasonable budget. It is really important to select the methodology
wisely so that it is sufficient to deliver the quality required for business success while
avoiding the steps that can waste time. Every step taken along the software
development has its own risk and variety of available techniques for improving
development process and increase the quality of result.
The term SDM can be considered as a framework or as an approach in
developing software. A SDM is a framework purposely for structuring, planning and
controlling the software development process. Usually SDM includes the pre-
definition of specific deliverables that are created and completed by a project team to
develop or maintain an application [1]. In addition, the term SDM refers to an approach
used to apply the software development methodology framework. Every SDM
approach performs as a foundation for applying specific frameworks to develop and
maintain software. There are several software development approaches that have been
used since the origin of information technology. Broadly these are several established
SDM approaches [30]:

(i) Waterfall: a linear framework

(ii) Spiral: a combined linear-iterative framework
(iii) Incremental: a combined linear-iterative framework or V Model
(iv) Prototyping: an iterative framework
(v) Rapid application development (RAD): an iterative framework
(vi) Scrum: iterative and incremental agile SDM
 9

(vii) eXtreme programming: is a pragmatic approach to program development

(viii) Adaptive software development (ASD): is a principle for creating software
that focuses on quickly making programs
(ix) Dynamic system development method (DSDM): is a robust Agile project
management and delivery framework
Basically all the SDM are categorized into two categories that known as heavyweight
methodology that is also known as traditional methodology and lightweight
methodology or called agile methodology [31].

2.2.1 Heavyweight Methodology

Software methodologies like Waterfall model, V-Model and Rational Unified Process
(RUP) are called traditional SDM and these are classified into the heavyweight
methodologies [32]. These methodologies are based on a sequential series of steps like
requirements definition, solution building, testing and deployment. Heavyweight
methodologies require defining and documenting a stable set of requirements at the
beginning of a project [6]. There are four main phases involved in heavyweight
methodology which consist of project requirements set up and determine the time
duration that will take to implement the various phases of development while at the
same time trying to forecast any troubles that may occur in the project.
After gathering the requirements, the next step is to design the architectural
planning phase in the form of diagrams or models. The next step is the development
phase where code is produced until the specific goals are reached. Development is
often broken down into smaller tasks that are distributed among various teams based
on skill. Usually the testing phase often overlaps with the development phase to ensure
issues are addressed early on. Once the project is nearly complete, the customer will
become part of the testing process and the project will be delivered after the customer
satisfies with it [6].
The heavyweight SDM are dependent on a set of predetermined processes and
also on-going documentation which is written as the work progresses and guides
further development [32]. The successful factor of a project depending on the
understanding of the requirements at the early phase of development and it means that
any requirement changes during the development phase can be somewhat problematic.
However by unpermitted requirement to change it makes easier to determine the costs
 10

of the project set a schedule and allocate resources. The followings are the
characteristics of heavyweight SDM [20]:
(i) Heavyweight SDM is usually used to develop the traditional projects that use
procedural programming.
(ii) It also uses common processes likes: analysis, design, implementation, and
testing.
(iii) The implementation of heavyweight SDM depends on the size of projects and
type of projects
(iv) It sometimes needs the longer duration to develop the large projects.
Heavyweight SDM is considered as the linear sequential model. The linear
sequential model suggests a sequential approach to software development that starts at
the system level and progresses through analysis, design, coding, testing, and support
[33]. Figure 2.1 illustrates the linear sequential model for software engineering.

System/information
engineering

Analysis Design Code Test

Figure 2.1: Linear Sequential Model [33]

As mentioned before, there are various SDM that fall into this category but this
research only focused on Modified Waterfall model.

2.2.1.1 Modified Waterfall Model

Historically, the waterfall model is the classical model of software engineering which
is widely used by the software developer. This model emphasizes proper planning at
the beginning of the stages and ensures there is no design fault before the developing
process [34]. Furthermore waterfall model does not permit any changes of requirement
in between the phases of development. The phases in waterfall model consist of six
 11

phases which are analysis, design, development, testing, implementation and

maintenance as shown in Figure 2.2.

Analysis

Design

Development

Testing

Implementation

Maintenance

Figure 2.2: Waterfall model [25]

Nowadays, the dynamic business environment makes organizations to

constantly change their software requirements in order to adjust with new
environment. The demand for fast delivery of software products is rapidly increase as
well as accepting changing requirements [18]. This aspect contributes to the failure of
meet up user requirement in this plan-driven development. Therefore, Modified
Waterfall model was introduced by Winston W. Royce which improvised the
sequential model or waterfall model that he developed before [2]. The Modified
Waterfall uses the same phases as the pure waterfall, but is not based on a
discontinuous basis. This enables the phases to overlap when needed which is
unacceptable in pure waterfall model [34].
In the Modified Waterfall model, each phase influences and depends on the
next and the previous phase respectively as shown in Figure 2.3. Verification and
validation process have been added to each of the phases making it is more flexible.
As a result, the defects and faults in software are removed in the development stage
which will reduce the overhead cost of making changes to a software project before
implementation stage [35].
 12

System
Requirements

Software
Requirements

Analysis

Program Design

Coding

UAT is implement at Testing

this stage

Operations

Figure 2.3: Modified Waterfall model [2]

The overlap process contributes to give a lot of flexibility to this methodology

because at the same time a number of tasks can function concurrently. The other
advantage of this methodology is a more relaxed approach if compared to the
traditional waterfall model. There is also drawback found while implementing
Modified Waterfall model such as the flexibility of turning back to previous phase will
expand the allocation time of software development process. The development team
may be bored to always moving back and forth between the phases to ensure the
software error will be fixed early. Table 2.1 shows the advantages and disadvantages
of Modified Waterfall model.
 13

Table 2.1: Advantages and disadvantages of Modified Waterfall model [34]

Advantages Disadvantages
More flexible than the pure waterfall model. Milestones are more ambiguous than the pure
waterfall.
If there is personnel continuity between the Activities performed in parallel are subject to
phases, documentation can be substantially miscommunication and mistaken assumptions.
reduced.
Implementation of easy areas does not need to Unforeseen interdependencies can create
wait for the hard ones. problems.

Although heavyweight methodologies, such as Modified Waterfall model

continue to dominate the software development almost few decades and much research
has done in traditional methodologies, lightweight methodology brings its own
uniqueness in order to satisfy the customer through early and continuous delivery of
the valuable software [18].

2.2.2 Lightweight Methodology

Many years ago, when the software development is at the early stage, most of the
users requirements were quite stable where the software development mostly
followed the plans without major changes. However, as the technology growth,
software development involved more critical and dynamic industrial projects that
make new difficulties emerged accordingly [3]. The followings are the difficulties
emerged while using the previous heavyweight methodology:
(i) Frequently changing requirements: customer requirements are always
changing due to evolving business needs. Most of the customers do not have a
clear vision about the specifications of their requirements at the early stages
because they sometimes realize what their true requirements are only when
they use an application that does not really meet their needs. This usually led
to cause the major software failure which is incorrect assumptions with regard
to software requirements or it can be said as faulty user expectation [36, 37].
(ii) Customer involvement: lack of customer involvement will usually leads to
higher chances of project failure [38] because in heavyweight SDM customer
will only involve at the requirement phase .
(iii) Cost assumptions and deadlines: In any software development, customers often
do not accept failure. Apart from that, companies usually offer low budgets,
 14

tight deadlines, while at the same time, requiring high demands because of
competition in the markets [39].
(iv) Miscommunications: one cause of the misunderstanding of requirements is the
miscommunication between developers and customers. It is hard to gather
requirements from customer as they were some barriers exist between the
developer and the customer as customer came from various background.
Therefore, the software practitioners are looking forward for a new SDM to
face the changes. Lightweight methodologies hold practices that enable programmers
to build solutions more quickly and efficiently, with better responsiveness to changes
in business requirements. Basically, it is mainly focused on development based on
short life cycles, involves the customer throughout the software development and also
strive for simplicity along with value the people [5]. Lightweight methodologies are
inherently flexible enough to be tailored to each project depending upon the
characteristics of each [40]. Lightweight methodology also known as Agile
methodology.
There have been several studies and ideas on improving and enhancing the
overall traditional software development process such as to defeat the rapid change in
organization and business need [41]. Basically, Agile methodology focuses on people,
customers satisfaction and rapid response to change [31, 42]. According to Agile
Manifesto [39], the followings are the major factors of Agile:
(i) Early customer involvement;
(ii) Iterative development;
(iii) Self-organizing teams; and
(iv) Adaptation to change.
There are various methodologies that categorized under lightweight
methodology. Currently, there are six methods that are identified as Agile
methodology, which are Crystal Methodologies, Dynamic Software Development
Method, Lean Software Development, Scrum, and eXtreme Programming [5, 6, 41,
42]. Table 2.2 shows the description of the Agile Methodology.
 15

Table 2.2: Description of the Agile Methodology [42]

Agile Methodologies Description

Crystal methodologies The most agile method, Crystal Clear, focuses on communication in
small teams developing software that is not life-critical.
Dynamic Software It divides projects into three phases: pre-project, project life-cycle, and
Development Method post project
Lean Software It consists of seven principles: eliminate waste, amplify learning, and
Development decide as late as possible, deliver as fast as possible, empower the team,
build integrity, and see the whole.
Scrum Focuses on project management in situations where is difficult to plan
ahead, with an importance on feedback mechanisms.
eXtreme Programming Focuses on best practice for development which consists of twelve
(XP) practices.

All the listed methodologies have their own uniqueness that will suit on
different software development situation. This research focused in details on XP as
one of the latest lightweight SDM.

2.2.2.1 eXtreme Programming (XP)

XP is actually a deliberate and disciplined approach to Agile software development

which was found to be most successful at smaller companies. The successful of XP
especially at smaller companies basically it is focused on customer satisfaction [27].
XP allows developers to confidently respond to changing customer requirements in
any software development phase [43]. The aim of this methodology is to deliver the
software to the customer needs on time. The key element of XP is teamwork where
managers, customers and developers collaborate in a team. There are five ways that
improve a software project in XP which are communication, simplicity, feedback,
respect, and courage.
XP is based on four main values which are known as simplicity,
communication, feedback and courage while the XP process is characterized by short
development cycles, incremental planning, continuous feedback, and reliance on
communication and evolutionary design [44]. The core of XP is made up of a simple
set of common-sense practices. These practices are planning game, small releases,
metaphor, simple design, testing, refactoring, pair programming, collective ownership,
continuous integration, 40-hourweek, on-site customer, coding standards, open
workspace and just rules [43]. The XP software development process starts with
 16

planning, and all iterations consist of four basic phases in its life cycle: designing,
coding, testing, and listening.
XP uses an iterative and incremental software process that carried out in
relatively short cycles. The life cycle of XP includes planning and also four basic
iterative development phases which are designing, coding, testing, and listening. In the
planning phase user stories are written, iterative implementation planning is
performed, efforts are estimated, and priorities of the features are decided [45].
Although simplicity is one of the XPs principles, it doesnt mean that it can exclude
designing process. Improper will lead to the complexity of certain project. Therefore
it is then important to create a design structure that organizes the logic in the system
to avoid too many dependencies in the system [46]. The other process in XP is coding
which is the most important phase where programmers begin to write the code at the
very beginning. Traditionally, testing is a phase of development that is carried out after
the main coding effort. In contrast, XP makes the programmers to write the tests before
the code. Every time new code is written on the XP project, a corresponding test case
must be written and implemented first. Listening is another important process in XP
lifecycle where good listening skill of the developer will enable them to understand
clearly what customer want thus will develop solutions based on customers need as
close as possible [43]. Figure 2.4 shows the flow chart of XP.

Figure 2.4: XP flow chart [43]

Based on the figure, it can be seen that at the starting point gathering
requirements is similar to traditional projects but different in the way it is documented.
It involves establishing business requirements and justification which results in a
vision document at a high level [47]. The iteration process is one that makes progress
 17

through successive improvement. Figure 2.5 shows how the iteration occur in Agile
along with possible documentation artifacts [47].

DEPLOY

DEPLOY
Iteration

Iteration
Iteration

Iteration

Iteration

(n+2)
(n+1)
(n)
(2)
(1)
Initiation

Iteration 0
(Requirements
Analysis/
Iteration Prep)

- Project Charter
- Vision statement
- Backlog - Help
- Business Requirement - Light Uses - Maintenance Plan
Document (Light) Cases/Stories
- Functional - Non-functional
Requirements Requirements
- Non-functional - Test
Requirements - Design Models
- Wireframe

Figure 2.5: Iterative project with possible documentation artifacts in Agile

The stage after initiation refer as Iteration 0. In this phase initial scoping and
requirements takes place. Preparation activities also occur such as assembling the
technical infrastructure, setting up test harness for test driven development and
assembling the team. After Iteration 0 comes a collection of iterations where each
iteration delivering a part of working functionality. Iterations are bound with time
allocations which tend to be between one to six weeks. The common concept in agile
methods is the concept that a deployment should be occurred as conclusion of each
iteration.
Although the advantages of XP had been proven but there are still some
disadvantages of XP. Table 2.3 shows the advantages and disadvantages of XP in
software development project.
 18

Table 2.3: Advantages and disadvantages of XP

Advantages Disadvantages
XP maximizes the communication among all XPs most problematic feature is the amount of
stakeholders [24]. on-site customer involvement it requires [24].

It welcome changing requirements, even late in The issue regarding what extent to which XP can
development [25]. adapt to changing requirements is still open to
discussion between the researcher [44].

2.2.3 Comparison between Heavyweight and Lightweight Methodology

Many years ago, software development methodologies were basically limited to the
process of coding and fixing the error found in the coding. At that moment, these
heavyweight methodologies continue to dominate the systems development [18].
Despite the success it has a lot of drawbacks, like linearity, inflexibility in changing
requirements, and high formal processes irrespective of the size of the project [31]. As
the system grew, they became prone to bugs and also unable to run efficiently.
Therefore, the newer methodologies arise with the aim of improving the software
development flexibility and efficiency. As there are various models of software
development life cycle, each has its own advantages and disadvantages depending
upon which have to be decided and which model should be selected [4].
There are many different characteristics between the heavyweight and
lightweight methodologies. According to Dyba & Dingsoyr [48], the differences
between heavyweight and lightweight development basis on the an unpredictable
world, as well as emphasizing the value competent people and their relationships bring
to software development. Research by Moniruzzaman & Akhter [18] shown the reason
of lightweight methodology adoption is mainly because of the acceleration time to
market the product. Table 2.4 shows the comparison of heavyweight and lightweight
methodologies [49].
 19

Table 2.4: Comparison of heavyweight and lightweight methodologies [49]

Issues Heavyweight Lightweight

Customers Committed, knowledgeable, Access to knowledgeable,
collocated, collaborative, collaborative, representative,
representative, empowered empowered customers

Requirements Mainly emergent, rapid change. Knowable early, largely stable.

Architecture Designed for current requirements Designed for current and

foreseeable requirements

Size Smaller Team and Products Larger Team and Products

Primary objective Rapid value High assurance

Developers Knowledgeable, collocated, Plan-driven, adequate skills,

collaborative access to external knowledge.

Release cycle In phases (multiple cycles) Big bang (all functionality at

once)

Development life cycle Linear or incremental Incremental

Style of development Adaptive Anticipatory

Documentation Light (replaced by face to face Heavy / detailed Explicit

communication) knowledge

Team members Co-location of generalist senior Distributed teams of specialists

technical staff
Client Involvement onsite and considered as a team Low involvement
member Active/proactive
Measure of success Business value delivered Conformance to plan

There were also many researchers that conducted comparative studies between
specific heavyweight and lightweight methodologies. For example, Liu & Umphress
[50] conducted comparative study between IEEE 1074 and XP [50], Abrahamsson et
al. [17] had conducted comparative study between several agile methodologies itself,
Balaji & Murugayan had performed comparative study between Waterfall model, V-
Model and also Agile modeling while Moniruzzaman & Hossain [18] had conducted
comparative study between Waterfall and Agile Modeling. Table 2.5 summarizes the
authors research work results.
 20

Table 2.5: Summary of comparative study from different authors

Author Comparative Study Result

Liu & Umphress, 2008 IEEE 1047 and XP XP is more suitable for developing grid
[50] software.

Abrahamsson et al., 2010 Among difference The better approach depends on different
[17] Agile modeling` situation of system development.

Balaji & Murugayan, 2012 Waterfall, V-Model and The better approach is depends on how
[25] Agile Modeling frequent the user requirement change.

Moniruzzaman & Hossain, Waterfall, Agile Agile modeling significantly improves

2013 [18] Modeling software development process in many
ways.

Apoorva & Deepty, 2013 Waterfall, V-shaped, The hybrid of all these methodologies is
[4] Rapid Application used with some modification will help to
Development (RAD), improve the quality of developed software.
Incremental, Spiral

2.3 Software Testing

Software testing is an integral part of the software development process. All the
activities involved in developing software need to be carefully coordinated in order to
meet the requirements. According to Mustafa & Khan [16], it is not unusual for
developers to spend 40 percent of the total project time to perform software testing.
Testing can be considered as a process of planning, preparing, executing and analyzing
the difference between the actual and the required status [16]. According to Singh [51],
testing is the process of executing a program with the intent of finding faults. In other
words, software testing is a process which is used to measure the quality of software
developed. Any software which is produced without any testing can be dangerous to
the users. Therefore the importance of software testing in software development are as
follows [52, 53]:
(i) To improve software quality;
(ii) For reliability estimation;
(iii) To measure usefulness of software; and
(iv) For verification and validation.
In average 50% of the software failure are because of lack of cooperation,
weak task backlog, and lack of software testing resources were common bridge causes
[54]. For many years until now developers are still using the same testing techniques
 21

some of which is crafted method rather than good engineering methods [55]. In order
to perform software testing, there are few software testing strategies that can be
followed such as unit testing, integration testing, system testing and user acceptance
testing. Figure 2.6 shows the relationship between all the software testing strategies
while performing testing activities [51]. However, this research focused on user
acceptance testing only where testing is done by the customer.

Integration Acceptance
Unit Testing System Testing
Testing Testing

Any testing technique (verification plus validation) is applicable. Testing Testing is done by

is done by testers only. the customer

Software is ready for the

customer

Figure 2.6: Software testing strategy [51]

2.3.1 Unit Testing

Unit testing can be considered as the basic level of software testing where it focuses
on smaller blocks of a program or system [56]. This process will give developer the
ability to verify whether the system unit functions as expected. Therefore, for any
function with a set of inputs, we can determine if the function is returning the proper
values. Although performing unit testing is to ensure that the unit implemented
structure matches the intended design structure but there are also problems with unit
testing [51]. The problem that usually occurred is how to run each unit independently
when there is a unit that may not be completely independent.
Unit testing is generally seen as a white box test class which purposely looking
and evaluating the code as implemented rather than conformance user requirements.
The followings are unit testing benefits according to Abhijit et al. [55]:
(i) Unit level testing is very cost effective.
(ii) It provides a much greater reliability improvement for resources expanded than
system level testing.
 22

(iii) Be able to test parts of a project without waiting for the other parts to be
available.
(iv) Achieve parallelism in testing by being able to test and fix problems
simultaneously by many engineers.
(v) Be able to detect and remove defects at a much less cost compared to other
later stages of testing.

2.3.2 Integration Testing

Integration testing is a systematic technique for constructing the program structure

while at the same time conducting tests to uncover errors associated with interfacing.
The objective is to take unit tested components and build a program structure that has
been dictated by design [55]. Integration testing can be divided into two categories
which are top down integration testing and also bottom up integration testing.

2.3.3 System Testing

System testing is performed once the unit testing and integration testing had been done.
In this testing all the developed software is test along with the expected environment
including hardware and other associated parts [51]. System testing is conducted on the
entire system in Software Requirement Specification (SRS). It tests the behaviour and
even the believed expectations of the customer. It is also intended to test up to and
beyond the bounds defined in the software/hardware requirements specification.

2.3.4 User Acceptance Testing (UAT)

UAT is a type of testing that carried out to verify whether the product is developed
according to the standards and specified criteria hence meets all the requirements that
specified by customer [57]. Generally, this testing is carried out by a user/customer
where the product is developed externally by another party. UAT categorized under
black box testing methodology where the user is only focusing on the overall system
function and compares it with the requirements specified by them. UAT is considered
 23

to be one of the most important testing by user before the system is finally delivered
or handed over to the end user.
Testing accomplishes a variety of things especially the quality of the
application. UAT does three things for a software development project [58]:
(i) They measure how compliant the system is with the business requirements.
(ii) They expose functionality/business logic problems that unit testing and system
testing have missed out since unit testing and system testing do not focus much
on functionality/business logic testing.
(iii) They provide a means of determining how done the system is.
According to Bordo [59] and Lewis [56], UAT process can be divided into six
activities such as analyzing business requirement, identify UAT scenarios, define the
test plan, create test case, run the tests and record the results.

2.4 Designing User Acceptance Testing

The steps of performing UAT are test strategy, test development, test
execution, defect management and delivery [60]. The test strategy, schedule and also
resources need to be defined at the beginning of the UAT process by the manager. It
is vital to decide the test strategy as it will affect the delivery phase because the UAT
is done just before delivery process. After completing the test strategy, the test
development is performed. This step consists of test plans and business scenario test
cases. It is followed by the test execution step which includes defects, reports, and
metrics. The following step is the defect management which consists of defect tracking
and verification. The last step is the delivery phase that includes the UAT plan with
verification and testing strategy. Figure 2.7 shows the UAT steps [60].
 24

Figure 2.7: UAT steps

One of the important questions in planning to perform UAT is what need to be

tested. This question is crucial as it will identify the objects that are needed for
planning the UAT. During planning the UAT, elements need to be considered are
documentation, business requirements, functional requirements, business processes,
workflows, transactions and the software [61]. It is impossible to test every possible
issue in each business system. Therefore, the most significant functionalities to be
tested are business functions. Business users should have the knowledge and
understanding of their business processes [62]. Even though choosing what to test is
so hard, but under certain circumstances a test stakeholder as well as the test team have
to consider the item to be tested [63].

2.4.1 The Importance of UAT

The importance of UAT is the ability to keep ongoing maintenance costs as low as
possible because the costs of fixing functionality and usability is cheaper if the
problem can be detected earlier. Performing UAT in various stages and with various
types of test respondents can provide ideal opportunities to identify and repair broken