See

discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/303987803

Effort Estimation in Co-located and Globally

Distributed Agile Software Development: A
Comparative Study
Conference Paper October 2016

CITATIONS

READS

39

2 authors, including:
Ricardo Britto
Blekinge Institute of Technology
31 PUBLICATIONS 67 CITATIONS
SEE PROFILE

Available from: Ricardo Britto

Retrieved on: 20 October 2016

Effort Estimation in Co-located and Globally

Distributed Agile Software Development: A
Comparative Study
Muhammad Usman

Ricardo Britto

Software Engineering Research Laboratory

Blekinge Institute of Technology
371 79, Karlskrona, Sweden
muhammad.usman@bth.se

Software Engineering Research Laboratory

Blekinge Institute of Technology
371 79, Karlskrona, Sweden
ricardo.britto@bth.se

AbstractContext: Agile methods are used both by both colocated and globally distributed teams. Recently separate studies
have been conducted to understand how effort estimation is
practiced in Agile Software Development (ASD) in co-located
and distributed contexts. There is need to compare the findings
of these studies.
Objectives: The objective of this comparative study is to identify
the similarities and differences in how effort estimation is
practiced in co-located and globally distributed ASD.
Method: We combined the data of the two surveys to conduct
this comparative study. First survey was conducted to identify
the state of the practice on effort estimation in co-located ASD,
while the second one identified the same in globally distributed
ASD context.
Results: The main findings of this comparative study are: 1)
Agile practitioners, both in co-located and distributed contexts,
apply techniques that use experts subjective assessment to
estimate effort. 2) Story points are the most frequently used
size metrics in both co-located and distributed agile contexts 3)
Teams prior experience and skill level are leading cost drivers in
both contexts. Distributed agile practitioners cited additional cost
drivers related to the geographical distance between distributed
teams. 4) In both co-located and distributed agile context, effort
is estimated mainly at iteration and release planning levels 5)
With regard to the accuracy of effort estimates, underestimation
is the dominant for both co-located and distributed agile software
development.
Conclusions: Similar techniques and size metrics have been used
to estimate effort by both co-located and distributed agile teams.
The main difference is with regard to the factors that are
considered as important cost drivers. Global barriers due to
cultural, geographical and temporal differences are important
cost and effort drivers for distributed ASD. These additional
cost drivers should be considered when estimating effort of a
distributed agile project to avoid gross underestimation.

I. INTRODUCTION
Efficiency and profitability are two objectives that have been
relentlessly pursued by the software industry over the years.
To do so, companies worldwide have developed software in
a globally distributed manner (Global Software Engineering GSE) to achieve benefits such as reduced time-to-market and
access to skillful people all over the world [17], [5], [2], [9].
Despite all the benefits argued to be achieved by means
of GSE, there are several challenges associated with this

operational model, which often impact the productivity [11]

and effectiveness [6] of distributed software development,
leading to delayed projects [10], [19].
To ease some of the issues associated with developing
software in a globally distributed manner, companies have
employed agile software development (ASD) methods [18], [1]
in globally distributed projects; agile methods are regarded as
being more suitable to deal with projects that present unclear
and uncertain requirements [1]. However, existing empirical
evidence suggests that practices associated with ASD methods
successfully applied in co-located projects are not readily
applicable in globally distributed projects [12], [13].
As aforementioned, existing literature reports that it is often
the case that software projects (both co-located and globally
distributed contexts) are delayed, which may indicate that
practitioners have fallen short of providing accurate and reliable effort estimates in both collocated and globally distributed
projects.
To understand the state of the practice on effort estimation,
two survey investigations were recently conducted, one focused on the agile co-located context [20], while the other focused on the agile globally distributed context [3]. Considering
that existing literature suggests that is necessary to adapt agile
practices to apply them in the globally distributed context, it
may be the case that the way effort is estimated for globally
distributed projects differs from the way it is estimated in colocated projects.
Therefore, the main objective and contribution of this paper
is to perform a comparative analysis to identify the similarities
and differences in the way effort estimation is carried out in
the agile co-located and globally distributed contexts. To do
so, we use the results of the two previously conducted survey
investigations [20], [3]. To the best of our knowledge, to date
there is no study that analyze the state of the practice on effort
estimation considering the two aforementioned contexts at the
same time.
The remainder of the paper is organized as follows: The
research methodology employed herein is described in Section
II. Section III describes and discusses the results of the study.

Finally, Section IV concludes our work and presents our view

on future work.
II. RESEARCH METHODOLOGY
As part of a larger project to investigate effort estimation
in Agile Software Development (ASD), two surveys were
performed to identify state of the practice on effort estimation
in ASD. First survey [20]1 focused on effort estimation in
co-located ASD, while the second one [3] 2 studied effort
estimation in globally distributed ASD context.
In the present study we combined the data from these
two surveys to analyze and compare agile effort estimation
practices in co-located and global contexts. Our goal is to
investigate how difference in terms of co-location and distribution impacts the effort estimation practice in ASD. The details
regarding the motivation for the choice of research method,
sampling techniques, and validity threats for both surveys are
reported elsewhere[3], [20].
The following research questions guided this comparative
study:
RQ1: Which techniques are used to estimate effort in
co-located and globally distributed ASD contexts?
RQ2: Which effort predictors (size metrics and cost
drivers) are used while estimating effort in co-located and
globally distributed ASD contexts?
RQ3: In what context effort estimation is performed in
co-located and globally distributed ASD?
RQ4: How accurate are the effort estimates in both
contexts?
RQ5: What are the main reasons for the inaccuracies in
effort estimates in both contexts?
With RQ1, we aim to identify and compare commonly used
effort estimation techniques in both context. RQ2 is designed
to identify the similarities and differences in relation to the
effort predictors used in each context. In ASD effort estimation
is performed at different planning levels (e.g. release, sprint),
and different development activities are accounted for in
estimates. RQ3 aims to identify the differences in the contexts,
in terms of planning levels and development activities, in
which effort is estimated in co-located and distributed ASD.
RQ4 attempts to compare accuracy levels of effort estimates in
both contexts. The purpose of RQ5 is to identify and compare
the reasons for the inaccurate effort estimates in both contexts.
Survey C and D both included two types of questions: 1)
Close ended questions where respondents provided their input
related to RQ1-RQ4 and their demographic data. These close
ended questions included an open field as well where respondents were also allowed to enter free text, if required. 2) Open
ended question related to the RQ6 above where respondents
enter free text to state the reasons for the inaccuracies in effort
estimates. Similar design of the questionnaire instruments for
both surveys allowed us to combine the data from both surveys
relatively easily to perform this comparative study.
1 In
2 In

the remainder of this paper, we refer to this survey as survey C.

the remainder of this paper, we refer to this survey as survey G.

III. RESULTS
In this section we present the results of the comparative
analysis. First, in Section III-A we present the demographics
of the two surveys respondents, and in Sections III-B to III-E
we present the results associated with our research question.
A. Respondents Demographics
In total, 111 practitioners participated in the two surveys
(60 in the agile co-located survey C [20] and 51 in the survey
G [3].
The respondents of survey C are from 5 continents and
16 different countries. Approximately half of them reported
having managerial roles such as product owner, agile coach
and team lead. A vast majority (82%) of these respondents
had over 5 years of software industry experience. In addition,
most of them are from companies that employ primarily
Scrum, XP and Kanban to develop software mainly in the
telecommunication domain.
The respondents of survey G are from 5 continents and
12 different countries. Like the respondents of survey C, most
respondents of survey G (76.5%) have over 5 years of software
industry experience, being distributed in many different roles.
Furthermore, in survey G we see the same trend of survey
C in relation to the most commonly used agile methods and
domains wherein software is developed.
B. RQ1: Estimation Techniques
Figure 1 presents the estimation techniques used by practitioners in both contexts. Planning poker is the estimation
technique reported the most in both surveys (73% co-located
agile and 63% globally distributed agile context). However,
there are different trends in relation to the second most cited
technique in each survey; estimation by analogy is the second
most frequent technique in the distributed context (47%), while
expert judgment is the second most reported technique in the
co-located context (48%).
Note that the respondents in both surveys were allowed to
select multiple estimation techniques, i.e. the percentages in
Figure 1 do not sum up to 100%3 . As a result, 52% of survey
Cs respondents and 49% of survey Gs respondents reported
the usage of two or more effort estimation techniques.
C. RQ2: Effort Predictors
Figure 1 shows that in both contexts story points is the
size metric reported the most by the respondents (62% in
the co-located context and 64% in the distributed context).
The only identified difference is in relation to the second
most frequently used size metric in each context; second
most frequently used size metric in co-located agile context is
function points and in distributed agile context it is use case
points.
When comparing the cost drivers reported in both surveys,
we identified that there is a set of them that is reported by respondent from both type of agile contexts. Nevertheless, there
3 Note that the same applies to the results associated with RQ2, RQ3, RQ4
and RQ6

Co-located agile context

Distributed agile context

80%

73%

70%

63%

60%

60%

42%

23%

20%

30%

ec
i fy
)

ica
6o

st

ea
s
pl
he
rs
(

na
nc
6o

No
n

Cu

fu

Ot

ire
m
en

om
er
co
m
m
un

l r
eq
u

ec
t d

ts

n
om
ai

ze

sk
si

Pr
oj

Ot
he
r (
p

Fig. 3. Cost drivers used

Us
e
C

as

e
Po
i

nt

Ta

r e
xp
rio

e/
s
er
6s

CO

CO

Te
am
's
ex
p

O
M

hi
De
lp

le
as

on

e
sp

et
h

ec
i fy
)

od

t
m
a8
s E

Ex

s8

on

pe
rt
Ju
d

y A
na

lo

ge
m
en

gy

g P
ok
er

8m
a8

er
ie
nc
e

le
ve
l
kil
l

02% 04%

00%
in

17% 15%

00%

08%
05%

10%

Es

19%

10%

13% 15% 13% 15%

20%

nn

40% 40%
35%

30%

31%

Pl
a

42%

40%

38%

40%

56%

50%

48%

47%

50%

Distributed agile context

67%

65%

Te
am
's
p

70%

72% 71%

e
sp

Co-located agile context

80%

70%

65%
59%

60%

Fig. 1. Effort Estimation Techniques

50%

45%

41%

40%
Co-located agile context

70%

62%

Distributed agile context

31%

30%

64%

22%

20%

60%

10%

50%

ce

ca
l
hi

ge
d
ng
ua

i st

an

er
en
ce

te
s
he
si
f t
La

el
o
m
od

Cu

ltu

re
d

i
ne
d
m
e
zo

s
oi
je
ct
p
Ob

f c
od
es
o
Lin

nt

s
Us
e
ca

se
p

oi

in
Fu

nc
9o

po

oi
or
y p
St

nt

ts

nt
s

Co

Ge
og
ra
p

04%

Pr
oc
es
s

02%

00%

Ti

08%

er
en
ce

es

be
tw
ee
n
4o

12%

10%

17%

ica

19%

20%

m
m
un

30%

si t

29%

28%

er
en
ce

00%

40%

Fig. 2. Size metrics used

are some cost drivers that were reported only by respondents

of survey G i.e. they are used only in globally distributed agile
context.
Figure 3 displays the cost drivers that were reported in both
contexts, while Figure 4 presents the cost drivers only cited by
the practitioners in the survey G. In the set of common cost
drivers, the results of both surveys are pretty similar, except
for the cost driver task size, which was much more frequently
reported by practitioners working in distributed agile context
(67% against 42%). About the cost drivers reported only in
survey G, not surprisingly the are related to the challenges
associated with global software engineering, such as time
zone, socio-cultural and geographical differences.
D. RQ3: Development context
The RQ investigates the differences in the context, in terms
of planning levels and developed activities accounted for in
estimates, in which effort estimation is carried out in colocated and distributed ASD.

Fig. 4. Cost drivers specific to distributed agile context

1) Effort estimation and planning levels: Figure 5 shows

that effort estimation is carried out mainly at the iteration
planning level in both co-located (82%) and distributed agile
(89%) contexts. Effort estimation is also performed at the
release planning level. The results show that relatively higher
percentage of the respondents from survey G (65%) cited
using effort estimation to support release planning as compared
to the respondents from survey C (42%). This indicates that
estimation may be more frequently performed during release
planning in distributed agile as compared to the co-located
agile context.
2) Effort estimation and software development activities:
The results (see Figure 6) show that in both co-located and
distributed agile contexts, implementation and testing are the
leading activities that are accounted for in effort estimates.
Design and analysis are cited by relatively fewer practitioners
in both surveys. A closer look at the results reveals that
in co-located agile context implementation (87%) by far is
the leading activity that is accounted for in effort estimates,
followed by testing (68%). Implementation (92%) is still the
leading estimated activity in distributed agile context as well.
However testing (85%), design (73%) and analysis (52%)

Co-located agile context

Co-located agile context

Distributed agile context

Distributed agile context

40%
89%

35%

35% 35%

30%

65%

25%

42%

22%
19%

20%
08%

08%

10%
07%
04%

2m
at
ed

by
5
0%
o

r m
or
e

r m
or
e
by
5
0%
o
Ov
er
es

r m
or
e

m
at

ed

by
0
5%
o
er
es
2
Un
d

Ov
er
es

2m
at
ed

by
2
5%
o

by
0
5%
o
ed

r m
or
e

r m
or
e

n
2o
ria
m
at

Sp

ot
o

by
2
5%
o
ed

m
at
er
es
2
Un
d

(0
-5
)%
va

r m
or
e

le
as

04%
00%

00%

Ot
he
r (
p

Fig. 5. Estimation and agile planning levels

12%

05%

e
sp

ec
t B
id
Pr
oj

10%

10%

er
es
2

di

in
nn
ily
P
la
Da

17%

15%

Un
d

08%

08%

ng

13%

ec
i fy
)

15%

Ite
ra
2

on

/S

Re
le
as

pr
in

e
Pl
a

t P
la
nn

nn

in

in

17%

18%

2m
at
ed

82%

Ov
er
es

100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
00%

Fig. 7. Estimation residuals

activities are also considered during effort estimation by a

relatively higher percentage of distributed agile practitioners
as compared to co-located agile practitioners.
Co-located agile context

100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
00%

87%

Distributed agile context

92%
85%
73%

68%
53%

52%
43%
31%
23%

ec
i fy
)

nc
e
le
as

e
sp

nt
en
a
ai

Ot
he
r (
p

Re
qu

ire
m
en

ts

an

al
ys

is

De
sig
n

ng

Te
s5

Im
pl

em
en
ta
5

on

12% 12%

Fig. 6. Estimation and development activity

E. RQ4: Estimation Accuracy

The results for both co-located and distribution agile surveys
are presented in Figure 7. The results show a similar pattern
for the respondents of both contexts, i.e. underestimating the
development effort is the dominant trend.
F. RQ5: Reasons for the Inaccurate Estimates
A total of 28 respondents from survey C and 24 from
survey G provided additional information on the reasons
for inaccurate effort estimates, which was captured in both
surveys through open-ended questions. The identified reasons
are presented in Table I4 .
Requirements-related issues, such as missing and changing
requirements, are cited as leading reasons for inaccurate
estimates in both contexts. These results support previous
evidence where both changing and new requirements were
identified as main reasons for effort overruns (e.g., [8]).
Team related issues, such as inexperienced and unskilled
team members, are also responsible for inaccurate estimates
4 Note that we present both the number of respondents who replied a
particular reason in each survey, along with the associated percentages.

in both contexts. The lack of skills and experience of team

members were noted as reasons for inaccurate estimates previously as well [14].
The third category represents global barriers and comprises issues specific to the GSE respondents. These include
differences related to time zone, geographical distance and
culture. Understandably 13 respondents from the distributed
agile context cited these differences as one of the main reasons
for inaccurate estimates.
The fourth category represents project management-related
issues such as poor change control, scope creep etc. These
issues are cited by relatively fewer respondents of both surveys. These issues when not managed properly may lead to
time and cost overruns, and therefore increase the risk for
estimation inaccuracies.
Overoptimism is the fifth category and comprises two
specific reasons: considering only the best case scenario and
willfully underestimating to win a contract. The overoptimism
bias is also observed in other studies on agile development
[4], [15]. Willfully underestimating to win a contract is not a
fair practice, and can be categorized as a breach of the code
of ethics for software engineers described in [7].
Lack of a formal estimation process was reported by only 3
respondents, all from survey C. This corroborates with findings
of another work on agile estimation [14]. Likewise, ignoring
testing effort is also reported by only two respondents, all
from survey C too. Ignoring testing effort could lead to
underestimation of the development effort.
The eighth and last category represents the lack of customer
involvement during story sizing sessions and is reported by
respondents of both surveys. The effective involvement of the
customer during requirements specification stage and story
sizing sessions can help resolve ambiguities in the user stories.
Effective customer collaboration and commitment have been
associated estimation accuracy in other studies too [16], [8].
IV. CONCLUSIONS
In this paper we conducted a comparative analysis of the
results of two recent surveys on the state of the practice

TABLE I
R EASONS FOR INACCURATE ESTIMATES
#
1
2
3
4
5
6
7
8

Category of reasons for inaccurate estimates

Requirements related issues (Missing and changing requirements, Poor user
stories, Overlooking Non-functional requirements, etc.)
Team related issues (Inexperience, knowledge sharing problem in team,
Unskilled team members)
Global barriers (time, distance, culture)
Project management issues (Poor change control, Management not guiding
the team, Scope creep)
Overoptimism (Best case scenario, Purposely underestimating to obtain work)
Lack of formal estimation process
Ignoring testing effort
Insufficient customer involvement during story sizing

on the effort estimation in the agile software development.

Since each survey had a different focus, i.e. one focused
on co-located agile software development and the other on
globally distributed agile software development, we herein
compared the findings of both studies. Our aim was to identify
similarities and differences in the way practitioners carry out
effort estimation in these two different agile development
contexts.
As a result, we identified that differences are related to the
secondary effort estimation techniques (analogy for distributed
agile practitioners and expert judgment for co-located agile
practitioners), the second most frequently used size metric
(use case point for distributed agile practitioners and function
points for co-located agile practitioners), and the reported cost
drivers. The main difference lays on the cost drivers; we
identified that there are many cost drivers related to global
barriers, such as cultural and time zone differences, that were
reported only by the practitioners in survey G, which was not
a surprising finding.
The issues related to these global barriers are cited as main
differentiating reason by respondents of survey G for inaccurate estimates in globally distributed agile projects. These
global cost drivers should be considered explicitly by project
managers while estimating effort in a globally distributed
agile project to avoid grossly underestimating the development
effort.
Note that the findings of presented herein are limited to
the results presented in the two surveys used as basis for this
paper. Furthermore, this paper is exposed to all the validity
threats reported by Usman et al. [20] and Britto et al. [3].
As part of our future work, we plan to conduct interviews of
agile practitioners working in both contexts to further clarify
the main differences identified herein, and to identify practices
to effectively deal with the identified reasons of inaccurate
estimates.
ACKNOWLEDGMENTS
We would like to thank CNPq (National Counsel of Technological and Scientific Development, Brazil), UFPI (Federal
University of Piaui, Brazil) and INES (National Institute of
Science and Technology for Software Engineering, Brazil) for
partially support this work.

Co-located
agile context

Distributed
agile context

9 (32%)

20 (83.3%)

5 (18%)

14 (58.3%)

13 (54.2%)

7 (25%)

3 (12.5%)

3
3
2
2

2 (8.3%)
2 (8.3%)

(10%)
(10%)
(7%)
(7%)

R EFERENCES
[1] K. Beck and C. Andres. Extreme Programming Explained: Embrace
Change. Addison-Wesley, 2nd edition, 2004.
[2] A. B. Bondi and J. P. Ros. Experience with training a remotely
located performance test team in a quasi-agile global environment.
In Proceedings of the 2009 Fourth IEEE International Conference on
Global Software Engineering - ICGSE09, pages 254261, Washington,
DC, USA, 2009. IEEE Computer Society.
[3] R. Britto, E. Mendes, and J. Borstler. An empirical investigation on effort
estimation in agile global software development. In Proceedings of the
10th IEEE International Conference on Global Software Engineering ICGSE15, pages 3845, 2015.
[4] L. Cao. Estimating agile software project effort: an empirical study.
2008.

[5] E. Conchuir, P. J. Agerfalk,

H. Holmstrom, and B. Fitzgerald. Global
software development: Where are the benefits? Communications of the
ACM, 52(8):127131, aug 2009.
[6] J. A. Espinosa, N. Nan, and E. Carmel. Do gradations of time zone
separation make a difference in performance? a first laboratory study. In
Second IEEE International Conference on Global Software Engineering
- ICGSE07., pages 1222, Aug 2007.
[7] D. Gotterbarn, K. Miller, and S. Rogerson. Computer society and acm
approve software engineering code of ethics. Computer, 32(10):8488,
1999.
[8] S. Grimstad, M. Jorgensen, and K. Molokken-Ostvold. The clients
impact on effort estimation accuracy in software development projects.
In Software Metrics, 2005. 11th IEEE International Symposium, pages
10pp. IEEE, 2005.
[9] J. Herbsleb and D. Moitra. Global software development. IEEE
Software, 18(2):1620, 2001.
[10] J. D. Herbsleb and A. Mockus. An empirical study of speed and
communication in globally distributed software development. IEEE
Transactions on Software Engineering, 29(6):481494, June 2003.
[11] J. D. Herbsleb, A. Mockus, T. A. Finholt, and R. E. Grinter. Distance,
dependencies, and delay in a global collaboration. In Proceedings of
the 2000 ACM Conference on Computer Supported Cooperative Work CSCW00, pages 319328, New York, NY, USA, 2000. ACM.
[12] E. Hossain, M. Ali Babar, and H. Y. Paik. Using scrum in global software
development: A systematic literature review. In Proceedings of 4th IEEE
International Conference on Global Software Engineering - ICGSE 09,
pages 175184, Limerick, Ireland, 2009.
[13] S. Jalali and C. Wohlin. Global software engineering and agile practices:
A systematic review. Journal of software: Evolution and Process,
24(6):643659, 2012.
[14] S. Keaveney and K. Conboy. Cost estimation in agile development
projects. 2006.
[15] V. Mahnic and T. Hovelja. On using planning poker for estimating user
stories. Journal of Systems and Software, 85(9):20862095, 2012.
[16] K. Molokken-Ostvold and K. M. Furulund. The relationship between
customer collaboration and software project overruns. In Agile Conference (AGILE), 2007, pages 7283. IEEE, 2007.
[17] N. Ramasubbu, M. Cataldo, R. K. Balan, and J. D. Herbsleb. Configuring
global software teams: A multi-company analysis of project productivity,

quality, and profits. In Proceedings of the 33rd International Conference

on Software Engineering - ICSE11, pages 261270, 2011.
[18] K. Schwaber and M. Beedle. Agile Software Development with Scrum.
Prentice Hall, -, 2001.

[19] D. Smite.
Global software development projects in one of the biggest
companies in latvia: Is geographical distribution a problem? Software

Process: Improvement and Practice, 11(1), 2006.

[20] M. Usman, E. Mendes, and J. Borstler. Effort estimation in agile
software development: A survey on the state of the practice. In
Proceedings of the 19th International Conference on Evaluation and
Assessment in Software Engineering, pages 12:112:10. ACM, 2015.