Empirical Research in Software Architecture
How far have we come?
Matthias Galster
Department of Computer Science and Software Engineering
University of Canterbury
Christchurch, New Zealand
mgalster@ieee.org
Abstract Context: Empirical research helps gain well-founded
insights about phenomena. Furthermore, empirical research
creates evidence for the validity of research results. Objective:
We aim at assessing the state-of-practice of empirical research in
software architecture. Method: We conducted a comprehensive
survey based on the systematic mapping method. We included all
full technical research papers published at major software
architecture conferences between 1999 and 2015. Results: 17% of
papers report empirical work. The number of empirical studies
in software architecture has started to increase in 2005. Looking
at the number of papers, empirical studies are about equally
frequently used to a) evaluate newly proposed approaches and b)
to explore and describe phenomena to better understand
software architecture practice. Case studies and experiments are
the most frequently used empirical methods. Almost half of
empirical studies involve human participants. The majority of
these studies involve professionals rather than students.
Conclusions: Our findings are meant to stimulate researchers in
the community to think about their expectations and standards of
empirical research. Our results indicate that software
architecture has become a more mature domain with regards to
applying empirical research. However, we also found issues in
research practices that could be improved (e.g., when describing
study objectives and acknowledging limitations).
Keywords software architecture, empirical research, state-of-
practice
I. INTRODUCTION
A. Problem and Motivation
The objective of empirical software engineering research is
to collect and use evidence to advance methods, processes,
techniques, and tools [1]. Evidence obtained from empirical
works helps us build trust and confidence in research findings
rather than relying on promotional anecdotes or rhetoric. Since
software architecture is primarily an applied discipline,
evidence includes insights about the usefulness and
applicability of research findings in practice.
In their 2010 article on applying empirical research to
software architecture, Falessi et al. promote the application of
the empirical paradigm to software architecture [1]. However,
there has been little effort to systematically gather, rigorously
analyze and widely disseminate empirical evidence in software
architecture [2]. olution (which
pr
and the applicability of the solution by a small example or
Danny Weyns
Department of Computer Science
Katholieke Universiteit Leuven, Belgium
Linnaeus University, Vxj, Sweden
danny.weyns@cs.kuleuven.be
demonstrator) is the dominant research type in architecture-
related areas such as component-based software engineering
[3]. In general, empirical models and methods do not appear to
be very popular in the architecture community [4].
A study from 2015 found that around 94% of papers at
recent editions of premier software engineering conferences
included an empirical method (e.g., case study, controlled
experiment) [5]. However, no comprehensive reports exist on
the state-of-practice of empirical research in software
architecture. Such reports could offer insights into the maturity
of the field of software architecture research and the amount
and type of evidence available to increase confidence in
research findings. Furthermore, such reports could pinpoint
areas in research practices that are still weak and potentially
require more encouragement and appreciation in the
community, and that may require more focused training of
architecture researchers.
B. Paper Goal and Research Questions
We aim at finding out whether the increasing trend of
applying empirical research in software engineering is also true
for software architecture research, or if evidence in software
architecture research primarily relies on anecdotes and rhetoric.
Therefore, the goal of this paper is to analyze (full technical)
research papers to better understand the state-of-practice of
empirical research from the point of view of researchers in the
context of software architecture. To operationalize this goal,
we define several research questions.
RQ1: How is empirical research applied in software
architecture? Empirical research can be used to evaluate a
newly proposed approach (e.g., a controlled experiment to
compare a new architecture recovery approach A to an
alternative approach B), or it can be used to increase our
understanding about a phenomenon (e.g., as an exploratory or
descriptive survey to understand what makes software
architecture decision making difficult). Therefore, we aim at
finding out whether empirical work in software architecture is
published as stand-alone studies (e.g., an exploratory survey)
or if empirical studies tend to be published together with
solution proposals (e.g., the proposal of a new approach
together with an empirical evaluation). Furthermore, we
investigate what empirical methods (case study, experiment,
etc.) are most prominent. This research question is exploratory
in nature and helps identify typical use cases of empirical
software architecture research.
 RQ2: What is the role of human participants in empirical
software architecture research? Software architecture activities
are often design and creative activities and therefore require
human input.
many trade-off decisions, which
depend on the experience and expertise of those involved.
Therefore, we analyze to what degree humans are involved in
empirical studies. With regards to human participants as
subjects in software engineering research, there has been a
debate about whether students are representatives for
professionals [6]. Also, due to limited availability of
participants, sometimes studies rely on colleagues or students
of the researchers conducting the study which may result in
biased findings [7]. Therefore, we investigate to what degree
empirical architecture research involves students and
professionals.
RQ3: To what degree does empirical software architecture
research acknowledge validity threats? To increase the
confidence in empirical results, we need to be aware of validity
threats. Without a discussion of validity threats, it is difficult to
judge the strengths and limitations of a study with regard to the
validity and applicability of its outcome [8]. Therefore, we
study if current empirical research reports validity threats, and
if so, what types of threats are discussed. For example, do
studies focus on internal validity (to control every aspect of a
study), external validity (to show a real-world effect, but
without fully understanding which factors actually caused a
certain result), or something else?
C. Contributions
Our research is meant to stimulate software architecture
researchers to think about their expectations and standards of
empirical research (i.e., our aim is not to criticize papers). We
provide the following contributions:
Overview of the state-of-practice over time of applying
empirical research in software architecture. This provides
insights in the growing maturity of empirical research and
awareness for the need of evidence. It also offers
researchers new perspectives and opportunities to study and
advance empirical work.
Analysis of how empirical research is used and why.
Software architecture researchers can use this information
to select methods and to benchmark their own work. This
can assist researchers in building new theories and
evaluating their work. Educators can identify issues when
training new researchers in applying empirical research.
Analysis of how trustworthy empirical software
architecture research is, in the sense to what degree validity
threats are acknowledged. Practitioners can use these
insights to form their view on software architecture
research (and the community).
We do not argue that more empiricism is always better or
that empiricism determines (or is even the key for) impact in
software architecture. As argued by Shaw, the nature of
validation can differ depending on the nature of a research
question [9]. Many seminal papers (e.g., Ward Cunningham's
paper on technical debt) have no empirical results but
significant impact. We discuss this further in Section V.A
when we relate research themes to our findings.
D. Paper Structure
In Section II we discuss background and related work on
empirical research in software engineering and software
architecture. We then outline our research method in Section
III before we present the results in Section IV. Our findings are
further discussed in Section V. We conclude in Section VI.
II. BACKGROUND AND RELATED WORK
In 2006, Zannier et al. investigated empirical software
engineering research at the International Conference on
Software Engineering (ICSE) [10]. The study picked 5% of all
papers at ICSE. It found that 70% of papers in the sample
included some form of evaluation. Experience reports rather
than rigorous empirical studies following a well-established
method were the most frequently type of work presented.
Furthermore, they found an absence of replications and few
papers that state a clear hypothesis. A study in 2015 on views
of internal and external validity in software engineering
research [5] reviewed 405 papers from premier software
engineering conferences (ICSE, ESEC/FSE and EMSE). That
study reported an increasing trend of empirical research.
Some studies on the use of particular empirical methods
exist. For example, Sjoberg et al. studied experiments in
software engineering [11] and found in 2005 that only around
2% of papers in leading software engineering conferences and
journals report controlled experiments. In 2013, systematic
literature reviews were empirically explored by Zhang and Ali
Babar [12]. Their study found that since 2004 systematic
literature reviews have become popular in software engineering
but it is challenging to balance effort and rigor when
conducting such reviews. An analysis of recent systematic
mapping studies was performed by Petersen et al. in 2015 [13].
With regards to empirical practices, that study found that most
mapping studies report validity threats. In 2015, de Magalhaes
et al. investigated replications in software engineering and
found that we still lack standardized concepts, guidelines and
methodologies to fully support replications [14].
Several analyses have been presented to investigate
empirical research in particular areas of software engineering.
For example, Daneva et al. briefly reviewed empirical work on
software requirements engineering [15]. The study reported a
significant growth in empirical requirements engineering
research with a jump in 2004 and 2005. Dyba and Dingsoyr
investigated the status of empirical research on agile software
development [16]. The study concluded that there is a need for
more and better empirical studies on agile software
development. In 2013, Weyns and Ahmad reported a
systematic literature review on the validation of architecture-
based approaches for self-adaptive systems [17]. They showed
that only 2.5% of studies in this area applied empirical
methods. In software architecture it has been claimed that it is
hard to apply empirical methods (e.g., due to difficulties
measuring architecture goodness, dependencies between design
decisions, social factors, types of required subjects in studies
with human participants) [18] but, to the best of our
knowledge, no study exists on the state-of-practice of empirical
research in this field. A special issue of the Empirical Software
Engineering journal was published in 2011 [2] and a workshop
on Empirical Assessment in Software Architecture (EASA)
was held at WICSA in 20081 and 20092. However, the
contributions of these initiatives focused on presenting
empirical research, rather than on reflecting on research and
providing an overview of the state-of-practice. Therefore, our
work complements these efforts and adds insights about how
and why empirical research is conducted in architecture.
III. RESEARCH METHOD
We performed a comprehensive survey following
procedures defined in the mapping study method [19]. A
systematic mapping study is a type of literature review that
aims to collect and classify research related to a specific topic,
in our case empirical software architecture research. Mapping
studies aim at structuring an area and showing how the work in
that area is distributed within the structure [13]. This requires a
rigorous process to identify relevant papers, and to apply
systematic data collection and analysis to ensure completeness
and representativeness.
We deviated from mapping study procedures in that we had
a clearly defined data extraction form derived from our
research questions, rather than key-wording and mapping out
existing papers. We deviated from the systematic literature
review method in that we did not apply quality assessment of
included studies since our aim was to obtain insights into all
research that has been published in software architecture. Also,
our scope was limited, so some issues of mapping studies or
systematic literature reviews did not apply (e.g., defining
search criteria and a search string for an automated search,
selecting electronic information sources). As suggested by
Petersen et al., our research process consisted of planning,
conducing and reporting [13] as discussed below.
A. Planning
1) Study identification: Since we were interested in
empirical research in software architecture, we analyzed
papers published at venues that focus on software architecture.
Therefore, our sample consists of all technical full papers from
the International ACM SIGSOFT Symposium on Component-
Based Software Engineering (CBSE), the European
Conference on Software Architecture (ECSA), the
International ACM SIGSOFT Conference on the Quality of
Software Architectures (QoSA) and the Working IEEE/IFIP
Conference on Software Architecture (WICSA). These four
conferences are the main venues for publishing software
architecture research. We included all editions of these
venues. We excluded the SATURN series since it is a non-
paper practitioner conference (we acknowledge however that
talks at SATURN may present empirical works, in particular
case studies). While this selection is limited, it still gives a
representative impression of the state-of-practice. Also, while
journals may follow best practices in research, conferences use
the most common practices of research in a community [5].
We manually identified 667 full technical research papers
(Table I). Papers were categorized as full technical research
papers based on paper categories in conference proceedings
1
http://wwwp.dnsalias.org/wiki/Wicsa7:Workshop:Empirical_Assessment_in_Software_Architecture
[accessed December 2015]
2
http://www.iso-architecture.org/wicsa2009/easa2009.pdf
[accessed December 2015]
and programs, and page length (e.g., eight pages at ECSA were
considered short papers).
TABLE I. NUMBER OF ANALYZED PAPERS PER VENUE
Venue #
CBSE3: 2004 (first edition) 2015 192
4
ECSA : 2007 (first edition) 2015 138
QoSA: 2005 (first edition) 2015 135
WICSA: 1999, 2001/02, 2004/05, 2007/08, 2011, 2014/15 202
We manually analyzed each paper to decide whether or not
a paper presents empirical work. We analyzed the whole paper,
rather than only the title, abstract or keywords. Also, we did
not only rely on methods claimed in papers. For example,
many papers claim case studies even though they only
present small examples or illustrations of a newly proposed
approach. On the other hand, some papers do not use the
terminology of well-established empirical methods, but still
present findings based on systematically collected and
analyzed data. Therefore, to decide whether a paper presents
empirical work, we used a broader definition of empirical
research as suggested by Sjoberg et al. [20]: Empirical research
includes any research based on systematically collected
evidence
natural, social, or cognitive phenomena by using evidence
based on observation or experience. It involves obtaining and
interpreting evidence, by, e.g., experimentation, systematic
observation, interviews or surveys, or by the careful
[20]. As a
consequence, we not only checked whether there was a well-
established empirical method, but also whether there was a
clear study objective, whether data was collected and analyzed
systematically, and if findings were linked back to the
objective. Furthermore, if the authors claimed a method, we
looked for clues that the claim about the method was true (e.g.,
references to literature related to the claimed method). An
example of a paper that presented a case study but was
excluded since it did not clearly discuss data extraction and
fUML-Driven Design and Performance Analysis of
Software Agents for Wireless Sensor Network
ECSA 2014. Finally, studies must provide insights beyond the
studied case or example and offer insights about the claim or
statement in a paper. An example of a paper that presented a
study but was excluded because it did offer insights beyond a
studied case (without discussing general insights or insights
about an approach) was Assessing a Multi-Site Development
Organization for Architectural Compliance
2) Data extraction: Table II lists the data items extracted
for each paper. We used topic-specific classifications (e.g.,
where topics
were defined upfront, together with topic-independent
classifications that can be used in other mapping studies (e.g.,
[13]. The first few data items were collected
not to answer a particular research question, but to provide
3
In 2004, the 6th International Symposium on Component-Based Software Engineering was held.
Before 2004, CBSE was organized as ICSE workshop. Therefore, we excluded these editions of CBSE.
4
In 2009/12, ECSA and WICSA were co-located; papers for 2009/12 were recorded for ECSA only.
more detailed demographic information about papers. We study, multi-method study6). We added the category
explain some of the data items in more detail below. To systematic empirical enquiry for empirical work that does
extract data, we manually reviewed the whole papers. not follow the structure or design of well-established
methods, but still formulates objectives, collects and
TABLE II. EXTRACTED DATA analyzes data, and presents insights in a systematic manner
(see also our notion of empirical research in Section
Item Description RQ III.A.1). We did not rely on self-classification of methods
Venue Publication venu (conference) n/a or keywords, but reviewed papers manually.
Discussion of validity: We recorded whether validity
Year Publication year n/a threats are discussed explicitly (i.e., in a dedicated section
Paper title The title of the paper n/a or paragraph) or implicitly (i.e., hidden in a paragraph of
the methodology, discussion or conclusion), or not at all.
Academic authors Number of academic authors n/a
Types of validity: If validity was discussed, we recorded
Industry authors Number of industry authors 5
n/a whether external and internal validity (or other kinds of
validity) are differentiated. Also, we include an option of
Citation count Citation count as of 12/2015 n/a
and limitations are discussed,
Objective but no differentiation of types of validity is made.
See below n/a
formulation
3) Data analysis: Most data were analyzed using
Study focus See below n/a descriptive statistics. Data about the study focus underwent
Reason See below RQ1 content analysis and open coding to identify themes of
architecture research in papers. Content analysis categorizes
Method See below RQ1 data and analyzes frequencies of themes within categories. It
Replication Does the paper present a replication? RQ1 allows to transfer qualitative information into quantitative
information. Tabulation helped us analyze data from content
Repetition Does the paper present a repetition? RQ1 analysis. As suggested in Petersen et al. [13], we visualized
Subjects Human and/or non-human RQ2 our findings in bar plots and pie diagrams.
Human subjects n/a or students and/or professionals RQ2 B. Conducting and Reporting the Study
Discussion of validity See below RQ3 The study was conducted as outlined in the planning phase.
Parts of the process were iterative and revised. Both authors
Types of validity See below RQ3 were involved in selecting studies and extracting data.
Researchers were co-located and discussed throughout the
Objective formulation: To get a better idea about how process to ensure a shared understanding of inclusion criteria
clear research objectives are defined, we extracted and extracted data items, as well as to resolve ambiguity and
information about whether studies present (high-level) conflicts. Spreadsheets were used to record data
goals, detailed research questions, or concrete hypotheses. (http://www.cs.kuleuven.be/publicaties/rapporten/cw/CW691.a
Study focus: The selection of a research method depends bs.html). The reporting follows guidelines proposed by
on the studied phenomena and the nature of the research Petersen et al. [13].
question. For example, evidence for the correctness of a
IV. RESULTS
model checker may require formal proofs rather than
empirical studies. Therefore, we identified themes that
A. Demographics
empirical software architecture research has been targeting.
This was recorded as free text. We found 115 papers that report empirical work (17% of
Reason: We extracted the main reason for why the all 667 full technical research papers). In Fig. 1 we show the
empirical work was conducted. We differentiate focus and contribution of empirical papers compared to all full technical
evaluation. Focus means that empirical work is the core of research papers per year at CBSE, ECSA, QoSA and WICSA.
the paper with the empirical results as the main Due to the varying total number of papers at each conference
contribution. In this case, the goal of the paper motivates every year, we only show relative rather than absolute
the empirical work (e.g., an exploratory case study or a numbers. For example, in 2015, 43% of all full technical
systematic literature review). Evaluation means that the research papers reported empirical work.
main contribution of the paper is a new approach and
empiricism is used to evaluate this new proposal (e.g., an
experiment to evaluate the efficiency of a new viewpoint).
Method: We recorded empirical research method applied.
We used basic types of empirical methods (case study,
experiment, survey, systematic literature review, interview

6
Here, surveys do not include literature reviews. Interview studies differ from surveys in that they use
interviews as data collection instrument and often use a relatively small sample. Multi-method studies
5
Authors from non-academic institutions were counted as industry authors. combine two or more research methods in one paper.

Figure 1. Relative contribution of empirical works in relation to all papers.
Fig 1. shows an increasing trend to publish empirical work
starting 2005. This could be due to the fact that some of the key
publications about empirical software engineering were
published in early 2000. This trend is similar to empirical
research in requirements engineering where empirical works
started to increase since 2004 [15]. To compare the distribution
of empirical works in software architecture to empirical works
in software engineering in general, we contrast our findings to
Siegmund et al. [5]. Siegmund et al. only investigated papers
after 2011. Only looking at architecture papers published in
2011 or later we notice that 28% of papers present empirical
work, compared to 94% of general software engineering
research. A possible explanation for this gap could be that
[5], while we meticulously reviewed papers to
identify empirical works. This could have resulted in a more
selective inclusion of papers (see our discussion on the misuse
of terms and our notion of empirical work in Section III.A.1).
In Fig. 2 we visualize the distribution of empirical works of
each conference. Again, due to the varying number of total
papers published at each conference every year, we only show
relative numbers. For example, in 2015, 47% of all full
technical research papers at WICSA reported empirical work.
Fig. 2 shows an increasing trend of publishing empirical works
at all architecture-related conferences. To interpret Fig. 2
correctly, it should be noted that in 2009 and 2012 WICSA and
ECSA were co-located. Also, WICSA started in 1999, CBSE
in 2004, QoSA in 2005, and ECSA in 2007.
With regards to the number of authors, the average
number of authors of empirical papers is 3.15 (min: 1; max: 7)
Figure 2. Relative distribution of empirical works over time per conference.
and the average number of authors of non-empirical papers is
3.04 (min: 1; max: 12). With regards to authors from industry
and academia, we noticed that empirical papers have an
average of 2.7 academic authors (min: 0; max: 7) and non-
empirical papers have an average of 2.5 academic authors (min:
0; max: 11). On the other hand, empirical papers have an
average of 0.5 of industry authors (min: 0; max: 6), similar to
non-empirical papers with 0.5 industry authors on average
(min: 0; max: 10). Also, 70% of empirical papers do not have
authors from industry and 74% of non-empirical papers do not
have authors from industry. Therefore, we did not find any
significant difference in the types of authors in empirical and
non-empirical papers. Furthermore, most empirical papers have
two (31%), three (23%) or four (28%) authors but only very
few empirical papers have a single author (6% of empirical
papers). This distribution of authors in empirical papers is
similar to the distribution of the authors of non-empirical
papers. It is also similar to what has been found for the number
of authors of the top-100 papers in software engineering [21].
With regards to citation counts, we noticed that the
average annual citation count (i.e., the total number of citations
for a paper divided by the number of years since its
publication) for empirical papers is 2.80 (min: 0; max: 10.75).
The average annual citation count for non-empirical papers is
3.09 (min: 0; max: 50.3). In comparison, the average annual
citation count of the top-100 research papers in software
engineering ranges from 21.8 to 154.2 [21].
Key insights
Similar to software engineering research in general,
there is also an increasing trend to publish empirical
work in software architecture.
Empirical papers do not necessarily have more or fewer
authors from industry than non-empirical papers.
Citations of empirical papers are not necessarily higher
than citations of non-empirical papers.

B. RQ1: How is Emprical Research Applied
In the majority of empirical papers (54%), empirical work
is the focus (i.e., these papers present an empirical study, e.g.,
an exploratory case study, without proposing a new approach).
In 46% of empirical papers, empirical work is used to evaluate
a newly proposed approach (present new approach together
with empirical assessment). Fig. 3 shows that the number of
papers that focus on empirical work for exploratory and
descriptive studies developed similarly to papers that present
empirical work to evaluate new approaches.
Figure 3. Reasons to apply empirical research over time.
We found that the most frequently applied research method
are experiments, followed by case studies, see Fig. 4.
Figure 4. Distribution of empirical research methods.
Most papers that have an empirical study as their focus
conduct case studies (Fig. 5). On the other hand, most
evaluation studies use experiments.
Figure 5. Research methods versus reasons for conducting empirical work.
We also investigated whether empirical works report
replications or repetitions of previous studies. Replications or
repetitions would increase the confidence in and strength of
findings regarding a phenomenon (a replication runs the same
study on a different sample or different subjects but with
identical conditions; a repetition repeats the same study with
the same sample and involves measuring the same cases
multiple times). We did not find repetitions study. We found
one paper where the authors present a replication of their study
The Supportive Effect of Traceability
Links in Architecture-Level Software Understanding Two
Controlled Experiments WICSA 2014).
Key insights
Empirical research is adopted equally to explore/study
phenomena and to evaluate (new) approaches.
Case studies and experiments are the most frequently
used empirical research methods.
Most exploratory studies use case studies. Evaluation
studies tend to apply experiments.
Replications and repetitions are almost non-existent in
empirical software architecture research.
C. RQ2: Role of Human Participants in Studies
Forty-seven percent of empirical studies use humans as
subjects or study participants, and 8% involve both human and
non-human subjects (e.g., software systems, design documents,
source code). Overall, we noticed a slightly increasing trend of
involving human subjects in studies. This could be due to the
types of research topics investigated in software architecture
(see also our discussion in Section IV). With regards to the
types of human subjects, we found that 68% of the papers that
involve humans use professionals and 25% of papers use
students. Seven percent of papers with humans involve both,
professionals and students. Fig. 6 shows the distribution of
types of participants and subjects across research methods (in
%; e.g., 100% of interview studies involve humans). Most
experiments involve non-human subjects.
Figure 6. Use of human and non-human participants.
 In Fig. 7 we illustrate the types of human participants or
subjects. We show the percentages across all studies of a
method, not only the ones that involve humans (e.g., 55% of all
case studies involve professionals). Therefore, the percentages
in Fig. 7 do not add up to 100% for a research method since the
remaining studies included non-human participants.
increasing trend to explicitly discussing validity threats. Fig. 8
shows the growing gap between papers that explicitly discuss
validity threats and papers that do not discuss them or only
implicitly (we show percentages for years from 2005 since this
is when the number of empirical works started to increase).

Figure 8. Relative distribution of validity threats in papers 2005-2015.

We also investigated what types of validity were discussed

Figure 7. Types of human participants in empirical architecture studies.
Key insights
Almost half of the empirical software architecture
studies involve humans as subjects or participants.
Experiments mostly use non-human subjects. Most case
studies involve human participants.
if there was an explicit discussion of validity threats. Table III
show the relative number of papers that address certain validity
threats, given the total number of papers that present a certain
research method (e.g., 39% of all papers that include a case
study discuss internal validity). From Table III we can
conclude that interview studies and multi-method studies
acknowledge internal and external validity most, but systematic
literature reviews and surveys often discuss validity threats
without clearly differentiating them. Reliability (i.e., whether
other researchers who conduct the same study would achieve
the same results) is discussed least.

Professionals are the most frequently used type of Key insights

human participants (except for experiments where
students are the dominating type of human participant). Empirical software architecture research increasingly
acknowledges and discusses validity threats.
Internal and external validity are the most frequently
D. RQ3: Validity Threats discussed types of validity.
Most empirical studies (60%) explicitly discuss study
limitations or validity threats, whereas a small number of Many papers discuss validity but do not differentiate
studies (8%) only implicitly acknowledge limitations. On the types of validity based on well-established categories.
other hand, there is still a significant number of empirical
papers that do not mention any limitations or validity threats at
all (32%). This number is slightly higher than the number
reported by Feldt and Magazinius (20.9%) in 2010 who
surveyed a sample of 43 papers published at ESEM in 2009 for
their reporting of validity threats [8]. Overall, we noticed an
TABLE III. VALIDITY TYPES PER EMPIRICAL METHOD
Systematic
Validity type Case study Experiment Interview study Multi-method SLR Survey
empirical enquiry
Internal 0.39 0.37 0.67 0.67 0.29 0.40 0.06

External 0.42 0.42 0.67 0.67 0.29 0.40 0.11

Construct 0.36 0.11 0.33 0.00 0.29 0.30 0.11

Reliability 0.21 0.00 0.17 0.00 0.14 0.00 0.00

Conclusion 0.12 0.11 0.17 0.33 0.14 0.00 0.00

Not differentiated 0.27 0.24 0.17 0.33 0.71 0.60 0.39


V. FURTHER INSIGHTS AND DISCUSSIONS
A. Research Themes
We acknowledge that conducting empirical studies is one
way to evaluate work and to increase our confidence into
whether something (e.g., a new approach) works or not. Other
forms of obtaining evidence could include formal proofs and
not require collecting and analyzing data (e.g., for showing that
a model checker works correctly). Hence, the type of research
performed in a discipline depends on the types of research
questions and types of problems tackled. Therefore, based on
content analysis and open coding of free text obtained from our
III), we identified a
set of themes that seem to be prominent in empirical software
architecture research: component selection and composition,
patterns, architecture reasoning and decisions, self-adaptation,
architecture description, technical debt, quality (in general),
and other. In Fig. 9 we show how these themes are addressed
by different research methods (in Fig. 9 we show the absolute
number of papers per theme). Experiments tend to study
quality issues (such as performance measurements, security,
reliability) whereas surveys have been used for
(such as architectural decision making, reasoning and
knowledge management). As mentioned in Section III.B, we
noticed a slight increase in human involvement in empirical
software architecture studies. This could be due to the fact that
themes like architecture description, architecture reasoning and
decisions require human input in practice and therefore
empirical studies need to reflect this in order to provide results
useful in the real world . Note that in this research we do not
investigate the development of research themes in software
architecture in further detail.
B. Implications for Researchers
There is an undeniable trend towards applying systematic
empirical methods in the field of software architecture. Even
though we did not evaluate the quality of empirical works in
Figure 9. Software architecture research themes addressed by research methods
.
detail (e.g., quality of study design, strength of evidence), our
findings allow some insights into the maturity of empirical
software architecture research. For example, many papers
claim to present case studies but only provide example
applications of an approach. Furthermore, standard
terminology of empirical methods is often mixed up (what is an
experiment, what is a sample or unit of analysis, etc.). Also,
discussions of validity types across research methods are quite
uneven. Furthermore, the way that studies formulate their
objective does often not comply with requirements for
specifying objectives in empirical research (e.g., experiments
should have hypotheses). Based on data we extracted for
we observe that 47%
of studies describe concrete research questions and around 41%
of empirical studies describe the objective of the study as
higher level goals. On the other hand, only a small portion of
empirical studies (the remaining 12%) describe detailed
hypotheses. We acknowledge that hypotheses are most
common for experiments, but even only 32% of all
experiments state a clear hypothesis. Therefore, while the
increasing number of empirical studies raises the maturity of
the field, this also puts an increasing demand on planning,
conducting and evaluating research and its results [22].
Finally, we still lack replications and repetitions to build
solid knowledge and evidence. This could be due to the fact
that we still miss standardized concepts and terminologies,
frameworks, guidelines and methodologies to fully support
replications [14]. Another reason could be that software
engineering researchers often do not appreciate the value of
replications and repetitions but consider them as a way to
[5]. Therefore, we should
acknowledge that replications and repetitions are key enablers
for further maturing software architecture as a research and
engineering discipline. Similar to other events (e.g., empirical
tracks at REFSQ and XP), we may encourage dedicated tracks
for such studies at software architecture venues.

C. Implications for Practitioners
The fact that empirical works are getting more attention
should increase the confidence of practitioners in research and
evaluations of new approaches. However, given that studies
often claim case studies but present examples, practitioners
should be careful when interpreting results based on claims
made by authors of research papers. Practitioners may still
need to carefully check relevance and rigor of presented works.
Furthermore, the active involvement of professionals as
study participants but also as authors of empirical papers
underpins the practical relevance of research questions and
methods used in the domain. We found that many empirical
works include not only human participants, but obtain evidence
from professionals. This shows that there are opportunities for
practitioners to get actively involved in empirical research in
collaborations with academics (in particular when providing
data and access to projects). However, these joint efforts are
subject to the typical success factors of industry-academia
collaborations (e.g., champions in industry organizations, buy-
in support
commitment to contribute to industry needs) [23].
D. Threats to Validity
Internal validity: Internal validity is about the extent to
which a causal conclusion based on a study is warranted. In our
study, it was not always straightforward to identify full
technical research papers. This is because often paper
categories are not separated in the conference proceedings, the
conference program or even in the call for papers. Also, some
venues (e.g., CBSE) may not differentiate strictly between
technical full research papers, experience papers, industry
papers, short papers, position papers, etc. However, irrelevant
papers were excluded during the review and screening process
based on the length of a paper (e.g., a 4-pages paper is most
likely a short paper; for WICSA 2002 and 2004 we excluded
working session papers). To increase validity, we applied
manual filtering and did not rely on a keyword-based paper
selection and data extraction.
External validity: External validity is about generalizing
our findings to all published software architecture research. We
excluded short papers even though there are short papers that
Software Reference
Architectures -
at ECSA 2013 which presents an exploratory survey with
practitioners). We also did not include journals. Journals may
have higher standards in their requirements for empirical work.
However, as argued in Section III, while journals may follow
best practices in research, conferences use the most common
practices of research in a community [5]. Also, we were
interested in empirical research in software architecture and the
architecture community currently does not have a dedicated
journal. This would make it difficult to separate architecture-
related research from research not related to software
architecture, without performing another filtering step (this
filtering step could then cause additional validity threats).
Difficulties with separating architecture-related work also
apply to general software engineering conferences, such as
ICSE, FSE or ASE (which also publish original research on
software architecture). Therefore, conference papers collected
from all editions of four primary architecture-related venues
are a representative sample to derive conclusions about the
state-of-practice in the architecture research community.
Reliability: Reliability is about ensuring that our results are
the same if our study would be conducted again. Researchers
may have been biased in their decision of whether a paper is
empirical or not, in particular if no established method and
study design was included in a paper. Therefore, data
extraction and analysis was done by two researchers following
guidelines and discussions while being co-located. Also, the
papers of several venues were analyzed by both researchers to
increase confidence. However, classification of papers may
have been subjective in some cases and biased by the
background and experience of the researchers. This is also
related to conclusion validity, which is concerned with the
ability to replicate the same findings.
Construct validity: Construct validity is about obtaining
the right measure and whether we defined the right scope in
relation to what is considered an empirical paper. Poor
reporting of studies in papers may have affected our selection
and data extraction. Furthermore, we have identified data items
to identify empirical studies (see our notion of empirical
research in Section III.A). This notion may be biased. To
reduce this threat, we utilized ideas from [5] and [20].
Note that in contrast to other literature surveys we are not
concerned with issues such as publication bias. Publication bias
may prevent negative or new and controversial results to be
published (theoretical validity). In our study we were only
interested in the state of published empirical work and not in
the actual findings of that work.
VI. CONCLUSIONS
Empirical research has received growing attention in
software architecture. Besides implications discussed in the
previous section, we feel that based on our findings, educators
can devise educational programs to better train architecture
researchers and to include empirical methods in the curriculum
of (post-graduate) study programs.
We did not perform an in-depth analysis of empirical work
depending on the publication venue but aimed at providing a
general overview of empirical research in software
architecture. However, our data would certainly allow such an
analysis, which could be subject to future work.
We acknowledge that not all empirical studies are created
or applied equally, and many may not even provide any useful
information or value [24]. However, this paper does not aim to
evaluate the value and quality of empirical research in software
architecture. We did not explicitly evaluate empirical works for
their study design quality, credibility, usefulness, value and
power of evidence. Therefore, other future work includes
analyzing additional data, such as the number of human
participants in empirical works with human subjects to get
insights into the strength of evidence, the use of data
repositories, and the relevance and rigor of applied methods.
To complete the overview of empirical research in software
architecture, we need to obtain insights about the perception of
the architecture community on empirical research, similar to
Siegmund et al. for general software engineering [5].
 REFERENCES
[1] D. Falessi, M. A. Babar, G. Cantone, and P. Kruchten, "Applying
Empirical Software Engineering to Software Architecture: Challenges
and Lessons Learned," Empirical Software Engineering, vol. 15, pp.
250-276, June 2010.
[2] M. A. Babar, P. Lago, and A. v. Deursen, "Empirical Research in
Software Architecture: Opportunities, Challenges and Approaches,"
Empirical Software Engineering, vol. 16, pp. 539-543, 2011.
[3] T. Vale, I. Crnkovic, E. de Almeida, P. A. da Mota Silveira Neto, Y. C.
Cavalcanti, and S. R. de Lemos Meira, "Twenty-eight Years of
Component-based Software Engineering," Journal of Systems and
Software, vol. 111, pp. 128-148, 2016.
[4] J. Maras, L. Lednicki, and I. Crnkovic, "15 Years of CBSE
Symposium: Impact on the Research Community," in 15th ACM
SIGSOFT Symposium on Component Based Software Engineering
(CBSE) Bertinoro, Italy: ACM, 2012, pp. 61-70.
[5] J. Siegmund, N. Siegmund, and S. Apel, "Views on Internal and
External Validity in Empirical Software Engineering," in 37th
International Conference on Software Engineering Florence, Italy:
IEEE Computer Society, 2015, pp. 9-19.
[6] I. Salman, A. T. Misirli, and N. Juristo, "Are Students Representatives
of Professionals in Software Engineering Experiments?," in 37th
International Conference on Software Engineering Florence, Italy:
IEEE Computer Society, 2015, pp. 666-676.
[7] M. Galster and D. Tofan, "Exploring Web Advertising to Attract
Industry Professionals for Software Engineering Surveys," in 2nd
International Workshop on Conducting Empirical Studies in Industry
Hyderabad, India: ACM, 2014, pp. 5-8.
[8] R. Feldt and A. Magazinius, "Validity Threats in Empirical Software
Engineering Research - An Initial Survey," in 22nd International
Conference on Software Engineering and Knowledge Engineering
Redwood City, CA: KSI, 2010, pp. 374-379.
[9] M. Shaw, "Writing Good Software Engineering Research Papers,"
Proc. 25th International Conference on Software Engineering, IEEE
Computer Society, 2003, pp. 726-736.
[10] C. Zannier, G. Melnik, and F. Maurer, "On the Success of Empirical
Studies in the International Conference on Software Engineering," in
28th International Conference on Software Engineering Shanghai,
China: ACM, 2006, pp. 341-350.
[11] D. Sjoberg, J. E. Hannay, O. Hansen, V. Kampenes, A. Karahasanovic,
N.-K. Liborg, and A. C. Rekdal, "A Survey of Controlled Experiments
in Software Engineering," IEEE Transactions on Software Engineering,
vol. 31, pp. 733-753, 2005.
[12] H. Zhang and M. Ali Babar, "Systematic Reviews in Software
Engineering: An Empirical Investigation," Information and Software
Technology, vol. 55, pp. 1341-1354, 2012.
[13] K. Petersen, S. Vakkalanka, and L. Kuzniarz, "Guidelines for
Conducting Systematic Mapping Studies in Software Engineering: An
Update," Information and Software Technology, vol. 64, pp. 1-18,
2015.
[14] C. de Magalhaes, F. da Silva, R. Santos, and M. Suassuna,
"Investigations about Replication of Empirical Studies in software
Engineering: A Systematic Mapping Study," Information and Software
Technology, vol. 64, pp. 76-101, 2015.
[15] M. Daneva, D. Damian, A. Marchetto, and O. Pastor, "Empirical
Research Methodologies and Studies in Requirements Engineering:
How Far did we Come?," Journal of Systems and Software, vol. 95, pp.
1-9, 2014.
[16] T. Dyba and T. Dingsoyr, "Empirical Studies of Agile Software
Development: A Systematic Review," Information and Software
Technology, vol. 50, pp. 833-859, 2008.
[17] D. Weyns and T. Ahmad, "Claims and Evidence for Architecture-based
Self Adaptation - A Systematic Literature Review," in 7th European
Conference on Software Architecture (ECSA) Montpellier, France:
Springer Verlag, 2013, pp. 249-265.
[18] D. Falessi, P. Kruchten, and G. Cantone, "Issues in Applying Empirical
Software Engineering to Software Architectures," Proc. First European
Conference on Software Architecture, Springer, 2007, pp. 257-262.
[19] K. Petersen, R. Feldt, S. Mujtaba, and M. Mattsson, "Systematic
Mapping Studies in Software Engineering," in Evaluation and
Assessment in Software Engineering (EASE 08) Bari, Italy: BCS, 2008,
pp. 1-10.
[20] D. Sjoberg, T. Dyba, and M. Jorgensen, "The Future of Empirical
Methods in Software Engineering Research," in Future of Software
Engineering (FOSE) Minneapolis, MN: IEEE Computer Society, 2007,
pp. 358-378.
[21] V. Garousi and J. Fernandes, "Highly-cited Papers in Software
Engineering: The top-100," Information and Software Technology, vol.
71, pp. 108-128, 2016.
[22] C. Wohlin and A. Aurum, "Towards a Decision-making Structure for
Selecting a Research Design in Empirical Software Engineering,"
Empirical Software Engineering, vol. 20, pp. 1427-1455, 2015.
[23] C. Wohlin, A. Aurum, L. Angelis, L. Philips, Y. Dietrich, T. Gorschek,
H. Grahn, K. Henningsson, S. Kagstrom, G. Low, P. Rovegard, P.
Tomaszewski, C. Van Toorn, and J. Winter, "The Success Factors
Powering Industry-Academia Collaboration," IEEE Software, vol. 29,
pp. 67-73, 2012.
[24] E. J. Weyuker, "Empirical Software Engineering Research - The Good,
The Bad, The Ugly," in International Symposium on Empirical
Software Engineering and Measurement Banff, AB: IEEE Computer
Society, 2011, pp. 1-9.