Research question: How does software development methodology and team size

influences developers productivity?

Design research

Threats to the research are related to operationalization and

measurement issues:
Operationalization issues The validity of the operationalization
Measurement issues Reliability, validity, sensitivity (see below)

Levels
(observed variables)

Independent
variables

Dependent
variables

(latent variables)

(latent variables)

Measures

Valid but
not reliable
Reliability threats

Valid and
reliable

Reliable but
not valid
Number of
developers

refers to the question whether the research can be repeated

with the same results.

Stability reliability Does the measurement vary over time?
Representative reliability Does the measurement give the same answer when
applied to all groups?
Equivalence reliability When there are many measures of the same construct, do
they all give the same answer?

Development
team size

All other variables which

are not the focus of
research are irrelevant
variables.

Validity threats
Face validity Research community good feel.
Content validity Are all aspect of the conceptual variable included in the
measurement?
Criterion validity validity is measured against some other standard or measure for
the conceptual variable.
Predictive validity The measure is known to predict future behavior that is related
to the conceptual variable.
Construct validity A measure is found to give correct predictions in multiple
unrelated research processes. This confirm both the theories and the construct validity
of the measure.
Conclusion validity is concerned with the relationship between the treatment and
the outcome of the research 8choice of sample size, choice of statistical tests).
Experimental validity (see reliability)

H1

Developer
productivity

H2
Measurement relationship
associate latent variables
with their measures
Causal relationships (H1,H2) define
cause-effect relationship between
latent variables (theoretical
propositions). Can be tested only by
evaluating relationships between
observed variables (hypotheses)!

Represent
the effect

Represent
the cause

Measurement issues

Internal Is concerned with the validity within the given environment and the
reliability of results. It relates to validity of research process design, controls and
measures.
External Is the question of how general the findings are. Can you carry over the
research results into actual environment?

Data is collected with a

research instrument, for
example questionnaire.

Collect data

Describe abstract
theoretical concepts.
They cannot be
directly measured.

Developer
efficiency

Requirements
change

{OSSD, RUP,
XP}

Development
team size

Number of
developers

Latent

Observed

Software
reliability

LOC

Analyze data

Mean time
between failure

Define ways of measuring

latent variables.
Each latent variable may
have multiple empirical
indicators.

How much does the measurement change with the change of the conceptual variable?

Qualitative
data

Use qualitative
data analysis

The objective of this

activity is to analyze
the collected data in
order to answer the
operationalized study
goal (research
question).

Quantitative
data

Chose data
analysis

Depends on data
and the goal of
the study.

Draw
conclusions

Consider reliability,
validity and sensitivity!

Consider
threats

Test

Real world

Toy world Laboratory

Simplify

Analyze <Object(s) of study> (what is studied/

observed?)
for the purpose of <Purpose> (what is the intention?)
with respect to their <Quality focus> (which effect is
studied?)
from the point of view of the <Perspective> (whose
view?)
in the context of <Context> (where is the study
conducted?).

Logic of sampling

Results are generalized

to population

Sample
frame

Sampling
process

Sample

Random sample: a sample in which a researcher

uses a random sampling process so that each
sampling element in the population will have an
equal probability of being selected.

A list of cases in a
population or the best
approximation of it.

Experiment design:
Define the set of tests
How many tests (to make effects visible)
Link the design to the hypothesis, measurement scales and statistics
Randomize, block(a construct that probably has an effect on response) and balance(equal number of subjects)
Random
assignment

Pretest

Posttest

Control
group

Classical

Yes

Yes

Yes

Yes

Yes

One shot case study

No

No

Yes

No

Yes

One group pretest posttest

No

Yes

Yes

No

Yes

Static group comparison

No

No

Yes

Yes

Yes

Two group posttest only

Yes

No

Yes

Yes

Yes

Time series design

No

Yes

Yes

No

Yes

Experiment design

Experimental
group

Hypothesis
formulation:
Hypoth. statement
H0: positive
Ha: One or two tailed

Design
notation
o x o
R o
o
x o
o

x
x
x

o
o
o
o
o
x

Don't be afraid to talk over

ideas with others!

Construct validity

External validity

External validity

Experimental validity or
reliability

Experimental validity or
reliability
Construct validity

Experiment design
notation:

Case research design:

Investigate a phenomenon in depth, get close to the
phenomenon, provide a rich description and reveal
its deep structure.

Enable the analysis of data across cases, which

enable the researcher to verify that findings are not
the result of idiosyncrasies of the research setting.
Cross case comparison allows the researcher to use
literal or theoretical replication.

In exploratory case studies, fieldwork,

and data collection may be undertaken
prior to definition of the research
questions and hypotheses.
Explanatory cases are suitable for doing
causal studies. In very complex and
multivariate cases, the analysis can make
use of pattern-matching techniques.
Descriptive cases require that the
investigator begin with a descriptive
theory, or face the possibility that
problems will occur during the project.

Case research objectives:

Discovery and induction:
Discovery is the description and conceptualization of the phenomena.
Conceptualization is achieved by generating hypotheses and developing
explanations for observed relationships
Statements about relationships provide the basis for the building of theory.

X = Treatment
O = Observation
R = Random assignment
o

Testing and deduction:

Testing is concerned with validating or disconfirming existing theory.
Deduction is a means of testing theory according to the natural science
model

Case study research

design components:
A study's question.
Its propositions, if any.
The unit of analysis.
The logic linking the
data to propositions.
The criteria for
interpreting the findings.

External validity

Repeated
ANOVA
ANOVA

Nom

3+
2

Int

We got an answer to
stated research question.
IV = Independent Variable
DV = Dependent Variable

3+

Linear
regression
The IV is the
variable that
defines
conditions

I nt +
I nt
Or
d
Or +O
d+ r d
In
t

S
The objective of this activity is to
report the study and its results so
that external parties are able to
understand the results in their
contexts as well as replicate the
study in a different context.

End empirical research

Varieties of case study research:

Case studies can be carried out by
taking a positivist or interpretivist
approach.
Can be deductive and inductive.
Can use qualitative or quantitative
methods.
Can investigate one or multiple cases.

Internal validity

Construct validity

Measures of central tendency

A measure of central tendency is a single number that is used to represent the average score in
the distribution.
Mode the most common score in a frequency distribution
Median the middlemost score in a distribution
Mean the common average
Measures of variability
A single number which describes how much the data vary in the distribution.
Range The difference between the highest and lower score in a distribution.
Variance The average of the squared deviations from the mean.
Standard deviation the square root of the variance, a measure of variability in the
same units as the scores being described.
Correlation and regression
Determine associations between two variables.
Correlation The strength of the relationship between two variables.
Regression Predicting the value of one variable from another based on the
correlation.

Inferential statistics or statistical induction comprises the use of statistics to make inferences
concerning some unknown aspect of a population.

Sampling distribution the distribution of means of samples from a population.

Sampling distribution has three important properties:
It has the same mean as the population distribution.
It has smaller standard deviation as the population distribution.
As the sample size becomes larger, the shape of the distribution approaches a normal
distribution, regardless of the shape of the population from which the samples are drawn.
Hypothesis testing - is the use of statistics to determine the probability that a given hypothesis
is true. The usual process of hypothesis testing consists of four steps.
Formulate the null hypothesis H0 (the hypothesis that is of no scientific interest) and the
alternative hypothesis Ha (statistical term for the research hypothesis).
Identify a test statistic that can be used to assess the truth of the null hypothesis.
Compute the P-value, which is the probability that a test statistic at least as significant as
the one observed would be obtained assuming that the null hypothesis were true. The
smaller the P-value, the stronger the evidence against the null hypothesis.
Compare the p-value to an acceptable significance value alpha (sometimes called an alpha
value). If p<=alpha, that the observed effect is statistically significant, the null hypothesis is
ruled out, and the alternative hypothesis is valid.
Statistical errors

The null
hypothesis
(H0)

H0 is TRUE

H0 is FALSE

Accept H0

Correct decision

Wrong decision
Type II error

Reject H0

Wrong decision
Type I error

Survey Process:
Study definition determining the goal of a survey.
Design operationalizing of the study goals into a set of questions (see theoretical model)
Execution the actual data collection and data processing.
Analysis interpretation of the data.
Packaging reporting about the survey results.

Ord

Here is the
distribution of
values of Z when
the hypothesis
tested is true.
(mean Z = 0)

Correct decision

Alpha is the probability

of rejecting the
hypothesis tested when
that hypothesis is true.
Here we have
set alpha = 0.05

Friedman

Int

Paired
(related) ttest

Related
Ord

Int

Wilcoxon
Matched
Pairs
One way
ANOVA

Indep.
Kruskal Wallis
3+

Ord
Independent
t-test
Int

A survey is a study by asking (a group of) people from a population about their opinion on specific
issue with the intention to define relationships outcomes on this issue.

Implementation operationalisation of the design so that the survey will be executable.

A smaller set of cases a
researcher selects from a
larger pool and generalizes
to the population

Population

Variables selection:
Independent and dependent variables
Observed variables
Measurement scales (nominal, ordinal, interval, ratio)
Selection of subjects:
Profile description
Quantity
Separation criteria

Case method facts:

Does not explicitly control or manipulate variables.
Studies a phenomenon in its natural context.
Makes use of qualitative tools and techniques for data
collection and analysis.
Case study research can be used in a number of different
ways.
Can be used for description, discovery and theory testing.

Context selection:
Online vs. Offline
Student vs. Professional
Specific vs. general

Single
case

ify

Multiple
case

Reality

Abstract

/ Fa
ls

Internal validity

Internal validity

om

port

Comparing different populations

among respondents, association
and trend analysis, consistency of
scores.

Conclusion validity

S up

Literature survey.

Compare case study results

to a representative
comparison baseline: sister
project, company baseline,
project subset with no
change.

Descriptive statistics are used to describe the basic features of the data in a study. They provide
simple summaries about the sample and the measures. Together with simple graphics analysis,
they form the basis of virtually every quantitative analysis of data.

Statistical significance the probability that an experimental result happened by chance.

Consider sources of
invalidity (internal,
external)

Disseminate
results

Questionnaires.
Interviews.
Project measurement.

Interviews
Parametric and
nonparametric
statistics, compare central
tendencies of treatments,
groups.

+
om

Mind

Operationalization

Quantitative (Hard numbers)

Easier to implement and collect data. Tick boxes.
Easier to make comparisons over time and between
places.
Can be a quick fix when organizations need
performance data to justify project investment.
Easier to process through a computer.
Easier for other stakeholders to examine and
comprehend.
Trends and patterns easier to identify.
Can distort the evaluation process as we measure
what is easy to measure.
Can lead to simplistic judgments and the wider
more complex picture is ignored.

Questionnaires.

Qualitative (Judgments)
Tends to be the poor relation.
Problems of opinion and perception when
making the judgment.
The data collected is more likely to create
differences of opinion over interpretation.
Not easily measurable.
As the benefits are longer term, they can be
outweighed by shorter term costs.
Can lead to inconsistent assessments of
performance between places over time and
between project elements.
Subjective opinions tend to be given less
status than quantitative ones.

World of propositions

Process and product

measurement.
Questionnaires.

+
om er
N th
O

World of theory

Process and product

measurement.

May rely on different

projects/organizations
keeping comparable data.
No control over variables
methods.
Can at most confirm
association but not causality.
Can be biased due to
differences between
respondents and
nonrespondents.
Questionnaire design may be
tricky (validity, reliability).

Use
quantitative
data analysis

Conclusions can be drawn

statistically or analytically.
Positivism is a
philosophy that
states that the
only authentic
knowledge is
scientific
knowledge, and
that such
knowledge can
only come from
positive
affirmation of
theories through
strict scientific
method.

Can confirm an effect

generalizes to many
projects/organizations.
Allow to use standard
statistical techniques.
Enable research in the
large.
Applicable to real world
projects in practice.
Generalization usually
easier.
Good for early exploratory
analysis.

With little or no
replication they may
give inaccurate results.
Difficult to interpret and
generalize (e.g., due to
confounding factors).
Statistical analysis
usually not possible.
Few agreed standards
on procedures for
undertaking case
studies.

The objective of this activity

is to run the study according
to the study plan.

Perform
research

General sable population

population to which you want to
ultimately generalize results.

Application in
industrial context
requires
Compromises.

rval

Sensitivity

Perform research on defined

sample

Can use existing experience.

I nt e

Reliability - does the measurement give

the same results under the same
conditions (consistency)?
Validity - does the measurement method
actually provide information about the
conceptual variable?
Sensitivity - how much does the
measurement change with the changes on
the conceptual variable?

Sources of invalidity

Design
survey

Design case
study

Can be incorporated in
normal development
activities.
Already scaled up to
life size if performed on
real projects.
Can determine whether
expected effects apply
in studied context.
Easy to plan.
Help answer why and
how questions.
Can provide qualitative
Insights.

Practitioner oriented methods

Delphi method
Action research
Laboratory oriented methods
Mathematical modeling
Computer simulation
Laboratory experiment
Technology oriented methods
Proof of technical concept
Literature based methods
Literature review
Conceptual study

Actual sample the sample

actually used in research.

Hypothesis testing
Hypotheses are tested by comparing
predictions with observed data
Observations that confirm a prediction do
not establish the truth of a hypothesis
Deductive testing of hypotheses look for
disconfirming evidence to falsify
hypotheses

Design
experiment

Other research
methods

Accessible population
population that you can actually
gain access.

Dependent

XP

Lines of code (LOC)

per developer per day

Independent

RUP

...

(observed variables)

OSSD
Software
development
methodology

Select
research
method

Can establish causal

relationships.
Can confirm theories.

Theoretical model is based on research question and represents set of concepts and
relationships between them!

Purpose

Theoretical model is used to

conceptualize the problem
stated in research question. It
is commonly represented with
causal model.

Pearson correlation
Linear regression

Create
theoretical
model

Spearman correlation
Linear regression

The goal of literature review is

to demonstrate a familiarity
with a body of knowledge and
establish credibility. Additional,
to show the path to prior
research and how a current
research is related to it.

In

What are the key success factors of object-oriented frameworks?

Does the proposed software improvement increases the efficiency of its users?
How does software development methodology and team size influence developers
productivity?

Review
Literature

Chi-squared
cross tabulation

Event

Research question examples:

Types of research questions are:

What?, Where?, Who?, When?,
How Much?, How many?, Why?,
How?

A good literature review is an

important part of any research.

Technology change is
implemented across a large
number of projects.
Description of results,
influence factors,
differences and
commonalities is needed.

One sample
t-test

Confidence in
theory is reduced

Theory is modified

Statistical methods can be used to summarize or describe a collection of data; this is called
descriptive statistics. In addition, patterns in the data may be modeled in a way that accounts for
randomness and uncertainty in the observations, and then used to draw inferences about the process
or population being studied; this is called inferential statistics. Both descriptive and inferential
statistics comprise applied statistics.

Requires low control

Change to be assessed
(e.g., new technology)
is wide-ranging
throughout the
development process.
Assessment in a typical
situation required.

va
l

Prediction is
NOT confirmed

Requires medium control

Control on who is using

which technology,
when, where, and
under which conditions
is possible.
To investigate self
standing tasks from
which results can be
obtained immediately.

t er

Laws are
formed

It is used to find out what is

already known about a question
before trying to answer it.

Requires high control

Control

In

Relationships
between
constructs
are identified

A clear and precise RQ guides

theory development, research
design, data collection and data
analysis.

Major
threats

Level of access

Theory is
NOT modified

Define research
question

When
appropriate

Event

Research is
performed

Research Question (RQ) is what

you are trying to find out by
undertaking the research
process.

Investigate information collected

from a group of people, projects,
organizations or literature.

Investigate a typical
case in realistic
representative conditions.

Pro's

Theory is
formed that
explains laws

The goal of the theory has to

be defined here. Based on the
research question an empirical
strategy has to be chosen.

Establishes causal
relationships, confirm
theories.

Con's

Predictions
from theory
can be
drawn, which
form
hypotheses

Event

Theory is
rejected

Data
collection

Prediction is
confirmed

Analysis
types

Laws are
formed

Statistics is a mathematical science pertaining to the collection, analysis, interpretation or

explanation, and presentation of data.

Nominal

Event

Research is a systematic
process for answering
questions to solve
problems and create new
knowledge.

Nominal

Can be pretty sure you are

getting at the correct
issues
But can you draw valid
conclusions?

Relationships
between
constructs
are identified

Start empirical
research

Empirical research is a
research approach in which
empirical observations (data)
are collected to answer
research question.

Discriminant of
Logistic Regression

Level of control

Explanatory
Why
research
How
How many
How much
Descriptive
research
When
Who
Where
Exploratory
research
What

Confidence
in theory is
increased

Chi-squared
Goodness of Fit

Can be pretty sure your

conclusions are valid ...
But are you getting at the
correct issues?

Data analysis
Coding scheme
(for open
question)
Data entry
Checking
Resolve
incomplete data
Statistical testing
of results

Ord

-3

-2

-1

Values of Zx

The Z score for an item, indicates how far and in what direction, that item deviates
from its distribution's mean, expressed in units of its distribution's standard
deviation.
Power is the probability
of rejecting the
hypothesis tested when
the alternative
hypothesis is true.

Here is the distribution of values of Z when a

particular alternative hypothesis is true
(mean Z = 1)
Beta is the probability of accepting
the hypothesis tested when the
alternative hypothesis is true.
-3

-2

-1

Beta = 0.74
Power = 0.26

Values of Zx

Mann
Whitney U

The critical value

of Z = 1.65

Bernd Freimut, Teade Punter, Stefan Biffl, & Marcus Ciolkowski 2002, State-of-the-Art in
Empirical Studies, Virtuelles Software Engineering Kompetenz-zentrum.
Johnston, R. & Shanks, G. Research Methods in Information Systems. 2003.
Neuman, W. L. 2005, Social research methods : qualitative and quantitative approaches,
5th ed. edn.
Winston Tellis 1997, "Introduction to Case Study", The Qualitative Report, vol. 3, no. 2.
www.wikipedia.org

Types of survey
Descriptive surveys are frequently conducted to enable descriptive assertions about
some population, i.e., discovering the distribution of certain features or attributes. The
concern is not about why the observed distribution exists, but instead what that
distribution is.
Explanatory surveys aim at making explanatory claims about the population. For
example, when studying how developers use a certain inspection technique, we might want
to explain why some developers prefer one technique while others prefer another. By
examining the relationships between different candidate techniques and several
explanatory variables, we may try to explain why developers choose one of the techniques.
Explorative surveys are used as a pre-study to a more thorough investigation to assure
that important issues are not foreseen. This could be done by creating a loosely structured
questionnaire and letting a sample from the population answer to it. The information is
gathered and analyzed, and the results are used to improve the full investigation. In other
words, the explorative survey does not answer the basic research question, but it may
provide new possibilities that could be analyzed and should therefore be followed up in the
more focused or thorough survey.

Question types
Open
Closed

Reporting response
rate
Total sample
selected
Number located
Number
contacted
Number returned
Number complete

This poster is licensed as Creative Commons Attribution-Noncommercial-No Derivative

Works 2.5 Slovenia License
Authors:
Gregor Polani
Email: info@itposter.net
University of Maribor
Faculty of Electrical Engineering and Computer Science
Institute of Informatics
Poster version: 0.6 (DRAFT)
http://researchmethods.itposter.net