Research Methods

1
This session.

Why do you need this.

Online resources
Literature reviews
The nature of research
Questionnaire design
Structured investigations

2
Why do you need this?

To understand content of directed reading within taught

modules
To be able to write essays or reviews of published
literature
To be able to plan and conduct your project where there is
an element of research in the investigation or the
evaluation of what you have done

3
Library student study skills books

http://www.library.dmu.ac.uk/Skills/Study/
Collection of self study books relevant to research and
projects
Most available in .pdf or .html format (free download)
Series includes
Research Methods
Designing a questionnaire
Information Citation and Control
Dissertation preparation and presentation
Research interviews
Planning a sample survey
Thinking it through: an introduction to critical thinking

4
MSc projects

All projects are required to have at least one academic

objective, usually an investigation of literature relevant to
some aspect of your project
Most projects will include some form of evaluation
E.g the production of a computer-based tool to teach letters of the
alphabet to nursery level children
A piece of software which purports to do this will be of limited
value without some evidence of how effective it is in doing so.
Demonstrating its effectiveness requires a structured study

De Montfort University, 2001 5

Project Skills

There is a non-assessed module in the second semester,

which covers
Personal time management
Project management
Project Selection
Written and verbal communication
Giving demonstrations
Learning Strategies

De Montfort University, 2001 6

Literature reviews

Conducted to ensure a researcher is familiar with all of

the what is known about a particular field
Often published in order to bring other researchers (and
MSc students) up to speed quickly in an unfamiliar field
Need to have the scope of the review carefully defined
Not too big such that adequate coverage is infeasible, and there is
too much literature to review and the review becomes unfocussed
(and thereby not useful)
Not too narrow such that there are too few papers to include

De Montfort University, 2001 7

Form of a literature review

Requires reading literature from a variety of sources

Forming some form of taxonomy or structure for your
review
Identifying where in your taxonomy the various
contributions from the literature fall
Critically reviewing the literature
Identifying different approaches, contradictions between
contributions, analysis of strengths and weaknesses
Not simply pasting quotes from different papers
Drawing your own conclusions, particularly concerning
completeness of coverage
Highlight implications for your work (if appropriate)
De Montfort University, 2001 8
Example of a literature review

Hand, C. "A Survey of 3D Interaction Techniques". Computer Graphics

Forum, 16(5): 269-281. (Dec 1997)
(Abstract)
Recent gains in the performance of 3D graphics hardware and rendering
systems have not been matched by a corresponding improvement
in our knowledge of how to interact with the virtual environments we
create; therefore there is a need to examine these further if we are to
improve the overall quality of our interactive 3D systems. This paper
examines some of the interaction techniques which have been
developed for object manipulation, navigation and application control
in 3D virtual environments. The use of both mouse-based techniques
and 3D input devices is considered, along with the role of feedback
and some aspects of tools and widgets.

De Montfort University, 2001 9

Example of a literature review

Hand, C. "A Survey of 3D Interaction Techniques". Computer Graphics

Forum, 16(5): 269-281. (Dec 1997)
(Abstract) rationale
Recent gains in the performance of 3D graphics hardware and rendering
systems have not been matched by a corresponding improvement
in our knowledge of how to interact with the virtual environments we
create; therefore there is a need to examine these further if we are to
improve the overall quality of our interactive 3D systems. This paper
examines some of the interaction techniques which have been
developed for object manipulation, navigation and application control
in 3D virtual environments. The use of both mouse-based techniques
and 3D input devices is considered, along with the role of feedback
and some aspects of tools and widgets.

De Montfort University, 2001 10

Example of a literature review

Hand, C. "A Survey of 3D Interaction Techniques". Computer Graphics

Forum, 16(5): 269-281. (Dec 1997)
(Abstract) Scope of review
Recent gains in the performance of 3D graphics hardware and rendering
systems have not been matched by a corresponding improvement in
our knowledge of how to interact with the virtual environments we
create; therefore there is a need to examine these further if we are to
improve the overall quality of our interactive 3D systems. This paper
examines some of the interaction techniques which have been
developed for object manipulation, navigation and application control
in 3D virtual environments. The use of both mouse-based techniques
and 3D input devices is considered, along with the role of feedback
and some aspects of tools and widgets.

De Montfort University, 2001 11

Example of a literature review

Hand, C. "A Survey of 3D Interaction Techniques". Computer Graphics

Forum, 16(5): 269-281. (Dec 1997)
(Abstract) Indication of taxonomy
Recent gains in the performance of 3D graphics hardware and rendering
systems have not been matched by a corresponding improvement in
our knowledge of how to interact with the virtual environments we
create; therefore there is a need to examine these further if we are to
improve the overall quality of our interactive 3D systems. This paper
examines some of the interaction techniques which have been
developed for object manipulation, navigation and application control
in 3D virtual environments. The use of both mouse-based techniques
and 3D input devices is considered, along with the role of feedback
and some aspects of tools and widgets.

De Montfort University, 2001 12

Sources of information

Scientific journals
Conference proceedings (refereed and unrefereed)
Magazines, newspapers
WWW

Important to differentiate between reported investigations

and reported opinion
Conclusions of any investigation contain some element of
informed opinion or judgement
Sources of information must be cited appropriately

De Montfort University, 2001 13

Project Academic Objectives

For example.. Review methods of usability evaluation

appropriate to the evaluation of internet-based groupware
systems for use by students undertaking a course by
distance learning
Requires review of methods described in the literature, using the
original sources (not just someone elses review)
Implications of problems/needs associated with evaluation of
groupware systems
Comparisons and contrasts between methods in light of identified
needs of groupware applications
Your conclusions about which method(s) to use and how

De Montfort University, 2001 14

How to do it

Collect and read current papers and reviews of a field

From the list of references cited in these, get copies of
those which appear relevant
Start to classify the papers you collect in one or several
ways to form the basis of a taxonomy
Identify authors who are prominent and check what else
they have published (WWW useful here)
Read contents of recent relevant journals in library
Subscribe to mailing lists for coming conferences and look
at contributors
search on-line bibliographic archives (e.g bids.ac.uk)

De Montfort University, 2001 15

Constructively criticising research..

Researchers have a vested interest in making a piece of

work appear significant and worthy of publication
They may not be as forthcoming about the limitations of
the work as about the benefits
Results should be reported in sufficient detail to enable the
reader to draw his/her own conclusions and thereby judge
the validity of the conclusions drawn by the author
Are the conclusions drawn justified by the evidence
provided?
Is the method of investigation appropriate or flawed?
Are the investigators aware of other similar contributions
in the field?
De Montfort University, 2001 16
What is research?

A systematic enquiry, which is reported in a form that

allows the research methods and outcomes to be
accessible to others
Concerned with seeking solutions to problems or answers
to meaningful questions
Meaningful questions are expressed in a way that indicates
what you will accept as an answer
Non-meaningful (in research terms) questions are not
answerable as a result of enquiry alone (eg judgemental or
metaphysical questions)
Positivism versus phenomalism
De Montfort University, 2001 17
Nature of research - positivist

Deals with positive facts and observable phenomena

Subscribes to the scientific method
Primary goal is not only description but prediction and
explanation
Classification of substances and events, and observation of
these, provide the basis for descriptive laws based on
consistencies in patterns and properties
Characterised by absolute or varying degree of
generalisability
Quantitative, as it draws on measurable evidence

De Montfort University, 2001 18

Postulates in Positivist Research

postulate of natural kinds: all instances of classes and

categories of phenomena exhibit the same properties
postulate of constancy: all phenomena remain the same or
change only very slowly over time
postulate of determinism: there is orderliness and
regularity in nature, constancy in terms of cause and effect

De Montfort University, 2001 19

Nature of research - phenomenalist

Considers that each phenomena is unique and is controlled

by variables such as time, location and culture
No two situations are identical
No reliance on postulates of natural kinds, constancy or
determinism
Essentially subjective, where the content of research and
the way it is pursued is indicative of researchers intention
Outcomes are descriptions which are expressed in
narrative and mainly in qualitative terms

De Montfort University, 2001 20

Stockholm archipelago

De Montfort University, 2001 21

Example of both approaches

Can the study of critical incidents (as opposed to accidents) in

marine navigation in the Stockholm archipelago provide
the basis for improvements in sea safety in the area?

Positivist approach: collect data via interview, classify

types of incidents, produce analyses, make
recommendations based on analysis

Phenominalist approach: analyse interviews in depth, seek

to draw conclusions about causal factors

De Montfort University, 2001 22

Positivist research methods include...

Descriptive research
Anything that is variable, varies to a defined degree, and
thus can be measured
Surveys, case studies, causal comparative studies,
correlational studies, developmental studies, trend studies
Experimental research
Deliberate manipulation of certain factors under highly
controlled conditions
Purpose is to identify causal connections through keeping
the levels of some variables constant and manipulating
others

De Montfort University, 2001 23

Questionnaire design

Relevant to requirements gathering for, and evaluation of,

projects
Self study pack by Arthur Rothwell covers:
Planning and logistics of questionnaire design
Layout issues
Forms of questions
Contents of questions

De Montfort University, 2001 24

Planning and logistics of questionnaire
design
Quantitative or qualitative?
Legal requirements: the Data Protection Act
Confidentiality and anonymity
Sample size
Volunteer respondents
Identifying subject areas
Determining appropriate length
Typical time scale
Main components of questionnaires

De Montfort University, 2001 25

Forms of question

Open and closed questions

Dichotomous items
Scaled items
Mid-point or no mid-point?
Ranking to show preference

De Montfort University, 2001 26

Content of items

Avoiding response set

Components of attitudes
Common types of faulty items
leading questions
context effects
double barelled questions
vague and ambiguous terminology
hidden assumptions
social desirability

De Montfort University, 2001 27

Leading questions and context effects

Would you agree that the governments policies on health are

unfair?
Item wordings should not contain value judgements

How many pints of beer did you drink last night?

Think how the context of the study would affect the
response, say in a
survey of young peoples life styles
survey of health behaviour and heart disease

De Montfort University, 2001 28

Double barreled questions

Do you believe the training programme was a good one and

effective in teaching you new skills?
avoid questions that involve multiple premises

De Montfort University, 2001 29

Vague and ambiguous terminology

How often do you clean your teeth?

Frequently
often
infrequently
never

what does frequently mean?

Give quantifiers to ensure all respondents understand the
same thing by the response categories

De Montfort University, 2001 30

Hidden assumptions, social desirability

When did you last borrow a video tape?

Avoid hidden assumptions - what are these?

Do you ever give to charity?

May lead to a positive response as otherwise something
negative about the respondent is being conveyed

De Montfort University, 2001 31

Structured Investigations - 1

An MSc project has produced a computer-based tool to

teach letters of the alphabet to nursery level children

Talk to the person next to you and discuss how you would
demonstrate the effectiveness of the tool through some
form of structured investigation

Consider the resources required and the feasibility of

conducting the study within a 3-month project

De Montfort University, 2001 32

Ways of approaching this.

You need access to people with expertise and, hopefully, a

group of nursery children
The product needs to be tested, and has to be ready in time
for this to happen
Subjective assessment by teachers comparing the tool with
Other computer based tools
Paper-based methods
Objective approaches
Before and after letter recognition tests with two matched groups
of children
Use of non-parametric statistical techniques

De Montfort University, 2001 33

Structured Investigations - 2

A project has produced a web-based presentation of a

college offering educational courses both as a 2D
collection of pictures of the building, and as a 3D virtual
building which you can walk through

How would you compare the relative efficiencies of each

approach?

Again, discuss with the person next to you

What you are trying to assess
How to carry out the assessment
De Montfort University, 2001 34
Ways of approaching this.

What to assess?
How much information a visitor to the site obtains with each type
of presentation
Usability issues e.g ease of navigation through the
representations of the college building
How to assess this?
Unstructured user trials, where the amount of building explored is
recorded
Structured user trials, where a task is given to find pieces of
information
Subjective assessment of visit by participant
Observed usability problems

De Montfort University, 2001 35