Design- Based Research:

A New Research Paradigm

for Open and Distance Learning

Feb, 2007
Terry Anderson, Ph.D.
Canada Research Chair
in Distance Education
 Presentation Overview
 Traditional opening joke
 Need for distance education research
 Sorry state of current research
 Methodological Orientations
 Quantitative
 Qualitative
 Critical
 Design-based
 Dissemination and building a research culture
 Why is Distance Education
Better Than Sex?
• If you get tired, you can stop, save your place and
pick up where you left off.
• You can finish early without feeling guilty.
• You can get rid of any viruses you catch with a $50
program from McAfee
• With a little coffee you can do it all night.
• You don’t usually get divorced if your spouse
interrupts you in the middle of it.
• And If you're not sure what you are doing, you can
always ask your tutor.
 Athabasca University,
Alberta, Canada
Fastest growing university
in Canada
34,000 students
700 courses
Graduate and
* Athabasca University Undergraduate programs
Athabasca University Largest Master of Distance
Education program
Only USA Accredited
University in Canada
 Founded Population Area Degrees Pacing
(sq.kms)

Athabasca 1972 Alberta 661,000Full Arts UGrad

University 3.2 m and Contin-
Canada 9.980,00 Science uous entry
32.8 m 0 MBA
MDE
Open 1971 Israel 22,144 Full Arts Paced
University 6.2 m & Sc.
Israel MBA,
MA,
MSc
45000
40000
35000
30000
25000
Athabasca
20000
OUI
15000
10000
5000
0
1987 1997 2002 2006
 Distance Education Research
 Can you think of at least one major
contribution to practice made by distance
education research in the last decade?
 Why does this particular research result make
a difference?
 Question ;
 context;
 methodology;
 clarity of presentation?
 Typical DE Research
 “Here is what we are doing at my university.
Isn’t it wonderful !!”
 “Here is what we are doing , why don’t you
come and research it?”
 Variable quality by journal, by conference, by
region, by practice.
 Defining Research
 “the systematic study of materials and sources in
order to establish facts and reach new conclusions”
(Oxford Compact Dictionary, 1991).
 adjectives describing research as
 disciplined, organized,
 transparent, problem orientated,
 public, creative,
 scientific, systematic,
 diligent, labourous
 and accessible
 Why Do Research in Distance
Education?
 Many unresolved questions of traditional distance education
 - attrition; F2F tutorial value, paced vs unpaced;
 new forms of distance education provision.
 what combinations of group based learning are worth the cost and
inconvenience?
 Do face to face tutorials really make a difference or is real time
video or audio conferencing just as effective?
 How much does expensive multimedia really enhance student
learning?
 how important are real time interactions compared to asynchronous
ones ?
 Do Web 2.0 and social software tools really encourage new forms of
communities of inquiry?
 Why Educational Research
‘Just Don’t Get No Respect’
 Most research is not valued by funders, other
academics or worse by practitioners
 Not funded financially
 Education 0.01 % of expenditures
 Health 3.0 %
 High tech companies 10.0 - 15.0 %
 Overall (Canada, 2002) 1.88%

Rodney Dangerfield
 Assessment of DE Research
 Many experimental research projects do not
display rigour in their design
 Many generalize inappropriately
 Cultural, linguistic and environmental factors
often not taken into consideration
 Few concerned with teacher and tutor support
 Few studies based on current learning,
pedagogical or psychological theories
 Olugbemiro Jegede (1999)
 A practitioner's perception of
educational research

 “answers are too narrow to be meaningful, too

superficial to be instrumental, too artificial to
be relevant, and, on top of that, they usually
come too late to be of any use.” van den Akker
1999
Barriers to Educational Research
 It’s nobody’s job
 How many in this room have at least 50% of their
job requirement to do research?
 Negligible industry support
 No large scale focus on particular problems
 Nobody keeping score in meaningful terms
 Pervasive lack of trust in research efficacy
 In sum, lack of an effective research culture
 (Burkhardt and Schoenfeld, 2003).
 Who Should Do Research?
 Action Research
 Focused on an authentic problem
 Designed and implemented by participants
 Includes reflection ands with dissemination
 Students as researchers
 Why are students excluded from this rich learning
experience?
 Professional researchers
 DE or educational technology researchers
 Faculty teaching DE within a different academic
perspective
 How do we Build a Culture of
Research in Distance Education?
Research Paradigms
 Quantitative ~ discovery of the laws that govern
behavior

 Qualitative ~ understandings from an insider

perspective

 Critical ~ Investigate and expose the power

relationships

 Design-based ~ interventions, interactions and

their effect in multiple contexts
 Quantitative Paradigm
 Key words like “evidence based” “systematic review”
“scientific research”
 employs a scientific discourse derived from the epistemologies
of positivism and realism
 Long tradition borrowed from the natural sciences
 Since context is so pivotal in education, a great number
of studies must be done to eliminate contextual
variance and combined using meta analysis.
 Inordinate support and faith in randomized controlled
studies
 “those who are seeking the strict way of truth
should not trouble themselves about any object
concerning which they cannot have a certainty
equal to arithmetic or geometrical
demonstration”
 (Rene Descartes,). 1496-1650
The challenge of meeting criteria for
quantitative study
 Control group assignment rarely possible
 Blind assessment not practical
 Contextual variables in natural contexts negate
transfer and replicability
 ‘What works’ in one context, at one time does
not guarantee it will work again
 Interventions are never controlled nor identical
 Educational results must always be interpreted
 Is meta analysis the gold
standard?
 Canadian example:
Quantitative Ex. – Meta-Analysis
 Ungerleider and Burns (2003)
 Systematic review of effectiveness and efficiency
of ICT
 The type of interventions studied were
extraordinary diverse –only criteria was a
comparison group and use of ICT
 “Only 10 of the 25 studies included in the in-
depth review were not seriously flawed, a
sobering statistic given the constraints that went
into selecting them for the review.”
Ungerleider, C., & Burns, T. (2003) . A systematic review of the
effectiveness and efficiency of networked ICT in education . P.38 Ottawa:
Industry Canada. Retireved Jan. 24, 2004 from
http://www.lnt.ca/technology/ict/SystematicReview.pdf
USA Department of Education (2003) guidelines for
Identifying and Implementing Educational Practices
Supported By Rigorous Evidence:
 Quantitative Summary
 Can be useful especially when testing well
established and consistent practice.
 The need to “control” context often makes
results of little value to practicing professionals
 In times of rapid change too early quantitative
testing may mask beneficial positive capacity
 Will we ever be able to afford blind reviewed,
random assignment studies?
 Qualitative Paradigm
 Many different varieties
 Generally answer the question ‘why’ rather
then ‘what’ or ‘how much’?
 Presents special challenges in DE due to
distance between participants and researchers
 Currently most common type of DE research
(Rourke and Szabo, 2002)
 1 Qualitative Example
st

 Dearnley (2003) Student support in

open learning: Sustaining the Process
 Practicing Nurses, weekly F2F tutorial
sessions
 Phenomenological study using
grounded theory discourse
Dearnley (2003)
 “Support structures to facilitate
personal and professional
development within this context need
to be in place and attention must be
given to the provision of effective
learner support.” (Dearnley, 2003)”
Johnson H. (2007). Dialogue and the Construction of
Knowledge in E-Learning: Exploring Students' Perceptions
of Their Learning While Using Blackboard's Asynchronous
Discussion Board. EURODL

 Four different ways of perceiving online

learning were identified
 Practical experience (Rosemary)
 Interconnections (Sarah, Katherine, Cindy)

 Expressing own thoughts (Anthony, David)

 Flexible learning (Larry)

 Qualitative Summary
 Measure of quality is “critical appraisal
concerning plausibility, internal consistency
and fit to prevailing wisdom”
 Burkhardt & Schoenfeld (2003)
 But what if the only producers and consumers
are researchers not practitioners?
 Often the only answer that makes it to the
practice is “it depends”!
 Critical Example
Friesen, N. (submitted). The Experience of Computer Use:
Expert Knowledge and User Know-How. GLIMPSE:
Phenomenology and Media.
 “I will also show how the actual experience of
computer use casts into doubt the educational
efficacy of computers understood as instruments of
cognitive amplification, or simply ‘mindtools.’ ”
 Did my failure to save my work or to properly
address my email message arise from a mismatch in
"system" and "user" models?
 “user knowledge of the system appears as embodied,
performative and emphatically provisional in nature.”

http://learningspaces.org/n/papers/Computer_Use.doc
 Do These Research Paradigms
Meet the Real Needs
of Practicing Distance Educators?
But what type of research has
most effect on practice?
 Kennedy (1999) - teachers rate relevance and
value of results from each of major
paradigms.
 No consistent results – teachers are not a
homogeneous group of consumers but they
do find research of value
 “The studies that teachers found to be most
persuasive, most relevant, and most
influential to their thinking were all studies
that addressed the relationship between
teaching and learning.”
But what type of research has
most effect on Practice?
 “The findings from this study cast doubt on
virtually every argument for the superiority
of any particular research genre, whether the
criterion for superiority is persuasiveness,
relevance, or ability to influence practitioners’
thinking.” Kennedy, (1999)
“In any dispute the intensity of feeling
is inversely proportional to the value
of the stakes at issue -- that is why
academic politics are so bitter.”

Wallace S. Sayre, 1905-1972

quoted in,
"Issawi's Laws of Social Motion" (1973)
 4th Paradigm
Design-Based Research
 Related to engineering and architectural research
 Focuses on the design, construction,
implementation and adoption of a learning
initiative
 Related to ‘Development Research’
 Closest educators have to a “home grown”
research methodology
 Interventionist within a real educational context
 Action Research on Steroids!
 The Contextual Turn
 Postmodern assertion of variability pervasively
induced by uniqueness of any particular context
 Context of:
 Place – virtual, home, classroom, institution etc.
 Actors – many individual differences, temporary and long
lasting
 Culture – including intra cultural heterogeneity
 “Learning, Cognition, Knowing and Context are
irreducibly co-constituted and cannot be treated as
isolated entities or processes” (Barab & Squire, 2004)
 Context creates
Content creates Context

deFigueiredo (2005) Learning Contexts a Blueprint for Research,

 The Complexity Turn
 Increased interest in viewing educational contexts as
complex environments:
 Not possible to precisely forecast or predict behaviour –
explain and promote - but not predict
 Interventions from the outside (teacher interventions) are
conditioned and recursively amplified or extinguished by
contextual variables
 Resulting in emergent behaviours of organisms
 Complexity “at the edge of chaos” provides opportunity for
creativity and necessary change
 See Underwood (2000) complexity theory; Pascale et. al
(2000) Surfing the Edge of Chaos, Bennet, (2004)
Organizational Survival
 Design

Design
Intervention

Evaluation &
Context
Assessment
(Bannan-Ritland, 2003)
4th Paradigm Design Studies
 iterative,

 process focused,
 interventionist,

 collaborative,

 multileveled,

 utility oriented,

 theory driven and generative

 (Shavelson et al, 2003)
Design Based Example 1 - Athabasca
 Tutor Model
 On call 2 hours/week
 Part time, problem with
knowledge of institution
 knowledge of a single
course
 Personal relationship
 Subject matter expertise
 Personal knowledge base
 Call Centre Model
 Advisor on call 40
hours/week
 Full time, steeped in
university environment
 All business curriculum
 Customer relationship
 Refers to academics for
subject matter expertise
 Formal FAQ and data
collection
 

Call Centres:
Answer 80% of student inquiries
Saves over $100,000 /year
 Stage 1: Informed Exploration
 Literature review, theoretical extrapolation and
expert and participant input
 Often an ideal provides a vision and a guide as
well as significant component of the
measuring stick by which the ideal, as
instantiated in actions within a real context, is
measured.
 Stage 1: Informed Exploration
 Review of call centre literature
 Interviews with current tutors and managers
 Data collection on current processes and costs
 Visit to other call centres, especially those in
related but uncompetitive contexts
 Stage 2: Enactment
 Production phase – highly visible
 Need for project management, tracking and
documentation
 Prototype articulation, design and construction
 Designs should be widely circulated and
critiqued
 Stage 2: Enactment
 Design and coded using Lotus Notes
 Project management, data collection on
development problems and costs,
 Pilot testing
 Multiple iterations
 Stage 3 Local Evaluation
 Interviews, focus groups with call centre staff
 Student satisfaction surveys
 Student interviews
 Analysis of transaction logs and FAQ
 Cost analysis
 Interviews with tutors, union
 Stage 3 Local Evaluation
 Multi-methodological evaluation of the
intervention
 Iterative moving from formative to summative
evaluation
Stage 4: Broader Impact Evaluation
 Generate and advance a particular set of theoretical
constructs that transcends the ..contexts in which they were
generated, selected or refined” (Barab & Squire, 2004)
 Use of thick description and qualitative transference
 Work to expand and develop theory
 Tools and conceptual models to understand and adjust both
the context and the intervention
 Value of an intervention lies in its capacity to effect positive
change – not in the scientific significance of the results
 Call for national and international clearinghouse of phase 4
evaluations (Collins et al. 2004)
 Stage Four - Trials in Multiple
Context
 Currently four help desks operating at Athabasca
 Continuing evaluation showing NSD between
perceptions of value by students between tutor and
call centre model
 Increased use of web services decreasing need for
either tutors or call centers ie Am I ready for AU
adaptive testing
 Further research analyzing institutional resistance to
change
 Design-Based Study #2
 A work very much in progress
 Social software solutions for continuous
enrollment courses
 But what type of interaction meets students
needs, is cost effective and is least restrictive
on freedom of both learners and teachers?
 Stage 1: Informed Exploration
 Review of literature on interaction and self directed
study
 Interviews with course developers, faculty in regard
to experience with social interventions
 Telephone interviews with others around the world
involved in continuous enrollment DE programming
 Survey of students in classes with social interventions
 Anderson, Annand and Wark 2005 Having your Cake
and eating it, 2005. Austariasia Journal of
Educational Technology
 Two Solitudes of
Distance Education

Collaborative,
Independent Study
Distance education
1st gen. correspondence
3rd gen. video, audio
2nd gen. telecourses
and computer conf
Type I
Type C
Information
Technology
Communications
Technology
AU Undergrad AU Grad
AU Future ??
 Type S Distance Education

Socially Collaborative,
Independent Study
Enhanced Distance education
1st gen. correspondence
Independent
Learning 3rd gen. video, audio
2nd gen. telecourses
Type S and computer conf
Type I
Social
Technology Type C
Information
Technology
Communications
Technology
AU Undergrad AU Grad
AU Future ??
 Learning Freedom
 Paulsen’s (1993) theory of cooperative freedom:
 Freedom of space
 Freedom of pace

 Freedom of time

 Freedom of media

 Freedom of content

 Freedom of access

 + Freedom of relationship (Anderson)

 But these freedoms are compromised by the
interactive, paced and collaborative form of 3 rd
generation DE that is based on Type C
technologies
 Is the value of the social, paced and
collaborative learning of sufficient magnitude
to justify these restrictions?
 Educational Social Technology
(EST)
 “Networked tools that support and encourage
learning through face-to-face and virtual social
interactions while retaining individual control over
time, space, presence, activity, identity and
community.” (Anderson, 2005)
 Social software are tools that support communication
using the five ‘devices’ of identity, presence,
relationships, conversations and groups. (Butterfield,
2003)
 Stage 1 Research questions
 What tools, learning activities and incentives
are needed to allow students in learner-paced,
continuous enrollment courses to provide
social, cognitive and teaching presence to each
other?
 Survey results
78% of self-paced student respondents indicated that they
would interact with other students as long as they were
able to proceed through the course at their own pace.
95% of student respondents reported a desire to access the
work of other students either currently or previously
enrolled in the courses.
Preferred Mode of that interaction:
• 70% preferred asynchronous media like email and
computer conferencing,
• 27% preferred a combination of synchronous and
asynchronous technologies
• only 3% preferred synchronous interaction alone (for
example, audio conferences or face-to-face interaction).
 Stage 1 summary
 Interviews and survey of students
 Review of the interaction literature
 Investigate social software tools
 Develop research questions and methodology
 Stage 2: Enactment
 Select and install an instance of elgg
Me2u@athabascau.ca
 Develop support strategies and documentation
 Pilot testing (faculty, undergraduate and
graduate classes)
 Develop new instructional designs
 Successive iterations
 Stage 2: Enactment
 Social software typically supports:
 Individual and group Wiki, Blog, Events,
Tasks, Polls.
 User profiles.

 Friend of friend

 Interest sharing, group formation and access

control
 RSS

 Connections on and off line

http://elgg.net
 Technologies of MDE 663 Fall 2006

M2U.Athabascau.ca Moodle

Blogging Content
Connections Admin
Asynchronous Int.

Portal
Products

Learning Objects
CMAP
Elluminate Furl
Real Time
Pacing Dissemination
Social Presence Knowledge Polling
 Stage 3 Local Evaluation
 Ethics clearance and resolving privacy issues
 Interviews, focus groups with developers and
faculty
 Student satisfaction surveys
 Student interviews
 Analysis of transaction logs
 Cost analysis using completion rate data
 Usefulness over 8 Educ Functions
BookMarks

Profiles

Web Conf

Cmap
Usefulness
RSS

Moodle Discussion

Blogs

Email

0 1 2 3 4 5

N= 9 of 13
 Supporting Social and
Feeling Connected
P ro files

Web Co nf

R SS
U sefulness
D iscussio n

B lo g s

E m ail

0 1 2 3 4 5 6

N= 9 of 13
 Mastering Knowledge Objectives
B o o kM arks

P ro files

Web Co nf

Cm ap
Kno w ledg e
R SS

D iscussio n

E m ail

B lo g s

0 1 2 3 4 5

N= 9 of 13
 Stage 1-3 Iterations
 Adding functionality
 Testing new designs and learning
activities
 Reacting to Organizational Change
 Think staff development and redefinition of
learning roles
 Think decentralization – students
developing unanticipated use of system
 Stage 4: Broader Impact
Evaluation & Theorizing
 Publishing of results
 Trials in different contexts
 Use in paced and unpaced courses
 Use in different disciplines
 Collaboration with international ELGG groups
 Synthesis of application in different contexts
 Adoption of Innovation framework
 Theorizing: Does Educational Social Software allow
scaleable, high quality, learning while maximizing
learner freedoms?
Design-based Research: Conclusion
 Methodology developed by educators for
educators
 Developed from American pragmatism – Dewey
(Anderson, 2005)
 Recent Theme Issues:
 The Journal of the Instructional Sciences, (13, 1, 2004),
 Educational Researcher (32, 1, 2003) and
 Educational Psychologist (39, 4, 2004)
 See bibliography at
http://cider.athabascau.ca/CIDERSIGs/DesignBase
dSIG/
 My article at www.cjlt.ca/abstracts.html
 Building the Research Culture
 Better dissemination within a combined
research/practitioner communities
 Better tools
 More funding; less fighting
 Online Journals
 International Review of Research on Open and
Distance Learning www.irrodl.org
CIDER.ATHABASCAU.CA
 A Tale of 3 books

Open Access Commercial publisher

E-Learning for the
72,000 downloads plus 934 copies sold at $52.00 21st Century
indiv. chapters Commercial Pub.
Buy at Amazon!!
1200 sold @
400 hardcopies sold @ $50.00 $135.00
Free at 2,000 copies in
cde.athabascau.ca/online_book Arabic Translation
@ $8.
 Conclusion
• DE Research is grossly under-resourced to meet
the magnitude of opportunity and demand.
• Paradigm wars are unproductive.
• Design-based research offers a promising new
research design model.
• Web 2.0 enhanced interaction (multiformat,
synch, asynch, immersive environments) may
offer key to more effective knowledge growth
and exchange.
 Conclusion
“Wisdom is fed equally from practical, theoretical,
and ethical knowledge and from a large quantity
of reflected experience”

Otto Peters, 2003 p. 137

Your Comments or Questions
Most Welcomed !

Terry Anderson
terrya@athabascau.ca
 Research “philosophy for
professionals” Ulrich 2006
reflective competence is:
•self-critical: the effort of systematically
examining one’s own premises through self-
reflection and dialogue, with a view to carefully
qualifying the meaning and validity of one’s
claims;
•emancipatory: working actively to help others
in emancipating themselves from one’s claims,
as well as from theirs; and
•ethically alert: making transparent to oneself
and to others the value implications of one’s
claims, and limiting these claims accordingly.