Note1 Advance 21st Century E-Learning - Soft

Executive Summary
1. Topic. This project investigated the principles for designing and developing advanced e-
Learning systems that employ a learner centered pedagogical approach, in contrast to the more
common delivery centered approach found in the majority of the e-Learning systems that are
developed by Korean e-Learning companies and organizations.
2. Research Purpose and Needs. A central argument in this report is that in order to advance
the overall quality of e-Learning (and learning and training more generally) in the Republic of
Korea, it will be necessary to identify major types and characteristics of advanced e-Learning
systems and the principles for developing and implementing these systems. The main goal of this
project is to provide information about the characteristics and design principles of different
types of advanced e-Learning systems that implement learner-centered approaches and that have
been found to foster cognitively and socially robust levels of student interaction and learning.
3. Research Content. Two main tasks were undertaken in this project. First, a comprehensive
literature review was conducted of advanced e-Learning projects from around the world and of
recent work on HCI (human-computer interaction) and LCD (learner centered design). Second,
a series of case study interviews were conducted with project directors, researchers, and
developers involved with advanced e-Learning systems in Korea and in the United States to
identify explicit and tacit knowledge about techniques for creating robust and effective learner-
centered e-Learning systems.
4. Research Results. A set of nine exemplary and representative learner-centered e-Learning
systems were identified in the literature review and from the case studies that met the criteria for
inclusion in the report. These e-Learning systems were discussed in a working framework that
consisted of six categories: (a) educational hypermedia, (b) modeling and simulation genre
systems, (c) virtual worlds for learning, (d) computer supported collaborative learning, (e)
intelligent systems, and (f) handheld devices. Based on an analysis of these systems, a framework
for advanced e-Learning systems based on principles from recent research in the science of
learning was proposed. The science of learning framework was illustrated using examples from
the literature review and case study e-Learning systems. Also provided are a set of suggested
initiatives to advance the development of advanced e-Learning systems in Korea, as well as a
consideration of implementation challenges for learner centered e-Learning in Korea and
significant opportunities for affecting educational reform initiatives.
5. Contributions. The results of this project should make contributions to e-Learning
conferences and workshops and also be disseminated in papers submitted to referred journals
and in a book.
6. Conclusion. It is hoped that the information in this report will be of value to a wide range of
individuals in the Korean e-Learning development community, including commercial, university,
and government content developers. This report could also be used as part of university classes
in instructional design and in professional development workshops that professional developers
take to become more familiar with the latest trends and approaches in e-Learning. The findings
and recommendations in this report should also be of value to personnel in the Korea IT
Industry Promotion Association and to industry groups such as the Korean e-Learning Industry
Association (KELIA) to inform policy decisions concerning opportunities and issues for future
funding projects, conferences, workshops, and so on. It is also hoped that the information in this
report will be of value to other policy makers involved with e-Learning in Korea.
KIPA FINAL REPORT: 21ST CENTURY E-LEARNING ii

Table of Contents
Executive Summary ....................................................................................................................................ii

1 Project Background ...........................................................................................................................1
1.1 Type I and Type II E-Learning Systems ...............................................................................2
1.2 Project Goals and Major Content...........................................................................................6
2 Literature Review Methodology .......................................................................................................7

3 Human-Computer Interaction and E-Learning: An Overview......................................................8
3.1 HCI and User Centered Design..............................................................................................9
3.2 From HCI to Learner-Centered Design................................................................................9
3.3 HCI and Type II E-Learning ................................................................................................10
4 Representative Exemplary Type II E-Learning Systems ............................................................10

4.1.1 Type II e-Learning Systems in Education.......................................................................11
4.1.2 Educational Hypermedia...................................................................................................13
4.1.2.1 System Description ................................................................................................... 13
4.1.2.2 LCD and Learning Scaffolds ................................................................................... 14
4.1.2.3 UCD, Usability, and Development......................................................................... 15
4.1.2.4 Research or Evaluation Findings ............................................................................ 16
4.1.2.5 Main References ........................................................................................................ 16
4.1.2.6 Online Resources....................................................................................................... 17
4.1.3 Virtual Worlds for Learning..............................................................................................18
4.1.3.5 Main References ........................................................................................................ 21
4.1.4 Computer Supported Collaborative Learning (CSCL)..................................................21
4.1.4.5 Main References ........................................................................................................ 24
4.1.5 Intelligent Systems..............................................................................................................25
4.1.5.5 Main References ........................................................................................................ 29
4.1.6 Handheld Technologies.....................................................................................................31
KIPA FINAL REPORT: 21ST CENTURY E-LEARNING iii

4.1.6.5 Main References ........................................................................................................ 35
4.2 Type II e-Learning in Industry and HRD ...........................................................................37
4.2.1 Overview..............................................................................................................................37
4.2.2 Modeling and Simulation Genre Systems: A Scenario-based e-Learning Program..37
4.2.2.5 Main References ........................................................................................................ 40
5 Type II E-Learning Case Studies....................................................................................................41

5.1 Case Study Interview Method ...............................................................................................41
5.2 Case Studies .............................................................................................................................41
5.2.1 Concord Consortium .........................................................................................................42
5.2.1.1 Case Study Participants............................................................................................. 42
5.2.1.4 Implementation.......................................................................................................... 44
5.2.1.5 E-Learning Industry Implications........................................................................... 44
5.2.1.6 Resources.................................................................................................................... 45
5.2.2 Harvard University .............................................................................................................46
5.2.2.2 General Description of System and Learning Scaffolds...................................... 46
5.2.2.4 Implementation.......................................................................................................... 47
5.2.2.6 Resources.................................................................................................................... 48
5.2.3 University of Michigan.......................................................................................................49
5.2.3.4 Implementation.......................................................................................................... 51
5.2.4 Center for Technology in Learning, SRI International.................................................52
5.2.4.4 Implementation.......................................................................................................... 54
5.2.4.6 Resources.................................................................................................................... 54
5.2.5 Crdu ....................................................................................................................................55
KIPA FINAL REPORT: 21ST CENTURY E-LEARNING iv

5.2.5.4 Implementation.......................................................................................................... 56
5.2.5.6 Resources.................................................................................................................... 57
6 Discussion.........................................................................................................................................58
6.1 Type II E-Learning Systems and Science of Learning Based Design Principles...........59
6.1.1 Provide Contextualized Learning.....................................................................................60
6.1.2 Provide Scaffolds and Tools.............................................................................................60
6.1.3 Consider the Preconceptions and Prior Knowledge .....................................................60
6.1.4 Make Organizing Conceptual Structure Explicit ...........................................................61
6.1.5 Make Thinking Visible.......................................................................................................61
6.1.6 Support Learning-By-Doing .............................................................................................62
6.1.7 Foster Thoughtful Collaborations amongst Communities of Learners .....................62
6.1.8 Provide Formative and Summative Assessment of Learning ......................................62
6.2 Application of Science of Learning Principles for the Design of E-Learning...............63
6.3 Lessons Learned Related to the Development of Type II E-Learning Systems ...........64
6.4 Research Limitations ..............................................................................................................65
6.5 Directions for Future Research and Policy Initiatives.......................................................65
7 Conclusion ........................................................................................................................................66
8 Acknowledgements..........................................................................................................................70
9 References ........................................................................................................................................71
10 Appendix............................................................................................................................................76
KIPA FINAL REPORT: 21ST CENTURY E-LEARNING v

1 Project Background
The meaning of knowing has shifted from being able to remember and repeat information
to being able to find and use it.
-- Herbert Simon, Nobel Laureate
Throughout the 20th century and now at the beginning of the 21st century, the dominate
approach to education both in Korea and in most advanced countries in the world may be
characterized as a teacher-centered or transmissionist approach. That is, the emphasis in
the classroom is on students acquiring facts and information, with teachers functioning as
the teller of facts, and the assessment of learning to determine if students can remember and
repeat the acquired information. However, views about learning have been changing
(National Research Council, 2000). As Nobel Laureate Herbert Simon has observed, the
goals for learning have now changed to having students who can find and use knowledge to
deal with complex and often dynamically changing situations.
Figure 1. Cover to the Ministry of Education White Paper.

A recent Korean Ministry of Education white paper Adapting Education to the Information Age
has called for reforms at all levels of education to prepare students for the 21st century global
knowledge economy (Kim, 2003). Also stressed in the white paper is the need for education
to shift away from the current teacher-centered approach to a learner-centered paradigm for
education in order to help students better develop creative thinking, problem solving, and
communication skills to handle changing situations. The White Paper also calls for more
extensive use of information and communication or e-Learning technologies in K-16 and life
long learning to help implement these 21st century educational reforms.
A considerable body of research over the past 20 years has demonstrated that in order to
achieve higher order learning goals such as those advocated in the Korean Ministry of
Education White Paper, different approaches to teaching and learning are needed (National
Research Council, 2000). For example, active approaches to learning are necessary in order
to help students develop deep understandings of important knowledge. It is also important
for teaching to take into consideration the pre-existing knowledge that students have, in
particular preconceptions that might make it difficult for them to understand the new
KIPA FINAL REPORT: 21ST CENTURY E-LEARNING 1

knowledge they are being taught. Allowing students to learn in ways that are similar to
situations in which they might use their knowledge is also valuable in order to foster the
ability to apply or transfer what they learn to new problems and new situations. The
importance of the social contexts of learning is also vital, with the recognition that often
advances in a students learning occur in collaborative situations with more knowledgeable
peers and teachers that then become cognitively internalized over time by the individual
learner. Many of these new approaches to teaching and learning can be enhanced by the use
of appropriately designed e-Learning systems.
1.1 Type I and Type II E-Learning Systems

Previous research related to e-Learning has been conducted under various names, such as
computer-assisted instruction, computer-based learning, educational technology,
instructional technology, learning technologies, hypermedia, computer supported
collaborative learning, and so on. In this project, the enabling technologies for e-Learning1
will be defined as computational media that consist of nodes of digitally encoded
information (e.g., text, pictures, multimedia, computer models and simulations, animations)
that are interconnected using hyperlinks, and network-mediated collaboration technologies.
Table 1. E-Learning Types and Characteristics
E-Learning
Paradigm Examples Learning Outcomes
Types
Delivery- Factual information, basic

Type I Tutorials, Drills
centered literacy
Problem-based Learning with Conceptual understanding,

Learner-
Type II Hypermedia, CSCL, virtual reality problem solving, knowledge
centered
systems, intelligent agents transfer
However, describing e-Learning in terms of the enabling technologies is not very useful as
this does not distinguish between the types of design features for various e-Learning
approaches, and more important, between different paradigms for teaching and learning.
Consequently in this report, a distinction is made between two general types e-Learning
systems (see Table 1). Type I e-Learning systems consist of more traditional approaches for
designing computer-based learning programs, such as tutorials and computer-based drills
(Roblyer, Castine, & King, 1988; Taylor, 1980). Systems such as these, which basically
employ a delivery-centered paradigm of instruction (similar to a teacher-centered
approach), have been found to help students learn basic factual information about a topic,
although they do not help students construct deep understandings of the content and
important concepts (Jacobson & Spiro, 1994). Despite this, Type I e-Learning systems such
as tutorials, one of the oldest approaches for designing computer-based learning systems, are
still the most widely used for commercial and government developed e-Learning programs
in Korea and globally. It is important to note that more recent Type I e-Learning tutorials
may be very sophisticated in terms of using multimedia, animations, and so on. However,
1 Few people know that the label eLearning or e-Learning was actually first coined by Jay Cross, a
Berkeley, U.S.A. based business consultant in 1998, and has been used very generally by many individuals since
that time to describe nearly any type of technology system used for educational or training purposes.

older and newer Type I e-Learning systems have a similar underlying design architecture that
employs mainly linear connections between different screens of information with some
simple branching based on answers to factual multiple choice questions.
Type I e-Learning systems differ from Type II approaches along a number of pedagogical
dimensions (see Table 2), such as learning mechanism, control, role of technology, role of student, type of
content, and learning outcome. The most fundamental difference concerns the learning
mechanism, whereby Type I systems primarily show and tell the students information to be
learned. Given this approach, it follows logically that the role for technology is to deliver
content and evaluations, with the content being didactic written texts or multimedia lectures
covering factual information. The role of the students is one of viewing or receiving what
the e-Learning system is showing and telling, hence the main control of a Type I system
must be in the technology. Finally, the learning outcomes in Type I e-Learning are typically
what a student achieves on so call objective tests that are primarily memory tests of factual
information retention.
Table 2. Pedagogical Dimensions of Type I and Type II E-Learning
Type I E-Learning Type II E-Learning

Pedagogical Dimension
Delivery-centered Paradigm Learner-centered Paradigm
Learning Mechanism Showing and Telling Learning-by-Doing
Cognitive Tools, Scaffolding

Learning, Providing Feedback,
Delivery of Content and
Role of Technology Non-linear Access to Information
Evaluations
Sources, and Supporting
Collaboration
Realistic Texts and Multimedia

Didactic Written Texts and
Cases, 2D and 3D Simulations
Type of Content Multimedia Lectures Covering
and Virtual Worlds, Dynamic
Factual Information
Computer Models
Actively Engaged in Problem

Passively View or Receive
Role of Student Solving, Projects, and
Content
Collaborative Activities
Control Technology Learner
Achievement on Objective Ability to Solve New, Often Open

Learning Outcomes Memory Tests of Factual Ended Problems, Performance
Information Retention Assessments
Representative of Type I e-learning systems in Korea are the VOD (Video-On-Demand)

services of the EBS (Educational Broadcasting Service) for educational television and for
Korean university entrance examination review courses. The VOD services run on current
multimedia computers that have access to the high speed Internet backbone that is available
throughout Korea. For example, Figures 2 to 4 are taken from is a social science VOD
program offered by the EBS. The content of the lecture, which is shown being delivered by
the teacher in Figure 2, deals with Industry Development and Environment for middle
school 3rd grade students. After the teacher gives the lecture, the content is summarized on

the screen (Figure 3) to help students memorize the main points, which are then tested for as
shown in Figure 4. The instruction in these VOD courses is clearly Type I, consisting of
cycles of audio/video delivery of content, summary texts, and factual tests to check if
content has been delivered (i.e., memorized) by the students (who really only need to
memorize the summary). A central goal of the EBS approach for VOD e-Learning is to
deliver the content in an efficient manner virtually anywhere in Korea and at any time.
Figure 2. A typical VOD screen of teacher speaking (i.e., delivering her lecture).
Figure 3. A summary screen of main points from the lecture.
Figure 4. A typical screen testing for memorization of content.

The use of VOD is also being used in higher education, as shown in Figure 5. While the
EBS VOD e-Learning system is a very impressive use of high speed multimedia Internet
technologies, it clearly illustrates a Type I delivery-centered e-Learning paradigm. The
learning mechanism is clearly one of showing and telling, as seen in Figures 2, 3, and 5 with
the technology being used to deliver the text and multimedia content. The students role in

this system is to view or receive, with the control of the e-Learning experience embedded in
the technology. Finally, the learning outcome is the score on a test of factual memory, as
seen in Figure 4.
Figure 5. University business course lecture delivered using VOD.

Type II systems differ dramatically from Type I systems in that they employ a learner-
centered rather than a delivery-centered paradigm (see Table 2). Given this different
learning mechanism, Type II systems view the role of technology as one that supports the
learning with tools that aid cognitive processing, provides scaffolds and real time feedback to
assist learning, enables non-linear access to a variety of information sources, and that
supports collaborative interactions. While the media types of content in Type II systems
might be similar to Type II (i.e., use of text and multimedia), that actual content in the digital
media is quite different. Type II content is typically authentic or real world texts or materials
or multimedia of realistic situations (not lectures) and may also be sophisticated computer
models and simulations or even virtual worlds. The role of students in Type II e-Learning is
one of active engagement in doing activities such as solving problems or collaboratively
working on projects, and consequently the control over the system is with the learner, not
the technology. Finally, the learning outcomes in Type II systems are actually much more
demanding than those used in Type I approaches as they require students to be able to solve
new, often open ended problems or for students to be assessed in terms of the ability to
actually perform or do what was learned, rather than to just pass a multiple choice memory
test.
An example of a Type II e-Learning approach, that is discussed in more detail below in
Section 4.2.9, involves a simulation environment for Samsung Construction engineers to
learn about advanced construction technology techniques. In this Goal Based Scenario
(GBS), as this approach is referred to, the engineers actively learn while working in a rich
multimedia e-Learning simulation that employs realistic problems, materials, and tasks as
content. The role of the technology in a GBS is to provide a variety of supports or scaffolds
for learning, feedback to the learner as he or she is engaged in tasks, access to realistic
information sources concerning advanced construction technologies, and supports for
interaction and collaboration. The role of the student in the GBS is to actively work on
realistic tasks and problems, with the various features of the program controlled by the
learner. The learning outcome for the GBS was successfully accomplishing the engineering
tasks in the system, and, more importantly, knowing how to use this new knowledge in the
real world projects that Samsung Construction is involved with.
Other examples of Type II e-Learning include hypermedia systems with non-linear links to
support problem-based learning (Jacobson & Archodidou, 2000; Jacobson & Spiro, 1995;

Shapiro & Niederhauser, 2003), computer supported collaborative learning systems (Guzdial
& Turns, 2000; Koschmann, 1996), adaptive or intelligent systems (Anderson, Corbett,
Koedinger, & Pelletier, 1995; Brusilovsky, 2001; Wenger, 1987), and virtual reality systems
(Dede, Salzman, Loftin, & Ash, 2000; Psotka, 1994). Type II e-Learning systems typically
employ more recent technologies (e.g., high performance computers with head mounted
displays for virtual reality systems, networked handheld computers, and intelligent agents),
but more important, they implement a learner-centered paradigm for teaching and learning
(see section 3). Research on Type II e-Learning systems has found these approaches can
foster advanced learning outcomes such as conceptual understanding of difficult knowledge
as well as subject specific problem solving and knowledge transfer (Anderson et al., 1995;
Jacobson & Kozma, 2000; National Research Council, 2000; President's Committee of
Advisors for Science and Technology (PCAST), 1997; Vosniadou, DeCorte, Glaser, &
Mandl, 1996).
Unfortunately, work on Type II e-Learning systems has been primarily conducted by basic
research groups at leading universities around the world, and consequently there is limited
expertise in the Korean e-Learning development community about how to develop and
implement these systems. To help address this issue, this project collected information about
representative exemplary systems and made an analytical synthesis of the important design
principles that are being articulated for Type II systems for e-Learning developers in Korea,
as is discussed below in this report.
1.2 Project Goals and Major Content

A central argument in this report is that in order to advance the overall quality of e-Learning
(and learning and training more generally) in the Republic of Korea, it will be necessary to
identify major types and characteristics of advanced e-Learning systems and the principles
for developing and implementing these systems. The main goal of this project is to provide
principled information about the characteristics and design principles of advanced Type II e-
Learning systems that implement learner-centered approaches and that have been found to
foster cognitively and socially robust levels of learner interaction and learning achievement.
In order to achieve this goal, the following tasks were undertaken in this project:
Conducted a comprehensive literature review of advanced e-Learning projects from
around the world and of recent work on HCI (human-computer interaction) and
LSD (learner centered design).2
Conducted case studies of advanced e-Learning systems in Korea and in the United
States that involved interviews with project directors, researchers, and developers to
identify explicit and tacit knowledge about techniques for developing robust and
effective learner-centered e-Learning systems.
Based on the information obtained in the literature review and the case studies, a set of
exemplary Type II e-Learning systems were identified that are discussed below. In addition,
a framework of design principles for Type II e-Learning systems is proposed that is based
upon a review of the HCI and learner centered design literatures and an analysis of the
exemplary e-Learning systems discussed in the report. The report concludes with a
discussion of challenges and opportunities related to the development and implementation
of Type II e-Learning approaches in Korea.
2 The review of the HCI literature was requested by the funding agency.

2 Literature Review Methodology
As discussed above, it is important that principles for the design and development of Type II
e-Learning systems be based on a solid understanding of the best and most current research
and development work that has been done in this area. Consequently, the first phase of the
project conducted comprehensive literature reviews in three main areas: (a) HCI and learner
centered design, (b) e-Learning in education (pre-college and college), and (c) e-Learning in
business and human resource development (HRD). The literature review methodology
consisted of ten steps:
1. Personal Contacts: Contact research colleagues for suggestions on recent papers,
reports, and books dealing with Type II e-Learning systems, in particular recent
research they may have conducted.
2. References in Known Documents: Examine the references in papers, reports, and books
professionally known to deal with Type II e-Learning systems.
3. Online Database Searches: Conduct searches of online bibliographic databases using
keywords and phrases relevant to research and development papers, reports, and
books on Type II e-Learning systems.
4. Filter Database Results: Critically evaluate search generated titles and abstracts in step 2
and select papers, reports, and books that appear to be most relevant to the goal of
this project.
5. References in Filtered Results: Examine the references in the papers, books, and reports
identified in step 4 for additional documents to include in the literature review set.
6. Obtain Documents: Obtain online or paper versions of papers and reports and buy or
obtain books through the Korea University library.
7. Potential Final Documents: Do readings of the papers, reports, and books identified in
steps 1 to 6 (i.e., potential final documents) as being the most relevant to the goals of
this project and make the determination of the final set of documents to be critically
analyzed in the literature review portion of the report.
8. Reference Database: Enter the reference information for the final set of papers, books,
and reports identified in step 7 into the projects master database of references.
9. Note Taking: Take notes on the papers, reports, and books in the final set of project
documents.

Table 3. Summary Data for Literature Review
E-Learning Literature Representative Databases

Representative Search Terms
Review Areas Searched
ERIC, ACM digital library, AACE

HCI and LCD HCI, LCD, interface design, information visualization
digital library
ERIC, PSYINFO, Google, Problem-based, case-based, transfer, scaffolding,

Education Harvard Graduate School of interactive/interaction, visualizations, representations,
Education-Learning Web hyperlink/hypermedia, simulation, virtual reality
Industry and Human AACE, AAOU, AECT, ASTD,

e-Learning, HCI, learner centered, GBS, hypertext,
Resources ICCE, ICDE, ACDL, ISPI, ISTE,
hypermedia, UI, Training, HRD
Development &ICDE
Table 3 provides summary data related to the three e-Learning areas being reviewed, the
databases searched, and representative search terms. Over 400 papers, books, and other
documents were reviewed by the project research staff and the report includes 80 documents
in the final reference list.
The next section provides a discussion of the main themes and principles for the design of
Type II e-Learning systems identified in the HCI and learner centered design documents
that were reviewed, followed by a non-technical description of exemplary Type II e-Learning
Systems the e-Learning literature review and from the case studies.
3 Human-Computer Interaction and E-Learning: An Overview3

In the past, computers were regarded as devices for particular mathematical, military,
scientific, or business applications that required specialized training and expertise to use.
Consequently, difficulties associated with using computers were not a big concern for the
education of the general population, and indeed, individuals with expertise in computational
devices could take a certain pride in their specialized skills. More recently, however, the
increasing processing power of computers has greatly increased their areas of applications
and the concurrent increasing affordability has exponentially increased the number of
individuals with interest in and access to these devices. As a result, the interaction between
humans and computers has shifted from the human adapting to the machine to having the
machine adapt to humans.
The field of human-computer interaction (HCI) studies how people can more easily and
conveniently use and interact with computer systems. To investigate these areas, HCI
encompasses multiple theoretical and research perspectives from computer science,
cognitive psychology, anthropology, education, design, engineering, mathematics, and even
physics (Jacko, 2003). Although defining a relatively new and growing research discipline is
difficult, a reasonable working definition of the field has been proposed by the ACM
SIGCHI (Special Interest Group-Computer-Human Interaction):
Human-computer interaction is a discipline concerned with the design, evaluation and implementation of
interactive computing systems for human use and with the study of major phenomena surrounding them.
(http://sigchi.org/cdg/cdg2.html#2_1)
3 The funding agency requested that a section of this report provide a discussion of HCI and e-Learning.

In brief, HCI is a multidisciplinary science that studies design and development issues to
improve the technology-based interactive interface environment.
3.1 HCI and User Centered Design

According to Soloway and Hsi (1998), one of the most important frameworks to emerge in
the field of HCI and software interface development has been User-Centered Design
(UCD). This perspective, which was articulated in one of the seminal books on UCD by
Norman and Draper (1986), focuses very specifically on addressing the needs of users of
computational media. Diverse users have diverse needs, and an important principle in UCD
is that the design of software interfaces should strive to align the representations and
affordances provided by a software system with likely expectations and ways of thinking and
acting from users. A corollary to this principle is that it is not enough for programmers to
get complex software systems to run. Just as important an issue in a software development
project is the ability of likely users to be able to easily use the software system, hence the
importance of usability testing techniques in UCD, such as interviews, questionnaires,
observations, focus groups, user logs, and user feedback (Nielsen, 1993; Norman & Draper,
1986; Shneiderman, 1998).
3.2 From HCI to Learner-Centered Design

As important as work in the field of HCI has been in advancing the ability of a wider range
of people to use and benefit from various types of computational media, the work on UCD
has tended to focus on the design of computational devices that were primarily used in the
business and industry sectors. It was argued in a set of papers by Elliot Soloway4 and his
associates that learners are different than users, and therefore, if computational media are
intended for use in educational settings that embrace new perspectives on learning and
teaching, then conventional techniques in HCI are not enough (Soloway, Guzdial, & Hay,
1994; Soloway & Hsi, 1998; Soloway et al., 1996). Soloway and his colleagues proposed the
term Learner-Centered Design (LCD) as a new perspective to extend HCI principles to
also include the support of educational interactions.
It was pointed out in Table 2 above that a learner centered paradigm involves a commitment
to having students learn-by-doing.5 However, it might seem paradoxical to expect students
to accomplish a task that might require them to know certain ideas that they are just in the
process of learning. To address this issue, a major strand of research involving learner
centered design has explored various ways to support or scaffold students to learn important
concepts, information, and skills while they are actively engaged in solving problems,
working on projects, or collaborating on other types of learning activities.
There have been a number of ways that LCD scaffolding techniques have been implemented
in recent advanced e-Learning systems. For example, in a report on the MODEL-IT system
developed at the University of Michigan, different types of scaffolding were implemented
that supported the learner or the encouraged reflection (Jackson, Krajcik, & Soloway, 2000).
One type, supportive scaffolding, assists the learner in performing a task, such as providing a
model of the task or guiding the learning, while a second type, reflective scaffolding, asks
questions to prompt the learner to metacognitively reflect on the task, the degree of success,
4One of the case studies discussed in this report involved discussions at the University of Michigan with Dr.
Soloway and members of his group; see section 5.3.
5 It must be stressed that this commitment to learning-by-doing is based on a growing body of scientific
research in the cognitive and learning sciences (National Research Council, 2000, 2001).

or ways to generalize from the task to other situations. Another scaffolding technique, which
may be implemented in different ways, supports the learner by highlighting or emphasizing
important conceptual aspects of the knowledge relevant to accomplishing the task. For
example, Section 4.1.2.2 discusses how conceptual scaffolding was integrated into the Knowledge
Mediator educational hypermedia system to support problem solving and learning for
transfer. These types of scaffolding are by no means exhaustive and are mentioned here as
representative examples of different scaffolding approaches. The Type II e-Learning systems
discussed in the body of the report will each be examined in terms of the various approaches
to scaffolding that they employ.
3.3 HCI and Type II E-Learning

At this point, the question may be asked: What is the specific value of HCI for developing
advanced e-Learning systems? There are three possible answers to this question:
1. HCI techniques such as LCD learning scaffolds and UCD usability are not important
in the development of Type II e-Learning systems.
2. HCI techniques such as LCD learning scaffolds and UCD usability are the most
important considerations in the development of Type II e-Learning systems.
3. HCI techniques such as LCD learning scaffolds and UCD usability are employed in
the development of Type II e-Learning systems, but another broader analytical
perspective is more important overall.
To help identify which of these possible answers is most reasonable, the discussion of the
literature review e-Learning systems and the case studies will present the information that
was obtained from available sources and will include two HCI related subsections: LCD
and Learning Scaffolds and UCD, Usability, and Development. The Discussion section
will revisit this issue and propose an answer.
In closing, this section has been a very brief introduction to HCI, UCD and LCD. Interested
readers are encouraged to consult the references to obtain more detailed information in
these areas.
4 Representative Exemplary Type II E-Learning Systems

As noted above, there has been considerable work on ways to design and use e-Learning
systems that are learner centered in their pedagogical approach. These systems are intended
to help achieve significant learning outcomes such as advanced problem solving, critical
thinking, and knowledge of conceptually challenging concepts, procedures, and skills, and so
on.
Based on the analysis of the documents obtained in the literature review portion of the
project, a working framework of six main categories for Type II e-Learning systems was
developed (see Table 4). This framework is intended to reflect different approaches that top
research and development groups in education and in business/HRD have been employing
as they explore ways to use various types of computational media to support learner centered
approaches to education and training. It should also be stressed that the working framework
categories are not rigidly exclusive of each other, and that a particular type of e-Learning
program might employ technologies and design elements derived from two or more
categories. For example, the MUVEES (Multi-User Virtual Environment Experiential
Simulator) project at Harvard University (discussed in sections 4.1.3 and 5.2) is most clearly
distinguished by its use of advanced 3D graphics to create virtual worlds for learners to

explore; hence it is assigned to Virtual Worlds for Learning category. However, MUVEES
systems also make extensive use of learner collaborations within the virtual environment that
employ techniques and perspectives related to the category of Computer Supported
Collaborative Learning (CSCL).
Table 4. A Working Framework for Type II e-Learning Systems and Programs Reviewed in Report
Category of Type II E-Learning Education Business/HRD Report Section
1. Educational Hypermedia Knowledge Mediator Discussion
BioLogica and
Case Study
Pedagogica
2. Modeling and Simulation Genre CDC Case Study
Systems Scenario
Discussion
Crdu Goal Based

Case Study
Scenario
Discussion & Case

3. Virtual Worlds for Learning MUVEES Project
Study
4. Computer Supported Discussion

WebCaMILE
Collaborative Learning
5. Intelligent Systems Cognitive Tutor Discussion
Environmental Discussion
6. Handheld Devices Detectives
Project WHIRL Case Study
From the systems identified in the literature review, a set of nine exemplary e-Learning
systems in the six Type II e-Learning category types were selected for inclusion in this report
(see Table 4). It is interesting to observe that several examples of educational Type II e-
Learning systems were identified for each of the category types (from which one was
selected for discussion in this report), while for business and HRD, the only Type II e-
Learning systems that met the criteria for inclusion in the report were both in the category
type Modeling and Simulation Genre Systems. Of these exemplary systems, six are presented
below in this section and four are discussed in section 5 Case Studies.6
4.1.1 Type II e-Learning Systems in Education
This section provides detailed illustrative discussions of exemplary Type II e-Learning
systems that are mainly used in pre-college or college education. Each of these systems has
been developed as part of funded research projects and the learning technology research
6Note that there are five case studies, but only four listed in Table 3. The Michigan case study with Dr. Elliot
Soloway involved more general design and development considerations related to a number of different e-
Learning systems that were either desktop modeling or handheld systems, and thus this case study is not
explicitly listed in the table.

related to each of these systems has been presented at the leading international conferences
and/or published in the top journals in the world dealing with e-Learning technologies. The
order for discussing these e-Learning systems follows the order in Table 4 using this schema:
System Description
LCD and Learning Scaffolds
UCD, Usability, and Development
Research or Evaluation Findings
Main References
Online Resources
The same schema is followed for the exemplary Type II business and HRD e-Learning
systems that are discussed in section 5.

4.1.2 Educational Hypermedia
Hypermedia may be defined as the connection of digitally encoded nodes of information that
might be text, pictures, audio, video, or animations/models using hyperlinks. Of course, the
most familiar example of hypermedia to many individuals is the World Wide Web, which is
essentially a globally distributed hypermedia system. The core features of hypermedia are its
nonlinear structure and the flexibility of information access. In traditional Type I e-Learning systems,
such as linear tutorials, the sequencing and presentation of content is controlled by the
author. However, the flexible and nonlinear structure of hypermedia shifts the degree of
control to the learner and in turn places greater cognitive demands on the learner as well
(Shapiro & Niederhauser, 2003; Tergan, 1997). Put another way, educational hypermedia is
fundamentally a learner-centered approach for using e-Learning technologies.
Research involving educational hypermedia over the past decade has been exploring various
ways to provide cognitive and metacognitive supports for novice and intermediate learners
(Brusilovsky, 2001; Dillon & Gabbard, 1998; Shapiro & Niederhauser, 2003; Spiro,
Feltovich, Jacobson, & Coulson, 1995; Tergan, 1997). This section discusses the main
elements of one particular approach for developing educational hypermedia, Knowledge
Mediator hypermedia systems (Jacobson & Archodidou, 2000; Jacobson, Maouri, Mishra, &
Kolar, 1996; Jacobson & Spiro, 1995), which have been critically acknowledged as one of the
most effective approaches for designing educational hypermedia in three separate large scale
literature reviews of research dealing with hypermedia and learning over the past seven years
(Dillon & Gabbard, 1998; Shapiro & Niederhauser, 2003; Tergan, 1997).
4.1.2.1 System Description
Knowledge Mediator hypermedia are problem-based learning (PBL) systems in which special
learning supports are provided to help learners understand difficult conceptual knowledge
and to enhance their ability to solve novel problems. Knowledge Mediator systems have
been developed in a variety of subject areas, such as the social impact of technology,
evolutionary biology, history, executive consulting, and medical PBL (Jacobson &
Archodidou, 2000; Jacobson et al., 1996; Jacobson & Spiro, 1995).
The main design principles for learning in Knowledge Mediator e-Learning hypermedia are
related to the functional elements of these systems, as shown in Figure 6. Three of these
elementsRepresentational Affordances of Technology, Knowledge-in-Context, and
Learning Scaffoldsare the design features that employ hypermedia link-node technologies.
The fourth element, Learning Tasks, refers to specific types of learning activities that are
optimized for these design features.
The representational affordances of hypermedia to show text, images, animations, and so
forth allow cases to be richly presented to the learners, as is illustrated in Figure 7. The
second functional element, Knowledge-in-Context, is another important feature of
Knowledge Mediator e-Learning systems. Here the focus is on solving problems using
multiple and contrasting case contexts that are more like the settings the learner is familiar
with or will experience in the real world, such as studying a case about ear infections and
relating that common childhood disease to a consideration of how antibiotic resistant
bacteria evolve (see Figures 7-9).

Figure 6. Functional elements of Knowledge Mediator e-Learning systems.
Figure 7. Case screen with pop-up list of the conceptual explanations available for the antibiotic resistance
case in a new SCORM compliant version of the Knowledge Mediator.
4.1.2.2 LCD and Learning Scaffolds
There are several learning scaffolds embedded in Knowledge Mediator e-Learning
hypermedia. One of the most important of these is conceptual criss-crossing, which is a special
type of problem solving with cognitive scaffolding. Conceptual criss-crossing provides the
learner with a problem or challenge question that requires the integration of knowledge
distributed across multiple cases and conceptual lessons in the e-Learning systems
knowledge base. A set of conceptually-based hyperlinks are provided for the problems that
are intended to support expert-like non-linear navigation through the knowledge space of
cases, conceptual lessons, abstract concepts, and context specific concept explanations. For

example, Figure 9 shows a challenge problem dealing with the role of the environment and
evolutionary changes and a set of conceptually-based hyperlinks to different case sections
related to the particular problem shown as Suggested links. These scaffolds are provided
because research indicates novices tend to focus on surface features or details of a question
and a case, yet solving problems requires a student to attend to both the conceptual deep
structure of the problem and the case materials, hence the scaffold of the conceptual
suggestions. Also, relevant information about a problem in a complex subject area may be
distributed across multiple information sources (e.g., cases). However, it difficult for novices
to cognitively interconnect this distributed knowledge, thus the learning scaffold of
suggested links is provided. Research has shown these types of conceptual scaffolds in
conceptual criss-crossing to be highly effective in fostering the difficult to achieve learning
goal of knowledge transfer (Jacobson & Archodidou, 2000; Jacobson et al., 1996; Jacobson
& Spiro, 1995).
4.1.2.3 UCD, Usability, and Development
One of the usability design challenges that received considerable attention in the
development of different versions of Knowledge Mediator systems was how to make the
cognitive scaffolding in the systems easy to use while still being effective. All research
systems were beta tested with learners who were representative of the target population and
appropriate revisions were made based on feedback and suggestions. Also, usability
questions were incorporated into the research instruments used with the main groups of
learners. The feedback that was obtained was used to inform subsequent visions of
Knowledge Mediator systems, such as readability of fonts, location of menu items, and ease
of using pop-ups and navigating in the system.
Figure 8. Screen showing the natural selection conceptual explanation for the antibiotic resistance case.

Figure 9. A Knowledge Mediator challenge problem with conceptual criss-crossing.
4.1.2.4 Research or Evaluation Findings
Research into learning with Knowledge Mediator e-Learning features has been the focus of a
series of studies over the past several years in which significant learning outcomes related to
problem solving, conceptual change, and knowledge transfer have been documented. A
particularly important finding was the enhanced ability of students who used an Evolution
Knowledge Mediator to solve novel evolutionary biology problems using the core ideas of
the neo-Darwinian theory of evolution (Jacobson & Archodidou, 2000). Also, compared to
students in a comparison group, the experimental students who had used the system for only
four to five hours over a two week period were still able to accurately solve difficult
evolutionary biology problems a year after using the system.
4.1.2.5 Main References
Brusilovsky, P. (2001). Adaptive Hypermedia. User Modeling and User Adapted Interaction, 6(2-3),
87-110.
Dillon, A., & Gabbard, R. (1998). Hypermedia as an educational technology: A review of the
quantitative research literature on learner comprehension, control, and style. Review of
Educational Research, 68(3), 322-349.
Jacobson, M. J., & Archodidou, A. (2000). The design of hypermedia tools for learning:
Fostering conceptual change and transfer of complex scientific knowledge. The Journal
of the Learning Sciences, 9(2), 149-199.
Jacobson, M. J., Maouri, C., Mishra, P., & Kolar, C. (1996). Learning with hypertext learning
environments: Theory, design, and research. Journal of Educational Multimedia and
Hypermedia, 5(3/4), 239-281.
Jacobson, M. J., & Spiro, R. J. (1995). Hypertext learning environments, cognitive flexibility,
and the transfer of complex knowledge: An empirical investigation. Journal of
Educational Computing Research, 12(5), 301-333.

Shapiro, A., & Niederhauser, D. (2003). Learning from hypertext: Research issues and
findings. In D. H. Jonassen (Ed.), Handbook of Research for Education Communications and
Technology, (2nd ed.). Mahwah, NJ: Lawrence Erlbaum Associates.
Spiro, R. J., Feltovich, P. J., Jacobson, M. J., & Coulson, R. L. (1995). Cognitive flexibility,
constructivism, and hypertext: Random access instruction for advanced knowledge
acquisition in ill-structured domains. In L. Steffe & J. Gale (Eds.), Constructivism in
education. Mahwah, NJ: Lawrence Erlbaum Associates.
Tergan, S. O. (1997). Conceptual and methodological shortcomings in
hypertext/hypermedia design and research. Journal of Educational Computing Research,
16(3), 209-235.
4.1.2.6 Online Resources
Online resources for Knowledge Mediator systems are in the process of being revised.
Access to papers related to Knowledge Mediator research:
http://mjjacobson.net/publications

4.1.3 Virtual Worlds for Learning7
The Type II e-Learning category of Virtual Worlds for Learning includes systems that may have
elements similar to sophisticated computer game-environments such as Unreal Tournaments
or Half-Life as well as systems that employ compelling 3D immersive technologies.
However, while computer games are dynamic and engaging, commercially available games
are intended to be entertaining and do not provide the type of environment that allows the
achievement of important educational and learning goals (Collins, 1996). Still, the potential
of utilizing the representational, collaborative, and motivational aspects of computer games
and advanced 3D technologies to help achieve substantive education learning goals is of
interest to many e-Learning researchers (Dede, 1995; Dede, Salzman, & Loftin, 1996; Dede
et al., 2000; Psotka, 1994; Squire, 2003; Winn, in press; Winn, Windschitl, & Hedley, 2001,
2001). In addition, there has been important work on employing advance technologies that
create immersive 3D virtual environments to experience representations about complex
scientific and mathematical ideas (Dede et al., 2000). In this section, one example of Virtual
Worlds for Learning is discussed, the MUVEES project at Harvard University (Dede &
Ketelhut, 2003; Dede, Nelson, & Ketelhut, 2004; Dede, Nelson, Ketelhut, Clarke, &
Bowman, 2004).
One of the most innovative of the projects in the area of Virtual Worlds for Learning is the
MUVEES (Multi-User Virtual Environment Experiential Simulator) at Harvard University.8
A current MUVEES project for science education is RIVER CITY that involves teams of
middle school students who are asked to collaboratively solve the health problems of a
simulated 19th century city (see Figure 10).
Figure 10. A screen shot of the River Worlds environment.
7 Some researchers in the field refer to the use of gaming technologies to create e-Learning systems as
educational computer games or simulation games. However, in this report the term virtual worlds for learning is used as
modern students, with their extensive experience with commercial computer games, will tend to play an
educational computer game rather than to learn with it.
8See the case study section 5.2.2 for a discussion with Dede and his group on development and
implementation issues related to an e-Learning system of this type.

When learners log into RIVER CITY, they see their avatars (computer generated characters on
the screen that they control and communicate through) and the main street of the city. The
RIVER CITY virtual world consists of a city with a river running through it, different forms
of terrain that influence water runoff, and various neighborhoods, industries, and
institutions, such as a hospital and a university (Figure 11). The avatars of the learners
themselves populate the city, along with computer-generated agents, digital objects that can
include audio or video clips, and the avatars of instructors. Students can communicatewith
one another with the group-chat function that is also shown in Figure 11.
Figure 11. River World screen with an avatar and different informational and communication displays.
Through data gathering, students observe the patterns that emerge and wrestle with
questions such as Why are many more poor people getting sick than rich people? Multiple
causal factors are involved, including polluted water runoff to low-lying areas, insects in
swampy areas, overcrowding, and the cost of access to medical care. Students are asked to
identify problems through observation and inference, form and test hypotheses, and deduce
evidence-based conclusions about underlying multi-causes.
MUVEES systems are explicitly designed to support learner-centered activities. The main
learning design principles in these systems are:
Learners do not passively observe situations, but actively investigate multivariate
problems.
Learners conduct authentic tasks in contexts similar to the real world.
Students learn important scientific knowledge and skills while they collect data on
water purity, crop yields, or pollution levels in order to solve problems posed in the
system.
Students learn in a collaborative problem-solving community in which they gain
knowledge and skills through co-interpreting data with other participants at varied
levels of skills. (Research has shown that novices may learn as much or more from
the observation of other, somewhat more advanced participants, as they would by
direct guidance from an expert.)

Learners are highly motivated since MUVEES systems resemble the entertainment
and communication media students experience and use outside of school, which can
reengage student interest in learning.
Learning scaffolds in MUVEES include various supports for science inquiry. For example,
learners are given the task of identifying what is making residents of River City sick, and then
given suggestions about where they can find information to try to identify the cause of the
illnesses. Another portion of the program scaffolds the process of hypothesis formation
where students are able to select a possible independent variable that might be related to the
dependent variable (which is a very difficult for students to deeply understand), such as
changing from summer to winter or draining a marsh by the city, and then collecting new
data in the virtual world that has been changed to reflect the new condition. A third learning
scaffold in MUVEES is a guide avatar (i.e., a teacher or subject expert) whose role is like that
of a coach. This avatar travels in River City with students and observers and supports their
learning processes by providing guidance, asking important questions at particular times,
moderating communications in a group, suggesting tips as necessary, correcting
misconceptions, and so on. Furthermore, related to this type of scaffolding support, teachers
can obtain information from the guide avatar that can help identify students learning needs
more concretely, which in turn should enhance the effectiveness of their teaching activities.
Participants can choose to interact with their team or with all participants in the RIVER CITY
environment at a given time. To aid their interactions, participants have access to one-click
interface features on the screen that enable the avatar to express (through stylized postures
and gestures) emotions such as happiness, sadness, and anger. These interface features also
allow looking upward or downward and seeing the world from a first-person perspective or
from a third-person viewpoint behind one's own body. Content in the right side of the
interface-window shifts based on what the participant encounters or activates in the virtual
environment. Dialogue is shown in the text box below these two windows; members of each
team can communicate regardless of their distance in the virtual space.
Another usability feature in River City is a map that is provided. With the map, students can
plan their discovery route both individually and collaboratively, as well as use and revise the
map as they geographically explore for clues to solve a problem. A birds eye view of the
River City map can also help students figure out how multiple factors might be related to
one another (which doubles as a type of learner scaffolding).

While research on the use of MUVEES systems is preliminary and ongoing, the early results
are quite promising. For example, in a study involving two public school classrooms in
Boston, MUVEES was examined in terms of usability, student motivation, student learning,
and classroom implementation issues (Dede & Ketelhut, 2003). One sixth- and one seventh-
grade classroom in different schools with high percentages of English as second language
(ESL) students used the RIVER CITY curriculum; with control classrooms using a similar
curriculum delivered via paper-based materials rather than technology. Preliminary results
indicated the MUVEES experimental condition was more motivating for students, including
lower ability students typically uninterested in classroom activities. Six out of seven
experimental students who scored less than 35% on the pre-test significantly improved their
content knowledge by the post test, while only two of five control students did so. Also,
students discovered multiple intriguing situations in MUVEES to investigate for which they

formed hypotheses about. For example, in the seventh grade classroom, five different
hypotheses about these situations emerged, such as posited causes ranging from population
density to immigration to water pollution. Another finding was that the MUVEES group
seemed to have greater positive effects for students with high perceptions of their own
thoughtfulness of inquiry. These students, on average, scored higher on the post content
test, controlling for SES, science GPA, ethnicity, and content pre-test score (Dede &
Ketelhut, 2003). Overall, these findings suggest that this approach to using e-Learning
technologies to implement a learner-centered set of educational experiences is promising in
terms of its potential to enhance student academic achievement, and that it may, in
particular, help teachers reach students struggling with motivation, self-worth, and lack of
content knowledge.
Dede, C. (1995). The evolution of constructivist learning environments: Immersion in
distributed, virtual worlds. Educational Technology, 35(5), 46-52.
Dede, C., Nelson, B., Ketelhut, D., Clarke, J., & Bowman, C. (2004). Design-based research
strategies for studying situated learning in a multi-user virtual environment. Paper presented at the
International Conference of Learning Science 2004, Los Angeles, CA.
Dede, C., Salzman, M., Loftin, R. B., & Ash, K. (2000). The design of immersive virtual
learning environments: Fostering deep understandings of complex scientific
knowledge. In M. J. Jacobson & R. B. Kozma (Eds.), Innovations in science and mathematics
education: Advanced designs for technologies of learning (pp. 361-413). Mahwah, NJ: Lawrence
Erlbaum Associates.
Psotka, J. (1994). Immersive tutoring systems: Virtual reality and education and training.
Retrieved from http://198.97.199.60/its.html.
Winn, W., Windschitl, M., & Hedley, N. (2001). Learning science in an immersive virtual
environment. Paper presented at the American Educational Research Association
(AERA) Annual Meeting 2001, Seattle, WA.
Winn, W., Windschitl, M., & Hedley, N. (2001). Using immersive visualizations to promote the
understanding of complex natural systems: Learning inside virtual Puget Sound. Paper presented
at the annual meeting of the National Association for Research on Science Teaching
(NARST), St. Louis, MO.
The main online resources for the MOVEES project may be found at:
http://muve.gse.harvard.edu/muvees2003/index.html
Information about the project, research papers, and pictures and digital video clips of sample
MOVEES experiences are available at the site.
4.1.4 Computer Supported Collaborative Learning (CSCL)
Computer supported collaborative learning (CSCL) refers to a technology based on
educational practice that encourages students to communicate and work together as a means
for fostering learning (Guzdial & Turns, 2000). This field has grown from wide research
roots such as socio-cultural theory, constructivism, and computer science (Koschmann,

1996; Pea, 1994; Scardamalia & Bereiter, 1994) and is based on the premise that computers
can support and facilitate group processes and dynamics in ways that compliment face-to-
face interactions or that even may not be achievable by face-to-face interactions. CSCL
systems are designed to support communicating ideas and information, accessing
information and documents, and providing feedback on problem-solving activities to
provide students scaffolding or support in learning together effectively. In this section, one
representative CSCL system, WebCaMILE, is discussed.
WebCaMILE (Collaborative and Multimedia Interactive Learning Environment) is a Web-
based collaboration tool for use by students to encourage learning (Guzdial, 1997; Guzdial et
al., 1996; Guzdial & Turns, 2000). The project, which is directed by Dr. Mark Guzdial at the
Georgia Institute of Technology, has received funding from the U.S. Advanced Research
Projects Agency, U.S. National Science Foundation, EduTech Institute, and GTE
Foundation.
The WebCaMILE "home" looks like a typical newsgroup in that it lists posted messages and
threads related discussions as in a conventional discussion board (see Figure 12). It also has
facilities for searching and filtering data on the page and for helping users find responses to
their notes. Each posting can be selected by clicking on a link from the home page, or
clicking on a link at the bottom of a note. A posting has its type, author, title, and body, and
may also contain links to digital media such as document or multimedia.
Figure 12. Main WebCaMILE message list screen.
One of the distinctive features of WebCaMILE is Thinking Types (see Figure 13). When a
user indicates that she wants to create a new note, she is prompted to identify the type of her
note, such as a New Idea, Rebuttal, or Alternative. These thinking types are modifiable for
each instance of WebCaMILE. For example, in a class on object-oriented modeling and
design, the note types include Analysis Issue, Design Issue, and Programming Issue. This
feature supports students as they think about a posting and its function among the whole
discussion, which enhances students metacognitive skills.

Figure 13. The Thinking Types screens in WebCaMILE.

The main learning principles in WebCaMILE are derived from socio-cognitive theory and
research. The system is explicitly learner centered as it is designed to support university
students who are involved with collaborative online discussions and projects for the courses.
The main LCD learning scaffolds in WebCaMILE are:
Prompts for students to identify the types of postings they are contributing to the
online collaboration and suggestions for things to say in those types of notes.
Directly addressable information (via a URL) to a particular WebCaMILE discussion
thread that provides students with anchors for their online discussions.
Prompts in the Thinking Types module that scaffold metacognition.
These three types of scaffolding are lacking in standard bulletin board systems (BBS) such as
those found in learning management systems (LMS).
WebCaMILE is designed as a system with a web client and a backend server that employs
standard HTML conventions. The interface design is intended to clearly show the main
functional elements of the system and to be easy to navigate. The program has been
iteratively revised based on user feedback over several years, from the initial non-Web
version of the program (CaMILE) to the most recent HTML implementation as
WebCaMILE.
The following are the main educational advantages of using WebCaMILE for CSCL over
regular discussion list systems that have been identified in research on the system (Guzdial &
Turns, 2000):

Increased quantity and quality of the notes posted on the system that are related to a
discussion thread.
Enhanced students metacognitive skills related to their understanding of ideas,
context of discussion, and contributions to the discussion community.
These findings are important as they indicate that students, even very bright ones at a top
technology university, need support in order to better learn in more open ended and
collaborative learner centered environments. Identifying additional ways to enhance learning
in CSCL is an ongoing area of concern and research in the field (Chan & Aalst, 2004).
Chan, C. K. K., & Aalst, J. V. (2004). Learning, assessment and collaboration in computer-
supported environments. In What we know about CSCL and implementing it in higher
education (pp. 87 - 112). Norwell, MA: Kluwer Academic Publishers.
Guzdial, M., Kafai, Y., Carroll, J. B., Fischer, G., Schank, R., & Soloway, E. (1995). Learner-
centered system design: HCI perspective for the future. In Proceedings of the Conference on
Designing Interactive Systems: Processes, Practices, Methods, & Techniques (pp. 143-147). Ann
Arbor, Michigan: ACM Press.
Guzdial, M. (1997). Information ecology of collaborations in educational settings: Influence
of tool. In R. Hall, N. Miyake & N. Enyedy (Eds.), Proceedings of computer-supported
collaborative learning '97 (pp. 83-90). Toronto, Ontario, Canada.
Guzdial, M., Kolodner, J. L., Hmelo, C., Narayanan, H., Carlson, D., Rappin, N., et al.
(1996). Computer support for learning through complex problem-solving. Communications of the
ACM, 39(4), 43-45.
Guzdial, M., & Turns, J. (2000). Computer-supported collaborative learning in engineering:
The challenge of scaling-up assessment. In M. J. Jacobson & R. B. Kozma (Eds.),
Innovations in science and mathematics education: Advanced designs for technologies of learning.
Mahwah, NJ: Lawrence Erlbaum Associates.
Koschmann, T. (1996). CSCL: Theory and practice on an emerging paradigm. Mahwah, NJ:
Lawrence Erlbaum Associates. (CSCL is an abbreviation for Computer Supported
Collaborative Learning).
Pea, R. D. (1994). Seeing what we build together: Distributed multimedia learning
environments for transformative communications. Journal of the Learning Sciences, 3(3),
285-299.
Scardamalia, M., & Bereiter, C. (1994). Computer support for knowledge-building

communities. The Journal of the Learning Sciences, 3(3), 265-283.
Suthers, D., & Weiner, A. (1995). Groupware for developing citical discussion skills. In J. L.
Schnase & E. L. Cunnius (Eds.), CSCL'95 Proceedings (pp. 341-348). Bloomington, IN:
Lawrence Erlbaum Associates.
Main WebCaMILE website: http://www.cc.gatech.edu/gvu/edtech/CaMILE.html

4.1.5 Intelligent Systems
From the earliest days of research involving computers and learning (i.e., computer-assisted
instruction [CAI]) in the 1960s), a goal has been to provide automated instruction that was
individualized for different students. However, the design approaches used in CAI, such as
linear tutorials and drill and practice programs (that continue to be used even in currently
produced Type I systems, as discussed above), were criticized for being rigid and failing to
provide flexible instructional approaches that matched the skill of human teachers
(Carbonell, 1970; Sleeman & Brown, 1982; Wenger, 1987). Another critique of CAI systems
is that they could not represent what a student knew, what the student needed to learn, or
how to adaptively teach the student. Consequently, developing the techniques to cognitively
model a students knowledge, as well as that of an expert and an appropriate pedagogy, has
been the focus of considerable research in the cognitive and learning sciences over the past
forty years, with the resulting systems often called intelligent systems, intelligent tutoring
systems, or adaptive systems (Brusilovsky, 2001; Sleeman & Brown, 1982; Wenger, 1987).
Dr. John R. Anderson at Carnegie Mellon University is one of most respected cognitive
scientists in the world and one of the leading researchers in this area since the 1980s
(Anderson, Corbett, Koedinger, & Pelletier, 1995; Anderson & Gluck, 2001; Anderson,
1983; Anderson, Conrad, & Corbett, 1989). His research group has developed a number of
intelligent tutoring systems (ITS) as research projects, and there has been a recent
commercialized version based on his groups research called the Cognitive Tutor that has
been released by Carnegie Learning, Inc.
The Cognitive Tutor is an intelligent learning system based on Andersons ACT-R theory
of cognition9 to inform the cognitive model the system has of what the student knows and
does not know. The use of this modeling approach allows instruction to be directed at what
still needs to be learned, and to ensure that students learning time is spent on learning
appropriate knowledge and skills in a more efficient manner (Koedinger, Corbett, Ritter, &
Shapiro, 2000). The Cognitive Tutor has been commercialized with full curricula in
Algebra I, Geometry, Algebra II, Integrated Math Series, and Quantitative Literacy through
Algebra. Each curriculum combines software-based, individualized computer lessons with
collaborative, real-world problem-solving activities. Students spend about 40% of their class
time using the software, and the balance of their time engaged in classroom problem-solving
activities (for a more detailed description, refer to the official website of Carnegie Learning
listed below).
This section provides a summary of the Cognitive Tutor Algebra I program that is
demonstrated on the Carnegie Learning website (see URL below). The Cognitive Tutor
Algebra I curriculum gives students the opportunity to work with various representations of
9According to the Anderson (1996), ACT-R theory provides insights into how students learn new skills and
concepts, and, in doing so allows teachers to see where students may need extra practice to master the new
work. ACT-R suggests that complex cognition arises from an interaction of procedural and declarative
knowledge. Declarative knowledge is encodes facts (such as Washington, D.C. is the capital of the United
States, 5 + 3 = 8); procedural knowledge encodes skills or how we do things (such how to drive or how to
perform addition). According to the ACT-R theory, the power of human cognition depends on how people
combine these two types of knowledge.

functions including tables, graphs, algebraic expressions, and text descriptions. As they
translate functions from one representation to the other, students learn to understand how
these representations are interconnected (see Figure 14).
Figure 14. Various representations of functions are interconnected in the Cognitive Tutor
Progressing through the curriculum, students learn to generalize specific instances into
algebraic formulas. Students complete the Worksheet (which functions like a spreadsheet) by
recording answers to questions posed in the Problem Scenario (see Figure 15).
Figure 15. The Problem Scenario screen, left, and a Worksheet screen on the right.
The Just-in-Time Help Messages feature, shown in Figure 16, is an important component of
the Cognitive Tutor. If a student becomes unsure of a solution path, help is readily

available by clicking on the help button. The tutor accesses the systems cognitive model of
the student and chooses the most helpful learning path before guiding the student to the
next step. The tutor provides progressive levels of helpif the first hint is insufficient, more
hints are offered. In addition to the help menus, assistance is offered automatically whenever
a student makes a common mistake. In such cases, the tutor responds with a context-
sensitive help message that is designed to clarify the lesson.
Figure 16. Just-in-Time Help Messages gives students immediate feedback when students make errors.
Another module of the Cognitive Tutor, Solver, encourages students to solve equations
within the context of problems (see Figure 17). Students learn techniques to solve problems
and may begin to discover the value of mathematical skills beyond the classroom.
Figure 17. The Solver module in the Cognitive Tutor.

Figure 18. A typical Cognitive Tutor graph screen.
Working with Graphs is an essential part of the Cognitive Tutor's approach to problem
solving (see Figure 18). Students set boundaries and intervals, label axes and plot points, and
lines.
Figure 19. A screen showing skills the student has learned.

The Skills module dynamically assesses and tracks each student's progress and level of
understanding on specific mathematical skills (see Figure 19). In any given lesson, there may
be up to 20 skills for a student to learn. The tutor constantly monitors the students actions
and each action is tied to the larger set of skills that comprise the domain, such as labeling a
graph or finding negative slopes.
As the Tutor guides a student down an individualized learning path, the student can also
access this information on demand, which encourages accountably for his or her learning
progress. Throughout the various problem-solving activities, students behavior and
knowledge growth are monitored and traced. This knowledge tracing dynamically updates
estimates of how well the student knows each production rule (Anderson et al., 1995). These
estimates are used to select problem-solving activities and to adjust pacing to adapt to
individual student needs. Using the Cognitive Tutor, students receive the benefits of

individualized instruction, ample practice, immediate feedback, and coaching. "Just-in-time"
help, "on-demand" help, and positive reinforcement put students in control of their own
learning and help to keep them on task (Koedinger et al., 2000).
The fundamental learning principle employed in the Cognitive Tutor is active learning-by-
doing with problem solving activities. To support this, important LCD learning scaffolds of
the system include monitoring student responses, providing feedback or help, and tracking
the development of skills. Another important design principle is the use of production rules
based cognitive modeling to allow the system to adaptively provide effective learning
experiences that challenge individual students with the proper level of tasks. The use of
cognitive modeling also allows the system to provide assessment scaffolding such as
immediate feedback to students, which is particularly important given research has
documented how learning is enhanced with timely and meaningful formative feedback
(VanLehn, Siler, & Murray, 2003), as well as how rare it is for students to receive such
feedback in typical classroom activities (National Research Council, 2001).
The papers reviewed for this report did not find explicit discussions of HCI and usability
issues related to the development of ITS developed earlier by the Anderson group
(Anderson et al., 1995). However, a review of the screen shots above indicates the current
commercial Cognitive Tutor has a simple, clear interface that should be easy for students
to navigate and to use.
From 1993 to 1997, a number of studies were conducted at Carnegie Mellon University to
evaluate the Cognitive Tutor (Koedinger, Corbett, Ritter, & Shapiro, 2000). Two of these
studies compared students using Cognitive Tutor programs at Pittsburgh public high
schools to classes at comparable Pittsburgh public high schools who were not using the
Cognitive Tutor programs. An additional study of the same design was performed with
students in the Milwaukee public schools. Students using the Cognitive Tutor have been
shown in these studies to:
Perform 30% better on questions from the Third International Math and Science
Study (TIMSS) assessment.
Demonstrate an 85% better performance on assessments of complex mathematical
problem solving and thinking.
Have a 70% greater likelihood of completing subsequent Geometry and Algebra II
courses.
Display a 15-25% advantage on the SAT and Iowa Algebra Aptitude Test.
Show equivalent results for both minority students and non-minority students.
Anderson, J., Corbett, A., Koedinger, K., & Pelletier, R. (1995). Cognitive tutors: Lessons
learned. Journal of the Learning Sciences, 4(2), 167-207.

Anderson, J., & Gluck, K. (2001). What role do cognitive architectures play in intelligent
tutoring systems? In D. Klahr & S. Carver (Eds.), Cognition & Instruction: Twenty-five
years of progress (pp. 227-262). Mahwah, NJ: Lawrence Erlbaum Associates.
Koedinger, K., Corbett, A., Ritter, S., & Shapiro, L. (2000). Carnegie Learning's Cognitive Tutor:
Summary research results. Retrieved from
http://www.carnegielearning.com/research/research_reports/CMU_research_results.
pdf.
Sleeman, D. H., & Brown, J. S. (1982). Introduction: Intelligent tutoring systems. In D. H.
Sleeman & J. S. Brown (Eds.), Intelligent tutoring systems (pp. 1-11). London: Academic
Press.
VanLehn, K., Siler, S., & Murray, C. (2003). Why do only some events cause learning during
human tutoring? Cognition and Instruction, 2(3), 209-249.
Wenger, E. (1987). Artificial intelligence and tutoring systems. Los Altos, CA: Morgan Kaufmann.
http://www.carnegielearning.com/

4.1.6 Handheld Technologies
It is remarkable that whereas in the United States there has been great interest in the
educational research community investigating the power of handheld computing technology
in education, in Korea, there has been little work to this area. However, in terms of
infrastructure, the Korea National Statistical Office statistics indicate that for users of mobile
phones (which are rapidly becoming the most sophisticated of the handheld technologies),
subscriber rates in 2000 were 90% or higher for individuals in their twenties and thirties and
79% or higher for individuals in their teens, forties, and fifties, which suggests Koreans are
technically sophisticated and comfortable with handheld and mobile technologies.
Based on the research to date (Clyde, 2004; Colella, 2000; Goldman, Pea, Maldonado,
Martin, & White, 2004; Inkpen, 1999; Jipping, Dieter, Krikke, & Sandro, 2001; Klopfer,
Yoon, & Rivas, in press; May, 2003; Relan, Guiton, Parker, & Wilkerson, 2003; Roschelle &
Pea, 2003; Tinker & Krajcik, 2001; Wilensky & Stroup, 1999; Zurita & Nussbaum, 2004), the
main merits of handhelds are:
Affordability.
Physical affordances to use in active and authentic learning activities.
Promote collaborative learning processes.
Support computational media with cognitively empowering representations (e.g.,
simulations, manipulate mathematical notations, modeling tools, diagramming tools).
Support network communication both among local peers (through beaming) and to
network connected servers.
Figure 20. Environmental Detectives handheld with GPS system.

In this section, ENVIRONMENTAL DETECTIVES is discussed as an example of an educational
handheld application (Klopfer, Squire, & Jenkins, 2002). This particular approach to using
the handheld device is in the context of a participatory simulation in which students
participate in a dynamic learning activity that provides the data to construct a simulation that
may then be later run on a computer (Colella, 2000; Klopfer et al., in press; Wilensky &
Stroup, 1999). This genre of educational software is exciting in that it leverages the
functionality of the new generation of handhelds technologies, which is much more
affordable for schools and students, while also making learner-centered pedagogical
approaches more practical in real classroom settings.

Students play the role of environmental engineers who are presented with the following
scenario at the beginning of the simulation (see screenshot in Figure 21).
During the construction of the underground garage of the new Stata Center, significant amounts of water are pumped
up from the ground in order to lower the groundwater table so that the garage can be constructed in a dry
environment. As a matter of regulation the water is tested for the 25 most commonly found chemicals in groundwater
at hazardous waste sites. As a result of the testing it is discovered that a toxin is present in the extracted water. You
call the President of the University to report and he asks, How dangerous is this toxin? Where did the contamination
come from and how widespread is it? Does MIT need to take some action (and what action might this be)? What do
you advise? You promise to call him back within three hours with your advice on the problem.
Figure 21. Scenario introduction screen.
By clicking Video button in Figure 22, students watch a 60 second digital video briefing
from the University president where they are enlisted to investigate the spill of the toxin, a
carcinogenic degreasing agent that is commonly found in machine shops, cafeterias, and
hospitals. The goal of the game is to locate the source of the spill, identify the responsible
party, design a remediation plan, and brief the president of the University on any health and
legal risks so that he will be prepared for a meeting with the EPA all within three hours. At
the end of the game, students make a five minute presentation to their peers outlining their
theory behind the spill.

Figure 23. GPS display of resources.
The spread of the toxin is simulated on a location-aware Pocket PC, which functions as a
tool which students can use to investigate the toxic spill. Each Pocket PC is equipped with a
GPS device, which allows players to sample chemical concentrations in the groundwater
depending on their location (see Figure 23). Players are given three reusable virtual drilling
apparatuses displayed in the handheld device that they can use to drill for water samples.
After drilling for a sample, players must wait three minutes for the sample to return, meaning
that students can only take three samples at a time, thus forcing them to develop sampling
strategies in order to optimize the amount of ground that they can cover in limited time.
Because the GPS data is only accurate within 10 meters, there is some built-in error to the
collected readings as well.
Environmental Detectives contains a multimedia database of resources that students can use
to learn more about the chemical, where it is found on campus, the health risks associated
with exposure to the toxin, how it flows through ground water, relevant EPA regulations,
remediation strategies for cleaning up the toxin, and the political and economic
consequences of EPA violations on campus. Students access these resources by obtaining
interviews from virtual university faculty and staff that are spread across campus at locations
roughly corresponding with actual operations. Because there is not enough time to interview
everyone or to drill more than a handful of wells, students must make choices between
collecting interviews, gathering background information, drilling wells, or adjusting and
reprioritizing goals as new information becomes available.
To conduct an interview, one of the main data resources in the game, students identify the
location that they want to visit. Clicking on the white dots gives them a short description of
what is at that location. Students walk to one of the locations indicated by a white dot. When
they get near the interview location, the Action button will turn from gray into a clickable
button and the status bar will tell you "Interview Available" (see Figure 24).

Figure 24. Interview screens
Digging wells and associated activities can be another important resource. Students can
virtually walk to the location they are interested in sampling and click the Dig button.
After the three minute wait, they return to the well location they can "Take Rig" and then
"Sample." Sample analysis offers three options according to its accuracy and time (see Figure
25). Students have to decide between getting more accurate results while waiting a relatively
long time versus getting less accurate results while waiting a shorter time. To review the
water sample data, they go to the Resources menu and choose Wells, which will display all of
the locations and readings.
Figure 25. Sampling screens.

The main learning design principles in this project explicitly focus on ways that handheld
software may support learner centered educational activities. Students are actively involved
with a realistic scenario and use the handheld systems for information collection and
communication. The Environmental Detectives software developed for the handheld
systems supports collaborative learning and problem solving.

However, it is interesting to note that while Environmental Detectives explicitly supports
learning-by-doing, evidently the interface or processing limitations of handheld devices did
not allow the implementation of conceptual or metacognitive types of scaffolding as did
other Type II e-Learning systems reviewed in this report. This issue of the potential
limitations of handheld devices for providing scaffolding is also discussed in section 5.2.3.3.
The HCI and usability features of the Environmental Detectives software make appropriate
use of the limited screen size that is provided by handheld systems. As can be seen in the
screen shots, the interface options are simple and clearly visible to the user. The availability
of color graphics and video on this new generation of handheld systems is also noteworthy,
both for the additional information that may be made available and for the potential
motivational enhancement that dynamic and engaging multimedia software may provide to
students. Another usability feature of the systems is the GPS loaded map. This map is similar
in some ways to the navigational map in MUVEES in that students can plan their
explorations with map, but with a real GPS the students can actually measure distances and
travel time on the handheld map.
To date, this participatory simulation has been run at three sites, including MIT, a nearby
nature center, and a local high school. Early research has shown that this mode of learning is
highly motivational, successful in engaging university and secondary school students in large
scale environmental engineering activities, and provides an authentic mode of scientific
investigation. The major research findings with the project are:
Augmented reality. Engaging and easy transition between PDA and real world.
Cooperation and competition in game play. Emergent cooperation prevails, but competition
was also found.
Gender patterns in points versus interviews. Boys were found to be number driven while
girls were found to be interpersonally driven.
Resources shaping and constraining actions. Informational resources in the game versus the
lack of real world sources of information.

Clyde, L. (2004). Computing in the palm of your hand. Teacher Librarian, 31(3), 42-44.
Colella, V. (2000). Participatory simulations: Building collaborative understanding through
immersive dynamic modeling. The Journal of the Learning Sciences, 9(4), 471-500.
Goldman, S., Pea, R., Maldonado, H., Martin, L., White, T. 2004. Functioning in the wireless
classroom. In Proceedings of the Third IEEE International Workshop on Wireless and Mobile
Technologies in Education. New York: IEEE Press.
Inkpen, K. M. (1999). Designing handheld technologies for kids. Personal Technologies Journal,
3(1&2), 81-89.

Jipping, M., Dieter, S., Krikke, J. & Sandro, S (2001). Using handheld computers in the
classroom: Laboratories and collaboration on handheld machines. In Proceedings of
the 2001 SIGCSE Technical symposium, SIGCSE Technical Bulletin, 33, 169-173.
Klopfer, E., Squire, K., & Jenkins, H. (2002). Environmental Detectives: PDAs as a window
into a virtual simulated world. In Proceedings of the IEEE International Workshop on
Wireless and Mobile Technologies in Education (WMT 02. Vxj, Sweden.
Klopfer, E. Yoon, S. & Rivas, L. (in press). Comparative Analysis of Palm and Wearable
Computers for Participatory Simulations. Journal of Computer Assisted Learning.
May, M. (2003). Hand-on power. The Scientist, 17(22), 1-4.
Relan, A., Guiton, G., Parker, N., Wilkerson, L. (2003, April). A hand-held requirement for
clinical education: Effectiveness of a systemic educational intervention in altering perception and use of
handhelds among medical students. Paper presented at the annual meeting of the American
Educational Research Association, Chicago, IL.
Roschelle, J. (2003). Unlocking the learning value of wireless mobile devices. Journal of
Computer Assisted Learning, 19(3), 260-272.
Tinker, R. & Krajcik, J. (2001). Portable technologies: Science learning in context. New York: Kluwer
Academic/Plenum Publishers.
Zurita, G. & Nussbaum, M. (2004). Computer supported collaborative learning using
wirelessly interconnected handheld computers. Computers & Education, 42, 289-314.

Promo video: http://education.mit.edu/ED/EnvDet.mov
Eric Klopfers intro class video: http://school.mit.edu/ED/intro.mov.

4.2 Type II e-Learning in Industry and HRD
4.2.1 Overview
It is widely accepted that there is considerable and increasing interest in e-Learning in Korea
and globally, particularly in industry and HRD areas. But the reasons for this interest in
many organizations seem more related to the perception that training costs may be
minimized using e-Learning rather than a view that e-Learning provides enhanced
knowledge and skill development over traditional training approaches (Dalton, Manning,
Hagen, Paul, & Tong, 2000).
Still, a surprising finding in this project was that despite a global investment in industry e-
Learning of an estimated $240 billion in 2004, only a handful of documents such as reports,
papers, or book chapters were found on Type II e-Learning systems used in industry or
HRD settings that met literature review criteria for inclusion in this report (e.g., provided a
discussion or illustration of major design features, provided quantitative or qualitative data
from research or evaluations of a particular system documenting enhanced learning
outcomes).10
Another surprising finding was that industry and HRD Type II e-Learning systems appear to
only have been developed in the second category type, Modeling and Simulation Genre
Systems, unlike the educational Type II e-Learning systems where there are numerous
examples of systems in all six categories (see Table 4). Thus it appears the Type II e-
Learning systems described in this report, Botulism in Argentina e-Learning program that is
discussed in this section and the Construction Engineering Goal Based Scenario developed by the
Korean e-Learning company Crdu that is discussed in case study section 5.5, represent
particularly innovative systems in this area.
4.2.2 Modeling and Simulation Genre Systems: A Scenario-
based e-Learning Program
The United States Centers for Disease Control and Prevention (CDC) has developed a series
of e-Learning case studies with interactive exercises that are intended to teach epidemiologic
principles and practices. They are based on real-life outbreaks and public health problems
and were developed in collaboration with the original investigators and experts from the
CDC. The case studies require students to apply their epidemiologic knowledge and skills to
problems confronted by public health practitioners at the local, state, and national level every
day. Each case study consists of limited information about an evolving public health
problem and a series of 20-25 questions. The questions help the student work methodically
through the problem and are trigger points for learning and discussion. In this section, the
Botulism in Argentina e-Learning program is discussed to illustrate this approach.
According to the developers, the CDC Botulism in Argentina e-Learning program blends the
characteristics of a simulation with a linear e-Learning approach, what the developers refer
to as a scenario-based e-learning model (SEM) (Gathany & Stehr-Green, 2003). The various
10This issue was discussed by the author of this report with a senior Vice-President of an Australian-based e-
Learning company. He believed that there were a number of organizations that in fact employed Type II e-
Learning approaches and technologies, but that they keep the design specifications and evaluation findings of
their systems confidential to maintain a competitive training advantage over rival organizations.

screens provide access, hyperlinks, to different types of information related to the case
scenario, including photographs that depict investigation team activities, an epidemic curve
that graphically illustrates the outbreak and investigation and that changes as the scenario
progresses, and an investigation outline that corresponds to the six steps of an
epidemiological outbreak investigation and a record of the learners progress, as well as
related items such as press releases and questionnaires. Figures 26 to 31 show various
screens from the program.
Figure 26. Opening screen for the CDC Botulism in Argentina e-Learning program.
Figure 27. The initial page of the disease outbreak scenario.
Figure 28. A conceptual information page to supplement the case study data.

Figure 29. Background biomedical information provided as text and with digital audio.
Figure 30. CDC screen with question and feedback.
Figure 31. Concluding screen of CDC case study with summary investigation information.
The CDC Botulism in Argentina program employs real-world case based materials that are
intended to be engaging to adult learners as they investigate and solve problems related to
the epidemiological outbreak scenario (see Figure 26). Rather than relying on an instructor to
give advice or guide participants, the program provides access to a variety of support tools
or learning scaffolds, such as hints and reference materials, which may be consulted before
they answer questions. Another important learning and design feature is the availability of

assessment scaffolding that provides immediate feedback to problems the learner works on.
Learners who metacognitively feel they lack certain background knowledge related to the
case may review short tutorial lessons about basic concepts and information in this area (see
Figures 27 and 28).
A central UCD and usability feature of the CDC program is the interface design that
employs various graphics of real world objects, such as the online depiction of a day-planner
notebook placed on a wooden desktop on the opening screen (see Figure 26). The notebook
interface metaphor is continued throughout most of the screens in the program, such as the
use of divider tabs that represent the steps in a routine outbreak investigation (Figures 27, 30,
and 31).
Another important usability feature of the program is that it allows each learner to work
through the case study at his or her own pace without the help of an instructor. Each learner
decides which supplemental learning activities to undertake; for example, curious learners
could explore topics of interest in greater detail through vignettes presented by professionals
that investigate outbreaks frequently, including an epidemiologist, laboratorian, or sanitarian.
No research data on learning outcomes associated with the use of the Botulism in Argentina
e-Learning program appears to be available. However, the program has received E-Learning
Courseware Certification from the American Society for Training and Development (ASTD)
and the 2002 Outstanding Practice Award from the Design and Development Division of
the Association of Educational Communications and Technology (AECT).
Dalton, J. P., Manning, H., Hagen, P. R., Paul, Y., & Tong, J. (2000). Online training needs a new
course. Cambridge, MA: Forrester Research.
Gathany, N., & Stehr-Green, J. (2003). Scenario-based e-learning model: A CDC case study.
Learning Circuits. Retrieved from
http://www.learningcircuits.org/2003/apr2003/gathany.htm.
Schank, R. C., Fano, A., Bell, B., & Jona, M. (1994). The design of goal-based scenarios. The
Journal of the Learning Sciences, 3(4), 305-345.
CDC Computer-based Epidemiologic Case Studies:
http://www.phppo.cdc.gov/phtn/casestudies/computerbased/default.htm
Botulism in Argentina Case Study downloads:
http://www.phppo.cdc.gov/phtn/casestudies/computerbased/botarg.htm

5 Type II E-Learning Case Studies
The second major component of this project involved case studies with groups in Korea and
the U.S. that have created exemplary examples of Type II e-Learning systems. The main goal
of the case studies was to discuss design, development, implementation, and research issues
each of the particular projects has experienced in order to obtain more real world
information concerning the creation of advanced e-Learning systems than is typically
provided in reports, papers, or book chapters. This section first describes the interview
method used for the case studies followed by a discussion of the five case studies.
5.1 Case Study Interview Method

The interviews for the U.S. and Korean case studies employed a semi-structured interview
approach. First, a set of questions (see Appendix) were put together related to the goal of the
case studies discussed above, that is to discuss design, development, implementation, and
research issues related to different types of Type 2 e-Learning systems. For example,
question 3 asked about the design goals for the technology system while questions 4-8 and
14-17 dealt with different aspects of the development phase of the project. An important
issue in the development process relates to HCI and usability, which were the focus of
questions 9 and 10. Implementation issues were raised in question 11-13 and research
findings in question 18. The remaining questions, 19-22, asked about compliance with
industry standards for learning management systems (LMS) and learning content
management systems (LCMS) or with other plans to market or make the technology more
widely available. The questions were intended to be general enough that they could apply to
each of the six categories of e-Learning listed in Table 4.11
The questions were used in the semi-structured interviews conducted for case studies. The
interviews, which were audio or video taped, were conducted in a conversational style that
typically began with the first two or three questions and then were allowed to proceed
naturally based on the responsibilities the particular interviewees had for each project. The
interviewer would ask follow-up questions to probe the responses made and would ask
questions at the end if information related to particular questions had not been covered
during the interview. The notes from the case study interviews were analyzed for common
themes and issues that are discussed below.
5.2 Case Studies

There were five case studies conducted in this project. Table 5 lists the research groups or
organizations that were selected for the case studies and the dates that the case study visits
were made. These section discuses each of the case studies in turn.
11Note that none of the questions specifically asked about Type I and Type II e-Learning issues, since all of the
case studies were selected because they were explicitly Type II approaches for the design of e-Learning systems.

Table 5. Case Study Groups
E-Learning Type II E-Learning Date of
Research Group or Company Country
Area Approach Visit
Concord Consortium: Horowitzs United Modeling and Simulation 09-21-
Education
Group States Genre Systems 2004
United 09-20-
Harvard University: Dedes Group Education Virtual Worlds for Learning
States 2004
Modeling and Simulation
University of Michigan: Soloways United Education, HCI, 09-22-
Genre Systems
Group States LCD 2004
Handheld Devices
Modeling and Simulations
Center for Technology in Learning, United Computer Supported 09-23-
Education
Stanford Research Center States Collaborative Learning 2004
Handheld Devices
HRD and Modeling and Simulation 10-25-
Crdu Korea
Industry Genre Systems 2004
5.2.1 Concord Consortium

The interviews for this case study took place at the Concord Consortium, which is a
nonprofit educational research and development organization based in Concord,
Massachusetts, U.S.A. The Concord Consortium develops innovative interactive materials
that exploit advanced functionality in newly available information technologies in order to
enhance student learning in science and mathematics. The Concord Consortium has received
extensive research funding over the years and is a partner, with the University of California-
Berkeley, on a new $10 million dollar initiative entitled Technology-Enhanced Learning in Science,
which is one of three Centers for Learning and Teaching recently established by the U.S.
National Science Foundation.
5.2.1.1 Case Study Participants
The main discussions were with Dr. Paul Horowitz (see Figure 32), a Senior Scientist at the
Concord Consortium who directs the Concord Consortium Modeling Center. He also runs
the Modeling Across the Curriculum, Models and Data, and Fostering Transfer projects. Dr.
Horowitz is a theoretical physicist with an A.B. from Harvard College and a Ph.D. in physics
from New York University. His educational research interests center around helping
students use mental models to learn and apply scientific principles. A meeting was also held
with the President and Founder of the Concord Consortium, Dr. Robert Tinker, who is
internationally recognized as a pioneer in constructivist uses of educational technology. He
developed the MBL (microcomputer-based labs) and Network Science concepts and has
directed numerous educational research projects. He holds a Ph.D. in experimental low-
temperature physics from MIT.
Figure 32. Dr. Paul Horowitz, Dr. Michael Jacobson (PI), and Dr. Robert Tinker at the Concord Consortium.

The conversations with Dr. Horowitz focused on his research and development work related
to modeling environments to help students learn difficult scientific ideas. However, rather
than focusing only on developing software that provided a reasonably powerful modeling
tool for a scientific domain such as genetics, Horowitz and his colleagues have a new type of
e-Learning system they refer to as hypermodels. Hypermodels not only provide sophisticated
dynamic representations of the scientific phenomena but also have special embedded
scaffolds to help support the students learning.
One example of a hypermodel is BioLogica, which is a program for teaching high school
genetics developed as part of a four year project funded by the U.S. National Science
Foundation. The BioLogica software, which is programmed in Java, allows students to
manipulate processes related to genetics at different linked levels, from the microscopic
genotypic level of DNA to the macro level phenotypic expression of an organisms traits
(see Figure 33). The BioLogica hypermodel has embedded curriculum and assessment
functions that are intended to support a learner centered, model-based learning approach to
help students understand complex scientific knowledge.
Figure 33. Representative screens from BioLogica of different levels of genetics.

Another ongoing e-Learning research and development project that Horowitz and his group
are conducting involves Pedagogica, which is a software environment for converting computer
models and modeling tools into hypermodels (see Figure 34). Pedagogica supports a special
type of learner centered pedagogy involving inquiry-based lessons that guide the students as
they use a particular modeling program. Pedagogica has features that may change the
appearance of a modeling tool and present only those options that are relevant to a
particular inquiry activity. The interface is also used to assess student progress and can
provide real-time feedback over the Internet to students as they work on their inquiry
activities.

Figure 34. A schematic of the Pedagogica environment.

Both BioLogica and Pedagogica are complex software programs that have been developed as
part of multi-year research and development projects. The programs are written in Java in
order to support cross-platform use, with Pedagogica also linking to a back end database
system. UCD usability evaluations of the programs were based on classroom observations
and assessments of students using the systems. Also, refinements of software were informed
by student and teacher comments and feedback.
5.2.1.4 Implementation
According to Dr. Horowitz, one of the biggest implementation challenges with BioLogica
(and with an earlier version of the program, GenScope), was teacher training to appropriately
use the software. Research involving the classroom use of these two e-Learning systems
found that students learning was best when the programs were used by outstanding teachers
(Hickey, Wolfe, & Kindfield, 2000; Horwitz & Christie, 2000). One of the issues being
explored in the Biologica and Pedagogica research is how to design learner centered
modeling tools and software for the average teacher and the technology situations of
typical classrooms in the United States.
5.2.1.5 E-Learning Industry Implications
In the meeting with Dr. Horowitz and Dr. Tinker, it was pointed out that all software
developed at the Concord Consortium is open source. That is, not only are the majority of
the e-Learning systems that are developed by the Concord Consortium free (because they
were developed with the support of public research funds), but the computer source code is
also freely available. Thus Korean e-Learning developers who are interested in modeling
systems such those developed by Horowitz and his group would be able to directly use the
actually software to create Korean versions of the systems, add functionality, and so forth.
Of course, enhancements made to open source software code are to be made available to the
rest of the open source community, although it is permitted to make commercial software
based on open source code.

Another e-Learning industry issue that was discussed was a subscription model for
educational e-Learning resources. Under this business model, individuals or organizations
would not buy e-Learning software, but rather would pay a monthly fee to have access to an
e-Learning service that would be constantly updated with content and resources for teachers
and students.
The high development costs for creating sophisticated e-Learning modeling software for
complex scientific areas such as genetics or physics were also discussed. Dr. Horowitz noted
the commercial development of sophisticated e-Learning modeling tools for high school
science such as BioLogica is probably not possible given the relatively small market, the
amount of time to develop the quality of software for complex content, and the tailoring
necessary for teachers. However, e-Learning companies who can build upon existing open
source software may be able to have a viable business model for their products. Another
option for e-Learning companies and developers is to use the authoring module of a system
like Pedagogica which has its own scripting language that could be used by subject matter
experts and instructional designers to create customized hypermodels. Finally, it was noted
that the advanced functionality of Pedagogica and BioLogica is not compatible with current
industry standards such as AISS and SCORM. However, this was not felt to be a problem as
AISS and SCORM standards are not widely used in K-12 educational settings, which is the
main target for the modeling tools being developed at the Concord Consortium.
5.2.1.6 Resources
Concord Consortium main website: http://www.concord.org/
BioLogica resources (including downloadable version of the program):
http://biologica.concord.org/
Pedagogica resources (including downloadable version of the program):
http://pedagogica.concord.org/

5.2.2 Harvard University
Chris Dede (see Figure 35) is the Wirth Professor of Learning
Technologies and Chair of the Learning and Teaching area at the
Harvard Graduate School of Education (HGSE). His current
research focuses on the use of shared virtual environments to
enhance science learning by middle school students and exploring the
learning effectiveness of students use of modeling environments in
science. Dede is a member of the board of directors of the Boston
Tech Academy, an experimental small high school in the Boston
Public School system, funded by the Gates Foundation, and he Figure 35. Dr. Chris
serves on the advisory boards of ThinkLink, FreshPond, bigchalk, Dede, Harvard
Celt, and World Book, as well as several U.S. Department of University.
Education Regional Educational Labs and Regional Technology
Centers. Also participating in the case study discussions were research assistants Dianne
Ketelhut and Cassie Bowman.
5.2.2.2 General Description of System and Learning Scaffolds
See Section 4.1.3.1 for a description of the MUVEES system developed by. Dedes group
and Section 4.1.3.2 for a discussion of learning scaffolds in the system.
The initial work on the MUVEE project: began in 1998 while Dr. Dede was a professor at
George Mason University. A second round of funding was obtained when Dr. Dede joined
the faculty at Harvard University. The multi-user virtual system (VR) was initially developed
using custom software developed by a computer science collaborator on the project. For the
second round work on the project, the content was ported to a commercially available
technology environment marketed by Active Worlds (see 5.2.6 Resources for the web
address). The new technology in Active Worlds provided enhanced authoring, 3D graphics
libraries, and collaboration features over the first generation version of the system.
Another incentive for the project staff to shift from custom programmed software to Active
Worlds was the cost. The Active Worlds basic system for using pre-developed worlds is
only $20 per year and for $500 per year, one may purchase a dedicated world to develop.
The cost of a universe, which supports multiple worlds and large number of users (400-
500), is $35,000 per year. The MUVEES grant paid $200,000 to support the development of
special HCI design features to support learning and for data collection functions to support
research.
In addition to the features of the system discussed above in section 4.1.2, Active Worlds
provides additional features that support interactions and experiences in a virtual
environment. For example, although multiple teams may use a world at the same time and
they may see each other, participants can only interact with avatars on their own team. Also,
individual experiences in the world may be tailored (e.g., one student might see world in
summer and the other in winter). The system collects log file records about what is said and
what a user does, thus allowing a teacher or researcher to determine if a user is taking notes,
talking or not talking, or just teleporting around but not doing the curricular activities.

Although the technology in the MUVEES system is sophisticated, the authoring of the
content for the environment required as much or more time as the technology development
phrase of the project. The original material for the students to learn that was developed at
George Mason University attempted to follow United States national and Virgin state
science standards related to the content area of experimental design. However, there was a
change in the state of Virginias content standards that removed experimental design as a
standard, so the project team elected to use disease transmission as the content area. It was
also decided to situate the virtual environment in a 19th century time period to minimize the
students over reliance on background knowledge about the present and to encourage
observations and looking for intervening clues. The project has also developed a non-
computer version of the curriculum to use in control treatment conditions. The project
currently has developed approximately 60 different worlds with different features such as
climate, time of day, and so on. However, only three main treatment worlds will be used as
part of a study being planned to explore the efficacy of different types of situated learning.
As the MUVEES system is still under development, the project team does not have
extensive data or experience with implementation (see section 4.1.2.4 for further
information). However, Dede anticipates that the implementation of multi-user virtual
environment systems could prove valuable for learners who are not motivated to learn a
particular subject. For example, a MUVE system could be used as a front end or initial
learning activity to get students engaged. MUVE systems also might be an option for
students who prefer a more active mode of learning. Another way MUVE systems may be
used is at the end of a unit of study, as an inquiry or project based type of activity. Also,
having the students give presentations on their experiences in a MUVE world would be
motivational as well as help the other students hear about different aspects of a MUVE
world they may not have experienced and/or different interpretations of the data that were
collected.
Given the very sophisticated and cutting edge nature of the technologies being used in the
development of MUVE systems for learning, one would expect that the time to develop a
new MUVE world would be very long and hence, from a commercial perspective, expensive.
However, the experience in the MUVE project suggests that a world can be developed in as
little as a month if one uses the existing Active World library of virtual objects. The
authoring of the scripts for what agents do and say and infusing the world with appropriate
learning content is much more labor intensive, but Dede felt a large research or commercial
development project would create different kinds of tools to assist the development process.
Consequently, it appears the development costs for creating multi-user virtual worlds is likely
to be comparable to the costs of creating other types of Type II and even Type I e-Learning
systems.
The issue of the compatibility of advanced e-Learning systems such as MUVE with
industry standards such as AISS and SCORM was also discussed. Dede indicated that
MUVE systems have design and implementation features that are not compatible with
current industry standards and conventions, but that is to be expected given the advanced
nature of the technologies being used in this area.
Another issue that was discussed concerned the use of MUVE systems, which are
fundamentally Type II e-Learning in their pedagogical design, in didactic educational systems

such as Korea. While in the short term it may not be possible to use MUVE systems in K-12
education in Korea, Dede speculated that the existence of the high bandwidth in Korea
might support a range of interesting possible uses of MUVE technologies. For example, this
type of technology might be used to foster cultural interactions like a Virtual Korea for
Korean and non-Korean avatars to interact and to share and experience cultural differences.
5.2.2.6 Resources
MUVEES project homepage: http://www.gse.harvard.edu/~dedech/muvees/
Active Worlds virtual reality website: http://www.activeworlds.com/

5.2.3 University of Michigan
The participants in the case study at the University of Michigan were
members of the Center for Highly Interactive Computing in Education
(HI-CE). The director of the HI-CE group is the distinguished
computer scientist, Dr. Elliot Soloway (see Figure 36), who has faculty
appointments as a professor in the School of Information, a professor in
the School of Education, and a professor in the Department of
Electrical Engineering and Computer Science.
Figure 36. Dr. Elliot
His research interests involve the use of technology in education and Soloway, University of
developing software that is tailored to the unique needs of different Michigan.
learners. Soloway is a principal investigator on the Center for Learning
Technologies in Urban Schools grant, which four partners: Detroit Public Schools, Chicago
Public Schools, the University of Michigan, and Northwestern University. The focus of the
center's activities is the creation of strategies for embedding and sustaining the use of
computing and communications technologies in the science curriculum at the middle school
level. The center has a four-year, $6 million grant from the U.S. National Science
Foundation. Additional participants in the case study meetings on HI-CE e-Learning
technologies included Dr. Joe Krajcik (Professor of Science Education), Dr. Chris Quintana
(Assistant Professor of Learning Technologies), and Ms. Lisa Scott Holt (graduate research
assistant with the HI-CE group).
Although the HI-CE group has developed a number of highly regarded software programs
over the years, such as the system for students to create scientific models, MODEL-IT
(Jackson et al., 2000), the case study meetings focused on the most recent work by the group
related to developing software tools for handheld computing devices to support science
education.
One example of a recent handheld system developed by the HI-CE group is SKETCHY,
which is a handheld application that allows students to graphically represent their ideas. The
program is a tool that has drawing and animation features that allows the student to create
up to 999 individual frames. The students could then use SKETCHY to organizing their
frames or drawings and then play then sequentially to create an animation. This program
has been used in research both at the University of Michigan and at SRI, International (see
section 5.2.4.1). Another handheld program under development is a video editing program
for Pocket PC devices. This program is intended to be initially used in science inquiry classes
where students would take video clips of experiments or data collection activities using
inexpensive digital video recorders and then edit the clips on the handheld system as part of
their project analysis.
An important issue discussed during the case study visit concerned how to provide learning
scaffolds on handheld devices. In a meeting with Dr. Chris Quintana, a member of the HI-
CE group, the usability factor of the very small screen size on a handheld device had been
found in his research to significantly constrain the types of scaffolds that could be effectively
made available on such devices. Quintana noted his current research is directly focusing on
this issue.

According to Soloway, a general development approach for handheld computers in the HI-
CE group is the 80% rule; that is, try to develop tools that support 80% of what teachers
and students do in a day. One consequence of applying this 80% rule has been the groups
recent interest in the development of multiple small applications that run on handheld
devices, rather than developing large, complex, and difficult to learn applications that do
20% of what teachers and students typically do (i.e., what might be a characterization of
typical e-Learning systems that have been developed for desktop computers). Soloway
suggested that the HI-CE handheld software development work might be described as
designing e-paper and e-pencils and things to think with.
In terms of technical development information, the programming languages for developing
the various handheld systems have been:
Pocket PC devices: Following .NET protocols using C#.
TI calculators: Assembly language.
Palm devices: Palm development platform.
The technical development process typically used one or two undergraduate or graduate
computer science students who were assigned to a particular project. The content
development process also employed science education experts who worked in conjunction
with teachers. The groups also were careful to align the content with state and national
science standards in the United States. The systems were developed and then used in
classrooms where qualitative and quantitative data on the use of the particular system was
collected.
The time to develop initial prototypes varied from project to project: Some small
applications took a month, while others, like the Windows CE video editing program, took a
year for two undergraduate computer science students to complete.
The initial handheld development work was on Palm OS handheld platform. However, new
projects by the HI-CE group are shifting to Windows CE and the Pocket PC devices
because of the support for the .NET architecture. Soloway noted that the .NET
architecture has been designed to be scalable and that the interface guidelines for developing
content are intended to allow the deployment of information across different types of
systems from handheld devices with small screens to larger screen desktop systems.
The meeting with Dr. Chris Quintana discussed research on handheld devices; in particular,
ways to provide the learner with scaffolding when using such systems. One of the challenges
with this area of research is that it is hard to put the scaffolds on the small screens, and so
part of his research is exploring different usability design approaches to deal with this issue.
This raises the possibility that while handheld devices are very well suited for use in learner
centered activities because of their size and affordability, it may be necessary to provide
learning scaffolds outside of the devices themselves, such as in the classroom context the
teacher constructs or on other desktop systems the students might be using in conjunction
with the handheld systems.

The HI-CE group has been using different handheld programs and systems in a variety of
classroom settings. In general, the students and the teachers find the handheld programs and
systems easy to use and effective for the particular learning activities. There are logistical
issues to deal with, such as where to locate the handheld charging stands and the connection
to the main classroom computer to pool data or information collected on the students
individual handheld devices. Also, as might be expected, the students sometimes mess
around or play with the handheld devices that are not on task learning activities.
Dr. Soloway indicated that he believes the Microsoft usability guidelines with the .NET
architecture will become increasingly important to e-Learning developers. The .NET
architecture provides software usability guidelines for different types of computers (e.g.,
handheld devices, desktop computers) and data structure guidelines for backend servers. He
believes the support for scalability across different computer platforms will prove useful for
e-Learning developers who need to create content to be accessed on different types of
systems and for developers of programs for which information collected from one platform
is made available on different platforms.
Another issue that has implications to the e-Learning industry is the place of handheld
devices in educational settings. Whereas researchers such as Soloway believe handheld
devices represent the future of e-Learning in education because of their low cost, ease of use,
and increasing functionality (high resolution color, wireless networking, faster processors),
other researchers in the field are not so sanguine about this view. Indeed one noted e-
Learning researcher has called handheld devices in the classroom paper weights that have
little to offer for enhancing student learning. Whether the future of e-Learning in schools is
complex software with many features and capabilities that runs on larger desktop and laptop
computers or simple single task applications running on small handheld devices is an
important and unanswered question at this time.

5.2.4 Center for Technology in Learning, SRI International
The Center for Technology in Learning (CTL) at SRI International is a nonprofit research
center whose mission is to improve learning and teaching through innovation and inquiry.
CTL research and development activities contribute to the knowledge base on effective
learning and teaching, and embody research insights in the innovative design, use, and
assessment of interactive learning environments. Research expertise related to technology
and learning at the CTL includes:
Evaluation of the interaction between technology and education reform efforts.
Design and support for distributed learning communities.
Strategic uses of handheld computing devices in education.
Technology supports for teaching and learning in mathematics and science.
Uses of technology in assessing science and mathematics learning.
Strategic Learning Consulting in learning sciences strategies for educational
technology firms.
The site visit to the CTL involved meetings with several distinguished researchers, including
Dr. Louise Yarnall, Dr. Edys Quellmalz, Dr. Mark Schlager, Dr. Jeremy Roschelle, Dr. Patti
Schank, and Ms. Tina Stanford, as well as with Co-Director of the CTL, Dr. Nora Sabelli.
Each of these meetings was quite informative about various innovative Type II e-Learning
systems and technology evaluation initiatives that the CTL has been involved with over the
past decade. However, for this case study, the discussion primarily focuses on the Wireless
Handhelds for Improving Reflection on Learning (WHIRL) research as this was a handheld
device project that employed some of the applications developed by the Michigan HI-CE
group. The conversations about this work provide an opportunity to contrast, compare, and
enhance on themes and issue from the Michigan case study.
Dr. Louise Yarnall (see Figure 37), a researcher on the handheld
Project WHIRL, is a Research Social Scientist at the Center for
Technology in Learning. She specializes in evaluation, assessment
design, and communication of educational technology reform and
research to diverse audiences. She is particularly interested in
examining how new technology may be used to improve young
people's skills of reading and analyzing real world data, particularly to
make policy decisions. At SRI, Dr. Yarnall oversees development of
handheld software to support classroom science inquiry and co-
designing the evaluation of the international GLOBE science
Figure 37. Dr. Louise
program. For these and other projects, Dr. Yarnall specializes in
Yarnall, Center for
educational technology design and both design and implementation Technology in
of case studies and surveys. She received her Ph.D. in Education at Learning.
the University of California Los Angeles in 2002.
Project WHIRL was a handheld project funded by the U.S. National Science Foundation
(Yarnall, 2004). The primary goal of this project was to foster formative assessment in the

classroom. This was initially done by working with teachers to support existing formative
assessments and then to explore ways that the handheld devices might enhance or augment
this type of feedback.
The project involved two teachers in South Carolina who taught in a diverse community that
included military families, retirees, as well as a range of students from different social
economic status (SES) backgrounds. Project WHIRL involved a multi-disciplinary design
team that employed a co-design model consisting of a facilitator from SRI, two developers
and technical support personnel, and an assessment specialist. The teams met bi-weekly on
average during the first year.
New software development in Project WHIRL was mainly done by two individuals, with
some assistance from a third programmer. The HCI and interface design considerations for
the project were greatly influenced by the constraints of the small screen on handheld
devices. Many different designs were pilot tested with students and feedback was obtained
from the teachers, thus allowing the projects applications for handheld devices to be
iteratively refined over the course of project. During the first year, the team piloted some of
the software being developed in a paper format before the actual applications were
programmed. Four of handheld applications that were used in the project were discussed in
the interview:
QUIZLER: A commercial program from a company in Colorado.
SKETCHY: Developed by the Michigan HI-CE group.
BOOMERANG: Developed by the Project WHIRL team.
DATA DOERS: Developed by the Project WHIRL team.
The teachers used QUIZLER to administer short quizzes on knowledge covered in different
units that provided formative feedback to the students. The project development team also
made additions to the QUIZLER grade book to support formative assessment and to help
diagnose misconceptions. The teachers would beam quizzes to selected students, who would
in turn start beaming the new quiz to the other students. There also was a collector
handheld that the students would beam their completed quizzes to when they were done.
SKETCHY is a handheld application that allows students to graphically represent their ideas.
The program is a tool that has drawing and animation features that allows the student to
create up to 999 individual frames. The students could then use SKETCHY to organizing their
frames or drawings and then play them sequentially to create an animation.
The idea for BOOMERANG came from QUIZLER, except with this application, it was the kids
who would pose questions as part of inquiry activities. The project team had observed that
the teachers were not actually doing inquiry but rather "hands on" activities. The challenge to
the project personnel was how to foster student questions and inquiry. The project team
then talked to the teachers and suggested that they provide terms to the students and then
have the kids pose questions using BOOMERANG. The teachers were initially skeptical about
this activity, but were later impressed with the quality of the questions that the kids asked. In
addition, the student questions often revealed prior knowledge and misconceptions, and thus
helped the teachers understand areas to focus the teaching on.

When students are engaged in regular laboratory work, they frequently make mistakes with
data, but it is often a few days before the teacher catches the problem. To help address the
lack of real time feedback in this situation, DATA DOER is an application that is intended to
help students collect and enter data correctly in a laboratory activity.
There were frequent technical problems that students and teachers encountered, so an
important aspect of the implementation was the availability of onsite technical support. On
occasion there was lost data when Palms powered off before being hot synched to backup
the data on a desktop computer.
In terms of the classroom use of handheld applications, the research found that the systems
were used about 25% of school year, with two tools per day being used on average. The
middle school teachers liked QUIZLER and BOOMERANG, but not DATA DOERS. It was
found that QUIZLER was used the most often, followed by BOOMERANG and SKETCHY, with
SKETCHY being the most popular with teachers and students. It was used also to get an
indication of the students prior knowledge and misconceptions.12 Teachers were found to
adjust their instruction based on the ideas revealed by the students work in SKETCHY.
There are three main implications of the Project WHIRL research for the e-Learning
industry. First, this project demonstrated the importance of a careful analysis of the teaching
and learning needs in real classrooms in order to effectively design and use e-Learning
technologies, in this case, handheld devices. Second, it is important that on site technical
support for the teachers and students be available in order to insure the successful use of this
type of technology. Third, and perhaps most important, the data collected in this research
demonstrated that small, relatively simple applications that do one or two tasks and that run
on affordable handheld devices can be very effective tools to support formative assessment
and learning in the classroom. This last point also is consistent with the Soloway thesis that
small, task focused applications on handheld devices may be a viable alternative e-Learning
paradigm to the more prevalent e-Learning paradigm that employs large, complex software
applications requiring powerful and expensive desktop or laptop systems to run.
5.2.4.6 Resources
Center for Technology in Learning website: http://ctl.sri.com/index.jsp
Project WHIRL website: http://ctl.sri.com/projects/displayProject.jsp?Nick=whirl
Handheld software information and download websites:
QUIZLER: http://www.pocketmobility.com/quizzler/index.html?105,82
SKETCHY: http://www.goknow.com/
12Teachers felt they could identify misconceptions, but the project team observed that the teachers tended to
just tell the students what was "right" rather than engage in pedagogical techniques that learning science
research has found to be most effective for changing misconceptions (National Research Council, 2000).

5.2.5 Crdu
Crdu, a Korean e-Learning services provider, was founded as an independent corporation
in 2000 four years after its origins as an open education team in the Samsung Human
Resources Development Center in 1996. Crdu currently provides e-learning services to
more than 900 domestic companies, governmental agencies, schools and public institutes
(total number of courses: 400 +, number of paying learners: 200,000 per annum). It has won
numerous awards, including (a) being selected as a superior institute of Internet
communication training by the Ministry of Labor in 2000, (b) receiving a Special Prize in the
Management of the Personnel by Korea Management Association in 2001, (c) being
awarded the grand prize at the e-Learning competition sponsored by the Korean Labor
Department, and (d) being awarded the Prime Minister's Prize at Korea E-Biz Competition
organized by the Korean Industrial Resource Department. In addition, Crdu recently
received the Korean E-Learning System of the Year for an innovative goal based scenario
(GBS) developed by the Content Development Team for the Construction Engineering
Academy of the Samsung Engineering and Construction Company. This GBS was the focus
of the case study meeting discussed below.
Dr. Jo Il-Hyun participated in the
case study. He had directed the
GBS project at Crdu and is now
on the faculty of Computer
Education at Chuncheon National
University of Education. His
current research focuses on e-
Learning and the constructive
instructional design model. He Figure 38. Ms. Kim Hye-Hee and Ms. Jun Young-Sun (Crdu) and
serves on the advisory board of Dr. Jo Il-Hyun (Chuncheon National University of Education).
Crdu. Also participating in the
meeting were two Associators from the Content Development Team at Crdu, Ms. Kim
Hye-Hee and Ms. Jun Young-Sun, who were currently heading up GBS activities at the
company (see Figure 38).
Figure 39. Two screens from Crdu Construction Engineering GBS.

The Crdu GBS was developed for the Construction and Engineering Academy of Samsung
Engineering and Construction Company (see Figure 39). The scenarios employed in this
program were divided into two types: Master Scenario and Sub Scenario. There were also
three types of learning resources for scaffolding learners to perform the given tasks: Tutorial,

Glossaries, and Data (including content from real situations such as a financial reports of
virtual companies, Web-site links to read relevant article printed in the Wall Street Journal,
meeting records, and so on). The GBS was intended to help construction engineers develop
problem-solving skills for real-world problems at their construction sites.
The initial version of the Construction Engineering GBS was developed over a four month
period in 2002. There has also been a recently completed second version of the GBS that
has additional functionality and features.
The impetus for the project began two years ago when the new CEO articulated a vision for
Samsung Construction that changed the company focus from low technology to high
technology construction techniques. In addition, the CEO was very critical of didactic (i.e.,
Type I) approaches for training construction engineers about advanced construction
technologies that did not use realistic content. Dr. Jo approached the CEO about GBS
systems as being a new approach that could be practically and affordably be implemented to
address these concerns. One of the challenges Dr. Jo faced was that neither the developers
nor the construction engineers to be trained had ever experienced an approach such as this.
The initial GBS that was developed in 2002 was primarily text based with graphics. A
formative evaluation was conducted at the beginning of the project to inform the design and
content of the system.
The project experienced two main early problems. First, the researcher who initially
articulated the concept of goal based scenarios, Dr. Roger Schank (see Schank, Fano, Bell, &
Jona, 1994), was more concerned about theoretical aspects of such systems and not very
specific about practical guidelines for actually developing GBS systems. Thus the Crdu GBS
development team had to define their own design guidelines and HCI for a GBS, in
particular, one for application in high technology construction techniques. The second
problem involved logistics and resources for developing the GBS. Given the need in a GBS
to provide as realistic an online environment as possible, the development team had to make
numerous trips and to conduct many interviews to collect realistic information from
workplace (e.g., collecting real forms, talking with subject matter experts [SME]). However,
despite the limited resources, a positive consequence of these interviews was that unlike
typical e-Learning projects in which instructional designers and SMEs are kept apart, in this
case the instructional designers were able to get in touch with real workers and to
consequently develop a much deeper understanding of not only the content but also the
contexts in which the knowledge was to be used.
The new version of the GBS employed more systematic authoring involving CIM (Critical
Incident Method). There were three main steps in the new authoring process. First, in order
to identify good and bad cases to be used in the new GBS, a workshop was held with SMEs
to get stories upon which to base the scenarios in the GBS. Second, interviews were held
with SMEs to obtain more detailed information and ideas about the content and tasks.
Third, the GBS writers and SMEs iteratively work together on the content to ensure its
accuracy.
The main problem that was expected in the implementation might be called the learning
paradigm issue. That is, it was anticipated that some users of the system would expect their
e-Learning experiences to be nicely wrapped and predictable, and that they might not be

comfortable using the less predictable story approach that is fundamental to a GBS.
However, in the evaluation of the GPS, the overall satisfaction was nearly 4 on a 5 point
Likert scale (Jo, 2004), which is quite high for the first time use of an innovative system.
Overall, the construction engineers who used the system found it enjoyable to use and
helpful for learning the material.
This GBS project at Crdu is important for three main reasons. First, it represents one of the
only examples of a learner centered Type II e-Learning system that has been developed by
the Korean e-Learning industry. Second, it is one of the few industry developed e-Learning
projects to actually conduct and report a high quality evaluation of learning associated with
the use of the system (Jo, 2004), which is very important in order for the industry to know
what types of e-Learning systems really are helping learners learn. Third, this project
demonstrates Korean e-Learning developers can in fact create Type II e-Learning systems
that are both educationally effective and of outstanding technical and design quality.
5.2.5.6 Resources
Crdu company website: www.credue.com

6 Discussion
As noted above, this project has attempted to delineate principles for designing and
developing advanced e-Learning systems that employ learner centered pedagogical
approaches, which are referred to as Type II e-Learning systems in this report. The
methodology employed in the project involved a comprehensive literature review to identify
selected exemplary Type II e-Learning systems and a series of case study visits to Type II e-
Learning developers in Korea and the U.S to discuss design, development, and
implementation issues.
The reader may recall that in section 3.3, the following question was asked: What is the
specific value of HCI for developing advanced e-Learning systems? It was then suggested
that there are three possible answers to this question:
1. HCI techniques such as LCD learning scaffolds and UCD usability are not important
in the development of Type II e-Learning systems.
2. HCI techniques such as LCD learning scaffolds and UCD usability are the most
important considerations in the development of Type II e-Learning systems.
3. HCI techniques such as LCD learning scaffolds and UCD usability are employed in
the development of Type II e-Learning systems, but another broader analytical
perspective is more important overall.
In looking at the range of information presented in the literature review examples and in the
case studies, it does not appear that the answer to this question is option 1, that it is not
important. Clearly themes from LCD and UCD were explicitly revealed in nearly all of the
examples and case studies. For example, LCD scaffolding of students as they were learning-
by-doing is illustrated by the thinking types scaffold in the WebCaMILE CSCL system
(section 4.1.4.2) and the conceptual criss-crossing scaffold in Knowledge Mediator
educational hypermedia (section 4.1.2.2). UCD usability is illustrated by the usability testing
that was done involving navigation techniques in the MUVEES virtual worlds for learning
system (section 4.1.3.3) and by the usability testing of BioLogica and Pedagogia based on
user feedback and classroom observations (section 5.2.1.3)
However, option 2HCI techniques are the most important for developing Type II e-
Learning systemsdoes not seem warranted either. A considerable amount of information
obtained in the data gathering portion of the project discussed various features of the
selected Type II e-Learning systems that are not explicitly considered in the LCD and UCD
literatures. For example, several of the systems included in the report stress the importance
of important cognitive factors of the learners or socio-cultural factors in the learning
environment that go beyond design aspects of the technology, such as the Cognitive Tutor
(section 4.1.5.1) and Knowledge Mediator (section 4.1.2.1) projects that focus on cognitive
changes in the learners or WebCaMILE (section 4.1.4.2) and the handheld devices in Project
WHIRL (section 5.2.4) that are concerned with collaborations and the use of technologies in
the overall learning environment. Another important feature in several of the e-Learning
systems was formative assessment for learning, as in the Cognitive Tutor (section 4.1.5.1)
and Project WHIRL (section 5.2.1.2). Thus it appears option 3 is the most appropriate
answer, which is that HCI techniques are useful, but that another perspective is more
important overall. But what should be the characteristics of the analytical framework?
As an alternative to HCI for developing Type II e-Learning systems, this other perspective
should provide a broad theoretical and research basis that is consistent with perspectives

from LCD and UCD techniques and research while also encompassing cognitive and socio-
cultural factors that are outside of the typical computer science focus of HCI. This
framework must also be based on the very best scientific research on learning that is
available, as well as one that has been applied to policy related issues concerning how to
achieve learning goals such as those identified in the Korean Ministry of Education White
Paper (Kim, 2003) that was discussed in the opening section of this report.
It is thus proposed that design principles for Type II e-Learning systems be grounded on
perspectives from the multi-disciplinary science of learning that is increasingly gaining in
prominence as a field of rigorous scientific inquiry (National Research Council, 2000,
2001).The science of learning framework for Type II e-learning systems is discussed in the next
section.
This portion of the report first discusses and illustrates the applicability of the science of
learning framework for Type II e-Learning with examples from the systems included in this
report. Next, a short set of lessons learned related to developing Type II e-Learning systems
are presented that are mainly drawn from the case studies, followed by suggestions for future
initiatives to further explore the development and use of Type II e-Learning systems in
Korea. The report concludes by noting certain systemic challenges to be faced with respect
to implementing learner centered pedagogical approaches such as Type II e-Learning
systems in Korea as well as significant opportunities for affecting educational reform
initiatives.
6.1 Type II E-Learning Systems and Science of Learning Based Design

Principles
In both the literature discussions of Type II e-Learning systems and the case studies, it is
interesting to observe how the designs of the systems were less influenced by the
affordances of a particular technology, such as computer supported collaborative learning or
virtual worlds for learning, than by research and theory related to human learning and
cognition. The science of learning framework for Type II e-learning systems that is presented and
illustrated in this section is drawn from the highly regard volume that was commissioned by
the U.S. National Science Council to survey and distill the best current scientific research
into how people learn and how to assess learning (National Research Council, 2000, 2001). It
is proposed here that the following science of learning principles are reflected collectively
and individually in nearly all of the Type II e-Learning systems reviewed in this report:
1. Provide contextualized learning based on multiple real world problems, cases, and
situations.
2. Provide scaffolds and tools to enhance learning and metacognitive skills.
3. Consider the preconceptions and prior knowledge of the learners engaged in learning
activities.
4. Make the organizing conceptual structure in the subject being learned explicit to the
learner.
5. Provide ways to make the students thinking visible to them and to their teachers.
6. Support learning-by-doing, rather than learning by telling and showing.
7. Foster thoughtful collaborations amongst communities of learners.

8. Provide formative and summative assessment of learning using measures of reasoning,
understanding, complex problem solving, and knowledge transfer, not just in the recall of
factual information.
Examples of each of these eight science of learning principles drawn from the reports
literature review and case studies are provided in turn.
6.1.1 Provide Contextualized Learning
Every e-Learning system discussed in this report attempted to involve the learner in a
technology represented or supported environment that included authentic or realistic
problems, cases, and situations. The use of cases, represented using text, pictures, and
multimedia, is found in the Knowledge Mediator system, the CDC Botulism simulation, and
the Crdu GBS simulation. Another technique for contextualizing the learning involved the
use of computer modeling, as with the BioLogica and Pedagogica systems developed at the
Concord Consortium. Still another approach for providing context is to have the students
use handheld devices as data collection or representational tools in the real world, as in
Environmental Detectives and Project WHIRL, or to construct a virtual world for students
to explore in and manipulate, as in the Harvard MUVEES project. Engagement in active
problem solving is another way to contextualize learning, which is found in all of these
systems, especially in the Cognitive Tutor ITS. By providing learning in meaningful
contexts, students (of all ages) are more motivated to learn as they see the relevance to their
lives of what they are studying. This also provides an additional cognitive benefit of fostering
transfer of learning from school to other situations in their lives.
6.1.2 Provide Scaffolds and Tools
The use of e-Learning technologies to provide either learning scaffolds to support learning
and thinking has been extensively discussed above with respect to learner centered design. A
related notion to scaffolding learning is to provide tools for accomplishing tasks (e.g., Data
Doer in Project WHIRL or features in Environmental Detectives for data collection). This
learning science principle is directly aligned with the LCD principle of scaffolding discussed
in section 3.2.13
6.1.3 Consider the Preconceptions and Prior Knowledge
The importance in the learning and teaching process of understanding the preconceptions
(sometimes call misconceptions) and the prior knowledge students have about a particular
subject area of study is vitally important (National Research Council, 2000). This is one of
the hardest e-Learning design features to implement because it requires fairly extensive
research or teaching experience to uncover the nature of these ideas. However, direct
examples for how e-Learning systems may be developed that explicitly consider
preconceptions and prior knowledge may be found in the research associated with the
Knowledge Mediator educational hypermedia system dealing with student preconceptions
about the processes of evolutionary biology and with cognitive research that formed the
basis of the Cognitive Tutoring systems related to mathematical knowledge involving algebra
and geometry. It should be noted some of the learner centered e-Learning systems do not
13 The credit for the introduction of the concept of scaffolding in the science of learning and in the area of
learning technologies or e-Learning may be attributed to Allan Collins in his seminal papers from the late 1980s
(Brown, Collins, & Duguid, 1989; Collins, 1989; Collins, Brown, & Newman, 1989).

explicitly implement this feature, such as the MUVEES or Environmental Detectives. In
such cases, the pedagogical responsibility for this important learning feature would have to
be provided by the teacher or in some other way in the learning environment.14 One
interesting way this was done was reported in Project WHIRL, where the students used
BOOMERANG to write-up and share their own questions about a particular topic. What the
teachers found was that not only did the students often ask very insightful questions, but
that many times the questions revealed their preconceptions or lack of knowledge in a
subject area, that in turn made it easier for the teacher to provide appropriate instruction.
6.1.4 Make Organizing Conceptual Structure Explicit
One of the distinguishing features of expertise is the well organized conceptual structure of
important or big ideas that experts have (Chi, Glaser, & Farr, 1988; National Research
Council, 2000). One of the most effective ways to ensure that students learn the important
or big ideas in a subject area is to make the conceptual structure explicit to the learner. This
science of learning principle is found in several of the e-Learning systems included in this
report. For example, the Knowledge Mediator educational hypermedia system provides a
pop-up list of important concepts along with specific explanations for how those concepts
apply in different cases, while the Cognitive Tutor employs a skills window that both
shows important mathematical ideas as well as the degree to which the student has
developed particular skills. Another approach to making important conceptual dimensions of
the subject area explicit is to use of tutorials, glossaries, or reference materials on important
ideas, as in the CDC Botulism and the Crdu GBS simulations. One of the more innovative
approaches to for making important ideas clear to the student is the hypermodel approach
in Pedagogica that serves as pedagogical wrapper around a particular modeling system. An
e-Learning developer in Pedagogica would decide which ideas are important to stress during
a particular lesson as students are given various open ended problems to solving that involve
the science modeling system being used, so that, for example, one particular learning activity
might involve the modeling tool to stress ideas in genetics related to the DNA level while
another learning activity might stress genetics ideas at the population level.
6.1.5 Make Thinking Visible
Another important science of learning principle is to make thinking visible (Collins et al.,
1989; National Research Council, 2000). By helping students become aware of their ideas
and to be aware of the ideas of others (including experts), learning can be greatly enhanced.
The systems in this report used different techniques to make thinking visible. For example,
in Biologica, students would set parameters to model different aspects of genetics and
evolution and then could visually see the consequences of their choices on simulated
creatures shown on the screen. The Cognitive Tutor makes extensive use of graphs to
make explicit aspects of the students mathematical thinking, while the Knowledge Mediator
hypermedia system employed modules to provided conceptual visualizations of novice and
expert ideas about evolution. In an intriguing use of a simple tool on a handheld device,
Project WHIRL found teachers would have students use SKETCHY to draw or sketch
individual frames that could then be animated in order to graphically express the students
ideas (i.e., make their ideas visible).
14 For the remaining science of learning principles for e-Learning, not every system in the report implemented
all the principles.

6.1.6 Support Learning-By-Doing
All of the e-Learning systems in the report involved the students in some type of reasonably
authentic or interesting task that required learning-by-doing, rather than learning by telling
and showing (i.e., a Type I approach to e-Learning). One of the approaches to learning-by-
doing that attempted to make the experience as realistic as possible was the Korean Crdu
GBS simulation project, where photos and videos from construction sites, newspaper
articles, actual work forms, interviews with subject matter experts, and so on were all made
available using web-based multimedia. An approach that sacrificed real world fidelity for
that of lesser fidelity in a virtual world, the MUVEES project still involved very active
learning while students were engaged in their River City task. Still another innovative
approach to the use of advanced e-Learning technologies may be seen in the Environmental
Detectives handheld project, which created what the developers call augmented reality.
That is, the students would use the GPS enabled handheld devices to explore the real
world, but would be doing so within the context of the task they were provided (an
environmental crisis at MIT) and with information presented on or obtained through the
handheld device that differed from what was actually available for real.
6.1.7 Foster Thoughtful Collaborations amongst
Communities of Learners
The recognition that human learning occurs in socio-cultural contexts, within particular
communities of practice, is being increasingly accepted in the science of learning (Brown et
al., 1989; National Research Council, 2000). As such, the use of technologies to explicitly
support collaborations between communities of learners has been receiving considerable
attention in the area of Type II e-Learning, as shown in several of the systems discussed in
this report. For example, WebCaMILE is specifically designed as a tool to support and
enhance collaborative interactions in college education in conjunction with particular types
of metacognitive scaffolds. The MUVEES system, while clearly an example of a virtual
world, explicitly supports collaborative interactions between the learners via their avatars in
the virtual space. In addition, both of the projects involving handheld devices, at the
University of Michigan and SRI, International, used the devices in the context of
collaborative learning interactions that involved students sharing and reflecting on each
others work.
6.1.8 Provide Formative and Summative Assessment of
Learning
One of the most important findings in science of learning research is the degree to which
reasonably rapid feedback or formative assessment can enhance students learning (National
Research Council, 2001). In typical classrooms in Korea, that average over 35 students, it is
clearly impossible for a single teacher to provide individualized formative assessment to each
student on a regular basis. However, as several of the projects in this report show, Type II e-
Learning systems can provide tools for such formative assessments in classroom settings.
For example, the Cognitive Tutoring system provides students with real time feedback on
the vast majority of challenges students experience while learning high school algebra and
geometry. Both the Knowledge Mediator and the CDC Botulism simulation have modules
that assess and provide feedback to learners, as does the Pedagogica hypermodeling system.
Project WHIRL identified a number of ways that even relatively simple applications on
handheld devices can be used for formative assessment, as with QUIZLER for quick tests of
understanding factual information and SKETCHY that provided a graphical way to display
students understanding.

However, even for the e-Learning systems that did not directly try to provide assessment of
students understanding by the system, the summative assessments given after using the
systems were generally very substantial. These summative assessments might be novel
problem solving tasks or public discussions of the results of the solutions students came up
with or the projects they were involved with that demonstrated their ability to reason,
understand, solve complex problems, or apply their knowledge to new situations. For
example, learners using the Knowledge Mediator system were tested on their understanding
of evolutionary biology concepts using transfer problems they had not studied using the
system, while students in the MUVEES project shared the solutions they proposed regarding
the causes of illness in River City and justified their solutions based on data they collected.
The summative assessment in the Crdu GBS simulation was not just how well these
construction engineers were judged to do in when using the system, but also how well their
supervisors felt they had learned the knowledge and could use it on the job. Overall, the
Type II e-Learning systems discussed in this report support the kinds of assessments being
recommended in the science of learning literatures. It should also be noted that these
assessments are showing the attainment of advanced learning skills that go far beyond mere
memorization of factual information as is the case with the use of Type I approaches to
instruction and to the design of e-Learning systems.
6.2 Application of Science of Learning Principles for the Design of E-

Learning
In considering ways that science of learning principles may be used to develop Type II e-
Learning systems, the choice of the phrase learning principles should be emphasized.
Given the complexities of learning environments and the interaction of an e-Learning
system with learning in a particular the setting, it is not possible to tell a developer an
abstract recipe of clear, unambiguous steps to follow to create different categories of Type
II e-Learning systems. It must also be stressed that each e-Learning implementation or
development project will always be unique, and the technologies are constantly changing as
well. However, the underlying principles of how people cognitively function and learn, how they
collaboratively interact with each other, how to develop robust user and learner centered
software tools, and so on remain relatively constant.
Indeed, developers who wish to learn how to develop Type II e-Learning systems are just
like any learners and the various science of learning principles discussed above will be
necessary for the acquisition of expertise in this area. That is, novice Type II e-Learning
developers will need to see and experience different examples of Type II e-Learning systems,
they will need to learn by trying to develop such systems, they will need feedback and
support on their work, opportunities to reflect on their work with colleagues, and so on. It
will also be important for e-Learning developers to contrast and compare ways that various
science of learning principles are actually implemented in different categories of Type II e-
Learning systems. For example, the principle of scaffolding is an important one, yet it is
implemented in very different ways in a modeling (or hypermodeling) system like Pedagogica
section, in the virtual environment of MUVEES, in Knowledge Mediator hypermedia, or in
the Crdu goal based scenario. However, perhaps the most fundamental design principle that
developers who are interested in developing Type II e-Learning systems should follow is the
necessity of moving away from a transmissionist model of telling and showing and to instead
embrace a model in which e-Learning technology supports learning-by-doing.

6.3 Lessons Learned Related to the Development of Type II E-Learning
Systems
It was anticipated at the outset that important information regarding the development of
Type II e-Learning systems would be best obtained by discussions with individuals involved
with actual projects as part of case study interviews. The rich sets of information obtain
from these case study interviews are presented in section 5. The purpose of this section is to
highlight five interesting perspectives that were identified in the case studies related to
development and/or implementation of Type II e-Learning systems.
First, it may not be necessary for developers interested in Type II e-Learning systems to
have to develop completely original source code, which is very time consuming and hence
expensive. For example, there are non-profit organizations such as the Concord Consortium
(see section 5.2.1) that develop all of their source code as open-source. This means that
other members of the open-source community may freely use this source code. Further, it is
permitted to make commercial systems based on proprietary adaptations to the open-source
code, but any enhancements to the open source code base are expected be shared with the
rest of the open source community. Within these very flexible parameters, it may be possible
for a Korean e-Learning developer, for example, to take the source code of BioLogica or
Pedagogica and make a commercial Korean version of those systems.15
A second approach to developing Type II e-Learning systems is to use environments that
provide their own programming or scripting languages. For example, in the Pedagogica
system (see section 5.2.1.5), there is a scripting language that allows the construction of
customized scaffolded sequences for students to learn particular concepts while using
different types of modeling tools. Other environments with powerful programming
functionality are the modeling tools NetLogo (http://ccl.northwestern.edu/netlogo/) and
StarLogo (http://education.mit.edu/starlogo/.
One of the more interesting findings to the author of this report was that the e-Learning
system most likely to be regarded as using an advanced type of technology, the MUVEES
project at Harvard, was not using extremely expensive, custom developed software from the
U.S. Department of Defense or the U.S. National Aeronautical and Space Administration
(NASA). Rather, this project is using a commercial system, Active Worlds. Korean e-
Learning developers who are interested in developing virtual worlds should investigate
commercial systems such as Active Worlds that allow the development of a custom virtual
world for around 500,000 won per year and a universe (with large numbers of individual
worlds and support for large numbers of users) in the range of 40,000,000 won per year
(which represents at least an exponential decrease in what it would have cost to develop
virtual systems of this type five to ten years ago).
Fourth, the issue of developing Type II e-Learning systems to be compliant with so called
commercial e-Learning standards such as AISS and SCORM was discussed in the case
studies. None of the case study interviews indicated that their systems were complaint with
current industry standards initiatives or that they planned to be. It was felt that current
industry standards were essentially intended to implement what are referred to in this report
as Type I e-Learning systems rather than to support the functionality of the various
innovative approaches to Type II e-Learning found in the case studies. Only one of the
15The exact legality of commercial versions of open-source systems, particularly in an international context,
should be done with appropriate legal consultation regarding intellectual property issues.

literature reviewed systems, the Knowledge Mediator, had attempted to be implemented
using the SCORM standard. It was found in that KIPA funded project that Knowledge
Mediator design features could not be completely implemented using current SCORM
specifications.
Fifth, and finally, there is compelling evident that the Korean e-Learning industry is quite
capable of developing sophisticated Type II e-Learning systems. The award winning
Construction Engineering simulation system developed by Crdu indicates that the technical
capacity and creativity exists in the Korean e-Learning community to develop sophisticated
and professional Type II e-Learning systems. However, it must be noted that there are
virtually no other examples of Type II e-Learning systems in Korea, either in educational or
industry settings, which this report was able to find and report on.
6.4 Research Limitations

There are certain limitations in the methodology employed in this project. For example,
although every effort was made to insure as comprehensive as possible survey of the
literature related to HCI, LCD, and Type II e-Learning systems, it is not possible to
guarantee that all papers, reports, and book chapters in this area were reviewed. This is
particularly an issue related to information concerning Type II e-Learning systems in
business, industry, and human resource development. It was very difficult to find papers and
reports in this area that met criteria for inclusion in this analysis such as being learner
centered or having an evaluative component that was publicly reported. While it may be that
sophisticated Type II e-Learning systems are being developed in the business, industry, and
HRD areas, the publicly available literature does not document this. Indeed, the few
available reports on these issues suggest that Type II approaches are in fact relatively rare in
the private sector (Dalton et al., 2000).
Another limitation in the methodology employed in this project concerns the criteria for
inclusion of a system in this report. It was decided to only report on selected exemplary
systems rather than to completely discuss all identified systems.16 This decision was made to
provide a more focused report that would leverage the expertise of the project staff in this
area based on direct professional experience with the reported systems and/or with the
research groups involved with their development. However, to avoid possible bias in this
approach, comprehensive references in the particular categories of Type II e-Learning
systems were also provided (e.g., while the Carnegie Tutor system was explicitly discussed in
the report, the major references in the field of intelligent and adaptive systems are also
provided).
6.5 Directions for Future Research and Policy Initiatives

A project such as this, which has attempted to provide a comprehensive set of materials
related to how advanced e-Learning technologies may be designed to support learner
centered pedagogical approaches, is but a first step towards helping the Korean IT industry
develop capabilities in this area of e-Learning. It was clear in discussions with the projects
advisory committee that there is a perceived need for more specific information on how to
actually develop advanced Type II e-Learning systems. However, despite the very
comprehensive literature review and case studies that were conducted in this project, the
available public information that was obtained was at a more general level dealing with
design principles and research findings.
16 This issue was discussed during a mid-project review meeting at the end of August, 2004.

One possible reason for this negative finding (i.e., the lack of specific design guidelines of
Type II e-Learning systems) is that IT systems such as these have only recently been
developed by basic research and university groups in the cognitive sciences, learning
sciences, and computer sciences. Historically there has been a gap between the initial
development of a new technology as part of basic research activities and the larger scale
implementation of that technology (e.g., the Wright brothers are generally credited with
developing the first flying machine in 1903, yet commercial aviation did not really develop
until the middle 1920s). It is thus not surprising that currently there is a gap between the
most advanced ways to develop Type II e-Learning technologies that basic research groups
have been (and are) investigating and what the commercial e-Learning providers are
knowledgeable about and capable of producing. A second reason for the lack of more
specific Type II e-Learning design guidelines is that the individuals and groups who have
developed the specific expertise and skills related to the creation of advanced e-Learning
systems are maintaining a degree of intellectual property control over this information. For
example, two of the case study research groups visited in this project, Concord Consortium
and University of Michigan, have commercial spin off companies that are marketing
commercial versions of the research software they each had developed.
To address these and other issues, Table 6 lists suggestions for possible KIPA initiatives. It
should be noted that these proposed initiatives use approaches that KIPA currently employs,
such funding the development of prototype e-Learning systems as in #1 or as sponsoring
conferences as suggested in #3. However, by shifting the focus from primarily the
technology, as in other KIPA initiatives this author is familiar with, to the support of learner
centered educational and training approaches with advanced technologies, these new
initiatives would clearly be different from previous KIPA requests for proposals involving e-
Learning systems.
7 Conclusion
This report has discussed ways that advanced learning technologies may be designed and
used to help foster powerful and highly effective learner centered or Type II e-Learning
approaches to education and training. Indeed, it may well be that the power of Type II e-
Learning will be first demonstrated in Korea in industry oriented contexts that require
advanced understandings of challenging new knowledge, as in the Crdu project discussed in
this report. This would allow the e-Learning industry in Korea to develop the technical and
design skills necessary to implement innovative Type II e-Learning systems and to seed a
broader awareness for how such approaches represent viable and desirable alternatives to
conventional didactic approaches to training in industry and to teaching in pre-college and
college education.
As noted in the beginning of this report, there are beginning to be calls to transform the
educational system in Korea from a very didactic and teacher centered paradigm to a learner
centered paradigm that fosters advanced learning goals such as problem solving, creativity,
and critical thinking (Kim, 2003). However, at present, the pre-college educational system in
Korea is dominated by the college entrance examination system that is primarily a test of
memorization of declarative information and certainly not a test of cognitive problem
solving skills or critical thinking. As such, e-Learning developers who are interested in
creating Type II learner centered tools for the pre-college education market are certainly to
be commended, but caution is advised until the Korean educational system aligns its high
stakes examinations with procedures that actually assess the kinds of higher ordered
knowledge and skills cultivated by the Type II e-Learning approaches advocated in this and
other reports (National Research Council, 2001). On the positive side, however,

development and appropriate use of rich and interesting Type II e-Learning technologies will
certainly be an essential component for implementing the Ministry of Educations goal for a
new learner centered educational paradigm to help prepare Korean students and citizens for
the emerging knowledge economy and the many challenges of the 21st century.

Table 6. Suggested Initiatives to Advance Type II E-Learning in Korea
Suggested Initiative Comments
1. Sponsor a program that would: A program such as this would motivate both university
research teams and e-Learning companies (and hopefully
a. Fund the development of prototype Type II e- partnerships between universities and e-Learning
Learning systems in the categories discussed in companies) to actually develop Korean Type II e-Learning
this report for both education (middle school systems. Given the majority of the systems identified in
through college) and industry/HRD (see Table this report have been developed in the U.S. and Europe,
4). a program such as this could provide needed Korean
versions of different categories of Type II e-Learning
b. Require successful proposals to participate in systems. Also, once these systems are developed, they
one day workshop at the conclusion the grant could then be used to demonstrate the potential of learner
period to discuss design and development centered approaches that are mediated with technology.
issues and to demonstrate the developed In addition, specific information about how to develop
Typed II e-Learning systems. Type II e-Learning systems can only be obtained by
actually making these systems, hence the need for the
end of grant workshop specified in (1b) where the
different projects would share their insights and
experiences with each other and with the Korean e-
Learning development community more generally.
2. Sponsor advanced e-Learning curriculum Currently, no Korean universities offer a coordinated set
development projects for universities with graduate of courses that prepare Korean masters and Ph.D.
programs that prepare students for the e-Learning students to design, develop, and implement advanced
field. These projects should develop courses and Type II e-Learning systems. Students in the U.S. are able
technology laboratories so that graduate students to obtain this type of training in new types of courses now
could not only read about categories of Type II e- being offered at leading U.S. universities in what are often
Learning, but also use, design, and develop such called Learning Sciences programs that combine
systems. cognitive science, computer science, education, and
research perspectives. By funding an advanced e-
Learning curriculum development initiative, KIPA could
help motivate universities to move more quickly to
develop the necessary academic programs to prepare
students for careers in a world class e-Learning industry
in Korea and as researchers in this important new field.
3. Sponsor a 2 or 3 day international conference on the A conference of this type could be valuable for both
design of advanced learner centered technologies practical and policy issues related to the Korean e-
with invitations to a selected group of internationally Learning industry and its impact on education and
regarded researchers, learning scientists, computer training/professional development in Korea. The explicit
scientists, and e-Learning industry developers to focus on learner centered uses of e-Learning
discuss design, development, and implementation technologies would be aligned with Ministry of Education
issues for a Korean e-Learning industry and efforts to foster educational reform using ICT. This focus
education audience. would also provide an opportunity to discuss the lack of
alignment of Type II e-Learning systems with current
educational practices in Korea and most Asia Pacific Rim
countries.
4. Sponsor KIPA awards for innovative Type II Korean A program that would provide awards (perhaps given at a
e-Learning systems. special banquet) for innovative Korean Type II e-Learning
systems could be a relatively low cost method to both
acknowledge outstanding work in this area as well as to
publicly link advanced ways to learn (i.e., learner
centered approaches) with advanced ways to develop e-
Learning technologies.

8 Acknowledgements
This project was supported by the Korean IT Industry Promotion Agency, although the
information and opinions expressed in this report are not necessarily endorsed or accepted
by the funding agency. The author of this report gratefully acknowledges the contributions
of research associates Ms. Kim Hyungshin and Ms. Lee Jihyun and research assistants Ms.
Kim Youngmi, Ms Park Soyoun, and Mr. Kwon Seungil; without their hard research work,
patience, and linguistic skills this bilingual project and report would not have been possible.
Also, special thanks are owed to the individuals who contributed their time and shared their
expertise in the case studies.

9 References
Anderson, J. (1996). ACT: A simple theory of complex cognition. American Psychologist, 51,
355-365.
Anderson, J., Corbett, A., Koedinger, K., & Pelletier, R. (1995). Cognitive tutors: Lessons
learned. Journal of the Learning Sciences, 4(2), 167-207.
Anderson, J., & Gluck, K. (2001). What role do cognitive architectures play in intelligent
tutoring systems? In D. Klahr & S. Carver (Eds.), Cognition & Instruction: Twenty-five
years of progress (pp. 227-262). Mahwah, NJ: Lawrence Erlbaum Associates.
Anderson, J. R. (1983). The architecture of cognition. Cambridge, MA: Harvard University Press.
Anderson, J. R., Conrad, F. G., & Corbett, A. T. (1989). Skill acquisition and the LISP tutor.
Cognitive Science, 13, 467-505.
Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995). Cognitive tutors:
Lessons learned. Journal of the Learning Sciences, 4(2), 167-207.
Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning.
Educational Researcher, 18, 32-42.
Brusilovsky, P. (2001). Adaptive Hypermedia. User Modeling and User Adapted Interaction, 6(2-
3), 87-110.
Carbonell, J. R. (1970). AI in CAI: An artificial intelligence approach to computer-assisted
instruction. IEEE Transactions on Man-Machine Systems, 11(4), 190-120.
Chan, C. K. K., & Aalst, J. V. (2004). Learning, assessment and collaboration in computer-
supported environments. In What we know about CSCL and implementing it in higher
education (pp. 87 - 112). Norwell, MA: Kluwer Academic Publishers.
Chi, M. T. H., Glaser, R., & Farr, M. J. (Eds.). (1988). The nature of expertise. Hillsdale, NJ:
Lawrence Erlbaum Associates.
Clyde, L. (2004). Computing in the palm of your hand. Teacher Librarian, 31(3), 42-44.
Colella, V. (2000). Participatory simulations: Building collaborative understanding through
immersive dynamic modeling. The Journal of the Learning Sciences, 9(4), 471-500.
Collins, A. (1989). Cognitive apprenticeship and instructional technology (Technical No. 474).
University of Illinois at Urbana-Champaign, Center for the Study of Reading.
Collins, A. (1996). Design issues for learning environments. In S. Vosniadou, E. DeCorte, R.
Glaser & H. Mandl (Eds.), International Perspectives on the Design of Technology- Supported
Learning Environments (pp. 347-361). Mahwah, NJ: Lawrence Erlbaum Associates.
Collins, A., Brown, J., & Newman, S. (1989). Cognitive apprenticeship: Teaching the crafts
of reading, writing, and mathematics. In L. Resnick (Ed.), Knowing, learning, and
instruction (pp. 453-494). Hillsdale, NJ: Lawrence Erlbaum Associates.
Dalton, J. P., Manning, H., Hagen, P. R., Paul, Y., & Tong, J. (2000). Online training needs a new
course. Cambridge, MA: Forrester Research.
Dede, C. (1995). The evolution of constructivist learning environments: Immersion in
distributed, virtual worlds. Educational Technology, 35(5), 46-52.
Dede, C., & Ketelhut, D. (2003). Designing for motivation and usability in a museum-based multi-user
virtual environment. Retrieved from
http://muve.gse.harvard.edu/muvees2003/documents/DedeKetelMUVEaera03fina
l.pdf.
Dede, C., Nelson, B., & Ketelhut, D. (2004). Design-based research on gender, class, race, and
ethnicity in a multi-user virtual environment. Paper presented at the annual meeting of the
American Educational Research Association, San Diego, CA.
Dede, C., Nelson, B., Ketelhut, D., Clarke, J., & Bowman, C. (2004). Design-based research
strategies for studying situated learning in a multi-user virtual environment. Paper presented at
the International Conference of Learning Science 2004, Los Angeles, CA.

Dede, C., Salzman, M., & Loftin, R. (1996). Early research results from using virtual reality to
enhance science education. Paper presented at the annual meeting of the American
Educational Research Association, Chicago, IL.
Dede, C., Salzman, M., Loftin, R. B., & Ash, K. (2000). The design of immersive virtual
learning environments: Fostering deep understandings of complex scientific
knowledge. In M. J. Jacobson & R. B. Kozma (Eds.), Innovations in science and
mathematics education: Advanced designs for technologies of learning (pp. 361-413). Mahwah,
NJ: Lawrence Erlbaum Associates.
Dillon, A., & Gabbard, R. (1998). Hypermedia as an educational technology: A review of the
quantitative research literature on learner comprehension, control, and style. Review of
Educational Research, 68(3), 322-349.
Gathany, N., & Stehr-Green, J. (2003). Scenario-based e-learning model: A CDC case study.
Learning Circuits. Retrieved from
http://www.learningcircuits.org/2003/apr2003/gathany.htm.
Goldman, S., Pea, R., Maldonado, H., Martin, L., & White, T. (2004). Functioning in the wireless
classroom. New York: IEEE Press.
Guzdial, M. (1997). Information ecology of collaborations in educational settings: Influence
of tool. In R. Hall, N. Miyake & N. Enyedy (Eds.), Proceedings of Computer-Supported
Collaborative Learning '97 (pp. 83-90). Toronto, Ontario, Canada.
Guzdial, M., Kolodner, J. L., Hmelo, C., Narayanan, H., Carlson, D., Rappin, N., et al.
(1996). Computer support for learning through complex problem-solving.
Communications of the ACM, 39(4), 43-45.
Guzdial, M., & Turns, J. (2000). Computer-supported collaborative learning in engineering:
The challenge of scaling-up assessment. In M. J. Jacobson & R. B. Kozma (Eds.),
Innovations in science and mathematics education: Advanced designs for technologies of learning.
Hickey, D. T., Wolfe, E. W., & Kindfield, A. C. H. (2000). Assessing learning in a
technology-supported genetics environment: Evidential and consequential validity
issues. Educational Assessment, 6(3), 155-196.
Horwitz, P., & Christie, M. A. (2000). Computer-based manipulatives for teaching scientific
reasoning: An example. In M. J. Jacobson & R. B. Kozma (Eds.), Innovations in science
and mathematics education: Advanced designs for technologies of learning (pp. 163-191).
Inkpen, K. (1999). Designing handheld technologies for kids. Personal Technologies Journal, 3(1
& 2), 81-89.
Jacko, J. A. S., A. (2003). The human-computer interaction handbook. New Jersey: Lawrence
Erlbaum Associates.
Jackson, S., Krajcik, J., & Soloway, E. (2000). Model-It: A design retrospective. In M. J.
Jacobson & R. B. Kozma (Eds.), Innovations in science and mathematics education: Advanced
designs for technologies of learning (pp. 77-115). Mahwah, NJ: Lawrence Erlbaum
Associates.
Jacobson, M. J., & Archodidou, A. (2000). The design of hypermedia tools for learning:
Fostering conceptual change and transfer of complex scientific knowledge. The
Jacobson, M. J., & Kozma, R. B. (Eds.). (2000). Innovations in science and mathematics education:
Advanced designs for technologies of learning. Mahwah, NJ: Lawrence Erlbaum Associates.
Jacobson, M. J., Maouri, C., Mishra, P., & Kolar, C. (1996). Learning with hypertext learning
environments: Theory, design, and research. Journal of Educational Multimedia and
Hypermedia, 5(3/4), 239-281.

Jacobson, M. J., & Spiro, R. J. (1994). A framework for the contextual analysis of
technology-based learning environments. Journal of Computing in Higher Education, 5(2),
3-32.
Jacobson, M. J., & Spiro, R. J. (1995). Hypertext learning environments, cognitive flexibility,
and the transfer of complex knowledge: An empirical investigation. Journal of
Educational Computing Research, 12(5), 301-333.
Jipping, M., Dieter, S., Krikke, J., & Sandro, S. (2001). Using handlheld computers in the
classroom: Laboratories and collaboration on handheld machines. In Proceedings of the
2001 SIGCSE Technical symposium, SIGCSE Technical Bulletin (Vol. 33, pp. 169-173).
Charlotte, NC.
Jo, I. H. (2004). A study on factors that influence learners' performance in GBS-based e-Learning
environment. Unpublished manuscript.
Kim, Y. C. (2003). Adapting Education to the Information Age: A White Paper. Seoul, Korea:
Ministry of Education & Human Resources Development, Korea Education &
Research Information Service. Retrieved from
http://www.keris.or.kr/english/pdf/2003-WhitePap.pdf.
Klopfer, E., Squire, K., & Jenkins, H. (2002). Environmental Detectives: PDAs as a window
into a virtual simulated world. In Proceedings of the IEEE International Workshop on
Wireless and Mobile Technologies in Education (WMTE 02. Vxj, Sweden.
Klopfer, E., Yoon, S., & Rivas, L. (in press). Comparative analysis of palm and wearable
computers for participatory simulations. Journal of Computer Assisted Learning.
Koedinger, K., Corbett, A., Ritter, S., & Shapiro, L. (2000). Carnegie learning's cognitive tutor:
http://www.carnegielearning.com/research/research_reports/CMU_research_result
s.pdf.
Koedinger, K., Corbett, A., Ritter, S., & Shapiro, L. (2000). Carnegie Learning's Cognitive Tutor:
http://www.carnegielearning.com/research/research_reports/CMU_research_result
s.pdf.
Koschmann, T. (1996). CSCL: Theory and practice on an emerging paradigm. Mahwah, NJ:
Lawrence Erlbaum Associates. (CSCL is an abbreviation for Computer Supported
Collaborative Learning).
May, M. (2003). Hand-on power. The Scientist, 17(22), 1-4.
National Research Council. (2000). How people learn: Brain, mind, experience, and school. (Expanded
Edition). J. D. Bransford, A. L. Brown, R. R. Cocking, & S. Donovan, (Eds).
Committee on Developments in the Science of Learning and Committee on
Learning Research and Educational Practice. Washington, DC: National Academy
Press.
National Research Council. (2001). Knowing what students know: The science and design of
educational assessment. J. Pellegrino, N. Chudowsky, & R. Glaser (Eds). Committee on
the Foundations of Assessment, Board on Testing and Assessment, Center for
Education, Division on Behavioral and Social Sciences and Education, National
Research Council. Washington, D.C.: National Academy Press.
Nielsen, J. (1993). Usability engineering. Boston: Academic Press.
Norman, D. A., & Draper, S. W. (Eds.). (1986). User centered system design: New perspectives on
human-computer interaction. Hillsdale, NJ: Lawrence Erlbaum Associates.
Pea, R. D. (1994). Seeing what we build together: Distributed multimedia learning
environments for transformative communications. Journal of the Learning Sciences, 3(3),
285-299.

President's Committee of Advisors for Science and Technology (PCAST). (1997). Report to
the President on the use of technology to strengthen K-12 education in the United States.
Washington, D.C.: Office of Science and Technology Policy, Executive Office of the
President. Retrieved from http://www.ostp.gov/PCAST/k-12ed.html.
Psotka, J. (1994). Immersive tutoring systems: Virtual reality and education and training. Retrieved
from http://198.97.199.60/its.html.
Relan, A., Guiton, G., Parker, N., & Wilkerson, L. (2003). A hand-held requirement for clinical
education: Effectiveness of a systemic educational intervention in altering perception and use of
handhelds among medical students. Paper presented at the annual meeting of the
American Educational Research Association, Chicago, IL.
Roblyer, M. D., Castine, W. H., & King, F. J. (1988). Assessing the impact of computer-based
instruction: A review of recent research. New York: Haworth Press.
Roschelle, J., & Pea, R. (2003). Unlocking the learning value of wireless mobile devices.
Journal of Computer Assisted Learning, 19(3), 260-272.
Scardamalia, M., & Bereiter, C. (1994). Computer support for knowledge-building
communities. The Journal of the Learning Sciences, 3(3), 265-283.
Schank, R. C., Fano, A., Bell, B., & Jona, M. (1994). The design of goal-based scenarios. The
Shapiro, A., & Niederhauser, D. (2003). Learning from hypertext: Research issues and
findings. In D. H. Jonassen (Ed.), Handbook of Research for Education Communications and
Technology, 2nd ed. (2nd ed.). Mahwah, NJ: Lawrence Erlbaum Associates.
Shneiderman, B. (1998). Designing the user interface: Strategies for effective Human-Computer
Interaction (3rd ed.). MA: Addison-Wesley.
Sleeman, D. H., & Brown, J. S. (1982). Introduction: Intelligent tutoring systems. In D. H.
Sleeman & J. S. Brown (Eds.), Intelligent tutoring systems (pp. 1-11). London: Academic
Press.
Soloway, E., Guzdial, M., & Hay, K. E. (1994). Learner-centered design: The challenge for
HCI in the 21st century. Interactions, 1(2), 36-38.
Soloway, E., & Hsi, S. (1998, April). Learner-centered design: Addressing, finally, the unique needs of
learners. Paper presented at the CHI 98 conference summary on human factors in
computing systems, Los Angeles, CA.
Soloway, E., Jackson, S. L., Klein, J., Quintana, C., Reed, J., Spitulnik, J., Stratford, S. J., et al.
(1996). Learning theory in practice: Case studies of learner-centered design. In M. J.
Trauber (Ed.), CHI96 Conference Proceedings (pp. 189-196). Vancouver, British
Columbia, Canada.: University of British Columbia.
Spiro, R. J., Feltovich, P. J., Jacobson, M. J., & Coulson, R. L. (1995). Cognitive flexibility,
constructivism, and hypertext: Random access instruction for advanced knowledge
acquisition in ill-structured domains. In L. Steffe & J. Gale (Eds.), Constructivism in
education. Mahwah, NJ: Lawrence Erlbaum Associates.
Squire, K. (2003). Design principles of next-generation digital gaming for education.
Educational Technology, 43(5), 1-14.
Taylor, R. (1980). The computer in the school: Tutor, tool, tutee. New York: Teachers College Press.
Tergan, S. O. (1997). Conceptual and methodological shortcomings in
hypertext/hypermedia design and research. Journal of Educational Computing Research,
16(3), 209-235.
Tinker, R., & Krajcik, J. (2001). Portable technologies: Science learning in Context. New York:
Kluwer Academic/Plenum Publishers.
VanLehn, K., Siler, S., & Murray, C. (2003). Why do only some events cause learning during
human tutoring? Cognition and Instruction, 2(3), 209-249.

Vosniadou, S., DeCorte, E., Glaser, R., & Mandl, H. (Eds.). (1996). International perspectives on
the design of technology-supported learning environments. Mahwah, NJ: Lawrence Erlbaum
Associates.
Wenger, E. (1987). Artificial intelligence and tutoring systems. Los Altos, CA: Morgan Kaufmann.
Wilensky, U., & Stroup, W. (1999). Learning through participatory simulations: Network-based design
for systems learning in classrooms. Paper presented at the Computer Supported
Collaborative Learning Conference (CSCL '99), Palo Alto, CA.
Winn, W. (in press). Learning in artificial environments: Embodiment, embeddedness and
dynamic adaptation.
Winn, W., Windschitl, M., & Hedley, N. (2001). Learning science in an immersive virtual
environment. Paper presented at the American Educational Research Association
(AERA) Annual Meeting 2001, Seattle, WA.
Winn, W., Windschitl, M., & Hedley, N. (2001). Using immersive visualizations to promote the
understanding of complex natural systems: Learning inside virtual Puget Sound. Paper presented
at the annual meeting of the National Association for Research on Science Teaching
(NARST), St. Louis, MO.
Yarnall, L., & Penuel, W. R. (2004). Designing handheld software to support classroom assessment.
Paper presented at the at the Annual Meeting of the American Educational Research
Association, San Diego, CA.
Zurita, G., & Nussbaum, M. (2004). Computer supported collaborative learning using
wirelessly interconnected handheld computers. Computers & Education, 42, 289-314.

10 Appendix
KIPA Case Study Semi-Structured Interview Questions
1. Summarize the main goals for the project.
2. What were/are your responsibilities for the project?
3. What were the design goals for the technology system?
4. How large are the technical development and content development groups?
5. What programming language was used to develop the systems?
6. Describe the technical development process that was employed.
7. Describe the content development process that was employed.
8. How long did it take to develop the initial prototype?
9. What usability testing was conducted?
10. What did the initial usability testing reveal and what changes were made in the system?
11. Describe the issues you encountered in the initial use of the system in the environment
for the targeted group of learners:
12. Expected uses of the technology
13. Unexpected uses of the technology
14. Describe any changes made in the system and the programming or content development
that was required to make these changes.
15. Describe the current status of the system in terms of its technical development.
16. Describe the current status of the system in terms of its content development.

17. Describe planned technical changes in the system and the expected programming or
content development efforts that will be required.
18. What have been the main research findings to date?
19. Are you aware of the technical specifications of currently proposed industry standards
for e-Learning systems and LCMS?
20. Does the e-Learning system developed for this project comply with current industry
standards for LMC and LCMS?
21. If not, are there plans to make the system standards compliant?
22. If there are no plans to make the system standards compliant, what plans are there to
make the technology system developed in this project more widely available?

Note1 Advance 21st Century E-Learning - Soft

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Note1 Advance 21st Century E-Learning - Soft

Uploaded by

Copyright:

Available Formats

Executive Summary

KIPA FINAL REPORT: 21ST CENTURY E-LEARNING ii

Executive Summary ....................................................................................................................................ii

2 Literature Review Methodology .......................................................................................................7

4 Representative Exemplary Type II E-Learning Systems ............................................................10

KIPA FINAL REPORT: 21ST CENTURY E-LEARNING iii

5 Type II E-Learning Case Studies....................................................................................................41

KIPA FINAL REPORT: 21ST CENTURY E-LEARNING iv

KIPA FINAL REPORT: 21ST CENTURY E-LEARNING v

Figure 1. Cover to the Ministry of Education White Paper.

KIPA FINAL REPORT: 21ST CENTURY E-LEARNING 1

1.1 Type I and Type II E-Learning Systems

Table 1. E-Learning Types and Characteristics

Delivery- Factual information, basic

Problem-based Learning with Conceptual understanding,

KIPA FINAL REPORT: 21ST CENTURY E-LEARNING 2

Table 2. Pedagogical Dimensions of Type I and Type II E-Learning

Type I E-Learning Type II E-Learning

Learning Mechanism Showing and Telling Learning-by-Doing

Cognitive Tools, Scaffolding

Realistic Texts and Multimedia

Actively Engaged in Problem

Control Technology Learner

Achievement on Objective Ability to Solve New, Often Open

Representative of Type I e-learning systems in Korea are the VOD (Video-On-Demand)

KIPA FINAL REPORT: 21ST CENTURY E-LEARNING 3

Figure 3. A summary screen of main points from the lecture.

Figure 4. A typical screen testing for memorization of content.

KIPA FINAL REPORT: 21ST CENTURY E-LEARNING 4

Figure 5. University business course lecture delivered using VOD.

KIPA FINAL REPORT: 21ST CENTURY E-LEARNING 5

1.2 Project Goals and Major Content

KIPA FINAL REPORT: 21ST CENTURY E-LEARNING 6

KIPA FINAL REPORT: 21ST CENTURY E-LEARNING 7

E-Learning Literature Representative Databases

ERIC, ACM digital library, AACE

ERIC, PSYINFO, Google, Problem-based, case-based, transfer, scaffolding,

Industry and Human AACE, AAOU, AECT, ASTD,

3 Human-Computer Interaction and E-Learning: An Overview3

KIPA FINAL REPORT: 21ST CENTURY E-LEARNING 8

3.1 HCI and User Centered Design

3.2 From HCI to Learner-Centered Design

KIPA FINAL REPORT: 21ST CENTURY E-LEARNING 9

3.3 HCI and Type II E-Learning

4 Representative Exemplary Type II E-Learning Systems

KIPA FINAL REPORT: 21ST CENTURY E-LEARNING 10

Category of Type II E-Learning Education Business/HRD Report Section

1. Educational Hypermedia Knowledge Mediator Discussion

Crdu Goal Based

Discussion & Case

4. Computer Supported Discussion

5. Intelligent Systems Cognitive Tutor Discussion

Project WHIRL Case Study

KIPA FINAL REPORT: 21ST CENTURY E-LEARNING 11

KIPA FINAL REPORT: 21ST CENTURY E-LEARNING 12

KIPA FINAL REPORT: 21ST CENTURY E-LEARNING 13

KIPA FINAL REPORT: 21ST CENTURY E-LEARNING 14

KIPA FINAL REPORT: 21ST CENTURY E-LEARNING 15

KIPA FINAL REPORT: 21ST CENTURY E-LEARNING 16

KIPA FINAL REPORT: 21ST CENTURY E-LEARNING 17

Figure 10. A screen shot of the River Worlds environment.

KIPA FINAL REPORT: 21ST CENTURY E-LEARNING 18

KIPA FINAL REPORT: 21ST CENTURY E-LEARNING 19