Overview of Activities for a Summative Evaluation Trial

Planning
Preparing
Implementing / Collecting
Data
Design evaluation
Obtain instruments
Set schedule for
instruction and testing
Create /modify syllabus
Outcomes analysis
Describe resources,
facilities, equipment
needed
Describe ideal entry
behaviors/characterist
ics of target group
Describe number of
groups and individuals
needed
Management analysis
Describe skills/
capabilities of
instructors or
managers
Describe number of
instructors needed

Obtain resources,
facilities, equipment

Adequate? (Observation,
Interview, Questionnaire)

Select sample
Verify entry behaviors
(data)

Learner performance?
(Pre-/Posttests)
Learner attitudes?
(Observation, Interview,
Questionnaire)
Learner performance in job
context (use, effect?)

Schedule learners

Select instructors
Verify skills
Schedule instructors

Validity of implementation?
Modifications?
(Observation, Interview)

Provide training for

instructors

Training effective?
(Observation, Interview)

Plan and develop any

training needed for
instructors/managers

Table 12.1

Summarizing and Analyzing

Data

Reporting Results
Describe limatations of
design

Describe problems by
resources, facilities, and
equipment
Item-by-objective analysis
for group and individual

Recommendation and
rationale
Explanation,
recommendations,
rationale for learner,
job and organization
impact

Cross-test summary by
group and individual by
objective
Attitude summary
Describe problems by
instructors by objective

Describe implementation
problems by objective and
activity

Recommendations and
rationale

Recommendations and
rationale

A Comparison of Formative and Summative Evaluation

Formative Evaluation
Summative Evaluation
Purpose
Locate weaknesses in instruction
Document strengths and
in order to revise it
weaknesses in instruction in
order to decide whether to
maintain or adopt it
Phases or stages

One-to-one
Small-group
Field trial

Expert judgment
Field trial

Instructional
Development
History

Systematically designed in-house

and tailored to the needs of the
organization

Produced in house or
elsewhere not necessarily
following a systems approach

Materials

One set of materials

One set of materials or

several competing sets

Position of
Evaluation

Member of design and

development team

Typically an external evaluator

Outcomes

A prescription for revising

instruction

A report documentingthe
design, procedures, results,
recommendation and rationale
Table 12.2

Congruence Analysis Information Summary Form

Statements of
Candidate
Candidate materials
Organizations
materials (Set 1)
(Set 2)
Characteristics
Organizations in
Stated goals and
Etc.
structional needs
objective in
(goals and main
materials
objectives)

Candidate materials
(Set 3)
Etc.

Entry behaviors of
organizations target
group

Stated entry
behaviors for
learners

Characteristics of
organizations target
group

Stated
characteristics orf
learners and
contexts

Characteristics of
performance
contexts
Organizations
resources available
for obtaining and
implementing
instruction
Organizations
facilities and
equipment available
for implementing
instruction

Cost of purchasing
and implementing
materials

Etc.

Etc.

Etc.

Etc.

Etc.

Etc.

Etc.

Etc.

Facilities required
to implement
materials (learning
centers,
equipment)

Table 12.3
Summative Evaluation Question from Principles of Motivation (Attention,
Relevance, Confidence, Satisfaction)
ARCS
Question Areas for Summative
Instruction
Instruction
Motivation
Evaluation
Model
1
2
Etc.
YES
NO
YES
NO
YES
NO
Attention :
1. Are strategies used to
gain and maintain the
learners attention
(e.g., emotional or
personal appeals,

Relevance :

Confidence
:

Satisfactio
n:

questions, thinking
challenges, human
interest examples, etc.)?
2. Is the instruction relevant
for given target groups
and how are learners
informed and convinced of
the relevance (e.g.,
information about new
requirements for
graduation, certification,
employment,
advancement, selfactualization,etc.)?
3. Are learners likely to be
confident at the outset
and throught-out
instruction so that they
can succed (e.g., learners
informed of purposes and
likely to possess
prerequisites; instruction
progresses from familiar
to unfamiliar, concrete to
abstract; vocabulary,
contexts and scope
appropriate; chalanges
present but realistic;
etc.)?
4. Are learners likely to be
satisfied from the learning
experience (e.g., relevant
external rewards such as
free time employment,
promotion, recognition;
actual antrinsic rewards
such as feelings of

success, accomplishment,
satisfaction of curiosity,
intellectual entertainment,
etc.)?
Table 12.4

Instruction
Intellectual Skills

1
YES

NO

Instructio
n
2
YES NO

Etc.
YES NO

Instructio
n
2
YES NO

Etc.
YES NO

1. Are learners reminded of prerequisite knowledge

they have stored in memory?
2. Are links provided in the instruction between
prerequisite skills stored in memory and new skills?
3.
Instruction
II. Verbal Information

1
YES

1. Is new information presented in a relevant context?

2. Are strategies provided for linking new information
to related information currently stored in memory
(e.g., presentation of familiar analogies, request for
learners to imagine something or to provide
examples from their own experiences)?
3.

NO

Instruction
III. Attitudes

1
YES

NO

Instructio
n
2
YES NO

Etc.
YES NO

Instructio
n
2
YES NO

Etc.
YES NO

1. Are the desired feelings clearly describes or

inferred?
2. Are the desired behaviors clearly describes or
inferred?
3.
Instruction
IV. Motor Skills

1
YES

1. Does the instruction address similar skills the

learner can already perform?
2. Does the instruction include a visual presentation
of the motor skill that illustrates is sequence and
timing?
3.
Table 12.5

NO

Checklist for Evaluation the Learning Component of the Instruction Strategies in Candidate
Materials
Learning Component
Candidate 1
Candidate 2
Candidate 3
Prese Attenti Prese Attenti Prese Attenti
nt
on
nt
on
nt
on
Preinstructional
A. Initial motivation
B. Objectives
C. Entry behaviors
1. Describe
2. Sample items
II. Information Presentation
A. Organizational structures
1. Headings
2. Tables and Illustrations
B. Elaborations
1. Analogies/synonyms
2. Prompts to imagine /
consider
3. Examples and nonexamples
4. Relevans characteristics of
examples
5. Summaries / review
III. Learners Participation
A. Relevant practice
B. Feedback
1. Answers
2. Example solutions
3. Common errors and mistakes
IV. Follow-Through Activities
A. Memory aids
B. Transfer strategy
TOTAL
Table 12.6