ASSESSMENT STRATEGIES

 RUBRIC = SCORING TOOLS: help students to focus on content and to

guide them in developing presentations, written and oral reports.
 CONCEPT MAPS: assist students in “seeing the big picture”.
 PORTFOLIOS: document student learning and improve student
metcognition.
 CONCEPT TESTS: conceptual multiple-choice questions useful in large
classes.
 KNOWLEDGE SURVEY: students answer whether they could answer a
survey of course content questions.
 EXAMS.
 ORAL PRESENTATIONS.
 PEER REVIEW: having students assess themselves and each other.
 WRITTEN REPORTS.

WHY USE THE RUBRICS?

Has a student ever said to you regarding an assignment, "But, I didn't know what you
wanted!" or "Why did her paper get an 'A' and mine a 'C?'" Students must understand
the goals we expect them to achieve in course assignments, and importantly, the
criteria we use to determine how well they have achieved those goals. Rubrics provide
a readily accessible way of communicating and developing our goals with students and
the criteria we use to discern how well students have reached them.

WHAT IS A RUBRIC?
Rubrics (or "scoring tools") are a way of describing evaluation criteria (or "grading
standards") based on the expected outcomes and performances of students. Typically,
rubrics are used in scoring or grading written assignments or oral presentations;
however, they may be used to score any form of student performance. Each rubric
consists of a set of scoring criteria and point values associated with these criteria. In
most rubrics the criteria are grouped into categories so the instructor and the student
can discriminate among the categories by level of performance. In classroom use, the
rubric provides an "objective" external standard against which student performance
may be compared

WHAT IS INVOLVED?

Instructor Preparation
Medium to High.
Time:
Continuous; but students catch on fairly
Preparing Your Students:
quickly.
Variable. As students use rubrics, they
become better writers and oral presenters;
Class Time:
hence the time instructors spend evaluating
students' work is reduced.
Disciplines: All.
Class Size: All. Rubrics are easy to use in small classes,
and are particularly useful in large classes to
 facilitate scoring large numbers of written or
oral assignments
Special
Classroom/Technical None.
Requirements:
Individual or Group
Both.
Involvement:
The level of analysis depends on the
instructor's intended goal of the assessment
task and the type of data desired about
students' performance. For detailed analysis
Analyzing Results: of students' responses, each section of the
rubric can be scored independently then
totaled. For a holistic analysis of students'
responses, all sections of the rubric can be
blended and an overall score assigned.
Rubrics must be readily available to students
before they begin an assignment or written
Other Things to Consider: test. Posting rubrics on the web and including
them in the course pack for in-class writing
promotes their usefulness.

Classroom Assessment Techniques

Scoring Rubrics
(Screen 2 of 6)

Description

Example 1. Scoring Rubric for Quizzes and Homework

Level of
General Approach Comprehension
Achievement
 Addresses the
question.  Demonstrates an accurate
 States a relevant, and complete understanding
justifiable answer. of the question.
Exemplary
 Presents  Backs conclusions with
(5 pts
arguments in a logical data and warrants.
quizzes)
order.  Uses 2 or more ideas,
 Uses acceptable examples and/or arguments
style and grammar that support the answer.
(no errors).
Adequate  Does not address  Demonstrates accurate
 the question
explicitly, although
does so tangentially. but only adequate
 States a relevant understanding of question
and justifiable because does not back
(4 pts answer. conclusions with warrants
quizzes)  Presents and data.
arguments in a logical  Uses only one idea to
order. support the answer.
 Uses acceptable  Less thorough than above
style and grammar
(one error)
 Does not address
the question.
 States no relevant
answers.  Does not demonstrate
Needs  Indicates accurate understanding of
Improvement misconceptions. the question.
(3 pts  Is not clearly or  Does not provide
quizzes) logically organized. evidence to support their
 Fails to use answer to the question.
acceptable style and
grammar (two or
more errors).
No Answer
   
(0 pts)

Assessment Purposes

 To improve the reliability of scoring written assignments and oral presentations

 To convey goals and performance expectations of students in an unambiguous
way
 To convey "grading standards" or "point values" and relate them to
performance goals
 To engage students in critical evaluation of their own performance.

Limitations

 The problem of criteria: One challenge (or potential limitation) in developing

and using rubrics is that of describing performance expectations, and defining
the criteria that differentiate several levels of performance. Quantitative
descriptors may be helpful in differentiating among performance levels, e.g.,
"provide 2 examples of evidence that support the conclusion...," or "zero
grammatical errors". Whereas variables such as vague or concise, must be
described clearly so that students see the differences between a statement that
is vague and a statement that is concise. By carefully describing our
performance expectations and defining the criteria we use to differentiate levels
of performance, our role as evaluators becomes easier, more informative, and
more useful to our students and us.
  The problem of practice and regular use: No assessment tool is effective if
it is not used on a regular basis. Rubrics are most effective when we practice
using them with our students over and over again. Developing effective rubrics
requires revision based on feedback from students. The best rubrics are
products of an iterative effort.

Example 2. Scoring Rubric for Grant Proposals (35 points possible)

Level of General Conceptual Argument Use of
Achievement Presentation Understanding Structure literature
(10 points (10 points (10 points and
possible) possible) possible) pertinent
resources
(5 points
possible)
Exemplary  (10 pts.)  (10 pts.)  (10 pts.)  (5 pts.)
 Provides a  Demonstrates  Provides  Follows
clear and a clear strong, clear, proper format
thorough understanding of convincing in providing
introduction and the LTER site and statements (i.e. citations.
background the proposed conclusions) of  Uses data
 States a research. the reasons the and/or
specific, testable  Uses a broad proposed information
research question range of research is relevant to
 Provides clear information to important and the proposed
explanation of build and support should be funded. research
proposed arguments.  Provides
research methods  Demonstrates relevant evidence
 Presents a good to support
rationale and understanding of conclusions.
significance of the implications of  Provides
proposed the data and/or reasons for the
research in the information. legitimacy of the
form of a well- evidence (i.e.
structured, logical warrants) that
argument. enable
 Uses conclusions.
acceptable style
and grammar (0
errors)
Adequate  (8 pts.)  (8 pts.)  (8 pts.)  (4 pts.)
 Provides an  Demonstrates  Provides  Follows
introduction and a partial statements (i.e. proper format
background that understanding of conclusions) in providing
is only somewhat the LTER site and explaining the citations, but
significant to the the proposed reasons the not
experiment. research. proposed consistently
 States a clear,  Uses a research is throughout
but untestable information from important and the proposal.
research only 2 or 3 should be funded,  Uses
question. sources to build but weak limited
  Provides an and support evidence to number of
adequate arguments. support sources of
explanation of  Demonstrates conclusions and data and/or
proposed a partial no warrants. information
research methods understanding of relevant to
 Shows some the implications of the proposed
effort to present the data and/or research
the rationale and information.
significance of
proposed
research in the
form of a well-
structured
argument.
 Uses adequate
style and
grammar (1-2
errors)
Needs  (6 pts.)  (6 pts.)  (6 pts.)  (3 pts.)
Improvement  Provides an  Does not  Provides  Does not
introduction and demonstrates an statements (i.e. follow proper
background that understanding of conclusions) format in
is insignificant to the LTER site and explaining the providing
the experiment. the proposed reasons the citations.
 States a research. proposed  Does not
vague, untestable  Uses less than research is uses data
research two sources to important and and/or
question. build and support should be funded, information
 Provides an arguments. but no evidence relevant to
unorganized  Does not to support the proposed
explanation of appear to conclusions and research
proposed understand the no warrants.
research methods implications of the
 Presents data and/or
rationale and information.
significance of
proposed
research in the
form of a weak,
unstructured
argument.
 Fails to use
acceptable style
and grammar (>2
errors)

Teaching and Learning Goals

Students learn to communicate about science in a variety of ways and especially
improve their writing skills. The quality of students' reasoning and logic increases.
Instructors gather a variety of data about students' understanding and performance.
Suggestions for Use
I design rubrics for the multiple forms of assessment I use in my courses: short
writing samples, essays, poster displays, research papers, public hearing papers, oral
presentations, weekly homework assignments, and concept maps. Each rubric stands
on its own, but the general criteria in many rubrics are similar. For example, rubrics for
written assignments have the same criteria for acceptable style and grammar;
responses must address the question, and arguments must be presented in a logical
order [compare Figures 1 and 3]. Alternatively, a rubric designed to evaluate the
mechanics of a poster display may include a checklist to guide the student in
developing all of the sections of the poster. Each component of the poster should then
have additional criteria for evaluation. For example: What are the criteria for the title
of a poster? Is the title informative? Are specific key words used?

Example 3. Scoring Rubric for Essay Questions

Level of General Reasoning,
Achievement Presentation Argumentation
 Provides a clear
and thorough
introduction and
 Demonstrates an
background
accurate and complete
 Addresses the
understanding of the
question
Exemplary question
 Addresses the
(10 pts)  Uses several arguments
question.
and backs arguments with
 Presents arguments
examples, data that support
in a logical order
the conclusion
 Uses acceptable
style and grammar (no
errors)
 Combination of
above traits, but less
consistently
 Uses only one argument
Quality represented (1-2
and example that supports
(8 pts) errors)
conclusion
 Same as above but
less thorough, still
accurate
 Does not address
the question explicitly,
though does so
tangentially  Demonstrates minimal
 States a somewhat understanding of question,
Adequate relevant argument still accurate
(6 pts)  Presents some  Uses a small subset of
arguments in a logical possible ideas for support of
order the argument.
 Uses adequate
style and grammar
(more than 2 errors)
Needs  Does not address  Does not demonstrate
 the question
 States no relevant
understanding of the
arguments
question, inaccurate
Improvement  Is not clearly or
 Does not provide
(4 pts) logically organized
evidence to support
 Fails to use
response to the question
acceptable style and
grammar
No Answer
   
(0 pts)

Step-by-Step Instructions
There are many routes to developing a useful scoring rubric, however, all of them
involve the following five steps:

 Develop the goals for your course and daily class meetings
 Select the assessment tasks that provide data aligned with the goals
 Develop performance standards for each of the goals
 Differentiate performances (categories) based on well-described criteria
 Rate (assign weight or value to) the categories

To provide a useful example of how these steps "play out" in a real world context, I will
describe how I developed rubrics for my own introductory biology and ecology courses.

1. I developed the goals for my course and daily class meetings.

Keep in mind the assessment tasks must be linked to student learning goals and
outcomes. So writing goals is the first step. These are examples of stems and sample
goals from introductory ecology or biology courses:

Students will be able to demonstrate their ability to:

 utilize science as a process

 communicate an understanding of and links among biological
principles
 write about, criticize and analyze concepts in biology
 use the process of scientific inquiry to think creatively and formulate
questions about real-world problems
 apply content knowledge in the resolution of real-world problems
 reason logically and critically to evaluate information
 argue science persuasively (in both written and oral format)
 illustrate the relevance of ecology to your lives by applying ecological
knowledge in the resolution of real-world problems

2. I selected the assessment tasks:

What type of assessment will provide me data about students' achievement

of each of these goals?
 Based on the goals for my courses, I selected different forms of extended
responses, both written and oral, and concept maps to gather the data that
would convince me that my students achieved the goals. The kinds of
questions I asked students and the types of projects I assigned, were
designed to promote students' reasoning. For example, for the first three
goals I have listed, various types of assessment that could be used to gather
the type of data desired.

 Utilize science -- performance assessment e.g., students conduct a

scientific investigation.
 Communicate an understanding of and links among biological
principles -- e.g., concept maps, Vee diagrams, extended written
responses (Novak and Gowin 1984, Novak 1998).
 Write about, criticize and analyze concepts in biology -- written critical
analysis of articles and papers.

3. I developed a set of performance standards:

The performance standards I used in my introductory biology course on

"logical reasoning" and "critically evaluating information" were different than
the performance standards I developed for my upper division biology majors.
The difference was based on the developmental stages of the students and
their experience in college-level science courses (Magolda 1992, King and
Kitchener 1994).

4. I differentiated performances based on criteria:

Examine the rubric for Quizzes and Homework. The criteria for responses fall
into two major categories: general approach and comprehension. Although
these two categories are not discrete as indicated by the dotted line between
them, students can see all of the itemized components of an exemplary
answer. These categories can be divided further. For example,
comprehension could be divided into content knowledge, conceptual
understanding, and reasoning and critical thinking skills (Freeman 1994).
Freeman (1994) includes communication skills as a category in rubrics.
Essentially, my rubrics cover the same categories; the difference is in the
number of columns used.

Notice, when it is possible to quantify the categories, I did so. So, for
example, the criteria for acceptable style and grammar in an exemplary
answer is based on no errors.

Our ability to differentiate among criteria is critical to the effectiveness of the

scoring rubric. So words like "good" are too subjective. The criteria must be
designed so that you and your students can discriminate among the qualities
 you consider important.

When we evaluate students' extended responses, we tend not to score them

point by point, however, by elaborating on the criteria that comprise the
different levels of performance, we provide the students substantive guidance
about what should be included in their extended responses.

5. I assigned ratings (or weights) to the categories.

 Exemplary (5 pts) - highest category of college-level work

 Adequate (4 or 3 pts) - acceptable college-level work
 Needs Improvement (3 or 1 pts) - not yet college level-work
 No answer: 0 points
Point values: Do you assign points on a 5, 3, 1 scale? or a 5, 4, 3 scale? I
have tried both. I chose 3 as the middle or as an adequate score. Most
student responses in this category can readily be improved through group
work, practice, effort and instruction. Therefore, in an effort to develop
students' self-efficacy and to promote their achievement of higher standards,
I chose the 5,4,3 point scheme.

On a five-point scale, the data do not enable me to discriminate between two

consecutive points, such as 3 and 4, in terms of evaluating the response.
Rather, three categories were readily distinguishable by my students and me,
therefore, little if any time was spent "arguing" for points. The criteria for
evaluation were clear and understood.

Variations
Student Roles
 I involve students in a dialogue about criteria that we use in any rubric. Students
gain a keen sense of my expectations for assessment by explicitly understanding the
criteria and by contributing to the modification of criteria in a rubric to enhance clarity.
 Consider letting students develop class rubrics for certain assessments. When
students understand and describe criteria for their own performance, they are often
better at attaining those standards. My students developed the rubric for the poster
displays of their laboratory research. To accomplish this, they walked around the
biology department looking at the variety of posters displayed on the walls and then
determined their own criteria for what makes a quality poster presentation. We
collated this information and designed a rubric for content as well as format.
 Students use rubrics when completing any assessment task for the course such as
writing in class, writing on an exam, designing homework, completing and
investigation, preparing a research paper.

Faculty Roles

 The critical factor for faculty to consider is that assessments must be linked to the
goals of the course. For example, if the goal is for students to demonstrate their ability
to design a testable hypothesis in a particular content area - asking students to
actually write a testable hypothesis would provide meaningful feedback. The recurring
question we must ask is, "Does this evidence convince us that students understand
how to write a testable hypothesis?"
 Include rubrics on your web site and in your course packs. Students should refer to
rubrics while they are completing any assessment task.
 Rubrics are dynamic and involve a major up-front investment of time.
 You must provide students repeated instruction on how to use rubrics as well as
how to achieve each of the criteria.

Share with students samples of "exemplary", "adequate", "needs improvement"

responses. Ask them to work in cooperative groups to analyze the strengths and
weakness of the written responses, using the rubric as a guide. With practice, students
learn to recognize and ultimately develop their own exemplary responses.

 The advantage of rubrics is that you and the students have well defined pathways
for gathering evidence that they have achieved the goals of the course. If either you or
your students are dissatisfied with the evidence or criteria, the rubrics should be
revised.

Analysis
Rubrics are scoring tools that enable me to assign points to students' assignments and
tests. Students' accumulation of points determines their grade in the course. Each
assignment, quiz, or test is weighted in terms of value in the overall course evaluation.
For example, daily writing samples (quizzes) are worth 5 points, twice weekly, 15
weeks per semester; hence a student can earn a maximum of 75 points for daily
performance. The pattern of students' performance is consistent from semester to
semester. At the beginning of each semester, many students' responses are below
college-level. As students begin to understand the criteria and practice writing, they
attain college-level work or exemplary performance on short, five-point assignments or
quizzes. A key strategy in promoting improvement by all students is peer review within
their cooperative groups.

The formative assessment I gather by using rubrics to evaluate students' responses

during the course is valuable. In-class writing assignments give me feedback about the
nature of the task and questions I ask students. The components of a question or task
that provide meaningful responses are readily identifiable from the rubric and provide
us insight into my students' strengths and weaknesses. I use these data to modify,
change directions, or add components to our instructional design and strategies.

Pros and Challenges

 Faculty develop and communicate assessable course goals.

 Faculty and students understand and agree upon the criteria for assessment.
 Rubrics provide ways to evaluate many different types of assessment.

However:

 Time - rubric development requires time up front, but the payoff is in increased
performance by the majority of students and which ultimately leads to less
instructor time in assessment.
 Criteria - qualitative scales are more difficult to define than quantitative scales.
  Practice - both students and faculty need to practice and refine the use of
rubrics for multiple types of assignments. Time to do this in class will affect
"coverage."

Theory and Research

What is assessment? Simply, assessment is data collection with a purpose. In each of
our courses, we engage in the process of gathering data about our students' learning.
The type of data we gather depends on the evidence we will accept that students have
learned what we want them to learn. Generally, the data we collect are intended to be
measures of students' knowledge, attitudes and performance. Ideally, these data are
also matched or "aligned" with the goals of the course and our daily or weekly course
activities.

Four functions of assessment data are described by Hodson (1992):

 formative assessment provides diagnostic feedback to students and instructors

at short-term intervals (e.g., during a class or on a weekly basis);
 summative assessment provides a description of students' level of attainment
upon completion of an activity, module, or course;
 evaluative assessment provides instructors with curricular feedback (e.g., the
value of a field trip or a writing assignment).
 educative assessment develops in students and instructors further insight and
understanding about their own learning and teaching. In effect, assessment IS
a form of learning (NRC 1996, p. 76).

We need confidence in the quality of the data we gather about our students if we want
to justify our subsequent decisions about teaching. Many of us who teach introductory
science courses are dissatisfied with the type of evidence we are collecting about our
students' learning. We admit that data from multiple choice tests measure inert bits of
knowledge and some comprehension but provide us incomplete and inadequate
feedback about our students' learning. We would like to use alternative forms of
assessment to gather multiple, substantive forms of data about active student
learning, such as understanding, analysis, reasoning, and synthesis (Ebert-May et al
1997). These kinds of assessments include short answer items, essays, minute papers,
oral communication, poster presentations, laboratory projects and research papers,
but because of large class sizes and individual research priorities we have limited time
to evaluate extended responses from students.

Assessment is learning. We and our students both benefit from meaningful assessment
information about the achievement of the broader course goals. Multiple assessment
strategies can be implemented to provide evidence that students have or have not
learned, have or have not accomplished the goals of the course. Rubrics help us set
well-defined standards for our students, provide students guidelines for achieving
those standards and facilitate grading extended written and oral responses. This
feedback provides us data to interpret and make informed decisions about our
students' learning and our own teaching practice, similar to the process of data
evaluation that we use daily in our scientific research.

Links
  Diane Ebert-May, Professor, Department of Plant Biology, Michigan State
University, East Lansing, Michigan.
email: ebertmay@pilot.msu.edu

Sources
Ebert-May D, Brewer C, Allred S. 1997. Innovation in large lectures-teachings for
active learning. Bioscience 47: 601-607.

Freeman, RHL. 1994. Open-ended questioning: a handbook for educators. Menlo Park,
California; Reading, Massachusetts; New York; Don Mills, Ontario; Wokingham,
England; Amsterdam; Bonn; Sydney; Singapore; Tokyo; Madrid; San Juan; Paris;
Seoul; Milan; Mexico City; Taipei. Addison-Wesley Publishing Company (The book is
published by Innovative Learningª, an imprint of the Addison-Wesley Alternative
Publishing Group.)

King PM, Kitchener KS. 1994. Developing reflective judgement: understanding and
promoting intellectual growth and critical thinking in adolescents and adults. San
Francisco (CA): Jossey-Bass Publishers.

MacGregor J. 1993. Student self-evaluation: fostering reflective learning. San Francisco

(CA): Jossey-Bass Publishers.

Magolda, MBB. 1992. Knowing and reasoning in college: gender-related students'

intellectual development. San Francisco (CA): Jossey-Bass Publishers.

National Research Council. 1996. National science education standards. Washington

(DC): National Academy Press.

Novak JD, Gowin DB. 1984. Learning how to learn. New York: Cambridge University
Press.

Classroom Assessment Techniques

Interviews
WHY USE INTERVIEWS?
In-depth "structured" interviews with a handful of carefully selected students will
enable you to readily judge the extent of understanding your students have developed
with respect to a series of well-focused, conceptually-related scientific ideas. This form
of assessment provides feedback that is especially useful to instructors who want to
improve their teaching and the organization of their courses.

WHAT IS AN INTERVIEW?
A formal interview consists of a series of well-chosen questions (and often a set of
tasks or problems) which are designed to elicit a portrait of a student's understanding
about a scientific concept or set of related concepts (Southerland, Smith & Cummins,
2000). The interview may be videotaped or audiotaped for later analysis.

WHAT IS INVOLVED?
 Several hours required to develop a set
Instructor Preparation ofgood questions, tasks and problems sets.
Time: Additional time to locate appropriate props
and recording equipment, if desired.
Interviews are most fruitful when the student
has developed a good rapport with you. It is
Preparing Your Students:
essential that the student feels relaxed and at
ease.
One-on-one or small group interviews may be
conducted in less than an hour in your office
Class Time: or other convenient "private space." Some
practice will reduce the time required to
conduct a good interview.
No disciplinary constraints. Appropriate for all
Disciplines:
STEM fields.
Normally, structured interviews are conducted
outside of class. It is important that subjects
Class Size: be carefully selected to represent a range of
ability and interest levels among students
enrolled in a course.
Special
Interview protocol, props, recording
Classroom/Technical
equipment and small private space.
Requirements:
The most useful interviews are those
conducted with individuals or small groups
Individual or Group
outside of class. Sometimes this is done well
Involvement:
in laboratory sections, but TAs will need
special training or assistance.
For "formative" assessment, the instructor
may want to review taped interviews with
special attention to potential
Analyzing Results:
"misconceptions." If used for "summative"
evaluation, a type of "scoring rubric" may be
developed.
Other Things to Consider: None.

Description
In its simplest form a structured interview is simply one person asking another person
a series of questions about a carefully selected concept/topic or asking her to perform
a task. Any materials to be used (props, problems, etc.), many of the questions to be
asked, and some responses from the interviewer to expected statements or actions of
the interviewee are carefully planned in advance. Importantly, however, the
interviewer is free to ask additional questions that focus on issues arising during the
course of the interview. It is this freedom to follow the interviewee, to ask for
clarifications, and to focus on errors, misconceptions, and gaps in knowledge, that
makes the interview so much more fruitful than more traditional methods of
assessment.
During a structured interview, the instructor uses a set of questions, called "probes"
(and sometimes selected photographs or other props) designed in advance of the
interview to elicit a portrait of the learner's understanding about a specific
concept/topic (e.g., evolution; molecular/kinetic theory; plate tectonics; binary stars;
Newton's laws). The student may be asked to use her own words to explain a concept
(e.g., "What is natural selection?") but is typically required to go beyond simple
recognition of a concept to construct a detailed personal explanation. Typically the
student is also asked to use that concept to solve a problem or other application task
(e.g., "Explain why cave fish have no color"). Valuable information is often obtained
not only from listening to what the interviewee says, but also from observing what she
does, including facial expressions and other body language.

Assessment Purposes
Structured interviews may serve many functions, among them:

 to investigate how well students understand a concept; to identify

misconceptions, areas of confusion, and/or gaps in understanding that may be
common among a group of students;
 to document how students can apply their knowledge in concrete settings (e.g.,
problem solving, trouble shooting);
 to document the general and content-specific procedures that students employ
in such application tasks and the sequences and manner in which these
processes are employed;
 to document how student understanding and problem-solving skills change over
time or with instruction; and
 to obtain verbal feedback from students about course structure, teaching
techniques and other aspects of the course or program of instruction.

It is also important to note that the goal of the interview is to describe how a student
understands a scientific concept or phenomenon, and not simply to provide a
measurement of the degree to which this understanding approximates the scientific
explanation. Thus, interviews are typically used to provide the instructor with insight
about students' understandings in order to refine and target instruction ("formative
assessment") rather than to evaluate the knowledge of individual students for
purposes of assigning a grade ("summative assessment").

Structured interviews are used to describe individual student's understandings of a

specific scientific concept or closely related groups of concepts. It is important to note,
however, that the degree of understanding to be assessed will differ depending on the
type of interview probe used. Can the student recognize the concept? Generate an
example? Apply the concept? Use the concept to predict phenomena or solve
problems? Different kinds of structured interviews measure different degrees of
understanding.

Limitations
Structured interviews are used to describe individual student's understandings, and are
best conducted individually with students; thus time is a major inhibiting factor in
using structured interviews to inform teaching. To prevent this issue from being
prohibitive, selective sampling of a broad range of students in a classroom may be
employed to make the technique more practical, yet still provide a portrait of how
different students in a class are engaging with course material.

A second limitation of structured interviews lies in the extreme content specificity of

students' thinking. For instance, when dealing with biological knowledge, the type of
organism included in an interview prompt has been shown to radically change the
nature of a student's response. Thus, if an instructor would like to probe a student's
reasoning pattern about a specific process (e.g., the change of coat color in response
to environmental cues) the nature of the exemplar (eg. the organism) included in the
probe must be taken into account (Tamir & Zohar, 1992). Similar specificity may be
expected in virtually all scientific disciplines.

Teaching Goals

 Analyzes problems from different viewpoints.

 Recognizes interrelationships among problems and issues.
 Applies principles and generalizations to new problems and situations.
 Demonstrates a basic knowledge of the concepts and theories of this subject.
 Demonstrates synthesis and integration of information and ideas.
 Uses appropriate synthetic and analytic methods to solve problems.
 Communicates effectively.

Suggestions for Use

Because structured interviews can provide a wealth of information about a student's
understanding, interviews would seem to be strong candidates for use in formal
("summative") evaluation. However, the use of such interviews for individual
evaluation is somewhat problematic. Using structured interviews in formal evaluation
requires extended sessions with each student, a luxury that few faculty can afford
except in relatively small classes. Instead, careful sampling and interviewing of a select
but diverse group of students may permit you to develop an overall portrait of the
various understandings that students in your class hold. This kind of "formative
assessment" can provide detailed feedback that is very helpful in improving your
teaching.

Structured interviews are very powerful tools for gaining insight into students' thinking.
They are especially useful in diagnosing "learning errors", "misconceptions", and
limitations in reasoning and critical thinking. With some training and practice, teaching
assistants may be encouraged to use interviewing strategies in small groups and
laboratory sections of large classes. Students themselves often find that knowledge of
interviewing is useful in collaborative learning environments.

Step-by-Step Instructions
Several types of interview strategies have been developed for use in STEM disciplines
(Southerland, Smith & Cummins, 2000). However, the "Interview about Instances and
Events" (White & Gunstone, 1992) is possibly the most widely used format for probing
understanding about single concepts. In this interview, the student is presented with a
set of 10-20 line-drawings, photographs or diagrams that depict examples and
counterexamples of natural objects (e.g., a mammal; a volcano; a planetary system; a
molecule) or events (eg. a burning candle; a moving automobile; a girl throwing a
baseball).
Revealing one drawing at a time, the student is asked to indicate whether it depicts an
example of the concept in question, and to provide a rationale or justification. For
example, consider a baseball in flight: Is there a "force" on the ball? What makes you
say that? Tell me more about that. Or consider a burning candle: Is this "sublimation"?
Why do you think that? Can you say some more about that? After each question, the
instructor gently probes further into the reasoning the student uses and encourages
him/her to elaborate on the responses to provide as complete a picture as possible of
the student's understanding. A few general suggestions for conducting successful
interviews:

1. The interview should begin with a focus question that requires application of the
concept to be investigated, without forcing the student into an explicit
definition. A more traditional assessment might ask the student to choose the
correct definition of the concept from among four choices or to write down a
definition of the concept. The more indirect approach of a structured interview
is usually more productive because it allows the student to evince her
understanding rather than relying on memorized, rote definitions. This also
enables the instructor to gain an idea of how the student applies the implicit
concept.
2. Do not force the student into a specific response to each graphic. If the student
needs to "waffle" in her answer, she should be allowed to do so. If the student
does not have an understanding of the concept that allows her to make a
decision about a specific instance, do not force her to choose. This lack of
understanding is an important piece of her "conceptual framework".
3. Specific definitions of the concept, if needed, should be sought only after
understanding the student's response to the focusing questions. Again, this
prevents students from early closure on a rote definition. Thus, in our example,
it would be inappropriate to ask, "Well, what is a force (or sublimation)?"
4. It is important for the interviewer to wait at least 3 to 5 seconds after each
prompt before trying to interpret the question or ask another. Classroom
research has shown that when this "wait time" is observed, both the length of
the student responses and the cognitive level of the responses increases (Rowe,
1974).

Variations
As mentioned previously, structured interviews are used to describe individual
students' understandings of specific scientific concepts and the degree to which they
can apply that understanding. Different interview probes allow for the investigation of
different degrees of student understanding.

Instances Interviews
In Interviews about Instances, a student is presented with a specific set of examples
and counterexamples of the concept of interest and is asked to identify which cases
are examples of the concept, and then to explain that decision. For practical reasons
the examples are usually graphics such as line pictures, drawings, or diagrams.

Prediction Interviews
Prediction Interviews require students to anticipate an outcome of a situation and
explain or justify that prediction. The strength of this kind of interview is that it focuses
on the ways a student can apply her personal meanings of the concept. And because
they require application, prediction interviews are very useful in teasing out what has
been learned by rote with minimal understanding from what is meaningful knowledge.
Sorting Interviews
In a Sorting Interview, the student is presented with a group of objects and asked to
sort them according to specific instructions. This exercise can be structured in many
different ways to match the purpose of the assessment. For example, the interviewer
may present a series of graphics depicting some natural phenomenon. The student
may then be asked to select any number of cards to be used in any order to explain
the phenomenon. Alternatively, a student may be presented with a set of genetics,
physics or chemistry problem cards and asked to sort them according to similarity
(e.g., Smith, 1992). As with other kinds of interviews described in this CAT, the
student is encouraged to talk about her reasoning as she attempts to construct an
explanation for her sorting.

Problem Solving Interviews

In a Problem Solving Interview, a student is asked to attempt to solve a problem while
"thinking aloud," explaining as much as possible about what she is doing, why she is
doing it, and what her symbols and actions mean. Problem solving interviews focus
more on a student's performance as a means to assess knowledge, although
understanding the student's conceptual framework remains the overarching goal in
conducting the interview.

Analysis
Note-taking during an interview can be beneficial, but it generally provides only a
superficial picture of a student's meaning. Instead, it is usually beneficial to record the
interviews, allowing for more intensive data analysis. As with most classroom
assessment activities, analysis of interview data may be accomplished in a variety of
ways, with some methods capturing a richer and more multilayered perspective than
others.

In order to analyze the results of structured interviews, we suggest that the instructor
attempt to put her expectations aside to the extent possible, and instead review the
tape or read the transcript with a fresh "eye," allowing important trends from the
learner's responses to emerge. Ideally, a sample of interview transcripts should be
reviewed several times, so that ideas emerging from one review can inform
subsequent readings. As strong trends are noted throughout several interviews,
negative examples (occasions for which the tentative trend fails to hold true) should be
searched for. This inductive approach to data analysis, i.e., looking for similarities and
differences in sets of data, allows for a more informative and reliable portrait of
learners to emerge.

For most instructors, a detailed analysis of transcribed interviews is a time-consuming

luxury that can't be afforded. However, a review of a taped interview can reveal much
about a student's understanding that is not readily discerned in the course of a more
casual discussion. Viewing or listening to taped interviews with colleagues or teaching
assistants can provide multiple perspectives of the same student, and offers a
collaborative opportunity to share a set of common problems.

Pros and Cons

  Structured interviews enable instructors to examine the degree of
understanding a learner has for a scientific concept or closely related group of
concepts.
 Interviews offer instructors a vehicle for focusing on how instruction is
interpreted and internalized by the learner.
 Faculty can use structured interviews as a powerful type of formative
assessment to improve courses and teaching methods.
 Collaborative analysis of interviews allows faculty groups to develop shared
understandings of typical difficulties students have with key concepts.

However,

 Structured interviews are designed to elicit how a student understands a

scientific concept. As such, they should be used in addition to, not instead of,
other forms of evaluation.
 Interviews are quite time-consuming. We suggest that faculty interview a broad
sample of students in a class in order determine how students are reacting to
and understanding concepts presented in class.
 The usefulness of the interview technique is largely determined by the nature
and quality of the probes and follow-up questions. Thus, a substantial amount
of planning may be required to design an informative interview.

Developing Concept Maps

Making Thinking Visible
Concept maps are visual representations of linkages/connections between a major concept and
other knowledge students have learned. Concept maps are excellent tools to provide instructors
with diagnostic pre-assessment prior to beginning a unit and formative assessments during
learning activities. Concept maps also provide immediate visual data to geoscience instructors on
student misconceptions and their level of understanding. Angelo and Cross (1993) indicate that
concept maps develop student abilities in certain critical areas. Among these are:

 The ability to draw reasonable inferences from observations

 The ability to synthesize and integrate information and ideas
 The ability to learn concepts and theories in the subject area

Classroom Assessment Techniques

Concept Mapping
WHY USE THE CONCEPT MAPS?
Concept maps assess how well students see the "big picture". They have been used for
over 25 years to provide a useful and visually appealing way of illustrating students'
conceptual knowledge.

WHAT IS THE CONCEPT MAPS?

A concept map is a diagram of nodes, each containing concept labels, which are linked
together with directional lines, also labeled. The concept nodes are arranged in
hierarchical levels that move from general to specific concepts.

WHAT IS INVOLVED?

Instructor Preparation Minimal if students construct maps; large for

Time: designing "fill-in" maps.
Students need training (about an hour) and
Preparing Your Students:
continual practice.
Class Time: Appropriate for all.
Disciplines: Appropriate for all.
Small classes if students construct maps
Class Size: individually; cooperative teams constructing
maps will work with large (or small) classes.
Special
Classroom/Technical None.
Requirements:
Individual or Group
Either.
Involvement:
Intensive for formal scoring for large classes;
Analyzing Results:
concept maps are generally not graded.
Other Things to Consider: Very demanding cognitive task for students.

Description
A concept map is a two-dimensional, hierarchical node-link diagram that depicts the
structure of knowledge within a scientific discipline as viewed by a student, an
instructor or an expert in a field or sub-field. The map is composed of concept labels,
each enclosed in a box or oval; a series of labeled linking lines, and an inclusive,
general-to-specific organization. By reading the map from top to bottom, an instructor
can:

1. gain insight into the way students view a scientific topic;

2. examine the valid understandings and misconceptions students hold; and
3. assess the structural complexity of the relationships students depict.

In addition to these applications in assessment, faculty have also used concept maps
to organize their ideas in preparation for instruction, as a graphic organizer during
class, and as a way to encourage students to reflect on their own knowledge and to
work together and share their understandings in collaborative group settings.
 Figure 1: Concept Map Of Concept Maps
Click here to see a larger version of this graph.

Assessment Purposes
To investigate how well students understand the disciplinary-acceptable connections
among concepts in a subject, to document the nature and frequency of students'
misconceptions, and to capture the development of students' ideas over time.

Limitations
Concept maps provide a useful and visually appealing way of depicting the structure of
conceptual knowledge that people have stored in long-term memory. As a result, they
offer a readily accessible way of assessing how well students see "the big picture."
They are not designed to tap into the kind of process knowledge that students also
need to solve novel problems or for the routine application of algorithmic solutions.
Because they probe an individual's or a group's cognitive organization, they are very
idiosyncratic and difficult to compare, either among individuals or groups, or across
time for the same individuals or groups.
 Figure 2: Jason's Concept Map on the Human Circulatory System
[From Mintzes, Wandersee & Novak, 1998]

Teaching Goals

 Learn terms, facts, and concepts of this subject

 Organizes information into meaningful categories
 Synthesize and integrate information , ideas, and concepts
 Think about the "big picture" and see connections among concepts
 Think creatively about this subject
 Improve long-term memory skills for accessible knowledge
 Develop higher-level thinking skills, strategies, and habit
 Use graphics effectively

Suggestions for Use

Development of Course Design.

Global concept maps developed by an experienced instructor, or a team of instructors,
can clarify the essential concepts and their linkages. They also make explicit the
connected knowledge of a field, which is implicit among experts in the discipline.
Effective instruction can then be designed that contains the structure of the concept
maps. Caveat: This use requires a high investment of instructor time for the initial
concept map construction.
 Figure 3: Schema for Using a Concept Map for Developing Course Design
[From Novak and Gowin, 1984]
Click here to see a larger version of this graph.

Instructional Tool.
The instructor can present "expert" concept maps to the whole class to highlight key
concepts and connections. These should be more detailed and flow from the global
maps executed for the course design. Concept maps can then serve as "advanced
organizers" (to preview material) and also for review. An instructor can continuously
refer to a concept map in class to show how to "grow" the connections, and to keep
the instruction focused. Caveat: At first, students will find concept maps very strange
and may even try to memorize them, rather than use them as a thinking tool.

Learning Tool.
Ask students to construct their own concept maps covering a section of the course
material from class or the textbook. Most (if not all!) of them will probably never have
seen a concept map before, and many have not developed the learning skills needed to
construct them. As a result, the instructor will need time (either in class, or perhaps in
the lab) to work with groups and individuals. The impact of student-created concept
maps is so powerful that it is worth the investment of time!

Step-by-Step Instructions

 Introduce a concept that is familiar to all students, such as "car", "chair" or

"food."
 Have students write down 10 other concepts that they associate with this main
concept (i.e. for food, "vegetables", "meat", "cereal", "milk", "steak",
"carrots"...).
 Ask them to rank the 10 concepts from "most general and inclusive" to "least
general and inclusive" or from "most important" to "least important"; this step
will require several minutes.
 Tell students to write the "most general" or "most important" concept near the
top of a large piece of paper (e.g., posterboard or butcher paper are excellent,
but regular notebook paper will suffice). Have them enclose this "superordinate
concept" in a box or oval. Use pencils instead of pens! (Post-its® are excellent
for this step.)
 Explain that you want them to connect concepts from their list, one pair at a
time, with directional links; and most importantly, to label the linking lines
 (e.g., Carrots vitamin A (linking word is, "contain") OR meat iron (linking
words are, "is a good source of"). Continue this process until all concepts
appear on the map.
 Give students plenty of time (20-30 minutes). Encourage them to include a lot
of branching and many levels of hierarchy. Put special emphasis on cross-
linking concepts in one area of the map with those in other areas. Suggest that
they may add as many additional concepts as they wish to make their maps
unique and personally meaningful. Remind them that the boxes or ovals should
contain only one or two words. Emphasize that "neatness doesn't count" and
that they may re-draw their maps as often as they wish.
 Circulate around the room as students construct their maps. Be supportive but
not directive. Remind students that a concept map is an distinctive
representation of their understanding, and that individual components on their
maps may or may not be scientifically accurate, but there is a large set of ways
to organize and represent what they know. Encourage creativity and stress that
there are no one "correct" answer.
 Select several students to share their maps with the class. You may need to
make a transparency to display the maps in large classes. Focus attention on
appropriate connections between concepts. Remind students that concept maps
may be a very helpful way to study; they can be used to condense many pages
of textbook verbiage into a succinct summary of what the author presents.
 In the next class, introduce a central concept from your course (e.g., "star",
"cell", "energy", "matter") and ask your students to construct a concept map on
this topic. Collect the maps and review them, but don't grade them. You may
want to suggest ways the maps could be improved.
 Return the maps to the students and suggest that they rethink some of their
ideas. We have used different colored pencils for each iteration so students may
depict and emphasize how their ideas change over time. The same map may be
used for several class periods, and students may be encouraged to add to,
delete, reorganize or even begin anew whenever they need to do so.

Variations
Collaborative Concept Mapping.
Sometimes the frustration levels can be very high when concept mapping is first
introduced, especially in large classes of relative novices. To counter some of this
anxiety and to encourage students to reflect on their own thinking, ask groups of 3 or
4 students to work together on a concept map. This exercise is often a very rewarding
and rich learning experience as peers argue, debate, and cajole each other. The result
is a genuine effort to negotiate the meaning of scientific concepts, attempting (as
scientists do) to reach consensus, or to stake out different points of view. The power of
the process resides in the interpersonal sharing of ideas, which are made explicit to the
instructor.

Fill-in Concept Mapping

You construct a concept map and then remove all of the concept labels (keep the
links!). You then ask the class to replace the labels in a way that makes structural
sense. Best done with small classes; a good way to introduce a new topic.

Select and Fill-in Concept Mapping

You create a concept map and then remove concepts from the nodes (about one-third
of them). These deleted concepts are placed in a numbered list on the map and
students choose among them. Scoring can be as simple as "percent correct."
Instructors of large classes may use multiple-choice type scanning sheets. The
assumption of this technique is that as students' thinking approximates that of the
instructor, the closer their connected knowledge is "expert-like." The key is to select
nodes that are at different levels of the hierarchy and have nearby or antecedent links.

Figure 4: Select and Fill-in Concept Map on Concept Maps

Click here to see a larger version of this graph. [See Figure 1 for the "answers"]

Selected Terms Concept Mapping

You provide a list of concept labels (10 to 20) and ask students to construct their maps
using only these labels. The focus here is on the linking relationships, and the
evolution of structural complexity of students' knowledge frameworks.

Seeded Terms Concept Mapping

In this approach, also known as "micromapping" (Trowbridge and Wandersee, 1996),
you furnish a small set of concept labels (5 to 10) and invite students to construct a
concept map using these, and an equal number of labels drawn from their own
knowledge of the topic.

Guided Choice Concept Mapping

Here you present a list of some 20 concept labels from which students select 10 to
construct their maps. When done over a period of time, the instructor focuses on which
concepts appear and which disappear. The assumption is that these changes represent
significant restructuring of the students' knowledge frameworks.

Analysis
To start, we suggest that you focus primarily on the qualitative aspects of students'
concept maps with emphasis on the accuracy or validity of the knowledge students
represent. Among the questions you might ask yourself are the following:

 Are the most important concepts depicted?

 Are the links among concepts scientifically acceptable?
 Is there a substantial amount of branching hierarchy and cross-linking?
 Do any of the propositions suggest that the student subscribes to significant
misconceptions?
 How have students' concept maps changed over the course of days or weeks?
As you and your students gain experience with concept mapping, you might consider
trying one or more of the quantitative "scoring rubrics" currently available (Novak and
Gowin, 1984). In the most well-established scoring scheme, 1 point is given for each
correct relationship (i.e., concept-concept linkage); 5 points for each valid level of
hierarchy; 10 points for each valid and significant cross-link; and 1 point for each
example.

Figure 5: Scoring Concept Maps

[From Novak and Gowin, 1984]
Click here to see a larger version of this graph.

Pros and Cons

 Concept maps help students focus on the "big picture", enabling them to devote
more of their time to conceptual understanding rather than rote learning
 Concept maps force students (and instructors!) to make valid connections
among concepts
 They provide a low tech (cheap!) vehicle that enables students to represent
graphically their knowledge, and to share it with the instructor and other
students
 They shift the emphasis from inert, static knowledge to contextually-embedded
knowledge; from isolated facts to theoretical frameworks of related concepts
 In addition to their role as assessment tools, concept maps offer a useful way to
help students "learn how to learn"; they also serve as useful vehicles for course
development and as graphic organizers before, during and after instruction

However:

 Comparisons among students are more difficult because concept maps tend to
reveal the idiosyncratic way that students view a scientific explanation, as a
result...
 Evaluation can become more time-consuming for the instructor, especially in
large classes, unless some variation (such as Select & Fill-in) is adopted
 If you score maps, you must use a consistent (and tested) scheme
  Students who have developed a strong facility for rote learning of verbal
knowledge sometimes find concept maps intimidating
 Constructing concept maps is a demanding cognitive task that requires training

Developing Portfolios to Assess Student Learning

Metacognition is an awareness of one's own thought processes. As such, it involves an almost

simultaneous, concious degree of self-awareness. "This is how I approach or think about a
situation"; "this is how I might best approach this particular concept in order to more fully
understand it"; "this is how I am thinking about this issue and it is or is not effective"; "these are
other possible approaches I might take instead" (Courts and McInerney, 1993 p.57)

Portfolios Defined
Portfolios are personalized long term documentation of student mastery of course material. An
essential element of portfolios are student reflections on their own learning and progression
towards the mastery of the material documented in the portfolio. As such, portfolios are windows
on the metacognitive process of students.

Types of Portfolios
In order to be useful as assessment tools the portfolio should include entries that demonstrate a
progression of student understandings and ultimately mastery of the concepts. Huba and Freed
(2000) identify two different portfolio types.

 All-Inclusive Portfolios that contain a complete record of all work done by a student in a
course or program.
 Selection Portfolios that are focused on documenting the achievement of mastery of
specific course goals/objectives set by the course instructor.

Assessment Using Portfolios

Because portfolios are by their nature long term records of student progress and achievement
they can be used to assess programs, courses or projects. Although the aspect of long-term change
in student content content knowledge and metacognition may not be as marked when portfolios
are used to assess courses, the process of portfolio development tends to focus students on
deciding what the essential elements of learning are and what samples of their work best display
these elements. In Chapter 5 of [Palomba and Banta, 1999] they discuss three critical elements of
student portfolios in promoting and supporting student learning. These are:

 Student involvement in the selection of entries

 Student preparation of written reflections about learning
 Continuing discussion with faculty regarding written reflections
 WHY USE PORTFOLIOS?
Portfolio assessment strategies provide a structure for long-duration, in-depth
assignments. The use of portfolios transfers much of the responsibility of
demonstrating mastery of concepts from the professor to the student.

WHAT ARE PORTFOLIOS?
Student portfolios are a collection of evidence, prepared by the student and
evaluated by the faculty member, to demonstrate mastery, comprehension,
application, and synthesis of a given set of concepts. To create a high quality
portfolio, students must organize, synthesize, and clearly describe their
achievements and effectively communicate what they have learned.

WHAT IS INVOLVED?

Minimal, after the course learning objectives

have been clearly identified. Can be high if
Instructor Preparation
multiple graders are to be trained (e.g.,
Time:
graduate teaching assistants) when used in
large classes.
Clear expectations must be provided to
Preparing Your Students:
students at the beginning of the course.
Class Time: None.
Disciplines: Appropriate for all.
Most applicable in small classes (n <30);
possible in large classes with pre-existing
Class Size:
infrastructure and less "open ended" character
of evidence allowed.
Special
Classroom/Technical None.
Requirements:
Individual or Group
Individual.
Involvement:
Analyzing Results: Intense and requires a scoring rubric.
Materials are presented in the natural
Other Things to Consider: language of the student and will vary widely
within one class.

Description
Student portfolios are a collection of evidence, prepared by the student and evaluated
by the faculty member, to demonstrate mastery, comprehension, application, and
synthesis of a given set of concepts. Accordingly, portfolio assessment strategies
substantially increase the rigor of an introductory science or mathematics course. For
example, in a physics course, this might include quantitative analysis of a video
showing motion. In a geology course, this might include an analysis of the impact of
agriculture on the community's water quality using locally acquired data. Students
must organize, synthesize, and clearly describe their achievements and effectively
communicate what they have learned. The evidence can be presented in a three-ring
binder, as a multimedia tour, or as a series of short papers.

A unique aspect of a successful portfolio is that it also contains explicit statements of

self-reflection. Statements accompanying each item describe how the student went
about mastering the material, why the presented piece of evidence demonstrates
mastery, and why mastery of such material is relevant to contexts outside the
classroom. Self-reflections make it clear to the reader the processes of integration that
have occurred during the learning process. Often, this is achieved with an introductory
letter to the reader or as a summary at the end of each section. Such reflections insure
that the student has personally recognized the relevance and level of achievement
acquired during creation and presentation of the portfolio. It is this self-reflection that
makes a portfolio much more valuable than a folder of student-selected work.

Assessment Purposes
The overall goal of the preparation of a portfolio is for the learner to demonstrate and
provide evidence that he or she has mastered a given set of learning objectives. More
than just thick folders containing student work, portfolios are typically personalized,
long-term representations of a student's own efforts and achievements. Whereas
multiple-choice tests are designed to determine what the student doesn't know,
portfolio assessments emphasize what the student does know.

Limitations
Portfolio assessments provide students and faculty with a direct view of how students
organize knowledge into overarching concepts. As such, portfolios are inappropriate for
measuring students' levels of factual knowledge (i.e., recall knowledge) or for drill-
and-skill activities and accordingly should be used in concert with more conventional
forms of assessment. Similarly, student work completed beyond the context of the
classroom is occasionally subject to issues of academic dishonesty.

Teaching Goals

 Develop ability to communicate scientific conceptions accurately

 Develop ability to write effectively using graphics as support
 Develop ability to relate principle concepts to real-world applications
 Develop ability to cite sources and references appropriately
 Develop ability to synthesize and integrate information and ideas
 Develop ability to be reflective and effectively conduct self-assessment
 Develop ability to think creatively and critically

Suggestions for Use

Portfolios are most appropriate when students need to integrate a number of complex
ideas, procedures, and relationships. Portfolios can move much of the responsibility of
assessment from the instructor to the student if the learner is instructed to
demonstrate and provide evidence that he or she has mastered a given set of learning
objectives. The most useful portfolios are composed of student solutions to
multifaceted tasks. Such tasks are typically complex, somewhat undefined, engaging
problems that require students to apply, synthesize, and evaluate various problem-
solving approaches.

Figure 1 - Illustrative Course Learning Objectives

  List of Course Learning Objectives for Introductory Environmental

Geology

1. The size of the human population, and the causes for change in its
size in various areas of the world.
2. The source, use, pollution and cleanup of the worlds water resources.
3. The origin and evolution of soils, and the way soils are affected by
agriculture.
4. Current and alternative sources of food.
5. The origin, advantages and disadvantages of current sources of
energy.
6. The origin, operation and potential for alternative sources of energy.
7. The causes of extinction and the processes which control the rate of
extinction.
8. Factors which control the use of land by people.
9. The geologic processes which cause earthquakes, and the potential for
predicting and preventing such events.
10. The origin, extraction and importance of ores.
11. The composition, management and recycle potential for solid &
hazardous waste material.
12. The origin, evolution and productivity of coastal areas.
13. The impact of human activities on coastal areas.
14. The origin, effect and remediation of atmospheric pollution.
15. How humans affect the earth's environment.

  List of Course Learning Objectives for First Semester Algebra-based

College Physics

1. Understand the nature of scientific knowledge and the various

disciplines of science.
2. Appreciate the historical and practical uses of units and measures.
3. Convert numerical quantities from one system of units to another and
within a given system.
4. Describe the various concepts and units used to describe motion.
5. Solve one-dimensional problems related to the acceleration of objects
due to gravity.
6. Diagram and describe quantitatively the motion of a projectile.
7. Appropriately apply vectors qualitatively to describe physical
situations.
8. Use vectors to quantitatively solve problems relating to motion.
 9. Create a free-body diagram to represent the total force on an object
(including friction).
10. State and apply the laws of motion developed by Isaac Newton.
11. Solve problems related to static equilibrium and rotational equilibrium.
12. Apply the Law of Universal Gravitation to objects moving in circles.
13. Calculate the work done on an object and its relationship to energy.
14. Quantitatively and qualitatively describe systems in which energy is
conserved.
15. Identify the various sources of energy and power.
16. Solve problems related to impulse and the Conservation of
Momentum.
17. Apply principles of fluid dynamics to describe phenomena in nature.
18. Distinguish between heat and temperature.
19. Identify the ways that heat can be transferred between two points.
20. Explain the distinguishing characteristics of solids, liquids, and gases.
21. State the laws of thermodynamics and their importance to technology.
22. Solve problems relating to periodic (cyclical) motion.
23. Describe the properties of sound waves with respect to pitch, volume,
and intensity.
24. Apply the Doppler Effect to physical situations quantitatively and
qualitatively.

Step-by-Step Instructions

 Carefully construct and distribute 12-25 overarching learning objectives for the
course.
 Decide if a portfolio supports student learning and assessment for these
objectives.
 Determine if the portfolio is primarily a learning activity or an assessment tool.
 Inform students of your expectations that students have the opportunity to
clearly demonstrate to the professor that course learning objectives have been
attained.
 Require that each piece of evidence must be clearly labeled as to which
objective the evidence pertains.
 Require that each piece of evidence must be accompanied by a written
paragraph of rationale and a separate written paragraph of self-reflection.
 Emphasize to students that it is their responsibility to clearly demonstrate
mastery of the learning objectives for this course.
 Score each item of evidence in the portfolio according to a scheme that has
been distributed to the students when the portfolios are initially assigned.

Variations
Showcase Portfolios
A showcase portfolio is a limited portfolio where a student is only allowed to present a
few pieces of evidence to demonstrate mastery of learning objectives. Especially useful
in a laboratory course, a showcase portfolio might ask a student to include items that
represent: (i) their best work; (ii) their most interesting work; (iii) their most improved
work; (iv) their most disappointing work; (v) and their favorite work. Items could be
homework assignments, examinations, laboratory reports, news clippings, or other
creative works. An introductory letter that describes why each particular item was
included and what it demonstrates makes this type of portfolio especially insightful to
the instructor.

Checklist Portfolios
A checklist portfolio is composed of a predetermined number of items. Often, a course
syllabus will have a predetermined number of assignments for students to complete. A
checklist portfolio takes advantage of such a format and gives the students the choice
of a number of different assignment selections to complete in the course of learning
science. For example, instead of assigning exactly 12 sets of problems from the end of
each text chapter, students could have the option of replacing several assignments
with relevant magazine article reviews or laboratory reports that clearly demonstrate
mastery of a given learning objective. Additionally, class quizzes and tests can become
part of the portfolio if that is what is on the checklist of items to be included. A sample
checklist might require a portfolio to have 10 correctly worked problem sets, two
magazine article summaries, two laboratory reports, and two examinations in addition
to self-reflection paragraphs where the student decides which objectives most closely
fit which assignments.

Open-Format Portfolios
An open-format for a portfolio generally provides the most insightful view of a
student's level of achievement. In an open-format portfolio, students are allowed to
submit anything they wish to be considered as evidence for mastery of a given list of
learning objectives. In addition to the traditional items like exams and assignments,
students can include reports on museum visits, analysis of amusement park rides,
imaginative homework problems, and other sources from the "real world". Although
these portfolios are more difficult for the student to create and for the instructor to
score, many students report that they are very proud of the time spent on such a
portfolio.

Use in Large Enrollment Courses

Portfolios can be used successfully in large courses provided there is an infrastructure
for students and instructors to utilize. Most importantly, the format of each item in the
portfolio needs to be in a similar format; the use of cover sheets, forms, and
prescribed notebooks often helps. Second, students creativity must be sacrificed to
some degree for the sake of uniformity. This can be accomplished by assigning student
tasks that have fewer multiple-correct solutions. Finally, if graduate teaching assistants
are involved, each assistant should take responsibility for a particular series of learning
goals, thus becoming an expert and seeing all student submissions. If announced to
the students, this helps curtail academic dishonesty and variation in scoring.

Analysis
Because each portfolio is individualized, student assessment must be compiled by
looking at the portfolio's contents relative to the course learning objectives. Each piece
of evidence should be graded according to a predetermined scheme. The items can be
scored discretely as a 0, 1, 2, or 3 based on the grader's judgment about the student's
presentation as related to the stated learning goals. (A larger scale can be used, but
the reliability of different faculty giving the student the same score decreases.)

Figure 2 - Illustrative Grading Criteria for Portfolios

  Grading Criteria
 Each individual piece of evidence will be graded according to the following
scale:

  Grading Rubric
The overall portfolio is scored as follows as an indication of the extent to
which the portfolio indicates that the student has mastered the 15 course
objectives listed elsewhere in the syllabus:
Grade:
Rubric:

Strong evidence in at least 12 objectives;

A
adequate in other three;
Strong evidence in at least 12 objectives;
B+
adequate in at least one other;
Strong evidence in 10 objectives; adequate
B
in all others;
Strong evidence in 9 objectives; adequate in
C+
others;
Strong evidence in 9 objectives; adequate in
C
at least one other;
D+ Adequate evidence in 12 objectives;
D Adequate evidence in 10 objectives;
Adequate evidence in less than 10
F
objectives;

  Submission and Possession of Evidence:

Submission of evidence for mastery of each objective is to be done during
laboratory class meetings and the portfolios will be securely maintained in the
laboratory. It is your responsibility to see that your portfolio is current and
accurate. No late submissions will be accepted. Any submissions remotely
suspected of plagiarism will receive a score of 0.

Evidence scored as a 0 or a 1 is rather straightforward based on the criteria listed in

figure 2. The most difficult judgment usually lies between awarding a score of 2 and a
score of 3. In particular, a score of 2 is awarded if the student has addressed the
learning objective correctly and clearly, but only at the literal-descriptive level; there is
little explicit integration across concepts or indication of relevance to the student. A
common characteristic of such evidence is that facts are not used to support an opinion
or position. Furthermore, evidence that does not clearly identify relevance to the
student's life or career path is also given a score of 2. To be awarded a score of 3, the
evidence must clearly indicate that the student understands the objective in an
integrated fashion. Such evidence provides the reader deep insight into the complexity
of the student's comprehension.

Viewing student portfolios from this perspective drastically changes the emphasis from
collections of facts to encompassing concepts. Such a grading procedure also shifts
responsibility for demonstrating competence from the instructor to the student.
Effectively shifting this responsibility affects comments placed in the portfolio by the
grader; comments are directed toward improving the next submission as well as
indicating the inadequacies of the current evidence.

Pros and Cons

 Portfolios put the responsibility of demonstrating knowledge and integration

across concepts on the students
 Portfolios provide a structure for long-duration assignments
 Portfolios encourage student creativity and allow for students to emphasize the
aspects of a concept most relevant to them in meaningful ways
 Portfolios engender self-reflection and self-assessment

However:

 Portfolios take longer to score than machine graded multiple-choice exams

 Portfolios involve student work outside of class
 Portfolios do not easily demonstrate students' knowledge-recall abilities
 Students who have been successful at memorizing their way to an "A" initially
find portfolios intimidating

QUIZZES, TESTS, AND

EXAMS
By Barbara Gross Davis, University of California, Berkeley.
From Tools for Teaching, copyright by Jossey-Bass. For purchase or reprint information,
contact Jossey-Bass. Reprinted here with permission, September 1, 1999.

Many teachers dislike preparing and grading exams, and most students dread taking
them. Yet tests are powerful educational tools that serve at least four functions. First,
tests help you evaluate students and assess whether they are learning what you are
expecting them to learn. Second, well-designed tests serve to motivate and help
students structure their academic efforts. Crooks (1988), McKeachie (1986), and
Wergin (1988) report that students study in ways that reflect how they think they will
be tested. If they expect an exam focused on facts, they will memorize details; if they
expect a test that will require problem solving or integrating knowledge, they will
work toward understanding and applying information. Third, tests can help you
understand how successfully you are presenting the material. Finally, tests can
reinforce learning by providing students with indicators of what topics or skills they
have not yet mastered and should concentrate on. Despite these benefits, testing is
also emotionally charged and anxiety producing. The following suggestions can
enhance your ability to design tests that are effective in motivating, measuring, and
reinforcing learning.
A note on terminology: instructors often use the terms tests, exams, and even quizzes
interchangeably. Test experts Jacobs and Chase (1992), however, make distinctions
among them based on the scope of content covered and their weight or importance in
calculating the final grade for the course. An examination is the most comprehensive
form of testing, typically given at the end of the term (as a final) and one or two times
during the semester (as midterms). A test is more limited in scope, focusing on
particular aspects of the course material. A course might have three or four tests. A
quiz is even more limited and usually is administered in fifteen minutes or less.
Though these distinctions are useful, the terms test and exam will be used
interchangeably throughout the rest of this section because the principles in planning,
constructing, and administering them are similar.
General Strategies

Spend adequate amounts of time developing your tests. As you prepare a test,
think carefully about the learning outcomes you wish to measure, the type of items
best suited to those outcomes, the range of difficulty of items, the length and time
limits for the test, the format and layout of the exam, and your scoring procedures.

Match your tests to the content you are teaching. Ideally, the tests you give will
measure students' achievement of your educational goals for the course. Test items
should be based on the content and skills that are most important for your students to
learn. To keep track of how well your tests reflect your objectives, you can construct
a grid, listing your course objectives along the side of the page and content areas
along the top. For each test item, check off the objective and content it covers.
(Sources: Ericksen, 1969; Jacobs and Chase, 1992; Svinicki and Woodward, 1982)

Try to make your tests valid, reliable, and balanced. A test is valid if its results
are appropriate and useful for making decisions about an aspect of students'
achievement (Gronlund and Linn, 1990). Technically, validity refers to the
appropriateness of the interpretation of the results and not to the test itself, though
colloquially we speak about a test being valid. Validity is a matter of degree and
considered in relation to specific use or interpretation (Gronlund and Linn, 1990). For
example, the results of a writing test may have a high degree of validity for indicating
the level of a student's composition skills, a moderate degree of validity for
predicting success in later composition courses, and essentially no validity for
predicting success in mathematics or physics. Validity can be difficult to determine.
A practical approach is to focus on content validity, the extent to which the content of
the test represents an adequate sampling of the knowledge and skills taught in the
course. If you design the test to cover information in lectures and readings in
proportion to their importance in the course, then the interpretations of test scores are
likely to have greater validity An exam that consists of only a few difficult items,
however, will not yield valid interpretations of what students know.

A test is reliable if it accurately and consistently evaluates a student's performance.

The purest measure of reliability would entail having a group of students take the
same test twice and get the same scores (assuming that we could erase their
memories of test items from the first administration). This is impractical, of course,
but there are technical procedures for determining reliability. In general, ambiguous
questions, unclear directions, and vague scoring criteria threaten reliability. Very
short tests are also unlikely to be highly reliable. It is also important for a test to be
balanced: to cover most of the main ideas and important concepts in proportion to the
emphasis they received in class.

If you are interested in learning more about psychometric concepts and the technical
properties of tests, here are some books you might review:

Ebel, R. L., and Frisbie, D. A. Essentials of Educational Measurement. (5th ed.)

Englewood Cliffs, N.J.: Prentice-Hall, 1990.

Gronlund, N. E., and Linn, R. Measurement and Evaluation in Teaching. (6th ed.)
New York: Macmillan, 1990.

Mehrens, W. A., and Lehmann, I. J. Measurement and Evaluation in Education and

Psychology. (4th ed.) New York: Holt, Rinehart & Winston, 1991.

Use a variety of testing methods. Research shows that students vary in their
preferences for different formats, so using a variety of methods will help students do
their best (Jacobs and Chase, 1992). Multiple-choice or shortanswer questions are
appropriate for assessing students' mastery of details and specific knowledge, while
essay questions assess comprehension, the ability to integrate and synthesize, and the
ability to apply information to new situations. A single test can have several formats.
Try to avoid introducing a new format on the final exam: if you have given all
multiple-choice quizzes or midterms, don't ask students to write an all-essay final.
(Sources: Jacobs and Chase, 1992; Lowman, 1984; McKeachie, 1986; Svinicki,
1987)

Write questions that test skills other than recall. Research shows that most tests
administered by faculty rely too heavily on students' recall of information (Milton,
Pollio, and Eison, 1986). Bloom (1956) argues that it is important for tests to measure
higher-learning as well. Fuhrmann and Grasha (1983, p. 170) have adapted Bloom's
taxonomy for test development. Here is a condensation of their list:

To measure knowledge (common terms, facts, principles, procedures), ask these

kinds of questions: Define, Describe, Identify, Label, List, Match, Name, Outline,
Reproduce, Select, State. Example: "List the steps involved in titration."

To measure comprehension (understanding of facts and principles, interpretation of

material), ask these kinds of questions: Convert, Defend, Distinguish, Estimate,
Explain, Extend, Generalize, Give examples, Infer, Predict, Summarize. Example:
"Summarize the basic tenets of deconstructionism."

To measure application (solving problems, applying concepts and principles to new

situations), ask these kinds of questions: Demonstrate, Modify, Operate, Prepare,
Produce, Relate, Show, Solve, Use. Example: "Calculate the deflection of a beam
under uniform loading."

To measure analysis (recognition of unstated assumptions or logical fallacies, ability

to distinguish between facts and inferences), ask these kinds of questions: Diagram,
Differentiate, Distinguish, Illustrate, Infer, Point out, Relate, Select, Separate,
Subdivide. Example: "In the president's State of the Union Address, which statements
are based on facts and which are based on assumptions?"

To measure synthesis (integrate learning from different areas or solve problems by

creative thinking), ask these kinds of questions: Categorize, Combine, Compile,
Devise, Design, Explain, Generate, Organize, Plan, Rearrange, Reconstruct, Revise,
Tell. Example: "How would you restructure the school day to reflect children's
developmental needs?"

To measure evaluation (judging and assessing), ask these kinds of questions:

Appraise, Compare, Conclude, Contrast, Criticize, Describe, Discriminate, Explain,
Justify, Interpret, Support. Example: "Why is Bach's Mass in B Minor acknowledged
as a classic?"

Many faculty members have found it difficult to apply this six-level taxonomy, and
some educators have simplified and collapsed the taxonomy into three general levels
(Crooks, 1988): The first category knowledge (recall or recognition of specific
information). The second category combines comprehension and application. The
third category is described as "problem solving," transferring existing knowledge and
skills to new situations.

If your course has graduate student instructors (GSIs), involve them in

designing exams. At the least, ask your GSIs to read your draft of the exam and
comment on it. Better still, involve them in creating the exam. Not only will they
have useful suggestions, but their participation in designing an exam will help them
grade the exam.

Take precautions to avoid cheating. See "Preventing Academic Dishonesty"

Types of Tests

Multiple-choice tests. Multiple-choice items can be used to measure both simple

knowledge and complex concepts. Since multiple-choice questions can be answered
quickly, you can assess students' mastery of many topics on an hour exam. In
addition, the items can be easily and reliably scored. Good multiple-choice questions
are difficult to write-see "Multiple-Choice and Matching Tests" for guidance on how
to develop and administer this type of test.

True-false tests. Because random guessing will produce the correct answer half the
time, true-false tests are less reliable than other types of exams. However, these items
are appropriate for occasional use. Some faculty who use true-false questions add an
"explain" column in which students write one or two sentences justifying their
response.

Matching tests. The matching format is an effective way to test students' recognition
of the relationships between words and definitions, events and dates, categories and
examples, and so on. See "Multiple-Choice and Matching Tests" for suggestions
about developing this type of test.

Essay tests. Essay tests enable you to judge students' abilities to organize, integrate,
interpret material, and express themselves in their own words. Research indicates that
students study more efficiently for essay-type examinations than for selection
(multiple-choice) tests: students preparing for essay tests focus on broad issues,
general concepts, and interrelationships rather than on specific details, and this
studying results in somewhat better student performance regardless of the type of
exam they are given (McKeachie, 1986). Essay tests also give you an opportunity to
comment on students' progress, the quality of their thinking, the depth of their
understanding, and the difficulties they may be having. However, because essay tests
pose only a few questions, their content validity may be low. In addition, the
reliability of essay tests is compromised by subjectivity or inconsistencies in grading.
For specific advice, see "Short-Answer and Essay Tests." (Sources: Ericksen, 1969,
McKeachie, 1986)

A variation of an essay test asks students to correct mock answers. One faculty
member prepares a test that requires students to correct, expand, or refute mock
essays. Two weeks before the exam date, he distributes ten to twelve essay questions,
which he discusses with students in class. For the actual exam, he selects four of the
questions and prepares well-written but intellectually flawed answers for the students
to edit, correct, expand, and refute. The mock essays contain common
misunderstandings, correct but incomplete responses, or absurd notions; in some
cases the answer has only one or two flaws. He reports that students seem to enjoy
this type of test more than traditional examinations.

Short-answer tests. Depending on your objectives, short-answer questions can call

for one or two sentences or a long paragraph. Short-answer tests are easier to write,
though they take longer to score, than multiple-choice tests.

They also give you some opportunity to see how well students can express their
thoughts, though they are not as useful as longer essay responses for this purpose.
See "Short-Answer and Essay Tests" for detailed guidelines.
Problem sets. In courses in mathematics and the sciences, your tests can include
problem sets. As a rule of thumb, allow students ten minutes to solve a problem you
can do in two minutes. See "Homework: Problem Sets" for advice on creating and
grading problem sets.

Oral exams. Though common at the graduate level, oral exams are rarely used for
undergraduates except in foreign language classes. In other classes they are usually
time-consuming, too anxiety provoking for students, and difficult to score unless the
instructor tape-records the answers. However, a math professor has experimented
with individual thirty-minute oral tests in a small seminar class. Students receive the
questions in advance and are allowed to drop one of their choosing. During the oral
exam, the professor probes students' level of understanding of the theory and
principles behind the theorems. He reports that about eight students per day can be
tested.

Performance tests. Performance tests ask students to demonstrate proficiency in

conducting an experiment, executing a series of steps in a reasonable amount of time,
following instructions, creating drawings, manipulating materials or equipment, or
reacting to real or simulated situations. Performance tests can be administered
individually or in groups. They are seldom used in colleges and universities because
they are logistically difficult to set up, hard to score, and the content of most courses
does not necessarily lend itself to this type of testing. However, performance tests can
be useful in classes that require students to demonstrate their skills (for example,
health fields, the sciences, education). If you use performance tests, Anderson (1987,
p. 43) recommends that you do the following (I have slightly modified her list):

 Specify the criteria to be used for rating or scoring (for example, the level of
accuracy in performing the steps in sequence or completing the task within a
specified time limit).
 State the problem so that students know exactly what they are supposed to do
(if possible, conditions of a performance test should mirror a real-life
situation).
 Give students a chance to perform the task more than once or to perform
several task samples.

"Create-a-game" exams. For one midterm, ask students to create either a board
game, word game, or trivia game that covers the range of information relevant to
your course. Students must include the rules, game board, game pieces, and whatever
else is needed to play. For example, students in a history of psychology class created
"Freud's Inner Circle," in which students move tokens such as small cigars and toilet
seats around a board each time they answer a question correctly, and "Psychogories,"
a card game in which players select and discard cards until they have a full hand of
theoretically compatible psychological theories, beliefs, or assumptions. (Source:
Berrenberg and Prosser, 1991)
 Alternative Testing Modes

Take-home tests. Take-home tests allow students to work at their own pace with
access to books and materials. Take-home tests also permit longer and more involved
questions, without sacrificing valuable class time for exams. Problem sets, short
answers, and essays are the most appropriate kinds of take-home exams. Be wary,
though, of designing a take-home exam that is too difficult or an exam that does not
include limits on the number of words or time spent (Jedrey, 1984). Also, be sure to
give students explicit instructions on what they can and cannot do: for example, are
they allowed to talk to other students about their answers? A variation of a take-home
test is to give the topics in advance but ask the students to write their answers in
class. Some faculty hand out ten or twelve questions the week before an exam and
announce that three of those questions will appear on the exam.

Open-book tests. Open-book tests simulate the situations professionals face every
day, when they use resources to solve problems, prepare reports, or write memos.
Open-book tests tend to be inappropriate in introductory courses in which facts must
be learned or skills thoroughly mastered if the student is to progress to more
complicated concepts and techniques in advanced courses. On an open-book test,
students who are lacking basic knowledge may waste too much of their time
consulting their references rather than writing. Open-book tests appear to reduce
stress (Boniface, 1985; Liska and Simonson, 1991), but research shows that students
do not necessarily perform significantly better on open-book tests (Clift and Imrie,
1981; Crooks, 1988). Further, open-book tests seem to reduce students' motivation to
study. A compromise between open- and closed-book testing is to let students bring
an index card or one page of notes to the exam or to distribute appropriate reference
material such as equations or formulas as part of the test.

Group exams. Some faculty have successfully experimented with group exams,
either in class or as take-home projects. Faculty report that groups outperform
individuals and that students respond positively to group exams (Geiger, 1991;
Hendrickson, 1990; Keyworth, 1989; Toppins 1989). For example, for a fifty-minute
in-class exam, use a multiple-choice test of about twenty to twenty-five items. For the
first test, the groups can be randomly divided. Groups of three to five students seem
to work best. For subsequent tests, you may want to assign students to groups in ways
that minimize differences between group scores and balance talkative and quiet
students. Or you might want to group students who are performing at or near the
same level (based on students' performance on individual tests). Some faculty have
students complete the test individually before meeting as a group. Others just let the
groups discuss the test, item by item. In the first case, if the group score is higher
than the individual score of any member, bonus points are added to each individual's
score. In the second case, each student receives the score of the group. Faculty who
use group exams offer the following tips:

 Ask students to discuss each question fully and weigh the merits of each
 answer rather than simply vote on an answer.
 If you assign problems, have each student work a problem and then compare
results.
 If you want students to take the exam individually first, consider devoting two
class periods to tests; one for individual work and the other for group.
 Show students the distribution of their scores as individuals and as groups; in
most cases group scores will be higher than any single individual score.

A variation of this idea is to have students first work on an exam in groups outside of
class. Students then complete the exam individually during class time and receive
their own score. Some portion of the test items are derived from the group exam. The
rest are new questions. Or let students know in advance you will be asking them to
justify a few of their responses; this will keep students from blithely relying on their
work group for all the answers. (Sources: Geiger, 1991; Hendrickson, 1990;
Keyworth, 1989; Murray, 1990; Toppins, 1989)

Paired testing. For paired exams, pairs of students work on a single essay exam, and
the two students turn in one paper. Some students may be reluctant to share a grade,
but good students will most likely earn the same grade they would have working
alone. Pairs can be self-selected or assigned. For example, pairing a student who is
doing well in the course with one not doing well allows for some peer teaching. A
variation is to have students work in teams but submit individual answer sheets.
(Source: Murray, 1990)

Portfolios. A portfolio is not a specific test but rather a cumulative collection of a

student's work. Students decide what examples to include that characterize their
growth and accomplishment over the term. While most common in composition
classes, portfolios are beginning to be used in other disciplines to provide a fuller
picture of students' achievements. A student's portfolio might include sample papers
(first drafts and revisions), journal entries, essay exams, and other work
representative of the student's progress. You can assign portfolios a letter grade or a
pass/not pass. If you do grade portfolios, you will need to establish clear criteria.
(Source: Jacobs and Chase, 1992)
Construction of Effective Exams

Prepare new exams each time you teach a course. Though it is timeconsuming to
develop tests, a past exam may not reflect changes in how you have presented the
material or which topics you have emphasized in the course. If you do write a new
exam, you can make copies of the old exam available to students.

Make up test items throughout the term. Don't wait until a week or so before the
exam. One way to make sure the exam reflects the topics emphasized in the course is
to write test questions at the end of each class session and place them on index cards
or computer files for later sorting. Software that allows you to create test banks of
items and generate exams from the pool is now available.

Ask students to submit test questions. Faculty who use this technique limit the
number of items a student can submit and receive credit for. Here is an example
(adapted from Buchanan and Rogers, 1990, p. 72):

You can submit up to two questions per exam. Each question must be typed or
legibly printed on a separate 5" x 8" card. The correct answer and the source (that is,
page of the text, date of lecture, and so on) must be provided for each question.
Questions can be of the short-answer, multiple-choice, or essay type.

Students receive a few points of additional credit for each question they submit that is
judged appropriate. Not all students will take advantage of this opportunity. You can
select or adapt student's test items for the exam. If you have a large lecture class, tell
your students that you might not review all items but will draw randomly from the
pool until you have enough questions for the exam. (Sources: Buchanan and Rogers,
1990; Fuhrmann and Grasha, 1983)

Cull items from colleagues' exams. Ask colleagues at other institutions for copies of
their exams. Be careful, though, about using items from tests given by colleagues on
your own campus. Some of your students may have previously seen those tests.

Consider making your tests cumulative. Cumulative tests require students to

review material they have already studied, thus reinforcing what they have learned.
Cumulative tests also give students a chance to integrate and synthesize course
content. (Sources: Crooks, 1988; Jacobs and Chase, 1992; Svinicki, 1987)

Prepare clear instructions. Test your instructions by asking a colleague (or one of
your graduate student instructors) to read them.

Include a few words of advice and encouragement on the exam. For example,
give students advice on how much time to spend on each section or offer a hint at the
beginning of an essay question or wish students good luck. (Source: "Exams:
Alternative Ideas and Approaches," 1989)

Put some easy items first. Place several questions all your students can answer near
the beginning of the exam. Answering easier questions helps students overcome their
nervousness and may help them feel confident that they can succeed on the exam.
You can also use the first few questions to identify students in serious academic
difficulty. (Source: Savitz, 1985)

Challenge your best students. Some instructors like to include at least one very
difficult question -- though not a trick question or a trivial one -- to challenge the
interest of the best students. They place that question at or near the end of the exam.
Try out the timing. No purpose is served by creating a test too long for even well-
prepared students to finish and review before turning it in. As a rule of thumb, allow
about one-half minute per item for true-false tests, one minute per item for multiple-
choice tests, two minutes per short-answer requiring a few sentences, ten or fifteen
minutes for a limited essay question, and about thirty minutes for a broader essay
question. Allow another five or ten minutes for students to review their work, and
factor in time to distribute and collect the tests. Another rule of thumb is to allow
students about four times as long as it takes you (or a graduate student instructor) to
complete the test. (Source: McKeachie, 1986)

Give some thought to the layout of the test. Use margins and line spacing that
make the test easy to read. If items are worth different numbers of points, indicate the
point value next to each item. Group similar types of items, such as all true-false
questions, together. Keep in mind that the amount of space you leave for short-
answer questions often signifies to the students the length of the answer expected of
them. If students are to write on the exam rather than in a blue book, leave space at
the top of each page for the student's name (and section, if appropriate). If each page
is identified, the exams can be separated so that each graduate student instructor can
grade the same questions on every test paper, for courses that have GSIs.
References

Anderson, S. B. "The Role of the Teacher-Made Test in Higher Education." In D.

Bray and M. J. Blecher (eds.), Issues in Student Assessment. New Directions for
Community Colleges, no. 59. San Francisco: Jossey-Bass, 1987.

Berrenberg, J. L., and Prosser, A. "The Create-a-Game Exam: A Method to Facilitate

Student Interest and Learning." Teaching of Psychology, 1991, 18(3), 167-169.

Bloom, B. S. (ed.). Taxonomy of Educational Objectives. Vol. 1: Cognitive Domain.

New York: McKay, 1956.

Boniface, D. "Candidates' Use of Notes and Textbooks During an Open Book

Examination." Educational Research, 1985, 27(3), 201-209.

Brown, I. W. "To Learn Is to Teach Is to Create the Final Exam." College Teaching,
1991, 39(4), 150-153.

Buchanan, R. W., and Rogers, M. "Innovative Assessment in Large Classes."

College Teaching, 1990, 38(2), 69-73.

Clift, J. C., and Imrie, B. W. Assessing Students, Appraising Teaching. New York:
Wiley, 1981.

Crooks, T. J. "The Impact of Classroom Evaluation Practices on Students." Review of

Educational Research, 1988, 58(4), 438-481.

Ericksen, S. C. "The Teacher-Made Test." Memo to the Faculty, no. 35. Ann Arbor:
Center for Research on Learning and Teaching, University of Michigan, 1969.

"Exams: Alternative Ideas and Approaches." Teaching Professor, 1989, 3(8), 3-4.

Fuhrmann, B. S., and Grasha, A. F. A Practical Handbook for College Teachers.

Boston: Little, Brown, 1983.

Geiger, T. "Test Partners: A Formula for Success." Innovation Abstracts, 1991, 13

(l1). (Newsletter published by College of Education, University of Texas at Austin)

Gronlund, N. E., and Linn, R. Measurement and Evaluation in Teaching. (6th ed.)
New York: Macmillan, 1990.

Hendrickson, A. D. "Cooperative Group Test-Taking." Focus, 1990, 5(2), 6.

(Publication of the Office of Educational Development Programs, University of
Minnesota)

Jacobs, L. C., and Chase, C. I. Developing and Using Tests Effectively: A Guide for
Faculty. San Francisco: Jossey-Bass, 1992.

Jedrey, C. M. "Grading and Evaluation." In M. M. Gullette (ed.), The Art and Craft
of Teaching. Cambridge, Mass.: Harvard University Press, 1984.

Keyworth, D. R. "The Group Exam." Teaching Professor, 1989, 3(8), 5.

Liska, T., and Simonson, J. "Open-Text and Open-Note Exams." Teaching

Professor, 1991, 5(5), 1-2.

Lowman, J. Mastering the Techniques of Teaching. San Francisco: Jossey-Bass,

1984.

McKeachie, W. J. Teaching Tips. (8th ed.) Lexington, Mass.: Heath, 1986.

Milton, O., Pollio, H. R., and Eison, J. A. Making Sense of College Grades: Why the
Grading System Does Not Work and What Can Be Done About It. San Francisco:
Jossey-Bass, 1986.

Murray, J. P. "Better Testing for Better Learning." College Teaching, 1990, 38(4),
148-152.

Savitz, F. "Effects of Easy Examination Questions Placed at the Beginning of

Science Multiple-Choice Examinations." Journal of Instructional Psychology, 1985,
 12(l), 6-10.

Svinicki, M. D. "Comprehensive Finals." Newsletter, 1987, 9(2), 1-2. (Publication of

the Center for Teaching Effectiveness, University of Texas at Austin)

Svinicki, M. D., and Woodward, P. J. "Writing Higher-Level Objective Test Items."

In K. G. Lewis (ed.), Taming the Pedagogical Monster. Austin: Center for Teaching
Effectiveness, University of Texas, 1982.

Toppins, A. D. "Teaching by Testing: A Group Consensus Approach." College

Teaching, 1989, 37(3), 96-99.

Wergin, J. F. "Basic Issues and Principles in Classroom Assessment." In J. H.

McMillan (ed.), Assessing Students' Learning. New Directions for Teaching and
Learning, no. 34. San Francisco: Jossey-Bass, 1988.

Why Use a Multiple Choice Test?

Multiple choice testing is an efficient and effective way to assess a wide range of
knowledge, skills, attitudes and abilities (Haladyna, 1999). When done well, it allows
broad and even deep coverage of content in a relatively efficient way. Though often
maligned, and though it is true that no single format should be used exclusively for
assessment (American Educational Research Association, the American Psychological
Association, and the National Council on Measurement in Education , 1999), multiple
choice testing still remains one of the most commonly used assessment formats
(Haladyna, 1999; McDougall, 1997).

What is a Multiple Choice Test?

The multiple-choice test is a very flexible assessment format that can be used to
measure knowledge, skills, abilities, values, thinking skills, etc. Such a test usually
consists of a number of items that pose a question to which students must select an
answer from among a number of choices. Items can also be statements to which
students must find the best completion. Multiple-choice items, therefore, are
fundamentally recognition tasks, where students must identify the correct response.

WHAT IS INVOLVED?

Medium to high if you are writing your own

Instructor Preparation
items the first time; low if you have validated
Time:
items.
Little or none. Especially in introductory
classes, it might be wise not to assume that
Preparing Your Students: students know strategies for taking multiple-
choice tests. Some time spent on test taking
strategies may be useful.
 Class Time: Depends on the length of the test.
Disciplines: Any.
Class Size: Any. Especially efficient in large classes.
Special
None. An optical scanner and scan sheets may
Classroom/Technical
be useful with large classes.
Requirements:
Individual or Group
Usually individual; team testing is possible.
Involvement:
For grading purposes, analysis is usually quick
and straightforward. For developing and
Analyzing Results:
refining items or for diagnostic purposes,
analysis can be a little more complex.
Logistical concerns such as your students
Other Things to Consider:
using scanning sheets correctly.

Description

A multiple choice test is constructed of multiple choice items, which consist of two,
sometimes three, parts as shown below.

The Stem of a multiple choice item is the part to which the student is to respond. In
everyday language, we'd call this the question, but since it could be a statement or
even an analogy or an equation, we'll use the technical term, Stem.

The Options are the choices from which examinees are to choose. There are two kinds
of options: the Key is the correct or best choice; the Distracters are incorrect or less
appropriate choices.

Sometimes Stimulus Materials can also be used with a multiple choice item, something
like a bar graph, a table, a map, a short text, etc. There will be more about stimulus
materials in a moment.

Assessment Purposes

Multiple-choice tests can be used to meet a variety of educational purposes, though

they most often are used in the classroom to measure academic achievement, and to
determine course grades. Other purposes to which they can be put include feedback to
students, instructional feedback, diagnosis of student misconceptions, and others. (See
the Suggestions for Use section.)
A Few Multiple Choice Myths

Multiple-choice tests are objective—Multiple-choice items are sometimes called

"objective" items, an unfortunate label. They can be just as subjective as an
essay question if they're poorly written . (Indeed, a well-crafted essay prompt
and scoring rubric can be much more objective than some multiple choice
items.) Subjectivity/objectivity does not reside in the format but in the
construction and scoring, thus objectivity must be planned into multiple-choice
questions (Dwyer, 1993).

Multiple-choice tests assess only superficial knowledge—It is perhaps because

faculty test as they were tested, not following state-of-the-art rules for testing,
that multiple choice has the reputation it does. Research has long shown that
college-level faculty do not write exams well (Guthrie, 1992; Lederhouse &
Lower, 1974; McDougall, 1997), and that both faculty and students notice the
side effects, like focusing on memorization and facts (Crooks, 1988; Shifflett,
Phibbs, & Sage, 1997).

Multiple choice tests are for grading only —This myth arises from the
misapprehension that assessment and instruction are separate stages of the
learning process. Indeed, there can be no instruction without sound
assessment, and, more importantly, both are critical for learning to occur. As
Crooks (1988) succinctly put it: "Too much emphasis has been placed on the
grading function of evaluation, and too little on its role in assisting students to
learn" (p. 468). There are a number of ways in which multiple choice items can
be designed and used to promote and refine learning, to inform instruction, as
well as to assign grades. Some of these will be highlighted throughout this CAT.

Teaching Goals

Student Learning Outcomes:

 Demonstrate recognition and recall of knowledge, skills and abilities

 Demonstrate analysis, synthesis, and evaluation
 Demonstrate critical thinking

Instructor Teaching Outcomes:

 Assess higher-order and lower-order thinking skills

 Assess content broadly and deeply
 Assess quickly and efficiently
 Identify student misconceptions

Step-by-Step Instructions

In content assessment, you should aim at finding out which students have mastered
the material and which have not, and to what extent they have or have not. The test
score should reflect that and only that. The "rules" of test development, then, work in
fundamentally two ways: to maximize the potential for the student to show you what
they know about the content, and to minimize the possibility that other factors will
influence the score. Such factors range from test forms that are hard to read and
navigate to fatigue to reading ability or computer literacy. In writing items, you must
eliminate "guessing" and "trick" questions. In test layout, that means avoiding student
confusion about how to complete the test.

Planning the Test: The critical issue at this stage is matching the test to your
teaching so that what was taught gets tested and what is tested gets taught. This is
fundamentally an issue of fairness as well as matching assessments to learning
outcomes. A great many of the complaints about testing stem from a mismatch here
("I didn't know that was going to be on the test!", "We didn't talk about that in
class!"). Constructing a test blueprint is an excellent planning tool for testing and for
teaching.

A test blueprint [sometimes called a Table of Specifications (e.g. Gronlund & Linn,
1990)] is a table that records the content areas for the test on one axis, which may be
listed by topic or text chapter or other divisions. The other axis of the table categorizes
the ways you expect your students to know that content. These can be traditional
taxonomies like Bloom's Taxonomy of Educational Objectives or your own categories.
Finally, each cell in the table records the relative weight you'll give to the intersection
of each content and way of knowing. These can be expressed as proportions or as
numbers of items. Here's an example of a test blueprint:

Example Test Blueprint for an Introductory Physics Course

Facts, Terminology, Problem Application and
Concepts Solving Integration
Work 10% 5% 5% 20%
Energy 10% 40% 10% 60%
Potential
4% 16% 0% 20%
Energy
Kinetic
3% 12% 5% 20%
Energy
Mechanical
3% 12% 5% 20%
Energy
Power 2% 8% 10% 20%
22% 53% 25%

Once you have the Test Blueprint constructed, you can use it to plan your teaching as
well as to write test items. It's useful in planning teaching because it records what
content you feel is most important and how you expect your students to know that
content. It also informs decisions relevant to your instructional planning. And, since
the goal is a match between teaching, testing and outcomes, the same emphases used
in teaching should be used in testing. The Test Blueprint thus serves as a powerful way
to align teaching and assessment (Nitko, 2001).

Writing the Items: Perhaps the first issue here is whether you will actually write your
own items or use items from test or item banks that accompany textbooks. Using
publishers' test banks comes with a number of risks. First, your test blueprint and the
publishers' may look different, so you'll need to ensure that the test still retains your
emphases. Second, it's doubtful that published item banks consist of tried and true,
high quality items (e.g. Hansen & Dexter, 1997; Sims, 1997). Neither authors nor
publishers usually receive compensation for providing item banks, and therefore little if
any item development work is likely conducted. Therefore, you should use published
items sparingly or, at least, use them very carefully with as much scrutiny as you
would your own items.

These tips and "rules" for item writing are synthesized from research-based sources
and/ or consensus-of-the-field sources (e.g. Haladyna & Dowling, 1989). For an
extensive treatment of this topic with many examples and checklists, see Nitko's
chapter 8 (2001).

 Determine how many total items you want. Considerations here include how
much material the test will cover, how deep the coverage needs to be, how
complex the items are, and how long students will have to take the test. [The
rule of thumb is one minute per question or possibly more if the items are
complicated (Gronlund, 1988; Oosterhof, 2001)].
 Use the Test Blueprint to determine how many items you need to write in each
cell.
 Avoid being overly general or overly specific with the content for each item.
This is somewhat dependent on your learning objectives, but you don't want to
be asking about broad, sweeping issues nor do you want to be asking about
minutia.
 Make sure each item tests one and only one concept. If the item is "double-
barreled", that is tests two or more concepts, you won't know which of the two
the student truly understands if she gets the item correct. (See Pros and Cons
for more on this issue.)
 Decide what item variations you're going to use (see below).

Rules for Writing Item Stems

 Either write the stem in the form of a question or, if a statement is in the stem
with its completion in the options, put the "blank" at the end of the stem, not in
the middle.

Poor: When looking at liquid in a test tube, the ____________ is the name of
the curved surface of the liquid.

Better: The curved surface of liquid in a test tube is called a .

(Answer: meniscus)

 Put the "main idea" of the item in the stem, not in the options.
 Streamline the stem to avoid extraneous language, but try to put as much of
the text in the stem leaving the options shorter.
 Avoid negatives like "except" or "not" or highlight them in italics, boldface or
underline if you use do use them.
 Highlight important words like "not", "only", "except" etc. if you use them at all.
Rules for Writing Options

 General Option Writing Tips

o Practically speaking, there's no "magic number" of options you should
use (Nitko, 2001). Write the number of options that make sense. It is
acceptable to mix and match the number of options. It's better to have a
three-option item than a four-option item with a poor distracter (Nitko,
2001).
o Options should be relatively equal in length. A much longer or much
shorter option can attract responses because they stand out visually.
o Make sure all options are grammatically congruent with the stem. For
example, the article “an” at the end of the stem will give a key that
starts with a vowel away.
o Put repeating words or phrases in the stem rather than in each option.
o Avoid overlapping options.

Poor: Water will be a liquid between ____________ and ____________

degrees centigrade.
a) 0, 100
b) -50; 0
c) 100; 150

(Note that a and b both include 0 and a and c both include 100 -- they overlap.)

Better: Water will be a liquid between ____________ and ____________

degrees centigrade.
a) 1; 99
b) -50; 0
c) 100; 150

(Note that there is now no overlap.)

o Avoid "all of the above." Its use muddles the interpretation of a

student's response (Nitko, 2001).

 Key-writing Tips
o Make sure the key is, in fact, correct according to the text and/ or
consensus in the field (Which, we hope, is also what's been taught in
class!).

 Distracter Writing Tips

o They're called distracters because they are strategically designed to
attract examinees who haven't completely mastered the content. This
isn't "tricky" or "deceptive" or "unfair". It stems from the premise that
the goal of testing is to find out who has learned the content and who
has not, perhaps along a continuum between the two. Students who
mastered the material should recognize the key and those who haven't
should not.
 o All distracters need to be plausible. We've all seen "Mickey Mouse" listed
as an option on tests, but that's seldom a believable possibility, and, in
terms of finding out who has mastered the material, a waste of ink.
o Use your knowledge of common student misunderstandings in writing
distracters. If you know, for example, that students often miss a step in
a calculation, include a distracter that would result from that
miscalculation. There are also many studies that have documented
common misconceptions in science concepts. Building these into
multiple-choice distracters is a possibility (Sadler, 1998).

 Avoid "linking" items where the answer to one item is found in or dependent on
another item. This is something to check for near final test assembly when you
are proofreading. It's very easy to do inadvertently when you're writing items
across several sessions.

Constructing the Test: Here are a few tips for assembling the test.

 Provide very clear, printed instructions and directions. If you're using a variety
of Multiple Choice formats, you may wish to write separate directions for each
section.
 Order the items by content, by item format, and then by increasing difficulty of
items (Gronlund, 1988). This rule is based on information processing principles.
It is easier mentally for students to answer all of the items about one content
before moving to another. They also perform similar mental tasks on similar
items before changing mental tasks with other formats. Finally, putting easy
items before hard items helps students gain some success early on.
 Randomize the ordering of keyed responses to overcome many of the rules of
thumb for guessing, like "always choose B or C". This also eliminates students
looking for or using patterns like AABBCCDD, ABADABA, etc.
 Make sure students don't get "lost" in the test because then their score doesn't
reflect what they know, but how well they navigated the test.
o Don't crowd items on the page.
o Avoid using double-sided pages because students may miss the back
page.
o Use navigational cues throughout like, "continue on the next page" or
"page 4 of 12".

Scoring the Test: You may use a full-credit model, where the student gets the item
correct or incorrect. You may also use a partial credit model, where the key receives
full credit and some distracters receive partial credit. More information on scoring
appears in the "Analysis" section.

Variations

This is actually one of the great strengths of multiple-choice items. There are several
good variations that make multiple choice a very flexible item format.

Correct Answer: The key is clearly right and the distracters are clearly wrong.
For example, computation items often fit this category.

The formula for calculating Work is:

 a) ½*m*v2
b) 1N*m
c) m*g*h
d) F*D*cos q

(Answer is a)

Best Answer: Here, the key is the best choice (most complete, most
comprehensive) while distracters, though "correct", may be incomplete.
What are the key elements that define Work in physics?
a) force and cause
b) force and displacement
c) force, cause and displacement

(Answer is C)

Interpretive Exercises: These rely on a stimulus material and consist of a multiple-

choice question or series of questions about the stimulus. Stimulus material can
consist of many things: text, a graph, a picture, a table, etc. This format is particularly
useful in tapping into several different ways of knowing quickly. Based on the same
stimulus material, you can ask a knowledge question, an application question and a
synthesis question, for example.

Problem A: You are asked to design a hoist system to move sacks of grain
from the entrance to a grain warehouse to a loft five meters above the
entrance. Each grain sack weighs 60 kg, and the warehouse operator wants to
move 20 bags per minute.

NOTE: Refer to Problem A when answering questions 1, 2 & 3.

1. What formula would you use to compute the amount of work required to
move one bag of grain from the ground floor to the loft?
a) ½*m*v2
b) 1N*m
c) m*g*h
d) F*D*cos q

(Answer is a)

2. How much work would be required to move the twenty bags of grain to
the loft?
a) 60000 Joules
b) 3000 Joules
c) 12000 Joules
d) 1000 Joules

(Answer is a)

3. If the time restraint were quickened to 30 seconds, how would power be

affected?
a) The power would double.
b) The power would be cut in half.
 c) The power would decrease, but by an indeterminate amount given the
present information.
d) The power would increase, but by an indeterminate amount given the
present information.

(Answer is a)

Analysis

There are essentially two kinds of analysis possible with multiple choice questions. The
first kind of analysis, scoring models, deals with scoring the test to determine student
achievement. The second kind of analysis, item analysis, deals with analyzing how well
each test item functioned.

Scoring Models: With multiple-choice questions, there are several potential ways to
score students' responses. Here are two:

 One is the full credit model where the response is either correct or incorrect.
The student's total score is then the sum of correct responses.
 A second is a partial credit model where some responses receive full credit and
others fractional credit. This works best with “best answer “questions or ones
where students may choose more than one answer. You can reward students
who pick a correct but not best answer, for example.

A final issue in scoring is the weighting of items. The easiest approach is to let each
item equal one point so all are equally weighted. However, based on the Test Blueprint
or other considerations, you may decide to prioritize some items over others and
weight them accordingly in computing the final score for the test.

Item Analysis: Once you have administered the test, it is possible to conduct several
analyses to ensure the quality of the questions for both this and future administrations
of the test. Many optical scanning systems common in university testing offices have
many of these options available for you automatically.

Item Difficulty—How hard or easy was the item for this group of students? This is
typically computed as the proportion of students who got the item correct. So a low
value means a hard question and a high value means an easy question on a scale from
0.00 to 1.00. It is best to have a mix of difficulties: some hard ones to challenge top
students; some easy ones so low-performing students will persist; and the bulk of
items at a moderate difficulty level. The average item difficulty across a test, for most
multiple-choice types, should be between 0.6 and 0.7 (Gronlund & Linn, 1990).

Item Discrimination—How well does the item differentiate among students who have
mastered the content and students who have not? This is calculated either as a
correlation coefficient between the item score and the total score or as a proportion of
high-scoring student who got the item right to low-scoring students who got the item
right. Either way, it is expressed on a scale from -1.00 to +1.00. Negative 1 means all
low scorers got the item right and all high scorers got the item wrong. Given that we
want students who have mastered the content getting each item correct, that's bad. A
positive 1 means the item worked exactly as it should. A zero means the item doesn't
distinguish between mastery and non-mastery students. That's also bad. There are a
number of statistical issues at work that cause +1.00 to be a rare occurrence, so that a
reasonable expectation for item discrimination indices is between 0.3 and 0.5
(Oosterhof, 2001). Again, a mix of different values in an exam is acceptable.

Distracter Analysis—this approach looks to see who is choosing each option for an
item. Usually the examinees are divided into low-scoring and high-scoring groups, and
the proportion of each choosing each option is reported. High-scoring students will
usually pick the correct response and low scoring students will usually pick a distracter.
If the opposite happens, that's a cue to revise the item. Something about a distracter
is attracting high performance students (Oosterhof, 2001).

One final statistic that is useful is the reliability coefficient. Reliability is the degree of
accuracy present in the score. For multiple-choice tests, this is indexed using a
reliability coefficient like Cronbach's Alpha or KR-21. Both range from 0.00 to 1.00 with
higher values indicating higher reliability. Though there are not strict cutoffs for
"acceptable" reliability coefficients because reliability is influenced by many diverse
factors, the consensus in the measurement field is that 0.60 or 0.70 would be an
acceptable lower value (Oosterhof, 2001). Having most item discrimination values at
or near .50, having a variety of item difficulties, and having more items will all increase
estimates of reliability (Gronlund, 1988; Nitko, 2001; Oosterhof, 2001). In essence,
you are calculating internal correlations of student responses between individual items.

These analyses help to identify items that are unfair, unclear, or poor in other ways.
This information can be used in a variety of ways.

 In the current administration of the test. You can use this information to discard
items in the current administration of the test, another state-of-the-art practice
(McDougall, 1997). I find students really respect the scores more when I can
explain to them that certain poor items or unfair items were removed based on
item analysis. For example, if I find many high-scoring students choosing a
certain distracter, that cues me to rethink whether it could be correct. Often I'll
give all students who chose that distracter credit also.
 In future administrations of the test. Reusing items is a very "cost-effective"
procedure. The overall quality of the test, however, will continue to increase if
you target poor items -- ones that are too easy or too difficult, that have zero
or negative discrimination, or ones with odd response patterns -- to be
reworked or discarded. I find it helpful when going over exams with classes or
individual students to have the item analyses handy so that I can listen for
reasons why those values came out as they did. Sometimes I'll even probe to
get explanations as to why someone chose a distracter.

Suggestions for Use

In addition to the varieties of item types, there is also variety of ways to use multiple-
choice questions and administer multiple-choice tests. Here are some ideas:

Multiple Choice Items and Student Learning

Diagnosing misconceptions through distracter analysis—If you've used common

student misconceptions or research-based misconceptions when writing distracters,
you can look to see how many students chose those distracters. As a simplistic
example, we know that in double digit addition, students will forget to "carry" values
from the ones column to the tens column. So we could make sure there's a distracter
that represents not carrying in several of our items. We can then look to see how
many students consistently choose that option. We may then choose to return to that
concept and re-teach it, or make other adjustments.

Item Writing as a Study Strategy —A good study strategy for students is to have
them write multiple-choice items. This is especially good if you also want to help
students learn strategies for taking multiple-choice tests. This puts them "behind the
scenes" and helps them identify material that might be on the test and how it might be
asked. It also is a great way for the instructor to gauge their depth of understanding
and what issues/ topics they are focusing on. This can be very useful, for example, if
you notice that the things they're focusing on aren't what you would focus on.

Share the Test Blueprint —I usually give my students my test blueprint as a study
guide. After all, it tells me what's important, what I want students to know, and how I
want them to know it. I use it to teach and to assess. Why can't they also use that
information (McDougall (1997)? This is yet another way to share goals and
expectations with students.

Mastery Learning Revisions—One approach to emphasize learning over "testing"

after the test is to offer students the opportunity to write about items they missed
(Jacobsen, 1993; Murray, 1990). For each item, students argue in writing why the key
isn't the right answer or why a distracter should be. Or they can write an explanation
of why the key is the best choice. They can earn partial credit back on each item for
which they make a valid argument. The advantages are that this avoids the emotional
pleas and "feeding frenzies" that occur when tests are returned. Students usually,
when forced to write about it, come to see where they went astray. Upon occasion, a
student will justify a distracter, something you can then adjust for the future. Students
appreciate being allowed to explain their answers (Jacobsen, 1993). The disadvantage
is that it usually requires releasing the items and keys to students, and you may not
wish to reuse those items in the future once they're "out".

Computers and Multiple Choice Items

Multiple choice testing owes much of its ubiquity to the invention of the optical scanner
in 1934, and technology has continued to play a role in multiple-choice developments.
It is no surprise, then, that computing has facilitated multiple choice testing. The major
technological applications are summarized below:

Item Banking—Item banking consists of storing items and information about items
(e.g. difficulty, discrimination, content codes, etc.) in electronic format. This allows
searching based on parameters, and tracking item characteristics over multiple
administrations. There are a variety of item banking software programs available (e.g.
FastTEST, C-Quest, Examiner, Exam Manager, MicroCat). This is the least complicated
technological application in testing, but it interfaces with others.

Computer Administered Tests—Here, students take tests on a computer. This

allows for multiple forms of a test to be automatically generated, administered, scored,
and recorded quickly and efficiently. Many colleges and universities have testing labs
where this can be accomplished. You can make sure no two students get exactly the
same test; students see their score immediately; you can arrange for the lab to
schedule and proctor the tests so that you don't use class time for testing. In short,
there are many advantages due to the automation the technology provides. However,
there may be issues with student anxiety: some software is restrictive in terms of
letting students return to answered items, changing responses, or viewing more than a
single question at a time.

Computer Adaptive Tests—Adaptive testing is not logistically feasible for most

college classes because test items need to be calibrated on, hopefully, hundreds of
examinees before the "adaptive" part can be done. In essence, each item is assigned a
difficulty rating based on the hundreds of responses, which is stored in the item bank.
Then, the computer administers a brief set of questions to get a preliminary estimate
of the student's knowledge. Based on that estimate, the computer adapts the test to
the student asking increasingly easier or harder questions in order to pinpoint the
student's ability. Once it has done so to a certain level of accuracy, the student
receives a score. The advantages are the accuracy and efficiency of this approach. One
of the disadvantages is that students do not receive the same test or even the same
number of items. Also, in only the very largest courses is item calibration feasible.
Again, software for this is available (e.g. BiLog, MultiLog, Parscale, ConQuest, Quest,
RASCAL). Trained psychometric staff would probably be necessary to do the
calibrating. Such expertise may be found in the university's testing center or in
educational psychology or psychology departments.

Internet Testing—Testing on the Internet is becoming more and more common. All
of the aforementioned techniques can be done in an Internet environment. At the
moment, the main disadvantage is security. There's no good, inexpensive way yet to
make sure that the person clicking the mouse is actually the student who is supposed
to be taking the test. In Internet banking, for example, the customer is part of the
security system. They have a password or PIN number, and it's in their interest not to
share that with others. In an academic testing situation, however, the "customer" or
student cannot be considered part of the security because it is in their interest to let
someone else have access to their PIN or password. So the Internet works well for
practice tests or low stakes assessments (e.g. ones that don't lead to grades) but is
likely not yet ready for high stakes assessment (ones on which grades are based).

Pros and Cons

Multiple choice testing brings an efficiency and economy of scale that can be
indispensable. Because students can respond to dozens of questions in a class period,
it allows broad coverage of content. Multiple-choice items are flexible enough to tap
nearly any level of Bloom's taxonomy. It also prevents students from trying anything
to get some points: either they know it or they don't. Of course, these advantages only
accrue when the items are well written and the test well constructed. That takes time,
planning, creativity, and thoughtfulness.

A key downfall and disappointment with multiple choice items is that we tend to write
items that are easy to write rather than items that take hard work to craft. This bias
causes the tests to focus on the recall of facts (knowledge & comprehension level) and
to leave out analysis, synthesis and evaluation questions, something that concerns
both faculty and students (Crooks, 1988; Shifflett, Phibbs, & Sage, 1997). But this is
only a drawback if we surrender to it. Just as we want to be thoughtful and explicit
about all aspects of our assessment (Crooks, 1988), we especially want to be
thoughtful about the items on our tests. Referring to the test blueprint while writing
items is an excellent way to avoid this pitfall (Gronlund & Linn, 1990).

Another criticism of multiple-choice items is that they decontextualize information, that

is, they pull knowledge -- often facts -- out of context (Shepard, 1989; Wiggins, 1989,
1992). Some argue that pulling information out of one's memory without some context
is an invalid assessment. This charge doesn't come with an easy defense. Given the
restricted nature of the format, it is often true that only a little bit of information like a
word problem or a short scenario can be provided (Shepard, 1989). Case studies,
portfolios, even essays are often considered more "authentic" or "real world" (Wiggins,
1989, 1992). Using interpretive exercises ameliorates this concern to some extent
because of the additional context the stimulus material provides. Some consider this
decontextualization an advantage, not a disadvantage, however. They argue that we
want to know whether the student knows the answer or not, not whether they can use
context to find the answer, etc. Authenticity in assessment actually falls on a
continuum. Any assessment can always be made more authentic or less authentic. One
could ask, "What is 2 +2?" or one could ask, "If Johnny has two apples and you have
two apples, how many apples do you have together?" or one could actually hand two
apples to one student and two apples to another and ask the question, and so on.

Through the analysis of distracters that have been carefully designed, it is possible to
gain insight into the misconceptions of students. However, simply because a student
answers correctly, it does not necessarily mean they have mastered the content
(Dufresne, Leonard, & Gerace, 2002). In other words, it is possible to answer correctly
for many reasons, including guessing, which results in “false positives”.. This issue has
led to a category of scoring models that correct for guessing (Gronlund & Linn, 1990),
which are used in some well-known testing programs like the Scholastic Assessment
Test (SAT). These techniques are not recommended for classroom use, however,
because they correct for blind guessing, not informed guessing, more common in
classroom tests (Gronlund & Linn, 1990).

A final complaint that needs careful consideration is the potential for language or
cultural bias in multiple-choice questions. Any assessment can be biased, so we should
always take care to avoid it. Following the item writing rules mitigates the potential for
bias. Eliminating extraneous verbiage and keeping the language simple and on the
students' level helps, and at least one study has shown that well-constructed items
don't preference different cognitive styles while poorly constructed ones do
(Armstrong, 1993).

Links

Kehoe, Jerard (1995). Writing multiple choice test items.

Practical Assessment, Research & Evaluation, 4(9).
http://ericae.net/pare/getvn.asp?v=4&n=9.

Frary, Robert B. (1995). More multiple choice item writing do's and don'ts.
Practical Assessment, Research & Evaluation, 4(11).
http://ericae.net/pare/getvn.asp?v=4&n=11.

Kehoe, Jerard (1995). Basic item analysis for multiple choice tests.
Practical Assessment, Research & Evaluation, 4(10).
http://ericae.net/pare/getvn.asp?v=4&n=10.
Kehoe, Jerard. Basic Item Analysis for Multiple choice Tests. -- "This Digest offers
some suggestions for the improvement of multiple choice tests using "item analysis"
statistics. These statistics are typically provided by a measurement services, where
tests are machine-scored, as well as by testing software packages."
http://www.robmccormack.com/qphelp/qphowto4.php

Teaching Effectiveness Program, University of Oregon. Writing Multiple choice

Questions that Demand Critical Thinking
http://darkwing.uoregon.edu/%7Etep/assessment/mc4critthink.php

Carneson, J., Delpierre, G. & Masters, K. Designing and Managing Multiple Choice
Questions
http://www.le.ac.uk/cc/ltg/castle/resources/mcqman/mcqman01.php

Office of Measurement Services, University of Minnesota.

http://www.ucs.umn.edu/oms/multchoice.phpx

Resources

Haladyna, T. M. (1999). Developing and validating multiple choice test items. 2nd Ed.
Mahwah, NJ: Lawrence Erlbaum Associates.

This text is primarily focused on large-scale testing, but much of it is still very
relevant to classroom testing.

Nitko, A. J. (2001). Educational assessment of students. 3rd Ed. Columbus, OH: Merrill
Prentice Hall.
http://vig.prenhall.com/catalog/academic/product/1,4096,0130137081,00.php

Though this book is for K-12 teachers, Nitko is unparalleled in his copious
examples and checklists when it comes to item writing. See especially chapter
8.

Gronlund, N. E., & Linn, R. L. (1990). Measurement and evaluation in teaching (6th
Ed.). New York: Macmillan.

This is a classic measurement text that expands on many of the issues

presented here. See especially chapters 7 & 8.

Oosterhof, A. (2001). Classroom applications of educational measurement (3rd Ed.).

Columbus, OH: Merrill Prentice Hall.

Another text written for K-12 teachers, it also has excellent information about a
broad range of topics relevant to multiple choice testing.

References

American Educational Research Association, American Psychological Association, &

National Council on Measurement in Education (1999). Standards for educational and
psychological testing. Washington, DC: AERA.
Armstrong, A. (1993). Cognitive-style difference in testing situations. Educational
Measurement: Issues and Practice, 12 (3), 17-22.

Brissenden & Slater (n.d.) Assessment Primer. Retrieved June 26, 2002 from the
Internet at http://www.flaguide.org/start/start.php

Crooks, T. J. (1988). The impact of classroom evaluation practices on students. Review

of Educational Research, 58(4), 438-481.

Dufresne, R. J., Leonard, W. J., & Gerace, W. J. (2002). Making sense of students'
answers to multiple-choice questions. The Physics Teacher, 40, 174-180.

Dwyer, C. A. (1993). "Innovation and reform: Examples from teacher assessment." In

R. E. Bennett & W. C. Ward (eds.) Construction versus choice in cognitive
measurement (pp. 265 - 289. Hillsdale, NJ: Lawrence Erlbaum Associates.

Gronlund, N. E. (1988). How to construct achievement tests (4th Ed.). Englewood

Cliffs, NJ: Prentice Hall.

Gronlund, N. E., & Linn, R. L. (1990). Measurement and evaluation in teaching (6th
Ed.). New York: Macmillan.

Guthrie, D. S. (1992). "Faculty goals and methods of instruction: Approaches to

classroom assessment." In Assessment and Curriculum Reform. New Directions for
Higher Education No. 80, 69-80. San Francisco: Jossey-Bass.

Haladyna, T. M. (1999). Developing and validating multiple choice test items (2nd
Ed.). Mahwah, NJ: Lawrence Erlbaum Associates.

Haladyna, T. M., & Dowling, S. M. (1989). Validity of a taxonomy of multiple-choice

item-writing rules. Applied Measurement in Education, 2(1), 51 - 78.

Hansen, J. D., & Dexter, L. (1997). Quality multiple-choice test questions: Item-
writing guidelines and an analysis of auditing test banks. Journal of Education for
Business, 73(2), 94-97.

Jacobsen, R. H. (1993). What is good testing?: Perceptions of college students. College

Teaching, 41(4), 153-156.

Lederhouse, J. E., & Lower, J. M. (1974). Testing college professor's tests. College
Student Journal, 8(1), 68-70.

McDougall, D. (1997). College faculty's use of objective tests: State-of-the-practice

versus state-of-the-art. Journal of Research and Development in Education, 30(3),
183-193.

Murray, J. P. (1990). Better testing for better learning. College Teaching, 38(4), 148-
152.

Nitko, A. J. (2001). Educational assessment of students. (3rd Ed.). Columbus, OH:

Merrill Prentice Hall.
Oosterhof, A. (2001). Classroom applications of educational measurement (3rd Ed.).
Columbus, OH: Merrill Prentice Hall.

Sadler, P. M. (1998). Psychometric models of student conceptions in science:

Reconciling qualitative studies and distracter-driven assessment instruments. Journal
of Research in Science Teaching, 35(3), 265-296.

Shifflett, B., Phibbs, K., & Sage, M. (1997). Attitudes toward collegiate level classroom
testing. Educational Research Quarterly, 21(1), 15-26.

Shepard, L. (1989). Why we need better assessments. Educational Leadership, 46 (7),

4-9.

Sims, R. L. (1997). Gender equity in management education: A content analysis of test

bank questions. Journal of Education for Business, 72 (5), 283-287.

Wiggins, G. (1989). A true test: Toward more authentic and equitable assessment. Phi
Delta Kappan, 70 (9) , 703-713.

Wiggins, G. (1992). Creating tests worth taking. Educational Leadership, 49(8), 26-33.

Assessment Strategies and Definitions

Book Response Journals

Similar to a learning log, the book response journal is a place for students to express
personal reactions and to wonder about events, themes, and ideas in a book. Children are
encouraged to react to everything they read. Teachers may use these journals to respond to
each child individually, sharing their questions, feelings, and ideas about literature and
making suggestions for further reading or related activities. Some teachers hold individual
reading conferences with their students and use these book response journals as part of the
conferences.

Comparison Charts

Comparison charts are one of a number of graphic organizers. They involve the
examinations of similarities and differences among ideas, events, characteristics, etc.
Comparison charts may take a number of forms and are an excellent way to engage
students individually or in groups as they seek to focus characters, events, or themes within
a single story or compare books, events, or properties within a given theme.

Conferences

There are many types of conferences including reading, writing, goal-setting, evaluation,
and coaching. The major purposes are to collaborate, assess, and guide.
Cooperative Learning Activities

Cooperative learning involves students working together in groups (often following a

teacher presented lesson), with group goals and individual accountability. Critical to the
process are two factors: 1) how to help another student without giving the answer; and 2)
how to work together toward a common goal.

Demonstrations

A demonstration transforms ideas into something concrete and observable through visual,
audio, art, drama, movement, and/or music. This could also include opportunities to
demonstrate and explain procedures and strategies such as a science experiment or a
solution to a non-routine math problem.

Discussion

A discussion provides a safe, open forum where children are encouraged to speak, listen,
and respond to opinions, feelings, and ideas regarding the designated topic.

Goal Setting

Setting goals with children provides the basis for monitoring student performance through
collaboration and self reflection.

Graffiti Walls

Graffiti walls are free form spaces for brainstorming or communicating words, phrases, or
ideas on a topic. These are often used as evolving records. A teacher may use them to
facilitate brainstorming on a particular theme at the beginning of a unit, as well as
encourage students to add new words or phrases relating to the theme as the unit
progresses. In addition to encouraging children to search for new and interesting words, the
graffiti wall serves as a class dictionary/thesaurus as students need novel words to enrich
their writing.

"I Learned" Statements

"I Learned" statements may be in either written or oral form. Their purpose is merely to
give students a chance to self-select one or more of the things they learned during a class
session, an investigation, or a series of lessons.

Interviews

An interview is structured or unstructured dialogue with students in which the student

reports his/her reaction or response to a single question or a series of questions. This
typically provides an opportunity for the teacher to determine the student's depth of
understanding rather than whether the student can provide the "correct" answer.
Questioning may follow a period of observation to discover if the student's perception of a
situation is the same as the observer's.

Investigations

Investigations may be related to a specific subject area or may involve several areas,
integrating curriculum. The most typical form of investigation is a collection of student
writing, diagrams, graphs, tables, charts, posters, experiments and other products. When
students become involved in practical or mathematic investigations, assessment activities
and/or questions can be presented to students without their awareness of any difference
between the assessment and instruction.

KWLs

A KWL is a technique used by teachers to assess what students "know," "wish to know,"
and "have learned about a particular topic," using a sheet divided into three columns
labeled K, W, L. At the beginning of a lesson, the KWL serves as a written record of the
students prior knowledge (K) on the topic, and allows the opportunity for the student to
note what they desire (W) to know about the topic. Following the lesson, the student can
self-assess what has actually been learned (L) about the topic.

Learning Logs

A learning log is a kind of journal that enables students to write across the curriculum. The
major reason for using them is to encourage children to be in control of their own learning
and to promote thinking through writing.

Oral Attitude Surveys

Attitude surveys note in a systematic manner students' self reflections regarding group and
individual performance and affective characteristics such as effort, values, and interest.
Providing an oral survey allows students to share their ideas, learn from others, and deepen
the way they think about the topics being discussed.

Oral Presentations

Oral presentations include speeches, storytelling, retellings, recitations, drama, videos,

debates, and oral interpretation and are evaluated according to a predetermined criteria.

Peer Evaluations

Peer evaluations consist of student analysis and assessment of peer proficiency using either
established or self-generated criteria. An activity must be very carefully structured if
students are to receive valid feedback from their peers.

Problem Solving Activities

In a problem solving activity, students must search for a means to find a solution, as well as
for a solution to the problem. A good evaluation of the problem solving activity requires
consideration of both the thinking process and the final product.

Products

Student products represent completed student work in a variety of forms; writing,

videotapes, audiotapes, computer demonstrations, dramatic performances, bulletin boards,
debates, etc. Students can demonstrate understanding, application, originality,
organizational skills, growth in social and academic skills and attitudes, and success in
meeting other criteria.

Response Groups

Response groups are opportunities for small numbers of children to discuss books or events
in depth with one another. Often these groups are organized and run by children themselves
because they all have read the same book or experienced the same event and want to
discuss it. Teachers participating in a response group will gain insight into their students'
thinking skills, group behaviors, and affective characteristics.

Self-Evaluations

A key concept in alternative assessment is having the student learn to recognize his/her own
progress by taking the time to reflect. Those who are able to review their own performance,
explain the reasons for choosing the processes they used, and identify the next step, develop
insight and self-involvement. Self-reflection, an important concept in any form of
assessment, is a particularly important component of a student portfolio

101 THINGS YOU CAN DO

THE FIRST THREE WEEKS OF CLASS
By Joyce T. Povlacs
Teaching and Learning Center, University of Nebraska-Lincoln

Introduction

Beginnings are important. Whether the class is a large introductory course for
freshmen or an advanced course in the major field, it makes good sense to start the
semester off well. Students will decide very early - some say the first day of class -
whether they will like the course, its contents, the teacher, and their fellow students.

The following list of "101 Things You Can Do..." is offered in the spirit of starting
off right. It is a catalog of suggestions for college teachers who are looking for a fresh
way of creating the best possible environment for learning. Not just the first day, but
the first three weeks of a course are especially important, studies say, in retaining
capable students. Even if the syllabus is printed and lecture notes are ready to go in
August, most college teachers can usually make adjustments in teaching methods as
the course unfolds and the characteristics of their students become known.

These suggestions have been gathered from UNL professors and from college
teachers elsewhere. The rationale for these methods is based on the following needs:
1) to help students make the transition from high school and summer or holiday
activities to learning in college; 2) to direct students' attention to the immediate
situation for learning - the hour in the classroom: 3) to spark intellectual curiosity - to
challenge students; 4) to support beginners and neophytes in the process of learning
in the discipline; S) to encourage the students' active involvement in learning; and 6)
to build a sense of community in the classroom.
Ideas For the First Three Weeks

Here, then, are some ideas for college teachers for use in their courses as they begin a
new semester.
Helping Students Make Transitions

1. Hit the ground running on the first day of class with substantial content.
2. Take attendance: roll call, clipboard, sign in, seating chart.
3. Introduce teaching assistants by slide, short presentation, or self-introduction.
4. Hand out an informative, artistic, and user-friendly syllabus.
5. Give an assignment on the first day to be collected at the next meeting.
6. Start laboratory experiments and other exercises the first time lab meets.
7. Call attention (written and oral) to what makes good lab practice: completing
work to be done, procedures, equipment, clean up, maintenance, safety,
conservation of supplies, full use of lab time.
8. Administer a learning style inventory to help students find out about
themselves.
9. Direct students to the Learning Skills Center for help on basic skills.
10. Tell students how much time they will need to study for this course.
11. Hand out supplemental study aids: library use, study tips, supplemental
readings and exercises.
12. Explain how to study for kind of tests you give.
13. Put in writing a limited number of ground rules regarding absence, late work,
testing procedures, grading, and general decorum, and maintain these.
14. Announce office hours frequently and hold them without fail.
15. Show students how to handle learning in large classes and impersonal
situations.
16. Give sample test questions.
17. Give sample test question answers.
18. Explain the difference between legitimate collaboration and academic
dishonesty; be clear when collaboration is wanted and when it is forbidden.
19. Seek out a different student each day and get to know something about him or
her.
20. Ask students to write about what important things are currently going on in
their lives.
21. Find out about students' jobs; if they are working, how many hours a week,
and what kinds of jobs they hold.

Directing Students' Attention

22. Greet students at the door when they enter the classroom.
23. Start the class on time.
24. Make a grand stage entrance to hush a large class and gain attention.
25. Give a pre-test on the day's topic.
26. Start the lecture with a puzzle, question, paradox, picture, or cartoon on slide
or transparency to focus on the day's topic.
27. Elicit student questions and concerns at the beginning of the class and list
these on the chalkboard to be answered during the hour.
28. Have students write down what they think the important issues or key points
of the day's lecture will be.
29. Ask the person who is reading the student newspaper what is in the news
today.

Challenging Students

30. Have students write out their expectations for the course and their own goals
for learning.
31. Use variety in methods of presentation every class meeting.
32. Stage a figurative "coffee break" about twenty minutes into the hour; tell an
anecdote, invite students to put down pens and pencils, refer to a current
event, shift media.
33. Incorporate community resources: plays, concerts, the State Fair. government
agencies. businesses, the outdoors.
34. Show a film in a novel way: stop it for discussion, show a few frames only,
anticipate ending, hand out a viewing or critique sheet, play and replay parts.
35. Share your philosophy of teaching with your students.
36. Form a student panel to present alternative views of the same concept.
37. Stage a change-your-mind debate. with students moving to different parts of
the classroom to signal change in opinion during the discussion.
38. Conduct a "living" demographic survey by having students move to different
parts of the classroom: size of high school. rural vs. urban. consumer
preferences...
39. Tell about your current research interests and how you got there from your
own beginnings in the discipline.
40. Conduct a role-play to make a point or to lay out issues.
41. Let your students assume the role of a professional in the discipline:
philosopher, literary critic, biologist. agronomist. political scientist. engineer.
42. Conduct idea-generating or brainstorming sessions to expand horizons.
43. Give students two passages of material containing alternative views to
compare and contrast.
44. Distribute a list of the unsolved problems. dilemmas. or great questions in
your discipline and invite students to claim one as their own to investigate.
45. Ask students what books they've read recently.
46. Ask what is going on in the state legislature on this subject which may affect
their future.
47. Let your students see the enthusiasm you have for your subject and your love
of learning.
48. Take students with you to hear guest speakers or special programs on campus.
49. Plan "scholar-gypsy" lesson or unit which shows students the excitement of
discovery in your discipline.

Providing Support

50. Collect students' current telephone numbers and addresses and let them know
that you may need to reach them.
51. Check out absentees. Call or write a personal note.
52. Diagnose the students' prerequisites learning by questionnaire or pre-test ant
give them the feedback as soon as possible.
53. Hand out study questions or study guides.
54. Be redundant. Students should hear, read. or see key material at least three
times.
55. Allow students to demonstrate progress in learning: summary quiz over the
day's work. a written reaction to the day's material.
56. Use non-graded feedback to let students know how they are doing: post
answers to ungraded quizzes and problem sets, exercises in class, oral
feedback.
57. Reward behavior you want: praise, stars, honor roll, personal note.
58. Use a light touch: smile, tell a good joke, break test anxiety with a
sympathetic comment.
59. Organize. Give visible structure by posting the day's "menu" on chalk- board
or overhead.
60. Use multiple media: overhead, slides, film, videotape, audio tape, models,
sample material.
61. Use multiple examples, in multiple media. to illustrate key points and .
important concepts.
62. Make appointments with all students (individually or in small groups).
63. Hand out wallet-sized telephone cards with all important telephone numbers
listed: office department, resource centers, teaching assistant, lab.
64. Print all important course dates on a card that can be handed out and taped to
a mirror.
65. Eavesdrop on students before or after class and join their conversation about
 course topics.
66. Maintain an open lab gradebook. with grades kept current. during lab time so
that students can check their progress.
67. Check to see if any students are having problems with any academic or
campus matters and direct those who are to appropriate offices or resources.
68. Tell students what they need to do to receive an "A" in your course.
69. Stop the work to find out what your students are thinking feeling and doing in
their everyday lives.

Encouraging Active Learning

70. Have students write something.

71. Have students keep three-week-three-times-a-week journals in which they
comment. ask questions. and answer questions about course topics.
72. Invite students to critique each other's essays or short answer on tests for
readability or content.
73. Invite students to ask questions and wait for the response.
74. Probe student responses to questions ant wait for the response.
75. Put students into pairs or "learning cells" to quiz each other over material for
the day.
76. Give students an opportunity to voice opinions about the subject matter.
77. Have students apply subject matter to solve real problems.
78. Give students red, yellow, and green cards (mate of posterboard) and
periodically call for a vote on an issue by asking for a simultaneous show of
cards.
79. Roam the aisles of a large classroom and carry on running conversations with
students as they work on course problems (a portable microphone helps).
80. Ask a question directed to one student and wait for an answer.
81. Place a suggestion box in the rear of the room and encourage students to
make written comments every time the class meets.
82. Do oral show of-hands multiple choice tests for summary review and instant
feedback.
83. Use task groups to accomplish specific objectives.
84. Grade quizzes and exercises in class as a learning tool.
85. Give students plenty of opportunity for practice before a major test.
86. Give a test early in the semester and return it graded in the next class meeting.
87. Have students write questions on index cards to be collected and answered the
next class period.
88. Make collaborate assignments for several students to work on together.
89. Assign written paraphrases and summaries of difficult reading.
90. Give students a take-home problem relating to the days lecture.
91. Encourage students to bring current news items to class which relate to the
subject matter and post these on a bulletin board nearby.

Building Community
 92. Learn names. Everyone makes an effort to learn at least a few names.
93. Set up a buddy system so students can contact each other about assignments
and coursework.
94. Find out about your students via questions on an index card.
95. Take pictures of students (snapshots in small groups, mug shots) and post in
classroom, office, or lab.
96. Arrange helping trios of students to assist each other in learning and growing.
97. Form small groups for getting acquainted; mix and form new groups several
times.
98. Assign a team project early in the semester and provide time to assemble the
team.
99. Help students form study groups to operate outside the classroom.
100. Solicit suggestions from students for outside resources and guest
speakers on course topics.

Feedback on Teaching

101. Gather student feedback in the first three weeks of the semester to
improve teaching and learning.

FIRST DAY OF CLASS: WHAT CAN/SHOULD WE DO?

By L. Dee Fink.
Reprinted with permission of the University of Oklahoma Instructional Development Program, July 19, 1999.

What can we do on the first day of class? What should we do?

One common answer is simply to start lecturing: "This is day one, here is lecture one,
away we go." Another possibility is: "Here is the syllabus, go buy your books and we
will see you at the next scheduled class period." Neither of these two options seems
desirable. But what are some other possibilities?

Several years ago a group of professors at the University of Oklahoma visited each
other on the first day of class and then discussed what they saw each other doing. But
the discussion quickly went from what they observed, to "What might be done?"
They eventually identified nine attractive possibilities, as described below. A
teacher should not feel obliged to do all of these, but doing even one or several of
them on the first day (or during the first week) would seem to accomplish a number
of important tasks for getting a class started in the right way.

1. Involve students quickly.

This can be done in a variety of ways:

o having them introduce themselves

 o allowing them to think and write silently
o having a whole-class or a small-group discussion, etc.

But letting students know right from the outset that they will be active
participants seems like a good approach.

2. Identify the value and importance of the subject.

Not all students come to all classes with a clear idea of why this subject is
important. The teacher may need to help them understand the significance of
the course. The sooner this is done, the sooner the students will be ready to
invest time and energy in the task of learning the subject matter.

3. Set expectations.

This can involve such things as what the teacher considers appropriate
amounts of study time and homework for the class, the importance of turning
homework in on time, expectations about in-class behavior, how the teacher
wants to relate to students, and how much interaction among students is
desired. The first day also offers an opportunity to find out what expectations
the students have of the teacher and of the class.

4. Establish rapport.

Almost any class will be more enjoyable for both the teacher and the students
if they know each other a bit. This exchange can be started with introductions,
sharing some background information, etc.

5. Reveal something about yourself.

Sometimes students can relate to the teacher more productively if they can see
him or her as a human being, i.e., as something more than just an authority
figure or subject matter expert. Sharing personal stories and being able to
laugh at yourself can help this process.

6. Establish your own credibility.

Sometimes this happens automatically, but at other times students need to

know about the teacher's prior work experience, travel experience, or research
and publications in an area. Having this knowledge can help students gain
confidence that the "teacher knows what she or he is talking about."

7. Establish the "climate" for the class.

Different teachers prefer different classroom climates: intense, relaxed,

formal, personal, humorous, serious, etc. Whatever climate you want, you
 should try to establish this early and set the tone for the rest of the semester.

8. Provide administrative information.

This often takes the form of going through the syllabus, presuming you have a
syllabus with this information in it: what reading material the students will
need; what kind of homework will be involved; what your office hours are;
where your office is located; how the class grade will be determined; what
your policies are regarding attendance, late papers, make-up exams, etc.

9. Introduce the subject matter.

Generally this introduction will be facilitated by starting with some kind of

overview of the subject.

o What is it?
o What are the parts of the subject?
o How is it connected to other kinds of knowledge?

Final Note:

Remember that it is imperative that you do on the first day whatever it is you want
the class to do the rest of the semester. If you want them to discuss, discuss on the
first day. If you want them to work in small groups, find something for them to do in
small groups on the first day.
Ten Things to Make the First Day (and the Rest) of the Semester
Successful
By Mary C. Clement, Berry College, GA
As published in The Teaching Professor, Volume 21, Number 7, August/September 2007

I like to arrive in the classroom well before the students. It gives me time to get
things organized. I create an entrance table (I use chairs or desks if there's no table)
that holds handouts for students to pick up. From day one the students learn the
routine: they arrive, pick up handouts on the entrance table, and read the screen for
instructions. They know what to do, and it saves time. Here's how I recommend
introducing the routine on day one.

1. Post your name and the name and section of the class on the screen, so that
when students walk in they know that they are in the right place.
2. Write: "welcome" on the screen and have directions that tell students what
they need to do immediately. Example: "As you enter, please tell me your
name. Then pick up a syllabus, a card, and a folder from the entrance table.
Fold the card so that it will stand on your desk, and write your first name on it
 in BIG letters. Add your last name and major in smaller print. Write your
name on the tab of the folder, (last name first, then first name). Read the
syllabus until class starts." Note: By asking students to tell you their name as
they enter, you can hear how the name is pronounced, and avoid the
embarrassment of pronouncing it for the first time yourself.
3. When it's time for class to start - start class! Late arrivals can catch up by
reading the screen.
4. For classes of 25 or less, I have students do brief, 10-second introductions. I
tell them there will be a verbal quiz after all the introductions and that they
can win stars if they know who is who. (Have fun with this, but remember
that these are adults and college is not like junior high.)
5. For larger classes, I have students introduce themselves to three or four
people around them, and then we might do "stand-ups" - stand up if you are a
Spanish major, stand up if you are an education major, and so on. I explain
that students need to know each other for our small group work, and in case
they have a question.
6. I collect the file folders and put them alphabetically by student name into a
big plastic carrying case. When students need to turn in assignments, they
find the box on the entrance table and they put their papers in their respective
folders. When papers are graded, they can pull their graded tests or
assignments from their folders. The beauty of this system is that time is never
wasted by passing out papers. For small classes, I put handouts in the folders
of absent students.
7. After the introductions and the explanation of the folder and box system, I
turn to the "Today we will" list that I've written on the board, posted on a
large paper flip-chart, or projected on the screen. I like to actually write this
list on the board, so I can return to it even while projecting my notes. A
"today we will" list outlines my plan for the day. For example, for the first
day, my "today we will list" says:
o See screen for instruction for card and folder.
o Introductions
o Turn in folders
o Go over syllabus completely
o Mini-lecture on _____________
o Interest inventory
o Do you know what to read/do before the next class?

Note: The "today we will" list lets me walk around the room, teach from the
projection system, and then look at the list for what I should do next. I tend
not to forget things if I have the list. As the semester progresses, the "today
we will" list might contain warm-up questions that then appear as test
questions. The list helps students who arrive late or leave early see what they
have missed.]

8. The mini-lesson/mini-lecture - whether it's a short overview of the first

reading assignment, some sample problems, or 10 interesting questions
 students will be able to answer at the end of the course, I strongly recommend
doing some course content on the first day. For classes that last longer than 50
minutes, I include a short student activity. I also think it's important to begin
with course material on day one so that students begin to see who you are and
how you teach. Since I teach courses in teacher education, I often talk about
my teaching career. I include a few stories about how times have changed and
about how some things in teaching never change.
9. Interest inventories are great for the first day of class. An interest inventory is
just a short list of questions about students' backgrounds and interests. It may
assess their prior learning as well. In addition to name and major, students can
write about a hobby, interest, or goal. Do not be too personal. You can have
them answer several questions about content - maybe solve a problem, write a
short paragraph or answer specific questions. Finally open-ended questions
are useful:
o What are your goals after graduation?
o What has a teacher done in the past that helped you to learn ______?
o Is there anything else that you want me to know about you and your
course of study?

You can always add one fun question:

o If your song played when you entered the room, what would that song
be?
10. Every good class has an introduction, a body, and a conclusion. I usually
teach the mini-lesson, and then save the last six to eight minutes of class for
the interest inventory and individual questions. This way, students don't have
to wait on others to finish. I instruct students to turn in their interest inventory
as they exit. As they are writing, I alphabetize their folders and put them in
the box on the table. Another good closure is to ask if they know what to
read/do before the next class, and if they know three people to ask about the
assignment if they have a question.

BREAKICER GAMES
« Fear in a Hat

Icebreaker Questions »

Candy Introductions

Candy Introductions is a get-to-know-you game that helps people learn new facts about
each other in an easy way.  They select various pieces of candy from a bag, and each candy
variety is associated with a fact about themselves which they will introduce to the others.
This game also goes by other names, including the M&M game, Candy Confessions, the
Skittles Game, the Gum Drop game, among others.

Candy Introductions can work with any group size.  The icebreaker works best when the
group size is limited to 12, so if you have more than 12, divide the larger group and run the
icebreaker within the smaller sized groups.  This icebreaker works best indoors, and is well
suited for classrooms or meeting rooms.  Materials required are: candy with about five
different variations (color or candy type), and an optional chalkboard/whiteboard.

Setup for Candy Introductions

Purchase several variety packs of candy, enough for each person to be able to have at least
five pieces.  They can be any candy type, but not too many choices (limit it to around five
or six different varieties).  Alternatively, you can buy gummy bears, life savers, gum drops,
skittles, m&ms, or any other candy that already has a variety of colors.

Instructions for How to Play 

Pass around the candy and tell each participant to choose anywhere from 1 to 5 pieces of
anything that they want.  Instruct them not to eat it yet, though.  After they have chosen
their candy, you will tell them what each candy type/color represents.

If there is a whiteboard or chalkboard present, write on the board the following:

 Red – Favorite hobbies

 Green – Favorite place on earth
 Blue – Favorite memory
 Yellow -Dream job
 Orange – Wildcard (tell us anything about yourself!)

If you don’t have the above colors, change the above to match the candy types that you
have.  Each person takes turns introducing himself or herself, beginning with their name
and then saying one fact for each candy type that they have.  This easy introduction game
should go relatively quickly (assuming they weren’t greedy and that they didn’t take too
many pieces of candy!)

Defend the Egg

Defend the Egg (also known as the Great Egg Drop) is a teambuilding activity that involves
collaboration, problem solving, and creative teamwork.  Groups build a structure out of
ordinary materials and try to protect a raw egg from breaking when dropped from a high
elevation.

This exercise in teambuilding can be messy, so choose an appropriate setting where making
a mess is acceptable.  The recommended group size is: teams of four or five people. 
Several materials are needed: raw eggs, several plastic straws, masking tape, newspaper,
and other materials of your choice.  This activity is for people ages 14 and up.

Setup for Defend the Egg

This game works well with teams that are not too large.  A good team size is four or five
people.  Pass out one egg and a limited supply of materials (e.g. four straws, a three foot
strip of tape, one section of a newspaper, etc.)  This activity is more challenging with less
materials provided, so decide how challenging you wish to make it.

Gameplay for Defend the Egg

Explain the rules: the mission is to protect the egg from cracking using teamwork,
creativity, and a good design.  You will drop each structure at least fifteen feet, and so the
goal is for each structure to be able to withstand such a fall.  Each team will only be given
limited resources, and so they must be wise with what they have.  They may not use any
other resources other than what is given to them.  Optionally, you can have other critera for
judging including:

 most creative design

 most stylish/visually appealing
 (any other awards you wish)

Decide on an appropriate amount of time (e.g. 20-25 minutes) and then instruct them to
begin!  Tell them to place their egg inside their structure. 

Be sure to supervise each team as they build their structure. 

When time is up, collect all the structures.  Now is dramatic finale in which the structures
are dropped (or thrown!) from at least 15 feet in elevation and then carefully inspected to
see if the eggs survived. The winners are the groups that successfully protected the egg.  If
you chose to have other awards, announce those winners also. 

This activity is useful to illustrate the importance of teamwork.  Ask everyone to reflect on
how their group accomplished the task, what worked, what was challenging, etc.

Variation

A much more challenging variation of this activity is to provide no materials (other than the
egg) and ask the participants to find materials from outdoors.

Did You Know? Bingo

Did You Know? Bingo (also known as the Autograph Game) is an icebreaker that helps
people learn interesting facts about each other.  People walk around the room and mingle
until they find people that match the facts listed on a bingo-style sheet.
This game is a get-to-know-you style icebreaker.  The recommended group size is: large or
extra large.  The game works best with a group of about 25 people.  It can be played
indoors or outdoors.  Materials required are: printed bingo sheets and pens.  Ages 12 and
up.

Setup for Did You Know? Bingo

The objective of this game is for people to wander around the room and to obtain the
signatures of people who have the facts listed on the bingo sheet.  Once a person
successfully obtains a full row (5 in a row), whether horizontally, vertically, or diagonally,
he or she shouts “BINGO!” and wins.

This game requires a little bit of setup.  Prepare a 5 by 5 table, with interesting facts written
inside the boxes.  These facts can include funny or bizarre things.  For example:

 Likes anchovies
 Has been to Hawaii
 Speaks more than two languages
 Has never been on a plane
 Has more than four brothers
 Has gone without a shower for more than three days

Be creative!  You can mark the center square “FREE SPACE” like traditional bingo
games.  After you have prepared the table, print out enough copies for the number of
players you are expecting.

Instructions for How to Play

Pass out a sheet to each person, along with a pen.  Explain the objective of the game and
the following rules: (1) each person you talk to may only sign your sheet once, and (2) to
win, you must get signatures to form 5 in a row horizonally, vertically, or diagonally.  Say
“Go!” and ask your participants to begin.

Once someone shouts “Bingo!” everyone returns and the person must introduce the people
who signed his or her sheet.  If desired, you can ask each person to explain their fact. This
icebreaker game is fun way to get to know humorous or unique facts about people.  Enjoy!

Icebreaker Questions

Icebreaker Questions is simply a list of 20 great questions that you can ask people to help
them feel more part of a group or team.  These questions are fun and non-threatening.  You
can use them as an icebreaker for meetings or classrooms, written on notecards and adapted
for other games, or simply as a fun activity to help people get to know each other better.

Instructions for Icebreaker Questions

A great way to help people open up is to ask them fun questions that allow them to express
their personality or interesting things about them.  Here is a list of twenty safe, useful
icebreaker questions to help break the ice:

1. If you could have an endless supply of any food, what would you get?
2. If you were an animal, what would you be and why?
3. What is one goal you’d like to accomplish during your lifetime?
4. When you were little, who was your favorite super hero and why?
5. Who is your hero? (a parent, a celebrity, an influential person in one’s life)
6. What’s your favorite thing to do in the summer?
7. If they made a movie of your life, what would it be about and which actor would you want
to play you?
8. If you were an ice cream flavor, which one would you be and why?
9. What’s your favorite cartoon character, and why?
10.  If you could visit any place in the world, where would you choose to go and why
11. What’s the ideal dream job for you?
12. Are you a morning or night person?
13. What are your favorite hobbies?
14. What are your pet peeves or interesting things about you that you dislike?
15. What’s the weirdest thing you’ve ever eaten?
16. Name one of your favorite things about someone in your family.
17. Tell us about a unique or quirky habit of yours.
18. If you had to describe yourself using three words, it would be…
19. If someone made a movie of your life would it be a drama, a comedy, a romantic-comedy,
action film, or science fiction?
20. If I could be anybody besides myself, I would be…
21. If you were a comic strip character, who would you be and why?
22. What thought or message would you want to put in a fortune cookie?
23. If you had to give up a favorite food, which would be the most difficult to give up?
24. What is one food you’d never want to taste again?
25. If you won a lottery ticket and had a million dollars, what would you do with it?
26. You’ve been given access to a time machine.  Where and when would you travel
to?
27. If you could be any superhero and have super powers, which one would you like to
have and why?
28. Mount Rushmore honors four U.S. presidents: Washington, Jefferson, Lincoln, and
Roosevelt.  If yuo could add any person to Mount Rushmore, who would you add
and why?
29. What award would you love to win and for what achievement?
30. If you could transport yourself anywhere instantly, where would you go and why?
31. In your opinion, which animal is the best (or most beautiful) and why?
32. What is one item that you really should throw away, but probably never will?
33. Growing up, what were your favorite toys to play with as a child? 
Superlative Game

The Superlative Game is a simple icebreaker that asks players to line up in ascending order
for various different categories (e.g. height, birthday month, etc.).  It’s very easy to learn
and play, and it doesn’t require a lot of time.

This game can be classified as a get-to-know-you style icebreaker.  The recommended

group size is medium, large, or extra large groups.  Players will be divided into multiple
teams of at least five people.  This game can be played both indoors or outdoors.  There are
no special materials required to play.  It works great with all ages.

Setup for the Superlative Game

The facilitator of the Superlative game needs to prepare a list of categories.  These
categories can be surface-level categories such as height (in ascending order), birthday
month (in ascending order, from January to December), shoe size, number of siblings (least
to most), etc. or you can also make deeper categories, depending on your goals.

Playing the Superlative Game

Split the group into at least three teams.  The ideal team size is at least five players per
team, but preferably no more than nine.  Explain the rules and consider using one of the
variations below (such as the no talking rule). 

Read the first category aloud, such as “Line up by increasing order of height.  Go!”  Each
team scrambles to get in the proper order.  When a team thinks they are done, they must all
sit down and raise their hands.  At this point, the facilitator checks the team and verifies
that they are in the proper order.  If they made a mistake, they get awarded zero points that
round and the facilitator checks the next group that sat down.  The first team to correctly
get in order and sits down gets awarded a point. 

The facilitator then reads off the next category, and this process repeats until the game is
over (no more categories).

Variations

There are many variations to this game that are worth considering.  These include:

1. No talking allowed.  All players must rely on body language and hand gestures to get in
the proper order.

2. Head and feet only.  You can only use your head and feet to signal where to go.  No
talking or use of arms/hands to communicate.
 Aligning Learning Activities and Assessment Strategies in the ESL
Classroom
Robert F. Tambini
robu_sensei [at] yahoo.com
Centenary College (New Jersey, USA)

Introduction
Accurately gauging the extent to which student progress meets the goals that have been set is one
of the most difficult tasks which educators must perform. In order to do this effectively, it is
essential that instructors design their lesson plans in such a way that expectations placed upon
students regarding performance standards, instructional objectives and learning activities
identified for use in the classroom, and assessment strategies employed in evaluating student
learning are well-aligned with one another.

As a means of demonstrating the form that this alignment might take, I will discuss this
alignment within the context of a lesson plan which I have used in since 1996. The class for
which this lesson plan was developed was (and remains) part of Centenary College's spring
Intensive English Language Program (IELP), a six-week program offered each year to
students of Centenary's sister college in Tokyo, Japan. The original class for which this
lesson plan was designed consisted of twenty 19 year-old female Japanese students at
Obirin Junior College. This lesson plan has been subsequently used in both the spring and
summer Intensive English Language Programs at Centenary College.

This discussion will begin with a presentation of the standards which informed the design
of the lesson plan, and a description of the performance standards set out in the plan which
students are expected to meet. The lesson plan itself follows. Finally, I will attempt to form
a conclusion addressing the success with which the various elements of the plan have been
aligned.It is important to note, and I wish to emphasize, that the particular lesson plan under
discussion here is an actual lesson plan which has seen three years use in the classroom, not
a hypothetical one constructed solely for the purpose of serving as the object of this study.

The Standards
Any English as a Second Language exercise is founded upon a single standard against which
students will be measured: the relative character and quality of the English used by students as
compared to the character and quality of the English used by native speakers. Of course, when we
assign levels to second language (L2) students, we are judging them not only against the standard
set by native speakers, but also relative to a) the received academic standard, and b) other L2
students. These standards, English as used by native speakers and English as used by L2 students,
are built of a large number of components, including (but not limited to) the following:

1. Vocabulary - number and quality of items known; ability to use items correctly and
appropriately
2. Grammar - knowledge of morphology and syntax; ability to apply knowledge correctly and
appropriately
3. Oral production - the ability to create utterances comprehensible to native speakers
4. Listening comprehension - the ability to comprehend utterances produced by native
speakers
5. Written production - the ability to manipulate symbols (letters, characters) and produce
language structures comprehensible in a visual context, and to express feelings, concepts
and ideas in writing
6. Reading comprehension - the ability to comprehend language symbols and structures in a
visual or tactile context, and to understand feelings, concepts, and ideas expressed in
writing

Within each of these areas any number of items might be identified by the teacher as the object of
a specific lesson. The ultimate goal of ESL teaching is mastery of the English language; however,
prior to becoming fluent in their use of English, students must first acquire a clear understanding
of and proficiency in each of the areas outlined above.

About the Plan

This lesson plan was developed for use in the Intensive English Language Program, a six-week ESL
program offered annually during the spring and summer semesters at Centenary College in
Hackettstown, New Jersey. This program is attended by students from Obirin Junior College,
Tokyo, Japan. Daily, the structure of the program is as follows:

 English Language and Conversation (three hours)

 Lunch (one hour)
 Language Lab (two hours)
 American Culture (two hours)

After dinner, students are expected to study and complete homework for approximately three
hours. Upon completion of the IELP, students are awarded three transfer credits in English which
they may apply as an elective toward their Obirin degrees. The lesson plan which will be described
below is used in support of a unit on "apartment hunting" in the conversation class.

The Objective
The objective of this lesson is to actively engage students in the process of seeking out and making
inquiries regarding the availability and suitability of living space in as authentic a manner as
possible. In addition to this introduction to the apartment hunting process, students are required
to make actual contacts and inquiries based on their identification of available and potentially
desirable living spaces advertised in local newspapers. This requirement is intended to force
students, who otherwise tend to become overwhelmed by and shy at the prospect of interacting
with a native speaker, to use the English skills they possess in an authentic, real-life situation
which many of them are likely to encounter again.

Pursuant to this objective, students will demonstrate: a) proficiency in their knowledge and
use of English vocabulary related to housing; b) the ability to form appropriate wh-
questions intended to elicit information which will assist in determining the suitability of a
particular apartment; c) critical thinking skills, as evidenced in descriptions (written and
oral) of the reasoning which led to the ultimate acceptance or rejection of a particular
housing unit.

The Learning Activities

Various learning activities, designed to introduce students to the subject matter and to improve
their proficiency relative thereto, are employed in support of this lesson. They are:

 In-class presentation of vocabulary items, accompanied by teacher explanations and

pronunciation practice/drills
 An in-class game in which students are provided with descriptions of four available units
and asked to match fictional prospective tenants with the apartment best suited to them
 In-class review of newspaper classified advertisements
 Library work, in which students locate the classified section of a local newspaper and
identify an apartment which they feel might be suited to their current circumstances (re:
rent, location, etc.)
 In-class, teacher-guided composition of wh- questions to be used during a telephone
inquiry regarding the apartment
 As homework, students make actual inquiry of the landlord/agency using the questions
they have composed
 Students report results of inquiry and determination of suitability both orally in class and
to the teacher in writing

Evaluating Student Performance

The assessment strategies employed in evaluating student performance related to this lesson are
as varied as the tasks which students are assigned. However, in the interest of brevity and for the
purpose of this study, I will concentrate on the strategy used in only one area of the lesson - the
actual contact between student and landlord/agency. As mentioned above, students must report
the results of the inquiry with the landlord/agency offering the apartment for rent. This report
takes two forms: a) an in-class presentation during which students describe their experiences
interacting with a native speaker, as well as any information which they were able to gather
regarding the apartment and rental terms; b) a written description of the conversation with the
landlord/agency, the student's assessment of his own performance during the interview, and a
justification for the student's decision to accept or reject the apartment.

Assessing Oral Perfomance

Assessment of the oral component of the report is conducted through teacher observation of
student performance and teacher questioning during both the in-class presentation and a follow-
up, one-on-one student/teacher conference. Students are evaluated primarily on the quality and
appropriateness of the wh- questions which they have formulated, and on the effectiveness with
which they were able to elicit information from the landlord/agency. The standard against which
students are measured are the examples that were provided in class; additionally, students are
judged on the quality and depth of their reflection on their own work.

Assessing Written Perfomance

Similarly, student written performance is also measured against the standard set by examples
discussed in class. These examples, whether drawn from the textbook, provided to students in the
form of handouts, or presented by the teacher during class, remain available throughout the
course of the lesson as a means by which students might judge the progress and quality of their
work. In addition, students receive an in-class explanation regarding those aspects of their report
on which they will be evaluated and the teacher's expectations in these areas. The criteria for
evaluation include: spelling, punctuation, word choice, grammatical correctness, sentence
structure and variety, clarity, coherence, organization, and overall effectiveness.

Providing Feedback

Feedback is an essential component of this lesson, and is provided in any number of ways: in-class
teacher response, one-on-one student/teacher conferences, written comments, etc. Most
important, however, is the fact that during this lesson the lines of communication are not filtered
through a one-way valve. Of course, feedback is provided to the student by the teacher; but,
students are also encouraged and expected to evaluate the quality and appropriateness of the
services they receive, and to offer constructive criticisms relative thereto. For example, the one-
on-one student/teacher conferences are not intended merely as opportunities for the teacher to
"preach from the mountaintop," but are rather meant to provide students with the chance to offer
input as to how their learning might be better facilitated.

Conclusion

Clearly, the alignment of performance standards, instructional objectives, and assessment

strategies is necessary if student learning and progress are to be evaluated accurately. I hope that,
through the presentation and discussion of the lesson plan presented above, I have been able to
demonstrate at least some of the manifold forms that this alignment might take. During the design
this lesson plan, the alignment of standards, objectives, and strategies was a matter of primary
concern. I believe that this alignment - insofar as is possible within the context of a single lesson
plan - has been successfully achieved: students are given the opportunity to learn, the teacher is
provided effective means to gauge that learning, and communication between student and
teacher is encouraged and supported.