Professional Documents
Culture Documents
Version 4.
Reference http://wiscinfo.doit.wisc.edu/teachingacademy/Assistance/course/questions.htm
The education strategy (March 2004) has highlighted that Bristol University:1. Is a research intensive university where the student learning experience is
informed by research and is delivered by research active staff.
2. Wishes to retain its high quality teaching.
3. Needs to widen access and to look at diverse progression routes into HE.
4. Needs to expand postgraduate numbers particularly on taught masters
programmes
5. To retain their highly rated international research output and to deliver high quality
teaching, requires that teaching is cost effective.
Whilst E-learning has a relevance to all of these points, this paper will focus on point 5 i.e.
the requirement for teaching to be cost effective. The contribution of E-learning to
assessment is a major area where substantial gains in efficiency can be achieved.
Currently staff spend a significant amount of time
1. setting examinations
2. ensuring that they match course learning objectives,
3. liasing with the external examiner
4. marking
5. exam related student administration
Much of this activity is repeated at least twice per year and sadly the outcome is often only
summative rather than formative. Several staff within the university have already adopted
CAA (computer aided assessment) as a means of addressing this problem and have
found pedagogic as well as efficiency gains. However this approach has not yet become
embedded. Reasons for this limited uptake include:1.
2.
3.
4.
5.
Knowledge
Comprehension
Application
Analysis
Synthesis
Evaluation
1
2
3
4
5
6
"Taxonomy" is simply a system of categorizing and organizing. In this case, the taxonomy
is hierarchical; each level is subsumed by the higher levels. In other words, a student
functioning at the "application" level has also mastered the material at the "knowledge"
and "comprehension" levels.
Before taking a closer look at the levels, it should be noted that Bloom's taxonomy is not
the only system of classification for cognitive domains. However, it is a widely accepted
system, and is useful in illustrating how certain question types are better choices for
assessing different levels of student mastery. What's important is the idea that student
knowledge is not all equal; there are levels of mastery ranging from simple recitation of
facts, to formulating informed opinions regarding complex issues. An awareness of these
levels can help one determine how well do students really know course content.
Learner Action
Recall content in the exact form that it was presented.
Memorisation of definitions, formulas, or procedures are
examples of knowledge-level functioning
Question Cues
List, define, label, identify, name
Comprehension
Application
Analysis
Synthesis
Evaluation
2. Matching Questions
a. Advantages:
Matching items can assess a large amount of information in a confined space on
the exam page, relative to multiple-choice questions; if developed carefully, the
probability of guessing is low. To decrease that probability further, avoid equal-sized
lists by including a few distractor items in the second (answer) column.
b.
c.
Assessing student understanding of related information. Examples of related items
include Symptoms and diagnosis, intervention and risks, terms and definitions,
tools and uses, and events and dates.
Multiplechoice
Short-answer
Calculation
Translation
Essay
Comprehensio
n
'Gobbets'
Problem
based
Blooms levels
Knowledge
Comprehensio
n
Knowledge
Comprehensio
n
Knowledge
Comprehensio
n
Application
Analysis
Knowledge
Comprehensio
n
Application
Analysis
Knowledge
Comprehensio
n
Application
Analysis
Knowledge,
Comprehensio
n, Application,
Synthesis
Knowledge
Comprehensio
n Application
Analysis
Synthesis
Evaluation
Knowledge,
Comprehensio
n, Application,
Analysis,
Synthesis,
Evaluation
Knowledge
Comprehensio
n Application
Analysis
Synthesis
Evaluation
6
"Simulation
testing"
"Performance"
Knowledge
Comprehensio
n Application
Analysis
Synthesis
Evaluation
Knowledge
Comprehensio
n Application
Analysis
Synthesis
Evaluation
a.
b.
c.
d.
3
5
8
10
4. Short-Answer Questions
a. Advantages:
Short-answer questions assess unassisted recall of information, rather than
recognition. Compared to essay questions, they are relatively easy to write.
b. Disadvantages:
Short-answer items are only suitable for questions that can be answered
with short responses. Additionally, because students are free to answer any
way they choose, short-answer questions can lead to difficulties in scoring if
8
the question is not worded carefully. Its important when writing short-answer
questions that the desired student response is clear.
c. Most Appropriate For:
assessing the correct spelling of items, or in cases when it is desirable to
ensure that the students have committed the information to memory
(medical students, for example, will require recall of information more than
recognition by the nature of their jobs).
4. Calculation Questions
a. Advantages:
For simple formula this is a very easy and obvious question type. It can
generate a large set of potential questions automatically so each test can be
different. Handling dimensions and marking that aspect separately is another
useful feature. Method also allows for inexact responses so making student
aware of imprecision.
b. Disadvantages:
For more complex formula the capabilities might be too limited. Only suitable
for formula based questions.
c. Most Appropriate For:
Scientific material based on formulae that have a single solution.
5. Essay Questions
a. Advantages:
Essay questions can effectively assess all six levels of Blooms Taxonomy.
They allow students to express their thoughts and opinions in writing,
granting a clearer picture of the level of student understanding. Finally, as
open-ended questions, they assess recall over recognition.
b. Disadvantages:
There are two main disadvantages to essay questions: time requirements
and grading consistency. Essays are time-consuming for students to
complete, and require careful instructions on the part of the test writer.
Scoring can be difficult because of the variety of answers, as well as the
halo effect (students rewarded for strong writing skills as opposed to
demonstrated mastery of the content). Students may also plagiarise
published work and present that as their essay. Currently available software
can be used to screen student essays to deter plagiarism.
c. Most appropriate for:
Assessment that cannot be accomplished with other question types.
Because essays are the only question types that can effectively assess the
highest levels of student mastery, they are the only option if the goal of
testing is the assessment of synthesis and evaluation levels.
Essay questions can be written for all of Bloom's taxonomy, but their true
advantage is the ability to assess the highest levels. Because essay
questions can be time-consuming for student to complete and difficult for
instructors to grade, this question type should be chosen carefully. If the
desired knowledge can be assessed with a different question type, it should
be considered. However, if higher-level processing is a goal of a course (as
it likely will be), well-developed essay questions are the key to accurate
9
assessment.
6. Problem based
a. Advantages :
Similar to an essay question can effectively assess all six levels of Blooms
Taxonomy. It is particularly suited to assessing how effectively students can
apply knowledge gained in a novel situation. The students ability to make
effective decisions is assessed and thus it is appropriate to Blooms level 5 &
6 (synthesis and evaluation). For example you can ask a question and the
next question depends upon the answer given to the former. The result is an
assessment based as much on the path taken through the questions as it is
on the final question answer. It can identify if a student can choose a good
strategy or not. You can get the student to enter into a dialogue with the
computer, and record the responses. Time stress can be applied which has
relevance in some areas.
b. Disadvantages :
Difficult questions to write, requiring detailed data/information to be
presented to the student in a structured way. The question is likely to need to
integrate information from several sources and have several pathways
open for the student to choose. (see figure 1)
c. Most appropriate for :
Suited to differential diagnosis of clinical cases. Presenting problem can be
presenting clinical symptoms, which requires provisional diagnosis (possibly
more than one). This needs to be refined by clinical examination and
laboratory tests (pathology, microbiology, biochemistry, histology, ECG,
ultrasound, X ray etc.) The diagnosis once refined may need further tests
before a conclusive diagnosis can be made. This assessment process is
highly relevant to the principles of differential diagnosis and evidence based
medicine.
Once a diagnosis has been made then a suitable treatment needs to be
decided upon and a prognosis made. Each of these steps can be assessed
and ranked in order of importance. Negative marking will be required if an
option is taken which would actually harm the patient.
10
d.
Figure 1 Outline of a problem based assessment
Presenting information
which defines the
problem
Provisional hypothesis
Modified hypothesis
Additional information
requested by the student
[To be self selected from
several sources, but which is
ranked in terms of
significance. If inappropriate
or excessive information is
requested then student looses
marks]
Decision
(several options)
Course of action
choice
11
7. "Simulation testing"
a. Advantages
Many things we teach actually are high level processes that must be learnt interpreting, translating, writing, manipulating, etc. Simulation of the actual
process can be a powerful tool for facilitating learning, and can be used for
both summative and formative assessment. However the assessment of
these processes can often be difficult and complex. The development of
virtual reality tools and E-learning provides an exciting opportunity to
overcome many of these problems and to be cost effective at the same time.
b. Disadvantages
Simulations can be costly to develop both in staff time and the cost of
appropriate equipment. Currently the aerospace and defence world uses
simulations (flight simulators) as part of both learning and assessment. But
in this very expensive example, it is worth building a simulation because the
cost of making a mistake on a real one is huge. This is not always (often) the
case. There are cheaper ways of simulating however, a good example being
the CD test currently used to test driving students hazard awareness skills.
Another example being developed at Glasgow University is an interactive
model used to test a students ability to pregnancy diagnose a cow.
c. Most appropriate for :
Testing students skills, which currently is often not done because of the
financial cost, demands upon staff time and safety implications.
8. "Performance"
a. Advantages
Almost by definition the performing arts require both formative and
summative assessment tools capable of making judgements about student
learning outcomes based upon a performance or a piece of creative work. Elearning tools such as video or image capture can play an important role in
storing the evidence of student outcomes (on which the assessment is
based) and in supporting formative feedback.
b. Disadvantages
There are few shortcuts for marking performance and elearning tools cannot
automatically mark work as they can for multiple choice type questions for
example. Performance marking can be seen as subjective and the marking
criteria needs to be properly thought out. Performances are ephemeral
events, so unless the performance is captured in some way, (for example
with video recording equipment) the performance is lost. If it is a group
performance, it can be difficult to mark individual students for what is
essentially a group effort.
c. Most appropriate for :
Performance tests practical application of theory and methodology, gauging
creative interpretation of source material. Performance can also test
communication skills, for example in medicine a students ability to
12
The University of Bristol/ILRT has been proactive both in developing itself and in buying in
technologies capable of delivering on-line assessment (see attached appendix 2).
Whatever the decision as to which tools will be chosen for the next step, it will be
important that they can be used to assess all 6 levels of Blooms taxonomy.
13
LEVEL 1: knowledge/recognition
Question: Newton's three famous laws of motion include these concepts (select three):
1.
2.
3.
4.
5.
6.
Speed
Velocity
Mass
Vector
Energy
Gravity
Question: For each of the following dates, match the event or person associated with it.
Question Items
Answer Items
1. 1066
2. 1100s
3. 1215
4. 1347
5. 1455
6. 1517
7. 1664
8. 1764
9. 1867
10. 1947
14
LEVEL 2: Comprehension
Question:
class is this drug?
1.
2.
3.
4.
5.
Indicate what
Alkyl phenol
Steroidal anaesthetic
Barbiturate
Non-steroidal analgesic
Muscle relaxant
15
LEVEL 3 : Application
Question: Indicate what is the effect of this suture pattern on the wound edges
i. Apposes edges
ii. Inverts edges
iii. Relieves tension at edges
Question:
Pet cats show hunting behaviour. According to control theory models of motivation
involving feedback mechanisms, if the functional consequence of this hunting behaviour
(food. is given to the cat, the behaviour should stop.
True or False
Level 3/4 Application/Analysis
The following data were obtained from an experiment in which four diets were fed to
growing rats. Three diets were formulated to contain protein (CP) entirely as casein,
fishmeal or kidney bean meal. The fourth diet was protein-free. Food intake, faecal and
urinary excretion were measured over 5 days. The following results were obtained.
1. casein
2. fishmeal
3. kidney bean
Incorrect Answer Feedback
Example of casein
16
Question: The following data were obtained from a digestibility trial with growing cattle weighing 300kg .
which equals
17
LEVEL 4 : Analysis
Question A sow is introduced to another sow on one occasion and shows no oestrous
response but, a few minutes later, shows a strong oestrous response when introduced to
a boar, even though the two are not allowed to mate. The same procedure is followed
again a week later and this time the sow shows no oestrous response to the boar. These
observations indicate that:
1.
2.
3.
4.
Question The graph below shows the relationship between levels of stereotypic circling
behaviour and baseline circulating corticosterone observed in 20 randomly selected singly
housed mice.
18
Hepatic failure
Chronic renal failure
Auto immune haemolytic anaemia
Parvovirus infection
Small intestinal foreign body
1. otters are highly motivated to swim and this motivation accumulates since they last
swam
19
20
Question In an experiment on learning, 10 rats were injected with lithium chloride 2h after
consuming their daily ration of familiar rat pellets. A further 10 rats were handled and given
a sham injection. The rats were then replaced in their individual home cages with ad
libitum familiar rat pellets and water. After 24 h they were given a choice between familiar
rat pellets, and pellets flavoured with a novel but palatable substance. What happens?
1. The lithium chloride stimulates the rats' appetite and the experimental rats eat
more of both types of pellets than control animals.
2. The lithium chloride induces a temporary sickness and a specific food aversion for
the familiar rat pellets is formed. In the test, only novel pellets are consumed.
3. The lithium chloride induces a temporary sickness but no specific food aversion
learning takes place because the pellets were familiar. Rats eat the familiar pellets
in the test.
4. No food aversion learning takes place, because the injection was given 2h after
food consumption. For learning to occur the injection would have had to be given
immediately after food consumption. Rats eat the familiar pellets in the test.
5. The lithium chloride induces a feeling of sickness that persists for 24h. In the test,
nothing is consumed.
21
LEVEL 6: Evaluation
This example is produced using CALnet and currently can only be used for instruction
since it does not include the facility for assessment. It is planned to provide this as an
add-on. Basically it consists of a problem solving exercise using clinical material in
this case a horse presenting with symptoms of Colic.
Stepwise the student needs to :
1. make a provisional diagnosis based upon the presenting symptoms
2. carry out a clinical examination by asking pertinent questions of the software.
In the assessed version (to be developed) students will be assessed on the
amount of irrelevant information they request.
Material that can be provided to the student can be quite diverse in its format
including, text, numeric data, audio, pictures and video.
3. make a diagnosis
4. Decide upon treatment
5. Assess likely outcome of the case.
The following screen shots give a flavour of how the case looks for a student.
22
23
24
The following are the answers provided in pop-up boxes when individual
treatments have been selected : Note some are appropriate, others would be bad
news for the case!
25
Analgesia
Analgesia is always indicated in colic to control pain and reduce self-inflicted injury
and exhaustion. The pain of impaction colic tends to be intermittent, so that
analgesics are usually given intermittently to effect.
Prokinetics
Drugs which promote peristalsis are contraindicated because in the presence of an
obstruction they tend to increase pain and, at worst, they may cause gut rupture.
Liquid Paraffin
Liquid paraffin is a mild laxative which softens the mass and lubricates the gut. It is
delivered by nasogastric tube in large volumes, twice daily if necessary, until the
mass starts to break down and liquid paraffin appears in the faeces. Because of its
viscosity it is easier to deliver a suspension of the oil in warm water or 0.9% saline
(50:50 volumes). This may be done under gravity using a funnel or by the cautious
use of a pump. Volume guide: pony 2-3L (total); horse 4-5L; large horse 6-7L.
Intravenous Fluids
Intravenous polyionic fluids may be useful in the first day or two of treatment to
ensure proper hydration of gut tissues. They are positively indicated in dehydrated
patients.
Anthelmintics
An optional treatment but it is unlikely that parasites are implicated here.
Nevertheless, the presence of strongyle larvae and tapeworms may interfere with
normal gut motility. The mare's history recounts use of a proprietary benzimidazole
every six weeks to control alimentary parasites, but at normal therapeutic doses
BMZs are not effective against migrating strongyle larvae or tapeworms. If
considering anthelmintics as part of colic therapy, use products which are effective
against strongyle larvae and tapeworms.
Antibiotics
If the impaction were associated with peritonitis, then antibiotics would be indicated.
However, they are not indicated in the clinical work up of this case.
Teeth rasping
The teeth should be examined as part of the clinical work up. Sharp points on the
cheek teeth can be associated with inadequate chewing of hard feed, which can
predispose or exacerbate impactions. Teeth rasping should be considered.
Pathological changes, particularly vascular compromise of the gut, is reflected in the
colour changes seen in peritoneal fluid. In this case: normal clear yellow fluid
26
27
29