See

discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/274577986

High Level Multiple Choice Questions in

Advanced Psychology Modules

Article in Psychology Learning and Teaching · September 2009

DOI: 10.2304/plat.2009.8.2.30

CITATION READS

1 436

3 authors:

Clare Harley Catriona Morrison

University of Leeds University of Bradford
37 PUBLICATIONS 669 CITATIONS 52 PUBLICATIONS 2,623 CITATIONS

SEE PROFILE SEE PROFILE

Richard M Wilkie
University of Leeds
102 PUBLICATIONS 967 CITATIONS

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Driving Rehabilitation View project

Understanding human steering control View project

All content following this page was uploaded by Clare Harley on 28 October 2015.

The user has requested enhancement of the downloaded file.

Psychology Learning and Teaching 8(2), 30–36

Reports
High-level multiple choice questions in advanced psychology modules
Richard M. Wilkie 1, Clare Harley and Catriona Morrison 
University of Leeds, UK

Traditional approaches to assessing students assume that multiple choice questions (MCQs) are
adequate in assessing only basic, low-level knowledge at the early stages of the higher education
(HE) curriculum. Increasingly, however, teachers of HE across a variety of subject areas are keen
to explore the opportunities for developing higher-level MCQ formats for assessing more advanced
stages of the curriculum. This has many benefits: students are unable to question-spot and are
required to demonstrate a breadth as well as a depth of knowledge, tests can be administered
electronically, and because feedback and marking can be instantaneous feedback can be quicker
than in a traditional paper-and-pencil assessment, without an onerous marking load for staff. Here
we report the use of high-level MCQs (hMCQ) in Level 2 of our BSc Psychology programme. We
demonstrate the success of this format in differentiating between students, and highlight important
factors in designing questions. We argue that this type of examination format offers an assessment
that discriminates between students and which can be simply evaluated to ensure there is a
suitable fit between the questions that make up the assessment tool and the student population
under evaluation.

Introduction are widely regarded as being able only to measure

Multiple choice questions (MCQs) have become
increasingly popular as the assessment method of
choice for psychology courses in UK HE institutions.
This position has been driven by a desire to ensure
students cover the full curriculum in their exam revision,
while avoiding an overly onerous task for staff in terms
of marking. Where test banks need to be set up, the staff
workload is fairly intensive; however, once those test
banks exist they can be used and reused, sampled from
year after year, with departments building up useful
statistics on which questions best distinguish the
knowledgeable and less knowledgeable students. In
most cases papers will be machine-read, and
increasingly, tools such as “Question Mark Perception”
are available to administer tests online that can offer
instant marks and even feedback on individual questions.
Because there is no marking load, this form of
examination gracefully scales to any size of student
cohort. This is highly appealing for courses where the
push to increase student numbers has resulted in
modules taking around 200 students.
For all their popularity, MCQs are not a panacea for
assessment problems. Their major drawback is that they
rudimentary, descriptive knowledge. In terms of Bloom’s
taxonomy (Bloom, 1956), MCQs are often considered to
test simple concrete abilities such as knowledge and
comprehension, but are not regarded as offering the
capacity to assess higher-level skills such as analysis,
synthesis or evaluation.
Recently this has been challenged, with exam setters
attempting more creative assessment methods. Higher-
level MCQs (hMCQs) use a format that requires students
to select an answer or answers from a list of choices, but
also requires students to engage in higher-level learning
processes like synthesis. For example, Connelly (2004)
reported the use of assertion reason question (ARQ)
examinations for use in a Masters of Business
Administration course. In this exam format students read
both assertion and reason statements and select from a
choice of answers combinations that state the statements
are either true or false. An example of an ARQ used in a
business school is shown in Table 1. Connelly showed
that ARQ tests were good substitutes for more
conventional MCQs/short-answer questions, and that
ARQ scores were a good predictor of student
performance in essays. Connelly concluded that MCQs
of this sort did tap deep learning and provided the

1
 Correspondence concerning this report should be addressed to the first author at: Institute of Psychological Sciences,
Faculty of Medicine and Health, University of Leeds, Leeds LS2 9JT, UK. Email: r.m.wilkie@leeds.ac.uk
30

Downloaded from plj.sagepub.com by guest on August 22, 2015

 Questions in advanced psychology modules

opportunity to assess high-level knowledge at advanced It should be made clear that the hMCQs used were a
stages of the HE curriculum. step beyond those used to examine factual
understanding. For example, at BSc Psychology Level 1
An additional criticism of MCQs has been that they an example of a MCQ worth one mark could be:
involve no active engagement on the part of the student
and the task is not considered “authentic”, “worthy” or MCQ. The retinal receptors linked to colour are the:
”intellectual” (Wiggins, 1990). Supporters of MCQs,
however, argue that high-level questions of the ARQ a. rods
type do require evaluation, synthesis and critical analysis b. cones
on the part of the student, and as such they are able to c. intermediate cells
achieve the highest level of assessment possible in d. ganglion cells
terms of Bloom’s taxonomy (Bloom, 1956).
In contrast one of our seven-mark questions has the
The Assessment Context following structure:
In a recent set of examinations we trialled a new format
of questions for the BSc Psychology Level 2 Perception hMCQ. Read the following abstract from <details of
and Language module. Originally the exam format had research paper> (worth 7 marks)
consisted of six short answer questions (SAQs) followed
by one long answer question (LAQ) (chosen from three <Abstract from research paper>
possible essay titles), with 60 minutes recommended for 1. <Assertion 1> and <Assertion 2>
each section. 2. <Assertion 3> and <Assertion 4>
3. <Assertion 5> and <Assertion 6>
The SAQs were deemed unsatisfactory, so after 4. <Assertion 7> and <Assertion 8>
exploring a range of formats we replaced the SAQs with
a set of 12 hMCQs. By doubling the number of questions Select the answer below (a–d) which correctly
we expanded the range of topics covered as well as identifies the statements above which are consistent
being able to introduce differing levels of complexity in with the information in the abstract AND your
questions. The time to complete each question was understanding of the subject area:
related to its complexity, therefore we weighted the
questions by assigning more marks to the more complex a. 1 and 2
questions. Twelve questions were designed that were b. 2 and 4
worth three, five or seven marks, with four questions in c. 1, 3 and 4
each mark group. In total 60 marks were available, and d. 2, 3 and 4
students were given 60 minutes to complete the section.
As with previous years, the second part of the exam was This style of question requires several stages of
to answer one long answer question, selected from a evaluation prior to answering. In the above example,
choice of three essay titles. students must comprehend the abstract in the context of

Table 12
Example ARQ question (adapted from Williams, 2006) where the answer requires a true/false assessment of the
‘assertion’, a true/false assessment of the ‘reason’, and if both are deemed true, then a final assessment must be made
to determine whether the ‘reason’ provides an accurate explanation of the ‘assertion’.

Assertion Reason
In a small, open economy, if the In a small, open economy, any
prevailing world price of a good is lower surplus in the domestic market will
than the domestic price, the quantity be absorbed by the rest of the world.
supplied by the domestic producer will BECAUSE
This increases domestic consumer
be greater than the domestic quantity surplus.
demanded, increasing domestic
producer surplus.

a. True; True; Correct reason c. True; False e. False; False

b. True; True; Incorrect reason d. False; True (The correct answer is d.)

2
 Adapted from Williams, G. (2006). Assertion-reason multiple-choice testing as a tool for deep learning: A qualitative
analysis. Assessment and Evaluation in Higher Education, 31(3), 287–301 and reprinted by permission of the publisher
(Taylor & Francis Group, http://www.informaworld.com).
31

Downloaded from plj.sagepub.com by guest on August 22, 2015

Wilkie, Harley and Morrison
their learning, evaluate the truthfulness of each assertion,
and decide which combination of assertions provides the
correct answer. As such this format of questioning
requires a level of analysis, synthesis and evaluation that
would not be assessed by the Level 1 MCQ format. We
decided not to apply negative marking, on the basis that
some answers contained both correct and incorrect
statements and therefore we could be penalising
students who had some, but not complete, understanding.
To evaluate the usefulness of the new exam format we
carried out a series of analyses to examine student
performance in the old SAQ exam and the new hMCQ
exam. Our prediction was that the hMCQs would be
better than the SAQs at discriminating between students
who had achieved different levels of learning, and as a
result we would see a wider distribution of marks for the
hMCQ exam. In addition we carried out a Rasch analysis
which would not only allow a direct comparison between
the performance of questions in the SAQ and hMCQ,
but also an evaluation of the difficulty of the questions
used. This allows an evaluation of the weightings applied
to the hMCQ questions.
Method
Data Collection: Participants and Procedure
The responses to the SAQs or hMCQs for a Visual
Perception and Language module were collected over
two consecutive years. The two hour exam comprised
two parts: In the first year, 152 students (92% single
honours, 81% female) completed the SAQ exam
including six SAQs. In the second year, 167 students
(89% single honours, 92% female) completed the hMCQ
exam that included 12 questions. For both sets of
students, the exam also required the student to complete
one LAQ. The overall module mark was an equally
weighted average of the LAQ and either the SAQ or
hMCQ score (see Table 2).
Data Analysis
Descriptive data were examined to determine the
distribution of exam grades for each method of
assessment. Rasch analysis methods were then applied
to examine SAQs and hMCQs across two different
Mean and standard deviation (SD) of exam marks (%) for the two exam periods3.
Mean SAQ or SD SAQ or
hMCQ hMCQ
Exam period 1: SAQ 55
Exam period 2: hMCQ 63
3
 SAQ = Short Answer Questions; LAQ = Long Answer Questions; hMCQ = high-level multiple-choice questions.
32
Downloaded from plj.sagepub.com by guest on August 22, 2015
years. Rasch analysis illustrates the comparative fit of
item location (question difficulty) against person
measures (student ability) along postulated latent traits
(e.g., ‘understanding of topic’). Therefore this analysis
method can be used as a tool for determining the source
of misfit between students and questions for improved
examination construction (Bhakta, Tennant, Horton,
Lawton, & Andrich, 2005). The Rasch model applies a
probabilistic unidimensional model asserting that: a)
easier questions are more likely to be answered correctly
by the students; and b) that the more able the student,
the more likely they are to answer the question correctly.
Student ability and question difficulty are estimated
jointly to produce parameter estimates called ‘logits’ (or
log-odds), which are independent of the student sample
and the question items included. The fit of each question
to the model can be assessed using the weighted mean
square (infit) and unweighted mean square (outfit)
statistics. The fit statistics have an expected value of 1.0
but can range between 0 to infinity; values indicating
good fit range between 0.7 and 1.3. Values greater than
1.3 can be interpreted as noise or lack of fit, whilst values
below 0.7 can be interpreted as redundancy or overlap
(Linacre, 2005).
In the current data set, student responses to each SAQ
were analysed as a rating scale using the exam score
for each question. The hMCQ responses were analysed
as dichotomous data, with the correct answer being
given a value of 1 and all other (incorrect) response
options assigned a value of 0.
Results and discussion
To evaluate the general student performance on each
exam we examined the distribution of grades for the
SAQ and the hMCQ exams (presented in Figure 1). It
can be seen that there was a greater spread of grades
awarded to students sitting the hMCQ than the SAQ.
The hMCQs extended the distribution of exam marks
across the higher and lower grades, using almost the full
range of possible scores (from 18% to 100%). This
indicates that the hMCQs may provide better
discrimination between students of different ability, since
exam grades using SAQs were clustered in a narrow
Table 2
Mean SD Overall
LAQ LAQ mark
9.4 60 9.1 58
17.7 61 10.1 62

Figure 1 The proportion of students achieving each
exam mark for the old SAQ, shown in white,
and the hMCQs, shown in black.
48
MCQ
42
SAQ
Proportion of Students (%)
36
30
24
18
12
6
0
0 20 40 60
Exam Mark (%)
boundary between high 2:2 and low 2:1 grades, with few
students managing to excel.
The results of the Rasch analysis allow us to determine
how well the individual questions on the SAQ and hMCQ
exams targeted the abilities of the students in each year.
Figure 2 shows the results of the Rasch analysis as a
‘person-item map’.
Both exams seem to target student ability reasonably
well, but the distributions of question difficulty and
student ability are narrower for the SAQ exam compared
to the hMCQ exam. The lack of variability of student
performance on the SAQ exam is reflected in the small
standard error values in Table 3 (embedded in Figure 2).
The narrowness of distribution for the SAQ means that
the exam questions were less able to assess a broad
range of student ability.
The SAQ questions LN1 and LN3 appear together on
the map (Figure 2), suggesting overlap between these
questions. Because LN3 had poor fit statistics it seems
that this question was redundant. The question LN2 also
had poor infit and outfit statistics, which indicates that it
contributed little to the Rasch model. The spread of the
hMCQ data was much broader than the SAQ data, and
Figure 2 shows there was no overlap of hMCQ questions.
Additionally, Table 3 confirms that all questions on the
hMCQ exam had good fit (with no overlap or redundancy)
and had greater variability (standard error) in student
responses than did the SAQ exam questions.
The hMCQs contained items that had been assigned
three levels of difficulty based on the predetermined
design of the questions. We used Rasch analysis to
Downloaded from plj.sagepub.com by guest on August 22, 2015
Questions in advanced psychology modules
determine whether the chosen weighting was
appropriate. In Figure 2 the questions are shown in the
actual order of difficulty, descending from most difficult
at the top and easiest at the bottom. The questions are
assigned a logit value based on the number of students
correctly answering each question and then rank
ordered. The predetermined weights assigned to each
question were: Questions 1 and 2 (VP1, VP2, LN1, LN2)
each received three marks; Questions 3 and 4 (VP3,
VP4, LN3, LN4) each received five marks; and Questions
5 and 6 (VP5, VP6, LN5, LN6) each received seven
marks. If this allocation of grades were accurate, it might
be expected that Questions 1 and 2 would appear at
the  bottom of the map, and Questions 3 and 4 in the
middle, whilst Questions 5 and 6 would appear at the top
of the map.
The distribution of hMCQ questions in Figure 2 shows
80 100 that the assumption of difficulty was not strongly linked
to student performance. Some questions were placed
appropriately – for example, VP2 and LN2 were only
worth three marks, are placed at the bottom of the map,
and were answered correctly by approximately 95% of
students. There were seven mark questions that were
clearly harder – for example, only 53% of students
correctly answered LN6 and VP5, which are much
higher on the map. There are also exceptions, however,
since only 45% answered VP1 correctly (worth three
marks) and this is near the top of the map, whilst LN5
(worth seven marks) was answered correctly by 79% of
students and lies in the middle of the map.
The original concept behind the hMCQ examination was
to design questions of varying difficulty, to target different
levels of student learning. For basic question items, only
retention of key facts is necessary, with little additional
understanding of the topic required, therefore it was
predicted that the majority of students would be
successful on these questions. Other questions that
targeted synthesis and analysis of more complex
materials were thought less likely to be answered
correctly by the students. It was anticipated that by using
a variety of questions, the hMCQ would better allow
discrimination between students who had achieved
different levels of learning, and the overall exam
performance would therefore reflect the level of learning.
The Rasch analysis revealed that some questions
were weighted particularly appropriately (e.g., VP2, LN4
and VP5) so we looked more closely to see how
students  performed in these ‘ideal’ questions (shown
in Figure 3). We grouped the responses to each question
on the basis of overall score on the test to see how
these  questions discriminated between students of
different abilities. The groups of students who
achieved  a  very good overall grade in the exam
performed at a similar level across each type of question
(achieving close to 100% in each), whereas the groups
33
Wilkie, Harley and Morrison

Figure 2 Person-item map. This figure illustrates exam question difficulty against student ability for each
assessment method, transformed to a logit scale. Students with better ability and the more difficult exam
questions are shown at the top of the map descending to students with lower ability and easier exam
questions at the bottom. The number of students at each point on the scale are represented cumulatively
by a series of symbols on the left side of each map: ‘#’ (3), ‘:’ (2) and ‘.’ (1); whereas exam questions are
positioned on the right of each map. The top left panel shows SAQ scores for the six exam questions
and the right panel shows hMCQ scores for the 12 multiple choice exam questions.

Short Answer Questions (SAQ) High-level Multiple Choice Questions (hMCQ)

Short
moreAnswer
able | Questions (SAQ)
more difficult High-levelmoreMultiple
able | Choice Questions (hMCQ)
more difficult
1 more :able
+ | more difficult 3 more able | .# more +difficult
1
.: :| + 3 .# +
.# |
.: |
.::.::T|T|
.# |
| | Number ofNumber of students:
students:
:: ::| | T| T| # = 3 # = 3
: :|T|T |T |T : = 2 : = 2
.:::| |
.::: | | . = 1 . = 1
::::::S|S| |
|
.:: |S VP3 |
.:: |S| VP3
.:::::: LN1 LN3 | | Questions:
.::::::
::::::| | LN1
LN2 LN3 | | Questions:
VP = Visual Perception
0 ::::::
.::::::| M+MLN2 2 .####### + | VP = Visual Perception
LN = Language
0 :::::::::
.:::::: M+M| 2 |
.####### +
::::: | VP2 S|
LN = Language
::::::::: | |
::::: |S |
::::: | | VP2
.::::::::: | VP4S|
:::::::::|S S| VP1 ########### | |
.::::::::: .:::| |T | | VP4
.:::
:::: S| | VP1 |S
########### |
.::: .|T| | VP1
|
T| |
-1 .::: | + 1 .############ + VP3 |S
less .able
| | less difficult M| VP5 | VP1
T| | LN3 |
-1 + 1 .############| + VP3
| LN6
less able | less difficult M| VP5
.######### |
| | LN3
| |
Table 3 | LN1 | LN6
.###### |
.######### |
Mean-square (MNSQ) fit statistics and variability for SAQ 0 S+M
and hMCQ exam questions |
| VP6
| |
Exam Infit Outfit Standard .#### | | LN1
.######
| |
question MNSQ MNSQ error 0 | S+M
| LN5
Short answer questions |
| VP6
|
VP1 1.22 1.24 0.6 .# | LN4
.####
T| |
VP2 1.19 1.20 0.6 -1 + |
| |
VP3 1.08 1.19 0.5 |
| LN5
LN1 0.86 0.92 0.5 |S
|
.# |
LN2 0.47 0.48 0.5 | .# | LN4
LN3 1.28 1.38 0.5 | T|
-1 | +
High-level multiple choice questions |
|
|
VP1 1.02 1.06 1.7 -2 + VP2 |
| |S
VP2 0.94 0.83 3.1 | .# |
VP3 1.02 1.01 1.7 | |
|
VP4 1.02 1.12 1.7 |
|
|T |
VP5 0.87 0.83 1.7 | |
VP6 0.86 0.84 1.8 | LN2 |
-2 | + VP2
LN1 1.00 1.00 1.7 -3 +
|
less able | less difficult
LN2 1.07 1.23 4.2 |
LN3 1.03 1.03 1.7 |
|
LN4 1.13 1.13 2.1 |
LN5 0.92 0.96 2.0 |T
|
LN6 1.08 1.15 1.7 | LN2
Note. Infit/Outfit statistics <.7 or >1.3 have been |
-3 +
emboldened to indicate that these questions have less able | less difficult
particularly poor fit.

34

Downloaded from plj.sagepub.com by guest on August 22, 2015


who achieved a lower overall grade contain fewer
students who answered the five mark and seven mark
questions correctly.
In Figure 3 the slopes of the lines indicate the
discrimination achieved by each question type, and it
is  clear that the gradient becomes steeper for the
questions worth more marks. Many of the students
(irrespective of their overall score) answered the three
mark question correctly, and this is appropriate given the
basic understanding required for such questions. We
also wanted to reward higher level skills, and some
questions performed very well at discriminating between
those who performed well at the test and those who
scored poorly. By using the forms of analysis outlined
here we can reflect on the relative difficulty of each
hMCQ item, and over time refine our questions to better
reflect a distribution of topics as well as depth of
understanding. This will allow us to increasingly
discriminate between students who have attained
different levels of knowledge and understanding, as well
as critical and analytical skills.
It is still not clear whether differentially weighted
questions (worth three, five or seven marks) are the best
way to reward question complexity, since this may have
had both positive and negative outcomes. Differential
weighting is a useful way of using a small number of
items to discriminate between students by awarding
Figure 3 The proportion of students who answered correctly a three mark, five mark and seven mark multiple
choice questions (VP2, LN4 and VP5 respectively.) The student’s data were split into five groups based
on their overall score for the whole hMCQ, from lowest (0–50%) to highest (81–100%) respectively.
100%
90%
Students who answered correctly (%)
80%
70%
60%
50%
40%
30%
20%
10%
0%
0–50
Downloaded from plj.sagepub.com by guest on August 22, 2015
Questions in advanced psychology modules
more marks for deeper understanding of topics. To
accurately estimate weightings, however, does require
good insight into how hard each question is for the
students, since misjudging the level of difficulty can
result in rewarding students for only superficial
understanding. In the current hMCQ design correctly
answering a seven mark question contributes 6% of the
overall exam mark, so such questions need to be pitched
correctly. Because of the uncertainty in estimating
question weighting, if an academic were considering
introducing a new hMCQ format we would recommend
reducing the contribution of hMCQ exam mark in the first
year of use until the performance of each question
(especially highly-weighted ones) can be confirmed. We
use a 50/50 split between the hMCQ and the LAQ in our
exam format, but in the first instance a 30/70 split may
be more prudent.
The differential marks are supposed to reflect the
amount of time spent on a question (rather than purely
as a measure of difficulty). More time taken on one
question would, however, reduce the amount spent on
other questions, effectively making them harder. Further
work is needed to evaluate how long students spent on
each question and how this relates to the marks made
available for that question.
The ultimate measure of whether implementation of the
new hMCQs has been a success is the opinion of the
VP2 (3 Marks)
LN4 (5 Marks)
VP5 (7 Marks)
51–60 61–70 71–80 81–100
Overall Score in MCQ
35
Wilkie, Harley and Morrison
teaching staff, the students, and – not least – the external
examiners. Needless to say teaching staff appreciate
the time saved as a result of not having to mark hundreds
hand-written exam essays, and although many initially
approached the exercise with reservations, we have
now implemented the use of hMCQs across the board at
Level 2, to staff approval. The students find the format
challenging but again the feedback has been very
positive in consultations via the Staff-Student Committee
and other means. Crucially, both external examiners
gave the new hMCQ exam format a ringing endorsement
and were impressed with the way in which this form of
exam discriminates between students.
Conclusion
We have shown that it is possible to generate a small
number of hMCQs that perform across a range of
student abilities, and effectively discriminate between
students with varying abilities. High-level MCQs are not
trivial to write, but we have demonstrated that a large
number of questions are not required in order for the
examination to be useful, as long as they tap high-level
skills. To make a reusable and validated hMCQ question
bank we recommend annual analysis of the performance
of individual questions to ensure a good spread of more
and less difficult questions which also discriminate
between students of varying abilities. Despite the time
costs associated with generating and assessing the
questions, once they are in place, they can remove the
heavy workload associated with marking large numbers
of essay-based questions.
36
Downloaded from plj.sagepub.com by guest on August 22, 2015
View publication stats
References
Bhakta, B., Tennant, A., Horton, M., Lawton, G., & Andrich,
D. (2005). Using item response theory to explore the
psychometric properties of extended matching questions
examination in undergraduate medical education. BMC
Medical Education, 5(1), 9.
Bloom, B. S. (Ed.). (1956). Taxonomy of educational
Objectives: The classification of educational goals:
Handbook I, cognitive domain. London: Longman Group.
Connelly, L. B. (2004). Assertion-reason assessment in
formative and summative tests: Results from two graduate
case studies. In R. Ottewill, E. Borredon, L. Falque, B.
Macfarlane, & A. Wall (Eds), Educational innovation in
economics and business VIII: Pedagogy, technology and
innovation (pp. 359–378). Dordrecht: Kluwer Academic
Publishers.
Linacre, J. M. (2005). A user’s guide to Winsteps/Ministeps
Rasch-Model programs. Chicago, IL: MESA Press.
Linacre, J. M. (2006). WINSTEPS. [Rasch measurement
computer program]. Chicago, IL: Winsteps.com
Wiggins, G. (1990). The case for authentic assessment.
ERIC Digest (ERIC Document Reproduction Service
No. ED328611).
Williams, J. (Retrieved 9 December, 2009 from http://
www.ericdigests.org/pre-9218/case.htm). Assertion-reason
multiple-choice testing as a tool for deep learning: A
qualitative analysis. Assessment and Evaluation in Higher
Education, 31(3), 287–301.
Manuscript received 22 September 2008.
Revision accepted for publication 6 May 2009.